Lambda Function to Upload Csv File in S3 Python

AWS Lambda & S3| Automate CSV File Processing From S3 Bucket And Push button In DynamoDB Using Lambda [Python]

Subscribe to my newsletter and never miss my upcoming articles

In this blog we are going to pick CSV file from S3 bucket once it is created/uploaded, process the file and push information technology to DynamoDB table.

Create Role For Lambda

1. Create policy mentioned beneath.

                         {                        "Version":                        "2012-10-17",                        "Statement": [      {                        "Sid":                        "VisualEditor0",                        "Effect":                        "Allow",                        "Action": [                        "dynamodb:CreateTable",                        "s3:PutObject",                        "s3:GetObject",                        "dynamodb:PutItem",                        "dynamodb:UpdateItem",                        "dynamodb:UpdateTable"                        "logs:CreateLogDelivery",                        "logs:PutMetricFilter",                        "logs:CreateLogStream",                        "logs:GetLogRecord",                        "logs:DeleteLogGroup",                        "logs:GetLogEvents",                        "logs:FilterLogEvents",                        "logs:GetLogGroupFields",                        "logs:CreateLogGroup",                        "logs:DeleteLogStream",                        "logs:GetLogDelivery",                        "logs:PutLogEvents"                        ],                        "Resources":                        "*"                        }  ] }                                          

2. At present create new role for lambda and attach this policy to the function.

Create S3 Bucket And Attach Tags

Creates a new S3 saucepan. To create a bucket, you must annals with Amazon S3 and have a valid Amazon Web Services Access Cardinal ID to cosign requests. Bearding requests are never immune to create buckets. Past creating the bucket, you become the bucket owner.

1. Lets import boto3 module

                         import boto3                                          

2. We will invoke the client for S3

                         client = boto3.client('s3')                                          

3. At present we will employ input() to accept bucket proper name to exist create equally user input and volition store in variable "bucket_name".
   Note:- Make certain to check the saucepan naming rules here

                         bucket_name=str(input('Please input bucket name to be created: '))                                          

4. Goto link where you volition detect all arguments listing. Based on your requirement you can put this arguments to list your S3 buckets. This document also mentions datatype of the parameter.
   Note:-Saucepan Name statement is mandatory and bucket proper name should be unique

                         response1 = client.create_bucket(  ACL='public-read-write',  Bucket=bucket_name  )                                          

5. Now we will use input() to ostend if user wants to go ahead with saucepan tagging via user input and will shop it in variable "tag_resp".

                         tag_resp=str(input('Printing "y" if y'all want to tag your bucket?: '))                                          

6. Now we will use if status and take user input for tags which needs to be divers for bucket.
   Nosotros volition store tag key in variable "tag_key" and tag value in "tag_value". To add tag to bucket we are going to use put_bucket_tagging() method, make sure to cheque official documentation here In method parameters nosotros are passing variable as "bucket_name","tag_key","tag_value".

                         if tag_resp == 'y': tag_key=str(input("Delight enter key for the tag: ")) tag_value = str(input("Delight enter value for the tag: ")) response2 = client.put_bucket_tagging( Bucket=bucket_name, Tagging={     'TagSet': [         {             'Fundamental': tag_key,             'Value': tag_value         }     ] })                                          

   To view entire github lawmaking delight click hither

Create DynamoDB Tabular array

1. Python code in ane module gains access to the code in another module by the process of importing it. The import statement combines ii operations it searches for the named module, then it binds the results of that search to a name in the local scope.

                         import boto3                                          

2. We will invoke the resources for DyanamoDB.

                         dynamodb = boto3.resource('dynamodb')                                          

3. Nosotros will use create_table() function to create table in Dynamo DB with following arguments listed below. Here nosotros will run into ii examples one with "primary keys simply" and another with "chief key and sort cardinal". You lot can find official documentation here.
   Example1:- Beneath code is to create table with primary key merely

                         tabular array = dynamodb.create_table(  TableName='user',  KeySchema=[      {                        'AttributeName':                        'id',                        'KeyType':                        'HASH'                        #Partition                        Cardinal                        Only                        }  ],  AttributeDefinitions=[      {                        'AttributeName':                        'id',                        'AttributeType':                        'S'                        }  ],  ProvisionedThroughput={                        'ReadCapacityUnits':                        1,                        'WriteCapacityUnits':                        1                        }, )                                          

   You can find working lawmaking for instance in Git Repo here

Lambda Function To Read CSV File From S3 Bucket And Push Into DynamoDB Table

1. Goto Lambda console and click on create office
2. Select "Author From Scratch" , Function name = csv_s3_Lambda, Runtime= Python and function we created with above policy attached to this blog and click on create function.
3. Goto code editor and kickoff writing the code.
4. Nosotros will import 3 modules

                         import boto3                                          

5. We volition invoke the client for S3 and resource for dynamodb

                         s3_client = boto3.client('s3') dynamodb = boto3.resources('dynamodb')                                          

6. First we will fetch bucket name from event json object

                         def lambda_handler(event, context):  bucket = issue['Records'][0]['s3']['bucket']['name']                                          

7. Now we volition fetch file name which is uploaded in s3 bucket from event json object

                         def lambda_handler(result, context):  bucket = consequence['Records'][0]['s3']['bucket']['proper noun']  csv_file_name = event['Records'][0]['s3']['object']['central']                                          

8. We will call now get_object() function to Retrieves objects from Amazon S3. To apply Become , y'all must have READ access to the object. If you grant READ access to the bearding user, you can render the object without using an authorization header. You lot tin view this role official documentation hither

                         def lambda_handler(event, context):  bucket = event['Records'][0]['s3']['bucket']['name']  csv_file_name = event['Records'][0]['s3']['object']['key']  csv_object = s3_client.get_object(Bucket=bucket,Key=csv_file_name)                                          

9. Lets decode the json object returned by function which will return cord

                         def lambda_handler(event, context):  bucket = event['Records'][0]['s3']['bucket']['name']  csv_file_name = event['Records'][0]['s3']['object']['key']  csv_object = s3_client.get_object(Saucepan=bucket,Key=csv_file_proper noun)  file_reader = csv_object['Body'].read().decode("utf-eight")                                          

10. Use Split() which volition split up a string into a list where each word is a list particular. Make certain to cheque official documentation here

                          def lambda_handler(upshot, context): bucket = event['Records'][0]['s3']['saucepan']['proper noun'] csv_file_name = effect['Records'][0]['s3']['object']['fundamental'] csv_object = s3_client.get_object(Saucepan=saucepan,Cardinal=csv_file_name) file_reader = csv_object['Trunk'].read().decode("utf-8") users = file_reader.split up("\n")                                          

11. Nosotros will use filter() method which filters the given sequence with the help of a function that tests each element in the sequence to exist true or not.. Make sure to bank check official documentation hither

                          def lambda_handler(event, context): saucepan = result['Records'][0]['s3']['bucket']['name'] csv_file_name = effect['Records'][0]['s3']['object']['cardinal'] csv_object = s3_client.get_object(Saucepan=bucket,Fundamental=csv_file_name) file_reader = csv_object['Body'].read().decode("utf-eight") users = file_reader.separate("\northward") users = list(filter(None, users))                                          

12. Now nosotros will traverse through the list pick elements i past i and push information technology to dynamodb tabular array using tabular array.put_item() . You tin can observe official documentation here.

                          def lambda_handler(result, context): bucket = consequence['Records'][0]['s3']['bucket']['name'] csv_file_name = event['Records'][0]['s3']['object']['key'] csv_object = s3_client.get_object(Bucket=saucepan,Central=csv_file_proper noun) file_reader = csv_object['Trunk'].read().decode("utf-eight") users = file_reader.carve up("\n") users = list(filter(None, users)) for user in users:     user_data = user.split(",")                                                  table.put_item(Particular = {         "id" : user_data[0],         "proper name" : user_data[i],         "salary" : user_data[2]     }) render 'success'                                          

    To view entire github code please click here

Set Event For S3 bucket

1. Open Lambda function and click on add trigger
2. Select S3 as trigger target and select the bucket we have created above and select result blazon every bit "PUT" and add suffix equally ".csv" Click on Add together.

Create CSV File And Upload It To S3 Bucket

1. Create .csv file with beneath information

                                                 1,ABC,200                        ii,DEF,300                        3,XYZ,400                                          

2. Now upload this file to S3 bucket and it will process the information and push this data to DynamoDB.

Youtube Tutorial

Resources Cleanup

• Delete Lambda Function
• Delete DynamoDB Tabular array
• Delete S3 Bucket Object First And Then Bucket
• Delete Lambda Role

Conclusion

In this web log nosotros are going to pick CSV file from S3 saucepan once information technology is created, procedure the file and push information technology to DynamoDB table.

Stay tuned for my adjacent web log.....

And so, did you find my content helpful? If you did or like my other content, feel free to buy me a coffee. Thanks.

Did yous observe this article valuable?

Back up Dheeraj Choudhary by condign a sponsor. Any amount is appreciated!

robinsonansuchan1983.blogspot.com

Source: https://dheeraj3choudhary.com/aws-lambda-and-s3or-automate-csv-file-processing-from-s3-bucket-and-push-in-dynamodb-using-lambda-python

Share this post