Recursion Pattern in AWS Step Function to process long-running task using AWS Lambda

In this video , I have explained a pattern to process long-running task by calling the AWS Lambda code recursively and using AWS Step Function to track the state and to implement the stopping condition of the recursion.

Prerequisite:
------------------------
Paginate Rest API via Offset and Limit method
   • Paginate Rest API via Offset and Limit method  
Accessing s3 Data in Spark outside the AWS account using hadoop-aws and aws-java-sdk packages
   • Accessing s3 Data in Spark outside the AWS...  
AWS Step Functions Simplified
   • AWS Step Functions Simplified  

Architecture:
-----------------------
https://github.com/SatadruMukherjee/D...


Lambda Function Code:
-----------------------------------------
import json
import requests
import uuid
import boto3
s3 = boto3.client('s3')
limit=1000


def lambda_handler(event, context):
print(event)
offset=0
body=event.get("body",0)
if body==0:
offset=0
else:
offset=event['body']["offset"]

isComplete=False
url="http://jsonplaceholder.typicode.com/p...{}&_limit={}".format(offset,limit)

print("URL for the current iteration : {}".format(url))

result=requests.get(url);
data_extracted=result.json();

length_of_no_of_records_extracted_now=len(data_extracted)

if(length_of_no_of_records_extracted_now!=0):
file_name="{Folder Name}/"+str(uuid.uuid4())+".json"
s3.put_object(
Body=json.dumps(data_extracted),
Bucket='{Bucket Name}',
Key=file_name
)

if(length_of_no_of_records_extracted_now==0):
isComplete=True

offset=offset+limit

response_json_set={'offset':offset,
'isComplete':isComplete
}




TODO implement
return {
'statusCode': 200,
'body': response_json_set
}

Step Function Code:
------------------------------------
{
"Comment": "A description of my state machine",
"StartAt": "Lambda Invoke",
"States": {
"Lambda Invoke": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke",
"OutputPath": "$.Payload",
"Parameters": {
"Payload.$": "$",
"FunctionName": "{Put the Lambda ARN here}"
},
"Retry": [
{
"ErrorEquals": [
"Lambda.ServiceException",
"Lambda.AWSLambdaException",
"Lambda.SdkClientException"
],
"IntervalSeconds": 2,
"MaxAttempts": 6,
"BackoffRate": 2
}
],
"Next": "Choice"
},
"Choice": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.body.isComplete",
"BooleanEquals": false,
"Next": "Lambda Invoke"
}
],
"Default": "Notify Developer team"
},
"Notify Developer team": {
"Type": "Pass",
"End": true
}
}
}

Check this playlist for more Data Engineering related videos:
   • Demystifying Data Engineering with Cloud C...  

🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL