Triggering Lambda function from SQS and store the message in S3 bucket

Lab Details

  1. This lab walks you through on how to create an SQS queue and send a message that will trigger a lambda function to store the message in S3 bucket.

  2. Duration: 45 minutes

  3. AWS Region: US East (N. Virginia)

Introduction

Amazon Simple Queue Service(SQS)

DefinitionFirst ever AWS service that was publicly available.

  • Amazon SQS is a reliable, easy to manage,  scalable queuing service. SQS is a simple and cost effective cloud application.

  • AWS SQS can be used to transmit any amount of data, at any level of throughput, without losing messages.It doesn’t interrupt the other services to run continuously.

  • SQS helps to reduce the admirative task by scaling high available messaging clusters. While we pay only for what we use. AWS SQS helps us to save important data which might be lost in case the entire application goes down or if any component becomes unavailable. 

  • Basically, SQS queue acts as a buffer between the application components that receive the data and the other parts that process the data in the system.

  • SQS are used for message-oriented architecture based application.Sometimes processing server cannot process the work fast enough (due to any possible reason) the work is queued so that the processing servers can work on it when they have available resources to process the request.This means that work is not lost due to insufficient resources.

  • Amazon SQS ensures that each message is delivered at least once.

There are two types of Queue:

 
 
 
 
 
 
 
 

A Simple Use Case

  • Consider an example of major flash sale on E-commerce where people buy and sell a wide range of products. Now the requirement is all the requests with respect from buyer and seller should be processed in the order that they have received in the queue. To fulfill this requirement, we’ll be using the FIFO queue for making transactions in one flow.

  • A mobile company is holding a flash sale for their new model with great features at the best price. It is expected that a huge number of buyers will place their orders. The company is holding limited stock for a limited period, so it’s important to track the order that arrived first. Your flash sale receives huge responses and only the buyers who place the order first will receive the product and remaining users get to try in the next sale. Once the request is received, they are sent to a FIFO queue before they are processed.

  • Let’s understand how messages get in and out of the queue. Assume the consumer asks for a batch of up to 300 messages, AWS SQS starts filling the batch with the oldest message (REQ A1). Now SQS keeps filling the queue until the batch is full. In our case, assume batch contains only three requests and now the Queue is empty. Once the message batch has left from the queue SQS considers the batch to be In-flight until the batch gets processed completely and deletes or the visibility timeout gets expires.

 

 When you have a single consumer, this is easy to process. The consumer gets a batch of messages, does its processing and deletes the messages. Now the consumer is ready to process and take up the next batch of the messages.You can also add auto scaling group to scale your processing power depending upon the requirement.

Note: SQS won’t release the next batch of messages until the first batch has been deleted.

Tasks

  1. Log into the AWS Management Console.

  2. Create a S3 bucket.

  3. Create an IAM role for lambda.

  4. Create a lambda function.

  5. Create a SQS queue.

  6. Added Lambda trigger.

  7. Test the lab.

Architecture Diagram



Join Whizlabs_Hands-On to Read the Rest of this Lab..and More!

Step 1 : Login to My-Account
Step 2 : Click on "Access Now" to view the course you have purchased
Step 3 : You will be taken to our Learn Management Solution (LMS) to access your Labs,Quiz and Video courses