Support Documents
×


Configuring DynamoDB Streams Using Lambda

Lab Details

  1. This lab walks you through the steps to launch an Amazon DynamoDB table, configuring DynamoDB Streams and trigger a Lambda function to dump the items in the table as a text file and then move the text file to an S3 bucket.

  2. You will practice it using Amazon DynamoDB, S3, and lambda.

  3. Duration: 1 hour

  4. AWS Region: US East (N. Virginia) us-east-1

Introduction

Amazon DynamoDB

  1. Amazon DynamoDB is a fully managed NoSQL database service where maintenance, administrative burden, operative and scaling are taken care of by AWS.

  2. We don't need to provide the specifications of how much we are going to save.

  3. It provides single-digit latency even for terabytes of data and is used for applications where very fast reads are required.

  4. It is used in applications like gaming where data needs to be captured and changes take place very quickly.

Amazon DynamoDB Streams

  1. Amazon DynamoDB stream is a feature that emits events when record modifications or changes occur in a DynamoDB table.

  2. DynamoDB streams captures the time level modification in the DynamoDB table in a time ordered sequence.

  3. When a DynamoDB stream is enabled, it captures the changes happening in the DynamoDB tables in an orderly manner.

  4. DynamoDB streams can be used to replicate the data from one DynamoDB Table of region 1 to another DynamoDB Table of region2.

  5. Events can be of the following types:

  • INSERT

  • UPDATE

  • REMOVE

  1. Each event carries the contents of the rows that are being modified.

  2. It records all the modifications through encrypted logs and stores them for 24 hours.

  3. Whenever there is a change, DynamoDB creates a Stream record along with the primary key attribute of the items.

  4. A DynamoDB streams consists of a stream record and an individual stream record, which represents a unique data modification in the DynamoDB table.

  5. Each stream record is assigned through sequence numbering, which gives us the exact order in which the modification occurred.

  6. Stream records are coupled into groups called shards. Each shard contains multiple stream records.

  7. The shard also contains information required to access the stream records.

  8. The stream records inside the shard will be time-limited with a lifetime of 24 hours and after that time the Stream records will be deleted automatically.

  9. The shard can be split into multiple shards if needed for processing the records parallelly with one parent shard and multiple child shards.

  10. The order of processing will preference the parent first, followed by child shards.

  11. Events are recorded in near real-time.   

  12. In real-time applications, we can access the stream records that contain events where changes take place.

  13. The DynamoDB Streams can be configured to catch additional information like capturing the image of the modified items before and after the modification takes place. 

  14. The DynamoDB stream records can be read and processed with the help of the DynamoDB Streams endpoint.

  15. Streams can be enabled while creating a DynamoDB table and can be disabled at any time.

  16. Performance of the table won't be affected by both enabling and disabling the Streams because DynamoDB streams operate asynchronously.

  17. Once the DynamoDB Stream is disabled, the data in the Stream will be available for 24 Hours. There is no methodology available for manually deleting the existing streams.

  18. DynamoDB streams can be used as an event source for Lamda functions so you can create applications that take actions based on the events in the DynamoDB table.

  19. The events that the DynamoDB captures can be analyzed by moving the data into other AWS services like S3 or Cloudwatch.

Architecture Diagram

  Task Details

  1. Log in to the AWS Management Console.

  2. Create Amazon DynamoDB table. 

  3. Insert items into the DynamoDB table.

  4. Create a Lambda function.

  5. Add Triggers to the DynamoDB table.

  6. Make changes to the contents of the DynamoDB table.  

  7. Streaming Cloudwatch Logs.

  8. Validation of the lab.