Lab
A Cloud Guru

Performing Real-Time Data Analysis with Amazon Kinesis

Real-time streaming data underpins many modern businesses and solutions, and one of the most useful services for handling streaming data is Amazon Kinesis. In this lab, we will build a Kinesis Data Stream, develop both a producer and consumer of events, and also deliver them to a destination using Kinesis Data Firehose.

Try for free Contact sales

Path Info

Level

Advanced

Duration

1h 0m

Published

Nov 08, 2023

Challenge

Create an Amazon Kinesis Data Stream

Create a Kinesis Data Stream which we can use for the lab. You can use either On-Demand or Provisioned capacity mode, and any name of your choice.
Challenge

Develop a Lambda Function to produce streaming events

Our new Kinesis Data Stream will need to be receiving events before we can consume them, so we need to build a producer.

Build a Lambda Function using a runtime of your choice that will put records into the Kinesis Data Stream using the AWS SDK. Use the provided Lambda execution role containing the name KinesisLambdaRole to provide the necessary permissions. Tip: You can produce a burst of events by using a Lambda Function with a loop, and using the function timeout.

If you are following along with the video, you can find the Python code used in the video at our GitHub repository.
Challenge

Develop a Lambda Function to consume streaming events

Now we need to start consuming some of the events on our Kinesis Data Stream. In this case, we'll use another AWS Lambda function.

Build a Lambda Function using a runtime of your choice that will print the records into the function's log stream. Use the Kinesis Data Stream as a trigger, and experiment with the different settings for the Batch Size and Batch Window. Use the provided Lambda execution role containing the name KinesisLambdaRole to provide the necessary permissions.

If you are following along with the video, you can find the Python code used in the video at our GitHub repository.
Challenge

Setup Kinesis Data Firehose to send events to an S3 Bucket

Finally, we're going to ingest the records received by the Data Stream and load them into an Amazon S3 bucket

Create a new Kinesis Data Firehose delivery stream, using your existing Kinesis Data Stream as the source, and the provided S3 bucket with the name including firehosedeliverybucket as the destination. Observe the records being delivered both to our event consuming Lambda Function, and the Amazon S3 bucket by Kinesis Data Firehose.

Note: It can take some time for the objects to appear in the S3 bucket.

Author

A Cloud Guru

The Cloud Content team comprises subject matter experts hyper focused on services offered by the leading cloud vendors (AWS, GCP, and Azure), as well as cloud-related technologies such as Linux and DevOps. The team is thrilled to share their knowledge to help you build modern tech solutions from the ground up, secure and optimize your environments, and so much more!

What's a lab?

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.