- Lab
- A Cloud Guru
Performing Real-Time Data Analysis with Amazon Kinesis
Real-time streaming data underpins many modern businesses and solutions, and one of the most useful services for handling streaming data is Amazon Kinesis. In this lab, we will build a Kinesis Data Stream, develop both a producer and consumer of events, and also deliver them to a destination using Kinesis Data Firehose.
Path Info
Table of Contents
-
Challenge
Create an Amazon Kinesis Data Stream
Create a Kinesis Data Stream which we can use for the lab. You can use either On-Demand or Provisioned capacity mode, and any name of your choice.
-
Challenge
Develop a Lambda Function to produce streaming events
Our new Kinesis Data Stream will need to be receiving events before we can consume them, so we need to build a producer.
Build a Lambda Function using a runtime of your choice that will put records into the Kinesis Data Stream using the AWS SDK. Use the provided Lambda execution role containing the name
KinesisLambdaRoleto provide the necessary permissions. Tip: You can produce a burst of events by using a Lambda Function with a loop, and using the function timeout.If you are following along with the video, you can find the Python code used in the video at our GitHub repository.
-
Challenge
Develop a Lambda Function to consume streaming events
Now we need to start consuming some of the events on our Kinesis Data Stream. In this case, we'll use another AWS Lambda function.
Build a Lambda Function using a runtime of your choice that will print the records into the function's log stream. Use the Kinesis Data Stream as a trigger, and experiment with the different settings for the
Batch SizeandBatch Window. Use the provided Lambda execution role containing the nameKinesisLambdaRoleto provide the necessary permissions.If you are following along with the video, you can find the Python code used in the video at our GitHub repository.
-
Challenge
Setup Kinesis Data Firehose to send events to an S3 Bucket
Finally, we're going to ingest the records received by the Data Stream and load them into an Amazon S3 bucket
Create a new Kinesis Data Firehose delivery stream, using your existing Kinesis Data Stream as the source, and the provided S3 bucket with the name including
firehosedeliverybucketas the destination. Observe the records being delivered both to our event consuming Lambda Function, and the Amazon S3 bucket by Kinesis Data Firehose.Note: It can take some time for the objects to appear in the S3 bucket.
What's a lab?
Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.
Provided environment for hands-on practice
We will provide the credentials and environment necessary for you to practice right within your browser.
Guided walkthrough
Follow along with the author’s guided walkthrough and build something new in your provided environment!
Did you know?
On average, you retain 75% more of your learning if you get time for practice.
