Automated Solution for Optimum Kinesis Shards in Kinesis Data Streams

Correct Answer - A and C.

AWS Application Auto scaling can be used to scale Kinesis Streams automatically.

For this, CloudWatch can be used to monitor Kinesis Data Stream shard metrics.

Based on the changes in these metrics, CloudWatch can initiate a notification to Application Auto Scaling.

This will trigger an API Gateway to call Lambda Functions to increase/decrease Kinesis Data Stream Shards' number based upon metric values.

Alternatively, you can use the Amazon Kinesis Scaling Utilities.

To do so, you can use each utility manually or automated with an AWS Elastic Beanstalk environment.

Option B is incorrect because Amazon Kinesis Scaling Utility alone is a manual process as per documentation.

Option D is incorrect as using UpdateShardCount will be a manual process.

For more information on Scaling Kinesis Data Streams using Application Auto Scaling and Amazon Kinesis Scaling Utilities, refer to the following URLs-

https://aws.amazon.com/blogs/big-data/scaling-amazon-kinesis-data-streams-with-aws-application-auto-scaling/ https://aws.amazon.com/blogs/big-data/under-the-hood-scaling-your-kinesis-data-streams/

The correct answers are A and B.

Explanation:

Kinesis Data Streams is a managed service provided by AWS for real-time processing of streaming data at scale. It can handle data in real-time from various sources and process it using multiple consumer applications. It stores the data for 24 hours by default, and the data can be analyzed using various services like Amazon EMR, Amazon Redshift, and Amazon Elasticsearch.

Kinesis Data Streams is designed to scale up or down automatically based on the volume of incoming data. The number of shards within a Kinesis Data Stream is a critical factor in determining the overall throughput of the stream. Therefore, it is essential to choose an optimum number of shards based on the volume of data that the stream is expected to handle.

The management team is looking for an automated solution to achieve this. There are multiple options to automate the scaling of Kinesis Data Streams, but the two most appropriate options are:

A. Use Application Auto Scaling: Application Auto Scaling is a service provided by AWS that can automatically adjust the number of resources used by your application to maintain performance and reduce costs. It can be used to automatically scale Kinesis Data Streams based on the volume of data being processed. With Application Auto Scaling, you can define a target utilization level for the stream, and AWS will automatically adjust the number of shards in the stream to match the desired utilization level. This helps to ensure that you are only paying for the resources that you actually need and can avoid overprovisioning.

B. Use Amazon Kinesis Scaling Utility to modify the number of Shards in Kinesis Data Streams: Amazon Kinesis Scaling Utility is a tool provided by AWS that allows you to automate the scaling of Kinesis Data Streams based on the volume of data being processed. It works by monitoring the incoming data and adjusting the number of shards in the stream to match the volume of data. This helps to ensure that you are only paying for the resources that you actually need and can avoid overprovisioning.

C. Use Amazon Kinesis Scaling Utility along with AWS Elastic Beanstalk to automatically modify the number of Shards in Kinesis Data Streams: AWS Elastic Beanstalk is a service provided by AWS that makes it easy to deploy and manage applications in the AWS Cloud. However, it is not directly related to Kinesis Data Streams, and therefore, it is not a suitable option for automating the scaling of Kinesis Data Streams.

D. Use UpdateShardCount to scale Shard count as per requirement: UpdateShardCount is an API provided by AWS that allows you to manually adjust the number of shards in a Kinesis Data Stream. While it is possible to use this API to scale the shard count, it is not an automated solution and requires manual intervention, which is not ideal in a real-time stock trading scenario.

In summary, the most appropriate options to automate the scaling of Kinesis Data Streams based on the volume of data being processed are Application Auto Scaling and Amazon Kinesis Scaling Utility. These services allow you to dynamically adjust the number of shards in the stream to match the volume of data, which helps to ensure that you are only paying for the resources that you actually need and can avoid overprovisioning.