Your company releases new features with high frequency while demanding high application availability.
As part of the application's A/B testing, logs from each updated Amazon EC2 instance of the application need to be analyzed in near real-time to ensure that the application is working flawlessly after each deployment.
If the logs show any anomalous behavior, the application version of the instance is changed to a more stable one.
Which of the following methods should you use for shipping and analyzing the logs in a highly available manner?
Click on the arrows to vote for the correct answer
A. B. C. D.Answer - C.
You can use Kinesis Streams for rapid and continuous data intake and aggregation.
The type of data used includes IT infrastructure log data, application logs, social media, market data feeds, and web clickstream data.
Because the response time for the data intake and processing is in real-time, the processing is typically lightweight.
The following are typical scenarios for using Kinesis Streams:
Accelerated log and data feed intake and processing - You can have producers push data directly into a stream.
For example, push system and application logs and they'll be available for processing in seconds.
This prevents the log data from being lost if the front end or application server fails.
Kinesis Streams provides accelerated data feed intake because you don't batch the data on the servers before you submit it for intake.
Real-time metrics and reporting - You can use data collected into Kinesis Streams for simple data analysis and reporting in real-time.
For example, your data-processing application can work on metrics and reporting for system and application logs as the data is streaming in rather than waiting to receive data batches.
For more information on Amazon Kinesis and SNS, please refer to the below link:
http://docs.aws.amazon.com/streams/latest/dev/introduction.htmlThe best option for shipping and analyzing the logs in near real-time in a highly available manner is to use Amazon Kinesis stream and analyze the logs in real-time (Option C).
Amazon Kinesis is a managed service that makes it easy to collect, process, and analyze real-time, streaming data. Kinesis allows you to collect and analyze data in real-time from multiple sources such as logs, IoT devices, clickstreams, and social media feeds.
In this scenario, the logs from each updated Amazon EC2 instance of the application need to be analyzed in near real-time to ensure that the application is working flawlessly after each deployment. Therefore, shipping the logs to Amazon Kinesis stream, which can provide real-time processing and analysis, is the most appropriate option.
By shipping the logs to an Amazon Kinesis stream, the logs can be immediately analyzed using various analytics tools such as Amazon Kinesis Analytics, AWS Lambda, and Amazon Elasticsearch. These tools can be used to detect any anomalous behavior and trigger a response to ensure high application availability.
Option A, shipping the logs to Amazon S3 for durability and using Amazon EMR to analyze the logs in a batch each hour, is not ideal for near real-time analysis. EMR is a batch processing service that can analyze large amounts of data at once, but it is not designed for real-time processing.
Option B, shipping the logs to Amazon CloudWatch Logs and using Amazon EMR to analyze the logs in a batch each hour, is also not ideal for near real-time analysis. Although CloudWatch Logs can collect and store logs, it is not designed for real-time analysis, and using EMR for batch processing would not allow for immediate analysis and response to any anomalous behavior.
Option D, shipping the logs to a large Amazon EC2 instance and analyzing the logs in real-time, would require manual setup and maintenance of the EC2 instance, which is not ideal for high availability. Additionally, using an EC2 instance for log analysis may not be as scalable or cost-effective as using a managed service like Amazon Kinesis.
Therefore, Option C, shipping the logs to an Amazon Kinesis stream and analyzing the logs in real-time, is the most appropriate and effective method for shipping and analyzing the logs in near real-time in a highly available manner.