AWS Certified DevOps Engineer - Professional Exam: Question Answer

Filtering and Processing Logs for Unique Visitors in AWS Auto Scaling Group | DOP-C01

Prev Question Next Question

Question

You have a large number of web servers in an Auto Scaling group behind a load balancer.

On an hourly basis, you want to filter and process the logs to collect data on unique visitors and then put that data in a durable data store to run reports.

Web servers in the Auto Scaling group are constantly launching and terminating based on your scaling policies.

But you do not want to lose any of the log data from these servers during a stop/termination initiated by a user or by Auto Scaling.

What two approaches will meet these requirements? Choose two answers from the options given below.

Answers

A. Install an Amazon Cloudwatch Logs Agent on every web server during the bootstrap process.Create a CloudWatch log group and define Metric Filters to create custom metrics that track unique visitors from the streaming web server logs. Create a scheduled task on an Amazon EC2 instance that runs every hour to generate a new report based on the Cloudwatch custom metrics.

B. On the web servers, create a scheduled task that executes a script to rotate and transmit the logs to Amazon Glacier. Ensure that the operating system shutdown procedure triggers a logs transmission when the Amazon EC2 instance is stopped/terminated. Use Amazon Data Pipeline to process the data in Amazon Glacier and run reports every hour.

C. On the web servers, create a scheduled task that executes a script to rotate and transmit the logs to an Amazon S3 bucket. Ensure that the operating system shutdown procedure triggers a logs transmission when the Amazon EC2 instance is stopped/terminated.Use AWS Data Pipeline to move log data from the Amazon S3 bucket to Amazon Redshift to process and run reports every hour.

D. Install an AWS Data Pipeline Logs Agent on every web server during the bootstrap process.Create a log group object in AWS Data Pipeline, and define Metric Filters to move processed log data directly from the web servers to Amazon Redshift and run reports every hour.

Show Answer

Explanations

Click on the arrows to vote for the correct answer

A. B. C. D.

Answer - A and C.

You can use the CloudWatch Logs agent installer on an existing EC2 instance to install and configure the CloudWatch Logs agent.

For more information, please visit the below link:

http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html

You can publish your own metrics to CloudWatch using the AWS CLI or an API.

For more information, please visit the below link:

http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html

Amazon Redshift is a fast, fully managed data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL and your existing Business Intelligence (BI) tools.

It allows you to run complex analytic queries against petabytes of structured data, using sophisticated query optimization, columnar storage on high-performance local disks, and massively parallel query execution.

Most results come back in seconds.

For more information on copying data from S3 to redshift, please refer to the below link:

http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-copydata-redshift.html

The two approaches that meet the requirements of processing logs from web servers in an Auto Scaling group are:

A. Install an Amazon Cloudwatch Logs Agent on every web server during the bootstrap process. Create a CloudWatch log group and define Metric Filters to create custom metrics that track unique visitors from the streaming web server logs. Create a scheduled task on an Amazon EC2 instance that runs every hour to generate a new report based on the Cloudwatch custom metrics.

B. On the web servers, create a scheduled task that executes a script to rotate and transmit the logs to an Amazon S3 bucket. Ensure that the operating system shutdown procedure triggers a logs transmission when the Amazon EC2 instance is stopped/terminated. Use AWS Data Pipeline to move log data from the Amazon S3 bucket to Amazon Redshift to process and run reports every hour.

Approach A: Install CloudWatch Logs Agent on Web Servers

In this approach, you will install the CloudWatch Logs Agent on every web server during the bootstrap process. This agent will stream the web server logs to CloudWatch log groups. You will then define Metric Filters on the CloudWatch log groups to create custom metrics that track unique visitors from the streaming web server logs. A scheduled task on an Amazon EC2 instance will run every hour to generate a new report based on the Cloudwatch custom metrics.

This approach provides near real-time access to log data and allows you to generate reports hourly. However, it requires additional setup time to install the CloudWatch Logs Agent on every web server.

Approach B: Transmit Logs to Amazon S3 Bucket

In this approach, you will create a scheduled task on the web servers that executes a script to rotate and transmit the logs to an Amazon S3 bucket. You will ensure that the operating system shutdown procedure triggers a logs transmission when the Amazon EC2 instance is stopped/terminated. AWS Data Pipeline will move the log data from the Amazon S3 bucket to Amazon Redshift to process and run reports every hour.

This approach provides durability of the log data and allows for easy transfer of the data to Amazon Redshift for further processing. However, it requires more configuration to set up the scheduled task and AWS Data Pipeline.

Conclusion

Both approaches meet the requirements of processing logs from web servers in an Auto Scaling group. Approach A provides near real-time access to log data, while Approach B provides durability of the log data and allows for easy transfer of the data to Amazon Redshift for further processing. The choice between the two approaches depends on your specific needs and constraints.

Prev Question Next Question