![]() The metadata can later be accessed by the helper scripts. We use the AWS::CloudFormation::Init type to include metadata for our Amazon EC2 instance. We describe the tag and definitions next. Then we call a set of provided CloudFormation helper scripts from our EC2 instances’s UserData to implement the definitions before we move on to doing the work we’ve set up the EC2 instance to do. We can do so by using some capabilities that CloudFormation provides.įirst, we use an AWS::CloudFormation::Init tag in the metadata section of our EC2 definition to define customizations. In our case, we also want to automatically install some software packages (notably, the CloudWatch Logs agent) configure some files and then start some services – all before the actual processing specified in our UserData section is triggered. We use CloudFormation tags to customize the properties of the EC2 instance: specifying the EC2 instance type, AMI, the IAM role, VPC, and so on. The last CloudFormation template, Run-ec2-instance.yaml, starts our EC2 instance. This subscription will trigger the Lambda function when new logs are written to the CloudWatch log group, and pass the new log to it. Then, we associate the Lambda function and the CloudWatch log group via an “AWS::Logs::SubscriptionFilter” tag, specifying the Lambda function and the log group it’s subscribing to. You can see the source code of the Lambda here. For each invocation it unzips the received log file, converts it from JSON into a dictionary, then writes it out to S3 with our naming convention (///). The Lambda function receives logs from CloudWatch. We specify these variables as inputs to the master CloudFormation script, which passes them to the Lambda function at execution time. The function (cloudwatch-log-lambda.py) requires two environment variables, s3BucketName and s3KeyPrefix, to tell it where the log files should be exported to. We’ve packaged our Python code into a Lambda deployment package for uploading and deployment by CloudFormation. Next, we define the Lambda function that will perform the actual export. In Cloudwatchlogsexport.yaml, we first set up the CloudWatch Logs log group itself ( "AWS::Logs::LogGroup"). Exporting logs from CloudWatch Logs to S3 In practice we first set up the CloudWatch log group and export to Amazon S3, and then set up and configure the EC2 instance. Run-ec2-instance.yaml: This template creates the EC2 instance, installs the CloudWatch Log Agent, configures it to export the desired logs, and performs a specified task on startup (in this case, calculating digits of Pi). Cloudwatch-log-lambda.zip: This zip file contains the code for the Lambda function, packaged along with its prerequisites.Ĥ. It then creates a CloudWatch Log subscription to automatically send the CloudWatch log streams to the Lambda function.ģ. Cloudwatchlogsexport.yaml: This template creates the CloudWatch log group the logs will be sent to, and defines the Lambda function that will perform the export from the log group to S3. Cwexport-master-template.yaml: This template creates a security group and IAM role for our EC2 instance, and calls two embedded CloudFormation templates to do the real work.Ģ. The implementation consists of the following four files, which we’ll discuss later:ġ. If you’d like to skip ahead and see the code in action, go to “Running the Solution.” The implementation These helper functions can be combined to install and update a variety of software packages, configure them, start services, and more. To configure the EC2 instance, we use a neat feature in CloudFormation, the CloudFormation helper functions. Note that there’s some delay from the time a log message is created on the EC2 instance to the time it appears in the S3 bucket. A Lambda function (4) that’s subscribed to the log group picks up each log and writes it to an existing Amazon S3 bucket (5). The agent routes the configured logs to a CloudWatch Logs log group (3). ![]() When everything is up and running, we have an EC2 instance running a CloudWatch Logs agent (2). We use a CloudFormation stack (1) to create the components shown. Later, we’ll discuss other ways to integrate these components into your production infrastructure. The following diagram and code samples show how this solution works in a stand-alone fashion. To export the logs, we add some components to the CloudFormation stack that builds the EC2 instance. We’ll build it using Amazon CloudWatch Logs, AWS Lambda, and some useful capabilities in AWS CloudFormation for customizing EC2 instances. This blog post shows you how to build a solution for this problem. We’d like to do it in our CloudFormation stacks, as that’s our execution standard. “Then we can store them and process them later, for optimization, audit, and security review, and so on. “We want to get execution logs from our EC2 instances into S3,” my customer said. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |