Auto Subscribing CloudWatch Log Groups to AWS Lambda Function

Serverless computing has enabled developers to develop their next-generation products from ideation to production, without waiting for, or worrying about, infrastructure. AWS Lambda and AWS CloudWatch Logs, both services provided by AWS, are one of the important components in serverless architectures. I’ll assume that you know what they are and how to use them and some jargon that accompanies them.

It’s no surprise that a large number of our customers are utilizing them and are generating huge amount of logs in Log Groups which they want to ingest into Sumo for monitoring and troubleshooting. Currently customers use AWS console and manually create a subscription filter to subscribe those Log groups to our Sumo Logic Lambda function for CloudWatch Logs. But what if they have tons of Log Groups consequently, this small task can soon become a DevOps nightmare.

The seriousness of the customer’s plight was instantly recognized and our team developed a Lambda function, LogGroup Lambda Connector for automating the process of creating AWS CloudWatch Log Group subscriptions.This solution is generic enough to be used with any lambda function not necessarily Sumo Logic Lambda functions (except for logs generated by function itself).For more information and step by step instructions checkout its documentation.

Ready to do even more with AWS Lambda? See how the Sumo Logic App for AWS Lambda drives continuous intelligence with real-time analytics or request a demo to see it in action yourself.Free Trial

Configuring and Deploying the LogGroup Lambda Connector using CloudFormation template

This lambda function provides following environment variables

1. LOG_GROUP_PATTERN: This is a javascript regex to filter out Log Groups and only the matched ones will be subscribed to the lambda function. Since the function uses constructor to create a RegExp object, the normal string escape rules (preceding special characters with \ when included in a string) are necessary. For example
   1. Input Pattern: test will result in /test/ and it will match testlogroup, logtestgroup and LogGroupTest
2. LAMBDA_ARN: This specifies ARN of the lambda function. To simplify it we have already provided json snippet and you have to specify FunctionName attribute of your lambda function in the given placeholder.
   

   { “Fn::Join”: [ “”, [ “arn:aws:lambda:”, { “Ref” : “AWS::Region” }, “:”, { “Ref” : “AWS::AccountId” }, “:function:<Your Lambda Function Name>” ] ]}

To automate the deployment we have created a cloudformation template(loggroup-lambda-cft.json) which creates following resources:

1. PermissionForEventsToInvokeLambda: Permission to CloudTrail events for invoking our lambda function(SumoLogGroupLambdaConnector).
2. SumoLGCnCreateLogGroupTrigger: CloudTrail Event Rule which triggers our lambda function(SumoLogGroupLambdaConnector) on CreateLogGroup event.
3. SumoLGCnLambdaExecutionRole: IAM Role for our lambda function function which defines permissions to create subscription filter and cloudwatch logs.
4. SumoLogGroupLambdaConnector: Lambda function responsible for creating a subscription filter on Log Groups matching filter criteria.

After completing above configuration you can upload the loggroup-lambda-cft.json via AWS console (see docs) or use the following command in aws cli.

aws cloudformation create-stack --stack-name "teststack" --template-body "file:////loggroup-lambda-cft.json" --region="us-east-2" --capabilities=CAPABILITY_IAM

Now to test our function we will create a log group say testloggroup assuming test is our log group pattern

We assume that our lambda function already exists(say TestLambda) whose name is configured in LAMBDA_ARN

Now observe after few seconds Subscriptions columns gets populated with subscription filter.

Following are some of its use cases:

Subscription Filters with Sumo Logic CloudWatch Logs Lambda Function

Currently CloudWatch Lambda function provides a Log Group (SumoCWLogGroup) which has a subscription filter (SumoCWLogSubsriptionFilter) associated with it. Now consider these scenarios, currently you have a log group /var/log/syslog from EC2 instance and now you have added /var/log/messages, /var/log/boot.log, /var/log/auth.log(new log groups) in awslogs configuration or you have recently deployed another lambda function and it’s logs are sent to /aws/lambda/newlambda log group. If you wanted to send those logs to Sumo and assuming you knew there pattern(/var/log/* and /aws/lambda/*) it’s easy to configure it in LOG_GROUP_PATTERN and specify SumoCWLogsLambda as function attribute in LAMBDA_ARN and you are done, any log groups matching this criteria will be subscribed to Sumo Logic CloudWatch Lambda function and will be ingested to sumo thereby avoiding the likelihood of forgetting to create those subscriptions.

One thing to note here is that you must provide CloudWatch logs permission to invoke your lambda function. See example below

aws lambda add-permission --function-name "helloworld" --statement-id "helloworld" --principal "logs.region.amazonaws.com" --action "lambda:InvokeFunction" --source-arn "arn:aws:logs:region:123456789123:log-group:TestLambda:*" --source-account "123456789012"

Subscription Filters with Kinesis

Though this function is mainly built to be used with lambda functions but you can specify any aws resource arn in LAMBDA_ARN environment variable.

For Kinesis update the role with below trust policy.

aws iam update-assume-role-policy --role-name KinesisRoleName --policy-document file://trust_policy.json

Here is the trust_policy.json

Associate the role with following permissions policy which defines what actions CloudWatch Logs can do on your account.

aws iam put-role-policy --role-name KinesisRoleName --policy-name Permissions-Policy-For-CWL --policy-document file://PermissionsForCWL.json

Below is the permissions_for_cwl.json

After setting up the environment variables and deploying log group lambda connector function, all the incoming log events from log groups matching the filter pattern are forwarded to your Kinesis stream which can be used for real time event processing.