Massively Integrating AWS CloudWatch Logs with Loggly Using AWS CloudFormation

Well, integrating AWS CloudWatch Logs with Loggly is something already documented. The official source is: https://www.loggly.com/blog/sending-aws-cloudwatch-logs-to-loggly-with-aws-lambda/

But there is a use case I found to be pretty common in the real world that is not well documented either by Amazon or by Loggly or even the general community: often times, we decide to visualize our AWS logs using an external tool such as Loggly once we have a lot of applications already in place, or even applications generating logs to multiple log groups. Configuring a Lambda function to publish the log entries in real time to Loggly is relatively simple, but this becomes complex, or at least a long and error-prone task, when you have to configure many log groups to be connected with Loggly.

At least, you only need one lambda function to send your logs to Loggly and share it across multiple log groups. The idea is to use the function from the Loggly blueprint. Following the Loggly blog, you enable the function for just one log group. And you can manually add more triggers to your function from the web console. This is just fine for a few log groups, but the purpose of this post is to show you how to do it for multiple log groups at once.

After some research, we found CloudFormation as a clever tool for this task, since we could declare a resource for each trigger required for our already existing Lambda function (the one from the blueprint mentioned in the Loggly blog).

If you are not familiar with CloudFormation, this tool allows you to define a group of AWS objects described in a file (in JSON or YAML). The general structure of the file is:
{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "resource1": {
      "Type": "Resource1Type",
      "Properties": {}
    },
    "resource2": {
      "Type": "Resource2Type",
      "Properties": {}
    },
    ...
    "resourceN": {
      "Type": "ResourceNType",
      "Properties": {}
    }
  }
}

The key point here is to know what is the CloudFormation resource type required to define a trigger for a Lambda function, corresponding to the one required to send the logs to Loggly. And the answer is: AWS::Logs::SubscriptionFilter. So, each trigger would have a structure similar to:
  "myLogFilter": {
    "Type": "AWS::Logs::SubscriptionFilter",
    "Properties": {
      "DestinationArn": { "Fn::Join": [ ":", [ "arn:aws:lambda",
        { "Ref": "AWS::Region" }, { "Ref": "AWS::AccountId" },
        "function:lambdaFunctionAsPerLogglyBlog" ] ] },
      "FilterPattern": "",
      "LogGroupName": "/aws/lambda/logGroup"
    }
  }

You will need to replace the values in italics with the names corresponding to your case.

But this is not enough, and only defining your subscription filters, one for each log group, generates an error that doesn't allow you deploy your formation, since you need to grant permissions to each log group to invoke the Lambda function that sends the logs to Loggly (the error message shown is: "Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function"). So we need to add one more kind of object to our CloudFormation template: a permission (AWS::Lambda::Permission). The format for this object is:
  "myPermission": {
    "Type": "AWS::Lambda::Permission",
    "Properties": {
      "FunctionName": { "Fn::Join": [ ":", [ "arn:aws:lambda",
        { "Ref": "AWS::Region" }, { "Ref": "AWS::AccountId" },
        "function:lambdaFunctionAsPerLogglyBlog" ] ] },
      "Action": "lambda:InvokeFunction",
      "Principal": { "Fn::Join": [ ".", [ "logs",
        { "Ref": "AWS::Region" },
        "amazonaws.com" ] ] },
      "SourceArn": { "Fn::Join": [ ":", [ "arn:aws:logs",
        { "Ref": "AWS::Region" }, { "Ref": "AWS::AccountId" },
        "log-group:/aws/lambda/*" ] ] },
      "SourceAccount": { "Ref": "AWS::AccountId" }
    }
  }

Again, replace the values in italics with whatever is valid for your situation.

One important thing to notice is that you can use wildcards in the "SourceArn" property, which allows you to define a single permission covering multiple log groups. Beware of this and make sure you grant permissions to all your allowed log groups and no more than that. You rather declare more permissions instead of having just one, with a too open wildcard.

And there is one more thing. AWS tries to create as much resources as possible in parallel. But the missing permissions error described above could arise if it tries to create a subscription filter and no permission is available for its log group to invoke the Lambda function. The solution to this issue is to add a dependency to each subscription filter on the corresponding permission. So the format for the subscription filter now is:
  "myLogFilter": {
    "Type": "AWS::Logs::SubscriptionFilter",
    "Properties": {
      "DestinationArn": { "Fn::Join": [ ":", [ "arn:aws:lambda",
        { "Ref": "AWS::Region" }, { "Ref": "AWS::AccountId" },
        "function:lambdaFunctionAsPerLogglyBlog" ] ] },
      "FilterPattern": "",
      "LogGroupName": "/aws/lambda/logGroup"
    },
    "DependsOn": "myPermission"
  }

Wrapping up, this is the general structure for your CloudFormation template:
{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Resources": {
    "myPermissionA": {
      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "FunctionName": { "Fn::Join": [ ":", [ "arn:aws:lambda",
          { "Ref": "AWS::Region" }, { "Ref": "AWS::AccountId" },
          "function:lambdaFunctionAsPerLogglyBlog" ] ] },
        "Action": "lambda:InvokeFunction",
        "Principal": { "Fn::Join": [ ".", [ "logs",
          { "Ref": "AWS::Region" },
          "amazonaws.com" ] ] },
        "SourceArn": { "Fn::Join": [ ":", [ "arn:aws:logs",
          { "Ref": "AWS::Region" }, { "Ref": "AWS::AccountId" },
          "log-group:/aws/lambda/a*" ] ] },
        "SourceAccount": { "Ref": "AWS::AccountId" }
      }
    },
    "myPermissionB": {
      "Type": "AWS::Lambda::Permission",
      "Properties": {
        "FunctionName": { "Fn::Join": [ ":", [ "arn:aws:lambda",
          { "Ref": "AWS::Region" }, { "Ref": "AWS::AccountId" },
          "function:lambdaFunctionAsPerLogglyBlog" ] ] },
        "Action": "lambda:InvokeFunction",
        "Principal": { "Fn::Join": [ ".", [ "logs",
          { "Ref": "AWS::Region" },
          "amazonaws.com" ] ] },
        "SourceArn": { "Fn::Join": [ ":", [ "arn:aws:logs",
          { "Ref": "AWS::Region" }, { "Ref": "AWS::AccountId" },
          "log-group:/aws/lambda/b*" ] ] },
        "SourceAccount": { "Ref": "AWS::AccountId" }
      }
    },
    "myLogFilterX": {
      "Type": "AWS::Logs::SubscriptionFilter",
      "Properties": {
        "DestinationArn": { "Fn::Join": [ ":", [ "arn:aws:lambda",
          { "Ref": "AWS::Region" }, { "Ref": "AWS::AccountId" },
          "function:lambdaFunctionAsPerLogglyBlog" ] ] },
        "FilterPattern": "",
        "LogGroupName": "/aws/lambda/logGroupX"
      },
      "DependsOn": "myPermissionA"
    },
    "myLogFilterY": {
      "Type": "AWS::Logs::SubscriptionFilter",
      "Properties": {
        "DestinationArn": { "Fn::Join": [ ":", [ "arn:aws:lambda",
          { "Ref": "AWS::Region" }, { "Ref": "AWS::AccountId" },
          "function:lambdaFunctionAsPerLogglyBlog" ] ] },
        "FilterPattern": "",
        "LogGroupName": "/aws/lambda/logGroupY"
      },
      "DependsOn": "myPermissionB"
    },
    "myLogFilterZ": {
      "Type": "AWS::Logs::SubscriptionFilter",
      "Properties": {
        "DestinationArn": { "Fn::Join": [ ":", [ "arn:aws:lambda",
          { "Ref": "AWS::Region" }, { "Ref": "AWS::AccountId" },
          "function:lambdaFunctionAsPerLogglyBlog" ] ] },
        "FilterPattern": "",
        "LogGroupName": "/aws/lambda/logGroupZ"
      },
      "DependsOn": "myPermissionA"
    }
  }
}