Skip to main content

Example: How to build an API over Cloudwatch data

Learn how to create an API to retrieve and analyze data from Cloudwatch.

info

If you are not familiar with RAW we recommend checking out our Getting started guide first. To use RAW, you need an account which you can create and use for free here.

Click one of the examples below to learn more and deploy it.

Search CloudWatch logs of a given log group
Learn how to create a REST API that queries CloudWatch logs of a given log group.

This example illustrates how to create a REST API that queries Cloudwatch logs of a given log group.

According to Amazon's documentation

Cloudwatch Logs enables you to centralize the logs from all of your systems, applications, and AWS services that you use, in a single, highly scalable service. You can then easily view them, search them for specific error codes or patterns, filter them based on specific fields, or archive them securely for future analysis. Cloudwatch Logs enables you to see all of your logs, regardless of their source, as a single and consistent flow of events ordered by time, and you can query them and sort them based on other dimensions, group them by specific fields, create custom computations with a powerful query language, and visualize log data in dashboards.

For the purposes of this example, we have created a log group called /aws/tutorial/raw with synthetic apache server logs using the Extended Log File format or EFL.

Here's a small sample of these logs:

1.158.55.127 - user.wasp [01/Dec/2022:00:00:18 +0200] "PUT /apps/cart.jsp?appID=9118 HTTP/1.0" 200 5048 "http://baker.com/category/" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/5352 (KHTML, like Gecko) Chrome/14.0.835.0 Safari/535
2"
45.212.64.242 - user.iceman [01/Dec/2022:00:01:15 +0200] "PUT /apps/cart.jsp?appID=2281 HTTP/1.0" 500 4982 "http://www.robertson-ponce.com/home.asp" "Opera/9.22.(X11; Linux x86_64; it-IT) Presto/2.9.188 Version/10.00"
61.212.29.252 - user.hulk [01/Dec/2022:00:01:24 +0200] "GET /wp-content HTTP/1.0" 200 4911 "http://mathis-dominguez.net/category.html" "Mozilla/5.0 (X11; Linux i686; rv:1.9.7.20) Gecko/2014-04-25 14:32:11 Firefox/11.0"

Log group /aws/tutorial/raw contains several log streams containing the aforementioned logs in chronological order without any common timestamps.

The purpose of this example is to show how we can initiate a query over logs withing this log stream and retrieve the respective results as soon as they are ready to be consumed. Normally, this is a two-step process; The first step is to submit a Cloudwatch query, using the respective syntax. Invocation details can be found here. Upon successful execution, the output is a query identifier which serves as input to the second step of the process, the results retrieval. That step may result in incomplete results due to the fact that this is a long-running process and may not have prepared all data by the time of invocation. To bypass this problem, we need to adopt a busy wait process until the response status is Complete.

Patterns

There are several useful recipes in this use-case example, such as proxying AWS API calls, chaining multiple invocations and introducing artificial delays to avoid AWS throttling exceptions.

We use AWS API calls to interact with AWS, providing a facade that simplifies the required signing process. Users only need to focus on the API calls.

Here is a template of AWS requests:

Aws.SignedV4Request(
    <aws_access_key_id>,
    <aws_secret_access_key>,
    <services>,                         // e.g. "logs", "iam", "ec2" - see https://docs.aws.amazon.com/general/latest/gr/rande.html 
    region = awsRegion,                 // e.g. "us-east-1", "eu-west-1", etc. - see https://aws.amazon.com/about-aws/global-infrastructure/regions_az/
    method = <http_verb>,               // e.g. "GET", "POST"
    bodyString = <body>,                // http payload in case of "POST"
    args = <list_of_query_parameters>,  // in case of "GET"
    headers = <list_of_headers>         // list of http headers

Starting a Cloudwatch query is implemented like this:

Aws.SignedV4Request(
    <aws_access_key_id>,
    <aws_secret_access_key>,
    "logs",
    region = awsRegion,
    method = "POST",
    bodyString = Json.Print({
      endTime: endTimeInEpochMillis,
      limit: resultsLimit,
      logGroupNames: logGroupNames,
      queryString: query,
      startTime: startTimeInEpochMillis
    }),
    headers = [
      {"x-amz-target", "Logs_20140328.StartQuery"},
      {"content-type", "application/x-amz-json-1.1"}
    ]
)

Retrieving information about a user group is performed with this call:

Aws.SignedV4Request(
    <aws_access_key_id>,
    <aws_secret_access_key>,
    "iam",
    method = "GET",
    args = [
      {"Action", "GetGroup"},
      {"Version", "2010-05-08"},
      {"GroupName", usergroup}
    ]
)

Sample usage

The endpoint requires the following input arguments:

  • start and end times in ISO 8601 format (e.g. "2022-11-28T00:00:00"),
  • a Cloudwatch insights query,
  • an AWS region.

The necessary AWS credentials are embedded in the source code and they correspond to a read-only user. Note that this not recommended due to security issues. Instead users are encouraged to use secrets, which are key/value pairs that are stored securely outside of the source code. Secrets can be accessed using the built-in function Environment.Secret

Here is an example of retrieving logs of user.thor, after properly parsing EFL-type Apache logs:

shell$ query='query='fields @timestamp, @message                                                                                                                                                                | parse @message /(?<@ip>([^\s]+)) - (?<@userid>([^\s]+)) \[(?<@mdate>([^\]]+))\] "(?<@verb>([^\s]+)) (?<@url>([^\s]+)) (?<@httpversion>([^\s]+))" (?<@response>([^\s]+)) (?<@bytes>([^\s]+)) "(?<@referer>([^\"]+))" "(?<@useragent>([^\"]+))"/                                                                                                                                                                                                                                          | filter @userid = "user.thor"                                                                                                                                                                                                               | sort @timestamp desc'                         
                         
shell$ curl --get --data-urlencode "startTime=2022-11-28T00:00:00" --data-urlencode "endTime=2022-11-30T00:00:00" --data-urlencode "CloudwatchInsightsQuery=$query" --data-urlencode "awsRegion=eu-west-1" -H 'authorization: Bearer xxx' 'https://<org_name>.api.raw-labs.com/aws/example/logs/search'

The expected list is returned:

[
...
  [
    {
      "field": "@timestamp",
      "value": "2022-11-28 13:06:14.000"
    },
    {
      "field": "@message",
      "value": " 167.248.51.52 - user.thor [28/Nov/2022:15:06:14 +0200] \"GET /apps/cart.jsp?appID=8333 HTTP/1.0\" 200 4920 \"http://wolf.biz/\" \"Mozilla/5.0 (Macintosh; PPC Mac OS X 10_7_8) AppleWebKit/5350 (KHTML, like Gecko) Chrome/13.0.869.0 Safari/5350\""                                                                                                                                                                                                                     
    },
    {
      "field": "@ip",
      "value": "167.248.51.52"
    },
    {
      "field": "@userid",
      "value": "user.thor"
    },
    {
      "field": "@mdate",
      "value": "28/Nov/2022:15:06:14 +0200"
    },
    {
      "field": "@verb",
      "value": "GET"
    },
    {
      "field": "@url",
      "value": "/apps/cart.jsp?appID=8333"
    },
    {
      "field": "@httpversion",
      "value": "HTTP/1.0"
    },
    {
      "field": "@response",
      "value": "200"
    },
    {
      "field": "@bytes",
      "value": "4920"
    },
    {
      "field": "@referer",
      "value": "http://wolf.biz/"
    },
    {
      "field": "@useragent",
      "value": "Mozilla/5.0 (Macintosh; PPC Mac OS X 10_7_8) AppleWebKit/5350 (KHTML, like Gecko) Chrome/13.0.869.0 Safari/5350"
    },
    {
      "field": "@ptr",
      "value": "CmEKIgoeNjg0MTMwNjU4NDcwOi9hd3MvdHV0b3JpYWwvcmF3EAYSORoYAgY1ICb0AAAAAEhlC2oABjhS7yAAAABCIAEomJWj3MswMJD1gYLMMDj6EkC+jidIorcHUMizBxgAELULGAE="
    }
  ],
...
]

⚠️ Never store sensitive information as clear text in the code. Instead use secrets, which are key/value pairs that are stored securely outside of the source code. Secrets can be accessed using the built-in function Environment.Secret.

Retrieve logs for all users of a given group
Learn how to create a REST API that combines data from IAM and CloudWatch Amazon services.

This example illustrates how to create a REST API that combines data from IAM and Cloudwatch Amazon services. The goal is to retrieve logs from all users of a given user group.

Note that this example is built on top of Cloudwatch search logs example, combining it with insights from IAM service. Please refer to that example for further details about Cloudwatch invocations.

For the purposes of this example, we have created the following set of users:

  • user.spiderman
  • user.ironman
  • user.hulk
  • user.black.panther
  • user.namor
  • user.luke.cage
  • user.wolverine
  • user.captain.america
  • user.odin
  • user.iceman
  • user.ant.man
  • user.dr.strange
  • user.clint.barton
  • user.wade.wilson
  • user.wasp
  • user.carol.danvers
  • user.thor

grouped into the following user-groups:

  • example.human
  • example.magician
  • example.mutant
  • example.superhuman

Moreover, we have created a log group called /aws/tutorial/raw with synthetic apache server logs using the Extended Log File format or EFL.

Here's a small sample of these logs:

1.158.55.127 - user.wasp [01/Dec/2022:00:00:18 +0200] "PUT /apps/cart.jsp?appID=9118 HTTP/1.0" 200 5048 "http://baker.com/category/" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/5352 (KHTML, like Gecko) Chrome/14.0.835.0 Safari/535
2"
45.212.64.242 - user.iceman [01/Dec/2022:00:01:15 +0200] "PUT /apps/cart.jsp?appID=2281 HTTP/1.0" 500 4982 "http://www.robertson-ponce.com/home.asp" "Opera/9.22.(X11; Linux x86_64; it-IT) Presto/2.9.188 Version/10.00"
61.212.29.252 - user.hulk [01/Dec/2022:00:01:24 +0200] "GET /wp-content HTTP/1.0" 200 4911 "http://mathis-dominguez.net/category.html" "Mozilla/5.0 (X11; Linux i686; rv:1.9.7.20) Gecko/2014-04-25 14:32:11 Firefox/11.0"

As one can see, the third field is the actual user name defined in IAM.

Patterns

There are several useful recipes in this use-case example, such as proxying AWS API calls, combining different AWS services, chaining multiple invocations and introducing artificial delays to avoid AWS throttling exceptions.

We use AWS API calls to interact with AWS, providing a facade that simplifies the required signing process. Users only need to focus on the API calls.

Here is a template of AWS requests:

Aws.SignedV4Request(
        <aws_access_key_id>,
        <aws_secret_access_key>,
        <services>,                         // e.g. "logs", "iam", "ec2" - see https://docs.aws.amazon.com/general/latest/gr/rande.html 
        region = awsRegion,                 // e.g. "us-east-1", "eu-west-1", etc. - see https://aws.amazon.com/about-aws/global-infrastructure/regions_az/
        method = <http_verb>,               // e.g. "GET", "POST"
        bodyString = <body>,                // http payload in case of "POST"
        args = <list_of_query_parameters>,  // in case of "GET"
        headers = <list_of_headers>         // list of http headers

Starting a Cloudwatch query is implemented like this:

Aws.SignedV4Request(
    <aws_access_key_id>,
    <aws_secret_access_key>,
    "logs",
    region = awsRegion,
    method = "POST",
    bodyString = Json.Print({
      endTime: endTimeInEpochMillis,
      limit: resultsLimit,
      logGroupNames: logGroupNames,
      queryString: query,
      startTime: startTimeInEpochMillis
    }),
    headers = [
      {"x-amz-target", "Logs_20140328.StartQuery"},
      {"content-type", "application/x-amz-json-1.1"}
    ]
)

Retrieving information about a user group is performed with this call:

Aws.SignedV4Request(
    <aws_access_key_id>,
    <aws_secret_access_key>,
    "iam",
    method = "GET",
    args = [
      {"Action", "GetGroup"},
      {"Version", "2010-05-08"},
      {"GroupName", usergroup}
    ]
)

Sample usage

The endpoint requires the following input argument:

  • a user group name

The necessary AWS credentials are embedded in the source code and they correspond to a read-only user. Note that this not recommended due to security issues. Instead users are encouraged to use secrets, which are key/value pairs that are stored securely outside of the source code. Secrets can be accessed using the built-in function Environment.Secret.

Here is a template of retrieving logs for users in a given group:

https://<organization>.api.raw-labs.com/aws/example/iam-logs/search?usergroup=<group_name>

Here is an example of retrieving logs for users in group example.mutant:

https://<organization>.api.raw-labs.com/aws/example/iam-logs/search?usergroup=example.mutant

The retrieved logs refer to users from the given example:

[
  ...
  [
    {
      "field": "@timestamp",
      "value": "2022-11-28 11:56:51.000"
    },
    {
      "field": "@message",
      "value": " 185.54.246.80 - user.spiderman [28/Nov/2022:13:56:51 +0200] \"GET /wp-admin HTTP/1.0\" 200 4994 \"http://smith-roy.com/homepage/\" \"Mozilla/5.0 (iPod; U; CPU iPhone OS 4_1 like Mac OS X; en-US) AppleWebKit/531.21.6 (KHTML, like Gecko) Version/4.0.5 Mobile/8B111 Safari/6531.21.6\""                                                                                                                                                                               
    },
    {
      "field": "@ip",
      "value": "185.54.246.80"
    },
    {
      "field": "@userid",
      "value": "user.spiderman"
    },
    ...
  ],
...
]

⚠️ Never store sensitive information as clear text in the code. Instead use secrets, which are key/value pairs that are stored securely outside of the source code. Secrets can be accessed using the built-in function Environment.Secret.

Ready to try it out?

Register for free and start building today!

Otherwise, if you have questions/comments, join us on Discord!