Skip to main content

Example: Sharing logs on S3 with an API

This example illustrates how to create a REST API to read and serve data from a log file and using API Keys to share this data with external users temporarily.

info

If you are not familiar with RAW we recommend checking out our Getting started guide first. To use RAW, you need an account which you can create and use for free here.

info

If you want to try this example, you can deploy the following endpoint:

Changing output of data depending of user scopes
Parse a log file and show subset of data depending of user scopes.

This example illustrates how to create a REST API where the data displayed will depend on the scopes/permissions of the user calling the endpoint.

For this example we will read a log file which has 3 levels of log statements INFO, WARN and ERROR. The data access rules are:

  • Users without relevant scopes can only see INFO statements.
  • Users with the scope monitoring can see INFO and WARN statements.
  • Users with the scope admin scope can see all statements.

Here's an excerpt of the log file:

2015-01-01T05:54:15 WARN vibration close to treshold, check instrumentation panel ASAP.
2015-01-01T05:54:58 INFO calibration at 100%, checking inner sub-systems.
2015-01-01T05:55:41 ERROR voltage not measured for more than 25 seconds, reboot machine.
2015-01-01T05:56:24 INFO cleaning procedure schedulled soon, performing sub task 111.
2015-01-01T05:57:07 INFO task 155 schedulled soon, preparing next task.

Parsing the file

We can use String.ReadLines and Regex.Groups to extract the timestamp, level and message from each line of text:

parse() = 
    let
        lines = String.ReadLines(
            "s3://raw-tutorial/ipython-demos/predictive-maintenance/machine_logs.log"
        ),
        parsed = Collection.Transform(lines, l ->
            let
                groups = Regex.Groups(l,"""(\d+-\d+-\d+T\d+:\d+:\d+) (\w+) (.*)"""),
                timestamp = Timestamp.Parse(List.Get(groups, 0),"yyyy-M-d\'T\'H:m:s"),
                level = List.Get(groups, 1),
                message = List.Get(groups, 2)
            in
                {timestamp: timestamp, level: level, message: message}
        )
    in
        parsed

The output of this function looks like this:

[
  {
    "timestamp": "2015-01-01T05:54:15.000",
    "level": "WARN",
    "message": "vibration close to treshold, check instrumentation panel ASAP."
  },
  {
    "timestamp": "2015-01-01T05:54:58.000",
    "level": "INFO",
    "message": "calibration at 100%, checking inner sub-systems."
  },
  {
    "timestamp": "2015-01-01T05:55:41.000",
    "level": "ERROR",
    "message": "voltage not measured for more than 25 seconds, reboot machine."
  },
  {
    "timestamp": "2015-01-01T05:56:24.000",
    "level": "INFO",
    "message": "cleaning procedure schedulled soon, performing sub task 111."
  },
  {
    "timestamp": "2015-01-01T05:57:07.000",
    "level": "INFO",
    "message": "task 155 schedulled soon, preparing next task."
  }
]

Usage

/aws/s3/logs-scopes

Here's an excerpt of the log file:

2015-01-01T05:54:15 WARN vibration close to treshold, check instrumentation panel ASAP.
2015-01-01T05:54:58 INFO calibration at 100%, checking inner sub-systems.
2015-01-01T05:55:41 ERROR voltage not measured for more than 25 seconds, reboot machine.
2015-01-01T05:56:24 INFO cleaning procedure schedulled soon, performing sub task 111.
2015-01-01T05:57:07 INFO task 155 schedulled soon, preparing next task.
2015-01-01T05:57:50 WARN inner temp increasing rapidly, please check internet connection.
2015-01-01T05:58:33 INFO cleaning procedure starting, calibrating.
2015-01-01T06:00:00 WARN machine 24 with error=error1
2015-01-01T05:54:15 ERROR inner temp not measured for more than 16 seconds, please call 041 123 456 789.
2015-01-01T05:54:58 INFO task 144 delayed, performing sub task 146.
2015-01-01T05:55:41 INFO task 139 starting, performing sub task 144.

The following is function to parse the logs. It uses Regex.Groups to extract the timestamp, level and message from each line of text.

main(start: timestamp, end: timestamp) =
  let
    lines = String.ReadLines(
        "s3://raw-tutorial/ipython-demos/predictive-maintenance/machine_logs.log"
    ),
    parsed = Collection.Transform(lines, l ->
      let
        groups = Regex.Groups(l,"""(\d+-\d+-\d+T\d+:\d+:\d+) (\w+) (.*)"""),
        timestamp = Timestamp.Parse(List.Get(groups, 0),"yyyy-M-d\'T\'H:m:s"),
        level = List.Get(groups, 1),
        message = List.Get(groups, 2)
      in
        {timestamp: timestamp, level: level, message: message}
    )
  in
    Collection.Filter(parsed,l -> 
      l.timestamp > start and l.timestamp < end
    )

The resulting API allows users to get log data from a start timestamp until an end timestamp.

Sharing data with API keys

API Keys provide a secure way to give access to private endpoints to users without the necessary permissions (scopes) to access them.

If the sharing-logs endpoint is defined with the scope sys-admin this means only users with that scope can access the endpoint. We can then create a new API key with that scope, sys-admin, and set the expiration time e.g. one month. We can now give this key to external users which we want to share this data with.

For more information about this please check the guide on securing endpoints

Ready to try it out?

Register for free and start building today!

Otherwise, if you have questions/comments, join us on Discord!