Example: Sharing logs on S3 with an API

This example illustrates how to create a REST API to read and serve data from a log file and using API Keys to share this data with external users temporarily.

info

If you are not familiar with RAW we recommend checking out our Getting started guide first. To use RAW, you need an account which you can create and use for free here.

info

If you want to try this example, you can deploy the following endpoint:

Parsing logs

Parse a log file using regexes.

Overview

Usage:

/sharing-logs?start=<timestamp>&end=<timestamp>

For instance, to ask for logs from the 3rd January 2015 until the 4th January 2015, use:

/sharing-logs?start=2015-01-03T00:00&end=2015-01-04T00:00

Code

main(start: timestamp, end: timestamp) =
  let
    lines = String.ReadLines(
        "s3://raw-tutorial/ipython-demos/predictive-maintenance/machine_logs.log"
    ),
    parsed = Collection.Transform(lines, l ->
      let
        groups = Regex.Groups(l,"""(\d+-\d+-\d+T\d+:\d+:\d+) (\w+) (.*)"""),
        timestamp = Timestamp.Parse(List.Get(groups, 0),"yyyy-M-d\'T\'H:m:s"),
        level = List.Get(groups, 1),
        message = List.Get(groups, 2)
      in
        {timestamp: timestamp, level: level, message: message}
    )
  in
    Collection.Filter(parsed,l ->
      l.timestamp > start and l.timestamp < end
    )

// The following test will run if you press the [Run Code] button directly.
main(Timestamp.Build(2015, 1, 4, 0, 0), Timestamp.Build(2015, 1, 5, 0, 0))

Here's an excerpt of the log file:

2015-01-01T05:54:15 WARN vibration close to treshold, check instrumentation panel ASAP.
2015-01-01T05:54:58 INFO calibration at 100%, checking inner sub-systems.
2015-01-01T05:55:41 ERROR voltage not measured for more than 25 seconds, reboot machine.
2015-01-01T05:56:24 INFO cleaning procedure schedulled soon, performing sub task 111.
2015-01-01T05:57:07 INFO task 155 schedulled soon, preparing next task.
2015-01-01T05:57:50 WARN inner temp increasing rapidly, please check internet connection.
2015-01-01T05:58:33 INFO cleaning procedure starting, calibrating.
2015-01-01T06:00:00 WARN machine 24 with error=error1
2015-01-01T05:54:15 ERROR inner temp not measured for more than 16 seconds, please call 041 123 456 789.
2015-01-01T05:54:58 INFO task 144 delayed, performing sub task 146.
2015-01-01T05:55:41 INFO task 139 starting, performing sub task 144.

The following is function to parse the logs. It uses Regex.Groups to extract the timestamp, level and message from each line of text.

main(start: timestamp, end: timestamp) =
  let
    lines = String.ReadLines(
        "s3://raw-tutorial/ipython-demos/predictive-maintenance/machine_logs.log"
    ),
    parsed = Collection.Transform(lines, l ->
      let
        groups = Regex.Groups(l,"""(\d+-\d+-\d+T\d+:\d+:\d+) (\w+) (.*)"""),
        timestamp = Timestamp.Parse(List.Get(groups, 0),"yyyy-M-d\'T\'H:m:s"),
        level = List.Get(groups, 1),
        message = List.Get(groups, 2)
      in
        {timestamp: timestamp, level: level, message: message}
    )
  in
    Collection.Filter(parsed,l -> 
      l.timestamp > start and l.timestamp < end
    )

The resulting API allows users to get log data from a start timestamp until an end timestamp.

Sharing data with API keys

API Keys provide a secure way to give access to private endpoints to users without the necessary permissions (scopes) to access them.

If the sharing-logs endpoint is defined with the scope sys-admin this means only users with that scope can access the endpoint. We can then create a new API key with that scope, sys-admin, and set the expiration time e.g. one month. We can now give this key to external users which we want to share this data with.

For more information about this please check the guide on securing endpoints

Ready to try it out?Register for free and start building today!

Otherwise, if you have questions/comments, join us on Discord!