Example: Sharing logs on S3 with an API
This example illustrates how to create a REST API to read and serve data from a log file and using API Keys to share this data with external users temporarily.
If you are not familiar with RAW we recommend checking out our Getting started guide first. To use RAW, pick your favorite template API and start building!
If you want to try this example, you can deploy the following endpoint:
Changing output of data depending of user scopes
- Overview
- Code
This example illustrates how to create a REST API where the data displayed will depend on the scopes/permissions of the user calling the endpoint.
For this example we will read a log file which has 3 levels of log statements INFO
, WARN
and ERROR
.
The data access rules are:
- Users without relevant scopes can only see
INFO
statements. - Users with the scope
monitoring
can seeINFO
andWARN
statements. - Users with the scope
admin
scope can see all statements.
Here's an excerpt of the log file:
2015-01-01T05:54:15 WARN vibration close to treshold, check instrumentation panel ASAP.
2015-01-01T05:54:58 INFO calibration at 100%, checking inner sub-systems.
2015-01-01T05:55:41 ERROR voltage not measured for more than 25 seconds, reboot machine.
2015-01-01T05:56:24 INFO cleaning procedure schedulled soon, performing sub task 111.
2015-01-01T05:57:07 INFO task 155 schedulled soon, preparing next task.
Parsing the file
We can use String.ReadLines
and Regex.Groups
to extract the timestamp, level and message from each line of text:
parse() =
let
lines = String.ReadLines(
"s3://raw-tutorial/ipython-demos/predictive-maintenance/machine_logs.log"
),
parsed = Collection.Transform(lines, l ->
let
groups = Regex.Groups(l,"""(\d+-\d+-\d+T\d+:\d+:\d+) (\w+) (.*)"""),
timestamp = Timestamp.Parse(List.Get(groups, 0),"yyyy-M-d\'T\'H:m:s"),
level = List.Get(groups, 1),
message = List.Get(groups, 2)
in
{timestamp: timestamp, level: level, message: message}
)
in
parsed
The output of this function looks like this:
[
{
"timestamp": "2015-01-01T05:54:15.000",
"level": "WARN",
"message": "vibration close to treshold, check instrumentation panel ASAP."
},
{
"timestamp": "2015-01-01T05:54:58.000",
"level": "INFO",
"message": "calibration at 100%, checking inner sub-systems."
},
{
"timestamp": "2015-01-01T05:55:41.000",
"level": "ERROR",
"message": "voltage not measured for more than 25 seconds, reboot machine."
},
{
"timestamp": "2015-01-01T05:56:24.000",
"level": "INFO",
"message": "cleaning procedure schedulled soon, performing sub task 111."
},
{
"timestamp": "2015-01-01T05:57:07.000",
"level": "INFO",
"message": "task 155 schedulled soon, preparing next task."
}
]
Usage
/aws/s3/logs-scopes
main() =
let
lines = String.ReadLines("s3://raw-tutorial/ipython-demos/predictive-maintenance/machine_logs.log"),
parsed = Collection.Transform(
lines,
(l) ->
let
groups = Regex.Groups(l, "(\\d+-\\d+-\\d+T\\d+:\\d+:\\d+) (\\w+) (.*)"),
timestamp = Timestamp.Parse(List.Get(groups, 0), "yyyy-M-d\'T\'H:m:s"),
level = List.Get(groups, 1),
message = List.Get(groups, 2)
in
{timestamp: timestamp, level: level, message: message}
)
in
Collection.Filter(
parsed,
(x) ->
if List.Contains(Environment.Scopes(), "admin") then
true
else
if List.Contains(Environment.Scopes(), "monitoring") then
x.level == "WARN" or x.level == "INFO"
else
x.level == "INFO"
)
Here's an excerpt of the log file:
2015-01-01T05:54:15 WARN vibration close to treshold, check instrumentation panel ASAP.
2015-01-01T05:54:58 INFO calibration at 100%, checking inner sub-systems.
2015-01-01T05:55:41 ERROR voltage not measured for more than 25 seconds, reboot machine.
2015-01-01T05:56:24 INFO cleaning procedure schedulled soon, performing sub task 111.
2015-01-01T05:57:07 INFO task 155 schedulled soon, preparing next task.
2015-01-01T05:57:50 WARN inner temp increasing rapidly, please check internet connection.
2015-01-01T05:58:33 INFO cleaning procedure starting, calibrating.
2015-01-01T06:00:00 WARN machine 24 with error=error1
2015-01-01T05:54:15 ERROR inner temp not measured for more than 16 seconds, please call 041 123 456 789.
2015-01-01T05:54:58 INFO task 144 delayed, performing sub task 146.
2015-01-01T05:55:41 INFO task 139 starting, performing sub task 144.
The following is function to parse the logs.
It uses Regex.Groups
to extract the timestamp, level and message from each line of text.
main(start: timestamp, end: timestamp) =
let
lines = String.ReadLines(
"s3://raw-tutorial/ipython-demos/predictive-maintenance/machine_logs.log"
),
parsed = Collection.Transform(lines, l ->
let
groups = Regex.Groups(l,"""(\d+-\d+-\d+T\d+:\d+:\d+) (\w+) (.*)"""),
timestamp = Timestamp.Parse(List.Get(groups, 0),"yyyy-M-d\'T\'H:m:s"),
level = List.Get(groups, 1),
message = List.Get(groups, 2)
in
{timestamp: timestamp, level: level, message: message}
)
in
Collection.Filter(parsed,l ->
l.timestamp > start and l.timestamp < end
)
The resulting API allows users to get log data from a start timestamp until an end timestamp.
Sharing data with API keys
API Keys provide a secure way to give access to private endpoints to users without the necessary permissions (scopes) to access them.
If the sharing-logs
endpoint is defined with the scope sys-admin
this means only users with that scope can access the endpoint.
We can then create a new API key with that scope, sys-admin
, and set the expiration time e.g. one month.
We can now give this key to external users which we want to share this data with.
For more information about this please check the guide on securing endpoints
Ready to try it out?
Pick a template!Otherwise, if you have questions/comments, join us in our Community!