Skip to main content

Security

Learn how to secure data and APIs.

In RAW, data and APIs can be secured by making them private and by using scopes.

Scopes are user-defined "tags" that are assigned to users or API keys. These can then be used to filter access to an endpoint, or to even change the result of an endpoint.

Filtering data using scopes​

Let us illustrate how to create an API where the data displayed will depend on the scopes/permissions of the user calling the endpoint.

For this example we read a log file which has 3 levels of log statements INFO, WARN and ERROR. The data access rules are:

  • Users without relevant scopes can only see INFO statements.
  • Users with the scope monitoring can see INFO and WARN statements.
  • Users with the scope admin scope can see all statements.
info

If you want to try this example, you can deploy the following endpoint:

Changing output of data depending of user scopes
Parse a log file and show subset of data depending of user scopes.

This example illustrates how to create a REST API where the data displayed will depend on the scopes/permissions of the user calling the endpoint.

For this example we will read a log file which has 3 levels of log statements INFO, WARN and ERROR. The data access rules are:

  • Users without relevant scopes can only see INFO statements.
  • Users with the scope monitoring can see INFO and WARN statements.
  • Users with the scope admin scope can see all statements.

Here's an excerpt of the log file:

2015-01-01T05:54:15 WARN vibration close to treshold, check instrumentation panel ASAP.
2015-01-01T05:54:58 INFO calibration at 100%, checking inner sub-systems.
2015-01-01T05:55:41 ERROR voltage not measured for more than 25 seconds, reboot machine.
2015-01-01T05:56:24 INFO cleaning procedure schedulled soon, performing sub task 111.
2015-01-01T05:57:07 INFO task 155 schedulled soon, preparing next task.

Parsing the file

We can use String.ReadLines and Regex.Groups to extract the timestamp, level and message from each line of text:

parse() = 
    let
        lines = String.ReadLines(
            "s3://raw-tutorial/ipython-demos/predictive-maintenance/machine_logs.log"
        ),
        parsed = Collection.Transform(lines, l ->
            let
                groups = Regex.Groups(l,"""(\d+-\d+-\d+T\d+:\d+:\d+) (\w+) (.*)"""),
                timestamp = Timestamp.Parse(List.Get(groups, 0),"yyyy-M-d\'T\'H:m:s"),
                level = List.Get(groups, 1),
                message = List.Get(groups, 2)
            in
                {timestamp: timestamp, level: level, message: message}
        )
    in
        parsed

The output of this function looks like this:

[
  {
    "timestamp": "2015-01-01T05:54:15.000",
    "level": "WARN",
    "message": "vibration close to treshold, check instrumentation panel ASAP."
  },
  {
    "timestamp": "2015-01-01T05:54:58.000",
    "level": "INFO",
    "message": "calibration at 100%, checking inner sub-systems."
  },
  {
    "timestamp": "2015-01-01T05:55:41.000",
    "level": "ERROR",
    "message": "voltage not measured for more than 25 seconds, reboot machine."
  },
  {
    "timestamp": "2015-01-01T05:56:24.000",
    "level": "INFO",
    "message": "cleaning procedure schedulled soon, performing sub task 111."
  },
  {
    "timestamp": "2015-01-01T05:57:07.000",
    "level": "INFO",
    "message": "task 155 schedulled soon, preparing next task."
  }
]

Filtering by user scope

Now that we have parsed the file we can start implementing our data restriction rules. We use the built-in function Environment.Scopes to get the current user permissions. This allows us filter the data accordingly:

Collection.Filter(
    parsed,
    (x) ->
Usage:

/logs-user-scopes


Here's an excerpt of the log file:

2015-01-01T05:54:15 WARN vibration close to treshold, check instrumentation panel ASAP.
2015-01-01T05:54:58 INFO calibration at 100%, checking inner sub-systems.
2015-01-01T05:55:41 ERROR voltage not measured for more than 25 seconds, reboot machine.
2015-01-01T05:56:24 INFO cleaning procedure schedulled soon, performing sub task 111.
2015-01-01T05:57:07 INFO task 155 schedulled soon, preparing next task.

Parsing the file​

We can use String.ReadLines and Regex.Groups to extract the timestamp, level and message from each line of text:

parse() = 
    let
        lines = String.ReadLines(
            "s3://raw-tutorial/ipython-demos/predictive-maintenance/machine_logs.log"
        ),
        parsed = Collection.Transform(lines, l ->
            let
                groups = Regex.Groups(l,"""(\d+-\d+-\d+T\d+:\d+:\d+) (\w+) (.*)"""),
                timestamp = Timestamp.Parse(List.Get(groups, 0),"yyyy-M-d\'T\'H:m:s"),
                level = List.Get(groups, 1),
                message = List.Get(groups, 2)
            in
                {timestamp: timestamp, level: level, message: message}
        )
    in
        parsed

The output of this function looks like this:

[
  {
    "timestamp": "2015-01-01T05:54:15.000",
    "level": "WARN",
    "message": "vibration close to treshold, check instrumentation panel ASAP."
  },
  {
    "timestamp": "2015-01-01T05:54:58.000",
    "level": "INFO",
    "message": "calibration at 100%, checking inner sub-systems."
  },
  {
    "timestamp": "2015-01-01T05:55:41.000",
    "level": "ERROR",
    "message": "voltage not measured for more than 25 seconds, reboot machine."
  },
  {
    "timestamp": "2015-01-01T05:56:24.000",
    "level": "INFO",
    "message": "cleaning procedure schedulled soon, performing sub task 111."
  },
  {
    "timestamp": "2015-01-01T05:57:07.000",
    "level": "INFO",
    "message": "task 155 schedulled soon, preparing next task."
  }
]

Filtering by user scope​

Now that we have parsed the file we can start implementing our data restriction rules. We use the built-in function Environment.Scopes to get the calling user scopes. This allows us filter the data accordingly:

Collection.Filter(
    parsed,
    (x) ->
        if List.Contains(Environment.Scopes(), "admin")
            then true
        else if List.Contains(Environment.Scopes(), "monitoring") 
            then x.level == "WARN" or x.level == "INFO"
        else
            x.level == "INFO"
)

Here is all the code together:

main() =
    let
        lines = String.ReadLines(
            "s3://raw-tutorial/ipython-demos/predictive-maintenance/machine_logs.log"
        ),
        parsed = Collection.Transform(
            lines,
            (l) ->
                let
                    groups = Regex.Groups(
                        l,
                        "(\\d+-\\d+-\\d+T\\d+:\\d+:\\d+) (\\w+) (.*)"
                    ),
                    timestamp = Timestamp.Parse(
                        List.Get(groups, 0),
                        "yyyy-M-d\'T\'H:m:s"
                    ),
                    level = List.Get(groups, 1),
                    message = List.Get(groups, 2)
                in
                    {timestamp: timestamp, level: level, message: message}
        )
    in
        Collection.Filter(
            parsed,
            (x) ->
                if List.Contains(Environment.Scopes(), "admin")
                    then true
                else if List.Contains(Environment.Scopes(), "monitoring") 
                    then x.level == "WARN" or x.level == "INFO"
                else
                    x.level == "INFO"
        )

API Keys​

API Keys provide a secure way to give access to private endpoints to users.

For more information on API Keys, check the API Key management documentation.

For instance, using the example above, we can create API keys with the scope monitoring or admin and set the expiration time, e.g. one month. We can now give this key to users, so they can see ERROR or WARN messages.