Using the REST API

Introduction

This document describes the REST API of RAW services:

  • Executor service: responsible for executing queries, creating views and packages.

  • Credentials service: responsible for managing credentials.

  • Storage service: responsible for managing the cache.

Preparation

First go to the Web Interface of the RAW instance that you want to use, navigate to Settings, Authentication and click Download configuration in the Interactive clients section.

This will download a configuration with the settings needed to use the REST API of this RAW instance. The following is an example:

[raw]
executor_url = https://just-ask.raw-labs.com/api/executor
creds_url = https://just-ask.raw-labs.com/api/creds
storage_url = https://just-ask.raw-labs.com/api/storage
auth = oauth2

[auth_oauth2]
provider = auth0
domain = raw.eu.auth0.com
audience = https://just-ask.raw-labs.com
grant_type = authorization_code
client_id = mI82ICJqcWkMe3TgTY3tJO4WecQ5XVcw
client_secret = <client_secret>

The REST endpoints for the different services are specified in the first part of the file. For instance, in this case, the endpoint for the Executor service is https://just-ask.raw-labs.com/api/executor. In the remaining of this document, we will refer to these endpoints by their names in the configuration file above, that is, by executor_url, creds_url, storage_url.

Security

All RAW REST endpoints are protected by OAuth2 authorization. The caller must first obtain a valid OAuth2 token with the authorization service and then provide it in the Authorization header of the request.

To obtain the access token, first install the RAW command line tool as described in Installation of the Command-Line Client. Then, to obtain an access token, run:

rawcli -c <configuration_file> login -p

This command will launch a web browser to perform an interactive login. After the login completes successfully, this command will write the access token to the console.

Retrials

The server can respond with 503 if a request could not be processed but can be retried safely.

For instance, if a query is received but no resources are yet available, or are still starting up, a response with HTTP Status Code 503 is sent. In this case, clients can safely retry (silently if desired) on 503, instead of returning the failure directly to the user.

Executor Service

Querying

Execute a query using the REST API

Request:

POST /1/public/query

MediaType:

text/plain

Body contains the query in UTF-8 encoding, by default the result of the query will be in hjson format, to specify other formats see the table bellow “Optional HTTP headers”

Example:
curl -X POST <executor-url>/1/public/query \
    --header "Authorization: Bearer <access token>" \
    --data 'SELECT * FROM ["Hello", "World"]'

Optional HTTP headers:

Name

Type

Description

X-RAW-output-format

string

Output format of the query, accepted values are:
“json”, “hjson”, “csv”, “sqlite”,
“parquet-avro”.
Default “hjson”.

X-RAW-windows-line-ending

boolean

If “true” output with windows line ending \n\r
if “false” output with unix line ending \n.
Default “false”.

X-RAW-max-bytes

int

Maximum number of bytes in the response

Response
200 OK:
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8

"Hello"
"World"
Query Invalid:

In case of semantic errors or runtime errors in the query, the server will return a 400 response, with a list of errors in the response.

HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{
   "validationErrors":[
      {
         "message":"bb is not declared",
         "positions":[{
            "begin":{ "line":1, "column":1 },
            "end":{ "line":1, "column":3 }
         }]
      }
   ],
   "runtimeError":null
}
408 Request Timeout:

The request took too long to execute and was cancelled. This error likely not retriable, as execution of the same request is likely to once again exceed the time limits of the server. As an alternative, use WebSockets to execute the request, which has no time bounds restriction for execution.

Executing a query using WebSockets

It is possible to use WebSockets to execute a query and get status information from the server while the query is executing.

Url:

/1/public/ws/query

The following example uses wscat to execute a query.

Example:
wscat -c  ws://<executor-url>/1/public/ws/query
Connected (press CTRL+C to quit)
> {"type": "queryRequest", "accessToken": "<access token>", "query": "SELECT * FROM [\"Hello\", \"World\"] ", "settings": {"X-RAW-output-format": "hjson"}, "options": {} }
< {"type":"userMessage","level":"debug","message":"Starting query compilation"}
< {"type":"userMessage","level":"debug","message":"Finished query compilation"}
< {"type":"userMessage","level":"debug","message":"Starting query execution"}
< {"type":"querySuccess","id":"b4fcf3c1-1a7e-44a4-b1ed-eed118cf5d68-10"}
< {"type":"userMessage","level":"debug","message":"Starting writing results"}
< "Hello"
"World"
< {"type":"userMessage","level":"debug","message":"Finished query execution"}
< {"type":"userMessage","level":"debug","message":"Finished writing results"}
< {"type":"queryStreamingEnd"}
<
Disconnected (code: 1000, reason: "")
Please note that:
  • Binary frames from the server will contain query results and can, for instance, be appended to a file.

  • Text frames are used as a json protocol to get status information about the query.

The rest of this section will describe all possible JSON messages to and from the server.

Messages to the server

Query request

Execute query request.

accessToken

string

the access token

query

string

the query to execute

settings

record

key-value pairs string->string of settings
for this query for e.g.:
{"X-RAW-output-format": "hjson"}

options

record

key-value pairs string->string of options

type

string

literal string queryRequest

Example:
{"accessToken": "<accessToken>", "type": "queryRequest", "query": "SELECT * FROM read(\"mock:delay=5s:file:///tmp/out\")", "settings": {"X-RAW-output-format": "hjson"}, "options": {} }
Messages from the server
User message

Message with status information on the query.

level

string

log level: debug, info, warning or error

message

string

the message

type

string

literal string userMessage

Example:
{"type":"userMessage","level":"debug","message":"Finished query execution"}
Query success

Sent when the query executed successfully.

id

string

id of the query

type

string

literal string querySuccess

Example:
{"type":"querySuccess","id":"b4fcf3c1-1a7e-44a4-b1ed-eed118cf5d68-15"}
Unauthorized

Sent when the access token is not valid or the user is not authorized

error

string

error message

type

string

literal string unauthorized

Example:
{"type":"unauthorized","error":"authentication error: invalid access token"}
Invalid Query

Sent when the query has errors.

errors

List[ExecutorError]

list of executor errors

type

string

literal string invalidQuery

Example:
{
    "type": "invalidQuery"
    "errors": [{
         "message":"bb is not declared", "positions":[{
            "begin":{ "line":1, "column":1 },
            "end":{ "line":1, "column": 3 }
         }]
    }]
}
Query streaming end

Sent before closing the connection after the execution of a query.

type

string

literal string queryStreamingEnd

Example:
{"type":"queryStreamingEnd"}

Validate a query

Validates a query for syntax and semantic errors.

Request:

POST /1/public/query-validate

MediaType:

text/plain

Body contains the query in UTF-8 encoding.

Example:
curl -X POST <executor-url>/1/public/query-validate \
  --header "Authorization: Bearer <access token>" \
  --data 'SELECT * FROM ["Hello", "World"]'
Response
200 OK:

Returns a JSON object with the result type of the query.

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{"type":"collection(string)"}
202 Accepted:

If the response takes too long to compute, the server will answer with a 202 containing a request id that can be used to call the REST endpoint /1/public/pending-request to obtain the response to the original request.

HTTP/1.1 202 Accepted
Content-Type: text/plain; charset=utf-8

2c2d8351-cddf-4eca-9b97-436713773168
Query invalid:

If the query is invalid, the server will return a 400 response, with a list of errors in the response.

HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{
   "validationErrors":[
      {
         "message":"bb is not declared",
         "positions":[{
            "begin":{ "line":1, "column":1 },
            "end":{ "line":1, "column":3 }
         }]
      }
   ],
   "runtimeError":null
}

Views

Create a new view

Request:

PUT /views/<name>

MediaType:

text/plain

Body contains the query in UTF-8 encoding.

Example:
curl -X PUT <executor-url>/1/public/views/helloworld \
  --header "Content-Type: text/plain" \
  --header "Authorization: Bearer <access token>" \
  --data 'SELECT * FROM ["Hello", "World"]'
Response
200 OK:

If the view is valid the server will return a json with the type of the view.

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{ "type": "collection(string)" }
202 Accepted:

If the response takes too long to compute, the server will answer with a 202 containing a request id that can be used to call the REST endpoint /1/public/pending-request to obtain the response to the original request.

HTTP/1.1 202 Accepted
Content-Type: text/plain; charset=utf-8

2c2d8351-cddf-4eca-9b97-436713773168
Invalid view:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{
   "validationErrors":[
      {
         "message":"bb is not declared",
         "positions":[{
            "begin":{ "line":1, "column":1 },
            "end":{ "line":1, "column":3 }
         }]
      }
   ],
   "runtimeError":null
}
View name is invalid:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"viewNameInvalid", "message":"View name is invalid" }
View already exists:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"viewAlreadyExists", "message":"name already exists as a view" }
Materialized view already exists:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"materializedViewAlreadyExists", "message":"name already exists as a materialized view" }

List views

Request:

GET /1/public/views?pattern=<pattern>

Query parameters:
  • pattern - search pattern following the format used by the SQL LIKE operator (eg., % for zero or more characters, _ for single character)

Body:

empty

Example:
curl <executor-url>/1/public/views \
    --header "Authorization: Bearer <access token>" \
    -G --data-urlencode "pattern=hello%"
Response
200 OK:
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

[
  {
    "name": "helloworld",
    "query": "SELECT * FROM [\"Hello\", \"World\"]",
    "createdAt": "2017-11-09T09:54:28.599"
  }
]

Retrieve details of a view

Request:

GET /1/public/views/<name>

Body:

empty

Example:
curl <executor-url>/1/public/views/helloworld \
  --header "Authorization: Bearer <access token>"
Response
200 OK:
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "name": "helloworld",
  "query": "SELECT * FROM [\"Hello\", \"World\"]",
  "createdAt": [2017, 11, 09, 09, 54, 8, 599000],
  "updatedAt": null
}
View not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"viewNotFound", "message":"View not found" }

Validate a view

Request:

GET /1/public/view-validate/<name>

Body:

empty

Example:
curl <executor-url>/1/public/view-validate/helloworld \
  --header "Authorization: Bearer <access token>"
Response
200 OK:

Returns a JSON object with the result type of the view.

HTTP/1.1 200 OK
Content-Type: application/json

{
    "declarations": {},
    "type": "collection(string)",
    "comment": null
}
202 Accepted:

If the response takes too long to compute, the server will answer with a 202 containing a request id that can be used to call the REST endpoint /1/public/pending-request to obtain the response to the original request.

HTTP/1.1 202 Accepted
Content-Type: text/plain; charset=utf-8

2c2d8351-cddf-4eca-9b97-436713773168
View invalid:

If the view is invalid, the server will return a 400 response, with a list of errors in the response.

View not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"viewNotFound", "message":"View not found" }

Execute a view

Request:

POST /1/public/query-view/<name>

Body:

empty

Example:
curl -X GET <executor-url>/1/public/query-view/helloworld \
    --header "Authorization: Bearer <access token>"

Optional HTTP headers:

Name

Type

Description

X-RAW-output-format

string

Output format of the query, accepted values are:
“json”, “hjson”, “csv”, “sqlite”,
“parquet-avro”.
Default “hjson”.

X-RAW-windows-line-ending

boolean

If “true” output with windows line ending \n\r
if “false” output with unix line ending \n.
Default “false”.

X-RAW-max-bytes

int

Maximum number of bytes in the response

Response
200 OK:
HTTP/1.1 200 OK
Content-Type: application/octet-stream

"Hello"
"World"
View Invalid:

In case of semantic errors or runtime errors in the view, the server will return a 400 response, with a list of errors in the response.

408 Request Timeout:

The request took too long to execute and was cancelled. This error likely not retriable, as execution of the same request is likely to once again exceed the time limits of the server. As an alternative, use WebSockets to execute the request, which has no time bounds restriction for execution.

Delete a view

Request:

DELETE /1/public/views/<name>

Body:

empty

Example:
curl -X DELETE  <executor-url>/1/public/views/helloworld \
  --header "Authorization: Bearer <access token>"
Response
204 OK:

If the view was deleted the server responds with a 204 No Content.

HTTP/1.1 204 No Content
View not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"viewNotFound", "message":"View not found" }

Materialized Views

Create a new materialized view

Request:

PUT /materialized-views/<name>

MediaType:

text/plain

Body contains the query in UTF-8 encoding.

Example:
curl -X PUT <executor-url>/1/public/materialized-views/helloworld \
  --header "Content-Type: text/plain" \
  --header "Authorization: Bearer <access token>" \
  --data 'SELECT * FROM ["Hello", "World"]'
Response
200 OK:

If the view is valid the server will return a json with the type of the materialized view.

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{ "type": "collection(string)" }
202 Accepted:

If the response takes too long to compute, the server will answer with a 202 containing a request id that can be used to call the REST endpoint /1/public/pending-request to obtain the response to the original request.

HTTP/1.1 202 Accepted
Content-Type: text/plain; charset=utf-8

2c2d8351-cddf-4eca-9b97-436713773168
Invalid view:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{
   "validationErrors":[
      {
         "message":"bb is not declared",
         "positions":[{
            "begin":{ "line":1, "column":1 },
            "end":{ "line":1, "column":3 }
         }]
      }
   ],
   "runtimeError":null
}
View name is invalid:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"viewNameInvalid", "message":"View name is invalid" }
View already exists:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"viewAlreadyExists", "message":"name already exists as a view" }
Materialized view already exists:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"materializedViewAlreadyExists", "message":"name already exists as a materialized view" }

List materialized views

Request:

GET /1/public/materialized-views?pattern=<pattern>

Query parameters:
  • pattern - search pattern following the format used by the SQL LIKE operator (eg., % for zero or more characters, _ for single character)

Body:

empty

Example:
curl <executor-url>/1/public/materialized-views \
    --header "Authorization: Bearer <access token>" \
    -G --data-urlencode "pattern=hello%"
Response
200 OK:
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

[
  {
    "name": "helloworld",
    "query": "SELECT * FROM [\"Hello\", \"World\"]",
    "statistics": null,
    "createdAt": "2017-11-09T09:54:28.599",
    "updatedAt": "2017-11-09T09:54:28.599",
  }
]

Retrieve details of a materialized view

Request:

GET /1/public/materialized-views/<name>

Body:

empty

Example:
curl <executor-url>/1/public/materialized-views/helloworld \
  --header "Authorization: Bearer <access token>"
Response
200 OK:
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
    "name": "helloworld",
    "query": "SELECT * FROM [\"Hello\", \"World\"]",
    "statistics": null,
    "createdAt": "2017-11-09T09:54:28.599",
    "updatedAt": "2017-11-09T09:54:28.599",
}
View not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"viewNotFound", "message":"View not found" }

Validate a materialized view

Request:

GET /1/public/materialized-view–validate/<name>

Body:

empty

Example:
curl <executor-url>/1/public/materialized-view-validate/helloworld \
  --header "Authorization: Bearer <access token>"
Response
200 OK:

Returns a JSON object with the result type of the view.

HTTP/1.1 200 OK
Content-Type: application/json

{
    "declarations": {},
    "type": "collection(string)",
    "comment": null
}
202 Accepted:

If the response takes too long to compute, the server will answer with a 202 containing a request id that can be used to call the REST endpoint /1/public/pending-request to obtain the response to the original request.

HTTP/1.1 202 Accepted
Content-Type: text/plain; charset=utf-8

2c2d8351-cddf-4eca-9b97-436713773168
Materialized view invalid:

If the view is invalid, the server will return a 400 response, with a list of errors in the response.

Materialized view not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"viewNotFound", "message":"View not found" }

Delete a materialized view

Request:

DELETE /1/public/materialized-views/<name>

Body:

empty

Example:
curl -X DELETE  <executor-url>/1/public//materialized-views/helloworld \
  --header "Authorization: Bearer <access token>"
Response
204 OK:

If the view was deleted the server responds with a 204 No Content.

HTTP/1.1 204 No Content
View not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"viewNotFound", "message":"View not found" }

Packages

Create a package

Request:

PUT /1/public/packages/<name>

MediaType:

text/plain

Body contains the package code in UTF-8 encoding.

Example:
curl -X PUT <executor-url>/1/public/packages/pkg1 \
  --header "Content-Type: text/plain" \
  --header "Authorization: Bearer token" \
  --data  'func1(a: int, b: int) := a + b;
           pkg_data := ["Hello", "World"];
           pkg_data'
Response
200 OK:
{"type":"collection(string)"}
202 Accepted:

If the response takes too long to compute, the server will answer with a 202 containing a request id that can be used to call the REST endpoint /1/public/pending-request to obtain the response to the original request.

HTTP/1.1 202 Accepted
Content-Type: text/plain; charset=utf-8

2c2d8351-cddf-4eca-9b97-436713773168
Invalid package:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{
   "validationErrors":[
      {
         "message":"bb is not declared",
         "positions":[{
            "begin":{ "line":1, "column":1 },
            "end":{ "line":1, "column":3 }
         }]
      }
   ],
   "runtimeError":null
}
Package already exists:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"packageAlreadyExists", "message":"Package already exists" }
Package name is invalid:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"packageNameInvalid", "message":"Package name is invalid" }

List packages

Request:

GET /1/public/packages?pattern=<pattern>

Query parameters:
  • pattern - search pattern following the format used by the SQL LIKE operator (eg., % for zero or more characters, _ for single character)

Body:

empty

Example:
curl <executor-url>/1/public/packages \
    --header "Authorization: Bearer <access token>" \
    -G --data-urlencode "pattern=pk%"
Response
200 OK:
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

[
  {
    "name": "pkg1",
    "query": "SELECT * FROM [\"Hello\", \"World\"]",
    "createdAt": "2017-11-09T09:54:28.599"
  }
]

Retrieve details of a package

Request:

GET /1/public/packages/<name>

Body:

empty

Example:
curl <executor-url>/1/public/packages/test \
  --header "Authorization: Bearer <access token>"
Response
200 OK:
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
  "name": "helloworld",
  "query": "SELECT * FROM [\"Hello\", \"World\"]",
  "createdAt": [2017, 11, 09, 09, 54, 8, 599000],
  "updatedAt": null
}
Package not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"packageNotFound", "message":"Package not found" }

Validate a package

Request:

GET /1/public/package-validate/<name>

Body:

empty

Example:
curl <executor-url>/1/public/package-validate/pkg1 \
  --header "Authorization: Bearer <access token>"
Response
200 OK:

Returns a JSON object with the result type of this package.

HTTP/1.1 200 OK
Content-Type: application/json

{
    "declarations": {
        "func1": [
            {
                "parameters": [
                    {
                        "identifier": "a",
                        "type": "int",
                        "required": true
                    },
                    {
                        "identifier": "b",
                        "type": "int",
                        "required": true
                    }
                ],
                "outputType": "int",
                "comment": null
            }
        ],
        "pkg_data": [
            {
                "parameters": null,
                "outputType": "collection(string)",
                "comment": null
            }
        ]
    },
    "type": "collection(string)",
    "comment": null
}
202 Accepted:

If the response takes too long to compute, the server will answer with a 202 containing a request id that can be used to call the REST endpoint /1/public/pending-request to obtain the response to the original request.

HTTP/1.1 202 Accepted
Content-Type: text/plain; charset=utf-8

2c2d8351-cddf-4eca-9b97-436713773168
Package invalid:

If the package is invalid, the server will return a 400 response, with a list of errors in the response.

Package not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"packageNotFound","message":"executor error: package not found"}

Execute a package

Request:

GET /1/public/query-package/<name>

Body:

empty

Example:
curl -X GET <executor-url>/1/public/query-package/pkg1 \
    --header "Authorization: Bearer <access token>"
Optional HTTP headers

Name

Type

Description

X-RAW-output-format

string

Output format of the query, accepted values are:
“json”, “hjson”, “csv”, “sqlite”,
“parquet-avro”.
Default “hjson”.

X-RAW-windows-line-ending

boolean

If “true” output with windows line ending \n\r
if “false” output with unix line ending \n.
Default “false”.

X-RAW-max-bytes

int

Maximum number of bytes in the response

Response
200 OK:
HTTP/1.1 200 OK
Content-Type: application/json
 [ "Hello",  "World"]
Package Invalid:

In case of semantic errors or runtime errors in the package, the server will return a 400 response, with a list of errors in the response.

408 Request Timeout:

The request took too long to execute and was cancelled. This error likely not retriable, as execution of the same request is likely to once again exceed the time limits of the server. As an alternative, use WebSockets to execute the request, which has no time bounds restriction for execution.

Query a declaration inside a package

Request:

GET /1/public/query-package/<package-name>/<declaration-name>

Body:

empty

Example:
curl -X GET <executor-url>/1/public/query-package/pkg1/foo \
    --header "Authorization: Bearer <access token>"

If the declaration is a function you can pass the arguments as a query string in the URL (URL parameters).

Example:
curl -X GET <executor-url>/1/public/query-package/pkg1/bar?arg1=10 \
    --header "Authorization: Bearer <access token>"
Optional HTTP headers

See Optional HTTP headers in Execute a package.

Response
200 OK:
HTTP/1.1 200 OK
Content-Type: application/json
 [ "Hello",  "World"]

For extra information see Response in the section “Execute a package”.

Delete a package

Request:

DELETE /1/public/packages/<name>

Body:

empty

Example:
curl -X DELETE  <executor-url>/1/public/packages/helloworld \
  --header "Authorization: Bearer <access token>"
Response
204 OK:

If the package was deleted the server responds with a 204 No Content.

HTTP/1.1 204 No Content
Package not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"packageNotFound", "message":"Package not found" }

OpenAPI

Get the OpenAPI json definition for all rest endpoints of a package.

Request:

GET /1/public/openapi/v3/package/<name>

Body:

empty

Example:
curl -X GET  <executor-url>/1/public/openapi/v3/package/pkg1 \
  --header "Authorization: Bearer <access token>"
Response
200 OK:

The server returns a json document describing the rest endpoints of the package following the as defined in the OpenAPI specification.

Package not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{ "code":"packageNotFound", "message":"Package not found" }
Invalid package:
  • See Create Package -> Response -> Invalid package

Resume pending request

Request:

POST /1/public/pending-request/

MediaType:

text/plain

Tries to obtain the response of a pending request, which was suspended due to validation taking longer than a certain threshold.

Example:
curl -X POST <executor-url>/1/public/pending-requests \
  --header "Authorization: Bearer <access token>" \
  --data '2c2d8351-cddf-4eca-9b97-436713773168'
Response
200 OK:

If the original request completed, return the response.

HTTP/1.1 200 OK
202 Accepted:

If the original request did not complete, the server will answer again with a 202 containing the same request-uuid.

HTTP/1.1 202 Accepted
Content-Type: text/plain; charset=utf-8

2c2d8351-cddf-4eca-9b97-436713773168

Query Logs

Search/List Query Logs

Request:

GET /1/public/query-logs

Lists top level query-log information. Searching of results can be made with the optional url arguments listed bellow.

Optional URL Arguments:
  • n - Number of log entries to fetch.

  • start - Start timestamp of the search.

  • end - End timestamp of the search.

  • state - State of the query.

  • keyword - Keyword for string search in the query code.

Example:
curl -X GET '<executor-url>/1/public/query-logs?n=3&start=2021-04-03T00:00&end=2021-05-03T00:00' \
  --header "Authorization: Bearer <access token>"

Note

start and end timestamps have to be in the ISO 8601 date and time format YYYY-MM-DDThh:mm:ss. For example “2021-05-07T10:31:03”

Response
200 OK:
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

[
   {
      "id":"222b2063-5004-4abe-87bc-0c0c2338bac1-8",
      "query":"select * from products",
      "createdAt":[2021,4,30,18,27,33,714933000],
      "state":null,
      "detail":"output-format: raw",
      "updatedAt":null
   },
   {
      "id":"222b2063-5004-4abe-87bc-0c0c2338bac1-7",
      "query":"read(\"s3://raw-tutorial/ipython-demos/patients.xml\")",
      "createdAt":[2021,4,30,18,27,29,866162000],
      "state":null,
      "detail":"output-format: raw",
      "updatedAt":null
   }
]

Get Query Log Entries

Request:

GET /1/public/query-logs/entries/

Fetches query-log entries for a certain query-id.

URL Arguments:
  • id - Id of the query to fetch the entries.

Example:
curl -X GET '<executor-url>/1/public/query-logs/entries?id=222b2063-5004-4abe-87bc-0c0c2338bac1-8' \
  --header "Authorization: Bearer <access token>"
Response
200 OK:
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
    "id":"222b2063-5004-4abe-87bc-0c0c2338bac1-8",
    "query":"read(\"s3://raw-tutorial/machine_log.log\")",
    "createdAt":[2021,4,30,18,27,33,714933000],
    "updatedAt":null,
    "state":null,
    "detail":"output-format: raw",
    "entries":[
        {
           "level":"DEBUG",
           "message":"Job group id: program-user-uid-222b2063-5004-4abe-87bc-0c0c2338bac1-8",
           "internalInfo":null,
           "timestamp":[2021,4,30,18,27,35,814933000]
        },
        {
           "level":"INFO",
           "message":"Starting to write output...",
           "internalInfo":null,
           "timestamp":"timestamp":[2021,4,30,18,27,35,814933000]
        },
        {
           "level":"TRACE",
           "message":"Compiler timing for emit: 1019 ms",
           "internalInfo":null,
           "timestamp":"timestamp":[2021,4,30,18,27,35,814933000]
        }
    ]
}

Credentials Service

S3 Buckets

Register a new S3 bucket

Request:

POST /1/public/s3buckets

Arguments:
  • name - Name of S3 bucket

  • credentials - Optional AWS credentials.
    • accessKey -

    • secretKey -

Example:
curl -X POST <creds-url>/1/public/s3buckets \
  --header "Authorization: Bearer <access token>" \
  --header "Content-Type: application/json" \
  --data '{
  "name": "acme-logs",
  "region": "eu-west-1",
  "credentials": {
    "accessKey": "<access key>",
    "secretKey": "<secret key>"
  }
}'
Response
201 Created:

empty

S3 credentials already exists:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"s3CredentialsAlreadyExists","message":"S3 credentials already exists"}

List registered S3 buckets

Request:

GET /1/public/s3buckets

Example:
curl <creds-url>/1/public/s3buckets \
  --header "Authorization: Bearer <access token>"

Response

200 OK:
["acme-logs","raw-tutorial"]

Retrieve details of a S3 bucket

Request:

GET /1/public/s3buckets/<bucketName>

Example:
curl <creds-url>/1/public/s3buckets/acme-logs \
  --header "Authorization: Bearer <access token>"
Response
200 OK:
{
  "name": "acme-logs",
  "credentials": {
    "accessKey": "<access key>",
    "secretKey": "<secret key>"
  }
}
S3 credentials not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"s3CredentialsNotFound","message":"S3 credentials not found"}

Delete a S3 bucket

Request:

DELETE /1/public/s3buckets/<bucketName>

Example:
curl -X DELETE <creds-url>/1/public/s3buckets/acme-logs \
  --header "Authorization: Bearer <access token>"
Response
204 No Content:

empty

S3 credentials not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"s3CredentialsNotFound","message":"S3 credentials not found"}

RDBMS Servers

Register a new RDBMS server databases

Request:

POST /1/public/rdbmsservers

Arguments:
  • name - Unique name used to refer to the database within raw.

  • vendor - Database vendor. Currently only postgresql is supported.

  • host - Host name of the database server.

  • port - The port of the database server.

  • database - Name of the database

  • username - [Optional] Used for authenticating with the database server.

  • password - [Optional]

Example:
curl -X POST <creds-url>/1/public/rdbmsservers \
  --header "Authorization: Bearer <access token>" \
  --header "Content-Type: application/json" \
  --data '{
  "name": "db1",
  "vendor": "postgresql",
  "host": "localhost",
  "port": 5432,
  "database": "db",
  "username": "john",
  "password": "1234"
}'
Response
201 Created:

empty

Database already exists:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"databaseCredentialsAlreadyExists","message":"database credentials already exists"}

List registered RDBMS databases

Request:

GET /1/public/rdbmsservers

Example:
curl <creds-url>/1/public/rdbmsservers \
  --header "Authorization: Bearer <access token>"
Response
200 OK:
["db1", "db2"]

Retrieve details of a RDBMS databases

Request:

GET /1/public/rdbmsservers/<dbName>

Example:
curl <creds-url>/1/public/rdbmsservers/db1 \
  --header "Authorization: Bearer <access token>"
Response
200 OK:
{
     "type": "postgresql",
     "host":"localhost","
     port":1234,
     "database":"db",
     "username":"user",
     "password":"passwd"
 }
Database not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"databaseCredentialsNotFound","message":"database credentials not found"}

Delete a RDBMS Server

Request:

DELETE /1/public/rdbmsservers/<serverName>

Example:
curl -X DELETE <creds-url>/1/public/rdbmsservers/db1 \
  --header "Authorization: Bearer <access token>"
Response
204 No Content:

empty

Database not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"databaseCredentialsNotFound","message":"database credentials not found"}

Dropbox tokens

Register dropbox token

Registers a dropbox token, failing if a token is already registered. The token should have been obtained from calling the /oauth2/token endpoint of the Dropbox API.

Request:

POST /1/public/dropbox-tokens

Arguments:
  • accessToken - The access token.

  • tokenType - Always “bearer”.

  • uid - The user id.

Example:
curl -X POST <creds-url>/1/public/dropbox-tokens \
  --header "Authorization: Bearer <access token>" \
  --header "Content-Type: application/json" \
  --data '{
  "accessToken": "n5HcoSK_1pUAAAAAAAAMpeqUU5Cuzlm9M0kA3eG4HDNrsgKSFHtyzyrG4Eymz5_W",
  "tokenType": "bearer",
  "uid":"246215797"
}'
Response
201 Created:

empty

Dropbox token already exists:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"dropboxTokenAlreadyExists","message":"dropbox token already exists"}

Get dropbox token

Request:

GET /1/public/dropbox-tokens

Body:

empty

Example:
curl <creds-url>/1/public/dropbox-tokens \
  --header "Authorization: Bearer <access token>"
Response
200 OK:
{
  "accessToken": "n5HcoSK_1pUAAAAAAAAMpeqUU5Cuzlm9M0kA3eG4HDNrsgKSFHtyzyrG4Eymz5_W",
  "tokenType": "bearer",
  "uid": "246215797"
}
Dropbox token not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"dropboxTokenNotFound","message":"dropbox token not found"}

Delete dropbox token

Request:

DELETE /1/public/dropbox-tokens

Example:
curl -X DELETE <creds-url>/1/public/dropbox-tokens \
  --header "Authorization: Bearer <access token>"
Response
204 No Content:

empty

Dropbox token not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"dropboxTokenNotFound","message":"dropbox token not found"}

Protected HTTP urls

Register a new HTTP credentials

Request:

POST /1/public/http-creds

Arguments:
  • prefixUrl - base protected url.

  • credentials - format depends of the type of credentials to register (see table bellow).

Example:
curl -X POST <creds-url>/1/public/http-creds \
  --header "Authorization: Bearer <access token>" \
  --header "Content-Type: application/json" \
  --data '{
            "prefixUrl": "https://www.protected.com",
            "credentials": {
                "type": "oauth-token",
                "token": "my-token"
            }
         }'

Supported credentials:

Type

Parameters

Name

Description

Example

Basic Auth

type

Defines type of credentials,
basic-auth for basic auth credentials

“basic-auth”

user

User name

“john_doe”

password

Password

“my_password”

Oauth client

type

oauth-client for oauth client credentials

“oauth-client”

clientId

client id

“my_client_id”

clientSecret

client secret

“my_secret”

tokenUrl

url to get oauth token

http://oauth.validate

useBasicAuth

Use basic auth style authentication
in get-token flow(optional)

true

Oauth token

type

oauth-token for oauth token credentials

“oauth-token”

token

Oauth token

“my_token”

refreshToken

Refresh token (optional)

“my_refresh_token”

tokenUrl

url to refresh token (optional)

http://oauth.validate

Response
201 Created:

empty

HTTP credentials already exists:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"httpCredentialsAlreadyExists","message":"HTTP credentials already exists"}

List registered HTTP credentials

Request:

GET /1/public/http-creds/

Example:
curl <creds-url>/1/public/http-creds \
  --header "Authorization: Bearer <access token>"
Response
200 OK:
["http://www.protected.com"]

Retrieve details of HTTP credentials

Request:

GET /1/public/http-creds?name=<name>

Example:
curl <creds-url>/1/public/http-creds?name=http%3A%2F%2Fwww.protected.com \
  --header "Authorization: Bearer <access token>"
Response
200 OK:
{
     "prefixUrl":"http://www.protected.com",
     "credentials":{
         "type": "oauth-token",
         "token": "token",
         "refreshToken":null,
         "tokenUrl": null
     }
 }
HTTP credentials not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"httpCredentialsNotFound","message":"HTTP credentials not found"}

Delete HTTP Credentials

Request:

DELETE /1/public/http-creds?name=<name>

Example:
curl -X DELETE  <creds-url>/1/public/http-creds?name=http%3A%2F%2Fwww.protected.com \
  --header "Authorization: Bearer <access token>"
Response
204 No Content:

empty

HTTP credentials not found:
HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=utf-8

{"code":"httpCredentialsNotFound","message":"HTTP credentials not found"

Storage Service

Get number of cache entries

Request:

GET /1/public/number_cache_entries

Example:
curl -X GET <storage-url>/1/public/number_cache_entries \
  --header "Authorization: Bearer <access token>"

Response

200 OK:

Returns an integer with the number of cache entries.

123

Storage Service

Get cache entries

Request:

GET /1/public/cache_entries

Example:
curl -X GET <storage-url>/1/public/cache_entries \
  --header "Authorization: Bearer <access token>" \

Response

200 OK:

Returns a JSON array with the cache entries.

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

[
  {
     "uid":"user-uid",
     "key":"cache:e00e7a1fa2545dfa005a764920a653374f55e6832306890f53a72266d0465836",
     "url":"file:/rawcache/user/raw/2/cache/user-uid/cache_e00e7a1fa2545dfa005a764920a653374f55e6832306890f53a72266d0465836/795fef27-99f1-4cad-9f2a-2efeed56f833",
     "cdate":1620296060.756161000,
     "size":22580,
     "lastAccessed":null
  },
  {
     "uid":"user-uid",
     "key":"cache:6edaf64e8e323eb7b6f310845a0882d80ca129f4e921c480f56870926c8b22a9",
     "url":"file:/rawcache/user/raw/2/cache/user-uid/cache_6edaf64e8e323eb7b6f310845a0882d80ca129f4e921c480f56870926c8b22a9/eb537024-6b14-4eb4-b344-0297aa051a40",
     "cdate":1620295320.623912000,
     "size":22580,
     "lastAccessed":null
  }
[

Get Quota Info

Request:

GET /1/public/cache_entries

Example:
curl -X GET <storage-url>/1/public/quota_info \
  --header "Authorization: Bearer <access token>" \

Response

200 OK:

Returns a JSON object with the quota info.

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
   "totalSize":10000000000,
   "cachesSize":203266,
   "temporaryLocationsSize":0,
   "tablesSize":0,
   "tableIndexesSize":0,
   "materializedViewsSize":0
}

Invalidate Caches

Request:

POST /1/public/cache_entries

Example:
curl -X GET <storage-url>/1/public/invalidate_caches \
  --header "Authorization: Bearer <access token>" \

Response

204 OK:

If the caches were invalidated the server responds with a 204 No Content.