Skip to main content

Accessing data from web services

Learn how to access data from web services.

info

If you want to try this example, you can deploy the following endpoint:

Tutorial 12 - Querying web services
Learn how to perform HTTP requests including passing HTTP headers, query parameters and authentication options.

Usage:

/tutorial/12-query-web-services

Quick Summary​

  • The HTTP package builds locations which are then used in conjunction with JSON or CSV packages to parse the result.
  • Paginated results can be handled with recursive functions.

Querying Web Services​

Snapi supports reading data directly from web services. For this, you will need to use the HTTP package, and usually the JSON or CSV packages to parse the result.

To do so, you must first build an HTTP location to make the request. For example, the following code builds an HTTP GET request. The args argument takes a list of name/value for the query parameters, while the headers argument receives the list of headers to pass in the request.

Http.Get(
    "https://example.org/web-service",
    args = [{"parameter", "value"}],
    headers = [{"Accept", "text/json"}]
)

Calling Http.Get does not directly trigger the request. Instead, it builds a location. The request is only executed when data is read using e.g. Json.Read or Csv.Read, as in:

let
    location = Http.Get(
        "https://example.org/web-service",
        args = [{"parameter", "value"}],
        headers = [{"Accept", "text/json"}]
    )
in
    Json.Read(location, type collection(int))

The program above does an HTTP GET request, passing query parameters and headers, and parses the output as a JSON structure. This assumes the response of the HTTP web service was 200. If you would like to configure "valid codes", refer to the expectedStatus optional parameter of the Http methods: the default value is [200], which means only the 200 code is accepted by default to indicate a success. Every other response will trigger an error, which can also be handled: refer to the Error Handling guide for more information.

Alternatively, you can also use Http.Read which returns a record with the response body and the status code.

Handling Pagination​

Web services can produce paginated results. For instance, a JIRA server exposes multiple REST APIs that offer programmatic access to its database. The responses are in JSON.

A call to JIRA's search REST API returns the set of issues matching a given search criteria. Its results are paginated.

info

If you want to try this example, you can deploy the following endpoint:

Querying a JIRA server
Learn how to query the JIRA REST APIs.

Usage:

/jira

For instance, with query fixVersion=9.0.0 query, jql returns all JIRA issues fixed in version 9.0.0:

[
  {
    "key": "JRASERVER-73294",
    "summary": "Update product documentation for Zero Downtime Upgrade (ZDU) related steps",
    "status": "Closed",
    "resolutiondate": "2022-11-22T14:25:58.000+0000"
  },
  {
    "key": "JRASERVER-74200",
    "summary": "Improve the Archiving a project and Archiving an issue documentation to account for the need of a re-index to assertively decrease Index size (and disk space)",
    "status": "Closed",
    "resolutiondate": "2022-11-22T14:18:20.000+0000"
  },
  {
    "key": "JRASERVER-74506",
    "summary": "Product document Running Jira applications over SSL or HTTPS has incorrect step for command line installation",
    "status": "Closed",
    "resolutiondate": "2022-11-21T10:05:10.000+0000"
  },
  ...
  ...
  ...
  {
    "key": "JRASERVER-72995",
    "summary": "Jira webhooks stop working after \"I/O reactor terminated abnormally\" error",
    "status": "Closed",
    "resolutiondate": "2022-03-31T10:35:40.000+0000"
  },
  {
    "key": "JRASERVER-73252",
    "summary": "Restarting the database causes cache replication issues",
    "status": "Closed",
    "resolutiondate": "2022-03-31T10:32:55.000+0000"
  }
]

The whole set of paginated results can be retrieved by looping and calling search with an increasing startAt argument. This should be done until the issues array returned is empty.

Here is an implementation to handle pagination. It internally uses a recursive function: note the use of let rec to indicate the recursive call. Finally, the results obtained are unioned together into a single collection:

jql(projectURL: string, query: string) =
    // recursive function (annotated with rec)
    let rec issues(startAt: int = 0): collection(issueType) =
        let reports = Json.Read(Http.Get(projectURL + "/rest/api/latest/search", args=[{"jql", query}, {"startAt", String.From(startAt)}]), jqlType)
        in
            if Collection.Count(reports.issues) == 0
            then reports.issues
            else Collection.Union(reports.issues, issues(startAt + 50)) // recursive call
    in issues(0)