Skip to main content

S3

Learn how to store S3 credentials in RAW.

How to create​

AWS access keys are a combination of an access key ID and secret access key. Using the S3 tab from the 'Credentials' list, we are able to set up public or private S3 buckets that reference data we want to process. To set up an S3 bucket, select the 'S3' button from the 'Credentials' tab, and you will be presented with the following form:

S3 form

First, let's prepare a public bucket:

S3 public

Select Save and you can see the generated entry in the list of Credentials:

S3 generated

By selecting Test Access, you can test the accessibility of the Credential:

S3 accessible

The same process is followed for specifying private buckets. Again, after selecting S3, enter the bucket name, the respective region, and our AWS credentials, access key and access secret respectively:

S3 private form

After selecting Save, the generated entry can be visuzalised in the list of Credentials:

S3 private generated

Checking whether the new Credential can be accessed by RAW is accomplished by clicking on Test Access:

S3 private accessible

If the specified bucket does not exist or is not accessible, we receive the following error message:

S3 private not accessible

How to use from Snapi​

An example of usage can be found in the following code:

athlete(name: string) =
  let
     summer = Csv.InferAndRead("s3://rawlabs-private-test-data/summer_olympics.csv"),
     winter = Csv.InferAndRead("s3://rawlabs-public-test-data/winter_olympics.csv"),
     total = Collection.Union(summer, winter)
  in
    Collection.Filter(total, x -> x.Athlete == name)

As can be seen above, the function athlete creates a union from reading CSV files residing in a private and public S3 bucket respectively. This joint data is then used to search athletes by name.