GBDX

S3 Access Course

This document describes how to access and manage the contents of an S3 location.

Last Updated: April 11, 2019

Definitions

Term Definition
AWS CLI The Amazon Web Services Command Line Interface for controlling Amazon Cloud Services.
Folder A Folder is a namespace inside a Prefix. The folder name is typically set by the task .
GBDX Dashboard A Graphical User Interface(GUI) for managing user accounts and account information.
GBDXtools An open-source Python-based project for using The GBDX Platform.
GUI Graphical User Interface
Object Individual files stored on S3 are called Objects. Objects are prepended with the user's Prefix and folder name (when there is one).
Prefix A Prefix is a storage location within the main bucket. It may also be referred to as a directory. Prefixes are associated with account IDs. The Prefix name is typically the account ID. Only users that belong to the account have access to the Prefix.
S3 Amazon's Cloud Storage Service
S3 Browser The Amazon Web Services S3 Management Console GUI.
S3 Bucket All GBDX customer files are stored an S3 customer bucket. The name of the bucket is gbd-customer-data.
S3 Location GBDX documentation uses the term "S3 location" to describe the location where an account's output data is stored. The S3 location is a "prefix" within the GBDX S3 customer bucket.
S3 Storage Service A set of GBDX APIs used to request temporary credentials
Worlkflow A series of tasks (algorithms) chained together and run to produce a specified output.

Overview

When a workflow is run on on The GBDX Platform, the output data is saved to an S3 location. This location is a Prefix within the the GBDX Customer bucket. There are several ways to access and manage the contents of your S3 location, described in this document.

The full path to your S3 contents is:

s3://{s3 bucket}/{prefix}/{folder}/{object}

S3 Management Tools

The contents of a customer's S3 location can be accessed and managed using the following tools:

Tool Description Dependencies Tutorial
AWS CLI A command line interface that supports downloading a file or the contents of a folder, uploading, and deleting files. Use the S3 Storage Service to get credentials to the S3 location first. S3 Storage Service AWS CLI tutorial
AWS S3 Browser An AWS S3 management console GUI that lets you download a file, delete files, upload files, and more. The GBDX Dashboard, located at https://dashboard.geobigdata.io. Log in to the dashboard with GBDX credentials (email address and password). S3 Browser tutorial
GBDXtools A python-based project that supports downloading, deleting, and uploading files from S3. GBDXtools setup or GBDX Notebooks GBDXtools tutorial](doc:manage-s3-contents-with-gbdxtools)
S3 Storage Service A set of API endpoints that provide temporary credentials to an S3 location. The temporary credentials are used with the AWS CLI. The service also supports getting the size of a prefix and getting a download url for an object. An API client; i.e. Postman S3 Storage Service

Supported Features

This table shows the actions that are supported by each tool.

AWS S3 Browser S3 Storage Service AWS CLI GBDXtools
Delete a file X X X X
Delete a folder or directory X X X
Download a file X X X
Download the contents of a directory X X
Download file as . . . X
Get a download url for an Object X X
Get the size of a Prefix X X
Upload a file X X X
Upload a folder or Directory X X X

S3 Management Tools Descriptions

AWS CLI

The AWS CLI lets you download, delete, and upload files or directories. To use the AWS CLI, it must be installed, and you must have temporary credentials from the S3 Storage Service.

Tutorial: Manage S3 Contents with AWS CLI
AWS CLI Installation: AWS CLI

AWS S3 Browser

A link to the AWS S3 Browser is available from the Accounts section of the GBDX Dashboard. When you sign into the Dashboard using your GBDX account credentials, the link will take you to the S3 contents for your account.

The AWS S3 Browser allows you to:

  • Download a file (individual files only)
  • Upload a file to a folder
  • Delete a file
  • Bulk delete files or folders

*Note: Bulk download is not supported by the AWS S3 Browser. Use the AWS CLI or GBDXtools to download the contents of a directory or folder.

Tutorial: Manage S3 Contents with the S3 Browser

GBDXtools

If you're a GBDXtools user, you can manage S3 contents using Python scripts. Downloading, deleting, and uploading files and directories are supported.
Tutorial: Manage S3 Contents with GBDXtools

S3 Storage Service

The S3 Storage Service is a set of API endpoints that allow you to:

  • Get temporary credentials to the S3 location. You can then use these credentials to manage your S3 contents with the AWS CLI.
  • Get the size of your S3 Prefix
  • Get a download url for a file

S3 Access Course


This document describes how to access and manage the contents of an S3 location.

Last Updated: April 11, 2019

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.