Definitions
| Term | Definition |
|---|---|
| AWS CLI | The Amazon Web Services Command Line Interface for controlling Amazon Cloud Services. |
| Folder | A Folder is a namespace inside a Prefix. The folder name is typically set by the task . |
| GBDX Dashboard | A Graphical User Interface(GUI) for managing user accounts and account information. |
| GBDXtools | An open-source Python-based project for using The GBDX Platform. |
| GUI | Graphical User Interface |
| Object | Individual files stored on S3 are called Objects. Objects are prepended with the user's Prefix and folder name (when there is one). |
| Prefix | A Prefix is a storage location within the main bucket. It may also be referred to as a directory. Prefixes are associated with account IDs. The Prefix name is typically the account ID. Only users that belong to the account have access to the Prefix. |
| S3 | Amazon's Cloud Storage Service |
| S3 Browser | The Amazon Web Services S3 Management Console GUI. |
| S3 Bucket | All GBDX customer files are stored an S3 customer bucket. The name of the bucket is gbd-customer-data. |
| S3 Location | GBDX documentation uses the term "S3 location" to describe the location where an account's output data is stored. The S3 location is a "prefix" within the GBDX S3 customer bucket. |
| S3 Storage Service | A set of GBDX APIs used to request temporary credentials |
| Worlkflow | A series of tasks (algorithms) chained together and run to produce a specified output. |
Overview
When a workflow is run on on The GBDX Platform, the output data is saved to an S3 location. This location is a Prefix within the the GBDX Customer bucket. There are several ways to access and manage the contents of your S3 location, described in this document.
The full path to your S3 contents is:
s3://{s3 bucket}/{prefix}/{folder}/{object}
S3 Management Tools
The contents of a customer's S3 location can be accessed and managed using the following tools:
| Tool | Description | Dependencies | Tutorial |
|---|---|---|---|
| AWS CLI | A command line interface that supports downloading a file or the contents of a folder, uploading, and deleting files. Use the S3 Storage Service to get credentials to the S3 location first. | S3 Storage Service | AWS CLI tutorial |
| AWS S3 Browser | An AWS S3 management console GUI that lets you download a file, delete files, upload files, and more. | The GBDX Dashboard, located at https://dashboard.geobigdata.io. Log in to the dashboard with GBDX credentials (email address and password). | S3 Browser tutorial |
| GBDXtools | A python-based project that supports downloading, deleting, and uploading files from S3. | GBDXtools setup or GBDX Notebooks | GBDXtools tutorial](doc:manage-s3-contents-with-gbdxtools) |
| S3 Storage Service | A set of API endpoints that provide temporary credentials to an S3 location. The temporary credentials are used with the AWS CLI. The service also supports getting the size of a prefix and getting a download url for an object. | An API client; i.e. Postman | S3 Storage Service |
Supported Features
This table shows the actions that are supported by each tool.
| AWS S3 Browser | S3 Storage Service | AWS CLI | GBDXtools | |
|---|---|---|---|---|
| Delete a file | X | X | X | X |
| Delete a folder or directory | X | X | X | |
| Download a file | X | X | X | |
| Download the contents of a directory | X | X | ||
| Download file as . . . | X | |||
| Get a download url for an Object | X | X | ||
| Get the size of a Prefix | X | X | ||
| Upload a file | X | X | X | |
| Upload a folder or Directory | X | X | X |
S3 Management Tools Descriptions
AWS CLI
The AWS CLI lets you download, delete, and upload files or directories. To use the AWS CLI, it must be installed, and you must have temporary credentials from the S3 Storage Service.
Tutorial: Manage S3 Contents with AWS CLI
AWS CLI Installation: AWS CLI
AWS S3 Browser
A link to the AWS S3 Browser is available from the Accounts section of the GBDX Dashboard. When you sign into the Dashboard using your GBDX account credentials, the link will take you to the S3 contents for your account.
The AWS S3 Browser allows you to:
- Download a file (individual files only)
- Upload a file to a folder
- Delete a file
- Bulk delete files or folders
*Note: Bulk download is not supported by the AWS S3 Browser. Use the AWS CLI or GBDXtools to download the contents of a directory or folder.
Tutorial: Manage S3 Contents with the S3 Browser
GBDXtools
If you're a GBDXtools user, you can manage S3 contents using Python scripts. Downloading, deleting, and uploading files and directories are supported.
Tutorial: Manage S3 Contents with GBDXtools
S3 Storage Service
The S3 Storage Service is a set of API endpoints that allow you to:
- Get temporary credentials to the S3 location. You can then use these credentials to manage your S3 contents with the AWS CLI.
- Get the size of your S3 Prefix
- Get a download url for a file