{"_id":"561d578e9242920d00df9f47","parentDoc":null,"__v":54,"category":{"_id":"5601afa02499c119000faf19","pages":["5601b42b6811d00d00ceb49b","5601b63c9137690d003357cb","5601b7b49137690d003357d0","5601b82a81a9670d006d166b","5601b84350ee460d00022261","5601b8c281a9670d006d166e","5601b9146811d00d00ceb4a3","5601ba086811d00d00ceb4a4","5601baf650ee460d00022264","5601bf6d7916060d00760b00","561d578e9242920d00df9f47","561d57ec2d6a450d00f0512d","561d58449242920d00df9f53","56438be2f49bfa0d002f561e","56438e101ecf381700343c48","5643902708894c0d00031f14","564391d1c92c470d002deb03","564394529eebf70d00490d35","56f194243eb62a34003e9fef"],"project":"55faeacad0e22017005b8265","__v":19,"version":"55faeacad0e22017005b8268","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-09-22T19:44:32.525Z","from_sync":false,"order":10,"slug":"s3-storage-service-guide","title":"S3 Storage Service Guide"},"version":{"_id":"55faeacad0e22017005b8268","project":"55faeacad0e22017005b8265","__v":35,"createdAt":"2015-09-17T16:31:06.800Z","releaseDate":"2015-09-17T16:31:06.800Z","categories":["55faeacbd0e22017005b8269","55faf550764f50210095078e","55faf5b5626c341700fd9e96","55faf8a7825d5f19001fa386","560052f91503430d007cc88f","560054f73aa0520d00da0b1a","56005aaf6932a00d00ba7c62","56005c273aa0520d00da0b3f","5601ae7681a9670d006d164d","5601ae926811d00d00ceb487","5601aeb064866b1900f4768d","5601aee850ee460d0002224c","5601afa02499c119000faf19","5601afd381a9670d006d1652","561d4c78281aec0d00eb27b6","561d588d8ca8b90d00210219","563a5f934cc3621900ac278c","5665c5763889610d0008a29e","566710a36819320d000c2e93","56ddf6df8a5ae10e008e3926","56e1c96b2506700e00de6e83","56e1ccc4e416450e00b9e48c","56e1ccdfe63f910e00e59870","56e1cd10bc46be0e002af26a","56e1cd21e416450e00b9e48e","56e3139a51857d0e008e77be","573b4f62ef164e2900a2b881","57c9d1335fd8ca0e006308ed","57e2bd9d1e7b7220000d7fa5","57f2b992ac30911900c7c2b6","58adb5c275df0f1b001ed59b","58c81b5c6dc7140f003c3c46","595412446ed4d9001b3e7b37","59e76ce41938310028037295","5a009de510890d001c2aabfe"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"v1","version_clean":"1.0.0","version":"1"},"user":"55fae9d4825d5f19001fa379","project":"55faeacad0e22017005b8265","updates":[],"next":{"pages":[],"description":""},"createdAt":"2015-10-13T19:12:14.736Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":0,"body":"[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"S3 Storage Service Overview\"\n}\n[/block]\nGBDX stores ancillary data and derived products in an Amazon Web Services (AWS) S3 bucket. When a workflow is run on the GBDX platform, a task called \"StageDataToS3\" is typically the last task run by the workflow. This task takes the processed data and places it in the AWS S3 bucket.\n\nThe purpose of the GBDX S3 Storage Service is to allow users to access this data. The service provides the temporary credentials required to access a Prefix, Folder, or Object in the S3 bucket. \n\nIn this course, we'll provide an overview of the S3 Storage Service. When you've completed this course, go to \n\n[Lesson: Get S3 Bucket Contents](doc:lesson-access-the-data-in-your-s3-bucket)\n\nThe step-by-step tutorial will show you how to request temporary credentials, and how to use AWS CLI to access the contents of your S3 bucket.\n\n## AWS S3 Bucket \n\nThe full name of the S3 bucket is: s3://gbd-customer-data/{prefix}/{folder}/{object}\nTemporary credentials are required by AWS to access the S3 bucket. . Most commonly, you'll request access to the prefix, but it's possible to request access to a specific folder or object within the prefix. \n\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Resource\",\n    \"h-1\": \"Definition\",\n    \"0-0\": \"Prefix\",\n    \"0-1\": \"A Prefix is a storage location within the main bucket. Storage buckets are associated with account IDs. Only users that belong to the account have access to the Prefix.\",\n    \"1-0\": \"Folder\",\n    \"1-1\": \"A Folder is a namespace inside a Prefix. For example, /my/folder/is a valid folder name. All objects would have {Prefix}/my/folder/ at the start of their name. The folder name is typically your Account ID.\",\n    \"2-0\": \"Object\",\n    \"2-1\": \"Individual files stored on S3 are called Objects. Objects are prepended with the user's Prefix and folder name (when there is one).\"\n  },\n  \"cols\": 2,\n  \"rows\": 3\n}\n[/block]\n## GBDX S3 Storage Service \n\nThe S3 Storage Service is a set of API endpoints that allow users to request temporary credentials to an S3 bucket. You'll need an OAuth2 token to make API requests. \n\n### API Requests:\n[Get Temp S3 Creds for Prefix](doc:temp-credentials-for-a-prefix) \n[Update a Prefix](doc:update-a-prefix) \n[Get Prefix Size](doc:get-prefix-size) \n[Get temp creds-Folder](doc:get-temporary-credentials-for-a-folder) \n[Get temp creds-Object](doc:get-temporary-credentials-for-an-object) \n[Get Download URL for an object](doc:temp-download-url-for-an-object) \n\n### Response Codes\nA 404 error code means the requested Prefix, Folder, or Object does not exist and is required.\n\n### Temporary Credentials\nThe credentials provided by the GBDX S3 Storage Service are temporary and session-based. When a request is made for credentials, the system will return:\n\n* Bucket (The S3 Bucket is gbd-customer-data)\n* Prefix (The Prefix is typically your account ID)\n* S3_access_key\n* S3_secret_key\n* S3_session_token\n\n### Session Duration\nThe default duration that the credentials are valid is 3600 seconds. The requester can change the duration when making an API request. The valid range for duration is 900 seconds to 129600 seconds. \n\nDuration Type | Value\n--- | ---\nMinimum |900 seconds (.25 hours) \nDefault | 3600 seconds (1 hour) \nMaximum |129600 seconds (36 hours)\n\n\n### Step by Step Tutorial\n\nOur step-by-step tutorial walks you through the process of getting temporary S3 credentials and accessing the contents of an S3 bucket using AWS CLI. [Lesson: Get S3 Bucket Contents](doc:lesson-access-the-data-in-your-s3-bucket) \n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Sample Python Script\"\n}\n[/block]\nPython users can use our [Python Examples](doc:python-examples) to access your S3 Bucket. \n\nThese examples assume the user has a working knowledge of Python 2.7. For more information, see https://www.python.org/doc/\n\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Step-by-step Tutorial\"\n}\n[/block]\nThis tutorial explains how to request temporary credentials and how to use AWS CLI to access the contents of an S3 bucket. \n\n[Lesson: Get S3 Bucket Contents](doc:lesson-access-the-data-in-your-s3-bucket)","excerpt":"Overview of the S3 Storage Service","slug":"s3-storage-service-course","type":"basic","title":"S3 Storage Service Course"}

S3 Storage Service Course

Overview of the S3 Storage Service

[block:api-header] { "type": "basic", "title": "S3 Storage Service Overview" } [/block] GBDX stores ancillary data and derived products in an Amazon Web Services (AWS) S3 bucket. When a workflow is run on the GBDX platform, a task called "StageDataToS3" is typically the last task run by the workflow. This task takes the processed data and places it in the AWS S3 bucket. The purpose of the GBDX S3 Storage Service is to allow users to access this data. The service provides the temporary credentials required to access a Prefix, Folder, or Object in the S3 bucket. In this course, we'll provide an overview of the S3 Storage Service. When you've completed this course, go to [Lesson: Get S3 Bucket Contents](doc:lesson-access-the-data-in-your-s3-bucket) The step-by-step tutorial will show you how to request temporary credentials, and how to use AWS CLI to access the contents of your S3 bucket. ## AWS S3 Bucket The full name of the S3 bucket is: s3://gbd-customer-data/{prefix}/{folder}/{object} Temporary credentials are required by AWS to access the S3 bucket. . Most commonly, you'll request access to the prefix, but it's possible to request access to a specific folder or object within the prefix. [block:parameters] { "data": { "h-0": "Resource", "h-1": "Definition", "0-0": "Prefix", "0-1": "A Prefix is a storage location within the main bucket. Storage buckets are associated with account IDs. Only users that belong to the account have access to the Prefix.", "1-0": "Folder", "1-1": "A Folder is a namespace inside a Prefix. For example, /my/folder/is a valid folder name. All objects would have {Prefix}/my/folder/ at the start of their name. The folder name is typically your Account ID.", "2-0": "Object", "2-1": "Individual files stored on S3 are called Objects. Objects are prepended with the user's Prefix and folder name (when there is one)." }, "cols": 2, "rows": 3 } [/block] ## GBDX S3 Storage Service The S3 Storage Service is a set of API endpoints that allow users to request temporary credentials to an S3 bucket. You'll need an OAuth2 token to make API requests. ### API Requests: [Get Temp S3 Creds for Prefix](doc:temp-credentials-for-a-prefix) [Update a Prefix](doc:update-a-prefix) [Get Prefix Size](doc:get-prefix-size) [Get temp creds-Folder](doc:get-temporary-credentials-for-a-folder) [Get temp creds-Object](doc:get-temporary-credentials-for-an-object) [Get Download URL for an object](doc:temp-download-url-for-an-object) ### Response Codes A 404 error code means the requested Prefix, Folder, or Object does not exist and is required. ### Temporary Credentials The credentials provided by the GBDX S3 Storage Service are temporary and session-based. When a request is made for credentials, the system will return: * Bucket (The S3 Bucket is gbd-customer-data) * Prefix (The Prefix is typically your account ID) * S3_access_key * S3_secret_key * S3_session_token ### Session Duration The default duration that the credentials are valid is 3600 seconds. The requester can change the duration when making an API request. The valid range for duration is 900 seconds to 129600 seconds. Duration Type | Value --- | --- Minimum |900 seconds (.25 hours) Default | 3600 seconds (1 hour) Maximum |129600 seconds (36 hours) ### Step by Step Tutorial Our step-by-step tutorial walks you through the process of getting temporary S3 credentials and accessing the contents of an S3 bucket using AWS CLI. [Lesson: Get S3 Bucket Contents](doc:lesson-access-the-data-in-your-s3-bucket) [block:api-header] { "type": "basic", "title": "Sample Python Script" } [/block] Python users can use our [Python Examples](doc:python-examples) to access your S3 Bucket. These examples assume the user has a working knowledge of Python 2.7. For more information, see https://www.python.org/doc/ [block:api-header] { "type": "basic", "title": "Step-by-step Tutorial" } [/block] This tutorial explains how to request temporary credentials and how to use AWS CLI to access the contents of an S3 bucket. [Lesson: Get S3 Bucket Contents](doc:lesson-access-the-data-in-your-s3-bucket)