{"_id":"5873cf7aa90f73230023e2f2","project":"55faeacad0e22017005b8265","parentDoc":null,"user":"55fae9d4825d5f19001fa379","version":{"_id":"55faeacad0e22017005b8268","project":"55faeacad0e22017005b8265","__v":33,"createdAt":"2015-09-17T16:31:06.800Z","releaseDate":"2015-09-17T16:31:06.800Z","categories":["55faeacbd0e22017005b8269","55faf550764f50210095078e","55faf5b5626c341700fd9e96","55faf8a7825d5f19001fa386","560052f91503430d007cc88f","560054f73aa0520d00da0b1a","56005aaf6932a00d00ba7c62","56005c273aa0520d00da0b3f","5601ae7681a9670d006d164d","5601ae926811d00d00ceb487","5601aeb064866b1900f4768d","5601aee850ee460d0002224c","5601afa02499c119000faf19","5601afd381a9670d006d1652","561d4c78281aec0d00eb27b6","561d588d8ca8b90d00210219","563a5f934cc3621900ac278c","5665c5763889610d0008a29e","566710a36819320d000c2e93","56ddf6df8a5ae10e008e3926","56e1c96b2506700e00de6e83","56e1ccc4e416450e00b9e48c","56e1ccdfe63f910e00e59870","56e1cd10bc46be0e002af26a","56e1cd21e416450e00b9e48e","56e3139a51857d0e008e77be","573b4f62ef164e2900a2b881","57c9d1335fd8ca0e006308ed","57e2bd9d1e7b7220000d7fa5","57f2b992ac30911900c7c2b6","58adb5c275df0f1b001ed59b","58c81b5c6dc7140f003c3c46","595412446ed4d9001b3e7b37"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"v1","version_clean":"1.0.0","version":"1"},"category":{"_id":"5601aee850ee460d0002224c","__v":20,"project":"55faeacad0e22017005b8265","version":"55faeacad0e22017005b8268","pages":["56023786930fe1170074bd2c","561d53a09463520d00cd11ef","561d546d31d9630d001eb5d1","561d54af31d9630d001eb5d3","561d54e56386060d00e0601e","561d554d9463520d00cd11f2","564246059f4ed50d008be1af","5643712a0d9748190079defb","564372751ecf381700343c1e","5643742008894c0d00031ed3","5643747a0d9748190079df01","564375c988f3a60d00ac86b0","56437d0f0d9748190079df13","56437e83f49bfa0d002f560a","56437f7d0d9748190079df15","5643810508894c0d00031ef5","5643826f88f3a60d00ac86cb","564382de88f3a60d00ac86ce","56e07ba14685db1700d94873","56e08c9b903c7a29001d5352"],"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-09-22T19:41:28.703Z","from_sync":false,"order":8,"slug":"tasks-and-workflows-guide","title":"Tasks and Workflows Guide"},"__v":0,"updates":[],"next":{"pages":[],"description":""},"createdAt":"2017-01-09T17:59:22.000Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":3,"body":"[block:callout]\n{\n  \"type\": \"warning\",\n  \"title\": \"The new Task Versioning feature was released Thursday, January 12, 2017.\",\n  \"body\": \"All existing tasks were given a version number of \\\"0.0.1\\\". All new tasks will require a version number..\"\n}\n[/block]\n\n# Task Versioning\nTask versioning lets a task owner update an existing task without overwriting previous versions. Multiple versions of the same task can be registered in the GBDX task registry. Versions are identified by version number. Versioning a task is as simple as re-registering the task and incrementing the version number.\n\nWhen a task is registered on the GBDX platform, the task owner will give it a name and version number. To upgrade the task, the owner will re-register the task with the same name and increment the version number. Each new version must be incremented to a higher number. Task owners are discouraged from including a version number in the task name. \n\n##How to add a version number to a task\n\nFor more information about registering a task in the task registry, see [Register a task](doc:register-a-task-with-the-task-registry).\n\nThe version number is a property in the task description JSON schema. When you register a task, you'll provide a name, description, and version number as part of the task definition. Registering a task means sending a \"POST\" request with the task definition JSON in the request body. \n\nIn this example, this is a new task named \"test-success\". This is the first version of the task, and the version number is 0.0.1.\n\n    \n        \"name\": \"test-success\",\n        \"version\": \"0.0.1\",\n   \n  \nTo register a new version of the same task, the owner will re-register the task with the same name and an incremented version number. \n\nFor example:\n\t\t\n     \"name\": \"test-success\",\n     \"version\": \"0.0.2\",\n\nor \n\n     \"name\": \"test-success\",\n     \"version\": \"1.0.0\",\n\n  \n  * Names are not case-sensitive. \"Test-Success\" and \"test-success\" will be read as the same task name. \n\n  * All characters in the name must match. For example, \"test-success\" and \"test_success\" will be registered as separate tasks.\n       \n These rules must be followed when adding a version number to a task:\n      \n1. The task version numbering convention adheres to [Semantic Versioning](http://semver.org/) standards which are \n\n\tmajor.minor.patch.  \n\nIn the example version number \"0.0.1\", \n\n    major = 0\n    minor = 0\n    patch = 1\n\nWe encourage task owners to follow this convention:\n\nmajor = breaking change\nminor = feature change\npatch = bug fix\n\n2. The version number must be enclosed in quotes, like this: \n\t\n\t\"version\": \"0.0.2\",\n\t\n3. The version number can only be incremented by the owner of the task. The \"owner\" is the account, so any user associated with the account that owns the task can version it. \n\nIf a user not associated with the account attempts to register a task with the same name, they will get a 403: Forbidden error.\n\n## Example Task with Version Number\n\nTo register a task with an incremented version number, POST the JSON request body with the task definition to https://geobigdata.io/workflows/v1/tasks\n\nThis example includes the version number.\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"\\n     \\\"inputPortDescriptors\\\": [\\n        {\\n            \\\"required\\\": true,\\n            \\\"description\\\": \\\"A string input.\\\",\\n            \\\"name\\\": \\\"inputstring\\\",\\n            \\\"type\\\": \\\"string\\\"\\n        },\\n        {\\n            \\\"name\\\": \\\"dependency_input\\\",\\n            \\\"type\\\": \\\"string\\\"\\n        }\\n    ],\\n    \\\"outputPortDescriptors\\\":[\\n        {\\n            \\\"name\\\": \\\"dependency_output\\\",\\n            \\\"type\\\": \\\"string\\\"\\n        }\\n    ],\\n    \\\"containerDescriptors\\\": [\\n        {\\n            \\\"type\\\": \\\"DOCKER\\\",\\n            \\\"command\\\": \\\"\\\",\\n            \\\"properties\\\": {\\n                \\\"image\\\": \\\"tdgp/test-success\\\",\\n                \\\"mounts\\\": [\\n                    {\\n                        \\\"local\\\": \\\"$task_data_dir\\\",\\n                        \\\"container\\\": \\\"/mnt/work\\\",\\n                        \\\"read_only\\\": false\\n                    }\\n                ]\\n            }\\n        }\\n    ],\\n    \\\"description\\\": \\\"Runs a no-op task that writes successful output status.\\\",\\n    \\\"name\\\": \\\"test-success\\\",\\n    \\\"version\\\": \\\"0.0.1\\\",\\n    \\\"properties\\\": null\\n}\",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\n## Naming Beta Tasks\n\nIn some cases, a task owner may want to register a beta version of a task. The simplest way to do this is to add the suffix \"_beta\" to the task name.\n\nBeta tasks will be registered as separate tasks. For a beta version, the suffix _beta should be added to the name.  \n​\nWe'll continue using the example task called \"test-success\". \n\n**​Latest  production-ready task**\nName: test-success\nVersion: 1.0.1\n\n**\"test-success\" beta task**\nName: test-success_beta\nVersion 1.0.2 (any version is ok here since this is registered as a separate task)\n\n***note: previous version #s of \"test-success\" will still be available in the task registry unless deleted.***\n\nIf a user requests \"test-success\" without a version number, they'll get the latest production-ready task. To access the beta task, the user would have to request \"test-success_beta\".\n\nWhen the beta version is ready for production, it can be registered as name: test-success, version: 1.0.2 (or the next incremented number for the production task).\n\n## Task Docker Migration\nOnce a task is registered, it triggers a Docker migration from DockerHub to AWS ECR (Amazon EC2 Container Registry). While the task is migrating, it will not be accessible.  It will not appear in the task registry until migration is complete. \n\nWhile a task is migrating, if that task is called by name only, the previous version will be used. \n\nFor example, there is a task called test-success with version number 0.0.1 available on the platform. The owner registers a new version of test-success with version number 0.0.2, and it's still migrating. \nIn this scenario:\n\n* A \"Get Task\" request will return return version 0.0.1 while 0.0.2 is in migration.\n* If a workflow is run that includes the task \"test-success\", it will use 0.0.1 while 0.0.2 is still in migration.\n\nDocker migration typically takes 5 to 10 minutes to complete. If it takes more than 30 minutes, the migration will fail. It will continue to retry the migration for two hours. \n\nIf the task Docker fails to migrate, the owner will need to re-register it. The version number will need to be incremented. If the version number isn't changed, it will result in an error. \n\n### Migration Failure\nIf a task fails during migration, it will result in an HTTP 500 and message \"{Your task name} has failed registration.\"\n\n## Updating a Task with a \"Put\" Request\nUpdating a Task using a \"Put\" request updates the task definition, but does not trigger a Docker migration. Anything in the task definition can be updated with a \"put\" request except the name and version number. \n\n## Deleting a Task\n\n***A request to delete a task must include the task's version number!***\n\nOnly the task owner can delete a task. When a \"delete task\" request is sent, it will delete the task definition from the registry, but the Docker image will not be deleted  from the AWS ECR. \n\nOnce a task is deleted, it can be re-registered using the same version number or an incremented version number.\n                                                                                                                  \n## Previous Task Versions\nAll versions of a task remain in the task registry unless the owner deletes them. If a user wants to see or use a previous version of a task, the task version must be included in the request. If a version is not specified, the latest version of the task will be used.\n\n\n## Get a Task by Version Number\nA \"Get Task\" request can include the version number for a task. \n\nTo Get a task with version number, include name:version. The request looks like this:\n\nworkflows/v1/tasks/test-success:0.0.1\n\nIf a GET task request is made without a version number, the latest version will be returned. \n\n\n## List all Tasks\nWhen a GET request is made to list all tasks in the task registry, each version of a task will be listed separately, with the version number appended.\n\nExample:\n\n    {\n        \"tasks\": [ \"test-success:0.0.1\",\n                   \"test-success:0.0.2\"\n                 ]\n     }\n\n## Using a Versioned Task in a Workflow\nWhen using a versioned task in a workflow, you can specify the version number to use. If a version number is not specified, the latest version number will be used. \n\nTo specify the version of a task to use, append colon:version to \"taskType\". The attribute \"version\" can also be included.\n\nThese are examples of both options;\n\n- If a user wants to use a specific task version, there are two options:\n    - Append the version to 'TaskType' property, Example: `{ \"taskType\": \"test-success:0.0.1\" }`\n    - Supply version attribute on the task object, Example: `{ \"taskType\": \"test-success\", \"version\": \"0.0.1\"  }`\n\n*Note: In a workflow, \"taskType\" is equivalent to the registered name of the task. If a task was registered with the name \"test-success\", the taskType in the workflow must be \"test-success\" or \"test-success:{version number}\". *\n                                                           \n## Error Conditions\n\nOperation | Error returned\n--- | ---\nA user who is not the task owner attempts to register a new version of a task | 403 Forbidden\nA task owner re-registers a task, but doesn't increment the version number | Error: The version number must be incremented\nA task owner re-registers a task, but uses a version number lower than the last registered version | Error: The version number be incremented\nA user sends a GET request for a task while the task is migrating | 403 Forbidden' and message \"{Your task name} has been submitted for registration\nA user submits a workflow that attempts to run a task that's in migration | 403 Forbidden' and message \"{Your task name} has been submitted for registration\nAn owner attempts to delete a task while it is migrating | HTTP 403 and message \"Cannot Delete {Your task name} until Migration is Complete\" \nThe docker image fails to migrate when a task is registered | 500: \"{Your task name}has failed registration.\"","excerpt":"","slug":"how-to-version-a-task","type":"basic","title":"How to Version a Task"}

How to Version a Task


[block:callout] { "type": "warning", "title": "The new Task Versioning feature was released Thursday, January 12, 2017.", "body": "All existing tasks were given a version number of \"0.0.1\". All new tasks will require a version number.." } [/block] # Task Versioning Task versioning lets a task owner update an existing task without overwriting previous versions. Multiple versions of the same task can be registered in the GBDX task registry. Versions are identified by version number. Versioning a task is as simple as re-registering the task and incrementing the version number. When a task is registered on the GBDX platform, the task owner will give it a name and version number. To upgrade the task, the owner will re-register the task with the same name and increment the version number. Each new version must be incremented to a higher number. Task owners are discouraged from including a version number in the task name. ##How to add a version number to a task For more information about registering a task in the task registry, see [Register a task](doc:register-a-task-with-the-task-registry). The version number is a property in the task description JSON schema. When you register a task, you'll provide a name, description, and version number as part of the task definition. Registering a task means sending a "POST" request with the task definition JSON in the request body. In this example, this is a new task named "test-success". This is the first version of the task, and the version number is 0.0.1. "name": "test-success", "version": "0.0.1", To register a new version of the same task, the owner will re-register the task with the same name and an incremented version number. For example: "name": "test-success", "version": "0.0.2", or "name": "test-success", "version": "1.0.0", * Names are not case-sensitive. "Test-Success" and "test-success" will be read as the same task name. * All characters in the name must match. For example, "test-success" and "test_success" will be registered as separate tasks. These rules must be followed when adding a version number to a task: 1. The task version numbering convention adheres to [Semantic Versioning](http://semver.org/) standards which are major.minor.patch. In the example version number "0.0.1", major = 0 minor = 0 patch = 1 We encourage task owners to follow this convention: major = breaking change minor = feature change patch = bug fix 2. The version number must be enclosed in quotes, like this: "version": "0.0.2", 3. The version number can only be incremented by the owner of the task. The "owner" is the account, so any user associated with the account that owns the task can version it. If a user not associated with the account attempts to register a task with the same name, they will get a 403: Forbidden error. ## Example Task with Version Number To register a task with an incremented version number, POST the JSON request body with the task definition to https://geobigdata.io/workflows/v1/tasks This example includes the version number. [block:code] { "codes": [ { "code": "\n \"inputPortDescriptors\": [\n {\n \"required\": true,\n \"description\": \"A string input.\",\n \"name\": \"inputstring\",\n \"type\": \"string\"\n },\n {\n \"name\": \"dependency_input\",\n \"type\": \"string\"\n }\n ],\n \"outputPortDescriptors\":[\n {\n \"name\": \"dependency_output\",\n \"type\": \"string\"\n }\n ],\n \"containerDescriptors\": [\n {\n \"type\": \"DOCKER\",\n \"command\": \"\",\n \"properties\": {\n \"image\": \"tdgp/test-success\",\n \"mounts\": [\n {\n \"local\": \"$task_data_dir\",\n \"container\": \"/mnt/work\",\n \"read_only\": false\n }\n ]\n }\n }\n ],\n \"description\": \"Runs a no-op task that writes successful output status.\",\n \"name\": \"test-success\",\n \"version\": \"0.0.1\",\n \"properties\": null\n}", "language": "json" } ] } [/block] ## Naming Beta Tasks In some cases, a task owner may want to register a beta version of a task. The simplest way to do this is to add the suffix "_beta" to the task name. Beta tasks will be registered as separate tasks. For a beta version, the suffix _beta should be added to the name. ​ We'll continue using the example task called "test-success". **​Latest production-ready task** Name: test-success Version: 1.0.1 **"test-success" beta task** Name: test-success_beta Version 1.0.2 (any version is ok here since this is registered as a separate task) ***note: previous version #s of "test-success" will still be available in the task registry unless deleted.*** If a user requests "test-success" without a version number, they'll get the latest production-ready task. To access the beta task, the user would have to request "test-success_beta". When the beta version is ready for production, it can be registered as name: test-success, version: 1.0.2 (or the next incremented number for the production task). ## Task Docker Migration Once a task is registered, it triggers a Docker migration from DockerHub to AWS ECR (Amazon EC2 Container Registry). While the task is migrating, it will not be accessible. It will not appear in the task registry until migration is complete. While a task is migrating, if that task is called by name only, the previous version will be used. For example, there is a task called test-success with version number 0.0.1 available on the platform. The owner registers a new version of test-success with version number 0.0.2, and it's still migrating. In this scenario: * A "Get Task" request will return return version 0.0.1 while 0.0.2 is in migration. * If a workflow is run that includes the task "test-success", it will use 0.0.1 while 0.0.2 is still in migration. Docker migration typically takes 5 to 10 minutes to complete. If it takes more than 30 minutes, the migration will fail. It will continue to retry the migration for two hours. If the task Docker fails to migrate, the owner will need to re-register it. The version number will need to be incremented. If the version number isn't changed, it will result in an error. ### Migration Failure If a task fails during migration, it will result in an HTTP 500 and message "{Your task name} has failed registration." ## Updating a Task with a "Put" Request Updating a Task using a "Put" request updates the task definition, but does not trigger a Docker migration. Anything in the task definition can be updated with a "put" request except the name and version number. ## Deleting a Task ***A request to delete a task must include the task's version number!*** Only the task owner can delete a task. When a "delete task" request is sent, it will delete the task definition from the registry, but the Docker image will not be deleted from the AWS ECR. Once a task is deleted, it can be re-registered using the same version number or an incremented version number. ## Previous Task Versions All versions of a task remain in the task registry unless the owner deletes them. If a user wants to see or use a previous version of a task, the task version must be included in the request. If a version is not specified, the latest version of the task will be used. ## Get a Task by Version Number A "Get Task" request can include the version number for a task. To Get a task with version number, include name:version. The request looks like this: workflows/v1/tasks/test-success:0.0.1 If a GET task request is made without a version number, the latest version will be returned. ## List all Tasks When a GET request is made to list all tasks in the task registry, each version of a task will be listed separately, with the version number appended. Example: { "tasks": [ "test-success:0.0.1", "test-success:0.0.2" ] } ## Using a Versioned Task in a Workflow When using a versioned task in a workflow, you can specify the version number to use. If a version number is not specified, the latest version number will be used. To specify the version of a task to use, append colon:version to "taskType". The attribute "version" can also be included. These are examples of both options; - If a user wants to use a specific task version, there are two options: - Append the version to 'TaskType' property, Example: `{ "taskType": "test-success:0.0.1" }` - Supply version attribute on the task object, Example: `{ "taskType": "test-success", "version": "0.0.1" }` *Note: In a workflow, "taskType" is equivalent to the registered name of the task. If a task was registered with the name "test-success", the taskType in the workflow must be "test-success" or "test-success:{version number}". * ## Error Conditions Operation | Error returned --- | --- A user who is not the task owner attempts to register a new version of a task | 403 Forbidden A task owner re-registers a task, but doesn't increment the version number | Error: The version number must be incremented A task owner re-registers a task, but uses a version number lower than the last registered version | Error: The version number be incremented A user sends a GET request for a task while the task is migrating | 403 Forbidden' and message "{Your task name} has been submitted for registration A user submits a workflow that attempts to run a task that's in migration | 403 Forbidden' and message "{Your task name} has been submitted for registration An owner attempts to delete a task while it is migrating | HTTP 403 and message "Cannot Delete {Your task name} until Migration is Complete" The docker image fails to migrate when a task is registered | 500: "{Your task name}has failed registration."