GBDX

How to Version a Task

Task Versioning

Task versioning lets a task owner update an existing task without overwriting previous versions. Multiple versions of the same task can be registered in the GBDX task registry. Versions are identified by version number. Versioning a task is as simple as re-registering the task and incrementing the version number.

When a task is registered on the GBDX platform, the task owner will give it a name and version number. To upgrade the task, the owner will re-register the task with the same name and increment the version number. Each new version must be incremented to a higher number. Task owners are discouraged from including a version number in the task name.

taskOwnerEmail is now a required field in the task definition

A value for this field is required when a new task is registered, and when a new version of a task is registered.

How to add a version number to a task

For more information about registering a task in the task registry, see Register a task.

The version number is a property in the task description JSON schema. When you register a task, you'll provide a name, description, and version number as part of the task definition. Registering a task means sending a "POST" request with the task definition JSON in the request body.

In this example, this is a new task named "test-success". This is the first version of the task, and the version number is 0.0.1.

    "name": "test-success",
    "version": "0.0.1",

To register a new version of the same task, the owner will re-register the task with the same name and an incremented version number.

For example:

 "name": "test-success",
 "version": "0.0.2",

or

 "name": "test-success",
 "version": "1.0.0",
  • Names are not case-sensitive. "Test-Success" and "test-success" will be read as the same task name.

  • All characters in the name must match. For example, "test-success" and "test_success" will be registered as separate tasks.

    These rules must be followed when adding a version number to a task:

  1. The task version numbering convention adheres to Semantic Versioning standards which are

    major.minor.patch.

In the example version number "0.0.1",

major = 0
minor = 0
patch = 1

We encourage task owners to follow this convention:

major = breaking change
minor = feature change
patch = bug fix

  1. The version number must be enclosed in quotes, like this:

    "version": "0.0.2",

  2. The version number can only be incremented by the owner of the task. The "owner" is the account, so any user associated with the account that owns the task can version it.

If a user not associated with the account attempts to register a task with the same name, they will get a 403: Forbidden error.

Example Task with Version Number

To register a task with an incremented version number, POST the JSON request body with the task definition to https://geobigdata.io/workflows/v1/tasks

This example includes the version number.


     "inputPortDescriptors": [
        {
            "required": true,
            "description": "A string input.",
            "name": "inputstring",
            "type": "string"
        },
        {
            "name": "dependency_input",
            "type": "string"
        }
    ],
    "outputPortDescriptors":[
        {
            "name": "dependency_output",
            "type": "string"
        }
    ],
    "containerDescriptors": [
        {
            "type": "DOCKER",
            "command": "",
            "properties": {
                "image": "tdgp/test-success",
                "mounts": [
                    {
                        "local": "$task_data_dir",
                        "container": "/mnt/work",
                        "read_only": false
                    }
                ]
            }
        }
    ],
    "description": "Runs a no-op task that writes successful output status.",
    "name": "test-success",
		"taskOwnerEmail": "TEST@TEST.com",
    "version": "0.0.1",
    "properties": null
}

Naming Beta Tasks

In some cases, a task owner may want to register a beta version of a task. The simplest way to do this is to add the suffix "_beta" to the task name.

Beta tasks will be registered as separate tasks. For a beta version, the suffix _beta should be added to the name.

We'll continue using the example task called "test-success".

​Latest production-ready task
Name: test-success
Version: 1.0.1

"test-success" beta task
Name: test-success_beta
Version 1.0.2 (any version is ok here since this is registered as a separate task)

note: previous version #s of "test-success" will still be available in the task registry unless deleted.

If a user requests "test-success" without a version number, they'll get the latest production-ready task. To access the beta task, the user would have to request "test-success_beta".

When the beta version is ready for production, it can be registered as name: test-success, version: 1.0.2 (or the next incremented number for the production task).

Task Docker Migration

Once a task is registered, it triggers a Docker migration from DockerHub to AWS ECR (Amazon EC2 Container Registry). While the task is migrating, it will not be accessible. It will not appear in the task registry until migration is complete.

While a task is migrating, if that task is called by name only, the previous version will be used.

For example, there is a task called test-success with version number 0.0.1 available on the platform. The owner registers a new version of test-success with version number 0.0.2, and it's still migrating.
In this scenario:

  • A "Get Task" request will return return version 0.0.1 while 0.0.2 is in migration.
  • If a workflow is run that includes the task "test-success", it will use 0.0.1 while 0.0.2 is still in migration.

Docker migration typically takes 5 to 10 minutes to complete. If it takes more than 30 minutes, the migration will fail. It will continue to retry the migration for two hours.

If the task Docker fails to migrate, the owner will need to re-register it. The version number will need to be incremented. If the version number isn't changed, it will result in an error.

Migration Failure

If a task fails during migration, it will result in an HTTP 500 and message "{Your task name} has failed registration."

Updating a Task with a "Put" Request

Updating a Task using a "Put" request updates the task definition, but does not trigger a Docker migration. Anything in the task definition can be updated with a "put" request except the name and version number.

Deleting a Task

A request to delete a task must include the task's version number!

Only the task owner can delete a task. When a "delete task" request is sent, it will delete the task definition from the registry, but the Docker image will not be deleted from the AWS ECR.

Once a task is deleted, it can be re-registered using the same version number or an incremented version number.

Previous Task Versions

All versions of a task remain in the task registry unless the owner deletes them. If a user wants to see or use a previous version of a task, the task version must be included in the request. If a version is not specified, the latest version of the task will be used.

Get a Task by Version Number

A "Get Task" request can include the version number for a task.

To Get a task with version number, include name:version. The request looks like this:

workflows/v1/tasks/test-success:0.0.1

If a GET task request is made without a version number, the latest version will be returned.

List all Tasks

When a GET request is made to list all tasks in the task registry, each version of a task will be listed separately, with the version number appended.

Example:

{
    "tasks": [ "test-success:0.0.1",
               "test-success:0.0.2"
             ]
 }

Using a Versioned Task in a Workflow

When using a versioned task in a workflow, you can specify the version number to use. If a version number is not specified, the latest version number will be used.

To specify the version of a task to use, append colon:version to "taskType". The attribute "version" can also be included.

These are examples of both options;

  • If a user wants to use a specific task version, there are two options:
    • Append the version to 'TaskType' property, Example: { "taskType": "test-success:0.0.1" }
    • Supply version attribute on the task object, Example: { "taskType": "test-success", "version": "0.0.1" }

Note: In a workflow, "taskType" is equivalent to the registered name of the task. If a task was registered with the name "test-success", the taskType in the workflow must be "test-success" or "test-success:{version number}".

Error Conditions

Operation Error returned
A user who is not the task owner attempts to register a new version of a task 403 Forbidden
A task owner re-registers a task, but doesn't increment the version number Error: The version number must be incremented
A task owner re-registers a task, but uses a version number lower than the last registered version Error: The version number be incremented
A user sends a GET request for a task while the task is migrating 403 Forbidden' and message "{Your task name} has been submitted for registration
A user submits a workflow that attempts to run a task that's in migration 403 Forbidden' and message "{Your task name} has been submitted for registration
An owner attempts to delete a task while it is migrating HTTP 403 and message "Cannot Delete {Your task name} until Migration is Complete"
The docker image fails to migrate when a task is registered 500: "{Your task name}has failed registration."