Index Name Requirements

Elasticsearch allows us to set a template for defining field mappings for vector items when it automatically creates an index. We do that by matching the index name to be created against an expression in the index template.

In our case, the template looks for index names that start with "vector-", so in order for any automatically created index to map to our vector index structure, it must begin with "vector-". For example, "vector-osm" or "vector-twitter-20150929".

Note: For user generated indices using the Write API, we will automatically prepend "vector-" to the index name if it is not already there.

Note: For user generated indices using the Write API, all indices begin with the format vector-user-provided-, and the user must provide the user-provided- portion themselves. Contact [email protected] to request a different start to the index name.

Index Name Restrictions

An Elasticsearch index must not contain any upper-case letters. Elasticsearch will return an error if it does.

Index Name Templates

As well, with our custom Java-based Elasticsearch writer, we can use placeholders in the index name and have those placeholders substituted with data from the item being ingested. The ES writer supports the following placeholders:

• {geohash}: replaced with the single-character geohash which covers the vector item's geometry. If no single character will cover it, the placeholder will be replaced with nohash. For example, for the index template vector-user-provided-{geohash}-footprint, if an item is covered by the geohash e, the index name the item would be written to would be vector-user-provided-e-footprint.

• {date}: replaced with the current system date in the format yyyyMMdd. For example, if the current system date is September 29, 2015, the index name template vector-user-provided-{date} will become vector-user-provided-20150929.

• {item_date}: replaced with the item's date in the format yyyyMMdd. For example, if the item to be indexed has an 'item_date' field with the value of September 29, 2015, the index name template 'vector-user-provided-{item_date}' will become vector-user-provided-20150929.

• {ingest_date}: replaced with the item's ingest date in the format yyyyMMdd. For example, if the item to be indexed is ingested on September 29, 2015, the index name template vector-user-provided-{ingest_date} will become vector-user-provided-20150929.

An index name template can include multiple placeholders. For example, if an item is covered by a single-character geohash of 'w' and is ingested on September 29, 2015, then the index name template vector-user-provided-{geohash}-{ingest_date} would become vector-user-provided-w-20150929.

Index Name Template Syntax

For any of the date type templates, it is possible to choose how specific the date in the index is. For example, if the item to be indexed has an 'item_date' field with the value of September 29, 2015, and the index should specify to the month, the index name template vector-user-provided-{item_date:yyyyMM} will become 'vector-user-provided-201509. If the index should specify to the year, the index name template vector-user-provided-{item_date:yyyy} will become vector-user-provided-2015.

To add any templates into the index name, a method of "double" escapes is necessary. Not following this convention will cause the API to return errors. The "double" escapes are simply taking the usual escapes:

• { escapes to %7B
• } escapes to %7D
• : escapes to %3A
  and escaping for the % at the front. So instead use:
• { escapes to %257B
• } escapes to %257D
• : escapes to %253A

The end result is an index name template of, for example, vector-user-provided-%257Bitem_date%253AyyyyMM%257D.