Index Name Requirements#
Elasticsearch allows us to set a template for defining field mappings for vector items when it automatically creates an index. We do that by matching the index name to be created against an expression in the index template.
In our case, the template looks for index names that start with "vector-", so in order for any automatically created index to map to our vector index structure, it must begin with "vector-". For example, "vector-osm" or "vector-twitter-20150929".
Note: For user generated indices using the Write API, we will automatically prepend "vector-" to the index name if it is not already there.
Note: For user generated indices using the Write API, all indices begin with the format vector-user-provided-
, and the user must provide the user-provided-
portion themselves. Contact [email protected] to request a different start to the index name.
Index Name Restrictions#
An Elasticsearch index must not contain any upper-case letters. Elasticsearch will return an error if it does.
Index Name Templates#
As well, with our custom Java-based Elasticsearch writer, we can use placeholders in the index name and have those placeholders substituted with data from the item being ingested. The ES writer supports the following placeholders:
-
{geohash}: replaced with the single-character geohash which covers the vector item's geometry. If no single character will cover it, the placeholder will be replaced with
nohash
. For example, for the index templatevector-user-provided-{geohash}-footprint
, if an item is covered by the geohashe
, the index name the item would be written to would bevector-user-provided-e-footprint
. -
{date}: replaced with the current system date in the format
yyyyMMdd
. For example, if the current system date is September 29, 2015, the index name templatevector-user-provided-{date}
will becomevector-user-provided-20150929
. -
{item_date}: replaced with the item's date in the format
yyyyMMdd
. For example, if the item to be indexed has an 'item_date' field with the value of September 29, 2015, the index name template 'vector-user-provided-{item_date}' will becomevector-user-provided-20150929
. -
{ingest_date}: replaced with the item's ingest date in the format
yyyyMMdd
. For example, if the item to be indexed is ingested on September 29, 2015, the index name templatevector-user-provided-{ingest_date}
will becomevector-user-provided-20150929
.
An index name template can include multiple placeholders. For example, if an item is covered by a single-character geohash of 'w' and is ingested on September 29, 2015, then the index name template vector-user-provided-{geohash}-{ingest_date}
would become vector-user-provided-w-20150929
.
Index Name Template Syntax#
For any of the date type templates, it is possible to choose how specific the date in the index is. For example, if the item to be indexed has an 'item_date' field with the value of September 29, 2015, and the index should specify to the month, the index name template vector-user-provided-{item_date:yyyyMM}
will become 'vector-user-provided-201509. If the index should specify to the year, the index name template vector-user-provided-{item_date:yyyy}
will become vector-user-provided-2015
.
To add any templates into the index name, a method of "double" escapes is necessary. Not following this convention will cause the API to return errors. The "double" escapes are simply taking the usual escapes:
- { escapes to %7B
- } escapes to %7D
- : escapes to %3A
and escaping for the % at the front. So instead use: - { escapes to %257B
- } escapes to %257D
- : escapes to %253A
The end result is an index name template of, for example, vector-user-provided-%257Bitem_date%253AyyyyMM%257D
.
Updated 2 years ago