{"_id":"56e1cba3cd6a8d0e00d1216c","category":{"_id":"56e1ccdfe63f910e00e59870","version":"55faeacad0e22017005b8268","project":"55faeacad0e22017005b8265","__v":5,"pages":["56e311b0cb6ef20e0084f24b","56e312096e602e0e00700b2d","56e312bcd46bc30e007bb99a","56e31309cb6ef20e0084f250","56e31357cb6ef20e0084f254"],"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-03-10T19:37:03.043Z","from_sync":false,"order":19,"slug":"write-ingest-vector-services-guide","title":"Write & Ingest Vector Services Guide"},"project":"55faeacad0e22017005b8265","user":"56267741db1eda0d001c3dbb","version":{"_id":"55faeacad0e22017005b8268","project":"55faeacad0e22017005b8265","__v":36,"createdAt":"2015-09-17T16:31:06.800Z","releaseDate":"2015-09-17T16:31:06.800Z","categories":["55faeacbd0e22017005b8269","55faf550764f50210095078e","55faf5b5626c341700fd9e96","55faf8a7825d5f19001fa386","560052f91503430d007cc88f","560054f73aa0520d00da0b1a","56005aaf6932a00d00ba7c62","56005c273aa0520d00da0b3f","5601ae7681a9670d006d164d","5601ae926811d00d00ceb487","5601aeb064866b1900f4768d","5601aee850ee460d0002224c","5601afa02499c119000faf19","5601afd381a9670d006d1652","561d4c78281aec0d00eb27b6","561d588d8ca8b90d00210219","563a5f934cc3621900ac278c","5665c5763889610d0008a29e","566710a36819320d000c2e93","56ddf6df8a5ae10e008e3926","56e1c96b2506700e00de6e83","56e1ccc4e416450e00b9e48c","56e1ccdfe63f910e00e59870","56e1cd10bc46be0e002af26a","56e1cd21e416450e00b9e48e","56e3139a51857d0e008e77be","573b4f62ef164e2900a2b881","57c9d1335fd8ca0e006308ed","57e2bd9d1e7b7220000d7fa5","57f2b992ac30911900c7c2b6","58adb5c275df0f1b001ed59b","58c81b5c6dc7140f003c3c46","595412446ed4d9001b3e7b37","59e76ce41938310028037295","5a009de510890d001c2aabfe","5a96f89c89442e002041144b"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"v1","version_clean":"1.0.0","version":"1"},"parentDoc":null,"__v":3,"updates":["5ad7a7f20cbbc80003232349"],"next":{"pages":[],"description":""},"createdAt":"2016-03-10T19:31:47.737Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":2,"body":"#Index Name Requirements#\n\n Elasticsearch allows us to set a template for defining field mappings for vector items when it automatically creates an index. We do that by matching the index name to be created against an expression in the index template.\n\n In our case, the template looks for index names that start with \"vector-\", so in order for any automatically created index to map to our vector index structure, it must begin with \"vector-\". For example, \"vector-osm\" or \"vector-twitter-20150929\". *Note: For user generated indices using the [Write API](doc:vector-services-write-vectors-reference-overview), we will automatically append \"vector-\" to the index name if it is not already there.*\n\n#Index Name Restrictions#\n\n An Elasticsearch index must not contain any upper-case letters. Elasticsearch will return an error if it does.\n\n#Index Name Templates#\n\n As well, with our custom Java-based Elasticsearch writer, we can use placeholders in the index name and have those placeholders substituted with data from the item being ingested. The ES writer supports the following placeholders:\n\n  - {geohash}: replaced with the single-character geohash which covers the vector item's geometry. If no single character will cover it, the placeholder will be replaced with `nohash`. For example, for the index template `vector-{geohash}-footprint`, if an item is covered by the geohash `e`, the index name the item would be written to would be `vector-e-footprint`.\n\n  - {date}: replaced with the current system date in the format `yyyyMMdd`. For example, if the current system date is September 29, 2015, the index name template `vector-test-{date}` will become `vector-test-20150929`.\n\n  - {item_date}: replaced with the item's date in the format `yyyyMMdd`. For example, if the item to be indexed has an 'item_date' field with the value of September 29, 2015, the index name template 'vector-test-{item_date}' will become `vector-test-20150929`.\n\n  - {ingest_date}: replaced with the item's ingest date in the format `yyyyMMdd`. For example, if the item to be indexed is ingested on September 29, 2015, the index name template `vector-test-{ingest_date}` will become `vector-test-20150929`.\n\nAn index name template can include multiple placeholders. For example, if an item is covered by a single-character geohash of 'w' and is ingested on September 29, 2015, then the index name template `vector-test-{geohash}-{ingest_date}` would become `vector-test-w-20150929`.\n\n#Index Name Template Syntax#\n\nFor any of the *date* type templates, it is possible to choose how specific the date in the index is. For example, if the item to be indexed has an 'item_date' field with the value of September 29, 2015, and the index should specify to the month, the index name template `vector-test-{item_date:yyyyMM}` will become 'vector-test-201509. If the index should specify to the year, the index name template `vector-test-{item_date:yyyy}` will become `vector-test-2015`.\n\nTo add any templates into the index name, a method of \"double\" escapes is necessary. Not following this convention will cause the API to return errors. The \"double\" escapes are simply taking the usual escapes:\n - { escapes to %7B\n - } escapes to %7D\n - : escapes to %3A\nand escaping for the % at the front. So instead use:\n - { escapes to %257B\n - } escapes to %257D\n - : escapes to %253A\n \nThe end result is an index name template of, for example, `vector-test-%257Bitem_date%253AyyyyMM%257D`.","excerpt":"","slug":"vs-elasticsearch-index-name-templates","type":"basic","title":"Vector Services Elasticsearch Index Name Templates"}

Getting Started

GBDX Operations

GBDX Stories

Data Sets on GBDX

Raster Data Access (RDA) Guide

GBDX Notebooks

Authentication Guide

Ordering Guide

Catalog V2 API Guide

Algorithms

Tasks and Workflows Guide

S3 Storage Service Guide

Thumbnail Service Guide

Web Application

AnswerFactory Application

Read & Query Vector Services Guide

Aggregations & Facets Vector Services Guide

Write & Ingest Vector Services Guide

IDAHO Guide

Helpful Links

Vector Services Elasticsearch Index Name Templates

#Index Name Requirements# Elasticsearch allows us to set a template for defining field mappings for vector items when it automatically creates an index. We do that by matching the index name to be created against an expression in the index template. In our case, the template looks for index names that start with "vector-", so in order for any automatically created index to map to our vector index structure, it must begin with "vector-". For example, "vector-osm" or "vector-twitter-20150929". *Note: For user generated indices using the [Write API](doc:vector-services-write-vectors-reference-overview), we will automatically append "vector-" to the index name if it is not already there.* #Index Name Restrictions# An Elasticsearch index must not contain any upper-case letters. Elasticsearch will return an error if it does. #Index Name Templates# As well, with our custom Java-based Elasticsearch writer, we can use placeholders in the index name and have those placeholders substituted with data from the item being ingested. The ES writer supports the following placeholders: - {geohash}: replaced with the single-character geohash which covers the vector item's geometry. If no single character will cover it, the placeholder will be replaced with `nohash`. For example, for the index template `vector-{geohash}-footprint`, if an item is covered by the geohash `e`, the index name the item would be written to would be `vector-e-footprint`. - {date}: replaced with the current system date in the format `yyyyMMdd`. For example, if the current system date is September 29, 2015, the index name template `vector-test-{date}` will become `vector-test-20150929`. - {item_date}: replaced with the item's date in the format `yyyyMMdd`. For example, if the item to be indexed has an 'item_date' field with the value of September 29, 2015, the index name template 'vector-test-{item_date}' will become `vector-test-20150929`. - {ingest_date}: replaced with the item's ingest date in the format `yyyyMMdd`. For example, if the item to be indexed is ingested on September 29, 2015, the index name template `vector-test-{ingest_date}` will become `vector-test-20150929`. An index name template can include multiple placeholders. For example, if an item is covered by a single-character geohash of 'w' and is ingested on September 29, 2015, then the index name template `vector-test-{geohash}-{ingest_date}` would become `vector-test-w-20150929`. #Index Name Template Syntax# For any of the *date* type templates, it is possible to choose how specific the date in the index is. For example, if the item to be indexed has an 'item_date' field with the value of September 29, 2015, and the index should specify to the month, the index name template `vector-test-{item_date:yyyyMM}` will become 'vector-test-201509. If the index should specify to the year, the index name template `vector-test-{item_date:yyyy}` will become `vector-test-2015`. To add any templates into the index name, a method of "double" escapes is necessary. Not following this convention will cause the API to return errors. The "double" escapes are simply taking the usual escapes: - { escapes to %7B - } escapes to %7D - : escapes to %3A and escaping for the % at the front. So instead use: - { escapes to %257B - } escapes to %257D - : escapes to %253A The end result is an index name template of, for example, `vector-test-%257Bitem_date%253AyyyyMM%257D`.