ElasticSearch is a great technology for a wide variety of use cases from autocompletion to log management, and is likely to be part of your stack for many complex projects.
We have found that
Index management can be hard to manage over time:
- ElasticSearch is not truly schemaless but it is smart enough to figure out the schema (aka
datatype) most of the time. Thanks to that, you can play with ElasticSearch without configuring anything at first.
- In a typical project, you will start with small documents, then add, delete and alter field content. You’ll also need to add some
mappingsto have control over the
datatypeof the fields. Then, you’ll probably need to update some field mapping at some point, only to discover you can’t, without re-indexing all of the index content. You’ll encounter the same issue, should you need to change your index static settings. Unfortunately if you did not prepare for it, this means production downtime.
- ElasticSearch will figure out the datatype of a given field using the dynamic mapping rules it has. The first document you index containing a given field is its only context to figure out the right
During the life of a project, without being managed, an index will surely break.
A Workflow to Manage These Problems
In order to have the flexibility to change your index
mappings or static
settings whenever you want, you need to be able to create a new index with the same content and the new
settings. Once reindexed, you need to use this new index instead of the original one. Ideally, this should happen with no downtime.
Keeping track of changing indices names at a given moment in your project can be hard to manage. ElasticSearch provides a solution to this with
aliases: you can search and write to aliases like you do with indices. Aliases are just redirects to one or more indices, possibly with some filtering. They also allow you to redirect all of the queries from an index to a new one, with no downtime.
Here are simplified, nominal examples of what you should do to manipulate index-like alias
- create a new index structure:
- update an existing index structure:
- delete an existing index structure:
One of the best ways to allow both machines and humans (for testing) to create indices that work in the long run, is to deploy
index templates prior to any creations or updates with this workflow.
Introducing Leastic for ElasticSearch
Leaistic provides high-level primitives for creating, updating, and deleting indices, index templates, and aliases working together to help with the problems listed above. It does not cover all ElasticSearch possibilities like
rollover index, multi-indices strategies, etc. If you start a new project with ElasticSearch, it will help you get your indices right.
Leaistic tries to be resilient against external changes in parallel with its operations, and has a per-alias locking mechanism to compensate for ElasticSearch’s lack of transactions for these operations. It is committed to providing helpful errors, and to rollback, when necessary, to the best possible state.
Read more about how we use ElasticSearch in our projects:
- How to build a real-time Node.js Performance Monitoring System
- Sematext Guest Post: PINO – The Fastest Node.js Logger for Production
Image: Gabriel Sollmann