HOWTO close indices or create aliases with elastic curator
Why do that¶
The PunchPlatform usually writes a large amount of logs in an elasticsearch cluster. The administrator hide some logs to the external customer (for business purpose or security reasons) or change the resilience configuration of data by using elasticsearch alias mecanism.
Curator 4.3 is a tool provided by PunchPlatform to change these settings (and developped by elastic).
Elasticsearch Background Information¶
Aliases¶
Aliases provide a link between indices where are stored logs and the data that kibana can requested.
You can check all links :
1 | $ curl <elasticsearch_url>:9200/_aliases?v |
Replica¶
Replica is a copy of an indice to an other node. It provides resiliency.
- Replica : 0 = > There is no replication
- Replica : 1 = > Replication factor 1 is enabled
To check the replica factor on indices :
1 | $ curl <elasticsearch_url>:9200/_cat/indices?v |
Opening and Closing Indices¶
In elasticsearch, indice are often opened, i.e. in order to be requested, or closed, i.e. in order to reduce memory or CPU consumption.
To check the closed and opened indices :
1 | $ curl <elasticsearch_url>:9200/_cat/indices?v |
Warning
Do not open too many indices in one time. It generates huge IO on systems and a lot of tasks for the master node. A best practice is to run curator 30 days per 30 days
Prerequisite¶
You need to have access to the operator environment.
What to do¶
Run curator for testing¶
The curator script needs a configuration file and execution file.
First we need to test:
1 | $ curator --config <configuration_file>.yml <execution_file>.yml --dry-run |
Second, check the result:
- the list of changed indices
- the output of the command
Third, run curator for real:
1 | $ curator --config <configuration_file>.yml <execution_file>.yml |
Execution file sample¶
Closing an Indice¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # actions: 1: action: close description: >- Close indices between last 30 days and 60 days (based on index name), for events- prefixed indices. options: delete_aliases: False timeout_override: continue_if_exception: False disable_action: False filters: - filtertype: pattern kind: prefix value: events- exclude: - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 30 exclude: - filtertype: age source: name direction: younger timestring: '%Y.%m.%d' unit: days unit_count: 60 exclude: |
Create an Alias¶
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # actions: 1: action: alias description: >- Alias indices older than 0 days, with a prefix of events-mytenant options: name: events-mytenant-kibana-2016.08.01 extra_settings: timeout_override: continue_if_exception: True disable_action: False ignore_empty_list: True add: filters: - filtertype: pattern kind: prefix value: events-mytenant-apache-2016.08.01 exclude: |