HOWTO close indices or create aliases with elastic curator
Why do that¶
The PunchPlatform usually writes a large amount of logs in an elasticsearch cluster. The administrator hide some logs to the external customer (for business purpose or security reasons) or change the resilience configuration of data by using elasticsearch alias mechanism.
Curator 4.3 is a tool provided by PunchPlatform to change these settings (and developed by elastic).
Elasticsearch Background Information¶
Aliases¶
Aliases provide a link between indices where are stored logs and the data that kibana can requested.
You can check all links :
curl <elasticsearch_url>:9200/_aliases?v
Replica¶
Replica is a copy of an Index to an other node. It provides resiliency.
- Replica : 0 = > There is no replication
- Replica : 1 = > Replication factor 1 is enabled
To check the replica factor on indices :
curl <elasticsearch_url>:9200/_cat/indices?v
Opening and Closing Indices¶
In elasticsearch, Index are often opened, i.e. in order to be requested, or closed, i.e. in order to reduce memory or CPU consumption.
To check the closed and opened indices :
curl <elasticsearch_url>:9200/_cat/indices?v
Warning
Do not open too many indices in one time. It generates huge IO on systems and a lot of tasks for the master node. A best practice is to run curator 30 days per 30 days
Prerequisite¶
You need to have access to the operator environment.
What to do¶
Run curator for testing¶
The curator script needs a configuration file and execution file.
First we need to test:
curator --config <configuration_file>.yml <execution_file>.yml --dry-run
Second, check the result:
- the list of changed indices
- the output of the command
Third, run curator for real:
curator --config <configuration_file>.yml <execution_file>.yml
Execution file sample¶
Closing an Index¶
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
#
actions:
1:
action: close
description: >-
Close indices between last 30 days and 60 days (based on index name), for events-
prefixed indices.
options:
delete_aliases: False
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: events-
exclude:
- filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 30
exclude:
- filtertype: age
source: name
direction: younger
timestring: '%Y.%m.%d'
unit: days
unit_count: 60
exclude:
Create an Alias¶
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python "NoneType"
#
actions:
1:
action: alias
description: >-
Alias indices older than 0 days, with a prefix of
events-mytenant
options:
name: events-mytenant-kibana-2016.08.01
extra_settings:
timeout_override:
continue_if_exception: True
disable_action: False
ignore_empty_list: True
add:
filters:
- filtertype: pattern
kind: prefix
value: events-mytenant-apache-2016.08.01
exclude: