Skip to content

HOWTO close indices or create aliases with elastic curator

Why do that

The PunchPlatform usually writes a large amount of logs in an elasticsearch cluster. The administrator hide some logs to the external customer (for business purpose or security reasons) or change the resilience configuration of data by using elasticsearch alias mecanism.

Curator 4.3 is a tool provided by PunchPlatform to change these settings (and developped by elastic).

Elasticsearch Background Information

Aliases

Aliases provide a link between indices where are stored logs and the data that kibana can requested.

You can check all links :

1
$ curl <elasticsearch_url>:9200/_aliases?v

Replica

Replica is a copy of an indice to an other node. It provides resiliency.

  • Replica : 0 = > There is no replication
  • Replica : 1 = > Replication factor 1 is enabled

To check the replica factor on indices :

1
$ curl <elasticsearch_url>:9200/_cat/indices?v

Opening and Closing Indices

In elasticsearch, indice are often opened, i.e. in order to be requested, or closed, i.e. in order to reduce memory or CPU consumption.

To check the closed and opened indices :

1
$ curl <elasticsearch_url>:9200/_cat/indices?v

Warning

Do not open too many indices in one time. It generates huge IO on systems and a lot of tasks for the master node. A best practice is to run curator 30 days per 30 days

Prerequisite

You need to have access to the operator environment.

What to do

Run curator for testing

The curator script needs a configuration file and execution file.

First we need to test:

1
$ curator --config <configuration_file>.yml <execution_file>.yml --dry-run

Second, check the result:

  • the list of changed indices
  • the output of the command

Third, run curator for real:

1
$ curator --config <configuration_file>.yml <execution_file>.yml

Execution file sample

Closing an Indice

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
actions:
  1:
    action: close
    description: >-
      Close indices between last 30 days and 60 days (based on index name), for events-
      prefixed indices.
    options:
      delete_aliases: False
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: events-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 30
      exclude:
    - filtertype: age
      source: name
      direction: younger
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 60
      exclude:

Create an Alias

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
actions:
  1:
    action: alias
    description: >-
      Alias indices older than 0 days, with a prefix of
      events-mytenant
    options:
      name: events-mytenant-kibana-2016.08.01
      extra_settings:
      timeout_override:
      continue_if_exception: True
      disable_action: False
      ignore_empty_list: True
    add:
      filters:
      - filtertype: pattern
        kind: prefix
        value: events-mytenant-apache-2016.08.01
        exclude: