Skip to content

Archives Housekeeping

Abstract

This chapter explains how to setup your archive(s) data lifecycle.

Overview

If you use the punch archives to store data on long term object storage, you will also need to define your strategy to clean too old data. Just like for the elasticsearch data housekeeping, the punch provides you with a ready to use archive housekeeper job, that you can include in one of your channel.

Configuration

First add the following job to the channel channel_structure.json descriptor file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
{
    "version" : "5.0",
    "jobs" : [
        {
            "type" : "shiva",
            "name" : "archives-housekeeping",
            "command" : "archives-housekeeping",
            "args": [
                "archives-housekeeping.json"
            ],
            "resources": [
                "archives-housekeeping.json"
            ],
            "cluster" : "common",
            "shiva_runner_tags" : ["standalone"],
            "quartzcron_schedule" : "0 0 * ? * * *"
        }
    ]
}

Along with an archives-housekeeping.json file to define your settings. Here is an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
   "archiving_pools": [
    {
      "destinations": ["file:///tmp/storage"],
      "pool": "mytenant-data",
      "retention": "3m",
      "max_deletion_percentage": 100,
      "es_cluster_id": "es_search",
      "es_index": "mytenant-archive",
      "user": "admin"
    }
  ]
}

Watch out! This service works with the Elasticsearch mapping template mapping_archive.json and especially the topic_name field. If you need to load this mapping, use this command:

1
2
3
$ curl -X POST localhost:9200/_template/mapping_archive \
       -H "Content-Type: application/json" \
       -d @resources/elasticsearch/mapping_archive.json

Parameters

Mandatory Parameters

  • destinations (string)

    Array of destinations. In case of a ceph cluster, this name is the absolute path to the ceph cluster configuration file with format ceph_configuration://.
    In case of a file-system storage, this name is the absolute path of the archive root directory with format file:///.

  • pool (string)

    Archiving pool name. Usually is the tenant name.

  • retention (string)

    Retention time. All data older than specified retention will be deleted.
    format: 30s, 1m, 2minutes, 1h, 1d...

  • max_deletion_percentage (decimal)

    Maximum deletion percentage compared with the whole cluster data. This feature exists to prevent accidental deletion: operation will fail if we ask to delete more that x% of data.
    default: 1.0

  • es_cluster_id (string)

    Elasticsearch cluster used to store objects meta-data.

  • es_index (string)

    Elasticsearch index containing objects meta-data (will be appended -*)

Optional Parameters

  • user (string)

    Only in case of a ceph cluster, name used to access to Ceph cluster.

Archive permission

In case of filesystem, do not forget to give read+write permissions to the archive folder. The shiva process owner must have the read write delete permissions on the archive data.