Skip to content

Archiving and Extracting

Abstract

Archiving is simple to setup but lets you choose subtle options. This guide provides you with a walkthrough to understand the various operations from archiving data to extracting it.

Requirements

Make sure you successfully followed the Punchlets Getting Started guide first. This guide assumes you are familiar with punchlines and punch injection files.

Generate Logs

All the configuration files used in this guide are located under the conf/samples/archiving folder. Start by injecting logs into a Kafka topic. Simply use the provided injector file.

punchplatform-log-injector.sh -c $PUNCHPLATFORM_CONF_DIR/samples/archiving/archiving_guide_injector.hjson

This will generate 1024 records into a mytenant_archiving_guide topic. Each record looks like this:

{"ip":"128.78.0.8","raw":"20-06-2019 01:18:00 host 128.78.0.8 bob","user":"bob","timestamp":"20-06-2019 01:18:00"}
{"ip":"128.78.3.42","raw":"20-06-2019 01:19:00 host 128.78.3.42 alice","user":"alice","timestamp":"20-06-2019 01:19:00"}
{"ip":"128.78.0.30","raw":"20-06-2019 01:20:00 host 128.78.0.30 ted","user":"ted","timestamp":"20-06-2019 01:20:00"}
...

In order to produce interesting data the injector file generates 1K records, with ip addresses among 16K possible values. This will be useful to illustrate how efficiently you can locate a given ip address in a batch file.

Tip

checkout the injection file, it is self explanatory.

Archive the Logs

Now that you have your (1024) logs into Kafka, archive them into an indexed file store. Before we execute the archiving punchline, let us have a look at the punchline settings. The Kafka input simply reads our topic. The file output is in charge of archiving. It is configured with an additional bloom filtering capability. Last we catch the indexing metadata generated by the file output and we index it into elasticsearch. That last step will be used later to efficiently query our archived data.

{
  name: archiving_guide_topology
  type: punchline
  version: "6.0"
  runtime: storm
  tenant: mytenant
  channel: demo
  dag:
  [
    {
      component: input
      type: kafka_input
      fail_action: exit
      settings:
      {
        brokers: local
        topic: mytenant_archiving_guide
        start_offset_strategy: earliest
        batch_size: 8
      }
      publish:
      [
        {
          stream: logs
          fields:
          [
            raw
            ip
            user
            timestamp
          ]
        }
      ]
    }
    {
      component: output
      type: file_output
      settings:
      {
        destination: file:///tmp/archives
        pool: mytenant
        topic: demo
        folders_hierarchy_pattern: %{partition}/%{date}
        create_root: true
        compression_format: NONE
        timestamp_field: timestamp
        fields:
        [
          raw
        ]
        bloom_filter_fields:
        [
          ip
        ]
        bloom_filter_expected_insertions: 10000
        bloom_filter_false_positive_probability: 0.1
      }
      subscribe:
      [
        {
          component: input
          stream: logs
        }
      ]
      publish:
      [
        {
          stream: logs
          fields:
          [
            metadata
          ]
        }
      ]
    }
    {
      component: elastic
      type: elasticsearch_output
      settings:
      {
        cluster_name: es_search
        per_stream_settings:
        [
          {
            stream: logs
            index:
            {
              prefix: mytenant-archive-
              type: daily
            }
            document_json_field: metadata
          }
        ]
      }
      subscribe:
      [
        {
          component: output
          stream: logs
        }
      ]
    }
  ]
  settings:
  {
    topology.worker.childopts: -server -Xms256m -Xmx256m
  }
}

In particular note in there :

  • timestamp_field : We want to use the timestamp incoming field as indication For future search. You may not have such a timestamp at hand but using the punch you typically have one. It is extremely useful so as to perform search based on the real timestamp event.

  • bloom_filter_fields : Bloom filtering is activated on the field "ip"

  • bloom_filter_expected_insertions : an estimate of the number of ip addresses

  • bloom_filter_false_positive_probability : we target to accept 10 % of false positive

For the sake of simplicity, the option create_root as been activated in this punchline. This option allows root directory (here /tmp/archives/) to be automatically created. We do not recommend to activate this option for anything else than tests. Setting this option to true may create unwanted directories if destinations are mistyped. Furthermore, not creating root directory allows to check that destination cluster is reachable.

To start punchline, simply run the following command:

punchplatform-punchline.sh archiving_guide_punchline.hjson

For the sake of the example it is configured to generate batch files containing 8 records. Of course this is a very small number but it will help us illustrating the various settings without filling your laptop with huge files. Since we processed 1024 logs, we thus generated 128 batch files. Checkout the resulting archive folder. You will have the following structure:

/tmp/archives/
└── mytenant
    └── 0
        └── 2019.12.05
            ├── puncharchive-demo-ZgCV1W4BT3gZiUK0mKWY-0-1575541381475.csv
            ├── ...
            └── puncharchive-demo-ZgCV1W4BT3gZiUK0mKWY-0-1575541381602.csv

3 directories, 128 files

The "%{partition}/%{date}" settings is the one responsible for that structure. Here we have only a single Kafka partition (0).

Query the Archive

In the following we go through the essential commands provided by the storectl cli tool. It allows you to query and extract your archived data.

To start using storectl, just run the storectl command :

storectl>  _____             _   
storectl> |  _  |_ _ ___ ___| |_ 
storectl> |   __| | |   |  _|   |
storectl> |__|  |___|_|_|___|_|_|
storectl>
storectl:mytenant>

Warn

This tool may not work as expected if Metadata stored in Elasticsearch doesn't comply to the archive mapping. You can find this template in $PUNCHPLATFORM_CONF_DIR/resources/elasticsearch/templates/platform/mapping_archive.json. The prefix provided in elasticsearch_bolt needs to be of form *-archive-* to apply this template. If you want to apply a custom prefix, you need to update this template.

Topic Status

Start by querying the status of your topic. Time filter here takes data for the last 20 days. Notice the short date notation. You can provide ISO dates as well.

topic-status \
    --from-date -20d --to-date now \
    --pool mytenant --topic demo \
    --es-index-pattern mytenant-archive
demo: 
 uncompressed_size: 44.3 kB
 batch_size: 1024
 compression_factor: 1.0
 size: 44.3 kB
 latest_date: 2020-02-20T10:24:02+01:00[Europe/Paris]
 batch_count: 128
 compression_ratio: 1.0
 earliest_date: 2020-02-20T10:24:01+01:00[Europe/Paris]

List Topics

List the last 20 days topic, together with some detailed info:

list-topics \   
    --from-date -20d --to-date now \
    --pool mytenant --details \
    --es-index-pattern mytenant-archive

topic_number: 1
topics: 
  demo: 
   uncompressed_size: 44.3 kB
   batch_size: 1024
   compression_factor: 1.0
   size: 44.3 kB
   latest_date: 2020-02-20T10:24:02+01:00[Europe/Paris]
   batch_count: 128
   compression_ratio: 1.0
   earliest_date: 2020-02-20T10:24:01+01:00[Europe/Paris]
pool: mytenant

List Batch Files

List the last 20 days batch files for a given topic:

list-objects \
    --from-date -20d --to-date now \
    --pool mytenant --topic demo \
    --es-index-pattern mytenant-archive
0/2019.12.05/puncharchive-demo-fYi41W4Btd7z87AblKC6-0-1575543674519.csv
0/2019.12.05/puncharchive-demo-fYi41W4Btd7z87AblKC6-0-1575543674520.csv
0/2019.12.05/puncharchive-demo-fYi41W4Btd7z87AblKC6-0-1575543674521.csv
0/2019.12.05/puncharchive-demo-fYi41W4Btd7z87AblKC6-0-1575543674522.csv
...
0/2019.12.05/puncharchive-demo-fYi41W4Btd7z87AblKC6-0-1575543674646.csv

Using Bloom Filters

If you activated bloom filters on one or several fields, you can extract efficiently only those batches that contain a target value. In our example bloom filter was set on the ip incoming tuple field. Here is how you can request for batches that contain a given ip:

list-objects  \
    --from-date -30d --to-date now \
    --pool mytenant --topic demo \
    --match 128.78.18.47 \
    --es-index-pattern mytenant-archive
0/2019.12.05/puncharchive-demo-fYi41W4Btd7z87AblKC6-0-1575543674595.csv

In this case only a single batch contain the target ip. This in turn allows you to perform efficient data extractions.

Tip

If the result is empty, it's because IPs are randomly generated for this example. You can get these IP by using the command below and then copy one of them in --match args

grep -r 128.78.18 /tmp/archives/

Extracting using punchline

See Archive Reader Node