Skip to content

5.6.0 to 5.7.0

This document explains what configuration changes MUST be performed during a PunchPlatform update from version 5.6.0 to 5.7.0.

Python 2.7 and Python 3

The Punchplatform is fully compatible with Python +2.7 and Python +3.6.
In the future major version (Dave 6.0.0). The Punchplatform will only work on Python +3.6.

Deployment

The deployer is now using ansible 2.9.0.
You must use ansible 2.9.0 either using pip install ansible == 2.9.0 or using the zip ansible-2.9.0-pippackage.zip.

Ansible 2.9.0 works with Python +2.7 and Python +3.5. However, Python 2.7 is deprecated. You can use the Python version of your choice.
The Punchplatform components are Python +3.6 compliant. It is strongly recommended to use Python +3.6.

Punch Plugin for kibana

Useless configurations have been removed :

  • kibana.plugins.punchplatform.ioc_enabled
  • kibana.plugins.punchplatform.ioc_index
  • kibana.plugins.punchplatform.ioc_max_size_mb

Archiving

punchplatform-objects-storage.sh

Option pool-status can only be applied to a ceph cluster. No more Filesystem pool status. No more option dump-objects and extract-scope.

Ceph archiving

File bolt settings for Ceph archiving have been simplified :

In past versions, two different configurations could be provided :

"destination": "ceph_configuration:///etc/ceph/client-main.conf"

or

"destination": "ceph:main"

These configurations worked the same way, it was just two different ways of writing the same thing

In 5.7.0 only ceph_configuration is preserved with a absolute path to ceph configuration. Check FileBolt documentation for more details

Analytics (PML)

Punchplatform Analytics has been refactored to PunchLine. The same goes to pml extension as punchline. This refactoring is still in progress and the keyword PML or analytics may still wander in our documentation...

With the addition of new features, some breaking changes were unfortunately unavoidable, as stated below.

Shells

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# punchplatform-analytics.sh has been refactored to punchlinectl
PATH_JOB_FILE=/full/path/to/your/job_file.punchline
punchlinectl --punchline $PATH_JOB_FILE

# punchlinectl now needs a working directory before executing any spark job
# the same goes for Plan
# By default
PUNCH_WORK_DIR=/tmp/spark

# this can be change by adding an argument to punchlinectl
WORK_DIR=/home/user/workdir
punchlinectl --punchline $PATH_JOB_FILE --punch-work-dir $WORK_DIR

Plan

Punchplatform Analytics (now Punchlinectl) plan feature has been improved. Only the model configuration file has been changed. Note that we tried our best to keep backward compatibility with the old configuration file.

Plans no longer depends on shiva to be executed in --last-committed !

Introduction of plan logs for each event made by the plan.

New configuration:

configuration key has been renamed to model and dates key is now a child of model key. duration key has been renamed to offset. name, channel, version and tenant are now mandatory parameters for plan configuration.

  • to be noted: nodes parameter alaways takes priority over es_cluster
  • by default ssl is set to false
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
{
    "name": "plan_name",
    "channel": "channel_name",
    "tenant": "tenant_name",
    "version": "5.0",
    "plan_settings": {
        "cron": "...",
        "persistence": [
            {
                "type": "elasticsearch",
                "nodes": [
                    {
                        "host": "server1",
                        "port": 9200
                    }
                ],
                "es_cluster": "es_search",
                "ssl": false,
                "credentials": {
                    // either
                    "user": "auser",
                    "password": "apass",
                    // or
                    "token": "a_token",
                    "token_type": "ApiKey"
                }
            }
        ]
    },
    "model": {
        ...,
        "dates": {
            ...
        }
    },
    "metrics": {
        "reporters": [
            {
                ...
            }
        ]
    }
}

Please review official plan documentation on how to write model file in 5.7.0

Nodes

Parameter cluster_name have been deprecated in all analytics Elastic nodes. It will be remove in Punchplatform next release. You must now use es_cluster or nodes parameter instead