Introducing the Standalone PunchPlatform¶

The Getting Started guide that follows leverages the standalone PunchPlatform.

In minutes you will run, operate, test and hopefully develop on a complete platform. It is also very useful in production, coupled with your production platform, to run prototype or test setups either in sandbox mode, or directly on the production traffic.

If you need one, check the PunchPlatform Site download area.

Important

What you cannot do is to go production with it, that’s not its purpose. For that, use the official deployment packages, taking care of properly installing the PunchPlatform components. The PunchPlatform team will provide production support only on versions installed using deployment packages.

Topics

Requirements
Tutorial Tracks
Two Minutes Tour : A Single-Process Log Channel
Run your LogManagement service
Uninstall
Troubleshooting
What to do next

Requirements ¶

At least 8 GB of memory. You need more memory to support many and/or more complex Storm or Spark topologies (e.g. input and output channels, tenants, …).

The standalone runs fine on the following systems:

Mac OS X,
Ubuntu 14.04LTS or later (64bits),
Debian 7 or later,
CEntOS/RHEL 7 (64bits),
SuSE 12 (64bits)

In all cases check you have the required dependencies:

bash : (well, basically: unix).
jq 1.5
curl
python 2.7 and virtualenv

on MacOs:¶

Note

we assume you have brew to deal with package installation.

$ brew install jq curl
$ sudo easy_install pip
$ pip install virtualenv

on Debian-based systems (Ubuntu, Debian):¶

$ sudo apt install curl jq python-minimal openjdk-8-jdk unzip python-pip python-virtualenv

CEntOS/RedHat:¶

$ sudo yum install zip unzip
$ curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py" && sudo python get-pip.py
$ pip install virtualenv
$ sudo wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 -O /usr/bin/jq && sudo chmod +x /usr/bin/jq
$ sudo yum install java-1.8.0-openjdk-devel

SuSE:¶

$ sudo wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 -O /usr/bin/jq && sudo chmod +x /usr/bin/jq
$ wget https://pypi.python.org/packages/d4/0c/9840c08189e030873387a73b90ada981885010dd9aea134d6de30cd24cb8/virtualenv-15.1.0.tar.gz#md5=44e19f4134906fe2d75124427dc9b716 ; tar -xvf virtualenv-15.1.0.tar.gz ; python virtualenv-15.1.0/setup.py install
$ wget https://bootstrap.pypa.io/get-pip.py ; python get.pip.py
$ download the oracle jdk 8 on oracle offcial website, unarchive the archive (tar.gz), move the content in the directory /usr/java, then :
$ sudo update-alternatives --install "/usr/bin/java" "java" "/usr/java/jdk1.8.0_131/bin/java" 1
$ sudo update-alternatives --set java /usr/java/jdk1.8.0_131/bin/java
$ sudo update-alternatives --install "/usr/bin/jps" "jps" "/usr/java/jdk1.8.0_131/bin/jps" 1

Note

a troubleshooting procedure for the standalone is available at Troubleshooting your standalone

That’s all !

Tutorial Tracks ¶

The standalone platform provides a number of ready to start with scenarii. Before you start with one, here is a quick overview. The following examples are part of the example tenant mytenant. Remember the PunchPlatorm is multi-tenant. Everything you do is defined as part of a well defined and isolated tenant.

Two Minutes Tour : A Single-Process Log Channel ¶

This chapter assumes you already downloaded the standalone PunchPlatform archive. Unzip it. That is, you should have a ~/punchplatform-standalone-x.y.z directory.

Note

You must install the Standalone as non-root user.

$ ./install.sh [-s]

Add the -s option for normal installation. (we advise you answer yes when asked for, it will give you direct access to the PunchPlatform commands)

This unarchives all our friends (Kafka, Storm, Elasticsearch, Punchplatform Admin, etc…), and patches their configuration files (cleanly) so that you have a simple local PunchPlatform setup. If you want to have a more complex setup (say with more kafka layers, or go to a real distributed setup), then you’ll have to update the configuration files. Check the Production Deployment.

Note

Additional components are available in the standalone. Use ‘./install.sh –help’ to get more information. As an example, if you plan to use Spark, you can simply specify --with-spark at install time.

Once installed, start a new terminal.

Assuming you are happy with a local setup, you can start the various open-source COTS (Zookeeper, Storm, Elasticsearch, Kibana). Do this:

$ punchplatform-admin.sh start

Check you have things running:

$ punchplatform-admin.sh status

Before going further, check the manual pages :

$ man punchplatform-admin.sh
$ man punchplatform-channel.sh
$ etc.

And configure your Kibana for first use:

$ punchplatform-setup-kibana.sh --import

Good. You have now a complete and running platform installed.

Run your LogManagement service ¶

The PunchPlatform can serve many purposes. One (the key one) is to act as a CyberSecurity LogManagement solution. We will go through that usage now.

Before following the few steps described hereafter, it’s good for you to understand the main (not to say the only) concept of the PunchPlatform : channel. A channel is an end to end setup to transport your data (that is : logs) from one end up to one or several backend(s) (in particular Elasticsearch) . In the pipe you plug in processing (log parsers).

This first tour will let you generate a simple setup where the input-processing-output functions are all deployed as part of a single-process storm job.

Doing so, you will understand most of the PunchPlatform concepts.

Let’s start in the configuration directory of PunchPlatform:

cd $PUNCHPLATFORM_CONF_DIR

In the tenants/mytenant/configurations_for_channels_generation/lmc folder, have a look at the few files. For example the sourcefire_channel.json file. It contains the human-friendly configuration for the sourcefire log channel.

Each file is associated with a “channel_structure_profile” pointing to the templates to be used for generating the detailed configuration files.

$ punchplatform-channel.sh --configure tenants/mytenant/configurations_for_channels_generation/lmc

This command resulted in creating the configuration files in $PUNCHPLATFORM_CONF_DIR/tenants/mytenant/channels/.

Check the PunchPlatform Admin on http://localhost:5000/#/channels. You should see shortly the channels appear.

You can now start the Sourcefire channel from the GUI, or by simply typing :

$ punchplatform-channel.sh --start mytenant/sourcefire

Check out the GUI. Your channel should now be started and exhibit a small green square. It basically tells you that your channel is alive and performing well.

Now inject some logs. The PunchPlatform Injector can take a single injection file or a directory, in which case it launches injection for all injection files.

$ punchplatform-log-injector.sh -c resources/injector/mytenant/sourcefire_injector.json

Check Kibana on http://localhost:5601 and Storm UI on http://localhost:8080,

When you are done stop injection (Ctrl+C) and stop your channel:

$ punchplatform-channel.sh --stop mytenant/sourcefire

You are ready for high performance stream event processing !

Uninstall ¶

Only if you want to uninstall your PunchPlatform!

First, stop the PunchPlatform.

$ punchplatform-admin.sh stop

Execute the uninstall.sh script.

$ $PUNCHPLATFORM_CONF_DIR/../uninstall.sh

(optional) Save any configuration you made in $PUNCHPLATFORM_CONF_DIR

Remove the punchplatform-standalone-x.y.z directory.

Troubleshooting ¶

If an error happens, please have a look at the Troubleshooting section. There is a section dedicated to the Regular Errors.

What to do next ¶

You need to get the basic concepts of a PunchPlatform in the Next section.