Introducing the Standalone PunchPlatform¶
The Getting Started guide that follows leverages the standalone PunchPlatform.
In minutes you will run, operate, test and hopefully develop on a complete platform. It is also very useful in production, coupled with your production platform, to run prototype or test setups either in sandbox mode, or directly on the production traffic.
If you need one, check the PunchPlatform Site download area.
What you cannot do is to go production with it, that’s not its purpose. For that, use the official deployment packages, taking care of properly installing the PunchPlatform components. The PunchPlatform team will provide production support only on versions installed using deployment packages.
At least 8 GB of memory. You need more memory to support many and/or more complex Storm or Spark topologies (e.g. input and output channels, tenants, …).
The standalone runs fine on the following systems:
- Mac OS X,
- Ubuntu 14.04LTS or later (64bits),
- Debian 7 or later,
- CEntOS/RHEL 7 (64bits),
- SuSE 12 (64bits)
In all cases check you have the required dependencies:
- bash : (well, basically: unix).
- jq 1.5
- python 2.7 and virtualenv
we assume you have brew to deal with package installation.
$ brew install jq curl $ sudo easy_install pip $ pip install virtualenv
$ sudo apt install curl jq python-minimal openjdk-8-jdk unzip python-pip python-virtualenv
$ sudo yum install zip unzip $ curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py" && sudo python get-pip.py $ pip install virtualenv $ sudo wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 -O /usr/bin/jq && sudo chmod +x /usr/bin/jq $ sudo yum install java-1.8.0-openjdk-devel
$ sudo wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 -O /usr/bin/jq && sudo chmod +x /usr/bin/jq $ wget https://pypi.python.org/packages/d4/0c/9840c08189e030873387a73b90ada981885010dd9aea134d6de30cd24cb8/virtualenv-15.1.0.tar.gz#md5=44e19f4134906fe2d75124427dc9b716 ; tar -xvf virtualenv-15.1.0.tar.gz ; python virtualenv-15.1.0/setup.py install $ wget https://bootstrap.pypa.io/get-pip.py ; python get.pip.py $ download the oracle jdk 8 on oracle offcial website, unarchive the archive (tar.gz), move the content in the directory /usr/java, then : $ sudo update-alternatives --install "/usr/bin/java" "java" "/usr/java/jdk1.8.0_131/bin/java" 1 $ sudo update-alternatives --set java /usr/java/jdk1.8.0_131/bin/java $ sudo update-alternatives --install "/usr/bin/jps" "jps" "/usr/java/jdk1.8.0_131/bin/jps" 1
a troubleshooting procedure for the standalone is available at Troubleshooting your standalone
That’s all !
The standalone platform provides a number of ready to start with scenarii. Before you start with one, here is a quick overview. The following examples are part of the example tenant mytenant. Remember the PunchPlatorm is multi-tenant. Everything you do is defined as part of a well defined and isolated tenant.
This chapter assumes you already downloaded the standalone PunchPlatform archive. Unzip it. That is, you should have a ~/punchplatform-standalone-x.y.z directory.
You must install the Standalone as non-root user.
$ ./install.sh [-s]
-s option for normal installation. (we advise you answer yes when
asked for, it will give you direct access to the PunchPlatform commands)
This unarchives all our friends (Kafka, Storm, Elasticsearch, Punchplatform Admin, etc…), and patches their configuration files (cleanly) so that you have a simple local PunchPlatform setup. If you want to have a more complex setup (say with more kafka layers, or go to a real distributed setup), then you’ll have to update the configuration files. Check the Production Deployment.
Additional components are available in the standalone. Use ‘./install.sh –help’ to get more information. As an example, if you plan to use Spark, you can simply specify
--with-spark at install time.
Once installed, start a new terminal.
Assuming you are happy with a local setup, you can start the various open-source COTS (Zookeeper, Storm, Elasticsearch, Kibana). Do this:
$ punchplatform-admin.sh start
Check you have things running:
$ punchplatform-admin.sh status
Before going further, check the manual pages :
$ man punchplatform-admin.sh $ man punchplatform-channel.sh $ etc.
And configure your Kibana for first use:
$ punchplatform-setup-kibana.sh --import
Good. You have now a complete and running platform installed.
The PunchPlatform can serve many purposes. One (the key one) is to act as a CyberSecurity LogManagement solution. We will go through that usage now.
Before following the few steps described hereafter, it’s good for you to understand the main (not to say the only) concept of the PunchPlatform : channel. A channel is an end to end setup to transport your data (that is : logs) from one end up to one or several backend(s) (in particular Elasticsearch) . In the pipe you plug in processing (log parsers).
This first tour will let you generate a simple setup where the input-processing-output functions are all deployed as part of a single-process storm job.
Doing so, you will understand most of the PunchPlatform concepts.
Let’s start in the configuration directory of PunchPlatform:
In the tenants/mytenant/configurations_for_channels_generation/lmc folder, have a look at the few files. For example the sourcefire_channel.json file. It contains the human-friendly configuration for the sourcefire log channel.
Each file is associated with a “channel_structure_profile” pointing to the templates to be used for generating the detailed configuration files.
$ punchplatform-channel.sh --configure tenants/mytenant/configurations_for_channels_generation/lmc
This command resulted in creating the configuration files in
Check the PunchPlatform Admin on http://localhost:5000/#/channels. You should see shortly the channels appear.
You can now start the Sourcefire channel from the GUI, or by simply typing :
$ punchplatform-channel.sh --start mytenant/sourcefire
Check out the GUI. Your channel should now be started and exhibit a small green square. It basically tells you that your channel is alive and performing well.
Now inject some logs. The PunchPlatform Injector can take a single injection file or a directory, in which case it launches injection for all injection files.
$ punchplatform-log-injector.sh -c resources/injector/mytenant/sourcefire_injector.json
When you are done stop injection (Ctrl+C) and stop your channel:
$ punchplatform-channel.sh --stop mytenant/sourcefire
You are ready for high performance stream event processing !
Only if you want to uninstall your PunchPlatform!
First, stop the PunchPlatform.
$ punchplatform-admin.sh stop
Execute the uninstall.sh script.
(optional) Save any configuration you made in $PUNCHPLATFORM_CONF_DIR
Remove the punchplatform-standalone-x.y.z directory.
If an error happens, please have a look at the Troubleshooting section. There is a section dedicated to the Regular Errors.