Skip to content

One Node Deployment

Sizing

The physical or virtual device must have at least:

  • CPU: 1 core
  • Memory: 4GB recommended (LTR,LMR) 8GB recommended (ML)
  • Storage: 20GB recommended

System description

image

Prerequisites

To start the 15-min installation, you need:

  • access to a unix server. Typically a VM.
  • a unix account with sudoers credentials.

    • In this guide we will use SSH_USER for the sake of the example, make sure you replace with your own in commands described below.
    • Check you can execute the ssh SSH_USER@localhost command (put your public key in the authorized_keys files)
  • the deployer package (ie punchplatform-deployer-X.Y.Z.zip), you can ask it in https://punchplatform.com/download-area/punchplatform-deployer/

  • A one node configuration zip provide by a punchplatform team member.

On the server (default: localhost):

  • Ubuntu 16.04 or Later, Centos (RedHat) 7, Macos
  • A configured DNS (or a /etc/hosts correctly updated).

  • Using ubuntu :

1
2
3
4
5
6
7
8
9
sudo apt update
sudo apt install \
  unzip \
  curl \
  git \
  jq \
  python \
  python-pip \
  sshpass
  • Using centos:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
sudo yum update
sudo yum install \
    vim \
    wget \
    unzip \
    curl \
    git \
    jq \
    python \
    sshpass \
    python-pip \
    apr

 sudo systemctl disable firewalld
 sudo systemctl stop firewalld

CentOS: No package available.

You have to enable the EPEL repo, use:
sudo yum --enablerepo=extras install epel-release
This command will install the correct EPEL repository for the CentOS version you are running.
After this you will be able to install python-pip.

  • Install ansible 2.3.0: (If you do not have internet access or local pip repository, use other variant below)
1
2
# ansible 2.3.0 
sudo pip install ansible==2.3.0

If you encounter problem with the previous command, set up this specific required version of ansible from your available repositories, you will find an offline setup tool in the deployment package of punchplatform :

1
2
3
4
5
6
unzip punchplatform-deployer-x.y.z.zip
cd punchplatform-deployer-x.y.z
cd deployment_dependencies
unzip ansible-2.3.0-pippackages.zip
cd ansible-2.3.0-pippackages
sudo ./install.sh

One node configuration

Choose your One node configuration

There are several available One node Configuration.

  • LTR-Light configuration: which gathers the lightest transport application stack (shiva,kafka,zookeeper), elasticsearch and some monitoring applications.

  • LTR configuration: which gathers the basic transport application stack (storm,kafka,zookeeper), elasticsearch and some monitoring applications.

  • LMR configuration: which adds Kibana to visualize your events, Kibana plug-in to ease your platform management, and more monitoring applications.

  • ML configurations: which adds Spark to run your ML and aggregations jobs, Kibana plug-in to ease your ML experience.

You can take them on the pp-academy/conf-samples repository on gitlab if your are allowed to, or ask them to a punchplatform team member.

One node Configuration Environment Setup

Once you have your configuration directory, for example : 1node_ml.zip :

1
2
3
4
5
6
7
8
unzip 1node_XXX.zip
mv 1node_XXX pp-deployment-conf   (cleaner)
cd pp-deployment-conf
echo "export PUNCHPLATFORM_CONF_DIR=`pwd`" >> ~/.bashrc
cd ..
mkdir pp-deployment-logs
cd pp-deployment-logs
echo "export PUNCHPLATFORM_LOG_DIR=`pwd`" >> ~/.bashrc
1
source ~/.bashrc

Check it worked as expected.

The result of the env command must look like:

1
2
3
env | grep PUNCH
PUNCHPLATFORM_LOG_DIR=/Users/dimi/pp-deployment-logs
PUNCHPLATFORM_CONF_DIR=/Users/dimi/pp-deployment-conf

Deployment procedure

Unzip the deployer package

1
unzip punchplatform-deployer-X.Y.Z.zip

Deployer Environment Setup

Update your PATH so as to have the punchplatform-deployer.sh available :

1
2
cd punchplatform-deployer-X.Y.Z
echo "export PATH=`pwd`/bin:$PATH" >> ~/.bashrc
1
source ~/.bashrc

Check it worked as expected.

1
2
echo  $PATH
PATH=/home/vagrant/punchplatform-deployer-5.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/vagrant/.local/bin:/home/vagrant/bin

Deploy the punchpatform 1-node

image

1
punchplatform-deployer.sh deploy -u SSH_USER -Kk

image

Check running as expected

Check everything is up and running as expected

Ubuntu :

LTR :

1
2
3
4
5
6
7
8
sudo supervisorctl status
 elasticsearch                    RUNNING   pid 31042, uptime 0:03:50
 kafka-local                      RUNNING   pid 28375, uptime 0:04:26
 metricbeat                       RUNNING   pid 2963, uptime 0:02:02
 storm-nimbus                     RUNNING   pid 24396, uptime 0:05:28
 storm-supervisor                 RUNNING   pid 24707, uptime 0:05:06
 storm-ui                         RUNNING   pid 24545, uptime 0:05:17
 zookeeper                        RUNNING   pid 21477, uptime 0:06:22

LMR :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
sudo supervisorctl status
 auditbeat                        RUNNING   pid 2958, uptime 0:02:06
 elasticsearch                    RUNNING   pid 31042, uptime 0:03:50
 kafka-local                      RUNNING   pid 28375, uptime 0:04:26
 kibana-mononode                  RUNNING   pid 1886, uptime 0:02:33
 filebeat                         RUNNING   pid 2963, uptime 0:02:07
 metricbeat                       RUNNING   pid 2969, uptime 0:02:02
 storm-nimbus                     RUNNING   pid 24396, uptime 0:05:28
 storm-supervisor                 RUNNING   pid 24707, uptime 0:05:06
 storm-ui                         RUNNING   pid 24545, uptime 0:05:17
 zookeeper                        RUNNING   pid 21477, uptime 0:06:22

ML :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
sudo supervisorctl status
 elasticsearch                    RUNNING   pid 31042, uptime 0:03:50
 kafka-local                      RUNNING   pid 28375, uptime 0:04:26
 kibana-mononode                  RUNNING   pid 1886, uptime 0:02:33
 metricbeat                       RUNNING   pid 2963, uptime 0:02:02
 storm-nimbus                     RUNNING   pid 24396, uptime 0:05:28
 storm-supervisor                 RUNNING   pid 24707, uptime 0:05:06
 storm-ui                         RUNNING   pid 24545, uptime 0:05:17
 spark                            RUNNING   pid 26477, uptime 0:03:22
 zookeeper                        RUNNING   pid 21477, uptime 0:06:22

Centos, Redhat

1
2
3
  sudo systemctl status elasticsearch.service
  sudo systemctl status kafka-local.service
  .........

Launch your pipelines!

Channels and jobs Start Procedure with operator.

image

Synchronize configuration between deployer user and punchplatform operator user using zookeeper

On deployer user :

1
2
3
cd ~/pp-conf
punchplatform-putconf.sh --zookeeper-cluster-name common -pf
punchplatform-putconf.sh --zookeeper-cluster-name common -t mytenant

On punchplatform operator user:

1
2
3
sudo su - admin1
punchplatform-getconf.sh -pf
punchplatform-getconf.sh -t mytenant

Use punchplatform user

Login as punchplatform operator user

1
sudo su - admin1

Start a channel

start the punchplatform-tenant/main_collector channel

1
2
3
 punchctl

 start --channel apache_httpd

Start a job (shiva, spark plan)

1
2
3
 punchctl

 start --channel apache_httpd

System Description

  • kibana port: 5601/TCP
  • elasticsearch REST API: 9200/TCP
  • storm UI port: 8080/TCP
  • Spark master : 8081/TCP
  • Spark slave : 80804/TCP

Troubleshooting

Failed to connect to the host via ssh

1
2
3
cd ~/.ssh
ssh-keygen -t rsa
cat id_rsa.pub >> authorized_keys 

Unable to ssh on localhost

Unable to ssh on localhost (Permission denied):

1
2
# try the following command:
ssh localhost

If it prompts you to enter a password AND you have not password:

1
2
3
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
retry
ssh localhost

punchplatform-deployer.sh is not found

punchplatform-deployer.sh is not found:

  • make sure you are using the deployment unix account (ie not punchadmin)
  • then:
1
source ~/.bashrc

warning LC_ALL, LC_NUMERIC

Read How to set locale to remove this warning.

punchplatform-channel.sh is not found

punchplatform-channel.sh is not found:

  • make sure you are using the operator unix account (i.e punchadmim)
1
sudo su punchadmin

Unable to locate package

If you have a error like Unable to locate package python-pip, update your source.list and add universe keyword on each line.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# sudo vim /etc/apt/sources.list

deb http://archive.ubuntu.com/ubuntu bionic main universe
deb http://archive.ubuntu.com/ubuntu bionic-security main universe 
deb http://archive.ubuntu.com/ubuntu bionic-updates main universe

# save and quit

sudo apt update
sudo apt install python-pip

or

No package jq available
1
2
3
wget -O jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
chmod +x ./jq
cp jq /usr/bin
No package python-pip available
1
2
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
sudo python get-pip.py