One Node Deployment¶
Deprecated Page: Will be updated soon
The physical or virtual device must have at least:
- CPU: 1 core
- Memory: 4GB recommended (LTR,LMR) 8GB recommended (ML)
- Storage: 20GB recommended
To start the 15-min installation, you need:
- access to a unix server. Typically a VM.
a unix account with sudoers credentials.
- In this guide we will use SSH_USER for the sake of the example, make sure you replace with your own in commands described below.
- Check you can execute the
ssh SSH_USER@localhostcommand (put your public key in the authorized_keys files)
the deployer package (ie
punchplatform-deployer-X.Y.Z.zip), you can ask it in https://punchplatform.com/download-area/punchplatform-deployer/
- A one node configuration zip provide by a punchplatform team member.
On the server (default: localhost):
- Ubuntu 16.04 or Later, Centos (RedHat) 7, Macos
A configured DNS (or a /etc/hosts correctly updated).
Using ubuntu :
sudo apt update sudo apt install \ unzip \ curl \ git \ jq \ python \ python-pip \ sshpass
- Using centos:
sudo yum update sudo yum install \ vim \ wget \ unzip \ curl \ git \ jq \ python \ sshpass \ python-pip \ apr sudo systemctl disable firewalld sudo systemctl stop firewalld
CentOS: No package
You have to enable the EPEL repo, use:
sudo yum --enablerepo=extras install epel-release
This command will install the correct EPEL repository for the CentOS version you are running.
After this you will be able to install python-pip.
- Install ansible 2.9.0: (If you do not have internet access or local pip repository, use other variant below)
If you encounter problem with the previous command, set up this specific required version of ansible from your available repositories, you will find an offline setup tool in the deployment package of punchplatform :
# ansible 2.9.0 sudo pip install ansible==2.9.0
unzip punchplatform-deployer-x.y.z.zip cd punchplatform-deployer-x.y.z cd deployment_dependencies unzip ansible-2.9.0-pippackages.zip cd ansible-2.9.0-pippackages sudo ./install.sh
One node configuration¶
Choose your One node configuration¶
There are several available One node Configuration.
LTR-Light configuration: which gathers the lightest transport application stack (shiva,kafka,zookeeper), elasticsearch and some monitoring applications.
LTR configuration: which gathers the basic transport application stack (storm,kafka,zookeeper), elasticsearch and some monitoring applications.
LMR configuration: which adds Kibana to visualize your events, Kibana plug-in to ease your platform management, and more monitoring applications.
ML configurations: which adds Spark to run your ML and aggregations jobs, Kibana plug-in to ease your ML experience.
You can take them on the pp-academy/conf-samples repository on gitlab if your are allowed to, or ask them to a punchplatform team member.
One node Configuration Environment Setup¶
Once you have your configuration directory, for example : 1node_ml.zip :
unzip 1node_XXX.zip mv 1node_XXX pp-deployment-conf (cleaner) cd pp-deployment-conf echo "export PUNCHPLATFORM_CONF_DIR=`pwd`" >> ~/.bashrc cd .. mkdir pp-deployment-logs cd pp-deployment-logs echo "export PUNCHPLATFORM_LOG_DIR=`pwd`" >> ~/.bashrc
Check it worked as expected.¶
The result of the
env command must look like:
env | grep PUNCH PUNCHPLATFORM_LOG_DIR=/Users/dimi/pp-deployment-logs PUNCHPLATFORM_CONF_DIR=/Users/dimi/pp-deployment-conf
Unzip the deployer package¶
Deployer Environment Setup¶
Update your PATH so as to have the
punchplatform-deployer.sh available :
cd punchplatform-deployer-X.Y.Z echo "export PATH=`pwd`/bin:$PATH" >> ~/.bashrc echo "source _punchplatform-bash-completion.sh" >> ~/.bashrc
Check it worked as expected.¶
echo $PATH PATH=/home/vagrant/punchplatform-deployer-5.3.0/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/vagrant/.local/bin:/home/vagrant/bin
Deploy the punchplatform 1-node¶
punchplatform-deployer.sh --deploy -u SSH_USER -Kk
Check running as expected¶
Check everything is up and running as expected
sudo supervisorctl status elasticsearch RUNNING pid 31042, uptime 0:03:50 kafka-local RUNNING pid 28375, uptime 0:04:26 metricbeat RUNNING pid 2963, uptime 0:02:02 storm-nimbus RUNNING pid 24396, uptime 0:05:28 storm-supervisor RUNNING pid 24707, uptime 0:05:06 storm-ui RUNNING pid 24545, uptime 0:05:17 zookeeper RUNNING pid 21477, uptime 0:06:22
sudo supervisorctl status auditbeat RUNNING pid 2958, uptime 0:02:06 elasticsearch RUNNING pid 31042, uptime 0:03:50 kafka-local RUNNING pid 28375, uptime 0:04:26 kibana-mononode RUNNING pid 1886, uptime 0:02:33 filebeat RUNNING pid 2963, uptime 0:02:07 metricbeat RUNNING pid 2969, uptime 0:02:02 storm-nimbus RUNNING pid 24396, uptime 0:05:28 storm-supervisor RUNNING pid 24707, uptime 0:05:06 storm-ui RUNNING pid 24545, uptime 0:05:17 zookeeper RUNNING pid 21477, uptime 0:06:22
sudo supervisorctl status elasticsearch RUNNING pid 31042, uptime 0:03:50 kafka-local RUNNING pid 28375, uptime 0:04:26 kibana-mononode RUNNING pid 1886, uptime 0:02:33 metricbeat RUNNING pid 2963, uptime 0:02:02 storm-nimbus RUNNING pid 24396, uptime 0:05:28 storm-supervisor RUNNING pid 24707, uptime 0:05:06 storm-ui RUNNING pid 24545, uptime 0:05:17 spark RUNNING pid 26477, uptime 0:03:22 zookeeper RUNNING pid 21477, uptime 0:06:22
sudo systemctl status elasticsearch.service sudo systemctl status kafka-local.service .........
Launch your pipelines!¶
Channels and jobs Start Procedure with operator.
Synchronize configuration between deployer user and punchplatform operator user using zookeeper¶
On deployer user :
cd ~/pp-conf punchplatform-putconf.sh --zookeeper-cluster-name common -pf punchplatform-putconf.sh --zookeeper-cluster-name common -t mytenant
sudo su - admin1 punchplatform-getconf.sh -pf punchplatform-getconf.sh -t mytenant
Use punchplatform user¶
Login as punchplatform operator user
sudo su - admin1
Start a channel¶
start the punchplatform-tenant/main_collector channel
channelctl start --channel apache_httpd
Start a job (shiva, spark plan)¶
channelctl start --channel apache_httpd
- kibana port: 5601/TCP
- elasticsearch REST API: 9200/TCP
- storm UI port: 8080/TCP
- Spark master : 8081/TCP
- Spark slave : 80804/TCP
Failed to connect to the host via ssh¶
cd ~/.ssh ssh-keygen -t rsa cat id_rsa.pub >> authorized_keys
Unable to ssh on localhost¶
Unable to ssh on localhost (Permission denied):
# try the following command: ssh localhost
If it prompts you to enter a password AND you have not password:
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys retry ssh localhost
punchplatform-deployer.sh is not found¶
punchplatform-deployer.sh is not found:
- make sure you are using the deployment unix account (ie not punchadmin)
warning LC_ALL, LC_NUMERIC¶
Read How to set locale to remove this warning.
channelctl is not found¶
channelctl is not found:
- make sure you are using the operator unix account (i.e punchadmin)
sudo su punchadmin
Unable to locate package¶
If you have a error like
Unable to locate package python-pip, update your source.list
universe keyword on each line.
# sudo vim /etc/apt/sources.list deb http://archive.ubuntu.com/ubuntu bionic main universe deb http://archive.ubuntu.com/ubuntu bionic-security main universe deb http://archive.ubuntu.com/ubuntu bionic-updates main universe # save and quit sudo apt update sudo apt install python-pip
No package jq available¶
wget -O jq https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 chmod +x ./jq cp jq /usr/bin
No package python-pip available¶
curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py" sudo python get-pip.py