Level 2: Storm¶
Before going through this guide, ensure you went through the level 1 : getting started deployer guide to get a zookeeper cluster.
This tutorial explains how to add a Storm cluster to an existing platform. It is interesting as Storm requires a zookeeper cluster. We will thus deploy a Storm cluster leveraging the zookeeper cluster you installed in the level 1 guide.
Again, the deployer can be your macos or linux laptop. The starting situation is unchanged.
Configure Storm¶
Edit first the punchplatform.properties
with your favorite
text editor file and include the following content
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | "storm" : { "clusters" : { "main": { "master" : { "servers" : ["server1", "server2", "server3"], "cluster_production_address" : "server1" }, "ui" : { "servers" : ["server1", "server2", "server3"], "cluster_admin_url": "server1:8080" }, "slaves" : ["server1", "server2", "server3"], "zk_cluster" : "common", "zk_root" : "storm-1.1.3-main", "storm_workers_by_punchplatform_supervisor" : 15, "workers_childopts" : "-Xmx1G", "supervisor_memory_mb" : 8192, "supervisor_cpu" : 4 } }, "install_dir": "/data/opt/apache-storm-1.2.2" } |
To get more details on each subsection, please read the punchplatform.properties documentation.
Next edit the punchplatform-deployment.settings
file and include the
following:
1 2 3 4 | { "storm_version" : "apache-storm-1.2.2", "storm_nimbus_nodes_production_interface" : "eth1", } |
To get more details on each subsection, please read the punchplatform-deployment.settings documentation.
Info
Adapt the properties to suit your environment in particular the
zookeeper_nodes_production_interface
, as well as unix user and
groups.
For example if you use a vagrant box the user is likely to be
'ubuntu' and the network interface 'enp0s8'. In doubt you can find out what is the right
network intefaces using the ifconfig
command.
Make sure you have the requirements installed on the deployer and target servers. In particular the deployer must have ssh access to the target server.
Check and Generate Your Configurations¶
First execute this command:
1 | $ punchplatform-deployer.sh --generate-inventory |
Or in short:
1 | $ punchplatform-deployer.sh -gi |
This command generates a complete set of so-called inventories from your two configuration files. It should output something like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | $ punchplatform-deployer.sh --generate-inventory LOG: checking your dependencies... LOG: all the required packages are installed ! LOG: audit the punchplatform configuration... LOG: found punchplatform.properties: /Users/dimi/Punch/deployer/conf/punchplatform.properties LOG: found punchplatform-deployment.settings: /Users/dimi/Punch/deployer/conf/punchplatform-deployment.settings LOG: generating deployment configuration from punchplatform.properties + punchplatform-deployment.settings... LOG: Following groups and tags can bu used as additional parameters like: punchplatform-deployer.sh --deploy [-l group1,group2] [-t tag1,tag2] - group: punchplatform_cluster tag: N/A - group: zookeeper_servers tag: zookeeper - group: storm_servers tag: storm LOG: configuration is ready for the deployment ! |
If that succeeds you can proceed.
Deploy Your Component(s)¶
From the deployer, make sure you can access your target server using ssh. Depending on your ssh configuration this may require a ssh password. Simply execute the following command.
1 | punchplatform-deployer.sh deploy -Kk --tags storm |
Note
You will be prompted (or not) for a ssh password. This depends on your ssh configuration.
Check your deployment¶
1 2 3 4 5 6 7 8 9 | ssh server1 # On Ubuntu sudo supervisorctl status storm-nimbus RUNNING pid 22239, uptime 0:00:35 storm-supervisor RUNNING pid 22186, uptime 0:00:44 storm-ui RUNNING pid 22213, uptime 0:00:37 zookeeper RUNNING pid 903, uptime 1:16:39 # On systemctl sudo systemctl status |
As you can see you now have a ready to use Storm cluster. Move on to the next level 3 guide to add an operator environment so that you can define and run punchplatform channels.