Skip to content

Level 2: Storm

Before going through this guide, ensure you went through the level 1 : getting started deployer guide to get a zookeeper cluster.

This tutorial explains how to add a Storm cluster to an existing platform. It is interesting as Storm requires a zookeeper cluster. We will thus deploy a Storm cluster leveraging the zookeeper cluster you installed in the level 1 guide.

Again, the deployer can be your macos or linux laptop. The starting situation is unchanged.

image

Configure Storm

Edit first the punchplatform.properties with your favorite text editor file and include the following content

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
"storm" : {
    "clusters" : {
        "main": {
            "master" : {
                "servers" : ["server1", "server2", "server3"],
                "cluster_production_address" : "server1"
                },
            "ui" : {
                "servers" : ["server1", "server2", "server3"],
                "cluster_admin_url": "server1:8080"
            },
            "slaves" : ["server1", "server2", "server3"],
            "zk_cluster" : "common",
            "zk_root" : "storm-1.1.3-main",
            "storm_workers_by_punchplatform_supervisor" : 15,
            "workers_childopts" : "-Xmx1G",
            "supervisor_memory_mb" : 8192,
            "supervisor_cpu" : 4
        }
    },
    "install_dir": "/data/opt/apache-storm-1.2.2"
}

To get more details on each subsection, please read the punchplatform.properties documentation.

Next edit the punchplatform-deployment.settings file and include the following:

1
2
3
4
{
  "storm_version" : "apache-storm-1.2.2",
  "storm_nimbus_nodes_production_interface" : "eth1",
}

To get more details on each subsection, please read the punchplatform-deployment.settings documentation.

Info

Adapt the properties to suit your environment in particular the zookeeper_nodes_production_interface, as well as unix user and groups.

For example if you use a vagrant box the user is likely to be 'ubuntu' and the network interface 'enp0s8'. In doubt you can find out what is the right network intefaces using the ifconfig command.

Make sure you have the requirements installed on the deployer and target servers. In particular the deployer must have ssh access to the target server.

Check and Generate Your Configurations

First execute this command:

1
$ punchplatform-deployer.sh --generate-inventory

Or in short:

1
$ punchplatform-deployer.sh -gi

This command generates a complete set of so-called inventories from your two configuration files. It should output something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
$ punchplatform-deployer.sh --generate-inventory
LOG: checking your dependencies...
LOG: all the required packages are installed !

LOG: audit the punchplatform configuration...
LOG: found punchplatform.properties: /Users/dimi/Punch/deployer/conf/punchplatform.properties
LOG: found punchplatform-deployment.settings: /Users/dimi/Punch/deployer/conf/punchplatform-deployment.settings

LOG: generating deployment configuration from punchplatform.properties + punchplatform-deployment.settings...
LOG: Following groups and tags can bu used as additional parameters like: punchplatform-deployer.sh --deploy [-l group1,group2] [-t tag1,tag2]
  - group: punchplatform_cluster     tag: N/A
  - group: zookeeper_servers      tag: zookeeper
  - group: storm_servers      tag: storm

LOG: configuration is ready for the deployment !

If that succeeds you can proceed.

Deploy Your Component(s)

From the deployer, make sure you can access your target server using ssh. Depending on your ssh configuration this may require a ssh password. Simply execute the following command.

1
punchplatform-deployer.sh deploy -Kk --tags storm

Note

You will be prompted (or not) for a ssh password. This depends on your ssh configuration.

Check your deployment

1
2
3
4
5
6
7
8
9
ssh server1
# On Ubuntu
sudo supervisorctl status
storm-nimbus                     RUNNING   pid 22239, uptime 0:00:35
storm-supervisor                 RUNNING   pid 22186, uptime 0:00:44
storm-ui                         RUNNING   pid 22213, uptime 0:00:37
zookeeper                        RUNNING   pid 903, uptime 1:16:39
# On systemctl
sudo systemctl status

As you can see you now have a ready to use Storm cluster. Move on to the next level 3 guide to add an operator environment so that you can define and run punchplatform channels.