Per Stream Destinations

The lumberjack, syslog and syslog udp bolts all support a configuration option that allows you to have a single bolt in charge of dispatching the data received on respectively storm stream1 stream2 .. streamN to a (respectively) destination1 destination2 destinationN.

Here is the way to configure this. Let us start with a regular mono-stream example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
     "type" : "syslog_udp_bolt",
     "bolt_settings" : {
         "destination" : [ {"host" : "127.0.0.1", "port" : 9999}]
     },
     "storm_settings" : {
       ...
       "subscribe" : [ 
           { "component" : "spout1", "stream" : "logs", "grouping": "localOrShuffle" } 
       ]
     }
 }

Say now you have two streams , and you need to send data from to 127.0.0.1:9999 and from logs2 to 127.0.0.1:8888. Here is how to configure it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
{
     "type" : "syslog_udp_bolt",
     "bolt_settings" : {
         "destinations" : [
           { 
             "stream" : "logs1",
             "destination" : [ {"host" : "127.0.0.1", "port" : 9999}]
           },
           { 
             "stream" : "logs2",
             "destination" : [ {"host" : "127.0.0.1", "port" : 8888}]
           }
         ]  
     },
     "storm_settings" : {
       ...
       "subscribe" : [ 
           { "component" : "spout1", "stream" : "logs1", "grouping": "localOrShuffle" } 
           { "component" : "spout2", "stream" : "logs2", "grouping": "localOrShuffle" } 
       ]
     }
 }

Info

make sure you understand spouts and bolts stream and field fundamental concepts.