Per Stream Destinations

The lumberjack, syslog and syslog udp output nodes all support a configuration option that allows you to have a single node in charge of dispatching the data received on respectively storm stream1 stream2 .. streamN to a (respectively) destination1 destination2 destinationN.

Here is the way to configure this. Let us start with a regular mono-stream example.

{
     "type" : "syslog_udp_bolt",
     "settings" : {
         "destination" : [ {"host" : "127.0.0.1", "port" : 9999}]
     },
       ...
       "subscribe" : [ 
           { "component" : "spout1", "stream" : "logs", "grouping": "localOrShuffle" } 
       ]
 }

Say now you have two streams , and you need to send data from to 127.0.0.1:9999 and from logs2 to 127.0.0.1:8888. Here is how to configure it:

{
     "type" : "syslog_udp_bolt",
     "settings" : {
         "destinations" : [
           { 
             "stream" : "logs1",
             "destination" : [ {"host" : "127.0.0.1", "port" : 9999}]
           },
           { 
             "stream" : "logs2",
             "destination" : [ {"host" : "127.0.0.1", "port" : 8888}]
           }
         ]  
     },
       ...
       "subscribe" : [ 
           { "component" : "spout1", "stream" : "logs1", "grouping": "localOrShuffle" } 
           { "component" : "spout2", "stream" : "logs2", "grouping": "localOrShuffle" } 
       ]
 }

Info

make sure you understand spouts and bolts stream and field fundamental concepts.