Skip navigation links

Package org.thales.punch.libraries.storm.spout

Punchplatform spouts package.

See: Description

Package org.thales.punch.libraries.storm.spout Description

Punchplatform spouts package.

Overview


The punchplatform spouts are standard storm spouts. It is easy to code your own and make them available to punch topologies. The few key concepts regarding the way the data is forwarded as streams and fields is described here. Refer to each spout javadoc page for details.

Data, Streams and Fields


User Data

Data comes in to a spout from an external source : socket, kafka, files. In some case you receive a single line (i.e. a string), in some other case you receive a map of key value elements.

In all case what the spout does is to take these value(s) and forward them as part of a storm stream, under the form of a so called tuple which actually is a key-value map.

It is your job to configure the spout to emit the fields you want as part of tuples inside streams. Punchplatform topology files allows you to design arbitrary DAGs. You really can invent the way your data is transported, processed and routed to one or several final destinations such as Elasticearch or an archiving backend.

System Data

You can also configure your spout to take care two additional reserved punchplatform streams whose names and semantics have a special meaning.

If you run production it is likely you need to properly leverage these two great punchplatform features. Your goal is to define the path so that both latency metrics and punch error travel up to their final destination.

Example

Here is an example Spout configuration. What you see here is that spout is configured to :


 "spouts" : [
  {
   "type" : "one_of_the_spout",
   "spout_settings" : { ... },
   "storm_settings" : {
    "publish" : [
     { "stream" : "logs", "fields" : ["log"] },
     { "stream" : "_ppf_metrics", "fields" : ["_ppf_latency"] },
     { "stream" : "_ppf_errors", "fields" : ["_ppf_error"] }
    ]
   }
  }
 
If instead you configure it like this :

 "spouts" : [
  {
   "type" : "one_of_the_spout",
   "spout_settings" : { ... },
   "storm_settings" : {
    "publish" : [
     { "stream" : "logs", "fields" : ["log"] }
    ]
   }
  }
 
Only the user data will be forwarded as tuple in your topology.
Skip navigation links

Copyright © 2022. All rights reserved.