Siddhi Rules

Overview

In this chapter we explain the details of invoking Siddhi rules from within a punchlet. Siddhi is a complex event processing rule engine. Here is how it is depicted on their documentation:

../../../../_images/siddhi-overview.png

Here is a rule example:

//
// Aggregate events on 3 second batch windows, and generate a single event
// with the sum of the price of each event.
//
// The data is received data on a stream "input"
// Every 3 seconds output data is generated on stream "output"
//
define stream input (symbol string, price float, volume int);
@info(name = 'query') \
  from input#window.timeBatch(3 sec) \
  select symbol,sum(price) as sumPrice, volume \
  insert all events into output ;

As explained, rules are traversed by streams of data. Each stream is named. In our example the data is expected to arrive on the input stream. That data consists of 3-values tuples of type (resp.) string, float, and int. Should the rule match (i.e. the select matches), the rule will emit a new event onto a new named data stream, here output. That output data consists in 3-values tuples of type (resp.) string, float and float.

With Siddhi rules you can express many (many) complex use cases from simple alerting to fraud detection including machine learning processing. Check out the SiddhiQl documentation.

Siddhi and Punch

Part of the work of deploying a Siddhi rule consists in preparing the input data, selecting the required fields, and formatting the output data. The punchplatform provides a Cep Bolt configuration guide. To use it you must design a topology and take care of sending the right storm stream and field to match the ones of your rule.

Using siddhi directly from within a punchlet is another option explained here. Using punch the pre and post processing is extremely easy to write, and requires a few lines of code.

In a punchlet you must simply feed the rule with input tuples that represent the rule input stream, and in turn collect the tuples generated on the output stream(s). The question though is : how do you get back the events generated by a siddhi rule ? Here is the magic : it is explained in the punch code that follows:

// This punchlet is a standalone example meant to illustrate
// how to pass input data to a Siddhi rule, and get back the results.
//
// The annotation hereafter associates a rule to this punchlet. This makes it
// easy to run it using the punchplatform-puncher.sh tool or directly
// from the Sublime text editor.
//
// @resource resources/siddhi/cep/low_battery.rule
{

  Tuple event;

  // For the sake of the example we will insert a sequence of events
  // in our rule. A real punchlet would typically send each incoming tuple
  // as event to the rule.
  Double[] batteryLevels = { 0.8 , 0.7, 0.6, 0.4, 0.3, 0.15, 0.10, 0.08 };

  for (Double batteryLevel : batteryLevels) {

    // input tuples must be a dictionary with a field identifying the
    // the siddhi input stream. That is why you have [input] here
    event:[input][local_uuid] = "uuid1";
    event:[input][device_id] = 3;
    event:[input][battery] = batteryLevel;

    // we need a placeholder to get the matched event back, if any.
    Tuple output;

    // All set, the Siddhi punch operator is ready to use.
    // Here "low_battery" corresponds to our rule file name.
    // the second argument indicates we are only interested in the
    // output stream. If you omit it, you will get both input and output
    // events.
    if (siddhi("low_battery", "output")

        // send the input tuple. What happens then is the Siddhi rule logic
        // maybe nothing, maybe some matches. In which case the rule will generate
        // some inputs.
        .send(event)

        // In case there are some match, you get them back into your output tuple.
        .into(output)) {

      // Do what you need with the output. In this example we just print it out.
      // In a real example you would typically propagate your event downstream
      // your punchplatform pipeline.
      print("the rule fired for battery level "+batteryLevel);
      print(output);

    } else {

      print("the rule did not fire for battery level "+batteryLevel);

    }
  }
}

To run this using the puncher, make sure you load your Siddhi rule as a visible resource:

$ punchplatform-puncher.sh -p ./battery.punch

It is even easier to directly cut and past this code in the Sublime Text editor. It is illustrated next.

../../../../_images/sublimeEditorSiddhi.png

On a real punchplatform you will deploy your punchlet as part of a Storm topology. Checkout the Storm & Punchlets chapter for instructions.

Performance Consideration

Running Siddhi rules from punchlets is extremely easy and powerful. However you pay some performance penalty because the (input and output) fields must be transformed back and forth from/into punch tuples. You do not have that extra processing using the CEP bolt.

You benefit however from a lot of flexibility to select what you emit downstream. You can use the punch language constructs to take care of additional enrichment or normalisation of your Siddhi rule output.