Skip to content

Topologies

The next concept to understand are topologies. A topology is a data pipeline, configured to fetch or receive data, process it and push it downstream. Why not just run one ?

1
cd $PUNCHPLATFORM_CONF_DIR/examples/topologies/files/csv_to_stdout_and_elasticsearch

Have a look at the csv_to_stdout_and_elasticsearch.json file. It is a topology file. It reads data from the local file AAPL.csv, then calls a small processing function AAPL.punch (a punchlet) to convert CSV into JSON, prints that JSON to standard output and inserts it into an elasticsearch cluster.

Go for it !

1
punchplatform-topology.sh csv_to_stdout_and_elasticsearch.json

You will get:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# [...]
{
  "logs": {
    "log": {
      "High": 172.0,
      "Low": 170.059998,
      "Volume": 171.509995,
      "Adj Close": 37687500,
      "Close": 171.509995,
      "Date": "2018-01-26",
      "Open": 172.0
      }
    }
}
# Ctrl+C to quit

The data displayed on your terminal comes from the punchlet. Notice the print function on the code below. The punchlet in action simply transforms a CSV input string into a JSON document which in turn will be indexed to elasticsearch.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
{
// Take care of the formatting line, not containing actual data
if ([logs][log].contains("Date")) {
  root.empty();
  return;
}

// Use the csv operator to do the job
if (!csv("Date","Open","High","Low","Close","Volume","Adj Close")
     .delim(",")
     .inferTypes()
     .on([logs][log])
     .into([logs][log])) {
     raise("unexpected format");
}

// Print the result. Do not do this in production!
print(root);
}

At this point, you should have access to your data within Kibana. We will take this opportunity to get familiar with what Kibana has to offer, i.e visualising our data.

  1. Open the kibana GUI in your browser http://localhost:5601
  2. Navigate to the Dev Tools tab on the left side.
  3. Execute the first request shown there: GET /_cat/indices?v. You should see one line with a stock-YYYY-MM-DD index, the one that holds our stock price data.
  4. Now, navigate to the "Management" tab. Click on "index patterns" to create a new index pattern. Use the pattern stock-* and click on "next step". Here, select "@timestamp" as time filter field name.
  5. Navigate to the "Discover" tab. On the Discover menu, at the top, select the stock-* index using the drop-down menu and chose a large time scope.
  6. Now, you should see your data, congratulation !

Mastering Elasticsearch and Kibana is ultimately required to take the best out of your punchplatform. If you are a newcomer: you can consider that an index in elasticsearch is equivalent to a table in a traditional SQL database. Each document you see represent a line in that table, and each field a column. What Kibana does is to allow you to explore and visualise your data easily."

As you see, topologies are quite simple to understand. They are (very) powerful. You can do all sort of stream computing with them. Now that you have a good understanding of topologies, let's move on.