Skip to content

Log Dispatcher Punchline

Refer to the central log management overview

Reference central site platform monitoring events management (image)

Key Design highlights

Transport / interface / HA

The remote collector site platform events are received from remote collector sites in lumberjack protocol, in a load-balanced way, through an inbuilt balancing mechanism of the sender site (here a Punch lumberjack output node acting as emitter in a forwarder punchline of the LTR - see LTR Reference architecture

To achieve High Availability, at least two "receiver/dispatcher" punchline are started. Each has a fixed IP address, by running on a fixed node. The way to ensure a fixed location for a punchline is obtained through use of a tagged Shiva runner node (using the tag equals to the hostname). In legacy configurations, a one-node storm 'cluster' could be used instead on each of a servers cluster (then a copy of the punchline is run inside each single-noded 'storm cluster')

With this multiple receivers, we also have scalability of input, because the lumberjack sender (lumberjack output node) is load-balancing over all available lmuberjack receivers, that are therefore combining their throughput capacity (as long as we scale also the target Kafka cluster as needed).

Multiple target indices

The different kind of platform events are written to separate target indices, depending on the nature of the data, and the desired retention duration (for example, operator start/storm commands must be kept for more than 1 year, to ensure that automated channels monitoring services are taking into account last known operator action on ALL applications of the platform, even those that are rarely stopped.)

The standard indices naming for dispatching events is indicated in Tenants configuration indices table

The switching rule is implemented in this punchlet :

tenants/platform/resources/punch/standard/Platform/monitoring_dispatcher.punch

Reference configuration example

Remote collecter monitoring events receiver/dispatcher channel structure example

As explained above, the receiving application must be run in multiple instances, located on each of the servers bearing the input addresses of the lumberjack flow. When leveraging shiva placement tags, this leads to this kind of channel_structure pattern:

tenants/platform/channels/ltr_monitoring/channel_structure.hjson

Remote collecter monitoring events receiver/dispatcher punchline example

The associated dispatcher punchline for ltr/collection-site forwarded platform events usually looks like this:

tenants/platform/channels/ltr_monitoring/ltr_events_handler.hjson

Central site monitoring events dispatcher punchline example

tenants/platform/channels/monitoring/local_events_dispatcher.yaml