Skip to content

Roadmap

Abstract

This chapter provides the high level roadmap together with an indicative agenda. Make sure to contact the punch team for details about any item, or if you would like to see a missing item appear.

Conventions

The various acronyms used to characterize punch releases are the following:

  • Alpha : is a non-production-ready early release. It introduces some new features, but other new features will be added later on. APIs and interface may not be stable. Alpha release are provided for pre-production, evaluation or innovation work.
  • Beta : is a release with all new features in. No new feature will be added. It is still under validation. APIs and interface are stable.
  • Release : is a production-ready release.
  • EOL means End of Life : No one should start a new project with an EOL release. It simply means newer releases are production ready and available. EOL releases are not improved with new features.
  • EOS means End of Support : too old releases are not supported anymore by the punch level 3 support team. Anticipate your migrations, the punch team is there to help. Some releases benefit from a longer term support. These are the:
  • LTS means Long Term Support. A LTS release is one that benefits from longer term support. LTS releases are interesting for projects requiring support going beyond the usual 2 years life-time.

Important

Each major release family, for example 6.x, will have at least one LTS release. For example the 6.3 release is LTS. 6.4 in contrast is not. What it means is simply that if you need a punch release for long term projects, without planning any new features or improvments, you can/should decide to go with the 6.3.

If you are ready to regularly update your release, you should instead decide to start with the 6.x release, meaning you can start from the 6.3 but plan to upgrade to 6.4 6.5 etc.. as time goes by.

Both strategies have pros and cons but the rule of thumb is: if you update regularly, you significantly make it easier to ultimatly switch to the next major release.

LTS releases benefit from bug fixes and critical security patches only, never from new features.

Here is what it looks like in our small ascii quick view: each '|' indicates the time at which an (alpha|beta|release) is released on our website.

Release    Alpha     Beta    Release    end-of-life     end-of-support

5.5        alpha|    beta|     rel|        eol|             eos|                       

Current and Planned Releases

Here is the current 6.x release planning.

        2020  |2021  |      |      |      |2022  |      |      |      |2023  |      |      |      |2024
            Q4|    Q1|    Q2|    Q3|    Q4|    Q1|    Q2|    Q3|    Q4|    Q1|    Q2|    Q3|    Q4|    Q1|
           Dec|   Mar|   Jun|   Sep|   Dec|   Mar|   Jun|   Sep|   Dec|   Mar|   Jun|   Sep|   Dec|   Mar|
              |      |      |      |      |      |      |      |      |      |      |      |
6.0 (eol)      
6.1 (eol)
6.2        rel|      |      |      |   eol|      |      |      |      |      |      |      |      |      |
6.3 LTS       |   rel|      |      |      |      |      |      |      |      |      |      |      |   eol|
6.4           |  beta|   rel|      |      |      |      |      |      |      |      |      |      |      |
7.0           |      | alpha|  beta|   rel|      |      |      |      |      |      |      |      |      |

Info

The end-of-support (eos) is not indicated. Support is of course gauranteed until eol but may be extended for LTS releases. Contact the punch team for details.

End-of-life (eol) is not indicated for non LTS releases. That does not means you do not have support, it simply means may eol may arrive soon after the next non-LTS release is delivered. For example a 6.4 might be replaced by a 6.5 six months later, at which time the 6.4 might (most probably) turn eol.

As a rule of thumb to go production for long term you should either choose to go with a LTS, or stick with the latest 6.X and be prepared to update (typically) once a year.

Dave 6.x

The dave 6.x release is built on top of elasticsearch 7.x. It runs on top of jdk 1.8. Support for java11 is planned for 6.4 and higher.

The 6.0 and 6.1 alpha releases introduced significant new features in particular a unified model for expressing all types of punchlines (moving away from the old style spouts and bolts storm-centric model), together with support for S3/minio and clickhouse, an improved internal design of the Shiva application scheduler and more fine grain features.

The 6.2 alpha release fixed important issues and introduced the final architecture to support python (spark) applications regarding their packaging, deployment and patching.

The 6.3 release has been elected as the LTS release. We strongly encourage all users to migrate from 6.0, 6.1 and 6.2 to 6.3.

The next 6.4 minor release will mainly introduce support for java11. Java11 brings in significant performance gains and will benefit cpu and ram intensive punch applications. For these reasons we decided to make it available as part of the 6.x major releases. This will be done in a backward compatible way. Customers will not have to migrate from java 1.8 to 11. They simply will have that new option.

Together with this internal improvement, the 6.4 release will introduce public punch APIs; and support for all types of nodes (storm, punch, spark or pyspark) in the punchline graphical editor. This editor is planned to switch from beta to alpha status.

Duke 7.x

The 7.x releases are built on top of Java 11, Spark3.0, elasticsearch 8.x. A beta 7.0.0 release is planned to be released Q1.

Warning

The duke 7 release will require java11 or higher.

The 7.x release will significantly leverage kubernetes, although a kubernetes-free release is still planned to accommodate tiny deployments and/or existing customers. Kubernetes support is achieved jointly with the Kast Thales companion team and asset.

Planned Features

                2021          |             |             |             |
                            Q1|           Q2|           Q3|           Q4|
                           Mar|          Jun|          Sep|          Dec|

PublicApis.                rel|             |             |             |  
PunchlineEditor           beta|        alpha|             |             |
ContainerImages            rel|             |             |             |
Spark3                   alpha|         beta|             |             |
KibanaFreeUIs            alpha|         beta|             |             |
Reporting                     |        alpha|         beta|          rel|
IotAlarming                   |        alpha|         beta|          rel|
IotDataProtectionAccess       |        alpha|         beta|          rel|
SupersetSupport           beta|             |             |             |
PlatformPlugins            rel|             |             |             |
PunchKubernetes               |        alpha|         beta|          rel|
MLMonitoring             alpha|         beta|          rel|             |

Some of these features are implemented jointly with several thales teams. Each is mentioned below together with the short feature description. These teams are the following:

  • Kast : the Thales SIX Kube Analytics STack team provides us with a native kubernetes stack that integrates most of the punch COTS. We are on our way to provide a completely kube-native punch in the next two-years period. Many of the punch services (punchlines, feedback UI, data protected and API gateways) will be available earlier on Kast as containerized apps.
  • IoTalk : the Thales TSN IoTalk team provides its customers with a big-data Iot as-a-service platform. It targets industrial monitoring applications and leverage already some punch features. Starting 2021 we start co-designing the reporting, IOT-alarming and IOT gateway APIs. These features are to a great extent generic and will benefit to most Punch customers.
  • AI Thales Labs : we work jointly with two TSN AI teams on various AI topics in particular making it robust, secured and monitored to deploy ml apps to the punch. Benefiting from MLFlow, from advanced punchline ML monitoring integrations, and the overall ability to design AI pipelines using punchlines are our key work items.

PublicApis

The punch python and java node API were significantly improved and available starting 6.3. We will expose these to public external (maven and python-pip) repositories to make it easier to bring in custom nodes on top of the punch.

PunchlineEditor

The punchline editor is a long beta/alpha development story. We are happy to announce its official release planning. This editor will make it easy to graphically develop a spark, pyspark, storm and punch punchlines. All these arte supported through a simple yaml format. The editor will be fully integrated into the existing punch Kibana plugins.

ContainerImages

We already package some of our apps as container images for those customers equipped with a container runtime. We plan to deliver official such images for the following services:

  • Punch API Gateway
  • Punch Single-Process Punchlines : storm/spark/pyspark/punch
  • Punch UIs

Spark3

Spark3 will be available starting at Punch 7.0. Spark2.4 will be supported for the entire 6.x releases lifetime and only deprecated in 7.x.

KibanaFreeUIs

As time goes by the punch becomes less and less an elastic-centric platform. The advent of Minio and Clickhouse are typical examples. We envision many of our customers will require Superset, a Kibana equivalent with a far greater support for various types of data backends. Similarly, our UIs were designed in a way to not depend too much on the Kibana framework.

We will start working on providing our UIs in Kibana-free frameworks to anticipate this overall move. Note that this does not mean in any way that we move away from Elasticsearch and Kibana. These remains punch essential components. In particular the punch feedback UIs will always be provided as a highly integrated Kibana visualisation.

Reporting

Reporting capabilities will be provided to generate custom reports. We target indicators and consolidated KPIs reports.

This is joint work with the IoTalk team.

IotAlarming

Today Elastalert and real-time punchlines together provide ways to implement all sorts of powerful alarming scenario on top of the punch. We will improve these services to offer a higher-level and fully integrated user experience for Iot users, interested in particular in time-series use cases.

This is joint work with the IoTalk team.

IotDataProtectionAccess

The Punch API Gateway provides a fully transparent elasticsearch API gateway equipped with data protections features. We will enrich this service with additional backend support.

This is joint work with the IoTalk team.

SupersetSupport

Superset is envisioned as a great kibana-like solution in particular to serve clickhouse and minio/S3 data.

This is joint work with the Kast team.

PlatformPlugins

In addition to deliver processing (i.e. business) nodes and deploy these inside punchlines, our customers require providing additional platform services to monitor their components, and to load in our API gateway their own plugins for supporting their backend. The so-called platform plugins will allow them to deliver java or python plugins to do just that, in a way to customise the punch to their specific usage yet benefit from integrated monitoring and additional API endpoints.

Ai2Production

These topics encompass several important features required to provide data scientist with state-of-the-art features even on on-prem secured platforms:

  • Support for Mlflow in both modes: rest service or embedded in a spark punchline
  • punchlines specific ML monitoring to track model qualities and deviations
  • (S3/Elasticsearch/filesystem) extended resource manager to provide users with all the required configuration management features necessary to easily design complete ML applications.

This is joint work with both the TSN AI teams, and the Kast team.

PunchKubernetes

The punch teams is now part of a joint effort to provide a Kubernetes native stack ready to equip on-prem or on-cloud deployments with security, integrated monitoring, and support for the many COTS at play in the punch : elasticsearch, minio, spark etc..

We plan to release a first kubernetes-native Punch at the end of 2021. The details of this rich work is under construction.

Information

This is much more than a punch-only story. The punch team now jointly work with the thales Kast team to provide on one-hand a kubernetes underlying stack and on the other hand extension packs (such as the punch) that provide business and higher-level specific features.

This is joint work with the Kast team.

Past Releases and Features

Past Releases

Punch releases 1.x and 2.x (from Art to Brad) are available on demand. The Craig 5.x releases is as follows:

        2020  |      |      |      |2021  |      |      |      |2022  |      |      |      |
            Q1|    Q2|    Q3|    Q4|    Q1|    Q2|    Q3|    Q4|    Q1|    Q2|    Q3|    Q4|
           Mar|   Jun|   Sep|   Dec|   Mar|   Jun|   Sep|   Dec|   Mar|   Jun|   Sep|   Dec|
              |      |      |      |      |      |      |      |      |      |      |      |
5.5        eol|      |      |      |   eos|      |      |      |      |      |      |      |
5.6        eol|      |      |      |   eos|      |      |      |      |      |      |      |
5.7           |      |   eol|      |      |      |      |   eos|      |      |      |      |

Past Features

For the records here are the past features list. This may be useful to you if you plan to move from older punch releases.

                2020          |             |             |             |
                            Q1|           Q2|           Q3|           Q4|
                           Mar|          Jun|          Sep|          Dec|


PunchlinesOnly            beta|          rel|
PunchResolver             beta|          rel|
ContinerdImages          alpha|         beta|          rel|
Minio                     beta|          rel|
ClickHouse                beta|             |          rel|
ShivaImprovments          beta|          rel|
RestApiGateway           alpha|         beta|          rel|
RestApiServer            alpha|         beta|          rel|
ParserCatalog            alpha|         beta|          rel|
Nifi                     alpha|         beta|          rel|
MlLifefCycle             alpha|         beta|          rel|
FeedBackGui              alpha|         beta|          rel|
CloudServices                          alpha|         beta|          rel|
DataProtectionAccess                   alpha|         beta|          rel|      
Spark3                                 alpha|         beta|          rel|      
Dask                                                  beta|          rel|      
CorrelationRuleEngine                   beta|          rel|      
CorrelationRuleIhm                     alpha|         beta|          rel|      

The following planned features were not pursued or were delayed:

  • Dask : dask remains a very interesting technology, but we decided to not proceed as python runtime are exploding right now. We envision to support python pipelines with other frameworks (pyspark and flink) that appear more in line with our use cases.
  • Spark3 : was delayed simply because of the lack of time and the requirements for java11. We are working on it.
  • CloudServices : we worked on many essential improvements that are required to ultimately reach that high-level objective. Yet the advent of Kubernetes and Kast made us delay the final design of this family of features.

PunchlinesOnly

Starting at 6.x all jobs (storm punch spark python) are expressed as punchlines. This has important benefits in particular on the HMI and front side as all punch applications will be editable from the punchline editor.

Important

The punchline schema is, in fact, the one of 5.x PML files. We decided to leverage that schema for all punch application. This said, old style format will be fully supported for backward compatibility. Refer to the migration guide.

PunchResolver

The punch resolver isolates user from low level security or access point configuration. It is required to deploy application on secured multi-tenant platforms to allow functional users to be relieved and protected from low level configuration items.

It is a significant improvement to the existing templating based punch approach.

ContainerdImages

The Punch is highly modular is often deployed on top of existing managed services. Starting at version 6.1, Punch API Gateway, Kibana plugins are provided as container images. You can deploy these directly into your docker rancher or k8 infrastructure.

Minio

The S3 minio store has been added in the 6.0 punch.

Clickhouse

Clickhouse is being evaluated. Punch already provides the required IO connectors (output nodes).

ShivaImprovements

Shiva is now running production and has turned out a key service to run big data, log management or machine learning use case on on-premise infrastructure. We decided to improve Shiva with a few simple but important capabilities:

  • resource aware scheduling
  • smooth upgrade service restarts
  • one shot application execution

RestApiGateway

For several reasons an API gateway is required.

  • Security : in order to isolate Kibana and external customers to directly access database or sensitive services such as Zookeeper, Elasticsearch, Clickhouse.
  • Services : in order to provide all Punch services like starting, stopping a channel, submitting a punchline ML application for execution.
  • Protection : in order to analyse the user submitted requests and take actions to protect user(s) from dangerous actions.
  • LifeCycle : enrichment files, parsers and machine learning models are examples of important resources deployed by users in their punchlines. Managing these together with their versioning and lifecycle requires a REST server, and a well-designed REST API and schema.

RestApiServer

A particularly important REST server use case is to expose resources such as parsers, models and enrichment files. This is for example used to automatically upload new enrichment files, automatically reloaded in running punchlines without service interruption.

The resource-server is the backend that provide these service. The Punch rest server leverages elasticsearch, S3 and a posix file system to store these resources.

Note

This item is in fact more than a server, it includes: 1. a REST API 2. REST API libraries 3. an actual server

ParserCatalog

The punch provides a number of ready to use parsers. The new ParserCatalog will expose these through an improved GUI in order for cybersecurity user to view, play, or deploy the parser into production.

Nifi

Starting at 6.x punch provides a nifi processor to run parsers in a Nifi runtime. This processor is improved in several ways to provide the same rich set of features than in the punch or storm engines.

Ml Life Cycle

The Mlflow component is integrated and provides dedicated and additional facilities to manage machine learning models. Note that the models themselves are stored in the Punch resource catalog.

FeedBack Gui

The so-called feedback Gui will provide users with an ergonomic way to interact with data produced by machine learning applications. It will let them label the data, and identify a false-positive, in a way for the overall application to recalibrate continuously the models.

Cloud Services

Punch deployed on Cloud will benefit from docker image punch services (already cited in a separate feature). What is required is also some easier ready-to-use service for :

  • creating a tenant
  • deploying automatically the Kibana plugins and punch Guis
  • Name and expose the customer data, as well as some of its internal monitoring data
  • automatically wire the customers' security RBAC attributes.

Theses are referred to as Cloud services.

Data Protection Access

One of the Punch REST Gateway service consists in exposing inner elasticsearch data directly through a REST API. This requires however safe strategies to protect users from submitting dangerous queries to the backend (elasticsearch or clickhouse for instance). This item consists in using machine learning strategies to improve the classification or user requests, in turn allowing the Punch gateway to accept or refuse the incoming requests.

CorrelationRuleEngine

The punch already includes the Elastalert engine. This integration has been fully revamped to make elastalert a punch service like any other, executed in shiva and soon as easily in kubernetes.

We plan to leverage the elastalert yaml rule format to provide additional capabilities such as machine learning alerting.

CorrelationRuleIhm

A rule engine IHM is planned to allow end-users to edit, save and load their rule from the IHM. This IHM will be fully integrated with the RBAC protections.

Information

This topics is now owned by a dedicated Thales solution built on top of the Punch. In turn resulting in a new thales SIEM product. Contact us for details.