Skip to content

Roadmap

Abstract

This chapter provides the high level roadmap together with an indicative agenda. Make sure to contact the punch team for details about any item, or if you would like to see a missing item appear.

Conventions

The various acronyms used to characterize punch releases are the following:

  • Alpha : is a non-production-ready early release. It introduces some new features, but other new features will be added later on. APIs and interface may not be stable. Alpha release are provided for pre-production, evaluation or innovation work.
  • Beta : is a release with all new features in. No new feature will be added. It is still under validation. APIs and interface are stable.
  • Release : is a production-ready release.
  • EOL means End of Life : No one should start a new project with an EOL release. It simply means newer releases are production ready and available. EOL releases are not improved with new features.
  • EOS means End of Support : too old releases are not supported anymore by the punch level 3 support team. Anticipate your migrations, the punch team is there to help. Some releases benefit from a longer term support. These are the:
  • LTS means Long Term Support. A LTS release is one that benefits from longer term support. LTS releases are interesting for projects requiring support going beyond the usual 2 years life-time.

Important

Each major release family, for example 6.x, will have at least one LTS release. For example the 6.3 release is LTS. 6.4 in contrast is not. What it means is simply that if you need a punch release for long term projects, without planning any new features or improvments, you can/should decide to go with the 6.3.

If you are ready to regularly update your release, you should instead decide to start with the 6.x release, meaning you can start from the 6.3 but plan to upgrade to 6.4 6.5 etc.. as time goes by.

Both strategies have pros and cons but the rule of thumb is: if you update regularly, you significantly make it easier to ultimatly switch to the next major release.

LTS releases benefit from bug fixes and critical security patches only, never from new features.

Here is what it looks like in our small ascii quick view: each '|' indicates the time at which an (alpha|beta|release) is released on our website.

Release    Alpha     Beta    Release    end-of-life     end-of-support

5.5        alpha|    beta|     rel|        eol|             eos|                       

Current and Planned Releases

Here is the current 6.x release planning.

        2020  |2021  |      |      |      |2022  |      |      |      |2023  |      |      |      |2024
            Q4|    Q1|    Q2|    Q3|    Q4|    Q1|    Q2|    Q3|    Q4|    Q1|    Q2|    Q3|    Q4|    Q1|
           Dec|   Mar|   Jun|   Sep|   Dec|   Mar|   Jun|   Sep|   Dec|   Mar|   Jun|   Sep|   Dec|   Mar|
              |      |      |      |      |      |      |      |      |      |      |      |
6.0 (eol)      
6.1 (eol)
6.2        rel|      |      |      |   eol|      |      |      |      |      |      |      |      |      |
6.3 LTS       |   rel|      |      |      |      |      |      |      |      |      |      |      |   eol|
6.4           |  beta|      |      |   rel|      |      |      |      |      |      |      |      |      |
7.0           |      | alpha|  beta|   rel|      |      |      |      |      |      |      |      |      |

Info

The end-of-support (eos) is not indicated. Support is of course guaranteed until eol but may be extended for LTS releases. Contact the punch team for details.

As a rule of thumb to go production for long term you should either choose to go with a LTS, or stick with the latest 6.X and be prepared to update (typically) once a year.

Dave 6.x

The dave 6.x release is built on top of elasticsearch 7.x. It runs on top of jdk 1.8. Support for java11 is planned for 6.4 and higher.

The 6.0 and 6.1 alpha releases introduced significant new features in particular a unified model for expressing all types of punchlines (moving away from the old style spouts and bolts storm-centric model), together with support for S3/minio and clickhouse, an improved internal design of the Shiva application scheduler and more fine grain features.

The 6.2 alpha release fixed important issues and introduced the final architecture to support python (spark) applications regarding their packaging, deployment and patching.

The 6.3 is the current stable production release. It has not been elected as the LTS release because of the elasticsearch license change.

We strongly encourage all users to migrate from 6.0, 6.1 and 6.2 to 6.3.

The next 6.4 LTS release will be built on top of the opensearch/opendashboard amazon distribution. That distribution if fully open source (apacheV2) and will provide us with the stable of future proof open source elasticsearch and kibana components.

Ella 7.x

The 7.x releases are built on top of Java 11, Spark3.0, elasticsearch 8.x. A beta 7.0.0 release is available.

Warning

The Ella 7 release will require java11 or higher.

The 7.x release will significantly leverage kubernetes, although a kubernetes-free release is still planned to accommodate tiny deployments and/or existing customers. Kubernetes support is achieved jointly with the Kast Thales companion team and asset.

The 7.x release also improve public punch APIs; and add support for all types of nodes (storm, punch, flink, spark or pyspark) in the punchline graphical editor. This editor is planned to switch from beta to alpha status.

Planned Features

                2021          |             |             |             |
                            Q1|           Q2|           Q3|           Q4|
                           Mar|          Jun|          Sep|          Dec|

PublicApis.                rel|             |             |             |  
PunchlineEditor           beta|        alpha|             |             |
ContainerImages            rel|             |             |             |
Spark3                   alpha|         beta|             |             |
KibanaFreeUIs            alpha|         beta|             |             |
Reporting                     |        alpha|         beta|          rel|
IotAlarming                   |        alpha|         beta|          rel|
IotDataProtectionAccess       |        alpha|         beta|          rel|
SupersetSupport           beta|             |             |             |
PlatformPlugins            rel|             |             |             |
PunchKubernetes               |        alpha|         beta|          rel|
MLMonitoring             alpha|         beta|          rel|             |

Some of these features are implemented jointly with several thales teams. Each is mentioned below together with the short feature description. These teams are the following:

  • Kast : the Thales SIX Kube Analytics STack team provides us with a native kubernetes stack that integrates most of the punch COTS. We are on our way to provide a completely kube-native punch in the next two-years period. Many of the punch services (punchlines, feedback UI, data protected and API gateways) will be available earlier on Kast as containerized apps.
  • IoTalk : the Thales TSN IoTalk team provides its customers with a big-data Iot as-a-service platform. It targets industrial monitoring applications and leverage already some punch features. Starting 2021 we start co-designing the reporting, IOT-alarming and IOT gateway APIs. These features are to a great extent generic and will benefit to most Punch customers.
  • AI Thales Labs : we work jointly with two TSN AI teams on various AI topics in particular making it robust, secured and monitored to deploy ml apps to the punch. Benefiting from MLFlow, from advanced punchline ML monitoring integrations, and the overall ability to design AI pipelines using punchlines are our key work items.

PublicApis

The punch python and java node API were significantly improved and available starting 6.3. We will expose these to public external (maven and python-pip) repositories to make it easier to bring in custom nodes on top of the punch.

PunchlineEditor

The punchline editor is a long beta/alpha development story. We are happy to announce its official release planning. This editor will make it easy to graphically develop a spark, pyspark, storm and punch punchlines. All these arte supported through a simple yaml format. The editor will be fully integrated into the existing punch Kibana plugins.

ContainerImages

We already package some of our apps as container images for those customers equipped with a container runtime. We plan to deliver official such images for the following services:

  • Punch API Gateway
  • Punch Single-Process Punchlines : storm/spark/pyspark/punch
  • Punch UIs

Spark3

Spark3 will be available starting at Punch 7.0. Spark2.4 will be supported for the entire 6.x releases lifetime and only deprecated in 7.x.

KibanaFreeUIs

As time goes by the punch becomes less and less an elastic-centric platform. The advent of Minio and Clickhouse are typical examples. We envision many of our customers will require Superset, a Kibana equivalent with a far greater support for various types of data backends. Similarly, our UIs were designed in a way to not depend too much on the Kibana framework.

We will start working on providing our UIs in Kibana-free frameworks to anticipate this overall move. Note that this does not mean in any way that we move away from Elasticsearch and Kibana. These remains punch essential components. In particular the punch feedback UIs will always be provided as a highly integrated Kibana visualisation.

Reporting

Reporting capabilities will be provided to generate custom reports. We target indicators and consolidated KPIs reports.

This is joint work with the IoTalk team.

IotAlarming

Today Elastalert and real-time punchlines together provide ways to implement all sorts of powerful alarming scenario on top of the punch. We will improve these services to offer a higher-level and fully integrated user experience for Iot users, interested in particular in time-series use cases.

This is joint work with the IoTalk team.

IotDataProtectionAccess

The Punch API Gateway provides a fully transparent elasticsearch API gateway equipped with data protections features. We will enrich this service with additional backend support.

This is joint work with the IoTalk team.

SupersetSupport

Superset is envisioned as a great kibana-like solution in particular to serve clickhouse and minio/S3 data.

This is joint work with the Kast team.

PlatformPlugins

In addition to deliver processing (i.e. business) nodes and deploy these inside punchlines, our customers require providing additional platform services to monitor their components, and to load in our API gateway their own plugins for supporting their backend. The so-called platform plugins will allow them to deliver java or python plugins to do just that, in a way to customise the punch to their specific usage yet benefit from integrated monitoring and additional API endpoints.

Ai2Production

These topics encompass several important features required to provide data scientist with state-of-the-art features even on on-prem secured platforms:

  • Support for Mlflow in both modes: rest service or embedded in a spark punchline
  • punchlines specific ML monitoring to track model qualities and deviations
  • (S3/Elasticsearch/filesystem) extended resource manager to provide users with all the required configuration management features necessary to easily design complete ML applications.

This is joint work with both the TSN AI teams, and the Kast team.

PunchKubernetes

The punch teams is now part of a joint effort to provide a Kubernetes native stack ready to equip on-prem or on-cloud deployments with security, integrated monitoring, and support for the many COTS at play in the punch : elasticsearch, minio, spark etc..

We plan to release a first kubernetes-native Punch at the end of 2021. The details of this rich work is under construction.

Information

This is much more than a punch-only story. The punch team now jointly work with the thales Kast team to provide on one-hand a kubernetes underlying stack and on the other hand extension packs (such as the punch) that provide business and higher-level specific features.

This is joint work with the Kast team.