org.thales.punch.pl.plugins.udf.api

Interface UdfRegistrationFactory

public interface UdfRegistrationFactory

Extend your SQL node class with this one to register our internal spark UDFs to the running SparkSession It also enable you to use custom UDFs function as long as the required jar is present inside the classpath. Below is an example of registering your custom udf.

        myOwnFunc is the function name you wish to give to your UDF function
 

        org.thales.punch.pl.plugins.sql.udf.KmeansSqdist is the Class Name where your UDF function resides...
 

        Double is the return type of your function written in Spark DDL format: for instance an Object of type  Array<Double> will be written as  ARRAY<DOUBLE>
 

 
  
     {
        type: sql
        component: sql
         settings: {
         	register_udf: [
              {
                function_name: myOwnFunc
                class_name: org.thales.punch.pl.plugins.sql.udf.KmeansSqdist
                schema_ddl: Double
              }
         	]
          statement_list: [
            {
              output_table_name: extract_analytics_json
              statement: SELECT json_tuple(output,'analytics') AS output_analytics, init_host_ip, session_out_byte, init_host_ip_tokenized, init_host_ip_hashed, full_vector, features AS source_features, prediction AS source_prediction FROM executor_data
            }
            {
              output_table_name: features_prediction_as_string
              statement: SELECT json_tuple(output_analytics, 'features', 'prediction'), init_host_ip, init_host_ip_tokenized, session_out_byte, init_host_ip_hashed, source_features, source_prediction FROM extract_analytics_json
            }
            {
              output_table_name: cast_columns_to_proper_types
              statement: SELECT punch_str_to_array_double(c0) AS output_features, CAST(c1 AS INTEGER) AS output_prediction, init_host_ip, init_host_ip_tokenized, session_out_byte, init_host_ip_hashed, source_features, source_prediction FROM features_prediction_as_string
            }
            {
              output_table_name: calculate_kmeans_sqdist
              statement: SELECT myOwnFunc(output_features, output_prediction, 'model.bin', 4) AS kmeans_sqdist, init_host_ip, session_out_byte FROM cast_columns_to_proper_types                  
            }
          ]
         }
         subscribe: [
             {
                 component: executor
                 stream: data
             }
         ]
         publish: [
           {
               stream: extract_analytics_json
           }
           {
               stream: features_prediction_as_string
           }
           {
               stream: cast_columns_to_proper_types
           }
           { 
               stream: calculate_kmeans_sqdist
           }
         ]
     }
 
 
 

Author:

jonathan

Method Summary

Modifier and Type	Method and Description
default Map<String,org.apache.spark.sql.types.DataType>	internalUdf()
default void	registerInternalUdfs()
default void	registerUserFunctions(List<UdfRegistrationPojo> punchUserUdfs) A Node class extending UdfRegistrationFactory class should call this method inside it's execute method

Method Detail

internalUdf

default Map<String,org.apache.spark.sql.types.DataType> internalUdf()

registerUserFunctions

default void registerUserFunctions(List<UdfRegistrationPojo> punchUserUdfs)

A Node class extending UdfRegistrationFactory class should call this method inside it's execute method

registerInternalUdfs

default void registerInternalUdfs()