Skip to content

Keras Update Stream

Overview

Compatible Pyspark only

Python node : launch with punchplatform-pyspark.sh or the punch UI job-editor with environment pyspark.

This is a demo node that take a trained model as input, continue to train it with some provided data, and submit the final model to the output stream. This aim to demonstrate the capability to re-train model within our pipelines.

This node have no configuration, and only work with the other keras nodes provided with the pml-pyspark.

Example configuration :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
{
            type: keras_update_train
            component: updater
            subscribe: [
                {
                    component: data_input
                    stream: x_train
                }
                {
                    component: data_input
                    stream: y_train
                }
                {
                    component: data_input
                    stream: x_test
                }
                {
                    component: data_input
                    stream: y_test
                }
                {
                    component: model_load
                    stream: model
                }
            ]
            publish: [
                {
                    stream: model
                }
            ]
        }

Example of complete pipeline :

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
{
  job: [
        {
            type: keras_data_input
            component: data_input
            publish: [
                {
                    stream: x_train
                }
                {
                    stream: y_train
                }
                {
                    stream: x_test
                }
                {
                    stream: y_test
                }
            ]
        }

        {
            type: keras_load_model
            component: model_load
            settings:{
                path: my_model.h5
            },
            publish: [
                {
                    stream: model
                }
            ]
        }

        {
            type: keras_update_train
            component: updater
            subscribe: [
                {
                    component: data_input
                    stream: x_train
                }
                {
                    component: data_input
                    stream: y_train
                }
                {
                    component: data_input
                    stream: x_test
                }
                {
                    component: data_input
                    stream: y_test
                }
                {
                    component: model_load
                    stream: model
                }
            ]
            publish: [
                {
                    stream: model
                }
            ]
        }

        {
            type: keras_save_model
            component: model_save
            settings:{
                path: my_model.h5
            },
            subscribe:[
                {
                    component: updater
                    stream: model
                }
            ]

        }
  ]

}