Troubleshooting Error on kibana plugin¶
Why do that¶
Even after having deploy the punchplatform, Our kibana plugins could possibly be nonfunctional, in order to make it functional the user have often to follow some additional steps.
Tips
You can look at kibana log whith the : journalctl -u kibana_
Steps¶
Create your tmp path¶
Error :
log [15:54:06.771] [info][plugin:punchplatform] ExecBashService.run() [1578498846740] PUNCHPLATFORM_CONF_DIR=/home/vagrant/pp-conf/tmp_path/conf PUNCHPLATFORM_CONF_URL=localhost:2181/common punchplatform-getconf.sh -pf -d /home/vagrant/pp-conf/tmp_path/conf
log [15:54:06.805] [info][plugin:punchplatform] ExecBashService.run() [1578498846804] PUNCHPLATFORM_CONF_DIR=/home/vagrant/pp-conf/tmp_path/conf PUNCHPLATFORM_CONF_URL=localhost:2181/common punchplatform-getconf.sh -t mytenant -d /home/vagrant/pp-conf/tmp_path/conf
log [15:54:09.235] [error][plugin:punchplatform] /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/ExecBashService.js:24:() [1578498846804] Error: {"code":1,"stdout":"","stderr":""}
log [15:54:09.238] [error][plugin:punchplatform] /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/PunchCommandService.js:76:() failed to get configuration for tenant=mytenant
error [15:54:06.739] Error: ENOENT: no such file or directory, scandir '/home/vagrant/pp-conf/tmp_path/conf'
at Object.readdirSync (fs.js:786:3)
at Object.readDirectory (/data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/helpers/file.js:14:8)
at PunchCommandService.getConfDir (/data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/PunchCommandService.js:82:23)
at /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/routes/routes.js:200:74
at process._tickCallback (internal/process/next_tick.js:68:7)
Solution :
In your kibana punchplatform.properties, you should have a "tmp_path": "/your/path/tmp".
You have to create this directory and a conf directory inside, and put your punchplatform.properties inside.
mkdir -pv /your/path/tmp/conf
cp $PUNCHPLATFORM_CONF_DIR/punchplatform.properties /your/path/tmp/conf
Push your tenant¶
Error :
log [16:02:56.425] [error][plugin:punchplatform] /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/PunchCommandService.js:76:() failed to get configuration for tenant=mytenant
log [16:03:12.995] [info][plugin:punchplatform] ExecBashService.run() [1578499392995] PUNCHPLATFORM_CONF_DIR=/home/vagrant/pp-conf/tmp_path/conf PUNCHPLATFORM_CONF_URL=localhost:2181/common punchplatform-getconf.sh -pf -d /home/vagrant/pp-conf/tmp_path/conf
log [16:03:13.028] [info][plugin:punchplatform] ExecBashService.run() [1578499393027] PUNCHPLATFORM_CONF_DIR=/home/vagrant/pp-conf/tmp_path/conf PUNCHPLATFORM_CONF_URL=localhost:2181/common punchplatform-getconf.sh -t mytenant -d /home/vagrant/pp-conf/tmp_path/conf
log [16:03:13.956] [error][plugin:punchplatform] /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/ExecBashService.js:24:() [1578499392995] Error: {"code":1,"stdout":"","stderr":""}
log [16:03:14.148] [error][plugin:punchplatform] /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/ExecBashService.js:24:() [1578499393027] Error: {"code":1,"stdout":"","stderr":""}
log [16:03:14.154] [error][plugin:punchplatform] /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/PunchCommandService.js:76:() failed to get configuration for tenant=mytenant
Solution :
Kibana plugins need your tenant configuration in zookeeper.
punchplatform-putconf.sh --zookeeper-cluster-name common --platform
Class not found¶
Error :
log [15:22:20.483] [info][plugin:punchplatform] ExecBashService.run() [1579792940482] PUNCHPLATFORM_CONF_DIR=/home/vagrant/tmp/conf PUNCHPLATFORM_CONF_URL=server1:2181,server2:2181,server3:2181/punchplatform-primary punchlinectl -r spark --punchline /home/vagrant/tmp/tenant1-background-1579792940482.pml
log [15:22:21.307] [error][plugin:punchplatform] /data/kibana/tenant1/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/ExecBashService.js:24:() [1579792940482] Error: {"code":1,"stdout":"","stderr":"Exception in thread \"main\" java.lang.NoClassDefFoundError: org/apache/spark/launcher/SparkAppHandle$Listener\n\tat java.lang.ClassLoader.defineClass1(Native Method)\n\tat java.lang.ClassLoader.defineClass(ClassLoader.java:756)\n\tat java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)\n\tat java.net.URLClassLoader.defineClass(URLClassLoader.java:468)\n\tat java.net.URLClassLoader.access$100(URLClassLoader.java:74)\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:369)\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:363)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:362)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:418)\n\tat sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:351)\n\tat org.thales.punch.pl.client.api.PunchLineLauncherBuilder.buildJob(PunchLineLauncherBuilder.java:602)\n\tat org.thales.punch.pl.client.main.PunchLineClient.main(PunchLineClient.java:88)\nCaused by: java.lang.ClassNotFoundException: org.apache.spark.launcher.SparkAppHandle$Listener\n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:382)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:418)\n\tat sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:351)\n\t... 14 more\n"}
log [15:22:21.309] [info][plugin:punchplatform] /data/kibana/tenant1/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/PmlCommandService.js:197:() remove temporary file= /home/vagrant/tmp/tenant1-background-1579792940482.pml
Solution :
Kibana plugins need spark binaries and operator analytics deployment Be sure to have at least a spark section in your punchplatform.properties and deployment.settings even if you don't want a spark cluster.
"spark" : {
"clusters" : {
},
"install_dir": "/data/opt/spark-2.4.3-bin-hadoop2.7"
},
"punchplatform_analytics_deployment_version": "punchplatform-analytics-deployment-5.8.0-SNAPSHOT",
Mapping Error¶
Error :
error [16:26:59.331] [illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [job.runtime.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead. :: {"path":"/tenant1-metrics-*/_doc/_search","query":{"ignore_unavailable":true},"body":"{\"query\":{\"bool\":{\"must\":[{\"term\":{\"type\":{\"value\":\"spark\"}}}],\"should\":[{\"match_phrase\":{\"name\":\"spark.application.end\"}},{\"match_phrase\":{\"name\":\"spark.application.start\"}}]}},\"size\":0,\"aggs\":{\"executions\":{\"terms\":{\"size\":10000,\"field\":\"job.runtime.id\",\"order\":{\"_key\":\"asc\"}},\"aggs\":{\"events\":{\"top_hits\":{\"size\":2,\"sort\":[{\"@timestamp\":{\"order\":\"desc\"}}]}}}}}}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Fielddata is disabled on text fields by default. Set fielddata=true on [job.runtime.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"tenant1-metrics-2020.01.23\",\"node\":\"-hX6vZYDSGKZqSpSLgADRA\",\"reason\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Fielddata is disabled on text fields by default. Set fielddata=true on [job.runtime.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\"}}],\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Fielddata is disabled on text fields by default. Set fielddata=true on [job.runtime.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Fielddata is disabled on text fields by default. Set fielddata=true on [job.runtime.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\"}}},\"status\":400}"}
at respond (/data/kibana/tenant1/kibana-6.8.6-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:308:15)
at checkRespForFailure (/data/kibana/tenant1/kibana-6.8.6-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:267:7)
at HttpConnector.<anonymous> (/data/kibana/tenant1/kibana-6.8.6-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)
at IncomingMessage.wrapper (/data/kibana/tenant1/kibana-6.8.6-linux-x86_64/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
at IncomingMessage.emit (events.js:194:15)
at endReadableNT (_stream_readable.js:1103:12)
at process._tickCallback (internal/process/next_tick.js:63:19)
Solution :
The field job.runtime.id must be a keyword field for elasticsearch
curl -X PUT "localhost:9200/<yourtenant>-job-metrics-*/_mapping?pretty" -H 'Content-Type: application/json' -d'
{
"properties": {
"job.runtime.id": {
"type": "keyword"
}
}
}
'
curl -X DELETE localhost:9200/<yourtenant>-job-metrics-*