Skip to content

TROUBLESHOOTING Error on kibana plugin

Why do that

Even after having deploy the punchplatform, Our kibana plugins could possibly be nonfunctional, in order to make it functional the user have often to follow some additional steps.

Tips

You can look at kibana log whith the : journalctl -u kibana_.service in systemctl, or find it in /var/log/supervisor/kibana-*stdout.log on supervisord

Steps

Create your tmp path

Error :

log   [15:54:06.771] [info][plugin:punchplatform]  ExecBashService.run() [1578498846740] PUNCHPLATFORM_CONF_DIR=/home/vagrant/pp-conf/tmp_path/conf PUNCHPLATFORM_CONF_URL=localhost:2181/common punchplatform-getconf.sh -pf -d /home/vagrant/pp-conf/tmp_path/conf
log   [15:54:06.805] [info][plugin:punchplatform]  ExecBashService.run() [1578498846804] PUNCHPLATFORM_CONF_DIR=/home/vagrant/pp-conf/tmp_path/conf PUNCHPLATFORM_CONF_URL=localhost:2181/common punchplatform-getconf.sh -t mytenant -d /home/vagrant/pp-conf/tmp_path/conf
log   [15:54:09.235] [error][plugin:punchplatform]  /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/ExecBashService.js:24:() [1578498846804] Error: {"code":1,"stdout":"","stderr":""}
log   [15:54:09.238] [error][plugin:punchplatform]  /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/PunchCommandService.js:76:() failed to get configuration for tenant=mytenant
error  [15:54:06.739]  Error: ENOENT: no such file or directory, scandir '/home/vagrant/pp-conf/tmp_path/conf'
    at Object.readdirSync (fs.js:786:3)
    at Object.readDirectory (/data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/helpers/file.js:14:8)
    at PunchCommandService.getConfDir (/data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/PunchCommandService.js:82:23)
    at /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/routes/routes.js:200:74
    at process._tickCallback (internal/process/next_tick.js:68:7)

Solution :

In your kibana punchplatform.properties, you should have a "tmp_path": "/your/path/tmp".

You have to create this directory and a conf directory inside, and put your punchplatform.properties inside.

mkdir -pv /your/path/tmp/conf
cp $PUNCHPLATFORM_CONF_DIR/punchplatform.properties /your/path/tmp/conf

Push your tenant

Error :

log   [16:02:56.425] [error][plugin:punchplatform]  /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/PunchCommandService.js:76:() failed to get configuration for tenant=mytenant
log   [16:03:12.995] [info][plugin:punchplatform]  ExecBashService.run() [1578499392995] PUNCHPLATFORM_CONF_DIR=/home/vagrant/pp-conf/tmp_path/conf PUNCHPLATFORM_CONF_URL=localhost:2181/common punchplatform-getconf.sh -pf -d /home/vagrant/pp-conf/tmp_path/conf
log   [16:03:13.028] [info][plugin:punchplatform]  ExecBashService.run() [1578499393027] PUNCHPLATFORM_CONF_DIR=/home/vagrant/pp-conf/tmp_path/conf PUNCHPLATFORM_CONF_URL=localhost:2181/common punchplatform-getconf.sh -t mytenant -d /home/vagrant/pp-conf/tmp_path/conf
log   [16:03:13.956] [error][plugin:punchplatform]  /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/ExecBashService.js:24:() [1578499392995] Error: {"code":1,"stdout":"","stderr":""}
log   [16:03:14.148] [error][plugin:punchplatform]  /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/ExecBashService.js:24:() [1578499393027] Error: {"code":1,"stdout":"","stderr":""}
log   [16:03:14.154] [error][plugin:punchplatform]  /data/kibana/admin/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/PunchCommandService.js:76:() failed to get configuration for tenant=mytenant

Solution :

Kibana plugins need your tenant configuration in zookeeper.

punchplatform-putconf.sh --zookeeper-cluster-name common --platform

Class not found

Error :

  log   [15:22:20.483] [info][plugin:punchplatform]  ExecBashService.run() [1579792940482] PUNCHPLATFORM_CONF_DIR=/home/vagrant/tmp/conf PUNCHPLATFORM_CONF_URL=server1:2181,server2:2181,server3:2181/punchplatform-primary punchlinectl -r spark --punchline /home/vagrant/tmp/tenant1-background-1579792940482.pml
  log   [15:22:21.307] [error][plugin:punchplatform]  /data/kibana/tenant1/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/ExecBashService.js:24:() [1579792940482] Error: {"code":1,"stdout":"","stderr":"Exception in thread \"main\" java.lang.NoClassDefFoundError: org/apache/spark/launcher/SparkAppHandle$Listener\n\tat java.lang.ClassLoader.defineClass1(Native Method)\n\tat java.lang.ClassLoader.defineClass(ClassLoader.java:756)\n\tat java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)\n\tat java.net.URLClassLoader.defineClass(URLClassLoader.java:468)\n\tat java.net.URLClassLoader.access$100(URLClassLoader.java:74)\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:369)\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:363)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:362)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:418)\n\tat sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:351)\n\tat org.thales.punch.pl.client.api.PunchLineLauncherBuilder.buildJob(PunchLineLauncherBuilder.java:602)\n\tat org.thales.punch.pl.client.main.PunchLineClient.main(PunchLineClient.java:88)\nCaused by: java.lang.ClassNotFoundException: org.apache.spark.launcher.SparkAppHandle$Listener\n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:382)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:418)\n\tat sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:351)\n\t... 14 more\n"}
  log   [15:22:21.309] [info][plugin:punchplatform]  /data/kibana/tenant1/kibana-6.8.6-linux-x86_64/plugins/punchplatform/server/services/PmlCommandService.js:197:() remove temporary file= /home/vagrant/tmp/tenant1-background-1579792940482.pml

Solution :

Kibana plugins need spark binaries and operator analytics deployment Be sure to have at least a spark section in your punchplatform.properties and deployment.settings even if you don't want a spark cluster.

  "spark" : {
    "clusters" : {
    },
    "install_dir": "/data/opt/spark-2.4.3-bin-hadoop2.7"
},
"spark_version" : "spark-2.4.3-bin-hadoop2.7",
"punchplatform_analytics_deployment_version": "punchplatform-analytics-deployment-5.8.0-SNAPSHOT",

Mapping Error

Error :

error  [16:26:59.331]  [illegal_argument_exception] Fielddata is disabled on text fields by default. Set fielddata=true on [job.runtime.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead. :: {"path":"/tenant1-metrics-*/_doc/_search","query":{"ignore_unavailable":true},"body":"{\"query\":{\"bool\":{\"must\":[{\"term\":{\"type\":{\"value\":\"spark\"}}}],\"should\":[{\"match_phrase\":{\"name\":\"spark.application.end\"}},{\"match_phrase\":{\"name\":\"spark.application.start\"}}]}},\"size\":0,\"aggs\":{\"executions\":{\"terms\":{\"size\":10000,\"field\":\"job.runtime.id\",\"order\":{\"_key\":\"asc\"}},\"aggs\":{\"events\":{\"top_hits\":{\"size\":2,\"sort\":[{\"@timestamp\":{\"order\":\"desc\"}}]}}}}}}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Fielddata is disabled on text fields by default. Set fielddata=true on [job.runtime.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"tenant1-metrics-2020.01.23\",\"node\":\"-hX6vZYDSGKZqSpSLgADRA\",\"reason\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Fielddata is disabled on text fields by default. Set fielddata=true on [job.runtime.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\"}}],\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Fielddata is disabled on text fields by default. Set fielddata=true on [job.runtime.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Fielddata is disabled on text fields by default. Set fielddata=true on [job.runtime.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\"}}},\"status\":400}"}
    at respond (/data/kibana/tenant1/kibana-6.8.6-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:308:15)
    at checkRespForFailure (/data/kibana/tenant1/kibana-6.8.6-linux-x86_64/node_modules/elasticsearch/src/lib/transport.js:267:7)
    at HttpConnector.<anonymous> (/data/kibana/tenant1/kibana-6.8.6-linux-x86_64/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)
    at IncomingMessage.wrapper (/data/kibana/tenant1/kibana-6.8.6-linux-x86_64/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

Solution :

The field job.runtime.id must be a keyword field for elasticsearch

curl -X PUT "localhost:9200/<yourtenant>-job-metrics-*/_mapping?pretty" -H 'Content-Type: application/json' -d'
{
  "properties": {
    "job.runtime.id": {
      "type": "keyword"
    }
  }
}
'

curl -X DELETE localhost:9200/<yourtenant>-job-metrics-*