Skip to content

API Reference

GET /v1/platform/properties

Get punchplatform.properties file.

Response

Status

Status Code Reason
200 Successful operation

Example request

curl --location --request GET 'http://localhost:4242/v1/platform/properties'

GET /v1/platform/tenant

Get tenant used by the gateway.

Response

Body

Field Description
tenant Tenant set in Gateway settings

Status

Status Code Reason
200 Successful operation

Example request

curl --location --request GET 'http://localhost:4242/v1/platform/tenant'

Example response

{
    "tenant": "mytenant"
}

POST /v1/puncher/{operator}

Execute a punchlet which using the given operator

Request

Path variable

Field Description
operator Operator to use ("grok", "dissect" or "punchlet")

Form

Field Description
input File containing log samples
pattern File contaning the pattern

Response

Status

Status Code Reason
200 successful operation
500 Invalid input or pattern

Example request

curl --location --request POST 'http://localhost:4242/v1/puncher/dissect' \
--form 'input=@/my_dissect_folder/input' \
--form 'pattern=@/my_dissect_folder/pattern'

DELETE /v1/{tenant}/archives

Delete all existing archives for a tenant

Request

Path variables

Field Description
tenant Tenant name

Response

Status

Status Code Reason
200 successful operation

Example request

curl --location --request DELETE 'http://localhost:4242/v1/mytenant/archives'

GET /v1/{tenant}/archives/{id}

Download an archive generated by a punchline

Request

Path variables

Field Description
tenant Tenant name
id Archive id to download

Response

HTTP Status

Status Code Reason
200 Successful download
404 Archive not found

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/archives/my_archive'

DELETE /v1/{tenant}/archives/{id}

Delete an archive

Request

Path variables

Field Description
tenant Tenant name
id Archive id to delete

Response

Status

Status Code Reason
200 Successful download
404 Archive not found

Example request

curl --location --request DELETE 'http://localhost:4242/v1/mytenant/archives/my_archive'

GET /v1/{tenant}/channels

Get all tenant channels status

Request

Path variables

Field Description
tenant Tenant name

Response

Body

Field Description
children Array of valid channels
children[].name Name of the channel
children[].status Status of the channel
errors Array of invalid channels
errors[].name Name of the channel
errors[].status Status of the channel

Status

Status Code Reason
200 successful operation

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/channels'

Example response

{
    "children": [
        {
            "name": "logstash",
            "status": "STOPPED"
        },
        {
            "name": "admin",
            "status": "STOPPED"
        },
        {
            "name": "apache_httpd",
            "status": "STOPPED"
        }
    ],
    "errors": []
}

GET /v1/{tenant}/channels/{channel}

Get status of a specific channel

Request

Path variables

Field Description
tenant Tenant name
channel Channel name

Response

Body

Field Description
status Channel status
jobs List of jobs of the channel
jobs[].cluster Job cluster name
jobs[].name Job name
jobs[].runtimeId Job related path
jobs[].status Job status

Status

Status Code Reason
200 successful operation
404 Channel not found

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/channels/admin'

Example response

{
    "status": "ACTIVE",
    "jobs": [
        {
            "cluster": "common",
            "name": "admin/common/elasticsearch-housekeeping",
            "runtimeId": "punchplatform/mytenant/channels/admin/elasticsearch-housekeeping",
            "status": "ACTIVE"
        },
        {
            "cluster": "common",
            "name": "admin/common/archives-housekeeping",
            "runtimeId": "punchplatform/mytenant/channels/admin/archives-housekeeping",
            "status": "ACTIVE"
        },
        {
            "cluster": "common",
            "name": "admin/common/channels-monitoring",
            "runtimeId": "punchplatform/mytenant/channels/admin/channels-monitoring",
            "status": "ACTIVE"        
        }
    ]
}

POST /v1/{tenant}/channels/{channel}/{action}

Request

Path variables

Field Description
tenant Tenant name
channel Channel name
action Action to operate on the channel ("start" or "stop")

Response

Status

Status Code Reason
201 Successful action
404 Channel doesn't exist

Example request

curl --location --request POST 'http://localhost:4242/v1/mytenant/channels/admin/start'

GET /v1/{tenant}/conf

List all tenant related files and directories in configuration folder

Request

Path variables

Field Description
tenant Tenant name

Response

Body

Field Description
PATH.parent Parent folder name
PATH.path Relative path
PATH.size Folder/File size
PATH.isFile true if PATH is a file, else false
PATH.canRead Able to read file/folder content
PATH.children Map of subfile / subfolder content

Status

Status Code Reason
200 successful operation

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/conf'

Example response

{
    "tenants": {
        "parent": "/conf",
        "path": "/tenants",
        "size": 4096,
        "isFile": false,
        "canRead": true,
        "children": {
            "mytenant": {
                "parent": "/conf/tenants",
                "path": "/tenants/mytenant",
                "size": 4096,
                "isFile": false,
                "canRead": true,
                "children": {
                    "test_file": {
                        "parent": "/conf/tenants/mytenant",
                        "path": "/tenants/mytenant/test_file",
                        "size": 162,
                        "isFile": true,
                        "canRead": true,
                        "children": {}
                    }
                }
            }
        }
    }
}

GET /v1/{tenant}/conf/{path}

Get a file from configuration folder

Request

Path variables

Field Description
tenant Tenant name
path Path to file

Response

Status

Status Code Reason
200 successful operation
400 "path" given is a directory
404 File not found

Example request

curl --location --request GET \
'http://localhost:4242/v1/mytenant/conf/tenants/mytenant/test_file'

POST /v1/{tenant}/extraction

Create extraction

Request

Path variables

Field Description
tenant Tenant name

Body

Field Description
description Extraction description
format Extraction format ("csv", "json")
index Elasticsearch index
fields List of fields to extract
fields[].name Field name
fields[].type Field type
filters Elastic filters to apply
tenant Tenant name
### Response
#### Status
Status Code Reason
200 successful operation

Example request

curl --location --request POST 'http://localhost:4242/v1/mytenant/extraction' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic YWRtaW46YWRtaW4=' \
--header 'Content-Type: application/json' \
--data-raw \
'{
     "description": "log-extraction",
     "format": "csv",
     "index": "platform-logs-*",
     "fields": [
         {
             "name": "@timestamp",
             "type": "date"
         },
         {
             "name": "content.message",
             "type": "string"
         }
     ],
     "filters": {},
     "tenant": "mytenant"
 }'

GET /v1/{tenant}/punchline

List punchlines saved.

Request

Path variables

Field Description
tenant Tenant name

Response

Body

Field Description
[].id Punchline ID
[].punchline Punchline file content
[].version Punchline version
[].platformId Punchline platform ID
[].@timestamp Punchline save timestamp

Status

Status Code Reason
200 successful operation

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/punchline'

Example response

[
    {
        "id": "BtDHgnEByFYZVYsBXB30",
        "punchline": {
            "runtime_id": "BtDHgnEByFYZVYsBXB30",
            "metrics": {
                "reporters": [
                    {
                        "http_hosts": [
                            {
                                "port": 9200,
                                "host": "localhost"
                            }
                        ],
                        "type": "elasticsearch",
                        "index_name": "mytenant-metrics"
                    }
                ]
            },
            "job": [
                {
                    "settings": {
                        "input_data": [
                            {
                                "name": "phil",
                                "musician": false,
                                "age": 21,
                                "friends": [
                                    "alice"
                                ]
                            },
                            {
                                "name": "alice",
                                "musician": true,
                                "age": 23,
                                "friends": [
                                    "dimi"
                                ]
                            },
                            {
                                "name": "dimi",
                                "musician": true,
                                "age": 53,
                                "friends": [
                                    "phil",
                                    "alice"
                                ]
                            }
                        ]
                    },
                    "component": "input",
                    "publish": [
                        {
                            "stream": "data"
                        }
                    ],
                    "description": "The batch_input node simply generates some data.\nYou simply write your data inline, it convert it as Dataset<Row>",
                    "type": "dataset_generator"
                },
                {
                    "settings": {
                        "truncate": false
                    },
                    "component": "show",
                    "subscribe": [
                        {
                            "component": "input",
                            "stream": "data"
                        }
                    ],
                    "type": "show"
                }
            ],
            "type": "spark",
            "tenant": "mytenant"
        },
        "version": -1,
        "platformId": "my-unique-platform-id",
        "@timestamp": "2020-04-12T11:37:15.766Z"
    }
]

GET /v1/{tenant}/punchline/executions

Get all tenant related punchlines executions.

Request

Path variables

Field Description
tenant Tenant name

Response

Body

Field Description
id Execution ID
timestamp Last event timestamp
duration Execution duration
tenant Tenant name
event Execution last event

Status

Status Code Reason
200 successful operation

Example request

curl --location --request GET \ 
'http://localhost:4242/v1/mytenant/punchline/executions'

Example response

[
    {
        "id": "wc2Yh3EBnOslsCeb6UAg-1587117889508",
        "timestamp": "2020-04-17T10:04:55.320Z",
        "duration": 229,
        "tenant": "mytenant",
        "event": "spark.application.start"
    }
]

GET /v1/{tenant}/punchline/executions/{id}/events

List all events of a specific execution.

Request

Path variables

Field Description
tenant Tenant name
id Execution ID

Response

Body

Field Description
[].timestamp Event timestamp
[].tenant Tenant name
[].event Execution event

Status

Status Code Reason
200 successful operation
404 Execution not found

Example request

curl --location --request GET \
'http://localhost:4242/v1/mytenant/punchline/executions/wc2Yh3EBnOslsCeb6UAg-1587117889508/events'

Example response

[
    {
        "timestamp": "2020-04-17T10:04:58.247Z",
        "tenant": "mytenant",
        "event": "spark.application.end"
    },
    {
        "timestamp": "2020-04-17T10:04:58.241Z",
        "tenant": "mytenant",
        "event": "spark.application.end"
    },
    {
        "timestamp": "2020-04-17T10:04:58.217Z",
        "tenant": "mytenant",
        "event": "spark.application.end"
    },
    ... 
]

GET /v1/{tenant}/punchline/executions/{id}/output

Get output of a specific execution.

Request

Path variables

Field Description
tenant Tenant name
id Execution ID

Response

Status

Status Code Reason
200 successful operation
404 Execution not found

Example request

curl --location --request GET \
'http://localhost:4242/v1/mytenant/punchline/executions/wc2Yh3EBnOslsCeb6UAg-1587117889508/output'

GET /v1/{tenant}/punchline/executions/{id}

Get executions for a specific punchline.

Request

Path variables

Field Description
tenant Tenant name
id Punchline ID

Response

Body

Field Description
[].id Execution ID
[].timestamp Last event timestamp
[].duration Execution duration
[].tenant Tenant name
[].event Execution last event

Status

Status Code Reason
200 successful operation
404 Punchline not found

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/punchline/executions/wc2Yh3EBnOslsCeb6UAg'

Example Response

[
    {
        "id": "wc2Yh3EBnOslsCeb6UAg-1587117889508",
        "timestamp": "2020-04-17T10:04:58.247Z",
        "duration": 3156,
        "tenant": "mytenant",
        "event": "spark.application.end"
    }
]

POST /v1/{tenant}/punchline/save

Save a punchline

Request

Path variables

Field Description
tenant Tenant name

Form

Field Description
file Punchline file

Response

Status

Status Code Reason
200 successful operation

Example request

curl --location --request POST 'http://localhost:4242/v1/mytenant/punchline/save' \
--form 'file=@/data/punchlines/dataset_generator.pml'

GET /v1/{tenant}/punchline/scan/analytics

Get all nodes available for Spark punchline.

Request

Path variables

Field Description
tenant Tenant name

Response

Status

Status Code Reason
200 successful operation

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/punchline/scan/analytics'

GET /v1/{tenant}/punchline/scan/pyspark

Get all nodes available for Pyspark punchline.

Request

Path variables

Field Description
tenant Tenant name

Response

Status

Status Code Reason
200 successful operation

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/punchline/scan/pyspark'

GET /v1/{tenant}/punchline/scan/storm

Get all nodes available for Storm punchline.

Request

Path variables

Field Description
tenant Tenant name

Response

Status

Status Code Reason
200 successful operation

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/punchline/scan/storm'

DELETE /v1/{tenant}/punchline/scan/reset

Force the reset of the scanner cache

Request

Path variables

Field Description
tenant Tenant name

Response

Status

Status Code Reason
200 successful operation

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/punchline/scan'

GET /v1/{tenant}/punchline/{id}

Get punchline.

Request

Path variables

Field Description
tenant Tenant name
id Punchline id

Response

Body

Field Description
index Elasticsearch index where punchline is stored
id Punchline id
source Punchline file content

Status

Status Code Reason
200 successful operation
404 Punchline not found

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/punchline/wc2Yh3EBnOslsCeb6UAg'

Example response

{
    "index": "mytenant-applications",
    "id": "wc2Yh3EBnOslsCeb6UAg",
    "source": {
        "@timestamp": "2020-04-17T10:04:37.537Z",
        "platform_id": "my-unique-platform-id",
        "dag": "{\"runtime_id\":\"wc2Yh3EBnOslsCeb6UAg\",\"metrics\":{\"reporters\":[{\"http_hosts\":[{\"port\":9200,\"host\":\"localhost\"}],\"type\":\"elasticsearch\",\"index_name\":\"mytenant-metrics\"}]},\"job\":[{\"settings\":{\"input_data\":[{\"name\":\"phil\",\"musician\":false,\"age\":21,\"friends\":[\"alice\"]},{\"name\":\"alice\",\"musician\":true,\"age\":23,\"friends\":[\"dimi\"]},{\"name\":\"dimi\",\"musician\":true,\"age\":53,\"friends\":[\"phil\",\"alice\"]}]},\"component\":\"input\",\"publish\":[{\"stream\":\"data\"}],\"description\":\"The batch_input node simply generates some data.\\nYou simply write your data inline, it convert it as Dataset<Row>\",\"type\":\"dataset_generator\"},{\"settings\":{\"truncate\":false},\"component\":\"show\",\"subscribe\":[{\"component\":\"input\",\"stream\":\"data\"}],\"type\":\"show\"}],\"type\":\"spark\",\"tenant\":\"mytenant\"}",
        "tenant": "mytenant"
    }
}

POST /v1/{tenant}/punchline/{id}

Execute a saved punchline

Request

Path variables

Field Description
tenant Tenant name
id Punchline id

Response

Body

Field Description
id Execution generated id

Status

Status Code Reason
200 successful operation
404 Punchline not found

Example request

curl --location --request POST 'http://localhost:4242/v1/mytenant/punchline/wc2Yh3EBnOslsCeb6UAg'

Example response

{
    "id": "wc2Yh3EBnOslsCeb6UAg-1587117889508"
}

DELETE /v1/{tenant}/punchline/{id}

Delete a saved punchline.

Request

Path variables

Field Description
tenant Tenant name
id Punchline id

Response

Status

Status Code Reason
201 successful operation
404 Punchline not found

Example request

curl --location --request DELETE 'http://localhost:4242/v1/mytenant/punchline/wc2Yh3EBnOslsCeb6UAg'

GET /v1/{tenant}/resources/list

List resources

Request

Path variables

Field Description
tenant Tenant name

Parameters

Field Description
all true or false. If false, list the last version of each resource. If true, list all resources versions. Default is false
pattern Exact resource name, or a name pattern with wildcard
simplify true or false. If true, the list contains only the name, the version and the timestamp of each resource. Default is false
filter key=value filter on resource properties. Can be repeated for multiple filters

Response

Body

Field Description
metadata List of resources metadata
name Name pattern requested
message Response description
tenant The tenant of the resource manager

Status

Status Code Reason
200 Resourcce listed

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/resources/list?all=true&pattern=tests/*&simplify=true&filter=test=true&filter=description=hello'

Example response

{
    "metadata": [
        {
            "schema": "V1",
            "@timestamp": "2020-07-06T13:33:10.487Z",
            "size": 4806,
            "name": "tests/test.jpg",
            "storage": {
                "type": "file",
                "url": "/tmp/punchplatform/manager/resources/mytenant/tests/0/test.jpg"
            },
            "version": 0,
            "tenant": "mytenant",
            "properties": {}
        },
        {
            "schema": "V1",
            "@timestamp": "2020-07-06T13:42:19.402Z",
            "size": 4806,
            "name": "tests/test.jpg",
            "storage": {
                "type": "file",
                "url": "/tmp/punchplatform/manager/resources/mytenant/tests/1/test.jpg"
            },
            "version": 1,
            "tenant": "mytenant",
            "properties": {}
        }
    ],
    "name": "tests/*",
    "message": "List resources",
    "tenant": "mytenant"
}

PUT /v1/{tenant}/resources/upload/{name}

Upload a resource

Request

Path variables

Field Description
tenant Tenant name
name Future name of the resource inside the data storage

Form

Field Description
input Input file to upload
properties key=value custom resource property. Can be repeated for multiple resources
version Force a version to upload. If not provided, automatically detect the last version to upload
embedded true or false. If true, embed the resource data content inside the metadata

Response

Body

Field Description
metadata Uploaded resource details
name Name inside the data storage
message Response description
tenant The tenant of the resource manager

Status

Status Code Reason
201 Resource uploaded and metadata successfully stored
504 Data storage or metadata storage cannot be reached

Example request

curl --location --request PUT 'http://localhost:4242/v1/mytenant/resources/upload/tests/test.jpg' \
--form 'input=@/home/lca/Pictures/kerbal_wall_1.jpeg' \
--form 'properties={"description":"hello world, and aliens","test":true}' \
--form 'version=42'

Example response

{
    "metadata": [
        {
            "schema": "V1",
            "@timestamp": "2020-07-06T14:37:55.855Z",
            "size": 4806,
            "name": "tests/test.jpg",
            "storage": {
                "type": "file",
                "url": "/tmp/punchplatform/manager/resources/mytenant/tests/42/test.jpg"
            },
            "version": 42,
            "tenant": "mytenant",
            "properties": {
                "test": true,
                "description": "hello world, and aliens"
            }
        }
    ],
    "name": "tests/test.jpg",
    "message": "Upload resources",
    "tenant": "mytenant"
}

GET /v1/{tenant}/resources/download/{name}

Download a resource

Request

Path variables

Field Description
tenant Tenant name
name Name of the resource inside the data storage

Parameters

Field Description
output If provided, also write the resource content into this output file path
version Specific version to download. If not provided, download the last version of the resource (equals to the highest version number)

Response

Status

Status Code Reason
200 Resource downloaded
404 Resource not found

Example request

curl --location --request GET 'http://localhost:4242/v1/mytenant/resources/download/tests/test.jpg?output=/tmp/output.jpg&version=3'

Example response

Response is the resource data content

hello world

DELETE /v1/{tenant}/resources/delete/{name}

Delete a resource

Request

Path variables

Field Description
tenant Tenant name
name Name of the resource inside the data storage

Parameters

Field Description
version If provided, only delete this version
recursive true or false. If true, the name of the resource can be a incomplete path. All sub-resources will be deleted inside this path. Default is false

Response

Body

Field Description
delete Deleted resources details
name Name inside the data storage
message Response description
tenant The tenant of the resource manager

Status

Status Code Reason
200 Resource deleted
404 Resource not found

Example request

curl --location --request DELETE 'http://localhost:4242/v1/mytenant/resources/delete/tests/test.jpg?version=42'

Example response

{
    "name": "tests/test.jpg",
    "message": "Delete resources",
    "delete": [
        {
            "@timestamp": "2020-07-03T15:47:08.140Z",
            "name": "tests/test.jpg",
            "version": 0
        }
    ],
    "tenant": "mytenant"
}

PUT /v1/{tenant}/resources/copy/{name}

Copy a resource

Request

Path variables

Field Description
tenant Tenant name
name Name of the resource inside the data storage

Form

Field Description
destination Copy location of the resource inside the data storage
version If provided, only copy this version
embedded true or false. If true, embed the resource data content inside the metadata

Response

Body

Field Description
metadata Copied resources details
name Name inside the data storage
message Response description
tenant The tenant of the resource manager

Status

Status Code Reason
201 Resource copied
404 Original resource not found
504 Data storage or metadata storage cannot be reached

Example request

curl --location --request PUT 'http://localhost:4242/v1/mytenant/resources/copy/tests/test.jpg' \
--form 'destination=copies/test.jpg' \
--form 'version=1'

Example response

{
    "metadata": [
        {
            "schema": "V1",
            "@timestamp": "2020-07-06T14:07:21.288Z",
            "size": 4806,
            "name": "copies/test.jpg",
            "storage": {
                "type": "file",
                "url": "/tmp/punchplatform/manager/resources/mytenant/copies/0/test.jpg"
            },
            "version": 0,
            "tenant": "mytenant",
            "properties": {}
        }
    ],
    "name": "tests/test.jpg",
    "message": "Copy resources",
    "tenant": "mytenant"
}

PUT /v1/{tenant}/resources/move/{name}

Copy a resource

Request

Path variables

Field Description
tenant Tenant name
name Name of the resource inside the data storage

Form

Field Description
destination New location of the resource inside the data storage
version If provided, only move this version
embedded true or false. If true, embed the resource data content inside the metadata

Response

Body

Field Description
metadata Moved resources details
name Name inside the data storage
message Response description
tenant The tenant of the resource manager

Status

Status Code Reason
201 Resource copied
404 Resource not found
504 Data storage or metadata storage cannot be reached

Example request

curl --location --request PUT 'http://localhost:4242/v1/mytenant/resources/move/tests/test.jpg' \
--form 'destination=moves/test.jpg' \
--form 'version=1'

Example response

{
    "metadata": [
        {
            "schema": "V1",
            "@timestamp": "2020-07-06T14:16:59.678Z",
            "size": 4806,
            "name": "moves/test.jpg",
            "storage": {
                "type": "file",
                "url": "/tmp/punchplatform/manager/resources/mytenant/moves/0/test.jpg"
            },
            "version": 0,
            "tenant": "mytenant",
            "properties": {}
        }
    ],
    "name": "tests/test.jpg",
    "message": "Move resources",
    "tenant": "mytenant"
}

PUT /v1/{tenant}/resources/register/{name}

Register the metadata of an existing resource

Request

Path variables

Field Description
tenant Tenant name
name Name of the resource inside the data storage

Form

Field Description
version Version of the resource to register
url Exact location of the resource inside the data storage
size Size of the resource to register
properties Custom properties of the resource to register
compression Compression type of the resource, if needed
encoding Encoding type of the data content

Response

Body

Field Description
metadata Registered resource details
name Name inside the data storage
message Response description
tenant The tenant of the resource manager

Status

Status Code Reason
200 Resource registered
404 Resource not found
422 If the resource version already exists

Example request

curl --location --request PUT 'http://localhost:4242/v1/mytenant/resources/register/tests/test.jpg' \
--form 'version=3' \
--form 'url=/tmp/punchplatform/manager/resources/mytenant/tests/0/test.jpg' \
--form 'properties={"description":"hello world, and aliens","test":true}' \
--form 'size=42' \
--form 'compression=gzip' \
--form 'encoding=UTF8'

Example response

{
    "metadata": [
        {
            "schema": "V1",
            "@timestamp": "2020-07-06T14:21:58.716Z",
            "size": 3,
            "name": "tests/test.jpg",
            "storage": {
                "type": "file",
                "encoding": "UTF8",
                "compression": "gzip",
                "url": "/tmp/punchplatform/manager/resources/mytenant/tests/0/nomods.png"
            },
            "version": 3,
            "tenant": "mytenant",
            "properties": {
                "test": true,
                "description": "hello world, and aliens"
            }
        }
    ],
    "name": "tests/test.jpg",
    "message": "Register resource",
    "tenant": "mytenant"
}

GET /v1/{tenant}/documentation/{page}

Display the documentation page content

Request

Path variables

Field Description
tenant Tenant name
page Path of the documentation page

Response

Status

Status Code Reason
200 Page content
404 Page not found

Example request

curl --location --request GET 'http://localhost:4242/v1/documentation/index.html'