HOWTO start a Batches requests service

Overview

For a functional description of the Batch Requests Service, please refer to Batches requests service overview.

This procedure will allow you to test the overall batch requests feature of PunchPlatform using a standalone deployment. The main feature is that we will import a list of reference values to search, then trigger a search within some test data, and finally retrieve the matching documents, all this without any need of interaction with the PunchPlatform HMI (although we will use it to disclose the batch and reference sets managements under-the-hood behaviour).

Prerequisites

You need to have a standalone PunchPlatform distribution deployed with –with-kibana-plugins and –with-shiva option and running.

Please refer to the Standalone quick-tour for details on standalone deployment and usage.

Activation of the Batch Requests Service

1) Checking the Batch Requests Service is UP

  • test standalone batch service status:

    punchplatform-batch-service.sh --status
    
  • test your plugin api:

    curl http://localhost:5001/punchplatform/api/v1/lists?tenant=mytenant | jq 'if . then "EVERYTHING IS OK" else "NOT WORKING" end'
    

2) (Optional) Deploying the Graphical User Interface kibana plugin into Kibana :

YOU DO NEED THIS STEP ONLY IF YOU DID NOT USE THE “–with-kibana-plugins” WHEN YOU INSTALLED YOUR STANDALONE.

  • stop kibana (if it was running)

    punchplatform-batch-service.sh --stop
    
  • install plugin

    cd $PUNCHPLATFORM_CONF_DIR
    external/kibana-5.6.4-linux-x86_64/bin/kibana-plugin install file:///home/cedric/lmc/active_loc/standalone/external/punchplatform-batches-requests-plugin-4.0.1-SNAPSHOT.zip
    

    This may take a minute or two.

  • start kibana

    punchplatform-batch-service.sh --start
    

Creation of test data, including an import of a ‘reference values list’ for searching

1) Create some test data in Elasticsearch

cd $PUNCHPLATFORM_CONF_DIR
punchplatform-channel.sh --configure tenants/mytenant/configurations_for_channels_generation/lmc
punchplatform-channel.sh --start mytenant/sourcefire
punchplatform-log-injector.sh -c resources/injector/mytenant/sourcefire_injector.json -n 100000 -t 2000

2) Create a list of reference IPs :

{   echo "189.23.213.149" ;  echo "189.23.213.150" ; echo "189.23.213.151" ; } | cat > /tmp/suspectips.list

3) Import the list of reference IPS into the Batches Service :

curl -XPOST -H "Content-Type: multipart/form-data" -F 'file=@/tmp/suspectips.list;type=application/octet-stream' 'http://localhost:5001/punchplatform/api/v1/lists?tenant=mytenant&description=suspect%20IPs&id=mySuspectIpsList&dataType=IP'

4) Ensure your list has been correctly imported :

  • Check and view your list in the GUI using a Web Browser :

  • check your list presence through API :

    curl 'localhost:5001/punchplatform/api/v1/lists?tenant=mytenant' | jq .
    
    
    # ==> returns something like this       :
    #       [
    #       {
    #           "list_id": "mySuspectIpsList",
    #           "list_description": "suspect IPs",
    #           "list_line_separator": "LINUX_LINE",
    #           "import_date": "2017-12-14T16:01:51.065+01:00",
    #           "data_type": "IP"
    #       }
    #       ]
    
  • get your list content (change the list id to match yours) :

    curl 'localhost:5001/punchplatform/api/v1/lists/mySuspectIpsList/data?tenant=mytenant' -o /tmp/listcontent
    cat /tmp/listcontent
    

Requesting an extraction of matching data

1) Create an extraction job using API

curl -XPOST -H "Content-Type: application/json" 'http://localhost:5001/punchplatform/api/v1/jobs?tenant=mytenant' -d '
{
         "id" : "my_job",
         "description" : "my description",
         "schedule" : { "plan" : "ANYTIME", "period" : "ONLY_ONCE" },
         "output" : {"format" : "CSV" },
         "settings": {
              "type": "extract",
              "filters_query": ["{ \"terms\": {\"target.host.ip\": [ \"__list_mySuspectIpsList\"]}}"],
              "index_patterns": [
                "mytenant-events-*"
              ],
              "extraction_scope_field": "ts",
              "extraction_scope_start": "2017-01-01T01:00:00.000+01:00",
              "extraction_scope_end": "2020-01-01T01:00:00.000+01:00",
              "requested_output_fields": [
                "ts",
                "_source"
              ]
         }
}
'

2) Check your job is persisted

  • check your job presence through API :

    curl -XGET "localhost:5001/punchplatform/api/v1/jobs/my_job?tenant=mytenant" | jq .
    
    # ==> returns something like this       :
    
    # {
    #               "current_job_representation": {
    #                       "id": "my_job",
    #                       "tenant": "mytenant",
    #                       "description": "my description",
    #                       "schedule": {
    #                       "plan": "ANYTIME",
    #                       "period": "ONLY_ONCE"
    #                       },
    #                       "job_status": "DEFINED",
    #                       "update_date": "2017-12-18T16:06:42.551+01:00",
    #                       "available_operations": [
    #                       "SUBMIT",
    #                       "EDIT",
    #                       "PURGE"
    #                       ],
    #                       "settings": {
    #                       "type": "extract",
    #                       "filters_query": [
    #                       "{ \"terms\": {\"target.host.ip\": [ \"__list_mySuspectIpsList\"]}}"
    #                       ],
    #                       "index_patterns": [
    #                       "mytenant-events-*"
    #                       ],
    #                       "extraction_scope_start": "2017-01-01T01:00:00+01:00",
    #                       "extraction_scope_end": "2020-01-01T01:00:00+01:00",
    #                       "extraction_scope_field": "ts",
    #                       "requested_output_fields": [
    #                       "ts",
    #                       "_source"
    #                       ],
    #                       "scroll_timeout": "3m",
    #                       "requests_timeout": "5m",
    #                       "maximum_events_to_extract": 10000000,
    #      "scroll_size": 10000,
    #                       "source_data_location": {
    #                       "type": "elasticsearch",
    #                       "cluster_details": {
    #                               "api_servers": [
    #                               "localhost"
    #                               ],
    #                               "api_port": 9200,
    #                               "cluster_name": "es_search"
    #                       },
    #                       "indexes": [
    #                               "mytenant-events-*"
    #                       ]
    #                       },
    #                       "expanded_filters_query": [
    #                       "{ \"terms\": {\"target.host.ip\": [ \"189.23.213.149\",\"189.23.213.150\",\"189.23.213.151\"]}}"
    #                       ]
    #                       }
    #               },
    #               "job_execution_representations": []
    # }
    
  • Check and view your job in the GUI :

2) Start your job

curl -H 'Content-Type: application/json' -XPOST "localhost:5001/punchplatform/api/v1/jobs/my_job/_SUBMIT?tenant=mytenant"
  • check status of your job through API :

    curl -s -XGET "localhost:5001/punchplatform/api/v1/jobs/my_job?tenant=mytenant" | jq .current_job_representation.job_status
    
        # ==> returns something like this :
    
        #  "DEFINED", "SUBMITTED", "RUNNING" or "COMPLETE"
    
  • repeat until “COMPLETE”

3) Get result of your job

curl -XGET "localhost:5001/punchplatform/api/v1/jobs/my_job/resultData?tenant=mytenant"

        # ==> returns something like this       :


        #  ts;_source
        #  2017-12-14T16:00:49.028+01:00;"{app={proto={name=tcp}}, col={host={port=9902, ip=127.0.0.1, name=pommeau}}, obs={process={name=SFIMS, id=119:4:1}, host={ip=127.0.0.1, name=host6}}, init={host={port=14267, ip=192.168.99.114}}, lmc={input={ts=2017-12-14T16:00:45.160+01:00}, parse={host={ip=127.0.0.1, name=pommeau}, ts=2017-12-14T16:00:48.361+01:00}}, channel=sourcefire, type=ids, message=Dec 14 16:00:44 host6 sourcefiredc: SFIMS: [119:4:1] http_inspect: BARE BYTE UNICODE ENCODING [Impact: Potentially Vulnerable] From ""IPS_inline_DE/MARC-3B4-SF3D45-1"" at Thu Dec 16:00:44 2017 UTC [Classification: Not Suspicious Traffic] [Priority: 3] {tcp} 192.168.99.114:14267->189.23.213.149:80, target={usr={loc={country=Brazil, country_short=BR, geo_point=[-46.6358, -23.5477]}}, host={port=80, ip=189.23.213.149}}, size=295, vendor=sourcefire, alarm={sev=3, name=Not Suspicious Traffic, description=http_inspect: BARE BYTE UNICODE ENCODING [Impact: Potentially Vulnerable] From ""IPS_inline_DE/MARC-3B4-SF3D45-1"" at Thu Dec 16:00:44 2017 UTC}, rep={host={ip=127.0.0.1, name=host6}, ts=2017-12-14T16:00:44.000+01:00}, tenant=mytenant, ts=2017-12-14T16:00:49.028+01:00}"


        # ... number of records depending on the random ips generated by the test flow

4) Delete your job

curl -XDELETE "localhost:5001/punchplatform/api/v1/jobs/my_job?tenant=mytenant"