org.thales.punch.libraries.punchlang.operator

Class FetchJsonOperator

java.lang.Object

org.thales.punch.libraries.punchlang.operator.FetchOperator<FetchJsonOperator>

org.thales.punch.libraries.punchlang.operator.FetchJsonOperator

public class FetchJsonOperator
extends FetchOperator<FetchJsonOperator>

Using the fetchJson operator you can fetch remote or local JSON files, and use them as resource tuple in your punchlet. Here is a first example:

 Tuple resource = fetchJson("https://w3c.github.io/csvw/tests/test001.json").asTuple();
 

This code will fetch the test001.csv file from the w3c web server, cache it and make it available to your punchlet.

You can use the refresh option to make the operator regularly refresh your resource. Here is a plian example to fetch a json document as a resource tuple.

 Tuple resource = fetchJson("https://w3c.github.io/csvw/tests/test001.json")
                  .asTuple();
 

And here is an example to reload that json every 60 seconds.

 Tuple resource = fetchJson("https://w3c.github.io/csvw/tests/test001.json")
                  .refresh("0/60 0/1 * 1/1 * ? *")
                  .asTuple();
 

This operaror extends the FetchOperator.

Fetching local file resources

The fetch operator can also be used with local punchlet file resources. Here is an example of a punchline punch node settings:

   {
     "type": "punchlet_node"
     "settings": {
       "punchlet_file_resources": [
           "standard/resources/enrichment.json"
       ],
       "punchlet": [
           "punchlet.punch"
       ]
     }
   }
 

You can then simply write :

  Tuple resource = fetchJson(standard/resources/enrichment.json).asTuple();
 

As just illustrated you can define JSON files. Refer to the fFetchCsvOperator variant for CSV files.

Fetching Json Arrays

It is common to have json arrays. For example :

 [
 {"id":"id32","name":"dimi","age":56},
 {"id":"id45","name":"phil","age":22},
 {"id":"id55","name":"sofia","age":22}
 ]
 

You can convert this array into a dictionary as follows:

    [result] = fetchJson("resource.json").hashKey("id").asTuple();
  

This will return to you :

  "id32": { "id":"id32", "age": 56, "name": "dimi" },
  "id45": { "id":"id32", "age": 22, "name": "phil" },
  "id55": { "id":"id32", "age": 22, "name": "sofia" }
  

You can also make your dictionary keys lower case using the lowerCaseKeys() method.

Large JSON files

By default the returned tuple is a regular punch Tuple, that provides you will all the methods. Using these to hold large hash table is not memory efficient though. If your files are big (more than several tenth of Mbs), you can use the compact option as illustrated next.

 Tuple resource = fetchJson("https://your.resource.server/resource.json")
                  .refresh("0/60 0/1 * 1/1 * ? *")
                  .hashKey("ip")
                  .compact()
                  .asTuple();
 

This option use a more compact strategy to represent the resource in memory. The returned tuple provides you only with the get(String key) method.

Plain String

Watch out : if your resource file contains a non Json content, (for example if put some CSV content into a JSON file), the file content will be returned to you as a plain String (leaf) value.

Field Summary

Fields inherited from class org.thales.punch.libraries.punchlang.operator.FetchOperator

compactionType, hashKey, logger, lowerCaseKeys, requiredResource, runtimeContext, s3Bucket, s3endpoint, s3KeyPath, s3Object, s3SecretPath, silent, url, uuid

Constructor Summary

Constructors

Constructor and Description
FetchJsonOperator(RuntimeContext r) Create a fetch operator to load JSON resources
FetchJsonOperator(RuntimeContext r, String url) Create a fetch operator for a remote resource.

Method Summary

Modifier and Type	Method and Description
Tuple	asTuple()
IResourceBuilder	getResourceBuilder() Implemented by subclasses to return the adequate resource builder.

Methods inherited from class org.thales.punch.libraries.punchlang.operator.FetchOperator

bestEffort, compact, compactDirect, compactInPlace, getResource, hashKey, loadAtStartup, lowerCaseKeys, refresh, required, s3AccessKey, s3AccessKeyPath, s3Bucket, s3Endpoint, s3Object, s3Secret, silent, url

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FetchJsonOperator

public FetchJsonOperator(RuntimeContext r,
                         String url)

Create a fetch operator for a remote resource.

Parameters:

r - the punchlet runtime context.

url - an HTTP or file URI.

FetchJsonOperator

public FetchJsonOperator(RuntimeContext r)

Create a fetch operator to load JSON resources

Parameters:

r - the punchlet runtime context

Method Detail

asTuple

public Tuple asTuple()

getResourceBuilder

public IResourceBuilder getResourceBuilder()

Description copied from class: FetchOperator

Implemented by subclasses to return the adequate resource builder. For example the FetchCsvOperator returns a builder to construct a tuple from a CSV document while the FetchJsonOperator returns builders to deal with JSON files.

This is called only once at init time.

Specified by:

getResourceBuilder in class FetchOperator<FetchJsonOperator>

Returns:

the target resource builder