FetchOperator (Punch Punch Language Runtime Library 6.4.4 API)

java.lang.Object
- org.thales.punch.libraries.punchlang.operator.FetchOperator<T>

Direct Known Subclasses:: CidrRangeOperator, FetchCsvOperator, FetchJsonOperator, IpRangeOperator

public abstract class FetchOperator<T>
extends Object

This is the base class for the FetchCsvOperator and the FetchJsonOperator, CidrRangeOperator operators, and more generally all operators that support the loading of remote CSV or JSON resource files.

Using these you can fetch remote or local CSV or JSON files, and use them as resource tuples as required by the operators.

In turn you do not need to restart your processing pipelines whenever you update your resources files, this will be automatically reloaded without service interruption.

Whatever be the operator, here is how you can control the dynamic reloading of remote resources. Here is a simple example using the fetchCsv punch operator.


 Tuple resource = fetchCsv("https://your.resource.server/resource.csv")
                  // make sure the resource is synchronously loaded at startup
                  .required()
                  // define the reload period
                  .refresh("0/60 0/1 * 1/1 * ? *")
                  // this part is specific to the fetchCsv operator
                  .asTuple();

Here is a similar example with the cidrRange operator:


 if (
         cidrRange("https://your.resource.server/resource.csv")
         .required()
         .refresh("0/60 0/1 * 1/1 * ? *")
         // this part is specific to the cidrRange operator
         .cidrField("range")
         .on("192.168.1.3")
         .into([match]
     ) {
     ...
 }

You can fetch remote resources from HTTP, HTTPS, local files or S3. Here is an example using S3:


 Tuple resource = fetchCsv()
                  .s3endpoint("https://play.min.io")
                  .s3AccessKey("Q3AM3UQ867SPQQA43P2F")
                  .s3SecretKey(""zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG")
                  .s3Bucket("myresources")
                  .s3File("myresources.csv.gz")
                  .required()
                  .refresh("0/60 0/1 * 1/1 * ? *")
                  .asTuple();

Memory Usage

If you have large JSON or CSV files (millions of objects) the conversion to punch Tuple will require a lot of memory. You can request the operator to leverages different strategies to drastically reduces memory usage:

compact	the compact stores only Strings and not nested Tuple. Conversion to tuples is lazy and performed in case of matches
compactDirect	This mode internally leverages a direct memory ByteBuffer. This is the most memory efficient strategy.
compactInPlace	This is a compact strategy except that a single resource Tuple is kept and refreshed in memory. This avoids to require aditional memory only for (periodically) loading the new resource tuple.

Error Handling

The loading of resources is performed asynchronously in a way not to impact the traffic. The punch engine publishes monitoring metrics so that you can monitor the correct loading of your resources. Four metrics are published :

punchlang.fetch.rtt	measure the round trip time in milliseconds to load the resource
punchlang.fetch.length	A sliding window histogram. Measure the length of the loaded resource in bytes. This value is computed based on the last 10 resource.
punchlang.fetch.success	Meter. The number of successful loading
punchlet.fetch.failure	Meter. The number of failed loading.

These metrics are published as part of the caller metrics context. For example if your punchlet is executed in a storm compatible engine, you will have "storm.punchlet.fetch.length" together with the caller tenant, component and platform identifiers. The resource URL is provided as well in a 'punchlang.resource.url' field.

Cron Expression Examples

Here are some simple examples:

*** * * ? * ***	Every second
0 * * ? * *	Every minute
0 /2 ? * *	Every even minute
0 1/2 * ? * *	Every uneven minute
0 /2 ? * *	Every 2 minutes
0 /3 ? * *	Every 3 minutes
0 /4 ? * *	Every 4 minutes
0 /5 ? * *	Every 5 minutes
0 /10 ? * *	Every 10 minutes
0 /15 ? * *	Every 15 minutes
0 /30 ? * *	Every 30 minutes
0 15,30,45 * ? * *	Every hour at minutes 15, 30 and 45
0 0 * ? * *	Every hour
0 0 /2 ? *	Every hour
0 0 0/2 ? * *	Every even hour
0 0 1/2 ? * *	Every uneven hour
0 0 /3 ? *	Every three hours
0 0 /4 ? *	Every four hours
0 0 /6 ? *	Every six hours
0 0 /8 ? *	Every eight hours
0 0 /12 ? *	Every twelve hours
*0 0 0 * ?**	Every day at midnight - 12am
*0 0 1 * ?**	Every day at 1am
*0 0 6 * ?**	Every day at 6am
*0 0 12 * ?**	Every day at noon - 12pm
*0 0 12 * ?**	Every day at noon - 12pm
*0 0 12 * SUN**	Every Sunday at noon
*0 0 12 * MON**	Every Monday at noon
*0 0 12 * TUE**	Every Tuesday at noon
*0 0 12 * WED**	Every Wednesday at noon
*0 0 12 * THU**	Every Thursday at noon
*0 0 12 * FRI**	Every Friday at noon
*0 0 12 * SAT**	Every Saturday at noon
*0 0 12 * MON-FRI**	Every Weekday at noon
*0 0 12 * SUN,SAT**	Every Saturday and Sunday at noon
*0 0 12 /7 * ?**	Every 7 days at noon
*0 0 12 1 ?**	Every month on the 1st, at noon
*0 0 12 2 ?**	Every month on the 2nd, at noon
*0 0 12 15 ?**	Every month on the 15th, at noon
*0 0 12 1/2 ?**	Every 2 days starting on the 1st of the month, at noon
*0 0 12 1/4 ?**	Every 4 days staring on the 1st of the month, at noon
*0 0 12 L ?**	Every month on the last day of the month, at noon
*0 0 12 L-2 ?**	Every month on the second to last day of the month, at noon
*0 0 12 LW ?**	Every month on the last weekday, at noon
*0 0 12 1L ?**	Every month on the last Sunday, at noon
*0 0 12 2L ?**	Every month on the last Monday, at noon
*0 0 12 6L ?**	Every month on the last Friday, at noon
*0 0 12 1W ?**	Every month on the nearest Weekday to the 1st of the month, at noon
*0 0 12 15W ?**	Every month on the nearest Weekday to the 15th of the month, at noon
*0 0 12 ? 2#1**	Every month on the first Monday of the Month, at noon
*0 0 12 ? 6#1**	Every month on the first Friday of the Month, at noon
*0 0 12 ? 2#2**	Every month on the second Monday of the Month, at noon
*0 0 12 ? 5#3**	Every month on the third Thursday of the Month, at noon - 12pm
0 0 12 ? JAN *	Every day at noon in January only
0 0 12 ? JUN *	Every day at noon in June only
0 0 12 ? JAN,JUN *	Every day at noon in January and June
0 0 12 ? DEC *	Every day at noon in December only
0 0 12 ? JAN,FEB,MAR,APR *	Every day at noon in January, February, March and April
0 0 12 ? 9-12 *	Every day at noon between September and December

Field Summary

Fields
Modifier and Type	Field and Description
`protected IResourceBuilder.CompactionType`	`compactionType` Make it true to limit memory usage
`protected String`	`hashKey` An optional hash key to be used to generate a hash table instead of an array.
`protected static org.apache.logging.log4j.Logger`	`logger`
`protected boolean`	`lowerCaseKeys` True to make search keys converted to lower case
`protected boolean`	`requiredResource` Set to true to prevent any processing in case of unavailable resource (at start time or afterwards).
`protected RuntimeContext`	`runtimeContext` The caller punchlet runtime context.
`protected String`	`s3Bucket` the S3 bucket.
`protected String`	`s3endpoint` the S3 end-point is the target S3 server HTTP.
`protected Path`	`s3KeyPath` the S3 access key.
`protected String`	`s3Object` the S3 resource object.
`protected Path`	`s3SecretPath` the S3 access key.
`protected boolean`	`silent` Set to true to prevent any processing in case of unavailable resource (at start time or afterwards).
`protected String`	`url` The remote or local resource URL.
`protected String`	`uuid` We need a unique identifier to identify each resource.

Constructor Summary

Constructors
Modifier Constructor and Description

protected FetchOperator(RuntimeContext r)
Ctor.

protected FetchOperator(RuntimeContext r, String url)
Ctor.

Constructors
Modifier	Constructor and Description
`protected`	`FetchOperator(RuntimeContext r)` Ctor.
`protected`	`FetchOperator(RuntimeContext r, String url)` Ctor.

Method Summary

All Methods Instance Methods Abstract Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`T`	`bestEffort()` Indicate that the resource must not be absolutely loaded before the traffic pass.
`T`	`compact()` In case you have very big resource files (30Mb, 100Mb or more), this option will represent the resource tuple in a way much more compact.
`T`	`compactDirect()` This option is similar than the compact method but leverage bytebuffers with direct memory allocation.
`T`	`compactInPlace()` Set to true to make resource tuples refreshed in place.
`Tuple`	`getResource()` This subl
`abstract IResourceBuilder`	`getResourceBuilder()` Implemented by subclasses to return the adequate resource builder.
`T`	`hashKey(String hashKey)` Use this method to produce a dictionary instead of an array.
`T`	`loadAtStartup()` Deprecated.
`T`	`lowerCaseKeys()` Make the lookup key values be converted to lower case value.
`T`	`refresh(String cron)` Make the operator periodically fetch the remote resource.
`T`	`required()` Indicate that the resource is absolutely required for the processing.
`T`	`s3AccessKey(String key)` This method is unsafe, use it only for test or ad-hoc investigations.
`T`	`s3AccessKeyPath(String path)` This method is unsafe, use it only for test or ad-hoc investigations.
`T`	`s3Bucket(String bucket)` The bucket where to load your resource file
`T`	`s3Endpoint(String endPoint)` Use this parameter for loading from a S3 compatible store.
`T`	`s3Object(String name)` The bucket where to load your resource file
`T`	`s3Secret(String secret)` This method is unsafe, use it only for test or ad-hoc investigations.
`T`	`silent()` Indicate that failures to load the resource must be silently ignored.
`T`	`url(String url)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - logger
```
protected static final org.apache.logging.log4j.Logger logger
```
  - uuid
```
protected String uuid
```
    We need a unique identifier to identify each resource. It can be the URL (in case of file or HTTP resource) or the S3 equivalent information.
  - url
```
protected String url
```
    The remote or local resource URL. Must start by "https://", "http://" or "file:/". Note that for S3 the end-point is expected to be "https://"
  - s3endpoint
```
protected String s3endpoint
```
    the S3 end-point is the target S3 server HTTP.
  - s3KeyPath
```
protected Path s3KeyPath
```
    the S3 access key. The only safe method is to provide it through secret files. You can provided hard coded values for test or unsecured platform usage.
  - s3SecretPath
```
protected Path s3SecretPath
```
    the S3 access key. The only safe method is to provide it through secret files. You can provided hard coded values for test or unsecured platform usage.
  - s3Bucket
```
protected String s3Bucket
```
    the S3 bucket.
  - s3Object
```
protected String s3Object
```
    the S3 resource object.
  - requiredResource
```
protected boolean requiredResource
```
    Set to true to prevent any processing in case of unavailable resource (at start time or afterwards).
  - silent
```
protected boolean silent
```
    Set to true to prevent any processing in case of unavailable resource (at start time or afterwards).
  - compactionType
```
protected IResourceBuilder.CompactionType compactionType
```
    Make it true to limit memory usage
  - lowerCaseKeys
```
protected boolean lowerCaseKeys
```
    True to make search keys converted to lower case
  - runtimeContext
```
protected final RuntimeContext runtimeContext
```
    The caller punchlet runtime context.
  - hashKey
```
protected String hashKey
```
    An optional hash key to be used to generate a hash table instead of an array. This is most often used, the point of a resource tuple is generally to be used as a hash map so as to quickly find elements.
- Constructor Detail
  - FetchOperator
```
protected FetchOperator(RuntimeContext r)
```
    Ctor.
    
    Parameters:
    
    r - the punchlet runtime context.
    
    url - an http or file url.
  - FetchOperator
```
protected FetchOperator(RuntimeContext r,
                        String url)
```
    Ctor.
    
    Parameters:
    
    r - the punchlet runtime context.
    
    url - an http or file url.
- Method Detail
  - required
```
public T required()
```
    Indicate that the resource is absolutely required for the processing. If this operator is not called, the resource loading will be attempted at start time anyway, but a failure will not prevent the processing of tuples
    
    Returns:
    
    this operator
  - silent
```
public T silent()
```
    Indicate that failures to load the resource must be silently ignored. An empty tuple should be returned in case of issue.
    
    Returns:
    
    this operator
  - bestEffort
```
public T bestEffort()
```
    Indicate that the resource must not be absolutely loaded before the traffic pass.
    
    Returns:
    
    this operator
  - loadAtStartup
```
@Deprecated
public T loadAtStartup()
```
    Deprecated.
    
    Deprecated. Use required() instead.
    
    Returns:
    
    this operator
  - s3Endpoint
```
public T s3Endpoint(String endPoint)
```
    Use this parameter for loading from a S3 compatible store.
    
    Parameters:
    
    endPoint - the S3 end point
    
    Returns:
    
    this operator
  - s3AccessKey
```
public T s3AccessKey(String key)
```
    This method is unsafe, use it only for test or ad-hoc investigations. Prefer loading the secret from a mounted file.
    Providing an access key is not required for anonymous accesses.
    
    Parameters:
    
    endPoint - the S3 end point
    
    Returns:
    
    this operator
  - s3AccessKeyPath
```
public T s3AccessKeyPath(String path)
```
    This method is unsafe, use it only for test or ad-hoc investigations. Prefer loading the secret from a mounted file.
    Providing an access key is not required for anonymous accesses.
    
    Parameters:
    
    endPoint - the S3 end point
    
    Returns:
    
    this operator
  - s3Secret
```
public T s3Secret(String secret)
```
    This method is unsafe, use it only for test or ad-hoc investigations. Prefer loading the secret from a mounted file.
    Providing an secret is not required for anonymous accesses.
    
    Parameters:
    
    endPoint - the S3 end point
    
    Returns:
    
    this operator
  - url
```
public T url(String url)
```
    Parameters:
    
    url - the HTTP HTTPS or file URL.
    
    Returns:
    
    this operator.
  - s3Bucket
```
public T s3Bucket(String bucket)
```
    The bucket where to load your resource file
    
    Parameters:
    
    endPoint - the S3 end point
    
    Returns:
    
    this operator
  - s3Object
```
public T s3Object(String name)
```
    The bucket where to load your resource file
    
    Parameters:
    
    filenameName - the S3 object name
    
    Returns:
    
    this operator
  - hashKey
```
public T hashKey(String hashKey)
```
    Use this method to produce a dictionary instead of an array.
    
    Parameters:
    
    hashKey - the key of one of the field to be used as hash key.
    
    Returns:
    
    this operator
  - lowerCaseKeys
```
public T lowerCaseKeys()
```
    Make the lookup key values be converted to lower case value. This does not affect the case of the content of the match.
    For example if you have the following resource :
```
 first user,dimi,54
 Second User,ced,43
 Third user,damien,43
 
```
    You will be able to query your tuple using "second user" and "third user".
    Returns:
    
    this operator
  - compact
```
public T compact()
```
    In case you have very big resource files (30Mb, 100Mb or more), this option will represent the resource tuple in a way much more compact. Basically instead of completely transforming your CSV of JSON files into ready to use nested tuples, only String representations are stored.
    On the good side, this drastically reduces the number of java objects, hence memory usage and garbage collection costs. On the down side every time you have a hit, a punch tuple will be constructed from that source String.
    In other words, the compact strategy reduces RAM (possibly by a factor two) but increases the CPU. It all depends if you have many hits. Only benchmarks can tell you if this option is interesting for you.
    Note that using that option only the Tuple.get(String) method will be supported. Compact tuples are read-only.
    
    Returns:
    
    this operator.
  - compactInPlace
```
public T compactInPlace()
```
    Set to true to make resource tuples refreshed in place. This avoids the reloading of a fresh copy followed by an atomic switch between the old and the new resource tuple.
    This strategy leverages the compact() resource encoding. The only difference is there is a single copy updated in place.
    
    Returns:
    
    this operator
  - compactDirect
```
public T compactDirect()
```
    This option is similar than the compact method but leverage bytebuffers with direct memory allocation. If your resources are only UTF_8, this is the most efficient method.
    
    Returns:
    
    this operator.
  - refresh
```
public T refresh(String cron)
```
    Make the operator periodically fetch the remote resource.
    
    Parameters:
    
    cron - a QUARTZ cron expression.
    
    Returns:
    
    this operator
  - getResource
```
public Tuple getResource()
```
    This subl
    
    Returns:
    
    the resource associated to this URL.
  - getResourceBuilder
```
public abstract IResourceBuilder getResourceBuilder()
```
    Implemented by subclasses to return the adequate resource builder. For example the FetchCsvOperator returns a builder to construct a tuple from a CSV document while the FetchJsonOperator returns builders to deal with JSON files.
    This is called only once at init time.
    
    Returns:
    
    the target resource builder

Class FetchOperator<T>

Memory Usage

Error Handling

Cron Expression Examples

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

logger

uuid

url

s3endpoint

s3KeyPath

s3SecretPath

s3Bucket

s3Object

requiredResource

silent

compactionType

lowerCaseKeys

runtimeContext

hashKey

Constructor Detail

FetchOperator

FetchOperator

Method Detail

required

silent

bestEffort

loadAtStartup

s3Endpoint

s3AccessKey

s3AccessKeyPath

s3Secret

url

s3Bucket

s3Object

hashKey

lowerCaseKeys

compact

compactInPlace

compactDirect

refresh

getResource

getResourceBuilder