org.thales.punch.libraries.storm.bolt.file

Class BatchArchiver

java.lang.Object

org.thales.punch.libraries.storm.bolt.file.BatchArchiver

public class BatchArchiver
extends Object

An BatchArchiver is in charge of writing the incoming data to one or several destination. It must be associated with an AvroParquetArchiveBuffer.

This class holds a current batch of tuples. It provides a method to marshall them all in a buffer once the file bolt receives the batch completion signal.

It also provide a bloom filter.

Author:

dimi

Constructor Summary

Constructors

Constructor and Description
BatchArchiver(org.thales.punch.libraries.objectstorage.indexing.BatchMetadata metaData, org.thales.punch.libraries.objectstorage.tuples.ArchiveFormatSettings archiveFormatSettings) Ctor.

Method Summary

Modifier and Type	Method and Description
void	archiveTuple(org.apache.storm.tuple.Tuple tuple, long epoch, Iterable<String> bloomFields) Marshal this tuple into the archive(s).
String	flush() Flush your archive.
org.apache.storm.tuple.Tuple	getAnchoredTuple() Getter for anchored tuple
org.thales.punch.libraries.objectstorage.indexing.BatchMetadata	getBatchMetadata() An BatchArchiver is handling a single batch.
String	getExpirationCause() Getter for expiration cause
boolean	hasExpired(long expirationTimeout, long lifeTimeout) Check if this batch has expired, given an expiration timeout.
BatchArchiver	prepare(Supplier<IArchiveBuffer> archiveBufferSupplier, org.thales.punch.libraries.metrics.api.IMetricContext metricContext) Prepare this instance.
BatchArchiver	setAnchoredTuple(org.apache.storm.tuple.Tuple tuple) In the exactly once mode, the first tuple of each batch is kept unacknowledged until the batch is flushed.
BatchArchiver	setBloomParameters(int expectedInsertions, double fpp) Use this to make your BatchArchiver instance compute bloom filters.
long	size()
void	updateBloom(org.apache.storm.tuple.ITuple tuple, Iterable<String> bloomFilterFields) Update the bloom filter with the new fields values

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BatchArchiver

public BatchArchiver(org.thales.punch.libraries.objectstorage.indexing.BatchMetadata metaData,
                     org.thales.punch.libraries.objectstorage.tuples.ArchiveFormatSettings archiveFormatSettings)

Ctor. To create a new archiver you must provide only the very essential elements.

The BatchMetadata must only be filled with the minimum required fields, a batch id, partition id, tenant and channel name.

The ArchiveFormatSettings describes how to encode the data (csv, fields etc).

The destinations tells the archiver where to store the generate archives.

Parameters:

metaData - the meta data

archiveFormatSettings - the format encoding

Method Detail

prepare

public BatchArchiver prepare(Supplier<IArchiveBuffer> archiveBufferSupplier,
                             org.thales.punch.libraries.metrics.api.IMetricContext metricContext)

Prepare this instance.

Parameters:

archiveBufferSupplier - the supplier so that one archiveBuffer per destination can be created.

metricContext - the metric context from bolt

Returns:

the prepared instance.

archiveTuple

public void archiveTuple(org.apache.storm.tuple.Tuple tuple,
                         long epoch,
                         Iterable<String> bloomFields)

Marshal this tuple into the archive(s). This method is fail fast and will exit in case of issue.

The epoch parameter let us you provide a timestamp that will be considered as the time of generation of your tuple.

Parameters:

tuple - an incoming tuple

epoch - an associated timestamp

bloomFields - Fields used for bloom filter

flush

public String flush()

Flush your archive. This operation may or may not fire a disk or network IO. Some archivers immediately process the incoming data and directly write files to archive.

Even for those, the flush is a required operation, and triggers the generation of the meta data document that summarizes the archiving operation, and return to you with what you need : the stats, etc ..

Some archiver write the same archive to several underlying destinations. In that case a single meta data is returned. It is guaranteed to be identical for all destination archives. I.e. the same batch file has been written on all the configured destinations.

Returns:

a json representation of the metadata generated by this flush.

setBloomParameters

public BatchArchiver setBloomParameters(int expectedInsertions,
                                        double fpp)

Use this to make your BatchArchiver instance compute bloom filters.

If you call it with a zero expectedInsertions, calling this has no effect.

Parameters:

expectedInsertions - the expected number of different values

fpp - the false positive target ratio

Returns:

this instance

updateBloom

public void updateBloom(org.apache.storm.tuple.ITuple tuple,
                        Iterable<String> bloomFilterFields)

Update the bloom filter with the new fields values

Parameters:

tuple - tuple

bloomFilterFields - bloomFilterFields

hasExpired

public boolean hasExpired(long expirationTimeout,
                          long lifeTimeout)

Check if this batch has expired, given an expiration timeout. Update the expiration cause.

Parameters:

expirationTimeout - Timeout before inactivity expiration

lifeTimeout - Timeout before life expiration

Returns:

True if batch expired

getExpirationCause

public String getExpirationCause()

Getter for expiration cause

getAnchoredTuple

public org.apache.storm.tuple.Tuple getAnchoredTuple()

Getter for anchored tuple

Returns:

the tuple that is used to acknowledge this batch.

setAnchoredTuple

public BatchArchiver setAnchoredTuple(org.apache.storm.tuple.Tuple tuple)

In the exactly once mode, the first tuple of each batch is kept unacknowledged until the batch is flushed. Use this method to provide that first tuple.

Parameters:

tuple - the first tuple of each batch

Returns:

this instance

getBatchMetadata

public org.thales.punch.libraries.objectstorage.indexing.BatchMetadata getBatchMetadata()

An BatchArchiver is handling a single batch. It is associated to a BatchMetadata that stores the batch identifier and many other properties.

Returns:

the batch meta data associated to this archiver.

size

public long size()

Returns:

the number of tuples in the batch