org.thales.punch.libraries.storm.bolt.file.buffer

Class AvroParquetArchiveBuffer

java.lang.Object

org.thales.punch.libraries.storm.bolt.file.buffer.AbstractArchiveBuffer

org.thales.punch.libraries.storm.bolt.file.buffer.AvroParquetArchiveBuffer

All Implemented Interfaces:

IArchiveBuffer

public class AvroParquetArchiveBuffer
extends AbstractArchiveBuffer

The AvroParquetArchiveBuffer writes its content into S3 cluster or Filesystem.

Author:

dimi

Field Summary

Fields inherited from class org.thales.punch.libraries.storm.bolt.file.buffer.AbstractArchiveBuffer

destinations, filename, maxWrites, minWrites, pool

Constructor Summary

Constructors

Constructor and Description
AvroParquetArchiveBuffer(int minWrites, int maxWrites, List<org.thales.punch.libraries.objectstorage.tuples.ArchiveDevice> destinations, org.thales.punch.libraries.record.client.builder.api.IRecordWriterBuilder<Map<String,Object>> builder, boolean createRoot)

Method Summary

Modifier and Type	Method and Description
void	close(List<String> successAddresses) Close this archive buffer.
String	getDigest() Return the digest of written bytes
long	getSize() Return the size, compressed if compression enabled, of written bytes
long	getUncompressedSize() Return the size, uncompressed if compression enabled, of written bytes
IArchiveBuffer	prepare(org.thales.punch.libraries.objectstorage.indexing.BatchMetadata metaData) Before using the archive buffer call this.
void	put(Map<String,Object> record) Add a single item in the archive buffer.

Methods inherited from class org.thales.punch.libraries.storm.bolt.file.buffer.AbstractArchiveBuffer

exit, getBatchOutputPath, handleDeviceFailure

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AvroParquetArchiveBuffer

public AvroParquetArchiveBuffer(int minWrites,
                                int maxWrites,
                                List<org.thales.punch.libraries.objectstorage.tuples.ArchiveDevice> destinations,
                                org.thales.punch.libraries.record.client.builder.api.IRecordWriterBuilder<Map<String,Object>> builder,
                                boolean createRoot)

Method Detail

prepare

public IArchiveBuffer prepare(org.thales.punch.libraries.objectstorage.indexing.BatchMetadata metaData)

Description copied from interface: IArchiveBuffer

Before using the archive buffer call this. It allows some implementation to complete the preparation, open files etc.

Specified by:

prepare in interface IArchiveBuffer

Overrides:

prepare in class AbstractArchiveBuffer

Parameters:

metaData - the provided context information.

Returns:

this instance

getSize

public long getSize()

Description copied from interface: IArchiveBuffer

Return the size, compressed if compression enabled, of written bytes

Returns:

compressed size

getUncompressedSize

public long getUncompressedSize()

Description copied from interface: IArchiveBuffer

Return the size, uncompressed if compression enabled, of written bytes

Returns:

uncompressed size

put

public void put(Map<String,Object> record)

Description copied from interface: IArchiveBuffer

Add a single item in the archive buffer. Watch out here an item means a csv line or one column.

Parameters:

record - the input record

close

public void close(List<String> successAddresses)

Description copied from interface: IArchiveBuffer

Close this archive buffer. You cannot use it anymore after calling this. You of course MUST call this once done writing to an archive.

Parameters:

successAddresses - in case of success the returned list of where the writes were successful.

getDigest

public String getDigest()

Description copied from interface: IArchiveBuffer

Return the digest of written bytes

Returns:

SHA256 Digest