org.thales.punch.kafka.platform

Class KafkaShivaUtil

java.lang.Object

org.thales.punch.kafka.platform.KafkaShivaUtil

public class KafkaShivaUtil
extends Object

This class groups the various values that are part of the Punch protocol implemented on top of Kafka used between the command line client or API, and the shiva daemon.

This protocol is used by the core punch administration to keep track of channels and applications status, as well as by the shiva application scheduler.

Why an extra layer ? simply to enforce the important properties required for the protocol to work correctly : auto commit, last commit offset strategy, key and value serializer etc ..

Author:

dimi

Field Summary

Fields

Modifier and Type	Field and Description
static int	MAX_RECORD_LENGTH Max default kafka record size is 1Mb.

Method Summary

Modifier and Type	Method and Description
static String	getCommandRecordKey(String appName)
static byte[]	getCompleteV2Record(String key, byte[] chunk, Map<String,List<byte[]>> map) This method receives the sequence of chunk from the same application start command, and aggregate them all.
static KafkaContinuousConsumerBuilder<String,byte[]>	getContinuousConsumerBuilder(String bootstrapServers, String groupId, String... topics) This is a light helper to create continuous consumers well-configured for the punch internal kafka admin and shiva protocol.
static org.thales.punch.libraries.utils.api.UtilZip.Visitor	getDefaultVisitor(String tenantName, List<String> whitelist, List<String> blacklist) To start a shiva application, a zip archive is sent to each worker.
static org.apache.kafka.clients.producer.KafkaProducer<String,byte[]>	getProducer(String bootstrapServers, String clientId) Get a simple Kafka producer.
static org.apache.kafka.clients.producer.KafkaProducer<String,byte[]>	getProducer(String bootstrapServers, String clientId, org.thales.punch.platform.api.pojo.JavaStoreSecurity securityStores) Get a simple Kafka producer.
static String	getV2CommandRecordKey(String appName, int seqNum, int lastNum) This one is used to send too big payloads.
static org.thales.punch.settings.api.ISettingsMap	getWorkerDeclarationMap(String workerId, List<String> tags) Return a json document that represents the declaration of a new worker.
static void	logRcv(String who, org.apache.kafka.clients.consumer.ConsumerRecord<String,byte[]> record)
static org.apache.kafka.clients.producer.RecordMetadata	send(String who, org.apache.kafka.clients.producer.KafkaProducer<String,byte[]> producer, org.apache.kafka.clients.producer.ProducerRecord<String,byte[]> record) Send and trace a gossip message.
static void	writeAdminMap(String topic, org.apache.kafka.clients.producer.KafkaProducer<String,byte[]> producer, org.thales.punch.settings.api.ISettingsMap map) Write a settings map to the administration topics.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_RECORD_LENGTH

public static final int MAX_RECORD_LENGTH

Max default kafka record size is 1Mb. Stay below.

See Also:

Constant Field Values

Method Detail

getDefaultVisitor

public static org.thales.punch.libraries.utils.api.UtilZip.Visitor getDefaultVisitor(String tenantName,
                                                                                     List<String> whitelist,
                                                                                     List<String> blacklist)

To start a shiva application, a zip archive is sent to each worker. This method returns the default file visitor that select the simplest configuration required for most tenants and platform configuration.

Every time a shiva application is started, the archive sent simply contains :

$PUNCHPLATFORM_CONF_DIR/punchplatform.properties

$PUNCHPLATFORM_CONF_DIR/resolv.hjson

$PUNCHPLATFORM_CONF_DIR/tenants/mytenant

This basically lets shiva worker launch correctly every required application.

To have better control you can define black and whitelist in the tenant configuration file. These will help you select what you need using regexp to exclud or include only some files.

Parameters:

blacklist -

whitelist -

Returns:

a visitor to select the right file.

getWorkerDeclarationMap

public static org.thales.punch.settings.api.ISettingsMap getWorkerDeclarationMap(String workerId,
                                                                                 List<String> tags)

Return a json document that represents the declaration of a new worker.

Parameters:

workerId - the worker unique identifier

tags - the worker tags

Returns:

the corresponding settings map

logRcv

public static void logRcv(String who,
                          org.apache.kafka.clients.consumer.ConsumerRecord<String,byte[]> record)

getCommandRecordKey

public static String getCommandRecordKey(String appName)

getV2CommandRecordKey

public static String getV2CommandRecordKey(String appName,
                                           int seqNum,
                                           int lastNum)

This one is used to send too big payloads.

Parameters:

appName - the application name

seqNum - the current sequence number

lastNum - the last sequence number

Returns:

a V2 header

send

public static org.apache.kafka.clients.producer.RecordMetadata send(String who,
                                                                    org.apache.kafka.clients.producer.KafkaProducer<String,byte[]> producer,
                                                                    org.apache.kafka.clients.producer.ProducerRecord<String,byte[]> record)

Send and trace a gossip message.

Parameters:

who - leader or worker, for tracing only

producer - the producer to use

record - the record to send

Returns:

the record metadata

getProducer

public static org.apache.kafka.clients.producer.KafkaProducer<String,byte[]> getProducer(String bootstrapServers,
                                                                                         String clientId)

Get a simple Kafka producer. Make sure you end up closing it.

Parameters:

bootstrapServers - the bootstrap.servers address

clientId - the client id

Returns:

the producer.

getProducer

public static org.apache.kafka.clients.producer.KafkaProducer<String,byte[]> getProducer(String bootstrapServers,
                                                                                         String clientId,
                                                                                         org.thales.punch.platform.api.pojo.JavaStoreSecurity securityStores)

Get a simple Kafka producer. Make sure you end up closing it.

Parameters:

bootstrapServers - the bootstrap.servers address

clientId - the client id

securityStores - the Object containing the truststore and keystore information. If not null, activate ssl.

Returns:

the producer.

getContinuousConsumerBuilder

public static KafkaContinuousConsumerBuilder<String,byte[]> getContinuousConsumerBuilder(String bootstrapServers,
                                                                                         String groupId,
                                                                                         String... topics)

This is a light helper to create continuous consumers well-configured for the punch internal kafka admin and shiva protocol.

It only sets the common characteristics, and enforce a consistent set of properties shared among all participant : the clients and the shiva daemons.

Parameters:

bootstrapServers - the kafka bootstrap server property

groupId - the groupId to use

topics - the topic(s) to listen to

Returns:

a builder

writeAdminMap

public static void writeAdminMap(String topic,
                                 org.apache.kafka.clients.producer.KafkaProducer<String,byte[]> producer,
                                 org.thales.punch.settings.api.ISettingsMap map)

Write a settings map to the administration topics.

Parameters:

producer - the admin producer

map - the settings m ap

getCompleteV2Record

public static byte[] getCompleteV2Record(String key,
                                         byte[] chunk,
                                         Map<String,List<byte[]>> map)

This method receives the sequence of chunk from the same application start command, and aggregate them all. When the last one is received it is returned as a complete byte array. If not yet complete null is returned.

Parameters:

key - the application key

chunk - the new chunk

map - the caller context map where chunks are stored using the application key.

Returns:

null or the complete byte array