punchplatform-kafka-consumers.sh

NAME¶

punchplatform-kafka-consumers.sh

Kafka debug commands

SYNOPSIS¶

punchplatform-kafka-consumers.sh COMMAND [--kafkaCluster <kafkaClusterId> ] [OPTIONS of kafka-consumer-groups.sh]

DESCRIPTION¶

punchplatform-kafka-consumers.sh is a wrapper for kafka_2.11-1.1.0/bin/kafka-consumer-groups.sh.

The native Kafka command is used to list all consumer groups, describe a consumer group, delete consumer group info, or reset consumer group offsets.

The punchplatform-kafka-consumers.sh wrapper helps by:

providing the punchplatform Kafka cluster actual connection parameters to this native Kafka command,
providing shortcut commands

The native command help is displayed along Punchplatform wrapper addition when -h is invoked.

EXAMPLES¶

List the consumer groups that have committed their offsets in Kafka:

punchplatform-kafka-consumers.sh --list --kafkaCluster myClusterIdBecauseIHaveSeveral

Display the current committed offset for the consumers group and their LAG (i.e. backlog of non-yet-processed messages:
```
punchplatform-kafka-consumers.sh --describe --group my.consumers.group.name
```
Reset the consumers offsets at the beginning of the topic. The reset will be applied by the consumers when they start. Can only be called when the consumers group is NOT RUNNING:
```
punchplatform-kafka-consumers.sh --reset-to-earliest --group my.consumers.group.name
```
Shift current position by -100000 offsets. This will cause replay of the last 100000 processed records. Can only be called when the consumers group is NOT RUNNING:
```
punchplatform-kafka-consumers.sh --shift-offsets -100000 --group my.consumers.group.name
```

OPTIONS¶

Kafka-native options¶

All these options will be passed through the punchplatform-kafka-consumers.sh wrapper to kafka-consumers.sh:

--all-topics

Consider all topics assigned to a group in the reset-offsets process.
--bootstrap-server

REQUIRED (for consumer groups based on the new consumer)
The server to connect to.
--by-duration

Reset offsets to offset by duration from current timestamp.
Format: PnDTnHnMnS
--command-config

Property file containing configs to be passed to Admin Client and Consumer.
--delete

Pass in groups to delete topic partition offsets and ownership information over the entire consumer group. For instance --group g1 --group g2.
Pass in groups with a single topic to just delete the given topic's partition offsets and ownership information for the given consumer groups. For instance --group g1 --group g2 --topic t1.
Pass in just a topic to delete the given topic's partition offsets and ownership information for every consumer group. For instance --topic t1.
WARNING: Group deletion only works for old ZK-based consumer groups, and one has to use it carefully to only
delete groups that are not active.
--describe

Describe consumer group and list offset lag (number of messages not yet processed) related to given group.
--dry-run

Only show results without executing changes on Consumer Groups.
Supported operations: reset-offsets.
--execute

Execute operation.
Supported operations: reset-offsets.
--export

Export operation execution to a CSV file.
Supported operations: reset-offsets.
--from-file

Reset offsets to values defined in CSV file.
--group

The consumer group we wish to act on.
--list

List all consumer groups.
--members

Describe members of the group. This option may be used with --describe and --bootstrap-server options only.
Example: --bootstrap-server localhost:9092 --describe --group group1 --members
--new-consumer

Use the new consumer implementation.
This is the default, so this option is deprecated and will be removed in a future release.
--offsets

Describe the group and list all topic
partitions in the group along with their offset lag.
This is the default sub-action of and may be used with --describe and --bootstrap-server options only.
Example: --bootstrap-server localhost:9092 --describe --group group1 --offsets
--reset-offsets

Reset offsets of consumer group.
Supports one consumer group at the time, and instances should be inactive. Has 2 execution options: --dry-run (the default) to plan which offsets to reset, and --execute to update the offsets. Additionally, the --export option is used to export the results to a CSV format. You must choose one of the following reset specifications: --to-datetime, --by-period, --to-earliest, --to-latest, --shift-by, --from-file, --to-current.
To define the scope use --all-topics or --topic. One scope must be specified unless you use --from-file.
--shift-by

Reset offsets shifting current offset by n, where n can be positive or negative.
--state

Describe the group state. This option may be used with --describe and --bootstrap-server options only.
Example: --bootstrap-server localhost:9092 --describe --group group1 --state
--timeout

The timeout that can be set for some use cases. For example, it can be used when describing the group to specify the maximum amount of time in milliseconds to wait before the group stabilizes (when the group is just created, or is going through some changes). (default: 5000)
--to-current

Reset offsets to current offset.
--to-datetime

Reset offsets to offset from datetime.
Format: YYYY-MM-DDTHH:mm:SS.sss
--to-earliest

Reset offsets to earliest offset.
--to-latest

Reset offsets to latest offset.
--to-offset

Reset offsets to a specific offset.
--topic

The topic whose consumer group information should be deleted or topic whose should be included in the reset offset process. In reset-offsets case, partitions can be specified using this format: topic1:0,1,2, where 0,1,2 are the partition to be included in the process. Reset-offsets also supports multiple topic inputs.
--verbose

Provide additional information, if any, when describing the group. This option may be used with --offsets/--members/--state and --bootstrap-server options only.
Example: --bootstrap-server localhost:9092 --describe --group group1 --members --verbose
--zookeeper

REQUIRED (for consumer groups based on the old consumer)
The connection string for the zookeeper connection in the form host:port.
Multiple URLS can be given to allow fail-over.

PunchPlatform options¶

These PunchPlatform additional commands provide shortcuts and connection parameters to Kafka-native command.

--kafkaCluster

Compute Zookeeper parameters for the provided Kafka cluster. The KafkaClusterId to provide is the same as in $PUNCHPLATFORM_OPERATOR_INSTALL_DIR/punchplatform.properties.
--reset-to-(earliest|latest)

This is a shortcut for native Kafka command args --reset-offsets --to-earliest/--to-latest --all-topics --execute. You must provide --group <consumerGroupName>
--shift-offsets

This is a shortcut for native Kafka command args --reset-offsets --shift-by <number> --all-topics --execute You must provide --group <consumerGroupName>

FILES¶

punchplatform.properties:

This json file defines the punchplatform settings.
It is used by punchplatform-kafka-topics.sh primarily to know the broker list.