Skip to content

punchplatform-kafka-consumers.sh

NAME

punchplatform-kafka-consumers.sh

Kafka debug commands

SYNOPSIS

punchplatform-kafka-consumers.sh COMMAND [--kafkaCluster <kafkaClusterId> ] [OPTIONS of kafka-consumer-groups.sh]

DESCRIPTION

punchplatform-kafka-consumers.sh is a wrapper for kafka_2.11-1.1.0/bin/kafka-consumer-groups.sh.

The native Kafka command is used to list all consumer groups, describe a consumer group, delete consumer group info, or reset consumer group offsets.

The punchplatform-kafka-consumers.sh wrapper helps by:

  • providing the punchplatform Kafka cluster actual connection parameters to this native Kafka command,
  • providing shortcut commands

The native command help is displayed along Punchplatform wrapper addition when -h is invoked.

EXAMPLES

  • List the consumer groups that have committed their offsets in Kafka:

    punchplatform-kafka-consumers.sh --list --kafkaCluster myClusterIdBecauseIHaveSeveral
    

  • Display the current committed offset for the consumers group and their LAG (i.e. backlog of non-yet-processed messages:

    punchplatform-kafka-consumers.sh --describe --group my.consumers.group.name
    

  • Reset the consumers offsets at the beginning of the topic. The reset will be applied by the consumers when they start. Can only be called when the consumers group is NOT RUNNING:

    punchplatform-kafka-consumers.sh --reset-to-earliest --group my.consumers.group.name
    

  • Shift current position by -100000 offsets. This will cause replay of the last 100000 processed records. Can only be called when the consumers group is NOT RUNNING:

    punchplatform-kafka-consumers.sh --shift-offsets -100000 --group my.consumers.group.name
    

OPTIONS

Kafka-native options

All these options will be passed through the punchplatform-kafka-consumers.sh wrapper to kafka-consumers.sh:

  • --all-topics

    Consider all topics assigned to a group in the reset-offsets process.

  • --bootstrap-server

    REQUIRED (for consumer groups based on the new consumer)
    The server to connect to.

  • --by-duration

    Reset offsets to offset by duration from current timestamp.
    Format: PnDTnHnMnS

  • --command-config

    Property file containing configs to be passed to Admin Client and Consumer.

  • --delete

    Pass in groups to delete topic partition offsets and ownership information over the entire consumer group. For instance --group g1 --group g2.
    Pass in groups with a single topic to just delete the given topic's partition offsets and ownership information for the given consumer groups. For instance --group g1 --group g2 --topic t1.
    Pass in just a topic to delete the given topic's partition offsets and ownership information for every consumer group. For instance --topic t1.
    WARNING: Group deletion only works for old ZK-based consumer groups, and one has to use it carefully to only
    delete groups that are not active.

  • --describe

    Describe consumer group and list offset lag (number of messages not yet processed) related to given group.

  • --dry-run

    Only show results without executing changes on Consumer Groups.
    Supported operations: reset-offsets.

  • --execute

    Execute operation.
    Supported operations: reset-offsets.

  • --export

    Export operation execution to a CSV file.
    Supported operations: reset-offsets.

  • --from-file

    Reset offsets to values defined in CSV file.

  • --group

    The consumer group we wish to act on.

  • --list

    List all consumer groups.

  • --members

    Describe members of the group. This option may be used with --describe and --bootstrap-server options only.
    Example: --bootstrap-server localhost:9092 --describe --group group1 --members

  • --new-consumer

    Use the new consumer implementation.
    This is the default, so this option is deprecated and will be removed in a future release.

  • --offsets

    Describe the group and list all topic
    partitions in the group along with their offset lag.
    This is the default sub-action of and may be used with --describe and --bootstrap-server options only.
    Example: --bootstrap-server localhost:9092 --describe --group group1 --offsets

  • --reset-offsets

    Reset offsets of consumer group.
    Supports one consumer group at the time, and instances should be inactive. Has 2 execution options: --dry-run (the default) to plan which offsets to reset, and --execute to update the offsets. Additionally, the --export option is used to export the results to a CSV format. You must choose one of the following reset specifications: --to-datetime, --by-period, --to-earliest, --to-latest, --shift-by, --from-file, --to-current.
    To define the scope use --all-topics or --topic. One scope must be specified unless you use --from-file.

  • --shift-by

    Reset offsets shifting current offset by n, where n can be positive or negative.

  • --state

    Describe the group state. This option may be used with --describe and --bootstrap-server options only.
    Example: --bootstrap-server localhost:9092 --describe --group group1 --state

  • --timeout

    The timeout that can be set for some use cases. For example, it can be used when describing the group to specify the maximum amount of time in milliseconds to wait before the group stabilizes (when the group is just created, or is going through some changes). (default: 5000)

  • --to-current

    Reset offsets to current offset.

  • --to-datetime

    Reset offsets to offset from datetime.
    Format: YYYY-MM-DDTHH:mm:SS.sss

  • --to-earliest

    Reset offsets to earliest offset.

  • --to-latest

    Reset offsets to latest offset.

  • --to-offset

    Reset offsets to a specific offset.

  • --topic

    The topic whose consumer group information should be deleted or topic whose should be included in the reset offset process. In reset-offsets case, partitions can be specified using this format: topic1:0,1,2, where 0,1,2 are the partition to be included in the process. Reset-offsets also supports multiple topic inputs.

  • --verbose

    Provide additional information, if any, when describing the group. This option may be used with --offsets/--members/--state and --bootstrap-server options only.
    Example: --bootstrap-server localhost:9092 --describe --group group1 --members --verbose

  • --zookeeper

    REQUIRED (for consumer groups based on the old consumer)
    The connection string for the zookeeper connection in the form host:port.
    Multiple URLS can be given to allow fail-over.

PunchPlatform options

These PunchPlatform additional commands provide shortcuts and connection parameters to Kafka-native command.

  • --kafkaCluster

    Compute Zookeeper parameters for the provided Kafka cluster. The KafkaClusterId to provide is the same as in $PUNCHPLATFORM_OPERATOR_INSTALL_DIR/punchplatform.properties.

  • --reset-to-(earliest|latest)

    This is a shortcut for native Kafka command args --reset-offsets --to-earliest/--to-latest --all-topics --execute. You must provide --group <consumerGroupName>

  • --shift-offsets

    This is a shortcut for native Kafka command args --reset-offsets --shift-by <number> --all-topics --execute You must provide --group <consumerGroupName>

FILES

  • punchplatform.properties:

    This json file defines the punchplatform settings.
    It is used by punchplatform-kafka-topics.sh primarily to know the broker list.

SEE ALSO