https://flink.apache.org/ logo
Join Slack
Powered by
Hello all, I am using Redpanda with Flink for stre...
# troubleshooting
k
Hello all, I am using Redpanda with Flink for streaming messages and processing them, I am giving group id but it does not seem to be taking it, any one know what is am doing wrong?
Copy code
KafkaSource<Map<String, Object>> logSource = KafkaSource.<Map<String, Object>>builder()
                .setBootstrapServers(BOOTSTRAP_SERVER)
                .setTopics(SOURCE_TOPIC)
                .setGroupId("group2")
                .setStartingOffsets(OffsetsInitializer.latest())
                .setValueOnlyDeserializer(new LogDeserializer())
                .build();
also try to list groups using 
rpk group list
, but that didn't show any groups
s
Do you actually commit the offsets? Check 
commit.offsets.on.checkpoint
, should be set to 
true
. Plus, checkpointing must be enabled.
k
So it is required to enable checkpointing to use consuer groups?
s
Yep! If you want the offsets to be committed. The group was likely created, you just don’t see it without any committed offsets. Or you can enable committing via 
enable.auto.commit
, but it’s not recommended.
k
I tried setting 
<http://auto.commit.interval.ms|auto.commit.interval.ms>
and 
enable.auto.commit
the redpanda ui shows the group but its status ia empty
Ok got it, thats why the lag value of the group change whenever their are any messages being pushed to the topic, but shows empty
I tried enabling checkpointing and setting 
commit.offsets.on.checkpoint
to true but after that the topic does not show any events.
Copy code
KafkaSource<Map<String, Object>> logSource = KafkaSource.<Map<String, Object>>builder()
                .setBootstrapServers("10.50.1.199:9092")
                .setTopics("raw_logs")
                .setProperty("group.id", "krutik1")
                .setProperty("commit.offsets.on.checkpoint", "true")
                .setStartingOffsets(OffsetsInitializer.latest())
                .setValueOnlyDeserializer(new LogDeserializer())
                .build();
now this does creates a group but the state of the group shows 
Empty
Note: there is a delay between the job start and group showing in Redpanda UI and also it is note consistent some times the group shows up after the job is canceled And also if two consumer in the same group consuming from same topic each consumer should consume unique records, for e.g. if I send 10 events in topic X and the two consumer are in same group consuming from that topic then each should not receive same event, But that is not happening in this case
Can anyone help on my above query?
a
I suggest you read this doc, Kafka source does not rely on committed offsets for fault tolerance. Committing offset is only for exposing the progress of consumer and consuming group for monitoring. From what I understand your assumption that Flink would rely on the consumer group to distribute the work between the sinks doesn't hold true, more details are in this doc. Flink has it own Partition discovery and assignment strategy.
2 Views
Open in Slack
PreviousNext
Powered by