Is there a behavior change between 0.9.0 and 0.10....
# troubleshootinga
Ashish
07/13/2022, 7:22 PMIs there a behavior change between 0.9.0 and 0.10.0 related to kafka client? In 0.9.0, we should see pinot committing the offset but 0.10.0, pinot never commits the offset to kafka and consumer lag keeps growing.
Ashish
07/13/2022, 7:33 PMpinot-server-realtime-0 server 2022/07/13 175556.463 INFO [AppInfoParser] [<segment name>] App info kafka.consumer for consumer-null-267 unregisteredserver-realtime-0 server 2022/07/13 175556.463 INFO [LLRealtimeSegmentDataManager_<segment name>] [segment name] Creating new stream consumer for topic partition <topic-partition> , reason: Idle for too <segment name>] ConsumerConfig values:
pinot-server-realtime-0 server allow.auto.create.topics = true
pinot-server-realtime-0 server auto.commit.interval.ms = 5000
pinot-server-realtime-0 server auto.offset.reset = latest
Ashish
07/13/2022, 7:34 PMpinot-server-realtime-0 server allow.auto.create.topics = true
pinot-server-realtime-0 server auto.commit.interval.ms = 5000
pinot-server-realtime-0 server auto.offset.reset = latest
pinot-server-realtime-0 server bootstrap.servers = [kafka:9092]
pinot-server-realtime-0 server check.crcs = true
pinot-server-realtime-0 server client.dns.lookup = use_all_dns_ips
pinot-server-realtime-0 server client.id = consumer-null-268
pinot-server-realtime-0 server client.rack =
pinot-server-realtime-0 server connections.max.idle.ms = 540000
pinot-server-realtime-0 server default.api.timeout.ms = 60000
pinot-server-realtime-0 server enable.auto.commit = true
Ashish
07/13/2022, 7:34 PMSo the consumer config values printed by pinot shows auto commit enabled with interval of 5 seconds. But it does not seem the auto commit is happening.
m
Mayank
07/13/2022, 7:39 PM
@Neha Pawar ^^
l
Laxman Ch
07/14/2022, 7:15 AM@Ashish: try setting
group.id""group.idnon-empty/non-nulla
Ashish
07/14/2022, 3:51 PMThanks, also found the same thing yesterday going over kafka release notes. Is there a plan to fix this in the code? Otherwise, every table configuration needs to have this extra settings.
m
Mayank
07/14/2022, 3:52 PM
It should be an easy fix. Care to take a shot at it? cc: @Neha Pawar
l
Laxman Ch
07/14/2022, 4:11 PMThis is a behavior change introduced by kafka version upgrade. Not sure if its correct to fix in Pinot.
if we really want to fix it in pinot, we need to set default group id when it was not explicitly configured by user (say, table name as the default group id)
pinot never commits the offset to kafka and consumer lag keeps growing.There is no lag really. This is just monitoring issue. Pinot uses low-level-consumer and manages the offsets and checkpoints on its own. It also commits the offset in kafka just for kafka monitoring tools to work. setting
group.id