https://pinot.apache.org/ logo
Join Slack
Powered by
Hi Folks, Does the Kinesis consumer handles shard ...
# troubleshooting
a
Hi Folks, Does the Kinesis consumer handles shard splitting? I have a realtime table which has stopped consuming events. Looking at the logs all the realtime segments are completed. There are no consuming segments. It correlates to shard split in Kinesis. It seems Pinot consumers stopped consuming after the shard is split. There are no error logs. Is there some manual step that is required if shard split happens?
k
Hi Aditya, The current release has a bug where new shards do not transition to consuming state. We have already raised PR to fix this issue - https://github.com/apache/pinot/pull/8053 https://github.com/apache/pinot/pull/7058
a
Thanks for fixing this. Is there any solution to resume consuming events for now? or shall we delete and recreate the relatime table, then I think it'll consume from all the available segments
k
Recreating is the only option for now. Triggering the validation manager also doesn't work because of this issue.
a
Got it, Thanks
@Kartik Khare Tried recreating table. The consumers started for shard-0,1,2 and 3. After consuming complete shard, segments were committed. Then the consumer stopped. It seems Kinesis consumer can only work for 4 default shards. I thought after recreating table the servers will first try to get all available shards (12 in my case)
k
That's not possibel. I have tested with 10 shards myself.
n
Recreating won't work. For kinesis we always consume shards in order, and only after parents close
You'll need the new patch
Merging it now, is it possible for you to take the latest master?
a
@Kartik Khare the stream I am working on had 4 initial shards which then further got split in 8 shards. Did you had 10 initial shards while testing? @Neha Pawar Do I need to build the master? I am using the docker image, is there a nightly docker image built on master?
n
@Xiang Fu what is the frequency of our docker image build pipeline? Is it for every commit or nightly? Can we build one adhoc right now?
a
@Neha Pawar No worries, I can download nightly image build tomorrow. I am working on test data, it's not critical. Thanks
👍 1
@Neha Pawar I have rebuilt server, broker and controller with the new docker image (digest a6c14285abf4). Deleted and recreated the realtime table. The issue persists, servers consumed from the initial shards 0,1,2,3 and committed the segments Consumers didn't resumed reading from their child shards. Shall I clear the zookeeper data too and start fresh?
n
let’s check with @Xiang Fu, hey Xiang would the latest docker image have this yet? was merged this morning: https://github.com/apache/pinot/pull/8053
not sure when the job runs
x
a6c14285abf4 should have the your change
image.png
n
cool
@Aditya, the new shards are detected by a periodic job. So once initial 4 shards reach end of life, the new shards should get detected by the periodic job (called RealtimeSegmentValidationManager). It runs hourly. You can check for the logs in controller
how does the ideal state look right now?
a
@Neha Pawar just checked the table. The new shards are detected! almost exactly 1 hr after the parent shards segments were committed Can we specify the period for RealtimeSegmentValidationManager? Is it possible to start consuming child shards immediately after finishing consuming the parent shard? (This may not be quick fix, if we get a shard iterator of type LATEST on parent shard it returns the child shard ids and offset)
n
It is possible to make the periodic job run faster. Will share config in a bit
And Kartik is working on making new shards detect immediately when the old shard reaches end of life
a
Thanks a lot!
n
controller.realtime.segment.validation.frequencyPeriod=5m
set this (or any frequency of your choice) into your controller.conf/controller configMap.
this is the page on the docs about it (thought not very verbose 🤦‍♀️ ) https://docs.pinot.apache.org/configuration-reference/controller#periodic-tasks-configuration
@Mark Needham would you be able to help with this doc? adding more description to the periodic task configs ans some example values to configure
👀 1
a
Thanks Is it possible to trigger RealtimeSegmentValidationManager from a swagger endpoint? Thinking from an operations point as it would not require controller restarts (which I believe is needed after updating conf)
n
Yes, look for Tasks tab in swagger
a
Tried GET /tasks/scheduler/jobKey Getting error response
Copy code
{
  "code": 404,
  "error": "Task scheduler is disabled"
}
Let me try figure this out
n
sorry wrong tab
it’s PeriodicTask
the one you tried is for minion tasks
a
found the tab, Thanks
Open in Slack
PreviousNext
Powered by