https://pinot.apache.org/ logo
Join Slack
Powered by
Is there a way to change the kafka cluster (which ...
# troubleshooting
e
Is there a way to change the kafka cluster (which will change offsets) to realtime table? How will pinot behave if we update the table config, will it handle the different offsets or do we need to do some other config changes?
m
I suppose offsets are cluster specific? If so, may be re-bootstrap might be safer?
👍 1
e
Does rebootstrapping lose data?
Also, what is reboostrapping mean? Drop and recreate the table?
m
Yeah, drop and recreate.
If this is a hybrid table, you can just consume from offset not in offline data (as opposed to full realtime retention).
e
Is it possible to do that by setting this config:`"stream.kafka.consumer.prop.auto.offset.reset": "smallest",`?
also, thanks @Mayank!
Would the following work? disable the table, then update the config and then restart the servers, and enable table?
m
Hmm, I'd recommend clean delete/recreate
Also, before recreate ensure to wait until IS/EV is gone.
e
What's IS/EV?
m
IdealState/ExternalView
e
ah:)
m
You can temporarily reduce retention to say 2d, to only consume last two days
I think there was a config way of doing that, let me find
e
On the new table or the old table?
is that as simple as setting retention in the table config?
m
No no, talking about two different things. Let me summarize:
Copy code
1. If offsets are cluster specific - then better to delete -> recreate.
2. If you have a hybrid table, you don't really want to consume all last n days of data in realtime, since it won't be used anyway. If you want to speed this up, you can play with retention (or there is a config way) to only consume last 2 days.
e
ah, thanks! this is a realtime only table
Would recreating it allow for downloading older segments from deepstore?
m
Oh, then 2 doesn't apply
e
or can we manually just do upload download segment api call to load older realtime segments?
m
No. Moreover, if offsets are differnt, those can't be used right
Because if that worked, then you wouldn't even need to delete technically
e
Oh, so we can't just download the older segments and reupload them to the newer table?
m
No, it won't work because we won't know what offsets the old segments correspond to, in the new cluster.
btw:
Copy code
public OffsetCriteria withOffsetAsPeriod(String periodString) {
      Preconditions.checkNotNull(periodString, "Must provide period string eg. 10d, 4h30m etc");
      _offsetCriteria.setOffsetType(OffsetType.PERIOD);
      _offsetCriteria.setOffsetString(periodString);
      return _offsetCriteria;
    }
e
ah got it
m
So we do support offset as period.
👍 1
e
would there be a way via a minion job I can move realtime segments from the old table to offline segments in the new table?
s
you cannot point to a different cluster, even though the schema maybe same. segments inone cluster not equivalent to the other one. so, you need to start afresh. Yes, 
smallest'
offset will work, but be careful that you may need to allow some time to consume beforre serving queries at a high rate (cpu is needed for consuming).
e
Thanks @Subbu Subramaniam! If anything I would like to contribute to some features to enable moving segments from realtime to offline, or from one table to another - are there any open issues or some example minion jobs you can point me to?
s
I believe @Neha Pawar already has something on this?
❤️ 1
But then even if you did the move to offline segmegnts, you will need to drop the realtime table and then re-consume. Yes, you can do that for a much shorter time period
e
sounds good. Would the code for @Neha Pawar’s minion job be in the pinot repo somewhere or is it in a different repo?
Ah I see it I think: in pinot-minion-builtin-tasks?
n
this one will move from realtime to offline, for the same table, like an automatic hybrid mode
🙌 1
e
Thanks @Neha Pawar!
r
We are also looking for help on this same topic, we need to change our kafka cluster, without any data loss in pinot. @Uday Vallamsetty, @Chad
c
@Neha Pawar @Mayank Any new suggestions on this? For context, @Rahul Jain's team is planning to migrate their kafka cluster to another kafka instance in production environment. They were expecting that in Pinot, it should be a simple change to update the config with new Kafka URL, but this did not work.
m
Discussing offline
e
Hey, we did this successfully - lost the context of the thread but do you need to migrate to a different kafka cluster? lmk - I can give a summary of what we did.
m
Sure @Elon, please share your approach here.
e
Ok:)
So the basic idea is that the offsets on the new kafka cluster have to be > than the last offset for the old cluster.
To do that we produced dummy messages on the new topic and had very short retention (10seconds) until the offsets were greater, we left a buffer while the current topic is still producing, ex. 100m offsets > than latest offset on the old topic.
You can use 
kafkacat
for that or a small executable that produces empty messages
Then we modified the table config table by table and restarted the servers.
One tricky part was the flip:
once the new cluster has offsets > than old cluster you wait for those messages to drain, then increase the retention period
Start producing data to the new kafka cluster and verify that all messages from the old cluster are ingested -
we used a simple script to check the latest value of the time column and row count. If it doesn't change for 10mins (for example) then we assume that the topic is drained
We restarted the servers after updating the table config to point to the new kafka cluster/schema registry just to ensure that the realtime ingestion had fresh metadata.
Does that sound doable for you?
m
(That’s some gangsta stuff right there 😂)
🤣 1
e
it was scarey as poop🙂
even though we did it table by table - we new we were doing something fck'd up:)
but it worked - people didn't even notice
m
@Rahul Jain ^^
e
I think the fear will naturally make you careful when doing it:) Just be 100% sure that old cluster is drained and new cluster offsets are much larger than old cluster offsets.
And you can do it table by table.
s
@Sajjad Moradi can pause/resume (just checked in) help here? How will the offsets in the new topic be handled?
s
It should be possible. When all the messages from the old topic are consumed, we can pause the table. After pause finishes, the consuming segments will have the end offset of the old topic in segment ZK metadata. Now we need to change the table config to point to the new topic. Also we need to manually adjust the end offset of the latest segment for each partition in segment ZK metadata. They need to be changed to the start offset of the corresponding partition on the new topic.
Does that make sense?
s
yes, but doing this manually is not possible. A better way may be to do two things: (1) Provide an option to resume from the latest. This will work if the new topic is not populated yet. It may not work for everyone. (2) Provide a way to incidicate new topic (or auto-recognize) and start from earliest in the new topic. Not sure how auto-recognize unless we keep pause history in znodes.
I think you should go over each of those issues that are in the open source, and add a note on how pause/resume solves them (or, what else is needed, and create issues for those).
s
You're right. What I said was the process with no addition to the current code and I think the manual work (only setting the offsets) is much less than the process that Elon described earlier.
I'll go over the issues and update on how pause/resume can address the issues, probably by next week.
e
Sounds good, I would do it this way in the future:) By pause/resume do you mean enable/disable the table?
s
No, the table is enabled and the queries can be executed, we just pause the stream consumption. Here are the endpoints that were added after the PR got merged yesterday:
image.png
🚀 1
e
that's amazing! this will save years off my life if we ever have to do it again:)
😄 2
s
Glad to hear that. As Subbu mentioned, this feature will address several issues that are tagged in the PR. If you have some suggestion on how we can improve this, you can put comments there. Here is the PR: https://github.com/apache/pinot/pull/8986
e
Thanks! I'll take a look. This is great stuff!
thankyou 1
Open in Slack
PreviousNext
Powered by