Hello all. I’m trying to figure out how to migrate...
# general
n
Hello all. I’m trying to figure out how to migrate a supervisor from one Kafka topic to another. I tried updating the supervisor to read from both topics, using a flag on the Kafka payload to signal to the indexer about which topic to use, but when the indexer tries to do the handoff I got the “Inconsistent metadata state” state. The only way I’ve seen to fix this is to reset the supervisor, but that will lead to me processing the same Kafka payloads multiple times. Is there a way to change a supervisor’s topic without resetting it? Alternatively is there any other way to migrate to another topic without the risk of double-processing the Kafka payloads?
i should note that both the old and the new topics are used by other consumers (including other druid datasources), so draining the messages on the topic is not really an option
j
Have you tried creating the new supervisor in suspended mode, and then using the "set offsets" panel to manually set the offsets? It's a manual process but in theory should allow you to start the new supervisor exactly where you want in the stream.
n
thanks, i think that will work. I didn’t think there was a way to get the latest offsets after suspending a supervisor but i can see now that the offsets are shown in the supervisor status after the last publish task finishes
👍 1
j
Fyi I haven't manually set offsets before, so please report back to let us know how this worked for you!
n
This is working, but a bit harder than i would like. Once a supervisor has been suspended, I can only get the latest offsets and the lag, and not the “last processed offset”. So my process has been to copy-paste those two numbers into a spreadsheet and subtract the lag from the latest offset to get the “last processed”, then i copy that number into the reset offsets dialog. With 64 partitions it’s a bit of a pain, but probably not enough of a pain for me to bother scripting up a solution since i’ll only have to do this a few more times.