I am looking for guide to use schema versioning. C...
# generals
sharofiddin
03/27/2024, 5:37 AMI am looking for guide to use schema versioning. Can consumer listen only messages with the same version it created? Suppose we have to consumers for one topic, first consumer (C1) processes messages with version V1 ans second (C2) processes V2, but I don't know how to configure C1 to listen only messages with schema version V1 and C2 with V2, is it possible?
👀 1
k
Kiryl Valkovich
03/27/2024, 5:44 AMEach message has a schema version it was produced with, and you can filter messages by this version.
This way each consumer can listen to all messages, but only process the ones it understands.
It’s probably the simplest solution. Would it work for you?
s
sharofiddin
03/27/2024, 5:47 AMYes it works, but when we update pulsar version or delete and recreate the topic, we have to restart applications with new schema versions in this case.
k
Kiryl Valkovich
03/27/2024, 5:48 AMWhat about dynamically configuring each consumer to set the schema version it should process?
Kiryl Valkovich
03/27/2024, 5:58 AMIn theory, if you don’t know the exact schema version number, you can hash the schema, and on the consumer side filter the proper version by hash to get the version number you can then use for filtering.
This way I see two options:
• Either someone who knows the proper schema number dynamically configures consumers.
• Either consumer knows the schema hash and determines the proper schema number.
Kiryl Valkovich
03/27/2024, 5:58 AMMaybe someone can suggest a better solution 🙂
s
sharofiddin
03/27/2024, 6:04 AMSuppose we dynamically configured consumer to consume schema version V3 , after awhile we upgraded pulsar version an recreated the topics which our consumer listens, and we don't need old versions of schema, so it starts with schema version V1 in new pulsar and we should change dynamically configured consumer's version. We can use new topic for each version but I think if we solve problem with schema versioning it will be more elegant. I am looking for pattern which migrations will be smoothly, sometimes we need to migrate our services gradually in this case both old and new schema versions should work seamless.
k
Kiryl Valkovich
03/27/2024, 6:05 AMwe upgraded pulsar version an recreated the topics which our consumer listensWhy would you recreate the topics after the Pulsar upgrade?
s
sharofiddin
03/27/2024, 6:08 AMYes, because new pulsar doesn't have this topic.
k
Kiryl Valkovich
03/27/2024, 6:09 AMSeems you mean not upgrade, but rather a fresh start without the old data.
s
sharofiddin
03/27/2024, 6:11 AMso far we are upgrading with this way 😞
k
Kiryl Valkovich
03/27/2024, 6:13 AMYeah, but why not use the option with the schema hash I suggested?
Calculate the hash, then configure the consumer to use this schema hash.
On consumer start, get all schemas by using the Pulsar Admin API and find the version number you need by hash.
🙌 1
s
sharofiddin
03/27/2024, 6:14 AMThank you
k
Kiryl Valkovich
03/27/2024, 6:15 AMYou’re welcome. Let me know if it works for you. 🙂