Hi team, I want to ask is datahub maintain schema versioning?

Hey Anung 👋 could you elaborate on your question? Metadata model schemas do not have versions. However, the datahub project is intentional about not publishing backward-incompatible changes to models. Once a model is created you should feel confident it won't have a backwards incompatible change published

@calm-lawyer-777 if your question was about the schema aspect of dataset, then yes; each update to the schema will result in a new versioned row being written to the metadata store(Eg MySQL). The UI doesn’t show this version history today.

Hi @green-football-43791 @mammoth-bear-12532 thank you for your kind reply. Yes it is about dataset schema, columns and datatypes. I have a use case to track changes to the metadata so users can have the history of changes.

Were you thinking about showing this in the UI as well? Or just offering API for it.

It's great to have it incorporated in the UI

+1

Hey @colossal-furniture-76714 - the versioning system at our storage layer is fixed. However, we intend to add a semantic layer on top that partitions version by column additions or removals, and groups versions together that fall into the same partition. Would that address your use case? Or do you imagine wanting to group multiple schema changes that happened over time into a single version?

Hey @green-football-43791, thanks for your answer. This might address my use case. Could you elaborate on "[...] and groups versions together that fall into the same partition"? When do they fall into the same partition? How is a partition defined? Our problem is that we do not hold a schema registry for the table, the engineers can change the schema and data for their technology area on the device and the database digest the doc string which it receives...