Hi all, I'd like to know if/how it's possible to track changes performed in the UI (e.g. changes in descriptions, new relationships, etc.) and display those changes again in the UI (e.g. by "turning on" displaying of the lastModified field defined for some aspects in order to see who performed certain change and when). As far as I can see, there's limited support for displaying differences in dataset structures coming from ingestion (limited in that it discovers if certain columns are newly added, but a newly added column comment is not considered as a change and the comment is displayed there as if it was there from the beginning). Any ideas? I suspect that adding support for displaying those changes would need to be added in the UI (i.e. the functionality is not available at the moment), but even then it's important to understand whether the history is at least already tracked in the backend, or not (and if not, how to change that)...

Hey @gentle-portugal-21014 you can use the Timeline API to get access to this kind of info, so yes, it already exists in the backend

Yet again, your help is most appreciated! :-)

@orange-night-91387 can speak more to the Timeline API decisions for just supporting datasets

We focused on datasets first as that was the highest priority, it hasn't been extended to other entities because that hasn't come up as a priority and we have limited capacity. Glossary terms specifically has someone from the community who was interested in contributing back working on it, not sure how much progress has been made on it though.

Thanks for the insights. @orange-night-91387, would you know who from the community intended to work on this (so that I can check his progress with him specifically)? Or should I ask someone else for this contact (like Maggie or Shirshanka)? Regarding possible extension - would you be able to point out particular commits where the support for timeline API in datasets was implemented, or files which had to be modified for supporting Datasets, so that we could do something similar for Glossary Terms? Finally, I'm confused regarding possible relation between the version history and the ChangeAuditStamps structure in PDL files. Is there any relation at all (like that this structure needs to be included for any aspects intended to be versioned), or are these concepts completely independent?

I believe it was Dan from this thread: https://datahubspace.slack.com/archives/C02QMLWJG12/p1668727481776099?thread_ts=1668521376.561869&cid=C02QMLWJG12 @little-megabyte-1074 have we heard anything more here after we met with Dan during office hours? This is the initial PR for timeline, the class names have changed since the logic has been reused for event generation: https://github.com/datahub-project/datahub/tree/master/metadata-io/src/main/java/com/linkedin/metadata/timeline/eventgenerator ChangeAuditStamps is different, we use mostly SystemMetadata and time based columns in the DB.

Hi @orange-night-91387, I haven't been successful with contacting @brainy-table-99728, but we started analyzing the PR to find out what needs to be modified. Some parts are rather obvious - there are specific Java classes for composing change events for specific Dataset aspects in https://github.com/datahub-project/datahub/tree/master/metadata-io/src/main/java/com/linkedin/metadata/timeline/eventgenerator/ like e.g. EditableDatasetPropertiesChangeEventGenerator.java, aspects relevant for GlossaryTerm entity might be added rather easily. Certain parts might be even reused - comparing assigned Domains, Owners and Tags might be performed the same way for GlossaryTerms as it's already done for Dataset (and the DOCUMENTATION category isn't very different either, in fact). The main point which is still not clear to us is where exactly is the code branch making sure that change events are indeed generated (and stored) for the Dataset entity, whereas no such events are generated in case of the GlossaryTerm or other entities (although the changes are initiated in the same parts of the UI, e.g. the Domain sidebar; let's focus to changes performed in the UI rather than in ingestion for now). Could you navigate us to this code branch, please?

The "events" are an interpretation of the existing data. We store historical changes to metadata as versions in the backend. TimelineServiceImpl is the spot where we determine what entities are supported for this processing. All entities have the data stored, they just don't all have the processing logic implemented, if that makes sense.

OK, thanks, we'll check it. :-) BTW, am I right that there's no particular reason for not tracking data lineage information changes in Timeline?

Yep, just that it hasn't been a priority.

Hi @orange-night-91387, my colleague managed to extend the Timeline API implementation based on your suggestions so that it supports categories DOCUMENTATION and OWNER for the GlossaryTerm entity - thanks for that, we'll create a PR with these changes. In addition, we alse added support for displaying the latest version number and its age (using the graphical element already used in the Dataset schema) plus the user having performed that change to the sidebar fragments for change categories DOCUMENTATION ("About"), OWNER ("Owner"), GLOSSARY_TERM ("Glossary Terms") and TAG ("Tags") for entity Dataset. Categories DOCUMENTATION and OWNER also work for entity Glossary Term thanks to our extension mentioned above. We implemented it also for category DOMAIN in the front-end part, but it seems that no change events are generated for DOMAIN even for entity Dataset, thus it doesn't make any difference. Note that we didn't add support for displaying metadata stored for the previous versions, only information about the latest version. I wonder if there's a general interest in this visual part, so that we should prepare another PR for it (I guess that it would be easy to make this displayed on demand similarly to the existing button for displaying version information for individual fields in TECHNICAL_SCHEMA, but we didn't implement that part) - please, let me know.

Hi there! I know another community member was actively working on this some time back. We had shared a UX design mockup with them but seems that this has not yet been completed: https://github.com/datahub-project/datahub/pull/6325

Hi @big-carpet-38439, thanks for this reference! The UI part of our implementation is much simpler than the outline from the mentioned PR. Your design with a dedicated History or Change Events tab is more universal and probably also more practical due to taking less space, but it requires touching parts of the source code my colleagues are not familiar with at the moment. Just please note that the picture attached by you to this PR doesn't match the underlying concept of version numbers bound to individual change category types rather than keeping one consistent versioning for all aspects of a certain entity (a dataset in your case). But that's just a sidenote, not really changing the overall idea of the UX. Anyway, your design basically suggests that creating a PR with our simpler implementation for the front-end wouldn't be very useful at the moment. We will prepare the PR for the back-end changes (support for Timeline for GlossaryTerm entity) as suggested in the first part of my previous post.

Thanks for raising this PR! We will certainly take a look