@high-hospital-85984 are there any updates on this?

what are the next steps

Yee is out for this week and next

Not that I know of @billions-lawyer-59647 . But I think you guys had some ideas about using some proto events that could potentially be used for this? Has that evovled in any way? Otherwise I think it would be nice to get a RFC on Flyte's side for how to support this feature. I'm super happy to partake as a "user" stakeholder, and once the plan is in place we can most probably allocate some resources to help with the implementation where we can.

cc @billions-apartment-18513 / @modern-pilot-97597 will know more let me follow up

from jonathan lamiel following the meeting this morning

https://www.metaintegration.net/ this is the data catalog I was referring to. Most big ETL vendors are selling it as their data catalog (I was at Talend). There we had a 2 way integration. meta integration have crawlers to harvest datasets a bit everywhere, and we use that within our tool too. (Kind of a dataset reference) for data lineage we mainly were sending events directly to meta-integration through API at runtime nothing really fancy.

https://www.alation.com/ this was our biggest competitors

And the one we are using internally, I don’t have access to it…, but it’s Onetrust

Realised we havent linked the document (very much WIP still): https://docs.google.com/document/d/1YMZkJr77Kg0IIJssFxs-QFMf2mpSEHIIP0hTVokqywY/edit

hey - thinking more about the suggestions in the doc, it sounds like you’re advocating for a 100% manually user-produced event stream. but a lot of things really do happen behind the scenes, is there really nothing useful to datahub that can come from the Flyte platform? for example, flyte tasks/workflows. at registration time, when a new version of a task or a task is registered for the first time, is there nothing useful that can be sent to datahub? flyte can emit an event for the creation of each task/workflow.

wrt the manual part of it, for sure that needs to be part of the experience