Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hello all,
<@U04HLNJMXJA> here is interested in Datahub and Flyte integration. In the past - <@U01J8L5AR2S> and <@UTU7ZNA56> have also worked on it - there are few other folks in the community.
I have also added folks from LinkedIn, who might also be interested (do not know yet).
<@UNR3C6Y4T> from my team was working with some folks, but he is out for the past 2 weeks and next 2 weeks. But, eventually he can help a lot.
There is a sample that we created sometime ago, with some community folks - <https://github.com/unionai/flyteevents-datahub>.
This is a prototype.
It uses Flyte Events egress and replicates them to Datahub. You have to run the `
```flytelineage```
script

Problems
• This is not resilient, in case of failures does not do well.
• We would ideally love this to be a framework - that folks can add new plugins if needed. For example Amundsen (spotify has an internal catalog) etc
• Currently it simply listens to all events and keeps them in memory for a workflow and then replicates it. This is not needed, it could use flyte remote to get all the data only on receiving a terminal event. 
• No one uses this in production

We are not datahub experts, i think there is a channel in datahub as well to integrate with flyte

There is one, yes, and this document seems to be the most recent sign of progress:

<https://docs.google.com/document/d/1YMZkJr77Kg0IIJssFxs-QFMf2mpSEHIIP0hTVokqywY/edit?usp=sharing>

Yes this document has been together mainly by me and <@UNR3C6Y4T>.

And I can probably proclaim myself to be a datahub expert in this context

We chatted a lot about this with <@UNR3C6Y4T> last fall, and got to a point where I would have needed to start experimenting with implementing some rudimentary support for emitting different types of events from Flyte. Life however got in the way and I have not been able the work on this since.

The idea we were discussing was indeed to emit event from Flyte that then would be translated into Datahub/Amundsen/OpenLineage/etc events by a catalog-specific translator service.

The event needs have been mapped out (to some extent) in <https://docs.google.com/document/d/1YMZkJr77Kg0IIJssFxs-QFMf2mpSEHIIP0hTVokqywY/edit?usp=sharing|this> document, but it might be quite a tall order for me to start implementing these in Flyte in my current situation. Any help on this front would be highly appreciated. I’d be happy to work on a reference event translator (for datahub), though!

<@UTU7ZNA56> can you guide <@U04HLNJMXJA> 

I can bring him up to speed, at least :smile:

But as said, the flyte code base is not familiar to me, so I would not be able to advice on that

Also what we need is what it should look like

<@U04HLNJMXJA> ,  ping me to set up a call of some sorts! :+1:

After a great call with <@UTU7ZNA56>, we stumbled in some situations, that I believe we can think how to solve together. Some of them: How and where Flyte send its callback functions? Is it possible to send a callback in OpenLineage format? How can we capture it on DataHub's side? Fredrik tried using a REST call, but we believe that's not the best way to do it...
There where some others considerations, that Fredrik can help me to remember
From there, I think we can open a new issue on Flyte's repository, is that right?

would early next week work for you?  what tz are you in?

Openlineage might not be the best format, as it seems to expect the whole package of metadata in the one and same JSON blob.This is not very flexible as you need to collect all info before you can send it onward. This in comparison with datahub’s MCP format which allows you to send incremental changes as they are discovered. 

And the REST calls were made from my toy ”translator” service using the datahub sdk, but i could trivially have used Kafka as well. Datahub only supports REST and Kafka as methods to transfer data into datahub

Thanks <@UNR3C6Y4T> if you can help bring some clarity! :pray: 

Thanks for the help, <@UTU7ZNA56>!
<@UNR3C6Y4T> next week works for me, I'm in GMT -3 timezone (Brasilia-DF timezone)

i have not.  <@U04HLNJMXJA> would you be open to a chat this week or next?

Hello guys, sorry the delay, I'm moving to a new house, and things are a little messy right now. I can talk, sure! We just need to sync our agendas.

<@UNR3C6Y4T> have you free this week for our talk? How about <@UTU7ZNA56> and <@UNZB4NW3S>? Can we talk this week? I'll help in this project as a personal contribution, it will be interesting if our chat can be scheduled after my working hours, 7pm or later in my timezone. I believe it's around 3pm in Yee's timezone.

It might be difficult to get everyones timezone to match, but you can invite me as optional

yeah that’s fine.  i can do weds this week?  3pm seattle time?

Cool, this work for me. If you want to sync agendas, my email is <mailto:victor.gso001@gmail.com|victor.gso001@gmail.com>

<@UNR3C6Y4T> will you be able to talk today?

Sorry man, had to leave...
How about today today 11am Seattle time?

I had a chat with <@UNR3C6Y4T> last week, and we talked about sync hour agendas to see if we can talk this week. <@UTU7ZNA56> when will you be available to talk? Let's try to do at a time that Yee can participate as well

that’s 11 for me here in seattle. works for me.

That's 3 pm for me. Depending on the day, works for me as well

could you make an invite for a day that works <@U04HLNJMXJA> ?

Yes, I can :slightly_smiling_face:
Just gonna look my schedule, and I send an invite.