Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hi!

I am working on a <http://dask.org/|dask> propeller plugin at the moment, which might be related to this. I wanted to give you a quick update  on what has happened so far as the work might be related:
• I've started a <https://github.com/bstadlbauer/dask-k8s-operator-go-client|small repo> creating a Go client for the <https://kubernetes.dask.org/en/latest/operator.html|k8s dask operator>
• I've started adding a <https://github.com/bstadlbauer/flyteplugins/tree/add-dask-plugin|flyteplugins branch> which adds the plugin. At the moment, there is a very rough draft which uses the `DaskJob` resource (similar to the `SparkApplication` resource) to deploy the cluster and run the code
• I've started a <https://github.com/bstadlbauer/flytekit/tree/add-dask-plugin|flytekit branch>, which at the moment only has a minimal skeleton which enables testing the created backend plugin
All of this is still on the POC level (things will be refactored/test added, etc.), but I would like to slowly progress this to have an MVP over the upcoming weeks.

Have a nice day :slightly_smiling_face:

set the channel topic: <https://docs.google.com/document/d/1-695lxz8a-GFz4cFamGkF1NhspKMkDGIDdkY9EDLMz8/edit#>

<@U03F38WVDHN> updated the doc with the things that we discussed yesterday.  left one place for you to add some examples.

Thank you! Really appreciate it! I’ll give a read today and add examples there

<@UNR3C6Y4T> <@UNZB4NW3S> do you mind if I also share this doc with Shopify folks? They might be also interested.  cc <@U03JAD86UMQ>

oh please do.  i think you should have editor (and share) perms

<@UNR3C6Y4T> I think the doc needs to be reordered

BTW, are you guys interested in presenting this in next KubeRay community meeting? cc <@UR1NJDZJ9>

i think we should… but only if we’re ready to.

and if not the next one, do you think we could do the one after that?

i’m curious to see how these other community syncs are run too

The next sync will be July 12 - it runs every two week on Tuesday night 6PM Western time

not sure if it’ll be me or kevin or ketan but one of us from our side

Sure I can give a bit context on how we will use this at Spotify  but I’ll leave you guys with the design doc

<@UNZB4NW3S> <@UNR3C6Y4T> regarding the KubeRay community demo, it seems tomorrow’s agenda will be pretty packed according to <@UR1NJDZJ9> for KubeRay 0.3 release. Are you guys OK with moving to the one on July 26? <@UR1NJDZJ9> could you please help confirm that there’ll be a slot available for the sync on July 26?

I can confirm the availability on July 26. If Flyte team can move demo to July 26, that would be great.

<@U03F38WVDHN> <@UR1NJDZJ9> - so we’re on for the 26th right?  we’re gonna start work on the presentation tomorrow.

and the details for the meeting?  is it a zoom meeting?

also is there a ray slack we should be on?

The meeting is 30mins long. If you can control the demo in 10-15mins. That would be great. I will send the google meeting link later

<@U03F38WVDHN> how much of the presentation do you want in on?

so the whole meeting is 30 mins? so 15 mins for presentation and 15 for questions/discussion?

or do we have to fit everything into 15 mins? and the other 15 is for someone else?

I think 15mins presentation and 5mins QA would be great

<@UNR3C6Y4T> I don’t think I have much to present here. I’ll just cheer for your guys and I can chime in and say a few words on how we will use it at Spotify

<@UR1NJDZJ9> are any of the prior talks on youtube? just want to see an example.

<@UNR3C6Y4T> I don’t think we have recorded meeting and uploaded to youtube. It’s pretty flexible, you can share docs etc, no strict rules on the styles and materials

Great job <@UNR3C6Y4T> : <@USU6W5ATA> and thank you <@U03F38WVDHN> 

I actually think we can release ray integration in alpha as a patch release as it needs no platform changes

This is really awesome work! Thank you all!

We finally did it - also wouldn't be possible without <@UR1NJDZJ9> 

<@U03F38WVDHN> - these are the Ray PRs we’re working on.
<https://github.com/flyteorg/flyteidl/pull/308>
<https://github.com/flyteorg/flytekit/pull/1093>
<https://github.com/flyteorg/flyteplugins/pull/279>
responding to your comment on the first one, but mind taking a quick look at the other two as well?

<@USU6W5ATA>’s shooting to merge these tomorrow, assuming they’re okay.  After that, I think we’re actually going to do a patch Flyte release (like the whole platform, since propeller and admin will need to pick up the new IDL)

do you think you’ll be able to help test/vet after that?

kevin’s still working on getting the dashboard hooked up, but you had mentioned something about metrics as well that you were interested in.  will think about ways to enable those as well (though can’t promise anything just yet)

Thanks! Yeah, I started looking into these PRs

Hey <@UNR3C6Y4T>, I drop a few comments in the PRs. They are mostly around the flexibility of the Ray cluster configuration. Essentially we will need more settings such as service account and image in Ray cluster config

&gt; do you think you’ll be able to help test/vet after that?
yep, I’m happy to dogfood it!

<@U03F38WVDHN> can we use the same serviceaccount as the workflow service account?

You mean the service account for flyte workflow? I think they are different

they are all on the same cluster and even the same namespace

that wont work for us though - flyte workflows are executed in their own infra and in a flyte workflow, we need to create a cluster in the Ray infra

so when you create the Ray cluster, do you assume you have Ray CRD installed on the flyte cluster?

thats the current setup right - cc <@USU6W5ATA>?

we might need to sync on this - in our setup, Ray and Flyte are run on different clusters. KubeRay also has an API server, I wonder if that will make things easier :thinking_face:

so Flyte does support running on different clusters

but my recommendation is to run just flyte in multi-cluster mode

this is so much better than using the same cluster

that way, data stuff cannot impact ML stuff

yes exactly! we have dedicated flyte cluster for orchestration, some teams using flink cluster for data, and now we have ray clusters

but a better model is to run flytepropeller on different clusters

keep the central control plane independent and just parachute flytepropeller to multiple clusters

otherwise the one propeller will start getting overloaded at (very high scale), but it will

and one good property of having flytepropeller manage your cluster is, it will know the scheduler how the resources are getting throttled and work correctly

&gt;  thats the current setup right - cc
yes, we have to install Ray CRD on flyte cluster

Discussion: In Spotify’s setup, Ray and Flyte are run on different clusters. KubeRay also has an API server, and they want to use Propeller to send a http request to Ray api server to create ray job. Therefore, we may need to add a ray webAPI plugin in propeller.

Start a new thread to discuss this. Is there any other better way to handle this?
cc <@UNR3C6Y4T> <@UNZB4NW3S> <@U03F38WVDHN>

yeah <@U03F38WVDHN> i know you brought this up in the past, but is there any chance you can add flyte to the ray cluster?

hmm, does that mean we need to install Propeller on our ray cluster? do you mind elaborating more on this? thank you!

yeah… though it’s not possible to scan the workflow right now is it <@UPBBNMXD1> do you know?

like if a workflow has a certain type of task, then it should go to a certain cluster, was that in the feature anand worked on?

are you asking how do we assign specific executions to specific flytepropeller clusters?

we don't have anything that's workflow-aware, right now it's just a static mapping: <https://docs.flyte.org/en/latest/deployment/multicluster.html#user-and-control-plane-deployment>

thanks katrina. so yeah never mind <@U03F38WVDHN> for the current plugin to be useful, the kuberay operator will need to be installed on the flyte cluster.  is this not possible?

yeah it’s not working with our current setup since Ray and Flyte are owned by different teams and operated on different GKE clusters.

and there’s no way to add the ray operator to the flyte cluster?

Can’t we load the creeds for the ray cluster and submit the job resource there?

<@UU43XA551> we can, but this causes a lot or problems, you folks did it for FlinkOperator if you remember and should work with Ray

But our recommended method is to just install flytepropeller on all k8s clusters. Think of propeller as integral to K8s

yeah i am asking because we already work in this way with a flink task and the FlinkOperator which is running on a different GKE cluster (more resources available for massive jobs)

would installing propeller to the ray-cluster automatically forward the ray tasks towards that propeller?