Folks, I am working on a new design that would allow users to run low latency workflows on Flyte and run existing simple python tasks and reduce / ammortize the overhead of running a pod. Would folks here be interested in helping me refine the design and the UX model? Goal: Users should be able to run sub second tasks and yet get the same level of visibility, fault tolerance and simplicity for their code. We might lose some isolation properties that running an individual pod offers, but we will continue to offer isolated environments, versioning and dynamic infrastructure.

Metaflow, which is a comparable framework (though more limited), recently added functionality for “Google cards for machine learning models”: https://towardsdatascience.com/dag-card-is-the-new-model-card-70754847a111. Does there exist anything similar for Flyte today, or is there a plan to implement something like this? If not, would it be of interest?

Feature request : Separate tutorial repo with jupyter notebooks Not sure if its already being considered or worked on, having the below tutorials in jupyter notebook format could be very useful for new users https://docs.flyte.org/projects/cookbook/en/latest/tutorials.html

Hey everyone, the graph UI in Flyte console is looking great ❤️ , do you think Dynamic Tasks will be supported as well in the future, I mean like expanding the nodes inside as it's possible for subworkflows? 🤔 🙏

We had a streaming pipeline that read the flyte events and from the execution we used to fetch the inputUri and from the closure the Outputuri, then fetch the files from gcs. Something like below:

final String inputUri = getNodeExecution().getInputUri();
    final String outputUri = getNodeExecution().getClosure().getOutputUri();

    gcsFileReader.readToLiteralMap(inputUri)
    gcsFileReader.readToLiteralMap(outputUri)

Also we had some special handling for the scenarios where the files are not there meaning that either the task does not have inputs/outputs. Now since the outputUri is deprecated, we are trying to use the NodeExecutionGetData rpc, but we notice some nil pointers here and there. How should we determine when the execution data exists, so we can make the rpc and succeed.

hey i have a request that would serve the Forecasting platform we’re building at Spotify on top of Flyte. We currently subscribe to a topic containing execution events to populate our UI with visibility on what forecasts have been run/where their outputs are located etc. That’s worked quite well! What Im wondering if a similar topic exists to capture the *Workflow* information created at registration time For context, our platform works as such: 1. A model developer registers to Flyte a workflow that represents a forecast model that takes in several assumptions as CSVs at runtime 2. A separate financial analyst writes those assumptions and uploads them to our platform 3. the model can be launched from the platform with the appropriate assumptions by anyone My thought is that if we have an understanding of workflow dependencies at registration time, we can have the platform enforce at assumption upload time that all the correct assumptions are uploaded for each model (with the correct schemas) according to the Flyte version selected. Im sure there are other ways to do this (create a hook at registration time that queries the API for all workflows belonging to the registered version), but through a topic is ideal for us so just want to know feasibility here. Thanks!

Based on the openapi documentation, there doesn't seem to be an offset parameter I can pass to utilize pagination on my site. Is that something you plan to implement? Secondly, when using pagination it would be nice to get a total count of all the entries so I can display the total number in a dashboard card.

is there any interest in platform-level override for default service account to use for k8s pods? right now if there is no k8s service account explicitly set in the launch plan spec, it uses ‘default’. likely constitutes a simple change here: https://github.com/flyteorg/flyteplugins/blob/d2f64cb8c49adccadb01f46960031df3219dce32/go/tasks/pluginmachinery/flytek8s/pod_helper.go#L72 @hallowed-mouse-14616: it seems likely that the pod template is respected here. does that sound right?

We have just submitted a RFC for the eviction of cached task outputs (and proposing the addition of a basic

flyteadmin

API for clearing the cache of completed executions): https://github.com/flyteorg/flyte/pull/2633 Thank you for the initial feedback to @hallowed-mouse-14616. As this change could have a slightly larger impact on Flyte, we'd be happy about some extra thoughts or ideas some of you might have - any comments would be appreciated!

Hi everyone. I don't know if I can use flyte-pod-webhook to send workflow status notifications to discord app?. Thanks all

Hi everyone. I was wondering if there is a way to submit the spark job to the amazon EMR cluster from a Flyte workflow. (My Flyte deployed on AWS cloud environment). Thanks all

Hello, curious if anyone else has had a need for a FlyteFile that has an associated lifetime - whereby it would be automatically cleaned up after a user-specified timeframe. This could be along the lines of compliance with data retention policies.

Question about flyte workflows roadmap/features: If i have a pipeline of 10 steps, could I simply click on the 5th step and just re-run that from the UI/cmd line ?

Within Spotify, we were discussing human in the loop feature and could see that some work has been done on this: https://github.com/lyft/flyte/issues/208 What is the best way to check it and share feedback?

Hi. What do you think of this feature request? https://github.com/flyteorg/flyte/issues/2883

Hey, I was thinking about adding

nodeSelector

to the Pod template propeller uses to create pods. Currently I am only using tolerations to allow my pods to schedule on certain nodes. However, let’s say I have multiple node pools for e.g., CPU/GPU tasks and I want to be verbose about which node I want to run my workloads on. Is tolerations sufficient or would I benefit from using

nodeSelectors

? It would be a fun contribution to work on, if you also think it would add value. This could e.g., be added in the

@task

decorator by the same means you are adjusting

requests/limits

. Let me know what you think! 🙂

Has the topic of creating a terraform provider for flyte ever been raised? I'm thinking in terms of being able to create projects, register workflows, configure launchplans, etc. not the flyte cluster itself.

Hey All, as discussed in our OSS Sync last week we're working on improving performance observability of Flyte components. I drafted an RFC on adding runtime metrics, offering overhead estimates, and orchestration metrics, using opentelemetry traces, to improve performance analysis in both production environments and during feature benchmarking. Please take a look at the RFC draft (with a formatted markdown version here) and leave any comments / questions / concerns. We are very excited to hear from the greater community and incorporate feedback! Also, trying a simple github PR formatted in markdown for this, we are always refining our RFC process - thoughts also welcome regarding this.

set the channel topic: If you're interested in discussing adding new features, please let us know! We'd love to hear your ideas and work together to make Flyte even better.

moving everything to the contribute channel

archived the channel