👋 I have several workflows that are dependent on the same external data, this external data appears at quasi regular intervals several times a day. I know when approximately I should wait for the next chunk of external data. What approach is best to make sure that polling is "deduplicated", i.e. that I don't query external service too often? Currently the pipeline is being run in luigi with a separate trigger application that performs the waiting only when the external data is most likely to appear and triggers the appropriate workflows. Obviously this can be reused with flyte but I was hoping that triggering can be implemented inside flyte too. The approaches I have in mind are the following: 1. create a special waiting task with caching to make sure that only one of these tasks is being run at the same time and once the data appears this fact will be retrieved from cache by different workflows 2. create a special sensor that somehow prevents making external requests for the same data too often (maybe creating a singleton state class for these sensor tasks would work?) What is the best solution for this? Perhaps I'm missing some other, simpler approach?

Hello. Need some help as our production env web broke for some reason. Last I remember I tried registering a workflow which registered successfully and a few mins later when I tried running it - it shows this error. Can someone lead me to what could be the issue? I asked my other colleagues - there was no operation on the cluster during that time. We have flyte deployed on our GCP k8s cluster.

I'm attempting to narrow down a bug in Flyte v1. When we do nothing but add one our private internal packages to a uv.lock that we pass into a workflow that makes use of map_task we get the following error (I'll post the full stacktrace in the thread):

ModuleNotFoundError: No module named 'site-packages.flytekit'

This causes a change down which seems to affect the container args (I'm not sure what these are), but we get this because if I change the dependencies on the image but not the tasks in the workflow I get an error asking me to re-register the task:

/template/Target/Container/args/23:
	- flytekit.core.python_auto_container.default_task_resolver 
	+ site-packages.flytekit.core.python_auto_container.default_task_resolver

... I believe the error "No module named 'site-packages.flytekit'" occurs because load_object_from_module() gets passed *site-packages.*flytekit.core.python_auto_container.default_task_resolver and not flytekit.core.python_auto_container.default_task_resolver but I don't know what leads to this import path being passed in. I'll post a minimal working example package that causes the break once I've factored one out, but does anyone know what's happening here?

Hi all! I'd like to submit a PR for something I found in the Flyte 2 documentation but not sure where the docs are in the source? I've contributed before but it's probably been a couple of years. Thanks!

occasionally my my workflow fails when a task gets this error:

/1] currentAttempt done. Last Error: USER::Grace period [3m0s] exceeded|containers with unready status: [primary]|Back-off pulling image "<http://mycompany.com/buck/horse:19.2.0|mycompany.com/buck/horse:19.2.0>": ErrImagePull: failed to pull and unpack image "<http://mycompany.com/buck/horse:19.2.0|mycompany.com/buck/horse:19.2.0>": failed to copy: read tcp 100.64.44.213:38748->10.37.121.189:443: read: connection reset by peer

this seems like a random node-related issue, is the best solution here to have all my tasks have retries set? (and is there a way to do this at the flyte level rather than changing all invocations at the flytekit level?)

Hi everyone, I am a new Flyte user and happy to join the Slack channel ! I noticed that in the v1 documentation on flyte.org site, some pages refer to other doc pages with broken links. For example, the page Data Input/Output links to https://docs.flyte.org/en/latest/concepts/data_management.html which no longer exists.

👋 Quick q -- On the Flyte 2 SDK changes -- are they a full break from Flyte 1 and require the Flyte 2 backend. Or are they intended to work with both Flyte 1 and Flyte 2?

❔I'm looking for some help with setting up Okta auth with flyte-core and an external authorization server following this doc. As I understand it (please correct me if I'm wrong), there are 4 main types of authentication to cover in a Flyte setup: 1. Human browser (HTTP) 2. Human CLI (GRPC) 3. Machine to machine (GRPC, external to k8s networks, e.g. CI systems) 4. Service to service (GRPC, within k8s network) Is there a way currently to enable 1,2 and 3 but disable 4 and rely solely on k8s networking policies to verify incoming traffic? Or maybe I need enable

selfAuthServer

for only

flytepropeller

to use the internal authorization server in just scenario 4? (Using flyte-core v1.16.1 in AWS EKS). Thanks in advance!

Hi, I'm trying to have an input parameter of type callable (python function) to the workflow (not task). When I register my workflow and open it on UI, I see the following inputs for the callable function param

custom_func

that I need to fill in. Can you please explain what file here means? is it a URL to the pickled python function stored somewhere where flyte can access it?

Hi, I would like to try Flyte 2.0 on a local sandbox cluster. Is there a way to create it? I tried

uv run --prerelease=allow flytectl demo start

but it started installing Flyte 1.16.1

Going to use Flyte v1.16.1

. Is it possible to do it?

Hi, I am facing issue when I submit a workflow to flyte which is setup in my local kubernetes following the helm installation guide. The execution pod is having the error. Is there anything I have missed out?

Reason:   Error                                                                                                                                                                                   │
│       Message:  y", line 1246, in invoke                                                                                                                                                                │
│     return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                      │
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                      │
│   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 814, in invoke                                                                                                                     │
│     return callback(*args, **kwargs)                                                                                                                                                                    │
│            ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                    │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/bin/entrypoint.py", line 773, in fast_execute_task_cmd                                                                                         │
│     _download_distribution(additional_distribution, dest_dir)                                                                                                                                           │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/core/utils.py", line 309, in wrapper                                                                                                           │
│     return func(*args, **kwargs)                                                                                                                                                                        │
│            ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                        │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/tools/fast_registration.py", line 310, in download_distribution                                                                                │
│     FlyteContextManager.current_context().file_access.get_data(                                                                                                                                         │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/utils/asyn.py", line 113, in wrapped                                                                                                           │
│     return self.run_sync(coro_func, *args, **kwargs)                                                                                                                                                    │
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                    │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/utils/asyn.py", line 106, in run_sync                                                                                                          │
│     return self._runner_map[name].run(coro)                                                                                                                                                             │
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                             │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/utils/asyn.py", line 85, in run                                                                                                                │
│     res = fut.result(None)                                                                                                                                                                              │
│           ^^^^^^^^^^^^^^^^                                                                                                                                                                              │
│   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 456, in result                                                                                                                     │
│     return self.__get_result()                                                                                                                                                                          │
│            ^^^^^^^^^^^^^^^^^^^                                                                                                                                                                          │
│   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result                                                                                                               │
│     raise self._exception                                                                                                                                                                               │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/core/data_persistence.py", line 614, in async_get_data                                                                                         │
│     raise FlyteDownloadDataException(                                                                                                                                                                   │
│ flytekit.exceptions.system.FlyteDownloadDataException: SYSTEM:DownloadDataError: error=Failed to get data from <s3://flyte-dev/flytesnacks/development/GJD6GFARVQLFIK64SWWSTFCM3I======/fast28f7e9edccfe> │
│ 6ce9484363d126d1965a.tar.gz to /root/ (recursive=False).                                                                                                                                                │
│                                                                                                                                                                                                         │
│ Original exception: Unable to locate credentials

Hi! How do I set up authentication? Or - Keycloak? With

flyte-binary

. I'm struggling with authentication set up of

flyte-binary

Helm chart version 1.16.1 against Keycloak 23.0.6. The

values.yaml

is below. In general, may be (I'm not sure), my question is about what the exact requirements are for Flyte, in order to set up the requirements in Keycloak (KC) of the version. Certain scopes, for example, are absent:

offline

is

offline_access

,

all

is absent in KC and so on. I'm looking into "Configuring authentication" - https://www.union.ai/docs/v1/flyte/deployment/flyte-configuration/configuring-authentication/ I set the things up for web interface and

flytectl

to work. I hope I understood what a things should be set up from Keycloak perspective concerning

audience

, for example. At leats web interface and

flytectl

works. However, an executions are submitted successfuly, but unable to start. Flytepropeller (may be it is Propeller) complains on auth. with logs producing speed 0,5+ Mb/second. At least I have in logs:

{"json":{"src":"interceptor.go:22","x-request-id":"a-fmktn2s6wf5f6rz7n5rb"},"level":"debug","msg":"authenticated user doesn't have required scope","ts":"2025-10-23T14:01:36Z"}
{"json":{"exec_id":"a5pwkw4w46snvz6xgvhj","ns":"flyteprj0005-development","res_ver":"217278987","routine":"worker-12","src":"auth_interceptor.go:213","wf":"flyteprj0005:development:workflows.hello_world.hello_world_wf"},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = authenticated user doesn't have required scope]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2025-10-23T14:01:36Z"}
{"json":{"exec_id":"a5pwkw4w46snvz6xgvhj","ns":"flyteprj0005-development","res_ver":"217278987","routine":"worker-12","src":"auth_interceptor.go:236","wf":"flyteprj0005:development:workflows.hello_world.hello_world_wf"},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"token_source_provider.go:257"},"level":"info","msg":"Fetched new token with expiry 2025-10-23 14:13:44.083157411 +0000 UTC m=+3572.209271697","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"token_source_provider.go:264"},"level":"info","msg":"retrieved token with expiry 2025-10-23 14:13:44.083157411 +0000 UTC m=+3572.209271697","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"token_utils.go:31"},"level":"debug","msg":"Token expiry : 2025-10-23 14:13:44.083157411 +0000 UTC m=+3572.209271697, Access token expiry : 2025-10-23 14:13:44 +0000 UTC, Are the equal : false","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"token_source_provider.go:238"},"level":"info","msg":"retrieved token from cache with expiry 2025-10-23 14:13:44.083157411 +0000 UTC m=+3572.209271697","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"handlers.go:271"},"level":"debug","msg":"Found existing metadata header flyte-authorization","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"handlers.go:299"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"handlers.go:254"},"level":"debug","msg":"gRPC server info in logging interceptor [8d666489-6d2f-48e6-b020-9347945c991b]method [/flyteidl.service.AdminService/CreateWorkflowEvent]\n","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"interceptor.go:22","x-request-id":"a-nbdd4tx7w6tr8xzk88w6"},"level":"debug","msg":"authenticated user doesn't have required scope","ts":"2025-10-23T14:01:36Z"}

My

values.yaml

for the Helm chart is below:

---

userSettings:
  hostName: "flyte"

configuration:
  logging:
    level: 6
    show-source: true
  database:
    username: "flyte_control_plane"
    password: "helm --set configuration.database.password=psql_pw"
    host: "postgres.test.daiger.ru"
    dbname: "flyte_control_plane"
  storage:
    type: "minio"
    metadataContainer: "flyte"
    userDataContainer: "flyte"
    provider: "s3"
    providerConfig:
      s3:
        region: "eu-west-1"  # Irrelevant for local but still needed.
        authType: "accesskey"
        endpoint: "<https://minio-api.test.k8s.daiger.ru:443>"
        accessKey: "helm --set configuration.storage.providerConfig.s3.accessKey=AWS_ACCESS_KEY_ID"
        secretKey: "helm --set configuration.storage.providerConfig.s3.secretKey=AWS_SECRET_ACCESS_KEY"
        disableSSL: "true"
        secure: "false"
  auth:
    enabled: true
    oidc:
      baseUrl: "<https://keycloak.daiger.ru/realms/daiger>"
      clientId: "flyteadmin_test"
      clientSecret: "helm --set configuration.auth.oidc.clientSecret=FLYTE_CREDENTIALS_OIDC_CLIENT_SECRET"
    internal:
      clientId: "flytepropeller_test"
      clientSecret: "helm --set configuration.auth.internal.clientSecret=FLYTE_CREDENTIALS_INTERNAL_CLIENT_SECRET"
      clientSecretHash: "helm --set configuration.auth.internal.clientSecretHash=FLYTE_CREDENTIALS_INTERNAL_CLIENT_SECRET_HASH"

    authorizedUris:
    - "<https://flyte.test.k8s.daiger.ru>"

  inline:
    auth:
      appAuth:
        authServerType: "External"
        externalAuthServer:
          baseUrl: "<https://keycloak.daiger.ru/realms/daiger>"
          metadataUrl: ".well-known/openid-configuration"
        thirdPartyConfig:
          flyteClient:
            clientId: "flytectl_test"
            redirectUri: "<http://localhost:53593/callback>"
            scopes:
            - "offline_access"
            - "offline"
            - "all"

            audience: "<https://flyte.test.k8s.daiger.ru>"
      userAuth:
        openId:
          baseUrl: "<https://keycloak.daiger.ru/realms/daiger>"
          scopes:
          - "profile"
          - "openid"
          clientId: "flyteconsole_test"

ingress:
  create: true
  ingressClassName: "nginx"
  commonAnnotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"

    nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
    nginx.ingress.kubernetes.io/proxy-buffers: "4 16k"
    nginx.ingress.kubernetes.io/proxy-busy-buffers-size: "32k"
    nginx.ingress.kubernetes.io/large-client-header-buffers: "8 16k"

  httpAnnotations:
    nginx.ingress.kubernetes.io/app-root: "/console"
  grpcAnnotations:
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  host: "flyte.test.k8s.daiger.ru"
  separateGrpcIngress: true
  tls:
  - secretName: "tls-flyte-test-k8s-daiger-ru"
    hosts:
    - "flyte.test.k8s.daiger.ru"

serviceAccount:
  create: true
  name: "flyte"
  imagePullSecrets:
    - name: "image-pull-secret"

What are a requirements to be provided from Keycloak perspective??? Or is there an evident error in the materials above?

dumb q: if i want to backup / restore the flyte postgres db, do just treat it like any old postgres DB that's used by a webapp? like i can just pg_dump / pg_restore? (ref https://flyte-org.slack.com/archives/C06H1SFA19R/p1761615884646699?thread_ts=1761615882.693109&cid=C06H1SFA19R )

sometimes we pass around FlyteDirectories and FlyteFiles that need to be fully downloaded to specific locations for cli applications to use. Are there builtin functions to do this sort of thing?

❔ Does anyone know how to delete local credentials created with the CLI PKCE auth, to force another log in while testing? TIA 🙏

Hey folks.. Does flyte have the ability to ensure that a given workflow can have only 1 instance running, irrespective of inputs (i.e. I can’t rely on caching)

Hi! currently, I am trying to use flyteconnector bigquery. However, after trying to open StructuredDataset using

bq_template = BigQueryTask(
    name="<name>",
    inputs={},
    query_template="SELECT * FROM <project_id>.<dataset_id>.<table>",
    output_structured_dataset_type=StructuredDataset,
    task_config=BigQueryConfig(ProjectID="<project_id>"),
)
@task(
    container_image=image_name,
)
def convert_bq_table_to_pandas_dataframe(ds: StructuredDataset) -> pd.DataFrame:
    return ds.open(pd.DataFrame).all()

@workflow
def full_bigquery_wf() -> pd.DataFrame:
    ds = bq_template()
    return convert_bq_table_to_pandas_dataframe(ds=ds)

So, what happen is when the bigquery task query data from bq it uses flyteconnector service account but after that when the python task try to extract pandas dataframe it is unable to do so.

google.api_core.exceptions.PermissionDenied: 403 Access Denied: Dataset <project_id>:<job_id>: User does not have permission to access results of another user's job.

I have already deploy flyteconnector and enable plugin as documentation mentioned. Any help would be greatly appreciate :).

Hi could you please give an example of fetch a workflow from the cluster and create and register a launch plan for this fetched workflow? Would it possible?

Hi team, is there any documentation related to configuring a

flyte-core

helm deployment using a custom MinIO S3 bucket, without IAM configuration? There's helm chart parameters to pass in accessKey and secretKey but we want to avoid baking long-term credentials into our source code. I checked all the pages under https://docs-legacy.flyte.org/en/latest/deployment/deployment/index.html and https://www.union.ai/docs/v1/flyte/deployment/flyte-deployment/. I also checked the example flyte core chart and read its README.md but I haven't seen if there's alternatives or usages for the accessKey secretKey fields

Hey everyone, looking into making my workflows more reliable by limiting how many executions can run concurrently. I am looking in the docs but haven't found a configuration that manages this. I'm ok with any parallelism within workflow executions My main goal is to handle peaks of incoming data (that trigger new workflow executions) better and prevent rising costs and jobs from failing due to timeouts and limited resources/failing to scale the cluster. Looking at https://www.union.ai/docs/v1/flyte/deployment/flyte-configuration/performance/#1-workers-the-workqueue-and-the-evaluation-loop and https://www.union.ai/docs/v1/flyte/deployment/configuration-reference/scheduler-config/#queue-configcompositequeueconfig

Btw, the links in the https://www.union.ai/docs/v1/flyte/deployment/configuration-reference/ subpages (scheduler, datacatalog, flyteadmin, propeller) don't point to the page sections but instead refer back to https://www.union.ai/docs/v1/flyte/deployment/configuration-reference/

Hello! Is there a plan to release a 1.16 patch with this fix? We would love to try it out when it's available!

Did flytectl

v0.9.0

got removed/deprecated for the

flytectl-setup-action

? I was using that in my CICD and it was still working last week. Today I'm getting:

Error: Unable to find flytectl version "v0.9.0" for platform "Linux" and architecture "x86_64".

in my GHA workflow when running this action.

hey all, im trying to get flyte working with okta for user + machine to machine auth. has anyone been able to make okta work with the Client Credential (

ClientSecret

) auth type? does anyone know if it will work without custom auth servers on our plan? been struggling with this for a few weeks to no avail

What could be potential reasons for Cache write error in a Flyte task? I am seeing this error in flyte console -

Failed to write output for this execution to cache.

. I looked into datacatalog logs for the corresponding flyte task, nothing unusual there. Datacatalog created, updated & deleted reservations for that task as other tasks. We are using flyte

1.15.3

Hi, I am facing a very strange issue, and I am out of ideas. We have 4 instances of Flyte, all of which are configured the same and run the latest version. Each of them is running on a different cluster, and we route traffic using Ingress Nginx Controller, which is configured in exactly the same way on all clusters. All instances use Azure AD SSO, and all use the same App Registration/credentials. However, for some reason, one of these 4 instances does not work. The issue is that when I access the URL, I get to the login page, and then successfully log in using the Azure AD SSO but after that, every request fails on 400 error

400 Bad Request
Request Header Or Cookie Too Large
nginx

I tried different browsers, incognito mode, wiping cookies, everything. This only happens on that one instance, and it works without any issues on the other 3. Any ideas?

Hi! 👋 We're running into an issue where executions are stuck in an

ABORTING

state and can't be fully terminated. The executions include a dynamic workflow step, and these dynamic workflows show 2 x tasks as

RUNNING

- the task descriptions specify they are

initializing

. I think these initializing dynamic tasks are somehow blocking the workflows from resolving the abort request. Any suggestions for ways we can trigger these workflows to transition to

ABORTED

? We're currently running flyte

1.15.3

.

Hi, is there a way to use podman instead of docker to build images when running

pyflyte run --remote

?

Hi, is there any plan to support python3.13/14 in flytekit any soon?

Hi everyone, I'm desperately trying to setup flyte-core with an s3 bucket and provide my access key and secret key via a secret. I can't find how to do that, the documentation isn't clear on what form should that secret take and the ai bot ins't helping and giving contradictory and false information. Can someone please provide an example? Thanks a lot