how can i run this in debug any idea, is there any env var

Hi there, I'm debugging one of my tasks and would like to use the outputs of a remote task. I'm following the snippet in the dashboard of the task that I want:

from flytekit.remote.remote import FlyteRemote
from flytekit.configuration import Config
remote = FlyteRemote(
    Config.for_endpoint("<http://my.endpoint.com|my.endpoint.com>"),
)
remote.get("<flyte://url-to-my-flyte-output>")

Redacted the urls. When I run this, I get a

<class 'flytekit.core.type_engine.LiteralsResolver'>

, how do I get the underlying data?

Hi there, I'm deploying Flyte to my K8s cluster using `flyte-binary`; the service is happily running but every time I try to start a "hello world" workflow using

pyflyte run

the execution fails with this error:

flytekit.exceptions.system.FlyteDownloadDataException: SYSTEM:DownloadDataError: error=Failed to get data from s3://<the bucket>/flytesnacks/development/K3QHP5EWEVPQRJJUI4CPMYMZQU======/fast5078eeda28546327dc735d6cad471fa3.tar.gz to ./ (recursive=False).

Original exception: Unable to locate credentials

My understanding is that the task pod does not have the credentials to access the metadata bucket. How is it supposed to get them in the first place? Since they are supplied via the Helm chart, I figured they would be automatically provisioned in the namespace where the task is running (

flytesnacks-development

in that case). Note that I had a similar issue with container images as I'm using a private registry. I was able to fix it by installing the credentials in a secret in the execution namespace and editing the serviceaccount the task pod is run under to such that it can make use of these credentials. In that case though, the format for the credentials and the way they are consumed is standard Docker access in K8s; however in the bucket case I have no idea how to supply the credentials to the execution pod - for example, in which format does it expect them? Thanks for your help!

I'm not seeing one, but just to be sure - was there ever a way to ensure that only X number of a specific task or (even better) a collection of tasks are running at the same time? I'm asking because we have a 4tb FSX lustre drive that is used by many tasks, but if you have many tasks operating on it at the same time - well the drive pretty much deadlocks... so it would be very useful to be able to limit this.

Hi all, nice to e-meet you! I'm running into a Flyte issue we haven't seen before. I have a top-level workflow that contains sub-workflows. One of them is failing leading to the error message "failed to create workflow in propeller etcdserver: request is too large." I looked at the flyte-propeller logs but could not find specific errors. Below are the inputs to this workflow in case that helps. Does anyone have any insight into what this error might indicate?

{
  "chunk_wait_seconds": 60,
  "start_datetime": "1/1/2013 12:00:00 AM UTC",
  "qhat_cc": {
    "union": "<gs://planet-forests-jira/FO-955/conformalizese_pv-forests-diligence-canopy-cover-v1.3.0-1x1.csv>"
  },
  "se_decimals_ch": {
    "union": 1
  },
  "overwrite": false,
  "spline_df_cc": {
    "union": 3
  },
  "lambda_ridge_cc": {
    "union": 0.4560787425514926
  },
  "qhat_ch": {
    "union": "<gs://planet-forests-jira/FO-955/conformalizese_pv-forests-diligence-canopy-height-v1.3.0-1x1.csv>"
  },
  "feature_scaler_path_ch": {
    "union": "<gs://pv-forests-diligence-training/libraries/diligence-v3-canopy_height.train.features.robust.scaler.pck>"
  },
  "cv_threshold_cc": {
    "union": 0.012505642062132475
  },
  "ramp_up_factor": 10,
  "gedify_model_path": {
    "union": "<gs://pv-forests-diligence-training/models/forest-observatory/model-registry/agb:v32/model.joblib>"
  },
  "se_decimals_cc": {
    "union": 0
  },
  "denoise_asset_keys": {
    "union": [
      "denoised",
      "denoised_se",
      "change_category"
    ]
  },
  "update_timeseries": true,
  "steps_to_skip": "(empty)",
  "model_config_paths_ch": {
    "union": [
      "<gs://pv-forests-diligence-training/models/diligence-v3-canopy_height-04b/config.yml>"
    ]
  },
  "cv_threshold_ch": {
    "union": 0.32753039812553936
  },
  "spline_df_ch": {
    "union": 3
  },
  "aic_threshold_ch": {
    "union": 9.270085537273262
  },
  "aic_threshold_cc": {
    "union": 7.70976136557189
  },
  "published_asset_keys": {
    "union": [
      [
        "data",
        "uncertainty",
        "change_category",
        "dayofyear",
        "score"
      ],
      [
        "data",
        "uncertainty",
        "change_category",
        "dayofyear",
        "score"
      ],
      [
        "data",
        "uncertainty",
        "dayofyear",
        "score"
      ]
    ]
  },
  "feature_scaler_path_cc": {
    "union": "<gs://pv-forests-diligence-training/libraries/diligence-v3-cover.train.features.robust.scaler.pck>"
  },
  "aoi": {
    "tag": "WKB (binary data not shown)"
  },
  "response_scaler_path_ch": {
    "union": "<gs://pv-forests-diligence-training/libraries/diligence-v3-canopy_height.train.response.robust.scaler.pck>"
  },
  "version": "v1.3.0.test",
  "denoise_prediction_version": {
    "union": "v1.1.0"
  },
  "response_scaler_path_cc": {
    "union": "<gs://pv-forests-diligence-training/libraries/diligence-v3-cover.train.response.robust.scaler.pck>"
  },
  "lambda_ridge_ch": {
    "union": 0.6031510586243339
  },
  "priority": 0,
  "model_config_paths_cc": {
    "union": [
      "<gs://pv-forests-diligence-training/models/diligence-v3-cover-04/config.yml>"
    ]
  },
  "end_datetime": "1/1/2025 12:00:00 AM UTC"
}

Hi team, We are currently working on integrating Ray with Flyte and would like to apply different annotations for the Ray head pod and worker pods. It seems that the current

pod_template

configuration applies the same annotations to all pods uniformly. Is there any existing workaround or recommended way to set annotations differently for head and worker pods? Thanks in advance!

Hey, I've got a workflow where each task is relatively light in compute/data needs, but the DAG itself is heavy. I have a workflow inside a dynamic inside a dynamic. Specifically, my outer level is a dynamic which creates 50 dynamics, this middle level is a dynamic which each create 35 workflows. The inner workflow itself is fitting a relatively simple ML model (think XGBoost). When I call this, at larger scales I often

[1/1] currentAttempt done. Last Error: USER::Pod was rejected: The node had condition: [DiskPressure…

. I've tried bumping the disk space on the nodepool to something large, but this does not help. Using lower max-parallelism helps to some extent, but I'd like these to execute in parallel at scale. Is this a known issue with nested dynamics? Is there something I can improve in my flyte deployment? Is this something that won't be an issue in flyte 2.0? This post by @clean-glass-36808 about deserializing dynamic workflows massively increasing CPU usage is possibly related https://flyte-org.slack.com/archives/CP2HDHKE1/p1753231403202179

Hey team, I have been having a issue recently where

flytectl demo start

repeatedly produces this error:

flytectl demo start
🧑‍🏭 Bootstrapping a brand new Flyte cluster... 🔨 🔧
🐋 Going to use Flyte v1.16.1 release with image <http://cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-8b999cef8739bd3a117d9dd3b9a16b06493605bd|cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-8b999cef8739bd3a117d9dd3b9a16b06493605bd>
🐋 Pulling image <http://cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-8b999cef8739bd3a117d9dd3b9a16b06493605bd|cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-8b999cef8739bd3a117d9dd3b9a16b06493605bd>
sha-8b999cef8739bd3a117d9dd3b9a16b06493605bd: Pulling from flyteorg/flyte-sandbox-bundled
52e0b83afc58: Pull complete
668cddf73609: Pull complete
c7e7273a9d9e: Pull complete
ff7dd37cdc54: Pull complete
9e34fcc4c01b: Pull complete
ce1037fe9e1d: Pull complete
e95b18ff6786: Pull complete
2a5fa435b225: Pull complete
Digest: sha256:26ed516df2aeadc7701d48ed0b8745ce033046e00b74e67555a11f20995a4271
Status: Downloaded newer image for <http://cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-8b999cef8739bd3a117d9dd3b9a16b06493605bd|cr.flyte.org/flyteorg/flyte-sandbox-bundled:sha-8b999cef8739bd3a117d9dd3b9a16b06493605bd>
🧑‍🏭 Starting container... 🔨 🔧
⏳ Waiting for cluster to come up... ⏳
Error: All attempts fail:
#1: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused - error from a previous attempt: read tcp 127.0.0.1:56220->127.0.0.1:6443: read: connection reset by peer
#2: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#3: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#4: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#5: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#6: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#7: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#8: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#9: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#10: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#11: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#12: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#13: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#14: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
#15: Get "<https://127.0.0.1:6443/livez>": dial tcp 127.0.0.1:6443: connect: connection refused
{"json":{},"level":"error","msg":"All attempts fail:\n#1: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused - error from a previous attempt: read tcp 127.0.0.1:56220->127.0.0.1:6443: read: connection reset by peer\n#2: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#3: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#4: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#5: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#6: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#7: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#8: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#9: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#10: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#11: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#12: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#13: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#14: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused\n#15: Get \"<https://127.0.0.1:6443/livez>\": dial tcp 127.0.0.1:6443: connect: connection refused","ts":"2025-10-07T13:19:18-06:00"}

My

config-sandbox.yaml

contains this:

admin:
  # For GRPC endpoints you might want to use dns:///flyte.myexample.com
  endpoint: dns:///localhost:30080
  insecure: true

My docker and flytectl versions are shown below:

docker version
Client: Docker Engine - Community
 Version:           28.5.0
 API version:       1.51
 Go version:        go1.24.7
 Git commit:        887030f
 Built:             Thu Oct  2 14:54:28 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.5.0
  API version:      1.51 (minimum version 1.24)
  Go version:       go1.24.7
  Git commit:       cd04830
  Built:            Thu Oct  2 14:54:28 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.28
  GitCommit:        b98a3aace656320842a23f4a392a33f46af97866
 runc:
  Version:          1.3.0
  GitCommit:        v1.3.0-0-g4ca628d1
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

and flytectl:

{
  "App": "flytectl",
  "Build": "f2a1ad7d4",
  "Version": "0.9.5",
  "BuildTime": "2025-10-07 13:30:45.459309309 -0600 MDT m=+0.000898025"

Are there any suggestions anyone can provide to fix this? Also LMK if you need any other information.

Is there any way to configure retries in flytekit for metadata operations? We have some transient DNS failures we'd like to see if we can retry on (in addition to retrying system level failures in propeller when the task gives up).

Hey team, I just wanted to bring up that I noticed that Google no longer seems to be indexing the Flyte docs at https://www.union.ai/docs/v1/flyte/user-guide/ with the term “flyte”. For instance, in the search in the attached screenshot, I would expect it to find this ImageSpec page, instead of 3 legacy doc pages. I’m not sure who manages Flyte SEO indexing, but just wanted to bubble it up. Thanks!

Is the Flyte v2 UI available in the OSS version yet?

Hey, I saw someone in the past (over 90 days ago) had configured Flyte to run on an Openshift cluster. I was wondering if anyone would be willing to share the rolebindings that they used to successfully create this, as Flyte is asking for way more permissions than I currently have on openshift, and it would save me a lot of time and headache of trying to create a very limited scope myself if someone already suffered through this problem. Thanks 🙂

👋 I have several workflows that are dependent on the same external data, this external data appears at quasi regular intervals several times a day. I know when approximately I should wait for the next chunk of external data. What approach is best to make sure that polling is "deduplicated", i.e. that I don't query external service too often? Currently the pipeline is being run in luigi with a separate trigger application that performs the waiting only when the external data is most likely to appear and triggers the appropriate workflows. Obviously this can be reused with flyte but I was hoping that triggering can be implemented inside flyte too. The approaches I have in mind are the following: 1. create a special waiting task with caching to make sure that only one of these tasks is being run at the same time and once the data appears this fact will be retrieved from cache by different workflows 2. create a special sensor that somehow prevents making external requests for the same data too often (maybe creating a singleton state class for these sensor tasks would work?) What is the best solution for this? Perhaps I'm missing some other, simpler approach?

Hello. Need some help as our production env web broke for some reason. Last I remember I tried registering a workflow which registered successfully and a few mins later when I tried running it - it shows this error. Can someone lead me to what could be the issue? I asked my other colleagues - there was no operation on the cluster during that time. We have flyte deployed on our GCP k8s cluster.

I'm attempting to narrow down a bug in Flyte v1. When we do nothing but add one our private internal packages to a uv.lock that we pass into a workflow that makes use of map_task we get the following error (I'll post the full stacktrace in the thread):

ModuleNotFoundError: No module named 'site-packages.flytekit'

This causes a change down which seems to affect the container args (I'm not sure what these are), but we get this because if I change the dependencies on the image but not the tasks in the workflow I get an error asking me to re-register the task:

/template/Target/Container/args/23:
	- flytekit.core.python_auto_container.default_task_resolver 
	+ site-packages.flytekit.core.python_auto_container.default_task_resolver

... I believe the error "No module named 'site-packages.flytekit'" occurs because load_object_from_module() gets passed *site-packages.*flytekit.core.python_auto_container.default_task_resolver and not flytekit.core.python_auto_container.default_task_resolver but I don't know what leads to this import path being passed in. I'll post a minimal working example package that causes the break once I've factored one out, but does anyone know what's happening here?

Hi all! I'd like to submit a PR for something I found in the Flyte 2 documentation but not sure where the docs are in the source? I've contributed before but it's probably been a couple of years. Thanks!

occasionally my my workflow fails when a task gets this error:

/1] currentAttempt done. Last Error: USER::Grace period [3m0s] exceeded|containers with unready status: [primary]|Back-off pulling image "<http://mycompany.com/buck/horse:19.2.0|mycompany.com/buck/horse:19.2.0>": ErrImagePull: failed to pull and unpack image "<http://mycompany.com/buck/horse:19.2.0|mycompany.com/buck/horse:19.2.0>": failed to copy: read tcp 100.64.44.213:38748->10.37.121.189:443: read: connection reset by peer

this seems like a random node-related issue, is the best solution here to have all my tasks have retries set? (and is there a way to do this at the flyte level rather than changing all invocations at the flytekit level?)

Hi everyone, I am a new Flyte user and happy to join the Slack channel ! I noticed that in the v1 documentation on flyte.org site, some pages refer to other doc pages with broken links. For example, the page Data Input/Output links to https://docs.flyte.org/en/latest/concepts/data_management.html which no longer exists.

👋 Quick q -- On the Flyte 2 SDK changes -- are they a full break from Flyte 1 and require the Flyte 2 backend. Or are they intended to work with both Flyte 1 and Flyte 2?

❔I'm looking for some help with setting up Okta auth with flyte-core and an external authorization server following this doc. As I understand it (please correct me if I'm wrong), there are 4 main types of authentication to cover in a Flyte setup: 1. Human browser (HTTP) 2. Human CLI (GRPC) 3. Machine to machine (GRPC, external to k8s networks, e.g. CI systems) 4. Service to service (GRPC, within k8s network) Is there a way currently to enable 1,2 and 3 but disable 4 and rely solely on k8s networking policies to verify incoming traffic? Or maybe I need enable

selfAuthServer

for only

flytepropeller

to use the internal authorization server in just scenario 4? (Using flyte-core v1.16.1 in AWS EKS). Thanks in advance!

Hi, I'm trying to have an input parameter of type callable (python function) to the workflow (not task). When I register my workflow and open it on UI, I see the following inputs for the callable function param

custom_func

that I need to fill in. Can you please explain what file here means? is it a URL to the pickled python function stored somewhere where flyte can access it?

Hi, I would like to try Flyte 2.0 on a local sandbox cluster. Is there a way to create it? I tried

uv run --prerelease=allow flytectl demo start

but it started installing Flyte 1.16.1

Going to use Flyte v1.16.1

. Is it possible to do it?

Hi, I am facing issue when I submit a workflow to flyte which is setup in my local kubernetes following the helm installation guide. The execution pod is having the error. Is there anything I have missed out?

Reason:   Error                                                                                                                                                                                   │
│       Message:  y", line 1246, in invoke                                                                                                                                                                │
│     return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                      │
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                      │
│   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 814, in invoke                                                                                                                     │
│     return callback(*args, **kwargs)                                                                                                                                                                    │
│            ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                    │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/bin/entrypoint.py", line 773, in fast_execute_task_cmd                                                                                         │
│     _download_distribution(additional_distribution, dest_dir)                                                                                                                                           │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/core/utils.py", line 309, in wrapper                                                                                                           │
│     return func(*args, **kwargs)                                                                                                                                                                        │
│            ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                        │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/tools/fast_registration.py", line 310, in download_distribution                                                                                │
│     FlyteContextManager.current_context().file_access.get_data(                                                                                                                                         │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/utils/asyn.py", line 113, in wrapped                                                                                                           │
│     return self.run_sync(coro_func, *args, **kwargs)                                                                                                                                                    │
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                    │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/utils/asyn.py", line 106, in run_sync                                                                                                          │
│     return self._runner_map[name].run(coro)                                                                                                                                                             │
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                             │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/utils/asyn.py", line 85, in run                                                                                                                │
│     res = fut.result(None)                                                                                                                                                                              │
│           ^^^^^^^^^^^^^^^^                                                                                                                                                                              │
│   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 456, in result                                                                                                                     │
│     return self.__get_result()                                                                                                                                                                          │
│            ^^^^^^^^^^^^^^^^^^^                                                                                                                                                                          │
│   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result                                                                                                               │
│     raise self._exception                                                                                                                                                                               │
│   File "/usr/local/lib/python3.12/site-packages/flytekit/core/data_persistence.py", line 614, in async_get_data                                                                                         │
│     raise FlyteDownloadDataException(                                                                                                                                                                   │
│ flytekit.exceptions.system.FlyteDownloadDataException: SYSTEM:DownloadDataError: error=Failed to get data from <s3://flyte-dev/flytesnacks/development/GJD6GFARVQLFIK64SWWSTFCM3I======/fast28f7e9edccfe> │
│ 6ce9484363d126d1965a.tar.gz to /root/ (recursive=False).                                                                                                                                                │
│                                                                                                                                                                                                         │
│ Original exception: Unable to locate credentials

Hi! How do I set up authentication? Or - Keycloak? With

flyte-binary

. I'm struggling with authentication set up of

flyte-binary

Helm chart version 1.16.1 against Keycloak 23.0.6. The

values.yaml

is below. In general, may be (I'm not sure), my question is about what the exact requirements are for Flyte, in order to set up the requirements in Keycloak (KC) of the version. Certain scopes, for example, are absent:

offline

is

offline_access

,

all

is absent in KC and so on. I'm looking into "Configuring authentication" - https://www.union.ai/docs/v1/flyte/deployment/flyte-configuration/configuring-authentication/ I set the things up for web interface and

flytectl

to work. I hope I understood what a things should be set up from Keycloak perspective concerning

audience

, for example. At leats web interface and

flytectl

works. However, an executions are submitted successfuly, but unable to start. Flytepropeller (may be it is Propeller) complains on auth. with logs producing speed 0,5+ Mb/second. At least I have in logs:

{"json":{"src":"interceptor.go:22","x-request-id":"a-fmktn2s6wf5f6rz7n5rb"},"level":"debug","msg":"authenticated user doesn't have required scope","ts":"2025-10-23T14:01:36Z"}
{"json":{"exec_id":"a5pwkw4w46snvz6xgvhj","ns":"flyteprj0005-development","res_ver":"217278987","routine":"worker-12","src":"auth_interceptor.go:213","wf":"flyteprj0005:development:workflows.hello_world.hello_world_wf"},"level":"debug","msg":"Request failed due to [rpc error: code = Unauthenticated desc = authenticated user doesn't have required scope]. If it's an unauthenticated error, we will attempt to establish an authenticated context.","ts":"2025-10-23T14:01:36Z"}
{"json":{"exec_id":"a5pwkw4w46snvz6xgvhj","ns":"flyteprj0005-development","res_ver":"217278987","routine":"worker-12","src":"auth_interceptor.go:236","wf":"flyteprj0005:development:workflows.hello_world.hello_world_wf"},"level":"debug","msg":"Request failed due to [Unauthenticated]. Attempting to establish an authenticated connection and trying again.","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"token_source_provider.go:257"},"level":"info","msg":"Fetched new token with expiry 2025-10-23 14:13:44.083157411 +0000 UTC m=+3572.209271697","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"token_source_provider.go:264"},"level":"info","msg":"retrieved token with expiry 2025-10-23 14:13:44.083157411 +0000 UTC m=+3572.209271697","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"token_utils.go:31"},"level":"debug","msg":"Token expiry : 2025-10-23 14:13:44.083157411 +0000 UTC m=+3572.209271697, Access token expiry : 2025-10-23 14:13:44 +0000 UTC, Are the equal : false","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"token_source_provider.go:238"},"level":"info","msg":"retrieved token from cache with expiry 2025-10-23 14:13:44.083157411 +0000 UTC m=+3572.209271697","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"handlers.go:271"},"level":"debug","msg":"Found existing metadata header flyte-authorization","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"handlers.go:299"},"level":"debug","msg":"Running authentication gRPC interceptor","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"handlers.go:254"},"level":"debug","msg":"gRPC server info in logging interceptor [8d666489-6d2f-48e6-b020-9347945c991b]method [/flyteidl.service.AdminService/CreateWorkflowEvent]\n","ts":"2025-10-23T14:01:36Z"}
{"json":{"src":"interceptor.go:22","x-request-id":"a-nbdd4tx7w6tr8xzk88w6"},"level":"debug","msg":"authenticated user doesn't have required scope","ts":"2025-10-23T14:01:36Z"}

My

values.yaml

for the Helm chart is below:

---

userSettings:
  hostName: "flyte"

configuration:
  logging:
    level: 6
    show-source: true
  database:
    username: "flyte_control_plane"
    password: "helm --set configuration.database.password=psql_pw"
    host: "postgres.test.daiger.ru"
    dbname: "flyte_control_plane"
  storage:
    type: "minio"
    metadataContainer: "flyte"
    userDataContainer: "flyte"
    provider: "s3"
    providerConfig:
      s3:
        region: "eu-west-1"  # Irrelevant for local but still needed.
        authType: "accesskey"
        endpoint: "<https://minio-api.test.k8s.daiger.ru:443>"
        accessKey: "helm --set configuration.storage.providerConfig.s3.accessKey=AWS_ACCESS_KEY_ID"
        secretKey: "helm --set configuration.storage.providerConfig.s3.secretKey=AWS_SECRET_ACCESS_KEY"
        disableSSL: "true"
        secure: "false"
  auth:
    enabled: true
    oidc:
      baseUrl: "<https://keycloak.daiger.ru/realms/daiger>"
      clientId: "flyteadmin_test"
      clientSecret: "helm --set configuration.auth.oidc.clientSecret=FLYTE_CREDENTIALS_OIDC_CLIENT_SECRET"
    internal:
      clientId: "flytepropeller_test"
      clientSecret: "helm --set configuration.auth.internal.clientSecret=FLYTE_CREDENTIALS_INTERNAL_CLIENT_SECRET"
      clientSecretHash: "helm --set configuration.auth.internal.clientSecretHash=FLYTE_CREDENTIALS_INTERNAL_CLIENT_SECRET_HASH"

    authorizedUris:
    - "<https://flyte.test.k8s.daiger.ru>"

  inline:
    auth:
      appAuth:
        authServerType: "External"
        externalAuthServer:
          baseUrl: "<https://keycloak.daiger.ru/realms/daiger>"
          metadataUrl: ".well-known/openid-configuration"
        thirdPartyConfig:
          flyteClient:
            clientId: "flytectl_test"
            redirectUri: "<http://localhost:53593/callback>"
            scopes:
            - "offline_access"
            - "offline"
            - "all"

            audience: "<https://flyte.test.k8s.daiger.ru>"
      userAuth:
        openId:
          baseUrl: "<https://keycloak.daiger.ru/realms/daiger>"
          scopes:
          - "profile"
          - "openid"
          clientId: "flyteconsole_test"

ingress:
  create: true
  ingressClassName: "nginx"
  commonAnnotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"

    nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
    nginx.ingress.kubernetes.io/proxy-buffers: "4 16k"
    nginx.ingress.kubernetes.io/proxy-busy-buffers-size: "32k"
    nginx.ingress.kubernetes.io/large-client-header-buffers: "8 16k"

  httpAnnotations:
    nginx.ingress.kubernetes.io/app-root: "/console"
  grpcAnnotations:
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  host: "flyte.test.k8s.daiger.ru"
  separateGrpcIngress: true
  tls:
  - secretName: "tls-flyte-test-k8s-daiger-ru"
    hosts:
    - "flyte.test.k8s.daiger.ru"

serviceAccount:
  create: true
  name: "flyte"
  imagePullSecrets:
    - name: "image-pull-secret"

What are a requirements to be provided from Keycloak perspective??? Or is there an evident error in the materials above?

dumb q: if i want to backup / restore the flyte postgres db, do just treat it like any old postgres DB that's used by a webapp? like i can just pg_dump / pg_restore? (ref https://flyte-org.slack.com/archives/C06H1SFA19R/p1761615884646699?thread_ts=1761615882.693109&cid=C06H1SFA19R )

sometimes we pass around FlyteDirectories and FlyteFiles that need to be fully downloaded to specific locations for cli applications to use. Are there builtin functions to do this sort of thing?

❔ Does anyone know how to delete local credentials created with the CLI PKCE auth, to force another log in while testing? TIA 🙏

Hey folks.. Does flyte have the ability to ensure that a given workflow can have only 1 instance running, irrespective of inputs (i.e. I can’t rely on caching)

Hi! currently, I am trying to use flyteconnector bigquery. However, after trying to open StructuredDataset using

bq_template = BigQueryTask(
    name="<name>",
    inputs={},
    query_template="SELECT * FROM <project_id>.<dataset_id>.<table>",
    output_structured_dataset_type=StructuredDataset,
    task_config=BigQueryConfig(ProjectID="<project_id>"),
)
@task(
    container_image=image_name,
)
def convert_bq_table_to_pandas_dataframe(ds: StructuredDataset) -> pd.DataFrame:
    return ds.open(pd.DataFrame).all()

@workflow
def full_bigquery_wf() -> pd.DataFrame:
    ds = bq_template()
    return convert_bq_table_to_pandas_dataframe(ds=ds)

So, what happen is when the bigquery task query data from bq it uses flyteconnector service account but after that when the python task try to extract pandas dataframe it is unable to do so.

google.api_core.exceptions.PermissionDenied: 403 Access Denied: Dataset <project_id>:<job_id>: User does not have permission to access results of another user's job.

I have already deploy flyteconnector and enable plugin as documentation mentioned. Any help would be greatly appreciate :).

Hi could you please give an example of fetch a workflow from the cluster and create and register a launch plan for this fetched workflow? Would it possible?