Hi all #CP2HDHKE1 I have set up a Flyte in GCP with Google’s OAuth2 Client Credentials successfully. I am able to access everything. It just asks for your google account as expected, you click your google account and it lets you in as expected. This is the config normal users use:

admin:
  endpoint: dns:///flyte.somedomain.com
  insecure: false
logger:
 show-source: true
 level: 0
storage:
  type: stow
  stow:
    kind: google
    config:
      json: ""
      project_id: brain-project # GCP Project ID
      scopes: <https://www.googleapis.com/auth/devstorage.read_write>
  container: bucket-flyte

Now, I want to trigger a workflow from a backend that is running inside the same cluster as Flyte, but I can’t figure out how I can get

flytectl

to connect and trigger the workflow. Among other things, I tried several flyte configs, but I always get some type of error. E.g.

Response: {"error":"invalid_client","error_description":"Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)."}

Does anyone know how I could achieve this? What config could I use? Any help is much appreciated 🙏

Hello, we have been keeping an eye on Flyte since a while and decided to run a POC today, but noticed that the GCP deployment page on Flyte docs has been changed - it no longer has the detailed steps that were there earlier. Is it possible to get those detailed instructions back?

I’m trying to run the above gcp terraform. I got these three errors. I’m not sure why it didn’t find the flyte namespace, I was able to point to it with

kubectl

. And I can’t find those two bucket names to know where to change them. Any ideas?

╷
│ Error: namespaces "flyte" not found
│
│   with kubernetes_secret.flyte-tls-secret,
│   on <http://ingress.tf|ingress.tf> line 40, in resource "kubernetes_secret" "flyte-tls-secret":
│   40: resource kubernetes_secret "flyte-tls-secret" {
│
╵
╷
│ Error: googleapi: Error 409: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again., conflict
│
│   with module.flyte_data.google_storage_bucket.buckets["flyte-gcp-data"],
│   on .terraform/modules/flyte_data/main.tf line 40, in resource "google_storage_bucket" "buckets":
│   40: resource "google_storage_bucket" "buckets" {
│
╵
╷
│ Error: googleapi: Error 409: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again., conflict
│
│   with module.flyte_user_data.google_storage_bucket.buckets["flyte-gcp-user-data"],
│   on .terraform/modules/flyte_user_data/main.tf line 40, in resource "google_storage_bucket" "buckets":
│   40: resource "google_storage_bucket" "buckets" {

Hi, new question, (sorry I’m having so many issues haha) I’m trying to get docker image pulling/pushing working. I’m getting this error right now:

Failed to check if the image exists with error : Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

Here is my ImageSpec:

misc_image_spec = ImageSpec(
    base_image="python3.10",
    packages=["pendulum==2.1.2"],
    env={"Debug": "True"},
    registry="<location>-docker.pkg.dev/<project>/<repository>",
)

And I took the “patch the default service account” path. I created a service account with access to the Artifact Registry, downloaded the json key, created the secret and patched the default service account with that

imagePullSecrets

with the key. However, after doing that I’m getting the same error. I’m not sure if the error is even auth related, or something else. Docker is running fine locally, as a note Edit: It seems like an issue with pyflyte interacting with my local docker, maybe? I’m not sure why I’m having so many docker issues with flyte..

Hello everyone: I'm happy to confirm that the reference implementation for GCP is now merged! https://github.com/unionai-oss/deploy-flyte/tree/main/environments/gcp/flyte-core It aims to incorporate learnings that community members like @elegant-toddler-67101 @kind-kite-58745 and others have shared here. Huge thanks to @cool-lifeguard-49380 for the detailed and careful reviews; it made the result much better. These Terraform modules can take you from an empty GCP project to a fully working Flyte environment in a matter of minutes 😎 If you find problems using them or you have ideas to make them better, feel free to create an Issue or submit a PR!

Hi, I am running into an error when trying to run the BigQuery Query example. The

no_io_wf

workflow is running fine, however, the

full_bigquery_wf

is failing on the

Python Task

with the following error:

ModuleNotFoundError: No module named 'flytekitplugins.bigquery'

Hi again, has anyone been able to connect to Artifact Registry? I have tried giving the

flyte-binary

and

flyte-worker

service accounts permission, and creating an

imagePullSecret

for both

default

and

flyte-binary

serviceaccount on the cluster, but still getting a

403 Forbidden

Error when the task is trying to pull the image

I’ve redeployed a

flyte-core

environment now, using the guide here, I’m now trying to setup the authentication using the guide here but after the following the steps I’m not getting any prompt to authorise from the UI. I noticed some indentation conflicts between the yaml in the deployment guide and authorisation guide, and had a similar issue with authorising

flyte-binary

due to an indentation issue, I’m not sure if it’s indentation causing issues again here, but any help would be greatly appreciated, thanks

Hello everyone, really cool to see everything happening with this project! My shop is currently on GCP, and I am trying to deploy Flyte to our k8s cluster running there. There is very good reference implementation here, for spinning everything up from scratch, but I am not so sure about using our already provisioned resources. If possible, could anyone please elaborate on what one would need to modify or change in order to do this? Thank you in advance! 🙏

Hi all, I was wondering if there was any documentation or examples for setting up Slack Notifications for workflows on a GCP deployment?

Hey! Possibly a stupid question about deploying flyte on GCP. After applying the terraform manifests successfully, is there a way to get started with flyte before setting up the A DNS record? I'm probably missing something very obvious, but when I try to reach the ingress directly, I get 404s

Hey! Trying to run a simple job using a non-default docker image (containing our dependencies), made after the image in flytekit. When I don't specify an image (and running the default one), the job successfully works -- it manages to write data to GCS to pass it between tasks. When I'm passing a custom image with additional dependencies, I'm getting 401 from GCS without container reaching my code. It feels like there is something wrong/off with the workload identity, but I'm not 100% sure. Any ideas about how to debug it efficiently?

Hi! I’m wondering what the default service account for a task will be, or how to find this out? I’m trying to run some `BigQueryTask`’s using the BigQuery plugin, but keep getting a permission error… I’ve given BigQuery permission to the flytepropeller and flyteworker service accounts, but to no avail. Anyone able to help?

https://github.com/flyteorg/flytekit/tree/master/plugins/flytekit-identity-aware-proxy#configuring-your-flyte-deployment-to-use-iap is there a way to configure flyte with ssl so that a Istio service mesh is not required for IaP?

Hi, does anyone happen to have a GCP version of eks-starter.yaml to use for a single cluster deployment?

how can i setup ci/cd auth for the pyflyte register command if i am using GCP oidc?

Hello, I'm configuring the token generation CLI according to these instructions but I'm unable to run commands like

flytectl get projects

. I get the error described in this comment, which states that it is due to

flytectl

not understanding the

proxyCommand

. Is there now a workaround for this error? I've looked but haven't been able to find one. I'm using

flytectl 0.8.14

Repost from: https://flyte-org.slack.com/archives/C01P3B761A6/p1709746035191139 Got a couple question on how to deploy flyte on GCP. Similar to https://flyte-org.slack.com/archives/C05A0JA1CCD/p1708700222304899

Hello, do we have any reference of how to configure Spark plugin + k8s connector on GKE ? i can see only aws related doc here https://docs.flyte.org/en/latest/deployment/plugins/k8s/index.html#deployment-plugin-setup-k8s

I'm running flyte on GKE with nginx ingress, and am currently migrating auth to Okta (I previously had auth working with Auth0). I followed all the steps in the Authentication setup for setting up flyte-core external auth server with Okta. Console auth + flytectl are working, and all pods are running without except for flytescheduler, which gives a somewhat ambiguous error in the flytescheduler-check init container:

panic: rpc error: code = Unauthenticated desc = token parse error [JWT_VERIFICATION_FAILED] Could not retrieve id token from metadata, caused by: rpc error: code = Unauthenticated desc = Request unauthenticated with IDToken

I've triple checked the flytepropeller setup for auth with client credentials in Okta and haven't been able to resolve the issue. I figured i'd ask if anyone has any pointers for troubleshooting this. Thanks!

We're trying to register workflows from within a Docker container using IAP. Previously, we used port-forwarding. We're able to register workflows using IAP from a local virtual environment, but when we try from within a container, the process hangs indefinitely. We saw a similar issue in this thread, but we would like to register workflows from a user account instead of a service account. In our Google Cloud Console logs for the load balancer, we see permission denied errors and no data under

authenticationInfo

when attempting to register from the container. Contrastingly, successful registration attempts from outside of the container show

principalEmail

with my email under

authenticationInfo

. We're also unable to run the script from step 8 of the configuring IAP instructions inside the container — this hangs indefinitely as well. Does anyone have suggestions to get this working?

Hey, this is probably a duplicate question but i was not able to find a solution from previous questions here. I have flyte-core deployed on gke with the gcp oidc/consent screen auth enabled and nginx. This works for local commands and browser authentication just fine. Now i try to write a python API that can execute workflows in the cluster. (API is not running in the cluster) How do i manage the authentication here ? I found https://pypi.org/project/flytekitplugins-identity-aware-proxy/ but that would mean switching ingress etc. I also tried

export FLYTE_CREDENTIALS_CLIENT_ID=GCP-OAUTH-ID
export FLYTE_CREDENTIALS_CLIENT_SECRET=GCP-OAUTH-SECRET
export FLYTE_CREDENTIALS_AUTH_MODE=basic

but i couldn't get this to work Thx in advance.

hello! I’m wondering if the GCP crowds have attempted or entertained the idea of using Spanner (with its native PG dialect). Any experiences or informed prognostications?

I am using GCP artifact registry what kind of configuration values does this

registry_config

take in the image spec . I can't seem to pull the images from GCP artifact registry

@cool-lifeguard-49380 I wanted to confirm what's the support status of using Google's IAP with Flyte? I see your contributions but just wanted to confirm what would be the limitations for users going that route? Thanks!

Hey folks! A question: I'm following the guide here: https://github.com/unionai-oss/deploy-flyte/blob/main/environments/gcp/flyte-core/README.md. I've followed the steps as stated; there is a DNS record configured and I can reach the console in browser. However, the grpc traffic doesn't seem to go through. The only suspicious thing I've found so far is the following: the certificate challenge is pending (for about 2 days now) and seems like the challenge is able to hit the ingress, however the acme-http-solver service sees errors like:

"cert-manager/acmesolver: validating request" host="" path="*" base_path="." token="*"                                                                                                                          
"cert-manager/acmesolver: invalid base_path" host="" path="*" base_path="." token="*" expected_base_path="/.well-known/acme-challenge"

I've tried skipping the ssl (as described in the docs linked above), but to no success. Can certificate challenge be the culprit? Or am I looking at the wrong place alltogether?

This seems a better channel for my problem. For the dns-domain, I use the comanyname.com . Not sure if it needed, but I can also create DNS server on GCP.

hey, not sure if this has been answered before but I can't search for anything older than 90 days, but have some questions about setting up gcp, I have everything working, but I have not set up the certs for the domain, I want to be able to connect directly while in the cluster for now but am having issues connecting with flytectl

Has anyone tried getting GCP load balancer working with Flyte? I had a hard time finding code samples. I found this in Flytekit but they're using Istio. I'm trying to see if there's a way to just configure the ingress without Istio.

Hello Flyte Community. This is Vlado from Akvelon. We're collaborating with the Google Kubernetes Engine (GKE) team on efforts to streamline the integration of open-source AI frameworks with GKE, ultimately enabling organizations to achieve scalable AI deployments Flyte is a workflow automation platform for complex, mission-critical data and ML processes at scale. It has reusable, type-safe, and testable building blocks to construct workflows, orchestration for both real-time and batch processes and scalability from your laptop to thousands of concurrent executions. You can read more and try it out in this Guide We would love to connect and hear your insights on the features that you feel we should prioritize to benefit both the Flyte community and GKE customers.