Hello! Does anyone know why syncs stopped working (workloads are not being created) when using GCS as storage?

storage:
    type: gcs
    secretName: airbyte-config-secrets
    bucket:
      log: ba-airbyte-bucket
      state: ba-airbyte-bucket
      workloadOutput: ba-airbyte-bucket
    gcs:
      projectId: bearded-analytics
      credentialsJsonSecretKey: gcp.json

Airbyte deployment is successful, old job logs are fetched just fine. Its that new ones are not being created and it's not possible to get their logs

rootCauseExceptionStack: [io.airbyte.commons.server.errors.IdNotFoundKnownException: Could not find attempt stats for job_id: 13978 and attempt no: 0, 	at io.airbyte.commons.server.handlers.AttemptHandler.getAttemptCombinedStats(AttemptHandler.kt:251),

I would really appreciate any help

I'm sorry, but there is no definitive solution in the knowledge sources for this exact error, but several documents point to common configuration issues when migrating to GCS storage in Helm v2 that cause similar symptoms (jobs not starting and 404/Not Found errors). ### Potential Causes and Solutions Based on the issues reported by other users with similar "IdNotFoundKnownException" and workload creation failures, you may want to check the following: * Missing Container Orchestrator Secret Configuration: A common reason why new workloads fail to start (while existing logs are still visible) is that the Container Orchestrator cannot access the GCS credentials. You may need to explicitly set the secret name and mount path for the orchestrator in your `values.yaml`:

yaml     global:       workloads:         containerOrchestrator:           secretName: "airbyte-config-secrets" # Ensure this matches your secret name           secretMountPath: "/secrets/gcs-log-creds"

[[[helm] GCP JSON not mounted](https://github.com/airbytehq/airbyte/issues/65153)] * Credential Path vs. Secret Key: While your configuration uses

credentialsJsonSecretKey

, some documentation for Helm v2 suggests using

credentialsJsonPath

instead, or ensuring that the case sensitivity for the storage type is correct: * Ensure

type: "gcs"

is lowercase in Helm v2 [[Container launcher not using airbyte-gcs-log-creds](https://github.com/airbytehq/airbyte/issues/68177)]. * The standard Helm v2 configuration often expects

credentialsJsonPath: /secrets/gcs-log-creds/gcp.json

[[State and Logging Storage Values](https://docs.airbyte.com/platform/deploying-airbyte/integrations/storage#values)]. * Case Sensitivity: In Helm v2, the

storage.type

field has been reported to be case-sensitive. Ensure it is set to lowercase

"gcs"

[[global.storage.secretName is not the correct key](https://github.com/airbytehq/airbyte/issues/48842)]. * Missing Stats Warnings: The error

Could not find attempt stats

often occurs because the job persistence layer cannot find records for a job that never successfully initialized or reported back. This is frequently a symptom of the orchestrator failing to launch the pod because it lacks valid storage credentials [[Missing stats for job warning](https://discuss.airbyte.io/t/4016); [All sync jobs hanging indefinitely](https://github.com/airbytehq/airbyte/issues/55812)]. If you recently upgraded Airbyte, one user noted that skipping minor versions (e.g., 0.40.0 to 0.40.4) caused

IdNotFoundKnownException

and recommended stepping through each minor version during the upgrade [[Update to 0.40.4 - all syncs fail](https://discuss.airbyte.io/t/2471)].

Doesnt work at all...