Hi Team,
We had a few questions regarding upgrading from v1 helm charts(1.6) to v2 helm charts(2.0.0)
We are running Airbyte 2.0.0 on Kubernetes and found an issue with existing/migrated connections whose connection.resource_requirements are missing or incomplete.
When connector resource defaults are enabled, we expected existing connections without explicit connection-level resource settings to inherit safe defaults. Instead, some existing v2 connections had missing ephemeral_storage_request / ephemeral_storage_limit values, and sync execution ended up producing invalid runtime env values such as:
CONNECTOR_STORAGE_LIMIT_BYTES=null
This caused replication jobs to fail and specifically the s3-destination connector.
We want to confirm whether this is expected, whether existing connections should be automatically backfilled, and whether emitting literal null runtime env values is a bug.
Environment
Airbyte version: 2.0.0
Deployment type: Kubernetes
Packaging: Helm chart v2
Database backend: PostgreSQL
Upgrade path: Airbyte 1.6.x to 2.0.0
Deployment style: v1 and v2 side-by-side during migration
Observed behavior
Some existing v2 connections had missing or incomplete resource_requirements in the config database.
When running syncs for those connections:
workloads were created with missing/blank ephemeral storage values
the runtime env ended up with CONNECTOR_STORAGE_LIMIT_BYTES=null
replication jobs failed
This appeared to affect existing/migrated connections rather than only newly created ones.
What we found:
For affected connections, connection.resource_requirements did not contain valid values for:
ephemeral_storage_request
ephemeral_storage_limit
We expected one of these to happen:
• Airbyte migration logic populates those fields for existing connections
• connector resource defaults are applied dynamically at runtime for existing connections
• missing values are omitted safely rather than converted into literal null env vars
Instead, we saw a failure path where missing values propagated into the workload runtime.
Temporary workaround
We were able to work around the issue by explicitly setting connection-level values in the config DB, for example:
{
"ephemeral_storage_request": "5G",
"ephemeral_storage_limit": "5G"
}
After those values were present, the storage-related failure stopped occurring for the affected connections.
Expected behavior
Our expectation is that Airbyte should not fail a sync because an existing connection lacks explicit resource_requirements.
One of the following should happen automatically:
• existing connections are backfilled during migration / upgrade
• existing connections inherit default connector resource settings at runtime
• missing values are treated as absent and not rendered as invalid null env vars
Questions:
• During a 1.6 -> 2.0 migration, is connection.resource_requirements expected to be automatically populated for existing connections?
• If connector resource defaults are enabled, should existing connections without explicit resource_requirements inherit those defaults automatically?
• Is it expected that missing storage values can become literal runtime env vars like CONNECTOR_STORAGE_LIMIT_BYTES=null, or is that a bug?
• Is there an official Airbyte migration or repair step for normalizing/backfilling resource_requirements on existing connections?
• Are ephemeral_storage_request and ephemeral_storage_limit required for all existing v2 connections in Kubernetes deployments?
Impact
This issue blocked successful sync execution for migrated/existing connections in Airbyte 2.0 and required manual DB-level remediation to proceed.
Thanks for any insight!