https://linen.dev logo
Join Slack
Powered by
bumping this again. Thanks in advance!
# replication-troubleshooting
g
bumping this again. Thanks in advance!
m
Hey Gaurav, thanks for the detailed follow-up! I'm not aware of an automatic backfill that backfills 
resource_requirements
on existing connections during a 1.6 to 2.0 upgrade. I think the workaround that you've used is correct for now. I'm looking internally to understand the migration logic better. Will get back to you when I have a clearer answer. Appreciate the patience 🙂
Quick update. We identified two issues: 1. Default behavior change in V2 Helm chart The V2 chart sets 
useConnectorResourceDefaults: true
by default. In V1, this was effectively 
false
. When enabled, connector-specific resource requirements from the registry take effect, meaning some connectors request 2-4Gi of memory per pod. This is likely causing OOM errors after upgrading. 2. Known bug with CHECK and DISCOVER jobs Even with 
useConnectorResourceDefaults: false
, that flag is only respected for sync/refresh jobs. CHECK and DISCOVER operations still use connector-specific resource requirements regardless of the flag setting. Suggested workaround: Can't guarantee this fully resolves everything, but it should help. Set 
useConnectorResourceDefaults: false
and also set explicit resource limits for check and discover workloads:
Copy code
global:  
  workloads:  
    resources:  
      useConnectorResourceDefaults: false  
      check:  
        memory:  
          limit: "2Gi"  
          request: "2Gi"  
      discover:  
        memory:  
          limit: "2Gi"  
          request: "2Gi"
Adjust the memory values based on your cluster capacity. The CHECK/DISCOVER bug is tracked internally and we'll update when there's a fix.
g
Thanks mark. i did try disabling the flag but i did not have the check and discover section for OOM protection. I can def add that and see what it does. but in general i think the connection sync fails at the s3 destination container. the check and discover had succeeded which is generally before the replication pod/connection sync step
The only thing that allowed me to sync the connection is this
Copy code
{
  "ephemeral_storage_request": "5G",
  "ephemeral_storage_limit": "5G"
}
I was wondering if this value 5G is more than enough and how would this impact if 5G isn't sufficient?
m
Hey Gaurav, for most workloads, 5G should be enough unless you're syncing large tables or high-volume streams. If 5G isn't sufficient (if sync produces more staged data than limit), k8s might evict the pod and the sync will fail.
g
hi mark, could you also help me understand why the connector resource defaults are not getting adhered from the yaml files. i set these values
Copy code
replication:
        cpu:
          request: "1"
        memory:
          request: "4Gi"
but pretty much this gets ignored. is this a bug that is going to be addressed soon? I see these three issues. https://github.com/airbytehq/airbyte/issues/68162 https://github.com/airbytehq/airbyte/issues/72833 https://github.com/airbytehq/airbyte/issues/73346 I can do the workaround as suggested by setting the connection level resource requirements in the database but this requires a lot of manual intervention for us. is there something in plan to address. Also wanted to ask, i don't see any new platform tags on git. the latest is still 2.0.0. any news on when a newer tag will be released
2 Views
Open in Slack
PreviousNext
Powered by