https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# replication-troubleshooting
s
This message was deleted.
k
The knowledge sources do not explicitly mention a specific fix for the combination of your 
values.yaml
configuration and the observed "ResourceConstraintException." However, based on the provided logs and documentation, here is an analysis of your issue: ### 1. Resource Constraint Failure The primary error in your logs is: 
io.airbyte.workers.exception.ResourceConstraintException: Unable to start the REPLICATION pod. This may be due to insufficient system resources.
Your logs show the following requested resources for the replication pod: * Init Container: 2Gi Memory, 1 CPU * Source: 1Gi Memory, 0.2 CPU * Destination: 1Gi Memory, 0.2 CPU * Orchestrator: 2Gi Memory, 1 CPU Even though you have configured "Economy Mode" in your 
values.yaml
with lower requests (e.g., 
replication
at 256Mi and 
mainContainer
at 100m CPU), the logs indicate the platform is still attempting to launch containers with much higher requirements (up to 2Gi memory and 1 CPU per container). This suggests your overrides in 
workloads.resources
might not be applying correctly or are being overridden by connector-specific defaults [[Forum Post: Kubernetes - Bigquery sync failed](https://discuss.airbyte.io/t/1222)]. ### 2. Facebook Marketing Performance Considerations If you successfully resolve the resource issue but find syncs are extremely slow or return 0 records initially, consider the following: * Rate Limiting: Facebook heavily throttles API tokens by default. It is recommended to use a Long-Lived Page Token to avoid syncs taking an excessive amount of time or failing due to rate limits [[Documentation: Facebook Pages Performance](https://docs.airbyte.com/integrations/sources/facebook-pages#performance-considerations)]. * Ad Account Scaling: You mentioned having ~70 ad accounts. If the orchestrator cannot manage the volume of requests within the allocated resources, the sync may hang or fail to process records [[Github Issue: Facebook marketing data sync issue](https://github.com/airbytehq/airbyte/issues/49977)]. ### Suggested Steps * Verify Resource Availability: Ensure your Oracle ARM instance (4 vCPU / 24GB RAM) has enough allocatable capacity. If other pods (Postgres, Temporal, etc.) are consuming memory, there may not be enough room for the replication pod's total request (which appears to be >4Gi for a single sync). * Check Override Application: Double-check if 
useConnectorResourceDefaults: false
is functioning as expected in your Airbyte version. The logs showing 
limits={memory=2Gi, cpu=1}
for the 
orchestrator
contradict your 
values.yaml
setting of 
limit: "1000m"
and 
limit: "2Gi"
. * Check for Internal Errors: The logs show a gap between the launch (11:39) and the post-replication attempt (12:10). If no records are being moved, it may be due to the source failing to initialize properly due to the resource constraints [[Github Issue: Ad Creative Is Incomplete](https://github.com/airbytehq/airbyte/issues/66027)].
A message from kapa.ai
The knowledge sources do not explicitly state exactly why your specific 
values.yaml
is being ignored, but they do highlight several known issues and patterns regarding resource configuration in Airbyte on Kubernetes. ### 1. Potential Issues with the 
values.yaml
Configuration Based on the provided GitHub issues, there are a few reasons why your "Economy Mode" settings might not be applying: * Connector-Specific Overrides in the Database: Even if you set 
useConnectorResourceDefaults: false
, resource requirements stored in the internal Airbyte database (
actor_definition
table) can take precedence. A known issue indicates that these persisted definitions may continue to override global workload settings, forcing the pod to use defaults (like the 1 CPU/2Gi memory limits seen in your logs) [[Github Issue: workload resource requests and limits not applied for replication jobs](https://github.com/airbytehq/airbyte/issues/68162)]. * Variable Name Mismatches: In some versions of the Helm chart, the platform expects specific environment variable names like 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
. If the chart version does not correctly map the 
workloads.resources
hierarchy to these internal variables, the settings are ignored [[Documentation: Platform: 2.0: Configuring Connector Resources](https://docs.airbyte.com/platform/2.0/operator-guides/configuring-connector-resources); [Github Issue: Cannot use Airbyte on Kubernetes due to OOM when setting up connector](https://github.com/airbytehq/airbyte/issues/48816)]. * CPU Request Ignoring Bug: There is a documented bug where if you only configure a 
request
without a corresponding 
limit
(or vice versa in some cases), the configuration is ignored and defaults are used instead [[Github Issue: Jobs Resources Support Setting Only Request](https://github.com/airbytehq/airbyte/issues/60809)]. ### 2. Facebook Marketing Data and Sync Progress Regarding the fact that you see "0 records loaded" despite having approved permissions and a system account: * Insufficient Initial Sync Time: Your logs show the replication pod failed with a 
ResourceConstraintException
. If the pod cannot properly start or is throttled immediately due to resource limits, it will fail to initialize the source connector. If the source cannot initialize, it cannot begin pulling records, resulting in 0 bytes transferred [[Github Issue: workload resource requests and limits not applied for replication jobs](https://github.com/airbytehq/airbyte/issues/68162)]. * Resource Exhaustion During Discovery: For a source with ~70 ad accounts, the "Discover" or "Check" phase requires sufficient memory to map the schema of all accounts. If the pod is restricted by the platform to lower resources than it actually needs to hold that metadata, it may exit silently or fail before the first record is emitted [[Github Issue: [source-facebook-marketing] Refresh schema and new connection fail due to timeout](https://github.com/airbytehq/airbyte/issues/53132)]. ### Recommendation for Resource Configuration To force the settings if the Helm chart is being ignored, the documentation suggests manually updating the connector definition in the Airbyte database:
Copy code
sql update actor_definition  set resource_requirements = '{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "1", "cpu_request": "0.2", "memory_limit": "2Gi", "memory_request": "1Gi"}}]}'  where docker_repository like '%source-facebook-marketing';
[[Documentation: Platform: 2.0: Configuring Connector Resources](https://docs.airbyte.com/platform/2.0/operator-guides/configuring-connector-resources)]
2 Views
Open in Slack
PreviousNext
Powered by