https://linen.dev logo
Join Slack
Powered by
Failure in source: Please reduce the number of fie...
# replication-ask-ai
a
Failure in source: Please reduce the number of fields requested. Go to the schema tab, select your source, and unselect the fields you do not need. @kapa.ai
k
This error is coming from the source connector (for example, Facebook Marketing) because the request is asking for too many fields at once. The fix is to reduce the number of selected fields in the connection’s schema. To do that in Airbyte: 1. In the Airbyte UI, go to Connections and open the failing connection. 2. Click the Schema tab. You’ll see all streams and their fields. [Configuring schemas; Select fields] 3. Expand the affected stream(s) (for example, 
ads_insights
, 
ads_insights_action_type
, etc.) to show their fields. [Select fields] 4. Deselect (toggle off) any fields you don’t actually need to sync. By default, all detected fields are selected, so you’ll likely need to unselect many of them. [Select fields] 5. Click Save changes. If prompted with Stream configuration changed, choose to Refresh the affected streams so the destination is rebuilt consistently. [Select fields] 6. Then click Save connection and re‑run the sync. In the Facebook Marketing issue you’re seeing, the connector explicitly advises: “Please reduce the number of fields requested. Go to the schema tab, select your source, and unselect the fields you do not need.” [FB sync issue] If, after significantly reducing fields, the error persists, the knowledge sources don’t provide further troubleshooting steps beyond this field‑reduction approach. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
2026-02-27 191512 destination INFO main i.a.c.i.d.s.S3ConsumerFactory(onCloseFunction$lambda$8):176 Stream not successful with status INCOMPLETE for namespace: null, name: ads Skipping deletion of any old objects marked for deletion. 2026-02-27 191512 destination INFO main i.a.c.i.d.a.AsyncStreamConsumer(close):200 class io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer closed 2026-02-27 191512 destination INFO main i.a.c.i.d.a.AsyncStreamConsumer(close):212 Some streams either received an INCOMPLETE stream status, or did not receive a stream status at all: null.ads 2026-02-27 191512 destination ERROR main i.a.c.i.u.ConnectorExceptionHandler(handleException):68 caught exception! io.airbyte.commons.exceptions.TransientErrorException: Some streams were unsuccessful due to a source error. See logs for details. at io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer.close(AsyncStreamConsumer.kt:215) ~[airbyte-cdk-core-0.46.1.jar:?] at kotlin.jdk7.AutoCloseableKt.closeFinally(AutoCloseableJVM.kt:48) ~[kotlin-stdlib-2.0.0.jar:2.0.0-release-341] at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.kt:215) [airbyte-cdk-core-0.46.1.jar:?] at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:119) [airbyte-cdk-core-0.46.1.jar:?] at io.airbyte.cdk.integrations.base.IntegrationRunner.run$default(IntegrationRunner.kt:113) [airbyte-cdk-core-0.46.1.jar:?] at io.airbyte.cdk.integrations.base.adaptive.AdaptiveDestinationRunner$Runner.run(AdaptiveDestinationRunner.kt:68) [airbyte-cdk-core-0.46.1.jar:?] at io.airbyte.integrations.destination.s3.S3DestinationRunner.main(S3DestinationRunner.kt:15) [io.airbyte.airbyte-integrations.connectors-destination-s3.jar:?] Stack Trace: io.airbyte.commons.exceptions.TransientErrorException: Some streams were unsuccessful due to a source error. See logs for details. at io.airbyte.cdk.integrations.destination.async.AsyncStreamConsumer.close(AsyncStreamConsumer.kt:215) at kotlin.jdk7.AutoCloseableKt.closeFinally(AutoCloseableJVM.kt:48) at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.kt:215) at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.kt:119) at io.airbyte.cdk.integrations.base.IntegrationRunner.run$default(IntegrationRunner.kt:113) at io.airbyte.cdk.integrations.base.adaptive.AdaptiveDestinationRunner$Runner.run(AdaptiveDestinationRunner.kt:68) at io.airbyte.integrations.destination.s3.S3DestinationRunner.main(S3DestinationRunner.kt:15). @kapa.ai
k
The logs show that the S3 destination is reporting the 
ads
stream as INCOMPLETE, and then raising a generic: > 
TransientErrorException: Some streams were unsuccessful due to a source error. See logs for details.
From the knowledge sources, this pattern means: • The real/root error is in the source connector, not in S3. The destination only knows that the stream never completed successfully, so it marks it 
INCOMPLETE
and throws 
TransientErrorException
. [S3 partial msg issue; FB sync issue] • For Facebook Marketing–type sources, one known cause is requesting too many fields, which leads to a source failure and then the same 
INCOMPLETE
+ 
TransientErrorException
pattern on the destination side. The recommended fix there is to reduce the number of selected fields in the schema. [FB sync issue] Given that: 1. Open the failing connection in Airbyte and check the logs for the source (not just the destination) for the same time window; that’s where the concrete error message will be. 2. If this is a Facebook Marketing / similar ads source and you see “Please reduce the number of fields requested” or similar in the source logs, follow the guidance: ◦ Go to the Schema tab of the connection. ◦ For the 
ads
(and related) streams, unselect fields you don’t need so fewer columns are requested. [FB sync issue] ◦ Save and re‑run the sync. 3. If the source logs show a different error (rate limit, API error, etc.), you’ll need to address that specific source‑side issue; the provided knowledge sources don’t include more detailed S3‑specific troubleshooting beyond pointing to the source error. [S3 troubleshooting] The knowledge sources do not provide a more specific root cause for 
null.ads
with S3 beyond “source error,” so you’ll need to rely on the source logs to identify the exact problem. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
The Facebook Marketing API is returning: > 
"Please reduce the amount of data you're asking for, then retry your request"
for the 
ads
endpoint, and this source failure is what causes the 
INCOMPLETE
stream and 
TransientErrorException
on the S3 destination. [FB sync issue] In Airbyte, the only documented mitigation is to reduce what you request from that stream: 1. In the Airbyte UI, open the failing connection. 2. Go to the Schema tab. 3. Find the 
ads
stream. 4. Unselect as many non‑essential fields as possible (for example, keep only 
id
, 
name
, 
status
, 
adset_id
, 
campaign_id
, 
created_time
, 
updated_time
to start, and deselect the rest like 
creative
, 
targeting
, 
tracking_specs
, 
conversion_specs
, 
recommendations
, etc.). 5. Save the schema changes and re‑run the sync. The GitHub issue with the same error message confirms that this is a Facebook API limitation and that the connector surfaces it as: > “Please reduce the number of fields requested. Go to the schema tab, select your source, and unselect the fields you do not need.” [FB sync issue] The knowledge sources do not provide a more granular rule (e.g., exact max number of fields), so you may need to iteratively reduce fields until the sync succeeds. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai when i re ran it worked.
k
Good to hear it worked on rerun. Based on the linked issue, this Facebook error can be intermittent: sometimes the same request that previously failed will succeed later, especially if the API load or internal limits change. [FB sync issue] If it starts happening frequently again, the recommended mitigation is still to reduce the number of selected fields in the 
ads
stream schema so each request is lighter. [FB sync issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
how to check value ## JOB_KUBE_NODE_SELECTORS of this @kapa.ai
k
JOB_KUBE_NODE_SELECTORS
is configured via your Helm 
values.yaml
, not directly inside Airbyte. To check its value, look at the 
jobs.kube.nodeSelector
section in the values file you used to deploy Airbyte. For Helm chart v1 / v2, the mapping is: • Environment variable: 
JOB_KUBE_NODE_SELECTORS
• Helm values key: 
jobs.kube.nodeSelector
[Values v2; Env→Helm map] So: 1. Open the 
values.yaml
you used for the deployment. 2. Find:
Copy code
jobs:
     kube:
       nodeSelector: {...}
3. Whatever is under 
nodeSelector
is the value that will be exposed as 
JOB_KUBE_NODE_SELECTORS
. [Values v2] If you’re running an existing release and don’t have the file handy, you can also inspect the generated ConfigMap that holds env vars (e.g., 
airbyte-airbyte-env
or similar) and look for the 
JOB_KUBE_NODE_SELECTORS
key. [Helm env CM] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
JOB_KUBE_NODE_SELECTORS @kapa.ai to update this value which pod should i restart/
k
The knowledge sources don’t explicitly say which pod must be restarted after changing 
JOB_KUBE_NODE_SELECTORS
, but they do clarify what it controls and how it’s configured. 
JOB_KUBE_NODE_SELECTORS
(or its Helm v2 equivalent 
global.jobs.kube.nodeSelector
) affects where Airbyte job pods (sync/check/discover/spec) are scheduled, not the long‑running “static” services. [Config jobs; Helm v2 mapping] In Helm chart v2, you set it like:
Copy code
global:
  jobs:
    kube:
      nodeSelector:
        type: jobs
and then job pods launched by the workload launcher / worker / orchestrator use that selector. [Multiple node groups] Because this is a Helm‑level configuration for how new job pods are created, the documented pattern is to update 
values.yaml
and redeploy the Helm release, which will roll the relevant components (e.g., 
workloadLauncher
, 
worker
, 
containerOrchestrator
/ 
orchestrator
) and ensure all new job pods use the updated selector. [Multiple node groups; Values reference] The docs do not provide a minimal “restart only pod X” instruction, so the safest documented approach is: • Change 
jobs.kube.nodeSelector
/ 
global.jobs.kube.nodeSelector
in 
values.yaml
. • Apply via Helm upgrade so the deployment is reconciled and future job pods pick up the new value. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai 2026-03-08 201608 source ERROR Encountered an exception while reading stream ads Traceback (most recent call last): File “/usr/local/lib/python3.11/site-packages/backoff/_sync.py”, line 105, in retry ret = target(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^ File “/airbyte/integration_code/source_facebook_marketing/api.py”, line 171, in call response = super().call(method, path, params, headers, files, url_override, api_version) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/site-packages/facebook_business/api.py”, line 336, in call raise fb_response.error() facebook_business.exceptions.FacebookRequestError: Message: Call was not successful Method: GET Path: https://graph.facebook.com/v23.0/act_5997433197001471/ads Params: {‘limit’: 6, ‘filtering’: ‘[{“field”“ad.updated time”,“operator”“GREATER_THAN”,“value”:“**********“}]’, ‘fields’: ‘id,name,status,adlabels,adset_id,bid_info,bid_type,creative,targeting,account_id,bid_amount,campaign_id,created_time,source_ad_id,updated_time,tracking_specs,recommendations,conversion_specs,effective_status,last_updated_by_app_id’, ‘summary’: ‘true’} Status: 500 Response: { “error”: { “code”: 1, “message”: “Please reduce the amount of data you’re asking for, then retry your request” } } getting this in meta ads
k
Meta (Facebook) is returning a 500 with: > “Please reduce the amount of data you’re asking for, then retry your request” for the 
ads
stream. This is the same error pattern documented for the Facebook Marketing source connector. [FB sync issue] In Airbyte, the connector surfaces this as: > “Please reduce the number of fields requested. Go to the schema tab, select your source, and unselect the fields you do not need.” [FB sync issue] To mitigate: 1. In the Airbyte UI, open the connection that uses the Facebook Marketing source. 2. Go to the Schema tab. 3. Find and expand the 
ads
 stream. 4. Deselect as many non‑essential fields as possible. For example, start by keeping only a minimal set (e.g., 
id
, 
name
, 
status
, 
adset_id
, 
campaign_id
, 
created_time
, 
updated_time
) and unselect heavier fields like 
creative
, 
targeting
, 
tracking_specs
, 
conversion_specs
, 
recommendations
, etc. 5. Click Save changes / Save connection and re‑run the sync. The knowledge sources do not specify an exact maximum number or combination of fields allowed by Meta; if the error persists, you’ll need to iteratively reduce the selected fields in 
ads
until the sync succeeds. [FB sync issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
does new version of connector solves it. @kapa.ai
k
The knowledge sources do not state that a newer version of the Facebook Marketing source connector definitively solves this “Please reduce the amount of data you’re asking for” error. What they do show: • The error is coming from Meta’s API and is surfaced by multiple connector versions (e.g., v4.0.1 and v5.0.1) when too much data is requested (many fields, certain breakdowns, or large time ranges). [FB sync issue; Custom stream bug; 9000 ads issue] • In one open issue for v5.0.1, the problem is still present when using 
product_id
as a breakdown, and it’s being investigated; there is no confirmation in the sources that a newer version has fixed it. [Custom stream bug] So, based on the available information: • Upgrading to a newer connector version may include improvements (e.g., better job splitting), but the sources do not confirm that any specific version fully eliminates this Meta API error. • The only documented and recommended mitigation remains: reduce the number of requested fields and/or narrow the request (time range, breakdowns) in the schema/config. [FB sync issue; 9000 ads issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
apiVersion: v1 kind: Secret metadata: name: airbyte-config-secrets type: Opaque stringData: database-host: database-port: “5432” database-name: airbyte database-user: postgres DATABASE_PASSWORD: 7pOd2fN s3-access-key-id: AQJVVR s3-secret-access-key: aEQszZ3pFNuLDb @kapa.ai do it for gcp.
k
Here is the equivalent Kubernetes secret configured for GCP instead of AWS S3:
Copy code
apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  database-host: ""
  database-port: "5432"
  database-name: airbyte
  database-user: postgres
  database-password: 7pOd2fN

  # GCP Secrets
  gcp.json: |
    {
      "type": "service_account",
      "project_id": "your-project-id",
      "private_key_id": "your-private-key-id",
      "private_key": "-----BEGIN PRIVATE KEY-----\n<YOUR_PRIVATE_KEY>\n-----END PRIVATE KEY-----\n",
      "client_email": "your-service-account@your-project-id.iam.gserviceaccount.com",
      "client_id": "your-client-id",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/your-service-account%40your-project-id.iam.gserviceaccount.com"
    }
[GCS storage secret; GCS enterprise secret] A few notes: • Replace the 
s3-access-key-id
and 
s3-secret-access-key
fields with the 
gcp.json
key containing your GCP service account credentials JSON blob. • The 
gcp.json
value should be the full service account JSON file content. • Alternatively, you can create the secret from a file using `kubectl`:
Copy code
kubectl create secret generic airbyte-config-secrets \
  --from-literal=database-host='' \
  --from-literal=database-port='5432' \
  --from-literal=database-name='airbyte' \
  --from-literal=database-user='postgres' \
  --from-literal=database-password='7pOd2fN' \
  --from-file=gcp.json \
  --namespace airbyte
[GCS enterprise secret] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
2026-03-22 010246 source INFO i.a.c.i.s.r.RelationalDbQueryUtils(lambda$queryTable$0):77 Queueing query: SELECT “ecb_violation_number”,“bin”,“bbl”,“issue_date”,“ecb_violation_status”,“hearing_status”,“severity”,“penality_imposed”,“penality_balance”,“aggravated_level”,“respondent_name”,“respondent_entity_id”,“is_open”,“is_default”,“recency_weight”,“isn_dob_bis_extract”,“violation_type”,“violation_number”,“violation_description”,“amount_paid”,“borough”,“block”,“lot”,“raw_data”,“content_hash” FROM “public”.“violations_ecb” 2026-03-22 010335 source INFO Malformed non-Airbyte record (connectionId = d8ac7cfb-75bc-4fc3-8ec2-e4b0416d8f75): Terminating due to java.lang.OutOfMemoryError: Java heap space @kapa.ai getting this on cocroach DB i have 25 GB data on cockroach DB
k
The error you're seeing is a classic Java heap space / Out of Memory (OOM) issue when syncing a large table (25 GB) from CockroachDB. The source worker is reading too many records into memory at once. Why This Happens The Source worker reads up to 10,000 records in memory. For large row sizes, this can quickly exhaust the JVM heap. For example, a table with an average row size of 0.5 MB would require 0.5 × 10,000 / 1,000 = 5 GB of RAM just for that buffer. [Scaling Airbyte] Java connectors (including CockroachDB, which is a JDBC-based connector) by default use only 1/4 of the host's allocated memory for the JVM heap. So even if your pod has 8 GB, the connector may only use 2 GB before OOMing. [Memory scaling] A similar case was reported with CockroachDB syncing an 18 GB / 57M row table, which consumed ~49.7 GB of RAM before OOM — suggesting a possible memory inefficiency in the connector itself. [CockroachDB OOM] What You Can Do 1. Increase memory for the connector pod Set higher memory limits for the job containers. You can do this by: • Setting 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
and 
JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables. [Memory scaling] • Or via the connection-level 
resource_requirements
in the database (for Kubernetes deployments):
Copy code
UPDATE connection 
    SET resource_requirements = '{"memory_limit": "20Gi", "memory_request": "13Gi"}' 
    WHERE id = '<your-connection-id>';
[Java heapspace issue] See the Configuring connector resources docs for details. [Scaling Airbyte] 2. Reduce the number of selected fields Your query is selecting 24 columns, some of which (like 
raw_data
, 
violation_description
, 
content_hash
) may contain large variable-length data. Reducing the number of fields in the schema tab will reduce per-row memory usage. [Scaling Airbyte] 3. Be aware of known limitations Even after increasing memory, OOMs can still occur unpredictably if rows have variable-length columns (TEXT, JSON, arrays) with high size variance, because the JDBC fetch size estimator only samples the first few rows and can underestimate actual row sizes. [JDBC fetchSize issue] The knowledge sources do not provide a guaranteed fix for CockroachDB specifically beyond increasing memory and reducing fields — the CockroachDB connector issue with large tables was reported as a potential memory leak that was recommended for a GitHub issue investigation. [CockroachDB OOM] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
2026-03-22 213216 source INFO Malformed non-Airbyte record (connectionId = d8ac7cfb-c-4fc3-8ec2-e4b0416d8f75): Terminating due to java.lang.OutOfMemoryError: Java heap space @kapa.ai
k
This is the same Java heap space OOM error from your previous CockroachDB sync. The source connector is running out of JVM memory while reading your large table. Based on the knowledge sources, here are the documented options to address it: Increase Memory for the Connector Option 1: Instance-wide (via 
values.yaml
) Set these environment variables in your Helm `values.yaml`:
Copy code
global:
  env_vars:
    JOB_MAIN_CONTAINER_MEMORY_REQUEST: "8Gi"
    JOB_MAIN_CONTAINER_MEMORY_LIMIT: "16Gi"
    JAVA_OPTS: "-XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=80.0 -XX:+UseG1GC"
[Java heapspace issue] The 
MaxRAMPercentage=80.0
is important — by default Java connectors only use 1/4 of the container's allocated memory, so even with a large limit, the JVM heap stays small unless you override this. [Memory scaling] Option 2: Connection-specific (via database)
Copy code
UPDATE connection 
SET resource_requirements = '{"memory_limit": "16Gi", "memory_request": "8Gi", "cpu_limit": "2", "cpu_request": "1"}' 
WHERE id = 'd8ac7cfb-c-4fc3-8ec2-e4b0416d8f75';
[Configuring connector resources] Also Reduce Selected Fields Your previous query selected 24 columns including potentially large ones like 
raw_data
, 
violation_description
, and 
content_hash
. Deselecting heavy fields in the Schema tab will reduce per-row memory consumption. [Scaling Airbyte] Important Caveat Even after increasing memory, OOMs can still occur with large tables if rows have variable-length columns (TEXT, JSON) whose sizes are underestimated by the JDBC fetch size sampler. The knowledge sources note this was flagged as a potential memory issue in the CockroachDB connector itself. [CockroachDB OOM] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
2026-03-22 213216 source INFO Malformed non-Airbyte record (connectionId = d8cfb-75bc-4fc3-8ec2-e4b0416d8f75): Terminating due to java.lang.OutOfMemoryError: Java heap space i created 64 GB instance. and stream size is 5.2GB. @kapa.ai
k
Even with a 64 GB instance and only a 5.2 GB stream, the OOM is still occurring. This is a known pattern — the root cause is almost certainly that Java connectors only use 1/4 of the container's allocated memory by default for the JVM heap, regardless of how much RAM the instance has. [Memory scaling] So on a 64 GB instance, the JVM heap may still be capped at ~16 GB unless you explicitly override it. What to Check / Fix 1. Override the JVM heap percentage Add this to your `values.yaml`:
Copy code
global:
  env_vars:
    JOB_MAIN_CONTAINER_MEMORY_REQUEST: "32Gi"
    JOB_MAIN_CONTAINER_MEMORY_LIMIT: "48Gi"
    JAVA_OPTS: "-XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=80.0 -XX:+UseG1GC"
The 
MaxRAMPercentage=80.0
is critical — without it, the JVM will only use 25% of the container's memory limit as heap, even if you've set a large limit. [Java heapspace issue] 2. Set connection-specific resource requirements
Copy code
UPDATE connection 
SET resource_requirements = '{"memory_limit": "48Gi", "memory_request": "32Gi"}' 
WHERE id = 'd8ac7cfb-75bc-4fc3-8ec2-e4b0416d8f75';
[Configuring connector resources] 3. Important caveat on orchestrator memory There is an open known issue where the container orchestrator ignores 
CONTAINER_ORCHESTRATOR_MEMORY_LIMIT
and 
REPLICATION_ORCHESTRATOR_MEMORY_LIMIT
and always uses 1Gi, which can cause OOMs even when source/destination have large memory allocations. [Container orchestrator OOM] This means even if your source connector pod has enough memory, the orchestrator itself may OOM and terminate the sync. 4. Reduce selected fields If you haven't already, deselect heavy columns like 
raw_data
, 
violation_description
, and 
content_hash
in the Schema tab to reduce per-row memory pressure. [Scaling Airbyte] The knowledge sources do not provide a guaranteed fix beyond these steps, particularly given the open orchestrator memory bug. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
This is the same Java heap space OOM issue as before. Even with a 64 GB instance and a 5.2 GB stream, the problem persists because Java connectors default to using only 1/4 of the container's allocated memory for the JVM heap. [Memory scaling] Key Steps to Fix 1. Set 
JAVA_OPTS
to increase JVM heap percentage This is the most critical fix. Without it, the JVM caps itself at ~25% of the container limit regardless of instance size:
Copy code
global:
  env_vars:
    JOB_MAIN_CONTAINER_MEMORY_REQUEST: "32Gi"
    JOB_MAIN_CONTAINER_MEMORY_LIMIT: "48Gi"
    JAVA_OPTS: "-XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=80.0 -XX:+UseG1GC"
[Java heapspace issue] 2. Set connection-specific memory via SQL
Copy code
UPDATE connection 
SET resource_requirements = '{"memory_limit": "48Gi", "memory_request": "32Gi"}' 
WHERE id = 'd8cfb-75bc-4fc3-8ec2-e4b0416d8f75';
[Configuring connector resources] 3. Check for the orchestrator memory bug There is an open known issue where the container orchestrator ignores 
CONTAINER_ORCHESTRATOR_MEMORY_LIMIT
and always uses 1Gi, causing OOMs even when source/destination pods have large allocations. [Container orchestrator OOM] This could explain why a 64 GB instance still OOMs on a 5.2 GB stream — the orchestrator itself may be the bottleneck, not the source connector pod. 4. Reduce selected fields Deselect heavy columns (e.g., 
raw_data
, 
violation_description
, 
content_hash
) in the Schema tab to reduce per-row memory pressure. [Scaling Airbyte] The knowledge sources do not provide a guaranteed resolution beyond these steps, particularly given the open orchestrator memory bug. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
Copy code
JAVA_OPTS="-XX:MaxRAMPercentage=80"
can this be set on DB level @kapa.ai
k
No, 
JAVA_OPTS
(including 
-XX:MaxRAMPercentage=80
) cannot be set at the database/connection level. Based on the knowledge sources, the only resource-related settings that can be configured at the database level are: • 
cpu_request
• 
cpu_limit
• 
memory_request
• 
memory_limit
These are set via SQL like:
Copy code
UPDATE connection 
SET resource_requirements = '{"memory_limit": "48Gi", "memory_request": "32Gi"}' 
WHERE id = '<your-connection-id>';
[Configuring connector resources] 
JAVA_OPTS
is a JVM-level environment variable and must be set at the instance level via your `values.yaml`:
Copy code
global:
  env_vars:
    JAVA_OPTS: "-XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=80.0 -XX:+UseG1GC"
[Java heapspace issue] This applies globally to all job containers. There is no documented way in the knowledge sources to set 
JAVA_OPTS
on a per-connection or per-connector basis. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
secretsManager the gcp i want o use i a, using airbyte 1.6 and give me valuees.yaml config for that. @kapa.ai
k
Here is the 
values.yaml
configuration for using Google Secret Manager with Airbyte 1.6 (Helm chart V1):
Copy code
global:
  secretsManager:
    type: googleSecretManager
    secretName: "airbyte-config-secrets"  # Name of your Kubernetes secret
    googleSecretManager:
      projectId: <project-id>
      credentialsSecretKey: gcp.json
[GCP secrets 1.6] Prerequisites Before applying this config, you need to create the Kubernetes secret containing your GCP service account credentials:
Copy code
apiVersion: v1
kind: Secret
metadata:
  name: airbyte-config-secrets
type: Opaque
stringData:
  gcp.json: |
    {
      "type": "service_account",
      "project_id": "your-project-id",
      "private_key_id": "your-private-key-id",
      "private_key": "-----BEGIN PRIVATE KEY-----\n<YOUR_PRIVATE_KEY>\n-----END PRIVATE KEY-----\n",
      "client_email": "your-service-account@your-project-id.iam.gserviceaccount.com",
      "client_id": "your-client-id",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/your-service-account%40your-project-id.iam.gserviceaccount.com"
    }
[GCP k8s secret 1.6] > ⚠️ Important: Modifying the configuration of connector secret storage will cause all existing connectors to fail. You will need to recreate those connectors to ensure they read from the new secret store. [GCP secrets 1.6] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by