https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I keep having this message in my sy...
# replication-ask-ai
s
@kapa.ai I keep having this message in my syncs and my airbyte syncs has multiple attempts Failures: [ { "failureOrigin" : "source", "internalMessage" : "Source process exited with non-zero exit code 143", "externalMessage" : "Something went wrong within the source connector", "metadata" : { "attemptNumber" : 1, "jobId" : 14, "connector_command" : "read" }, "stacktrace" : "io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 143\n\tat io.airbyte.container.orchestrator.worker.SourceR
k
Exit Code 143 in Airbyte Syncs The error 
Source process exited with non-zero exit code 143
is a known issue in Airbyte. Exit code 143 typically means the process received a SIGTERM signal (i.e., it was terminated), which can happen in a few scenarios: Common Causes 1. Job cancellation: Exit code 143 is normal when a sync job is manually cancelled. As noted in Airbyte's own log messages: "This warning is normal if the job was cancelled." [sync cancellation] 2. Out of Memory (OOM): A very common cause is the sync process running out of memory, especially when handling large amounts of data. This causes the container/process to be killed with a SIGTERM. [exit code 143 OOM] 3. Large data volumes: Syncs processing large datasets (hundreds of MBs or GBs) are particularly prone to this issue. [large sync failures] Recommended Solutions 1. Increase memory allocation (OSS/self-hosted only): Increase the memory limits for your sync jobs by raising the 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
and 
JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables on your scheduler and worker pods. Values like 4GB–8GB are commonly suggested. [exit code 143 fix; custom source OOM] 2. Reduce batch/data size: If possible, reduce the date range or scope of the sync to process smaller chunks of data at a time. [OOM large records] 3. Check for large individual records: Very large individual records (e.g., 500MB+ JSON payloads) can cause memory pressure during processing. [OOM large records] 4. Retry the sync: In some cases, the issue is transient and a retry resolves it. [sync failures] Next Steps To better diagnose the issue, check your full sync logs for any OOM-related messages or signs of the process being killed due to resource constraints. If you are on Airbyte Cloud, you may need to contact Airbyte support, as memory configuration is not user-adjustable there. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai i am also seeing
Copy code
ERROR DefaultDispatcher-worker-1 i.a.c.l.t.DestinationTaskLauncher$WrappedTask(execute):129 Caught exception in task io.airbyte.cdk.load.task.internal.InputConsumerTask@44639c80 io.airbyte.cdk.TransientErrorException: Input was fully read, but some streams did not receive a terminal stream status message. If the destination did not encounter other errors, this likely indicates an error in the source or platform. Streams without a status message: [stripe_source.invoice_items, stripe_source.refunds, stripe_source.balance_transactions, stripe_source.accounts, stripe_source.customers, stripe_source.shipping_rates, stripe_source.disputes, stripe_source.customer_balance_transactions, stripe_source.credit_notes, stripe_source.invoices, stripe_source.payment_intents, stripe_source.prices, stripe_source.payment_methods, stripe_source.charges, stripe_source.transactions, stripe_source.subscription_schedule, stripe_source.subscriptions, stripe_source.subscription_items, stripe_source.invoice_line_items, stripe_source.transfer_reversals, stripe_source.plans, stripe_source.coupons, stripe_source.transfers, stripe_source.events, stripe_source.products, stripe_source.promotion_codes]
	at io.airbyte.cdk.load.state.SyncManager.markInputConsumed(SyncManager.kt:116) ~[bulk-cdk-toolkit-legacy-task-loader-0.2.0.jar:?]
	at io.airbyte.cdk.load.state.PipelineEventBookkeepingRouter.close(PipelineEventBookkeepingRouter.kt:300) ~[bulk-cdk-toolkit-legacy-task-loader-0.2.0.jar:?]
	at io.airbyte.cdk.load.task.internal.InputConsumerTask.execute(InputConsumerTask.kt:117) ~[bulk-cdk-toolkit-legacy-task-loader-0.2.0.jar:?]
	at io.airbyte.cdk.load.task.internal.InputConsumerTask$execute$1.invokeSuspend(InputConsumerTask.kt) ~[bulk-cdk-toolkit-legacy-task-loader-0.2.0.jar:?]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-2.1.10.jar:2.1.10-release-473]
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100) [kotlinx-coroutines-core-jvm-1.10.2.jar:?]
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124) [kotlinx-coroutines-core-jvm-1.10.2.jar:?]
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89) [kotlinx-coroutines-core-jvm-1.10.2.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586) [kotlinx-coroutines-core-jvm-1.10.2.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820) [kotlinx-coroutines-core-jvm-1.10.2.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717) [kotlinx-coroutines-core-jvm-1.10.2.jar:?]
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704) [kotlinx-coroutines-core-jvm-1.10.2.jar:?]

Stack Trace: io.airbyte.cdk.TransientErrorException: Input was fully read, but some streams did not receive a terminal stream status message. If the destination did not encounter other errors, this likely indicates an error in the source or platform. Streams without a status message: [stripe_source.invoice_items, stripe_source.refunds, stripe_source.balance_transactions, stripe_source.accounts, stripe_source.customers, stripe_source.shipping_rates, stripe_source.disputes, stripe_source.customer_balance_transactions, stripe_source.credit_notes, stripe_source.invoices, stripe_source.payment_intents, stripe_source.prices, stripe_source.payment_methods, stripe_source.charges, stripe_source.transactions, stripe_source.subscription_schedule, stripe_source.subscriptions, stripe_source.subscription_items, stripe_source.invoice_line_items, stripe_source.transfer_reversals, stripe_source.plans, stripe_source.coupons, stripe_source.transfers, stripe_source.events, stripe_source.products, stripe_source.promotion_codes]
	at io.airbyte.cdk.load.state.SyncManager.markInputConsumed(SyncManager.kt:116)
	at io.airbyte.cdk.load.state.PipelineEventBookkeepingRouter.close(PipelineEventBookkeepingRouter.kt:300)
	at io.airbyte.cdk.load.task.internal.InputConsumerTask.execute(InputConsumerTask.kt:117)
	at io.airbyte.cdk.load.task.internal.InputConsumerTask$execute$1.invokeSuspend(InputConsumerTask.kt)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)
	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
k
This second error is directly related to the first one. The 
TransientErrorException: Input was fully read, but some streams did not receive a terminal stream status message
is a destination-side error that occurs because the source (Stripe) failed to emit terminal stream status messages before it was killed (exit code 143). In other words, the two errors are connected: 1. The source process was terminated (exit code 143, likely OOM) 2. Because the source was killed abruptly, it never sent the required terminal status messages to the destination 3. The destination (which uses CDK v3) then throws this 
TransientErrorException
because it expects those messages [streams without status] This is a known issue. As confirmed in a GitHub issue, this error means the destination did not receive the required 
STREAM_STATUS
messages that CDK v3 destinations expect. The error message itself even hints at this: "If the destination did not encounter other errors, this likely indicates an error in the source or platform." [HubSpot associations issue] What This Means for You Since you're using source-stripe with a large number of streams (26 streams listed in the error), the most likely root cause remains the OOM / exit code 143 issue from the source. The destination error is a downstream consequence. Recommended Actions 1. Address the root cause first: Focus on resolving the exit code 143 issue (increase memory, reduce sync scope, etc.) as described in the previous answer. Once the source completes successfully and emits proper status messages, the destination error should also resolve. 2. Check for Stripe-specific memory issues: There is a known memory leak in 
source-stripe
related specifically to 
invoice_line_items
and 
subscription_items
streams, where memory continuously grows until the container is OOMKilled. [Stripe memory leak] Both of these streams appear in your error list, which strongly suggests this known bug may be affecting you. 3. Update your source-stripe connector: If you are on an older version of 
source-stripe
, check if a newer version has addressed the memory leak. The known issue was observed on version 
5.12.0
. 4. Reduce the number of streams: As a temporary workaround, try syncing fewer streams at a time to reduce memory pressure. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai should request cpu larger or smaller than limit cpu
k
In Kubernetes resource configuration, the request should always be less than or equal to the limit. The request represents the minimum guaranteed resources, while the limit represents the maximum allowed. For Airbyte's CPU settings specifically: • 
JOB_MAIN_CONTAINER_CPU_REQUEST
— the job container's minimum CPU usage • 
JOB_MAIN_CONTAINER_CPU_LIMIT
— the job container's maximum CPU usage [configuring connector resources] So the request should be smaller than or equal to the limit. For example, from a real-world configuration seen in the knowledge sources:
Copy code
JOB_MAIN_CONTAINER_CPU_REQUEST: "1"
JOB_MAIN_CONTAINER_CPU_LIMIT: "2"
[custom source OOM] > ⚠️ Important note on millicore notation: There is a known bug where specifying CPU values using Kubernetes millicore notation (e.g., 
4000m
) causes sync failures, because Airbyte misinterprets the value. As a workaround, use whole CPU units (e.g., 
'4'
instead of 
'4000m'
). [millicore bug] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai what should be my limit for jobs
k
The knowledge sources don't provide specific recommended CPU/memory limit values that apply universally — the right limits depend on your workload. However, here's what the sources do tell us: General Guidance Airbyte recommends starting with the default resource allocations and only increasing them if you observe resource-related issues via logs or observability tools. [configuring connector resources] Values Seen in Practice From real-world examples in the knowledge sources: | Setting | Example Values | | - | - | | 
JOB_MAIN_CONTAINER_CPU_REQUEST
| 
1
or 
2
| | 
JOB_MAIN_CONTAINER_CPU_LIMIT
| 
2
or 
4
| | 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
| 
2Gi
or 
4Gi
| | 
JOB_MAIN_CONTAINER_MEMORY_LIMIT
| 
4Gi
or 
8Gi
| [Helm config example; custom source OOM] For Your Specific Case Given that you're syncing 26 Stripe streams and experiencing OOM-related failures, a reasonable starting point would be:
Copy code
JOB_MAIN_CONTAINER_CPU_REQUEST: "2"
JOB_MAIN_CONTAINER_CPU_LIMIT: "4"
JOB_MAIN_CONTAINER_MEMORY_REQUEST: "4Gi"
JOB_MAIN_CONTAINER_MEMORY_LIMIT: "8Gi"
> ⚠️ Remember to use whole CPU units (e.g., 
"4"
) rather than millicore notation (e.g., 
"4000m"
) due to a known parsing bug. [millicore bug] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai i cant parse configs to workload launcher
Copy code
limits:
      cpu: "2"
      memory: "2Gi"
    requests:
      cpu: "2"
      memory: "1Gi"
k
There is a known issue where resource limits set via 
values.yaml
or environment variables are ignored by the workload launcher in some Airbyte versions. [resource limits ignored] A workaround is to set the resource environment variables directly on the 
workload-launcher
deployment using 
extraEnv
, rather than relying on the ConfigMap:
Copy code
workloadLauncher:
  extraEnv:
    - name: JOB_MAIN_CONTAINER_CPU_LIMIT
      value: "4"
    - name: JOB_MAIN_CONTAINER_CPU_REQUEST
      value: "2"
    - name: JOB_MAIN_CONTAINER_MEMORY_LIMIT
      value: "8Gi"
    - name: JOB_MAIN_CONTAINER_MEMORY_REQUEST
      value: "4Gi"
[resource limits ignored] After applying this, you can verify the values are actually reaching the workload-launcher pod by running:
Copy code
kubectl exec -n airbyte deploy/airbyte-abctl-workload-launcher -- env | grep JOB_MAIN
[resource limits ignored] > ⚠️ Also note: your current config has 
requests.cpu
equal to 
limits.cpu
(
"2"
), which is valid, but 
requests.memory
(
1Gi
) is lower than 
limits.memory
(
2Gi
), which is the correct pattern (request ≤ limit). 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
what abouut jobs
@kapa.ai this is not working workload-launcher: extraEnv: - name: JOB_MAIN_CONTAINER_CPU_LIMIT value: "4" - name: JOB_MAIN_CONTAINER_CPU_REQUEST value: "2" - name: JOB_MAIN_CONTAINER_MEMORY_LIMIT value: "8Gi" - name: JOB_MAIN_CONTAINER_MEMORY_REQUEST value: "4Gi"
k
This is a known and actively investigated bug. The workload-launcher ignoring resource configurations is a confirmed issue in recent Airbyte versions. [resource limits ignored] Here are the alternative workarounds you can try: 1. Try the 
global.jobs.resources
Helm structure Some users have reported success with this approach after forcing a recreation of the worker pod:
Copy code
global:
  jobs:
    resources:
      requests:
        cpu: "2"
        memory: "4Gi"
      limits:
        cpu: "4"
        memory: "8Gi"
[workload resource requests] 2. Set resource requirements directly in the database As a last resort, one user reported success by setting 
resource_requirements
directly at the connection level in the database when nothing else worked: [workload resource requests] > _"I end up with setting 
resource_requirements
directly in database on a connection level and it worked well."_ 3. Verify what the pod actually sees First confirm whether the env vars are even reaching the workload-launcher pod:
Copy code
kubectl exec -n airbyte deploy/airbyte-abctl-workload-launcher -- env | grep JOB_MAIN
If the values are present in the pod's environment but still not applied to replication pods, this confirms the bug where the workload-launcher uses its own internal defaults (2Gi/2CPU) instead of the configured values. [resource limits ignored] Current Status This issue has been escalated to Airbyte's internal engineering team for investigation (tracked internally as 
airbytehq/oncall#11290
). [resource limits ignored] There is no confirmed fix available yet, so the database-level override may be your most reliable option in the meantime. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
this is also not working
Copy code
jobs:
    resources:
      requests:
        cpu: "2"
        memory: "4Gi"
      limits:
        cpu: "4"
        memory: "8Gi"
@kapa.ai this is also not working
Copy code
jobs:
    resources:
      requests:
        cpu: "2"
        memory: "4Gi"
      limits:
        cpu: "4"
        memory: "8Gi"
k
Unfortunately, this confirms what the knowledge sources describe — this is a known, unresolved bug. The root cause is that the workload-launcher uses its own internal defaults instead of reading from Helm values or environment variables, regardless of which configuration path you use. [resource limits ignored] At this point, the most reliable workaround reported by users is setting 
resource_requirements
directly in the Airbyte database at the connection level: > _"Nothing of the above worked so I ended up with setting 
resource_requirements
directly in the database on a connection level and it worked well."_ [workload resource requests] What You Can Do Now 1. Database-level override: Set 
resource_requirements
directly in the database for your connection(s). This has been confirmed as working by multiple users when all other methods failed. [workload resource requests] 2. Verify env vars are reaching the pod: Before anything else, confirm whether the variables are even visible inside the workload-launcher pod:
Copy code
kubectl exec -n airbyte deploy/airbyte-abctl-workload-launcher -- env | grep JOB_MAIN
If the values are present but still not applied to replication pods, this definitively confirms the internal bug. [resource limits ignored] 3. Wait for a fix: This has been escalated to Airbyte's internal engineering team (
airbytehq/oncall#11290
), but there is no confirmed fix available yet. [resource limits ignored] The knowledge sources do not provide the exact SQL to update the database directly, so I'd recommend reaching out to Airbyte support or the GitHub issue thread for the specific database update syntax. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai where do i see the job pod request ~ nodeSelector: { + cloud.google.com/gke-spot: "true"
k
To check what node selectors and resource requests are actually being applied to your job pods, you can inspect the pods directly in Kubernetes:
Copy code
kubectl get pods -n airbyte
kubectl describe pod <job-pod-name> -n airbyte
This will show you the actual 
nodeSelector
, 
resources.requests
, and 
resources.limits
applied to the pod. Configuring nodeSelector for Job Pods For job pods (sync/check/discover), the node selector is controlled via the 
JOB_KUBE_NODE_SELECTORS
environment variable or via the Helm values under 
global.jobs.kube.nodeSelector
. [configuring Airbyte jobs] In your `values.yaml`:
Copy code
global:
  jobs:
    kube:
      nodeSelector:
        cloud.google.com/gke-spot: "true"
Or via 
extraEnv
on the workload-launcher:
Copy code
workload-launcher:
  extraEnv:
    - name: JOB_KUBE_NODE_SELECTORS
      value: "cloud.google.com/gke-spot=true"
[values.yaml reference] > ⚠️ Note: As discussed in our previous conversation, there is a known bug where resource configurations may not be properly applied to job pods by the workload-launcher. The same may apply to node selectors. I'd recommend verifying the actual pod spec after applying the config to confirm it's taking effect. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai i am seeing this in the log what does it mean [destination] image: airbyte/destination-bigquery:3.0.17 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.2}, additionalProperties={}) 2026-03-12 002343 platform INFO [orchestrator] image: airbyte/container-orchestrator:2.0.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=2Gi, cpu=1}, additionalProperties={}) 2026-03-12 002415 platform INFO Attempting to update workload: 2a878654-2783-4d81-bc3d-c197a9bf6f12_30_2_sync to LAUNCHED.
k
This log is actually good news — it shows you the actual resource requirements being applied to your sync job pods. Airbyte logs these details when launching replication pods. [debugging connection resources] Here's what each line means: | Container | CPU Request | CPU Limit | Memory Request | Memory Limit | | - | - | - | - | - | | destination-bigquery | 0.2 | 1 | 1Gi | 2Gi | | orchestrator | 1 | 1 | 2Gi | 2Gi | What This Tells You The resources shown here are the defaults being applied by the workload-launcher, not your custom values. This confirms the bug we discussed earlier — your custom resource configurations are not being picked up. [resource limits ignored] Specifically, the destination-bigquery container is running with only 1Gi memory request and 2Gi limit, which is likely insufficient for syncing 26 Stripe streams to BigQuery and is consistent with the OOM/exit code 143 failures you've been seeing. Next Step Since the log confirms your custom values are not being applied, the database-level override remains your most reliable option. You can use the SQL approach to set resource requirements at the connector definition or connection level:
Copy code
update actor_definition 
set resource_requirements = '{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "4", "cpu_request": "2", "memory_limit": "8Gi", "memory_request": "4Gi"}}]}' 
where id = '<id-from-step-1>';
[configuring connector resources] After applying any change, re-run a sync and check the log again for the 
Launching replication pod
line to confirm the new values are being used. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
i set my main container memory limit to 4 gb
@kapa.ai i have a stripe sync, it is syncing, and have a successful status message, but i notice data hasn't been loaded
k
This is a known and concerning issue — a sync showing "Succeeded" status but with no data actually loaded to the destination. Here's what the knowledge sources tell us about this: Why a "Successful" Sync May Have No Data 1. Extracted records ≠ Loaded records Airbyte distinguishes between extracted records (read from source) and loaded records (confirmed by destination). If an attempt fails mid-extraction, the sync summary may show records extracted but none loaded. A successful status means all records from the source were written to the destination across all attempts — but this can be misleading if the source itself emitted 0 records. [connection timeline] 2. Source emitting 0 records silently This has been reported with Stripe specifically — the schema/table is created but no records are inserted, and no errors are surfaced. [Stripe copies schema but no records] In one case, the logs showed:
Copy code
recordsSynced=0, bytesEmitted=0, recordsCommitted=0
The sync completed "successfully" but nothing was actually moved. 3. Truncate refresh generation ID mismatch There is an open bug where a truncate refresh sync silently produces empty final tables due to a generation ID mismatch in TypeAndDedupe — raw tables are populated correctly, but final tables are overwritten with empty temp tables, and the sync still reports success. [generation ID mismatch] What to Check 1. Review the sync summary metrics in the Connection Timeline — check the actual number of extracted records and loaded records. If both are 0, the source emitted nothing. [connection timeline] 2. Check the start date configured for your Stripe source — if the start date is set too recently or incorrectly, the source may legitimately return 0 records. [Stripe no records] 3. Check the job logs for lines like 
recordsEmitted=0
or absence of 
SELECT / FETCH / EMITTED RECORDS
log entries, which would confirm the source sent nothing. 4. Given your previous OOM issues: Since your source process was being killed (exit code 143), it's very likely the source was terminated before emitting any records, causing the destination to receive nothing — yet the job may still report success after retries. This aligns directly with the errors you were seeing earlier in our conversation. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by