Bernardo Fernandes
06/25/2025, 7:34 AMPower School
06/25/2025, 8:01 AMKhánh Phạm An
06/25/2025, 8:09 AMAlec Sharp
06/25/2025, 8:13 AMSudeesh Rajeevan
06/25/2025, 8:17 AMMatheus Dantas
06/25/2025, 8:31 AMEitan Hyams
06/25/2025, 11:40 AMNevo
06/25/2025, 12:06 PMChris
06/25/2025, 12:16 PMsai ram komma
06/25/2025, 12:32 PMSlackbot
06/25/2025, 1:10 PMLui Pillmann
06/25/2025, 1:54 PMJacob Batt
06/25/2025, 2:20 PMLucas Segers
06/25/2025, 2:46 PMSérgio Marçal
06/25/2025, 4:08 PMLakin Wecker
06/25/2025, 4:40 PMKonathala Chaitanya
06/25/2025, 6:54 PMKonathala Chaitanya
06/25/2025, 7:29 PMMohd Asad
06/25/2025, 7:58 PMhelm repo add airbyte
<https://airbytehq.github.io/helm-charts>
helm install my-airbyte airbyte/airbyte --version 1.7.0
The core components are running fine. However, when I create a source and destination and trigger a sync, a new replication job pod is created. This pod includes three containers—`source`, destination
, and `orchestrator`—and it requests a total of 4 CPUs, which is too high for my environment.
I attempted to reduce the CPU and memory usage by setting the following values in my `values.yaml`:
global:
jobs:
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
I also tried setting these environment variables:
JOB_MAIN_CONTAINER_CPU_REQUEST
JOB_MAIN_CONTAINER_CPU_LIMIT
JOB_MAIN_CONTAINER_MEMORY_REQUEST
JOB_MAIN_CONTAINER_MEMORY_LIMIT
Despite these changes, the replication job pods are still requesting 4 CPUs. I’m looking for a reliable way to reduce their resource requests to around
1.5 to 2 CPUs
in total.Sree Shanthan Kuthuru
06/26/2025, 1:19 AMGlen Aultman-Bettridge
06/26/2025, 1:34 AMDavid Hu
06/26/2025, 2:07 AM2025-06-25 19:05:16 replication-orchestrator INFO Failures: [ {
"failureOrigin" : "destination",
"failureType" : "system_error",
"internalMessage" : "java.lang.NullPointerException: Cannot invoke \"io.airbyte.protocol.models.v0.AirbyteGlobalState.getSharedState()\" because the return value of \"io.airbyte.protocol.models.v0.AirbyteStateMessage.getGlobal()\" is null",
"externalMessage" : "Cannot invoke \"io.airbyte.protocol.models.v0.AirbyteGlobalState.getSharedState()\" because the return value of \"io.airbyte.protocol.models.v0.AirbyteStateMessage.getGlobal()\" is null",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 18128,
"from_trace_message" : true,
"connector_command" : "write"
},
"stacktrace" : "java.lang.NullPointerException: Cannot invoke \"io.airbyte.protocol.models.v0.AirbyteGlobalState.getSharedState()\" because the return value of \"io.airbyte.protocol.models.v0.AirbyteStateMessage.getGlobal()\" is null\n\tat io.airbyte.cdk.load.message.DestinationMessageFactory.fromAirbyteProtocolMessage(DestinationMessageFactory.kt:166)\n\tat io.airbyte.cdk.load.message.ProtocolMessageDeserializer.deserialize(DestinationMessageDeserializer.kt:50)\n\tat io.airbyte.cdk.load.task.internal.ReservingDeserializingInputFlow.collect(ReservingDeserializingInputFlow.kt:46)\n\tat io.airbyte.cdk.load.task.internal.InputConsumerTask.execute(InputConsumerTask.kt:120)\n\tat io.airbyte.cdk.load.task.DestinationTaskLauncher$WrappedTask.execute(DestinationTaskLauncher.kt:124)\n\tat io.airbyte.cdk.load.task.TaskScopeProvider$launch$job$1.invokeSuspend(TaskScopeProvider.kt:35)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)\n\tat kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:124)\n\tat kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:89)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:820)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)\n\tSuppressed: io.airbyte.cdk.TransientErrorException: Input was fully read, but some streams did not receive a terminal stream status message. If the destination did not encounter other errors, this likely indicates an error in the source or platform. Streams without a status message: [Account, User, ApiKey, SubscriptionRetentionOffer, UserSubscription, UserNotificationConfig, VideoProjectNotification, ImageProject, SubscriptionCancellation, VideoProject, PaymentIntent, WebhookEventLog, SubscriptionInvoice, Webhook, Tag, WebhookSubscribedEvent, VideoProjectTag, ImageCustomLora]\n\t\tat io.airbyte.cdk.load.state.SyncManager.markInputConsumed(SyncManager.kt:125)\n\t\tat io.airbyte.cdk.load.state.PipelineEventBookkeepingRouter.close(PipelineEventBookkeepingRouter.kt:248)\n\t\tat io.airbyte.cdk.load.task.internal.InputConsumerTask.execute(InputConsumerTask.kt:133)\n\t\t... 10 more\n",
"timestamp" : 1750903514303
}, {
"failureOrigin" : "replication",
"internalMessage" : "Broken pipe",
"externalMessage" : "Something went wrong during replication",
"metadata" : {
"attemptNumber" : 0,
"jobId" : 18128
},
"stacktrace" : "java.io.IOException: Broken pipe\n\tat java.base/sun.nio.ch.UnixFileDispatcherImpl.write0(Native Method)\n\tat java.base/sun.nio.ch.UnixFileDispatcherImpl.write(UnixFileDispatcherImpl.java:65)\n\tat java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:137)\n\tat java.base/sun.nio.ch.IOUtil.write(IOUtil.java:102)\n\tat java.base/sun.nio.ch.IOUtil.write(IOUtil.java:72)\n\tat java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:300)\n\tat java.base/sun.nio.ch.ChannelOutputStream.writeFully(ChannelOutputStream.java:68)\n\tat java.base/sun.nio.ch.ChannelOutputStream.write(ChannelOutputStream.java:105)\n\tat java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:309)\n\tat java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:381)\n\tat java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:357)\n\tat java.base/sun.nio.cs.StreamEncoder.lockedWrite(StreamEncoder.java:158)\n\tat java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:139)\n\tat java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:219)\n\tat java.base/java.io.BufferedWriter.implFlushBuffer(BufferedWriter.java:178)\n\tat java.base/java.io.BufferedWriter.flushBuffer(BufferedWriter.java:163)\n\tat java.base/java.io.BufferedWriter.implWrite(BufferedWriter.java:334)\n\tat java.base/java.io.BufferedWriter.write(BufferedWriter.java:313)\n\tat java.base/java.io.Writer.write(Writer.java:278)\n\tat io.airbyte.container.orchestrator.worker.io.AirbyteMessageBufferedWriter.write(AirbyteMessageBufferedWriter.kt:29)\n\tat io.airbyte.container.orchestrator.worker.io.LocalContainerAirbyteDestination.acceptWithNoTimeoutMonitor(LocalContainerAirbyteDestination.kt:132)\n\tat io.airbyte.container.orchestrator.worker.io.LocalContainerAirbyteDestination.accept(LocalContainerAirbyteDestination.kt:91)\n\tat io.airbyte.container.orchestrator.worker.DestinationWriter.run(ReplicationTask.kt:96)\n\tat io.airbyte.container.orchestrator.worker.DestinationWriter$run$1.invokeSuspend(ReplicationTask.kt)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n",
"timestamp" : 1750903516065
}
Alex C
06/26/2025, 7:04 AMAlex C
06/26/2025, 7:20 AMLuke Alexander
06/26/2025, 8:17 AMhttps://
so they are not formatted correctly, how to fix this?Slackbot
06/26/2025, 8:35 AMJacob
06/26/2025, 9:41 AMFabrizio Spini
06/26/2025, 10:12 AMCenk Batman
06/26/2025, 10:31 AMKonathala Chaitanya
06/26/2025, 11:59 AM