I just upgraded to 0.50.32 and I’m having sporadic failures across mutiple connections all with this error:

2023-10-25 16:36:43 INFO i.a.w.g.ReplicationWorkerHelper(getReplicationOutput):244 - failures: [ {
  "failureOrigin" : "replication",
  "internalMessage" : "java.lang.NullPointerException",
  "externalMessage" : "Something went wrong during replication",
  "metadata" : {
    "attemptNumber" : 4,
    "jobId" : 35934
  },
  "stacktrace" : "java.lang.RuntimeException: java.lang.NullPointerException\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startSource(ReplicationWorkerHelper.kt:134)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$run$1(BufferedReplicationWorker.java:161)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:229)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: java.lang.NullPointerException\n\tat java.base/java.io.Reader.<init>(Reader.java:168)\n\tat java.base/java.io.InputStreamReader.<init>(InputStreamReader.java:124)\n\tat io.airbyte.commons.io.IOs.newBufferedReader(IOs.java:74)\n\tat io.airbyte.commons.io.LineGobbler.<init>(LineGobbler.java:134)\n\tat io.airbyte.commons.io.LineGobbler.gobble(LineGobbler.java:73)\n\tat io.airbyte.workers.internal.DefaultAirbyteSource.start(DefaultAirbyteSource.java:92)\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startSource(ReplicationWorkerHelper.kt:132)\n\t... 6 more\n",</init></init></init>
  "timestamp" : 1698251684601
}, {
  "failureOrigin" : "replication",
  "internalMessage" : "java.lang.NullPointerException",
  "externalMessage" : "Something went wrong during replication",
  "metadata" : {
    "attemptNumber" : 4,
    "jobId" : 35934
  },
  "stacktrace" : "java.lang.RuntimeException: java.lang.NullPointerException\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startDestination(ReplicationWorkerHelper.kt:116)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$run$0(BufferedReplicationWorker.java:160)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:229)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: java.lang.NullPointerException\n\tat java.base/java.io.Reader.<init>(Reader.java:168)\n\tat java.base/java.io.InputStreamReader.<init>(InputStreamReader.java:124)\n\tat io.airbyte.commons.io.IOs.newBufferedReader(IOs.java:74)\n\tat io.airbyte.commons.io.LineGobbler.<init>(LineGobbler.java:134)\n\tat io.airbyte.commons.io.LineGobbler.gobble(LineGobbler.java:73)\n\tat io.airbyte.workers.internal.DefaultAirbyteDestination.start(DefaultAirbyteDestination.java:142)\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startDestination(ReplicationWorkerHelper.kt:114)\n\t... 6 more\n",</init></init></init>
  "timestamp" : 1698251743538
} ]

This is happening across Postgres, Typeform, and Salesforce Sources and BigQuery and Postgres destinations.

Hmm, I don't know enough to give you a confident answer yet. The error logs you provided indicate a

NullPointerException

during the replication process, but without more context or information, it's hard to pinpoint the exact cause of this issue. However, based on the information extracts, there are a few things you could try: 1. Check your source and destination configurations: Make sure that your source and destination configurations are correct. This includes checking the connection details, credentials, and any specific settings related to the source or destination. For example, for Postgres sources, you might want to refer to the [Postgres troubleshooting guide](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting). 2. Upgrade your Airbyte version: If you're not already on the latest version of Airbyte, consider upgrading. There might have been bug fixes or improvements that could resolve your issue. Here's the guide on [how to upgrade Airbyte](https://docs.airbyte.com/operator-guides/upgrading-airbyte/). 3. Check your system resources: Make sure that your system has enough resources (memory, disk space, etc.) to handle the data replication process. Insufficient resources could potentially lead to failures. If none of these suggestions help, I recommend reaching out to the Airbyte community on their [forum](https://discuss.airbyte.io) or [Slack](https://airbytehq.slack.com) for more specific assistance.

@kapa.ai Also getting this error: 2023-10-25 181002 INFO i.a.w.g.ReplicationWorkerHelper(getReplicationOutput):244 - failures: [ { 76 “failureOrigin” : “replication”, 77 “internalMessage” : “io.airbyte.workers.exception.WorkerException: Failed to create pod for read step”, 78 “externalMessage” : “Something went wrong during replication”, 79 “metadata” : { 80 “attemptNumber” : 0, 81 “jobId” : 35946 82 }, 83 “stacktrace” : “java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Failed to create pod for read step\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startSource(ReplicationWorkerHelper.kt:134)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$run$1(BufferedReplicationWorker.java:161)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:229)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: io.airbyte.workers.exception.WorkerException: Failed to create pod for read step\n\tat io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:188)\n\tat io.airbyte.workers.process.AirbyteIntegrationLauncher.read(AirbyteIntegrationLauncher.java:221)\n\tat io.airbyte.workers.internal.DefaultAirbyteSource.start(DefaultAirbyteSource.java:83)\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startSource(ReplicationWorkerHelper.kt:132)\n\t... 6 more\nCaused by: java.lang.NullPointerException: Cannot invoke \“java.lang.Integer.intValue()\” because the return value of \“io.airbyte.workers.process.KubePortManagerSingleton.take()\” is null\n\tat io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:131)\n\t... 9 more\n”, 84 “timestamp” : 1698257402506 85 }, { 86 “failureOrigin” : “replication”, 87 “internalMessage” : “io.airbyte.workers.exception.WorkerException: Failed to create pod for write step”, 88 “externalMessage” : “Something went wrong during replication”, 89 “metadata” : { 90 “attemptNumber” : 0, 91 “jobId” : 35946 92 }, 93 “stacktrace” : “java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Failed to create pod for write step\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startDestination(ReplicationWorkerHelper.kt:116)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$run$0(BufferedReplicationWorker.java:160)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:229)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: io.airbyte.workers.exception.WorkerException: Failed to create pod for write step\n\tat io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:188)\n\tat io.airbyte.workers.process.AirbyteIntegrationLauncher.write(AirbyteIntegrationLauncher.java:256)\n\tat io.airbyte.workers.internal.DefaultAirbyteDestination.start(DefaultAirbyteDestination.java:135)\n\tat io.airbyte.workers.general.ReplicationWorkerHelper.startDestination(ReplicationWorkerHelper.kt:114)\n\t... 6 more\nCaused by: java.lang.NullPointerException: Cannot invoke \“java.lang.Integer.intValue()\” because the return value of \“io.airbyte.workers.process.KubePortManagerSingleton.take()\” is null\n\tat io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:131)\n\t... 9 more\n”, 94 “timestamp” : 1698257402509 95 } ]

Based on the error logs you've provided, it seems like the issue is related to the creation of pods for both the read and write steps during the replication process. The error messages indicate that the system failed to create these pods, which led to the replication failure. The root cause of this issue appears to be a

NullPointerException

when invoking

java.lang.Integer.intValue()

. This is because the return value of

io.airbyte.workers.process.KubePortManagerSingleton.take()

is null. This could potentially be due to a resource allocation issue or a problem with the Kubernetes configuration. Unfortunately, the provided extracts do not contain a direct solution to this issue. However, they do suggest that similar issues have been encountered by other users. For instance, in [this forum post](https://discuss.airbyte.io/t/3856), a user reported failures when running many jobs concurrently, which also resulted in pod creation failures. Although the exact error messages are not the same, the underlying issue of pod creation failure in a Kubernetes environment is similar. In [another forum post](https://discuss.airbyte.io/t/3221), a user resolved a pod creation issue by adding a specific configuration in the Kubernetes environment. While this might not directly apply to your case, it does suggest that adjusting the Kubernetes configuration might help. Given the complexity of this issue, I would recommend reaching out to the Airbyte community on their [discussion forum](https://discuss.airbyte.io) or [Slack channel](https://airbytehq.slack.com) for further assistance. Be sure to provide them with the detailed error logs you've shared here.

Hi ! I'm having the same issue over here when I try to upgrade to airbyte

0.50.34

It does not seem to be a problem with running many jobs concurrently because I can replicate the error with only one connection syncing. It's also with BigQuery destination

2023-11-15 15:28:13 INFO i.a.w.p.KubePodProcess(<init>):648 - Reading pod IP...
2023-11-15 15:28:13 INFO i.a.w.p.KubePodProcess(<init>):650 - Pod IP: ****
2023-11-15 15:28:13 INFO i.a.w.p.KubePodProcess(<init>):653 - Creating stdin socket...
2023-11-15 15:28:13 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):693 - Setting stderr...
2023-11-15 15:28:13 INFO i.a.w.p.KubePodProcess(close):813 - (pod: airbyte / *****) - Closed all resources for pod
2023-11-15 15:28:13 INFO i.a.w.g.ReplicationWorkerHelper(getReplicationOutput):244 - sync summary: {
  "status" : "failed",
  "startTime" : 1700061968026,
  "endTime" : 1700062093223,
  "totalStats" : {
    "bytesEmitted" : 0,
    "destinationStateMessagesEmitted" : 0,
    "destinationWriteEndTime" : 0,
    "destinationWriteStartTime" : 1700061968040,
    "meanSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBeforeSourceStateMessageEmitted" : 0,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "recordsEmitted" : 0,
    "replicationEndTime" : 1700062093217,
    "replicationStartTime" : 1700061968026,
    "sourceReadEndTime" : 0,
    "sourceReadStartTime" : 1700061968041,
    "sourceStateMessagesEmitted" : 0
  },
  "streamStats" : [ ],
  "performanceMetrics" : {
    "processFromSource" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "readFromSource" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "processFromDest" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "writeToDest" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "readFromDest" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    }
  }
}
  "failureOrigin" : "replication",
  "internalMessage" : "java.lang.NullPointerException",
  "externalMessage" : "Something went wrong during replication",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 19139
  },
  "stacktrace" : "java.lang.RuntimeException: java.lang.NullPointerException
\tat io.airbyte.workers.general.ReplicationWorkerHelper.startDestination(ReplicationWorkerHelper.kt:117)
\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$run$0(BufferedReplicationWorker.java:167)
\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsync$2(BufferedReplicationWorker.java:236)
\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
\tat java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.lang.NullPointerException
\tat java.base/java.io.Reader.<init>(Reader.java:168)
\tat java.base/java.io.InputStreamReader.<init>(InputStreamReader.java:124)
\tat io.airbyte.commons.io.IOs.newBufferedReader(IOs.java:74)
\tat io.airbyte.commons.io.LineGobbler.<init>(LineGobbler.java:134)
\tat io.airbyte.commons.io.LineGobbler.gobble(LineGobbler.java:73)
\tat io.airbyte.workers.internal.DefaultAirbyteDestination.start(DefaultAirbyteDestination.java:100)
\tat io.airbyte.workers.general.ReplicationWorkerHelper.startDestination(ReplicationWorkerHelper.kt:115)
\t... 6 more
",
  "timestamp" : 1700062093121
} ]
...
2023-11-15 15:28:13 INFO i.a.c.i.LineGobbler(voidCall):149 -
2023-11-15 15:28:13 INFO i.a.c.i.LineGobbler(voidCall):149 - ----- END REPLICATION -----

I try to check the error on multiple runs. I have the feeling that it's right at the creation of the

read

job pod and the

write

job pod that something is happening I checked the ressources provided but I'm not finding anything helpful for now ... @Gabriel Levine did you solve this problem on your side ?

I ended up reverting to 0.50.30 and it resolved the issue