We are deploying airbyte as part of our kubernetes cluster i Airbyte #replication-troubleshooting

We are deploying airbyte as part of our kubernetes...

Mohit Reddy

10/13/2022, 3:48 AM

We are deploying airbyte as part of our kubernetes cluster in a namespace which by default injects a sidecar to every pod which gets created in it. This causes the following logs -

Copy code

2022-10-13 03:37:17 INFO i.a.w.p.KubePodProcess(<init>):572 - Pod IP: 10.105.236.201
2022-10-13 03:37:17 INFO i.a.w.p.KubePodProcess(<init>):579 - Using null stdin output stream...
2022-10-13 03:37:17 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):98 - Unexpected error while checking connection: 
java.lang.NullPointerException: null
	at java.io.Reader.<init>(Reader.java:168) ~[?:?]
	at java.io.InputStreamReader.<init>(InputStreamReader.java:112) ~[?:?]
	at io.airbyte.commons.io.IOs.newBufferedReader(IOs.java:120) ~[io.airbyte-airbyte-commons-0.40.14.jar:?]
	at io.airbyte.commons.io.LineGobbler.<init>(LineGobbler.java:99) ~[io.airbyte-airbyte-commons-0.40.14.jar:?]
	at io.airbyte.commons.io.LineGobbler.gobble(LineGobbler.java:67) ~[io.airbyte-airbyte-commons-0.40.14.jar:?]
	at io.airbyte.commons.io.LineGobbler.gobble(LineGobbler.java:28) ~[io.airbyte-airbyte-commons-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:65) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar

To get around this, we disable injecting the sidecar by adding an annotation to the jobs - https://docs.airbyte.com/operator-guides/configuring-airbyte/#jobs-specific (specifically CHECK_JOB_KUBE_ANNOTATIONS). We recently upgraded airbyte from 0.39.1 to 0.40.14 and this has started to fail again i.e. the annotation is not being applied. Any help here?

✍️ 1

Mohit Reddy

10/13/2022, 3:52 AM

This is from one of the workers -

Copy code

bash-4.2# echo $CHECK_JOB_KUBE_ANNOTATIONS
<http://linkerd.io/inject=disabled|linkerd.io/inject=disabled>
bash-4.2#
bash-4.2#
bash-4.2#
bash-4.2# echo $JOB_KUBE_ANNOTATIONS
<http://linkerd.io/inject=disabled|linkerd.io/inject=disabled>

Mohit Reddy

10/13/2022, 4:40 AM

I think the issue is this - https://github.com/airbytehq/airbyte/blob/759a27ec453debbaefa623292995befc6db3c0b9/airbyte-workers/src/main/resources/application.yml#L91 CHECK_JOB_KUBE_ANNOTATION v/s CHECK_JOB_KUBE_ANNOTATIONS env var 😞

user

10/13/2022, 3:53 PM

@[DEPRECATED] Marcos Marx turned this thread into Zendesk ticket 2808 to ensure timely resolution!

user

10/13/2022, 5:22 PM

Hi! I think you're onto it, I will ping the dev team and get back to you!

🙏 1

Mohit Reddy

10/13/2022, 7:10 PM