Also asked the same question in the https://github.com/airbytehq/airbyte/pull/8880

I'm also wondering if the Airbyte team has any experience of copying files using this client? https://github.com/fabric8io/kubernetes-client e.g.:

Ok, I just tried to replace "kubectl cp" with commands from the Fabric8 Kubernetes Client (

.upload(tmpFile)

), it seems that this will not help to get around the lack of privileges:

java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Error while getting spec from image image_name
...
Caused by: io.airbyte.workers.exception.WorkerException: An error has occurred.
...
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
...
Caused by: io.fabric8.kubernetes.client.http.WebSocketHandshakeException
...
Suppressed: java.lang.Throwable: waiting here
...
Caused by: java.net.ProtocolException: Expected HTTP 101 response but was '403 Forbidden'
        at okhttp3.internal.ws.RealWebSocket.checkUpgradeSuccess$okhttp(RealWebSocket.kt:224) ~[okhttp-4.9.3.jar:?]
        at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:170) ~[okhttp-4.9.3.jar:?]
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519) ~[okhttp-4.9.3.jar:?]
        ... 3 more

Maybe it's stupid to install ssh and copy files using scp?

Or is it possible to use the shared persistence volume instead of copying files between pods using "kubectl cp"?

Thanks folks, in the end we could not solve this problem and decided to use a solution based on custom Python connectors that will be orchestrated by Airflow. But the reason, of course, is not in Airbyte itself, but in the security settings on a specific local Kubernetes cluster, which we could not bypass. I liked your product itself, and I think that I can use it on other projects. Good luck guys!