Hi I m trying to set up a postgres source and postgres desti Airbyte #replication-troubleshooting

Hi, I'm trying to set up a postgres source and pos...

Alberto Colon

10/13/2022, 2:31 PM

Hi, I'm trying to set up a postgres source and postgres destination. We are using airbyte (0.40.14) on arm64 in a k3s cluster. When I try to create the postgres source, a pod is created like this:

rce-postgres-check-9688c843-03b2-40af-9b99-98b9d45e18f9-0-ixmtc   0/4     Init:Error   0               6m58s   10.42.7.8     b020114-47b9054   <none>           <none>

The logs of this pod:

Timeout while attempting to copy to init container, exiting with code 1...

and in the airbyte UI we get this error (attachment): Do I need to setup something else in the env file for kubernetes? I really don't have any clue about this bug/error. Thanks in advance...

👀 1

✍️ 1

user

10/13/2022, 3:53 PM

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2802 to ensure timely resolution!

Harshith (Airbyte)

10/13/2022, 4:01 PM

Hey looks to me like the copy data to pods is not working. Can you check if the required permissions are given?

Alberto Colon

10/13/2022, 4:44 PM

Thanks for answering. Where I can edit those permissions?

Alberto Colon

10/13/2022, 4:44 PM

Copy code

AIRBYTE_VERSION=0.40.14

# Airbyte Internal Database, see <https://docs.airbyte.io/operator-guides/configuring-airbyte-db>
DATABASE_HOST=airbyte-db-svc
DATABASE_PORT=5432
DATABASE_DB=airbyte
# translate manually DATABASE_URL=jdbc:postgresql://${DATABASE_HOST}:${DATABASE_PORT/${DATABASE_DB}
DATABASE_URL=jdbc:<postgresql://airbyte-db-svc:5432/airbyte>
JOBS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.29.15.001
CONFIGS_DATABASE_MINIMUM_FLYWAY_MIGRATION_VERSION=0.35.15.001

# When using the airbyte-db via default docker image:
CONFIG_ROOT=/configs
DATA_DOCKER_MOUNT=airbyte_data
DB_DOCKER_MOUNT=airbyte_db

RUN_DATABASE_MIGRATION_ON_STARTUP=true

# <http://Temporal.io|Temporal.io> worker configuration
TEMPORAL_HOST=airbyte-temporal-svc:7233
TEMPORAL_WORKER_PORTS=9001,9002,9003,9004,9005,9006,9007,9008,9009,9010,9011,9012,9013,9014,9015,9016,9017,9018,9019,9020,9021,9022,9023,9024,9025,9026,9027,9028,9029,9030,9031,9032,9033,9034,9035,9036,9037,9038,9039,9040

# Workspace storage for running jobs (logs, etc)
WORKSPACE_ROOT=/workspace
WORKSPACE_DOCKER_MOUNT=airbyte_workspace

LOCAL_ROOT=/tmp/airbyte_local

# Miscellaneous
TRACKING_STRATEGY=segment
WEBAPP_URL=airbyte-webapp-svc:80
API_URL=/api/v1/
INTERNAL_API_HOST=airbyte-server-svc:8001
FULLSTORY=disabled
IS_DEMO=disabled

WORKER_ENVIRONMENT=kubernetes
LOG_LEVEL=INFO

# S3/Minio Log Configuration
S3_LOG_BUCKET=airbyte-dev-logs
S3_LOG_BUCKET_REGION=
S3_MINIO_ENDPOINT=<http://airbyte-minio-svc:9000>
S3_PATH_STYLE_ACCESS=true

# GCS Log Configuration
GCS_LOG_BUCKET=

# State Storage Configuration
STATE_STORAGE_MINIO_BUCKET_NAME=airbyte-dev-logs
STATE_STORAGE_MINIO_ENDPOINT=<http://airbyte-minio-svc:9000>

# Docker Resource Limits
JOB_MAIN_CONTAINER_CPU_REQUEST=
JOB_MAIN_CONTAINER_CPU_LIMIT=
JOB_MAIN_CONTAINER_MEMORY_REQUEST=
JOB_MAIN_CONTAINER_MEMORY_LIMIT=

NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_LIMIT=
NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_REQUEST=
NORMALIZATION_JOB_MAIN_CONTAINER_CPU_LIMIT=
NORMALIZATION_JOB_MAIN_CONTAINER_CPU_REQUEST=

# Worker #
WORKERS_MICRONAUT_ENVIRONMENTS=control-plane

# Worker pod tolerations, annotations and node selectors
JOB_KUBE_TOLERATIONS=
JOB_KUBE_ANNOTATIONS=
JOB_KUBE_NODE_SELECTORS=

# Job image pull policy
JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_POLICY=

# Launch a separate pod to orchestrate sync steps
CONTAINER_ORCHESTRATOR_ENABLED=true
CONTAINER_ORCHESTRATOR_IMAGE=

CONNECTOR_SPECIFIC_RESOURCE_DEFAULTS_ENABLED=true

# Cloud Storage Configuration
WORKER_LOGS_STORAGE_TYPE=MINIO
WORKER_STATE_STORAGE_TYPE=MINIO

# Open Telemetry Configuration
METRIC_CLIENT=
OTEL_COLLECTOR_ENDPOINT=

# Temporal Activity configuration
ACTIVITY_MAX_ATTEMPT=
ACTIVITY_INITIAL_DELAY_BETWEEN_ATTEMPTS_SECONDS=
ACTIVITY_MAX_DELAY_BETWEEN_ATTEMPTS_SECONDS=
WORKFLOW_FAILURE_RESTART_DELAY_SECONDS=

USE_STREAM_CAPABLE_STATE=true

Alberto Colon

10/13/2022, 4:44 PM

This is my env file btw

Alberto Colon

10/13/2022, 7:12 PM

Copy code

2022-10-11 19:52:51 [32mINFO[m i.a.w.p.KubePodProcess(<init>):551 - Copying files...
2022-10-11 19:52:51 [32mINFO[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):258 - Uploading file: source_config.json
2022-10-11 19:52:51 [32mINFO[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):266 - kubectl cp /tmp/f970b42d-6dda-410e-ae64-8575a5f25dec/source_config.json airbyte/rce-postgres-check-795890cb-dbed-4aee-8365-9bcedaed9e28-0-vhfdy:/config/source_config.json -c init
2022-10-11 19:52:53 [32mINFO[m i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):269 - Waiting for kubectl cp to complete
2022-10-11 19:52:53 [1;31mERROR[m i.a.w.g.DefaultCheckConnectionWorker(run):98 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: java.io.IOException: kubectl cp failed with exit code 126
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:138) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.lang.RuntimeException: java.io.IOException: kubectl cp failed with exit code 126
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:285) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:552) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	... 5 more
Caused by: java.io.IOException: kubectl cp failed with exit code 126
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:279) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:552) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	... 5 more
2022-10-11 19:52:53 [32mINFO[m i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):164 - Completing future exceptionally...
io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:99) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.airbyte.workers.exception.WorkerException: java.io.IOException: kubectl cp failed with exit code 126
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:138) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	... 3 more
Caused by: java.lang.RuntimeException: java.io.IOException: kubectl cp failed with exit code 126
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:285) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:552) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	... 3 more
Caused by: java.io.IOException: kubectl cp failed with exit code 126
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:279) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:552) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	... 3 more
2022-10-11 19:52:53 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2022-10-11 19:52:53 [33mWARN[m i.t.i.a.POJOActivityTaskHandler(activityFailureToResult):307 - Activity failure. ActivityId=a6dbcea0-10d5-3cc6-8a9c-b6389c0367eb, activityType=RunWithJobOutput, attempt=1
java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:132) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:214) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:180) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.activity.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:120) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:204) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:164) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.8.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:99) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	... 1 more
Caused by: io.airbyte.workers.exception.WorkerException: java.io.IOException: kubectl cp failed with exit code 126
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:138) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	... 1 more
Caused by: java.lang.RuntimeException: java.io.IOException: kubectl cp failed with exit code 126
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:285) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:552) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	... 1 more
Caused by: java.io.IOException: kubectl cp failed with exit code 126
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:279) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:552) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	... 1 more

Alberto Colon

10/13/2022, 7:12 PM

This is the log that I'm getting.

Harshith (Airbyte)

10/17/2022, 4:49 AM

Got it. Can you share the logs of worker? also could you create a issue in github so that team can look into that and get back to you

Alberto Colon

10/18/2022, 3:57 PM

Copy code

io.grpc.StatusRuntimeException: UNAVAILABLE: Unable to resolve host airbyte-temporal-svc
Caused by: java.lang.RuntimeException: java.net.UnknownHostException: airbyte-temporal-svc
Caused by: java.net.UnknownHostException: airbyte-temporal-svc
WARNING: [Channel<1>: (airbyte-temporal-svc:7233)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host airbyte-temporal-svc, cause=java.lang.RuntimeException: java.net.UnknownHostException: airbyte-temporal-svc
Caused by: java.net.UnknownHostException: airbyte-temporal-svc
WARNING: [Channel<1>: (airbyte-temporal-svc:7233)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host airbyte-temporal-svc, cause=java.lang.RuntimeException: java.net.UnknownHostException: airbyte-temporal-svc
Caused by: java.net.UnknownHostException: airbyte-temporal-svc
Registering AWS S3 publish helper -> S3 configuration (airbyte-dev-logs:job-logging/workspace/a9dbeae4-fe1d-4262-bbac-366f439e7861/0/logs.log in region null; compressed: false)
Publishing to S3 (bucket=airbyte-dev-logs; key=job-logging/workspace/a9dbeae4-fe1d-4262-bbac-366f439e7861/0/logs.log/20221017165522_airbyte-worker-7d8cfbf4b6-dmdlh_edfa156a237f4770b2ac979902d42386):
Log4j2Appender says: Attempting to start pod = ion-postgres-check-a9dbeae4-fe1d-4262-bbac-366f439e7861-0-jylbh for airbyte/destination-postgres:0.3.26 with resources io.airbyte.config.ResourceRequirements@3c3e9801[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
2022-10-17 16:55:23 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = ion-postgres-check-a9dbeae4-fe1d-4262-bbac-366f439e7861-0-jylbh for airbyte/destination-postgres:0.3.26 with resources io.airbyte.config.ResourceRequirements@3c3e9801[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=]
Publishing to S3 (bucket=airbyte-dev-logs; key=job-logging/workspace/a9dbeae4-fe1d-4262-bbac-366f439e7861/0/logs.log/20221017165622_airbyte-worker-7d8cfbf4b6-dmdlh_edfa156a237f4770b2ac979902d42386):
io.airbyte.workers.exception.WorkerException: java.io.IOException: kubectl cp failed with exit code 126
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:138) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:285) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:552) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:279) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:552) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:99) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
Caused by: io.airbyte.workers.exception.WorkerException: java.io.IOException: kubectl cp failed with exit code 126
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:138) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:285) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:552) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:279) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:552) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:132) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
Caused by: io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:99) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
Caused by: io.airbyte.workers.exception.WorkerException: java.io.IOException: kubectl cp failed with exit code 126
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:138) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:285) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:552) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:134) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
io.temporal.failure.ActivityFailure: scheduledEventId=6, startedEventId=7, activityType='RunWithJobOutput', activityId='451df1a0-2771-3178-b89d-8571db2ca446', identity='1@airbyte-worker-7d8cfbf4b6-dmdlh', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
	at io.airbyte.workers.temporal.check.connection.CheckConnectionWorkflowImpl.run(CheckConnectionWorkflowImpl.java:40) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.support.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.java:79) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
Caused by: io.temporal.failure.ApplicationFailure: message='io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.', type='java.util.concurrent.ExecutionException', nonRetryable=false
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:132) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.runWithJobOutput(CheckConnectionActivityImpl.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
Caused by: io.temporal.failure.ApplicationFailure: message='Unexpected error while getting checking connection.', type='io.airbyte.workers.exception.WorkerException', nonRetryable=false
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:99) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
Caused by: io.temporal.failure.ApplicationFailure: message='java.io.IOException: kubectl cp failed with exit code 126', type='io.airbyte.workers.exception.WorkerException', nonRetryable=false
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:138) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:86) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:60) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:36) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:285) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:552) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
....

Alberto Colon

10/18/2022, 3:57 PM

@Harshith (Airbyte) made an issue here: https://github.com/airbytehq/airbyte/issues/18079

user

10/19/2022, 11:38 AM

Thanks a lot. Team will get back to you soon

Alberto Colon

11/10/2022, 3:59 AM

Still need help with this.

user

11/10/2022, 9:07 AM

Sure asked one of my colleagues to check it. They should get back soon.

Marcos Marx (Airbyte)

11/22/2022, 6:11 PM

@Alberto Colon are this happening to all sources and destinations? I don’t think it isn’t an issue with your deployment. Did you try to deploy using Helm Chart instead of Kustomize?

Alberto Colon

11/22/2022, 6:23 PM

Hi, this is a problem with the kubectl version that the image is using. Is not supported by ARM64. I had to do a new image for the kubectl. The airbyte team should look at this.

Alberto Colon

11/22/2022, 6:25 PM

https://hub.docker.com/layers/airbyte/worker/0.40.22/images/sha256-7f692d594d549864c6590de7a56af7854b5c8f973c398228429566c01e86fffa?context=explore

Alberto Colon

11/22/2022, 6:26 PM

looking at the new version is still using AMD in the Dockerfile

Alberto Colon

11/22/2022, 6:28 PM

For the worker Dockerfile I had to do this:

Alberto Colon

11/22/2022, 6:28 PM

Copy code

ARG JDK_IMAGE=airbyte/airbyte-base-java-image:1.0
FROM ${JDK_IMAGE} AS worker

RUN amazon-linux-extras install -y docker
RUN yum install -y jq tar && yum clean all

RUN curl -LO "<https://dl.k8s.io/release/v1.24.7/bin/linux/arm64/kubectl>" \
        && chmod +x kubectl && mv kubectl /usr/local/bin/

# Don't change this manually.  Bump version expects to make moves based on this string
ARG VERSION=0.40.17

ENV APPLICATION airbyte-workers
ENV VERSION ${VERSION}

WORKDIR /app

# Move worker app
ADD bin/${APPLICATION}-${VERSION}.tar /app

# wait for upstream dependencies to become available before starting server
ENTRYPOINT ["/bin/bash", "-c", "${APPLICATION}-${VERSION}/bin/${APPLICATION}"]

Marcos Marx (Airbyte)

11/22/2022, 6:28 PM

Can you create a Github issue for that?

Alberto Colon

11/22/2022, 6:29 PM

https://github.com/airbytehq/airbyte/issues/18079

Alberto Colon

11/22/2022, 6:29 PM

Already did

Alberto Colon

11/22/2022, 6:29 PM

of course I didn't add the stuff that I did extra because I dont know if thats the correct path.

Marcos Marx (Airbyte)

11/22/2022, 6:30 PM

Can you add it? Probably will help to understand the problem and possible solutions!

Alberto Colon

11/22/2022, 6:30 PM

Ok, right now airbyte works with that Dockerfile I built.

Alberto Colon

11/22/2022, 6:30 PM

I have other issues now... hehe

Alberto Colon

11/22/2022, 6:38 PM

https://github.com/airbytehq/airbyte/issues/18079

Alberto Colon

11/22/2022, 6:38 PM

Updated the answer

89 Views

Open in Slack

Previous Next