https://linen.dev logo
Join Slack
Powered by
Hi All, I have recently deployed the airbyte solu...
# contributing-to-airbyte
k
Hi All, I have recently deployed the airbyte solution on my local machine (Mac OS M1), while trying to connect to a Postgres Source or Destination I'm getting following error:
Copy code
2022-01-19 18:03:25 INFO i.a.w.t.TemporalAttemptExecution(get):118 - Docker volume job log path: /tmp/workspace/ee299288-f423-45c1-a9d4-b3d678894763/0/logs.log
2022-01-19 18:03:25 INFO i.a.w.t.TemporalAttemptExecution(get):123 - Executing worker wrapper. Airbyte version: version not set
2022-01-19 18:03:27 INFO i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):177 - Completing future exceptionally...
io.airbyte.workers.WorkerException: Error while getting checking connection.
	at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:84) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:27) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:174) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.RuntimeException: java.io.IOException: Cannot run program "/tmp/scripts11060788679633185033/image_exists.sh": error=0, Failed to exec spawn helper: pid: 129, exit value: 1
	at io.airbyte.workers.process.DockerProcessFactory.checkImageExists(DockerProcessFactory.java:205) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:96) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:60) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	... 3 more
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 129, exit value: 1
	at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
	at java.lang.ProcessImpl.<init>(ProcessImpl.java:314) ~[?:?]
	at java.lang.ProcessImpl.start(ProcessImpl.java:244) ~[?:?]
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1110) ~[?:?]
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1073) ~[?:?]
	at io.airbyte.workers.process.DockerProcessFactory.checkImageExists(DockerProcessFactory.java:193) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:96) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:60) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	... 3 more
2022-01-19 18:03:27 INFO i.a.w.t.TemporalAttemptExecution(get):144 - Stopping cancellation check scheduling...
2022-01-19 18:03:27 WARN i.t.i.s.POJOActivityTaskHandler(activityFailureToResult):363 - Activity failure. ActivityId=12a45b30-5c21-336e-8a78-dd18f28ee557, activityType=Run, attempt=1
java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Error while getting checking connection.
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:142) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.run(CheckConnectionActivityImpl.java:81) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:286) ~[temporal-sdk-1.6.0.jar:?]
	at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:252) ~[temporal-sdk-1.6.0.jar:?]
	at io.temporal.internal.sync.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:209) ~[temporal-sdk-1.6.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:193) ~[temporal-sdk-1.6.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:151) ~[temporal-sdk-1.6.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73) ~[temporal-sdk-1.6.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
I have checked this issue online so I got some discussion on github https://github.com/airbytehq/airbyte/issues/2017, there was a comment:
Copy code
h7kanna commented on 27 Nov 2021
the above error is because of the airbyte worker not able to create a process on Mac M1

Add this to the worker environment in docker-compose and try once

JAVA_OPTS: "-Djdk.lang.Process.launchMechanism=vfork"
I have tried this to fix the issue, but the issue didn't solve, now the server is restarting many times. Configuration on my machine are following: Exported env variables:
Copy code
export DOCKER_BUILD_PLATFORM=linux/arm64
export DOCKER_BUILD_ARCH=arm64
export ALPINE_IMAGE=arm64v8/alpine:3.14
export POSTGRES_IMAGE=arm64v8/postgres:13-alpine
export JDK_VERSION=17
Java version:
Copy code
kshitijchaurasiya@ckshitij:~/SandboxDev/new_airbyte/airbyte$ java --version
java 17.0.2 2022-01-18 LTS
Java(TM) SE Runtime Environment (build 17.0.2+8-LTS-86)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.2+8-LTS-86, mixed mode, sharing)
Any help will be appreciated 🙂 .
g
@Augustin Lafanechere (Airbyte) could you help here? 😄
c
Hi @kshitij chaurasiya, adding 
- JAVA_OPTS="-Djdk.lang.Process.launchMechanism=vfork"
in the environment variable list of 
airbyte-worker
in 
docker-compose.yaml
worked for me to fix the same problem you mentioned. Did you set the 
JAVA_OPTS
in 
.env
or specifically in 
docker-compose.yaml
as I suggest?
g
Hi @[DEPRECATED] Augustin Lafanechere, Thank you so much for the replying, I have added the JAVA_OPTS in docker-compose.yaml under airbyte-worker section only but after that I'm getting the server issue as below, and not able to access the UI :
Copy code
airbyte-server      | 2022-01-23 17:00:05 ERROR i.a.s.ServerApp(main):277 - Server failed
airbyte-server      | java.util.NoSuchElementException: No value present
airbyte-server      | 	at java.util.Optional.orElseThrow(Optional.java:377) ~[?:?]
airbyte-server      | 	at io.airbyte.server.ServerApp.getServer(ServerApp.java:179) ~[io.airbyte-airbyte-server-0.35.7-alpha.jar:?]
airbyte-server      | 	at io.airbyte.server.ServerApp.main(ServerApp.java:275) [io.airbyte-airbyte-server-0.35.7-alpha.jar:?]
Copy code
airbyte-scheduler   | 2022-01-23 17:10:46 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...
airbyte-scheduler   | 2022-01-23 17:10:48 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...
airbyte-webapp      | 172.27.0.1 - - [23/Jan/2022:17:10:49 +0000] "GET / HTTP/1.1" 200 4506 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" "-"
airbyte-webapp      | 2022/01/23 17:10:50 [error] 92#92: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 172.27.0.1, server: localhost, request: "POST /api/v1/workspaces/list HTTP/1.1", upstream: "<http://172.27.0.7:8001/api/v1/workspaces/list>", host: "localhost:8000", referrer: "<http://localhost:8000/>"
airbyte-webapp      | 172.27.0.1 - - [23/Jan/2022:17:10:50 +0000] "POST /api/v1/workspaces/list HTTP/1.1" 502 494 "<http://localhost:8000/>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" "-"
airbyte-webapp      | 2022/01/23 17:10:50 [error] 92#92: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 172.27.0.1, server: localhost, request: "POST /api/v1/workspaces/list HTTP/1.1", upstream: "<http://172.27.0.7:8001/api/v1/workspaces/list>", host: "localhost:8000", referrer: "<http://localhost:8000/>"
airbyte-webapp      | 172.27.0.1 - - [23/Jan/2022:17:10:50 +0000] "POST /api/v1/workspaces/list HTTP/1.1" 502 494 "<http://localhost:8000/>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" "-"
airbyte-webapp      | 172.27.0.1 - - [23/Jan/2022:17:10:50 +0000] "GET / HTTP/1.1" 200 4506 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" "-"
airbyte-scheduler   | 2022-01-23 17:10:51 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...
airbyte-webapp      | 2022/01/23 17:10:51 [error] 92#92: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 172.27.0.1, server: localhost, request: "POST /api/v1/workspaces/list HTTP/1.1", upstream: "<http://172.27.0.7:8001/api/v1/workspaces/list>", host: "localhost:8000", referrer: "<http://localhost:8000/>"
airbyte-webapp      | 172.27.0.1 - - [23/Jan/2022:17:10:51 +0000] "POST /api/v1/workspaces/list HTTP/1.1" 502 494 "<http://localhost:8000/>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" "-"
airbyte-webapp      | 2022/01/23 17:10:51 [error] 92#92: *8 connect() failed (111: Connection refused) while connecting to upstream, client: 172.27.0.1, server: localhost, request: "POST /api/v1/workspaces/list HTTP/1.1", upstream: "<http://172.27.0.7:8001/api/v1/workspaces/list>", host: "localhost:8000", referrer: "<http://localhost:8000/>"
airbyte-webapp      | 172.27.0.1 - - [23/Jan/2022:17:10:51 +0000] "POST /api/v1/workspaces/list HTTP/1.1" 502 494 "<http://localhost:8000/>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" "-"
airbyte-scheduler   | 2022-01-23 17:10:53 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...
airbyte-scheduler   | 2022-01-23 17:10:55 INFO i.a.s.a.SchedulerApp(waitForServer):210 - Waiting for server to become available...
I have tried many times to redeploy as well but getting the same issue. Do you have face this kind of issue on your setup?
Hi @kshitij chaurasiya no I don't have this problem. I suggest you to try a fresh redeploy again after cleaning volumes: 
docker-compose down
docker-compose rm -v
docker-compose up
a
Hi @[DEPRECATED] Augustin Lafanechere, Actually I got the issue it was due to docker resource management, so by increasing the docker memory it start working 🙂. Thank you so much for the help 🙂 .
a
Hi, This is my first day in the Airbyte open source community. I think it's a great tool. If possible, can someone please name one connector on M1 that's working? One source to destination. So far, I have tried Csv and API as a source, they both failed. Workaround to run connectors would be appreciated. Additionally, can we create our connectors using python or JS? Any links would be helpful. Thank you.
Hi @karan jagota, welcome, did you tried what I mentionned here
Yes, The server still may be starting up.
Okay. Thank you. It worked. But I have to docker-compose thrice. It somehow lost connection in between. Don't know why? But it's a solid workaround. Thank you. I have successfully transferred data from CSV to Postgres, the exchange rate API to Postgres, and the JSON file.
2 Views
Open in Slack
PreviousNext
Powered by