Sree Charan
01/18/2022, 5:25 PMDanny Dinges
01/19/2022, 1:41 AMkshitij chaurasiya
01/19/2022, 6:05 PM2022-01-19 18:03:25 INFO i.a.w.t.TemporalAttemptExecution(get):118 - Docker volume job log path: /tmp/workspace/ee299288-f423-45c1-a9d4-b3d678894763/0/logs.log
2022-01-19 18:03:25 INFO i.a.w.t.TemporalAttemptExecution(get):123 - Executing worker wrapper. Airbyte version: version not set
2022-01-19 18:03:27 INFO i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):177 - Completing future exceptionally...
io.airbyte.workers.WorkerException: Error while getting checking connection.
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:84) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:27) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:174) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.RuntimeException: java.io.IOException: Cannot run program "/tmp/scripts11060788679633185033/image_exists.sh": error=0, Failed to exec spawn helper: pid: 129, exit value: 1
at io.airbyte.workers.process.DockerProcessFactory.checkImageExists(DockerProcessFactory.java:205) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:96) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:60) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
... 3 more
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 129, exit value: 1
at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
at java.lang.ProcessImpl.<init>(ProcessImpl.java:314) ~[?:?]
at java.lang.ProcessImpl.start(ProcessImpl.java:244) ~[?:?]
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1110) ~[?:?]
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1073) ~[?:?]
at io.airbyte.workers.process.DockerProcessFactory.checkImageExists(DockerProcessFactory.java:193) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:96) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:60) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
... 3 more
2022-01-19 18:03:27 INFO i.a.w.t.TemporalAttemptExecution(get):144 - Stopping cancellation check scheduling...
2022-01-19 18:03:27 WARN i.t.i.s.POJOActivityTaskHandler(activityFailureToResult):363 - Activity failure. ActivityId=12a45b30-5c21-336e-8a78-dd18f28ee557, activityType=Run, attempt=1
java.util.concurrent.ExecutionException: io.airbyte.workers.WorkerException: Error while getting checking connection.
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:142) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
at io.airbyte.workers.temporal.check.connection.CheckConnectionActivityImpl.run(CheckConnectionActivityImpl.java:81) ~[io.airbyte-airbyte-workers-0.35.5-alpha.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityInboundCallsInterceptor.execute(POJOActivityTaskHandler.java:286) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.sync.POJOActivityTaskHandler$POJOActivityImplementation.execute(POJOActivityTaskHandler.java:252) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.sync.POJOActivityTaskHandler.handle(POJOActivityTaskHandler.java:209) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:193) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:151) ~[temporal-sdk-1.6.0.jar:?]
at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:73) ~[temporal-sdk-1.6.0.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
I have checked this issue online so I got some discussion on github https://github.com/airbytehq/airbyte/issues/2017, there was a comment:
h7kanna commented on 27 Nov 2021
the above error is because of the airbyte worker not able to create a process on Mac M1
Add this to the worker environment in docker-compose and try once
JAVA_OPTS: "-Djdk.lang.Process.launchMechanism=vfork"
I have tried this to fix the issue, but the issue didn't solve, now the server is restarting many times.
Configuration on my machine are following:
Exported env variables:
export DOCKER_BUILD_PLATFORM=linux/arm64
export DOCKER_BUILD_ARCH=arm64
export ALPINE_IMAGE=arm64v8/alpine:3.14
export POSTGRES_IMAGE=arm64v8/postgres:13-alpine
export JDK_VERSION=17
Java version:
kshitijchaurasiya@ckshitij:~/SandboxDev/new_airbyte/airbyte$ java --version
java 17.0.2 2022-01-18 LTS
Java(TM) SE Runtime Environment (build 17.0.2+8-LTS-86)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.2+8-LTS-86, mixed mode, sharing)
Any help will be appreciated 🙂 .Gorkem Yurtseven
01/19/2022, 7:59 PMAriyo Kabir
01/20/2022, 10:50 AMMoshe Mazuz
01/20/2022, 11:17 AMNick Miller
01/20/2022, 6:01 PMIncremental - Append
. For typical SQL DBs it seems to reduce down to SELECT * from table where cursor_column >= cursor_last_ran;
. I’ve previously built implementations with postgres similar to this, and one issue I’ve encountered with this that if there is a long running transaction/insert that the SELECT *
may finish and update cursor_last_ran
before the long running transaction finishes.
A pretty common default would be a column called updated_at
with a trigger/default value for NOW()
. In postgres if you’re running inserts/updates, the updated_at
column is updated when the row is updated. But that isn’t visible to the rest of the database until the transaction is committed, which could potentially be many seconds/minutes/hours later. The problem is that the updated_at
value stays the same time after commit. Is this something Airbyte has an approach to handle?Wadii Zaim
01/21/2022, 2:58 PM./gradlew format
but keep having Python call failed: .venv/bin/python -m flake8 . --config /Users/***/projects/airbyte/tools/python/.flake8
in my source directory. From previous threads I tried removing the .venv but doesn’t work.
Any idea please ?Guido Turturici
01/21/2022, 7:12 PMsource-bigcommerce
with the catalog/products
stream, but we are dealing with the acceptance-tests
. If we pull direct from main branch and run the acceptance test, we are having some issues directly there without making any changes.
PR
Do you have any idea if this connector have some previous issues with those testing or could be something about our environment?
Thanks in advance 🙂Jens
01/25/2022, 8:16 AMkananindzya
01/25/2022, 5:20 PMHarsha Teja Kanna
01/25/2022, 10:03 PMramin
01/26/2022, 8:39 AMKelvin Omereshone
02/02/2022, 9:29 AMkananindzya
02/02/2022, 11:24 AMAmeya Bapat
02/03/2022, 7:39 AMairbyte-integrations/connectors/source-jdbc/src/main/java/io/airbyte/integrations/source/jdbc/AbstractJdbcSource.java:265
LOGGER.info("Actual query : {}", sql);
I ran ./gradlew :airbyte-integrations:connectors:source-snowflake:build
and ./gradlew :airbyte-integrations:connectors:source-jdbc:build
and VERSION=dev docker-compose up
but it is not showing my Looger.info() statements in job's log.
I can see the previous and later logger statements but mine. Am i missing any step?Arian Giles Garcia
02/03/2022, 8:05 PMSUB_BUILD=PLATFORM ./gradlew clean build
, it gets stuck and keeps printing the Database is not ready yet. Please wait a moment, it might still be initializing...
message over and over again.. Do you know what I might be doing wrong? 🤔 Thank you very much! 🙂Naveen Sai Patnana
02/07/2022, 4:17 PM2022-02-07 09:24:27 WARN i.t.i.r.GrpcSyncRetryer(retry):56 - Retrying
after failure
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded
after 69.999905719s. [closed=[],
open=[[remote_addr=airbyte-temporal/192.168.144.6:7233]]]
at
io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
~[grpc-stub-1.42.1.jar:1.42.1]
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
~[grpc-stub-1.42.1.jar:1.42.1]
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
~[grpc-stub-1.42.1.jar:1.42.1]
at
io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getWorkflowExecutionHistory(WorkflowServiceGrpc.java:2642)
~[temporal-servicecl
ient-1.6.0.jar:?]
at
io.temporal.internal.client.WorkflowClientLongPollHelper.lambda$getInstanceCloseEvent$0(WorkflowClientLongPollHelper.java:143)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61)
~[temporal-serviceclient-1.6.0.jar:?]
at
io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51)
~[temporal-serviceclient-1.6.0.jar:?]
at
io.temporal.internal.client.WorkflowClientLongPollHelper.getInstanceCloseEvent(WorkflowClientLongPollHelper.java:131)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.client.WorkflowClientLongPollHelper.getWorkflowExecutionResult(WorkflowClientLongPollHelper.java:72)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.client.RootWorkflowClientInvoker.getResult(RootWorkflowClientInvoker.java:93)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:243)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:225)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:315)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:270)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178)
~[temporal-sdk-1.6.0.jar:?]
at jdk.proxy2.$Proxy40.run(Unknown Source) ~[?:?]
at
io.airbyte.workers.temporal.TemporalClient.lambda$submitSync$3(TemporalClient.java:148)
~[io.airbyte-airbyte-workers-0.35.15-alpha.jar:?]
at
io.airbyte.workers.temporal.TemporalClient.execute(TemporalClient.java:439)
~[io.airbyte-airbyte-workers-0.35.15-alpha.jar:?]
at
io.airbyte.workers.temporal.TemporalClient.submitSync(TemporalClient.java:147)
~[io.airbyte-airbyte-workers-0.35.15-alpha.jar:?]
at
io.airbyte.workers.worker_run.TemporalWorkerRunFactory.lambda$createSupplier$0(TemporalWorkerRunFactory.java:83)
~[io.airbyte-airbyte-workers-0.35.15-alpha.jar:?]
at io.airbyte.workers.worker_run.WorkerRun.call(WorkerRun.java:51)
[io.airbyte-airbyte-workers-0.35.15-alpha.jar:?]
at io.airbyte.workers.worker_run.WorkerRun.call(WorkerRun.java:22)
[io.airbyte-airbyte-workers-0.35.15-alpha.jar:?]
at
io.airbyte.commons.concurrency.LifecycledCallable.execute(LifecycledCallable.java:94)
[io.airbyte-airbyte-commons-0.35.15-alpha.jar:?]
at
io.airbyte.commons.concurrency.LifecycledCallable.call(LifecycledCallable.java:78)
[io.airbyte-airbyte-commons-0.35.15-alpha.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Raj
02/07/2022, 5:30 PMMohamed Magdy
02/08/2022, 5:03 PMWill Sargent
02/09/2022, 2:25 PMBlake Enyart
02/12/2022, 4:06 AMThanh Le
02/12/2022, 11:00 AMNaveen Sai Patnana
02/14/2022, 12:23 PMEthan Veres
02/16/2022, 3:04 PMMadhu Prabhakara
02/20/2022, 8:18 PM*> Task :airbyte-integrations:connectors:source-smartsheets:flakeCheck* FAILED
[python] .venv/bin/python -m flake8 . --config /home/ubuntu/airbyte/tools/python/.flake8
./source_smartsheets/source.py:112:17: F841 local variable 'columns' is assigned to but never used
FAILURE: Build failed with an exception.
Any clue on what might be wrong....? I am pretty new to this so it possible that I am doing something terribly wrong...kananindzya
02/21/2022, 11:44 AMactor
?
I just noticed that all credentials stored as plain text in configuration
field.Keshav Agarwal
02/24/2022, 3:35 AMdocker cp airbyte-server:/tmp/workspace/${NORMALIZE_WORKSPACE}/build/run/airbyte_utils/models/generated/ models/
I get
Error: No such container:path: airbyte-server:/tmp/workspace/38450/0/build/run/airbyte_utils/models/generated/
Am I following the right doc or is there any mistake here?Keshav Agarwal
02/24/2022, 4:06 AMdocker run --rm -i -v airbyte_workspace:/data -w /data/$NORMALIZE_WORKSPACE/normalize --network host --entrypoint /usr/local/bin/dbt airbyte/normalization debug --profiles-dir=. --project-dir=.
I get
Running with dbt=0.21.1
dbt version: 0.21.1
python version: 3.8.12
python path: /usr/local/bin/python
os info: Linux-5.4.0-99-generic-x86_64-with-glibc2.2.5
Using profiles.yml file at /data/38450/0/normalize/profiles.yml
Using dbt_project.yml file at /data/38450/0/normalize/dbt_project.yml
Configuration:
profiles.yml file [ERROR invalid]
dbt_project.yml file [OK found and valid]
Required dependencies:
- git [OK found]
1 check failed:
Profile loading failed for the following reason:
Runtime Error
Credentials in profile "normalize", target "prod" invalid: Runtime Error
Could not find adapter type clickhouse!
Anything that can I do?Pras
02/24/2022, 6:45 AM