hi team! thanks for a wonderful product, best wishes for the future. I got a small issue trying to figure it out, I am running Airbyte under docker hosted on AWS A1-ARM64 instance (A1.xlarge) Everything is running fine and working great until it comes to the basic normalization that fails with the following error in the logs:

airbyte-worker      | 2022-10-11 07:02:14 INFO i.a.w.p.DockerProcessFactory(create):163 - Preparing command: docker run --rm --init -i -w /data/1/0/normalize --log-driver none --name normalization-normalize-1-0-jmoyq --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e AIRBYTE_ROLE= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_VERSION=0.40.14 airbyte/normalization:0.2.22 run --integration-type postgres --config destination_config.json --catalog destination_catalog.json
airbyte-worker      | 2022-10-11 07:02:16 normalization > [FATAL tini (7)] exec /airbyte/entrypoint.sh failed: Exec format error
airbyte-worker      | 2022-10-11 07:02:16 INFO i.a.w.g.DefaultNormalizationWorker(run):82 - Normalization executed in 36 seconds.
airbyte-worker      | 2022-10-11 07:02:16 INFO i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):164 - Completing future exceptionally...
airbyte-worker      | io.airbyte.workers.exception.WorkerException: Normalization Failed.
airbyte-worker      |   at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:92) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
airbyte-worker      |   at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:27) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
airbyte-worker      |   at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.14.jar:?]
airbyte-worker      |   at java.lang.Thread.run(Thread.java:833) ~[?:?]

I ran the same installation on my localhost (Apple M1) and all was working fine and got good results. Used the below for deployments: - git clone https://github.com/airbytehq/airbyte.git - cd airbyte - docker-compose up Appreciate any tips if possible.

Hi. It fails in the normalization step i.e. in dbt. Are you using the same airbyte versions in both cases AND the same source version?

yes, everything is exactly the same on the server vs local (where local passes)

That is very odd. some thing is 100% different some where. what is the source?

i have ran the same exact commands on both deployments: - git clone https://github.com/airbytehq/airbyte.git - cd airbyte - docker-compose up did not change anything in the configs/env @Eli Sigal i usually have this issue with A1-arm64 due to a lack of support for V8, but nothing in the logs that shows it.

it`s not a v8 issue since airbyte source finished the job and the dbt model did not. Do you have any logs to share? We are missing some thing here

i made a new deployment on a linux instead of ubuntu and it failed again with same reason, can i send you the logs on private? @Eli Sigal

Yes sure

@[DEPRECATED] Marcos Marx turned this thread into Zendesk ticket 2720 to ensure timely resolution!

to add the findings in here, the issues seems to be related to deployments under A1-Arm64V8 AWS instances, the failiur is always the same:

at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Normalization Failed.
	at io.temporal.serviceclient.CheckedExceptionWrapper.wrap(CheckedExceptionWrapper.java:56) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.sync.WorkflowInternal.wrap(WorkflowInternal.java:448) ~[temporal-sdk-1.8.1.jar:?]
	at io.temporal.activity.Activity.wrap(Activity.java:51) ~[temporal-sdk-1.8.1.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:138) ~[io.airbyte-airbyte-workers-0.40.10.jar:?]
	at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$3(NormalizationActivityImpl.java:121) ~[io.airbyte-airbyte-workers-0.40.10.jar:?]
	at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:273) ~[io.airbyte-airbyte-commons-temporal-0.40.10.jar:?]
	... 14 more
Caused by: java.util.concurrent.ExecutionException: io.airbyte.workers.exception.WorkerException: Normalization Failed.
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.get(TemporalAttemptExecution.java:132) ~[io.airbyte-airbyte-workers-0.40.10.jar:?]
	at io.airbyte.workers.temporal.sync.NormalizationActivityImpl.lambda$normalize$3(NormalizationActivityImpl.java:121) ~[io.airbyte-airbyte-workers-0.40.10.jar:?]
	at io.airbyte.commons.temporal.TemporalUtils.withBackgroundHeartbeat(TemporalUtils.java:273) ~[io.airbyte-airbyte-commons-temporal-0.40.10.jar:?]
	... 14 more
Caused by: io.airbyte.workers.exception.WorkerException: Normalization Failed.
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:92) ~[io.airbyte-airbyte-workers-0.40.10.jar:?]
	at io.airbyte.workers.general.DefaultNormalizationWorker.run(DefaultNormalizationWorker.java:27) ~[io.airbyte-airbyte-workers-0.40.10.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:161) ~[io.airbyte-airbyte-workers-0.40.10.jar:?]
	... 1 more

Hi! Great to see the investigative work here. Could you please open a GitHub issue with your findings?

Hi! Great to see the investigative work here. Could you please open a GitHub issue with your findings?

Hi. An issue was opened by @Mark Elayan and it was resolved by changing the instance to x86 so the issue that airbyte does not support AWS A1-ARM64 instance (A1.xlarge) We validated by also trying mysql server

sorry guys, didnt get to open the issue yet, after further investigating, it seems like the dbt for basic transformation is not compatible with the used ARM64 arch, which fails to start the container and drops the job. Will add all the findings in the issue with logs soon.

Thanks so much for the update, Mark! When you get around to it - could you please link the issue here as well? That way we can keep better track of edge cases like this :)

Thanks so much for the update, Mark! When you get around to it - could you please link the issue here as well? That way we can keep better track of edge cases like this :)