Hi team, I have locally modified an existing connector and uploaded its image to docker hub in order to use it on my production airbyte setup, which runs on GCP. When I add this new connector locally on my, rather old, 0.39.42-alpha, I have no problem. However on my production airbyte, which run 0.40.23 I receive

Internal Server Error: Get Spec job failed.

I went through the advice given on these sides https://discuss.airbyte.io/t/internal-server-error-get-spec-job-failed/2508 https://discuss.airbyte.io/t/unable-to-add-connectors-get-spec-job-failed/2184 https://github.com/airbytehq/airbyte/issues/12030 but nothing fixed it yet. To be clear.

python main.py spec

and

python main.py --config --secrets/config.json

work as indented. So does

docker run --rm -i sbrickel/instagram:dev spec

. Does anyone has any advice for me. Or an idea why I do not get the same error when adding the new connector locally? I did

docker pull sbrickel/instagram:dev

in my VM as well, as suggested by someone in the github issue Thank you

Is it possible for you to upgrade your local version to see if it happens on a new/local version? One possibility that is probably worth ruling out is that the code was enhanced with better spec checking, which may only appear in a newer version ..

hmm, try running a docker pull on the machine that is running airbyte before trying to add the connector

@Marcelo Pio de Castro I already did a docker pull, unfortunately it did not help. @Alex Marquardt (Airbyte) I guess I will update my local version and see if the issue occurs locally as well. Thank you both

To be clear, I don’t think this will solve your issue - If the newer local version throws the same errors , then you will be able to locally debug the issue is with your spec file (assuming that is where the issue lies)

No worries, I got that point 😁

Can you get the worker logs? it could give more details. I generally get the spec error if my image doesn't exist (as it was not pulled or created) or I made an error on the spec. One thing that we use a lot is run the acceptance test, it checks the spec and the other commands, so if it passes, you connector is good, and the problem is elsewhere

I hope you are still reading this @Marcelo Pio de Castro, I was on my Christmas break. I could download the worker logs and it does say that it found the image locally (see attached log file) The acceptance test did encounter two failures FAILED ../../test_input/test_core.py:TestBasicRead:test_read[inputs0] - Failed: Timeout >300.0s FAILED ../../test_input/test_full_refresh.py:TestFullRefresh:test_sequential_reads[inputs0] - Failed: Timeout >1200.0s I will try to debug, but appreciate any advice 😄

Oh, it just timeout, so it is not an error. You should change the test stream to a smaller one or change the default timeout to run it beyond default

that is strange, why do I still get the

Internal Server Error: Get Spec job failed.

error message than when I

docker pull

on my VM?

Nice! Never would have thought about that.

At least I am not alone 😅

@Sebastian Brickel You are a life saver...just ran into this 😂

you are welcome @Dustin Pearson