Maybe this needs to go to the forums but seeing the activity

Question

Maybe this needs to go to the forums but seeing the activity here with channel renames etc prompted me to update our install docker images to the latest version That was a mistake They start but the server doesn t respond and I see messages in the airbyte server about timeout waiting for namespace default to be initialized in temporal The logs for airbyte temporal show the namespace registration attempt and that it was already registered Seems a docker connection issue or something Strange

Marcos Marx (Airbyte) · Answer

Are you using latest version on master < Steve Palm> also can you upload the server logs here

Steve Palm · Answer

I just pulled the latest from github I ll admit I haven t looked at this since I first installed it so I am not remembering all the pieces disappointed

Steve Palm · Answer

I ll try to collect some meaningful logs and come back

Léon Stefani · Answer

Had this kind of issue when updating correctly the docker compose but not the env file from github s master if this can help < Steve Palm> slightly smiling face

Marcos Marx (Airbyte) · Answer

Latest version introduced some new ` env` variables so it is necessary to update that

Steve Palm · Answer

< Léon Stefani> That doesn t update with git pull Sorry to be dense how to update it

Marcos Marx (Airbyte) · Answer

Following this instructions would work fine <https docs airbyte com operator guides upgrading airbyte upgrading on docker>

Steve Palm · Answer

Ok I had notes from what I did before I will look at those official docs Sorry for not checking that first I didn t think they would have changed Silly me slightly smiling face Thanks

Steve Palm · Answer

Ok I pulled it from the repo so those docs say that git pull is all that is needed but it seems that is not the case

Marcos Marx (Airbyte) · Answer

gt Ok I pulled it from the repo so those docs say that git pull is all that is needed but it seems that is not the case should be if it is not working please try again to spin up the docker container

Steve Palm · Answer

I downloaded the env from github and compared it with mine no difference I have brought it down and up several times I ll keep working on it Seems the server cannot connect to temporal the error is a timeout message ```2022 09 30 18 15 06 INFO i a c t TemporalUtils getTemporalClientWhenConnected 221 Waiting for temporal server 2022 09 30 18 15 06 WARN i a c t TemporalUtils getTemporalClientWhenConnected 232 Waiting for namespace default to be initialized in temporal 2022 09 30 18 15 13 WARN i t i r GrpcSyncRetryer retry 56 Retrying after failure io grpc StatusRuntimeException DEADLINE EXCEEDED Deadline exceeded after 4 987951421s at io grpc stub ClientCalls toStatusRuntimeException ClientCalls java 271 ~ grpc stub 1 48 0 jar 1 48 0 at io grpc stub ClientCalls getUnchecked ClientCalls java 252 ~ grpc stub 1 48 0 jar 1 48 0 at io grpc stub ClientCalls blockingUnaryCall ClientCalls java 165 ~ grpc stub 1 48 0 jar 1 48 0 at io grpc health v1 HealthGrpc$HealthBlockingStub check HealthGrpc java 252 ~ grpc services 1 48 0 jar 1 48 0 at io temporal serviceclient WorkflowServiceStubsImpl lambda$checkHealth$2 WorkflowServiceStubsImpl java 286 ~ temporal serviceclient 1 8 1 jar at io temporal internal retryer GrpcSyncRetryer retry GrpcSyncRetryer java 61 ~ temporal serviceclient 1 8 1 jar at io temporal internal retryer GrpcRetryer retryWithResult GrpcRetryer java 51 ~ temporal serviceclient 1 8 1 jar at io temporal serviceclient WorkflowServiceStubsImpl checkHealth WorkflowServiceStubsImpl java 279 ~ temporal serviceclient 1 8 1 jar at io temporal serviceclient WorkflowServiceStubsImpl lt init gt WorkflowServiceStubsImpl java 186 ~ temporal serviceclient 1 8 1 jar at io temporal serviceclient WorkflowServiceStubs newInstance WorkflowServiceStubs java 51 ~ temporal serviceclient 1 8 1 jar at io temporal serviceclient WorkflowServiceStubs newInstance WorkflowServiceStubs java 41 ~ temporal serviceclient 1 8 1 jar at io airbyte commons temporal TemporalUtils lambda$createTemporalService$0 TemporalUtils java 93 ~ io airbyte airbyte commons temporal 0 40 10 jar at io airbyte commons temporal TemporalUtils getTemporalClientWhenConnected TemporalUtils java 237 ~ io airbyte airbyte commons temporal 0 40 10 jar at io airbyte commons temporal TemporalUtils createTemporalService TemporalUtils java 89 ~ io airbyte airbyte commons temporal 0 40 10 jar at io airbyte commons temporal TemporalUtils createTemporalService TemporalUtils java 104 ~ io airbyte airbyte commons temporal 0 40 10 jar at io airbyte commons temporal TemporalUtils createTemporalService TemporalUtils java 108 ~ io airbyte airbyte commons temporal 0 40 10 jar at io airbyte server ServerApp getServer ServerApp java 240 ~ io airbyte airbyte server 0 40 10 jar at io airbyte server ServerApp main ServerApp java 333 ~ io airbyte airbyte server 0 40 10 jar ```

Marcos Marx (Airbyte) · Answer

Did you try to access localhost 8000 and see if it s working Temporal sometimes took a few secons minutes to start

Steve Palm · Answer

That error above repeated for over 15min

Steve Palm · Answer

I hit localhost 8000 with lynx and get a You need to enable Javascript to run this app This is on a remote host I didn t have that port forwarded over the ssh tunnel

Steve Palm · Answer

Hitting it with a web browser shows

Steve Palm · Answer

Oops Something went wrong Cannot reach server The server may still be starting up

Marcos Marx (Airbyte) · Answer

what `docker logs airbyte temporal` gives to you

Steve Palm · Answer

This is on a cpanel host and I had trouble getting it set up initially I had to add a custom network segment to the yaml file networks default driver bridge ipam driver default config subnet 172 18 0 1 16

Steve Palm · Answer

Then it worked without problems until this update

Steve Palm · Answer

I see these in airbyte webapp

Steve Palm · Answer

```2022 09 30 18 31 00 error 39 39 15 connect failed 111 Connection refused while connecting to upstream client 172 18 0 1 server localhost request POST api v1 workspaces list HTTP 1 1 upstream host localhost 8800 referrer 172 18 0 1 30 Sep 2022 18 31 00 +0000 POST api v1 workspaces list HTTP 1 1 502 497 Mozilla 5 0 Macintosh Intel Mac OS X 10 15 7 AppleWebKit 537 36 KHTML like Gecko Chrome 105 0 0 0 Safari 537 36 Edg 105 0 1343 53 ```

Steve Palm · Answer

airbyte cron is also reporting errors about connecting to Temporal

Steve Palm · Answer

and airbyte worker

Steve Palm · Answer

Seems it is a docker connection issue that is new for some reason

Marcos Marx (Airbyte) · Answer

you changed the default port to 8800

Steve Palm · Answer

webapp has ports 8000 80

Steve Palm · Answer

8800 is the local end of my ssh tunnel to 8000 on the other end

Steve Palm · Answer

From the remote host cli I can use nc to connect to port 7233 temporal I get an back

Steve Palm · Answer

I got a cli in airbyte server but seems no commands are there that could be used to check connectivity to temporal

Steve Palm · Answer

doing a `yum search netcat` gives me timeout errors so it may be networking related Something in docker changed maybe

Steve Palm · Answer

inside airbyte server that is

Marcos Marx (Airbyte) · Answer

ohhh you re in redhat or not ubuntu system

Steve Palm · Answer

This is inside the docker container

Steve Palm · Answer

It is redhat

Steve Palm · Answer

or something

Marcos Marx (Airbyte) · Answer

oh <https github com airbytehq airbyte blob master docs archive faq deploying on other os md> I m not sure if Airbyte runs out of box in Redhat CentOS probably you must configure some other steps to make it work I strongly recommend to you use Ubuntu

Steve Palm · Answer

You miss that this was working perfectly fine until the update pull So maybe something else changed Were any ports changed added etc That was the whole point of docker was to isolate from the host environment I thought I can revisit firewall rules but this was working before

Steve Palm · Answer

iptables L for chain DOCKER shows d3306 7233 80 8001 and 9000 as ACCEPT

Marcos Marx (Airbyte) · Answer

The most dramatic changed was the introduction of Micronaut framework in latest version but I don t see this as the problem Could you pull the correct release version `git pull v0 40 10` to ensure you re using the correct docker compose and env file

Steve Palm · Answer

Is there something missing from that command I get ```fatal v0 40 10 does not appear to be a git repository```

Steve Palm · Answer

I came back to this after doing a lot of other chores etc and saw this in the airbyte server log ```2022 09 30 19 04 08 WARN i a c t TemporalUtils getTemporalClientWhenConnected 232 Waiting for namespace default to be initialized in temporal 2022 09 30 19 04 15 WARN i t i r GrpcSyncRetryer retry 56 Retrying after failure 2022 09 30 21 44 19 INFO i t s WorkflowServiceStubsImpl lt init gt 188 Created GRPC client for channel ManagedChannelOrphanWrapper delegate=ManagedChannelImpl logId=39 target=airbyte temporal 7233 2022 09 30 21 44 24 INFO i a c t TemporalUtils getTemporalClientWhenConnected 249 Temporal namespace default initialized 2022 09 30 21 44 24 INFO i a s ServerApp migrateExistingConnectionsToTemporalScheduler 295 Migration to temporal scheduler has already been performed 2022 09 30 21 44 24 INFO i a s ServerApp getServer 266 Starting server 2022 09 30 21 44 24 INFO i a c EnvConfigs getEnvOrDefault 1072 Using default value for environment variable WORKER ENVIRONMENT DOCKER 2022 09 30 21 44 24 INFO o e j u l Log initialized 169 Logging initialized 146392ms to org eclipse jetty util log Slf4jLog 2022 09 30 21 44 24 INFO o e j s Server doStart 360 jetty 9 4 31 v20200723 built 2020 07 23T17 57 36 812Z git 450ba27947e13e66baa8cd1ce7e85a4461cacc1d jvm 17 0 4 1+9 LTS Sep 30 2022 9 44 24 PM org glassfish jersey server wadl WadlFeature configure WARNING JAXBContext implementation could not be found WADL feature is disabled 2022 09 30 21 44 25 INFO o h v i u Version lt clinit gt 21 HV000001 Hibernate Validator 6 1 2 Final 2022 09 30 21 44 25 INFO o e j s h ContextHandler doStart 860 Started o e j s ServletContextHandler 7744195 null AVAILABLE 2022 09 30 21 44 25 INFO o e j s AbstractConnector doStart 331 Started ServerConnector f559c74 HTTP 1 1 http 1 1 0 0 0 0 8001 2022 09 30 21 44 25 INFO o e j s Server doStart 400 Started 147758ms 2022 09 30 21 44 25 INFO i a s ServerApp start 139 | | | | | Now ready at <http localhost 8000 > Version 0 40 10```

Steve Palm · Answer

SO eventually it came up around 2hr 45min later but the web interface still gives an error it cannot contact the server

Steve Palm · Answer

While the server came up and I eventually could log in to the web interface just by waiting a long time However when it spins up the connectors they have the same problem so syncs fail

Marcos Marx (Airbyte) · Answer

< Steve Palm> sorry the long delay I ll return soon slightly smiling face

Steve Palm · Answer

< Marcos Marx Airbyte > Looping back After a lot of googling and testing it seemed something was just strange inside Docker systemctl restart docker and container communication seems much better now and things are working Sorry for the rabbit trails but being a production server I couldn t just reboot to keep testing wink A sync is running now

Marcos Marx (Airbyte) · Answer

Ohhhh this is amazing Sorry to not return to you sooner I m OOO disappointed

Steve Palm · Answer

No worries thanks for your contact I hope I remember this next time slightly smiling face