Title
s

Steve Palm

09/30/2022, 6:05 PM
Maybe this needs to go to the forums, but seeing the activity here with channel renames/etc prompted me to update our install (docker images) to the latest version. That was a mistake. They start, but the server doesn't respond and I see messages in the airbyte-server about timeout waiting for namespace default to be initialized in temporal. The logs for airbyte-temporal show the namespace registration attempt and that it was already registered. Seems a docker connection issue or something? Strange.
m

Marcos Marx (Airbyte)

09/30/2022, 6:07 PM
Are you using latest version on master @Steve Palm also can you upload the server logs here?
s

Steve Palm

09/30/2022, 6:07 PM
I just pulled the latest from github. I'll admit, I haven't looked at this since I first installed it so I am not remembering all the pieces. 😞
I'll try to collect some meaningful logs and come back.
l

Léon Stefani

09/30/2022, 6:09 PM
Had this kind of issue when updating correctly the docker-compose but not the .env file from github’s master if this can help @Steve Palm 🙂
m

Marcos Marx (Airbyte)

09/30/2022, 6:11 PM
Latest version introduced some new
.env
variables so it is necessary to update that
s

Steve Palm

09/30/2022, 6:11 PM
@Léon Stefani That doesn't update with git pull? Sorry to be dense, how to update it?
m

Marcos Marx (Airbyte)

09/30/2022, 6:12 PM
s

Steve Palm

09/30/2022, 6:13 PM
Ok, I had notes from what I did before, I will look at those official docs. Sorry for not checking that first, I didn't think they would have changed. Silly me. 🙂 Thanks!!
Ok, I pulled it from the repo, so those docs say that 'git pull' is all that is needed, but it seems that is not the case.
m

Marcos Marx (Airbyte)

09/30/2022, 6:18 PM
Ok, I pulled it from the repo, so those docs say that ‘git pull’ is all that is needed, but it seems that is not the case.
should be; if it is not working please try again to spin up the docker container
s

Steve Palm

09/30/2022, 6:21 PM
I downloaded the .env from github and compared it with mine, no difference. I have brought it down and up several times. I'll keep working on it. Seems the server cannot connect to temporal, the error is a timeout message.
2022-09-30 18:15:06 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):221 - Waiting for temporal server...
2022-09-30 18:15:06 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):232 - Waiting for namespace default to be initialized in temporal...
2022-09-30 18:15:13 WARN i.t.i.r.GrpcSyncRetryer(retry):56 - Retrying after failure
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.987951421s. 
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.48.0.jar:1.48.0]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.48.0.jar:1.48.0]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.48.0.jar:1.48.0]
	at io.grpc.health.v1.HealthGrpc$HealthBlockingStub.check(HealthGrpc.java:252) ~[grpc-services-1.48.0.jar:1.48.0]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.lambda$checkHealth$2(WorkflowServiceStubsImpl.java:286) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.checkHealth(WorkflowServiceStubsImpl.java:279) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubsImpl.<init>(WorkflowServiceStubsImpl.java:186) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:51) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.temporal.serviceclient.WorkflowServiceStubs.newInstance(WorkflowServiceStubs.java:41) ~[temporal-serviceclient-1.8.1.jar:?]
	at io.airbyte.commons.temporal.TemporalUtils.lambda$createTemporalService$0(TemporalUtils.java:93) ~[io.airbyte-airbyte-commons-temporal-0.40.10.jar:?]
	at io.airbyte.commons.temporal.TemporalUtils.getTemporalClientWhenConnected(TemporalUtils.java:237) ~[io.airbyte-airbyte-commons-temporal-0.40.10.jar:?]
	at io.airbyte.commons.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:89) ~[io.airbyte-airbyte-commons-temporal-0.40.10.jar:?]
	at io.airbyte.commons.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:104) ~[io.airbyte-airbyte-commons-temporal-0.40.10.jar:?]
	at io.airbyte.commons.temporal.TemporalUtils.createTemporalService(TemporalUtils.java:108) ~[io.airbyte-airbyte-commons-temporal-0.40.10.jar:?]
	at io.airbyte.server.ServerApp.getServer(ServerApp.java:240) ~[io.airbyte-airbyte-server-0.40.10.jar:?]
	at io.airbyte.server.ServerApp.main(ServerApp.java:333) ~[io.airbyte-airbyte-server-0.40.10.jar:?]
m

Marcos Marx (Airbyte)

09/30/2022, 6:22 PM
Did you try to access localhost:8000 and see if it’s working? Temporal sometimes took a few secons/minutes to start.
s

Steve Palm

09/30/2022, 6:23 PM
That error above repeated for over 15min
I hit localhost:8000 with lynx and get a 'You need to enable Javascript to run this app.' This is on a remote host, I didn't have that port forwarded over the ssh tunnel.
Hitting it with a web browser shows:
Oops! Something went wrong… Cannot reach server. The server may still be starting up.
m

Marcos Marx (Airbyte)

09/30/2022, 6:25 PM
what
docker logs airbyte-temporal
gives to you?
s

Steve Palm

09/30/2022, 6:33 PM
This is on a cpanel host, and I had trouble getting it set up initially, I had to add a custom network segment to the .yaml file: networks: default: driver: bridge ipam: driver: default config: - subnet: 172.18.0.1/16
Then it worked without problems until this update.
I see these in airbyte-webapp:
2022/09/30 18:31:00 [error] 39#39: *15 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: localhost, request: "POST /api/v1/workspaces/list HTTP/1.1", upstream: "<http://172.18.0.5:8001/api/v1/workspaces/list>", host: "localhost:8800", referrer: "<http://localhost:8800/>"
172.18.0.1 - - [30/Sep/2022:18:31:00 +0000] "POST /api/v1/workspaces/list HTTP/1.1" 502 497 "<http://localhost:8800/>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.53" "-"
airbyte-cron is also reporting errors about connecting to Temporal
and airbyte-worker
Seems it is a docker connection issue that is new for some reason.
m

Marcos Marx (Airbyte)

09/30/2022, 6:36 PM
you changed the default port to 8800?
s

Steve Palm

09/30/2022, 6:37 PM
webapp has: ports: - 8000:80
8800 is the local end of my ssh tunnel to 8000 on the other end
From the remote host cli I can use nc to connect to port 7233 (temporal). I get an @ back.
I got a cli in airbyte-server, but seems no commands are there that could be used to check connectivity to temporal.
doing a
yum search netcat
gives me timeout errors, so it may be networking related? Something in docker changed maybe?
(inside airbyte-server that is)
m

Marcos Marx (Airbyte)

09/30/2022, 6:49 PM
ohhh, you’re in redhat or not ubuntu system?
s

Steve Palm

09/30/2022, 6:49 PM
This is inside the docker container
It is redhat
or something
m

Marcos Marx (Airbyte)

09/30/2022, 6:52 PM
oh, https://github.com/airbytehq/airbyte/blob/master/docs/archive/faq/deploying-on-other-os.md I’m not sure if Airbyte runs out-of-box in Redhat/CentOS probably you must configure some other steps to make it work. I strongly recommend to you use Ubuntu
s

Steve Palm

09/30/2022, 6:54 PM
You miss that this was working perfectly fine until the update pull. So maybe something else changed. Were any ports changed/added/etc? That was the whole point of docker was to isolate from the host environment I thought. I can revisit firewall rules, but this was working before.
iptables -L for chain DOCKER shows d3306, 7233, 80, 8001 and 9000 as ACCEPT.
m

Marcos Marx (Airbyte)

09/30/2022, 6:56 PM
The most dramatic changed was the introduction of Micronaut framework in latest version; but I don’t see this as the problem. Could you pull the correct release version?
git pull v0.40.10
to ensure you’re using the correct docker-compose and .env file?
s

Steve Palm

09/30/2022, 6:59 PM
Is there something missing from that command? I get
fatal: 'v0.40.10' does not appear to be a git repository
I came back to this after doing a lot of other chores/etc and saw this in the airbyte-server log:
2022-09-30 19:04:08 WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):232 - Waiting for namespace default to be initialized in temporal...
2022-09-30 19:04:15 WARN i.t.i.r.GrpcSyncRetryer(retry):56 - Retrying after failure
 .
 .
 .
2022-09-30 21:44:19 INFO i.t.s.WorkflowServiceStubsImpl(<init>):188 - Created GRPC client for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=39, target=airbyte-temporal:7233}}
2022-09-30 21:44:24 INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):249 - Temporal namespace default initialized!
2022-09-30 21:44:24 INFO i.a.s.ServerApp(migrateExistingConnectionsToTemporalScheduler):295 - Migration to temporal scheduler has already been performed
2022-09-30 21:44:24 INFO i.a.s.ServerApp(getServer):266 - Starting server...
2022-09-30 21:44:24 INFO i.a.c.EnvConfigs(getEnvOrDefault):1072 - Using default value for environment variable WORKER_ENVIRONMENT: 'DOCKER'
2022-09-30 21:44:24 INFO o.e.j.u.l.Log(initialized):169 - Logging initialized @146392ms to org.eclipse.jetty.util.log.Slf4jLog
2022-09-30 21:44:24 INFO o.e.j.s.Server(doStart):360 - jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 17.0.4.1+9-LTS
Sep 30, 2022 9:44:24 PM org.glassfish.jersey.server.wadl.WadlFeature configure
WARNING: JAXBContext implementation could not be found. WADL feature is disabled.
2022-09-30 21:44:25 INFO o.h.v.i.u.Version(<clinit>):21 - HV000001: Hibernate Validator 6.1.2.Final
2022-09-30 21:44:25 INFO o.e.j.s.h.ContextHandler(doStart):860 - Started o.e.j.s.ServletContextHandler@7744195{/,null,AVAILABLE}
2022-09-30 21:44:25 INFO o.e.j.s.AbstractConnector(doStart):331 - Started ServerConnector@f559c74{HTTP/1.1, (http/1.1)}{0.0.0.0:8001}
2022-09-30 21:44:25 INFO o.e.j.s.Server(doStart):400 - Started @147758ms
2022-09-30 21:44:25 INFO i.a.s.ServerApp(start):139 - 
    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
--------------------------------------
 Now ready at <http://localhost:8000/>
--------------------------------------
Version: 0.40.10
SO eventually it came up (around 2hr 45min later), but the web interface still gives an error it cannot contact the server.
While the server came up, and I eventually could log in to the web interface (just by waiting a long time). However, when it spins up the connectors they have the same problem so syncs fail.
m

Marcos Marx (Airbyte)

10/03/2022, 2:54 PM
@Steve Palm sorry the long delay. I’ll return soon 🙂
s

Steve Palm

10/05/2022, 8:22 PM
@Marcos Marx (Airbyte) Looping back... After a lot of googling and testing, it seemed something was just strange inside Docker. 'systemctl restart docker' and container communication seems much better now and things are working. Sorry for the rabbit trails, but being a production server I couldn't just reboot to keep testing. 😉 A sync is running now.
m

Marcos Marx (Airbyte)

10/05/2022, 8:59 PM
Ohhhh this is amazing! Sorry to not return to you sooner, I’m OOO 😞
s

Steve Palm

10/05/2022, 9:01 PM
No worries, thanks for your contact. I hope I remember this next time. 🙂