https://linen.dev logo
d

Davin Chia (Airbyte)

04/09/2021, 9:09 AM
is anyone seeing slow image pull times when running the airbyte server on master? mine has gone from seconds -> minutes
u

user

04/09/2021, 10:59 AM
seems like my temporal cluster is just spinning in place
u

user

04/09/2021, 11:00 AM
Copy code
airbyte-webapp      | 172.25.0.1 - - [09/Apr/2021:10:58:32 +0000] "GET /connections HTTP/1.1" 200 2858 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36" "-"
airbyte-webapp      | 172.25.0.1 - - [09/Apr/2021:10:58:32 +0000] "GET /favicon.ico HTTP/1.1" 200 1150 "<http://localhost:8000/connections>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36" "-"
airbyte-server      | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/workspaces/get
airbyte-server      | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 POST 200 /api/v1/workspaces/get - {"workspaceId":"5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6"}
airbyte-server      | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/web_backend/connections/list
airbyte-server      | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/source_definitions/list
airbyte-server      | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 POST 200 /api/v1/source_definitions/list - {"workspaceId":"5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6"}
airbyte-server      | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/source_definitions/list_latest
airbyte-server      | 2021-04-09 10:58:33 DEBUG i.a.s.c.SpecCachingSynchronousSchedulerClient(createGetSpecJob):81 - {workspace_app_root=/tmp/workspace/server/logs} - cache miss: airbyte/source-postgres:dev
airbyte-server      | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 POST 200 /api/v1/source_definitions/list_latest - {"workspaceId":"5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6"}
airbyte-server      | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/destination_definitions/list
airbyte-server      | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 POST 200 /api/v1/destination_definitions/list - {"workspaceId":"5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6"}
airbyte-server      | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/destination_definitions/list_latest
airbyte-server      | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 POST 200 /api/v1/destination_definitions/list_latest - {"workspaceId":"5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6"}
airbyte-webapp      | 172.25.0.1 - - [09/Apr/2021:10:58:33 +0000] "GET /fonts/inter/Inter-Regular.woff2 HTTP/1.1" 304 0 "<http://localhost:8000/index.css>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36" "-"
airbyte-webapp      | 172.25.0.1 - - [09/Apr/2021:10:58:33 +0000] "GET /fonts/inter/Inter-Medium.woff2 HTTP/1.1" 304 0 "<http://localhost:8000/index.css>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36" "-"
airbyte-server      | 2021-04-09 10:58:37 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/sources/list
airbyte-server      | 2021-04-09 10:58:37 DEBUG i.a.s.c.SpecCachingSynchronousSchedulerClient(createGetSpecJob):81 - {workspace_app_root=/tmp/workspace/server/logs} - cache miss: airbyte/source-postgres:dev
airbyte-temporal    | {"level":"info","ts":"2021-04-09T10:59:18.936Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Starting","wf-task-queue-name":"/_sys/temporal-sys-batcher-taskqueue/1","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:188"}
airbyte-temporal    | {"level":"info","ts":"2021-04-09T10:59:18.945Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Started","wf-task-queue-name":"/_sys/temporal-sys-batcher-taskqueue/1","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:202"}
airbyte-webapp      | 172.25.0.1 - - [09/Apr/2021:11:00:17 +0000] "GET /fonts/inter/Inter-Bold.woff2 HTTP/1.1" 304 0 "<http://localhost:8000/index.css>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36" "-"
airbyte-server      | 2021-04-09 11:00:17 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/workspaces/get
airbyte-server      | 2021-04-09 11:00:17 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 POST 200 /api/v1/workspaces/get - {"workspaceId":"5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6"}
airbyte-temporal    | {"level":"info","ts":"2021-04-09T11:00:20.862Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Starting","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/1","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:188"}
airbyte-temporal    | {"level":"info","ts":"2021-04-09T11:00:20.871Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Started","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/1","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:202"}
airbyte-server      | 2021-04-09 11:00:21 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/destinations/list
airbyte-server      | 2021-04-09 11:00:21 DEBUG i.a.s.c.SpecCachingSynchronousSchedulerClient(createGetSpecJob):81 - {workspace_app_root=/tmp/workspace/server/logs} - cache miss: airbyte/destination-postgres:dev
u

user

04/09/2021, 11:00 AM
dev image is on my machine so this should be quick
u

user

04/09/2021, 11:01 AM
could this be blocking on some other job?
u

user

04/09/2021, 11:11 AM
ah figured it out. somehow my
.env
file was corrupted
u

user

04/09/2021, 2:13 PM
hmm this is still happening
u

user

04/09/2021, 2:14 PM
Copy code
airbyte-scheduler   | 2021-04-09 14:14:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):176 - {workspace_app_root=/tmp/workspace/scheduler/logs} - WORKER_ENVIRONMENT not found, defaulting to DOCKER
airbyte-scheduler   | Exception in thread "main" java.lang.IllegalArgumentException: Env variable not defined: LOCAL_ROOT
airbyte-scheduler   | 	at io.airbyte.config.EnvConfigs.getPath(EnvConfigs.java:195)
airbyte-scheduler   | 	at io.airbyte.config.EnvConfigs.getLocalRoot(EnvConfigs.java:99)
airbyte-scheduler   | 	at io.airbyte.config.EnvConfigs.getLocalDockerMount(EnvConfigs.java:124)
airbyte-scheduler   | 	at io.airbyte.scheduler.app.SchedulerApp.getProcessBuilderFactory(SchedulerApp.java:165)
airbyte-scheduler   | 	at io.airbyte.scheduler.app.SchedulerApp.main(SchedulerApp.java:191)
u

user

04/09/2021, 2:14 PM
is the error
u

user

04/09/2021, 2:18 PM
actually @charles I suspect this might be related to https://github.com/airbytehq/airbyte/pull/2675
u

user

04/09/2021, 2:19 PM
I dropped the commit from master and this started working again
u

user

04/09/2021, 2:44 PM
Is this related to the CI being super slow? https://github.com/airbytehq/airbyte/runs/2304346313?check_suite_focus=true (Acceptance tests running for multiple hours)
u

user

04/09/2021, 2:56 PM
mmm probably stuck waiting for the scheduler. maybe kill for now?
u

user

04/09/2021, 3:32 PM
what commit?
u

user

04/09/2021, 3:42 PM
I tried to look into it but brain too fried
u

user

04/09/2021, 3:43 PM
okay.
u

user

04/09/2021, 3:43 PM
so there was a bug in that commit
u

user

04/09/2021, 3:43 PM
but the next commit should have fixed it.
u

user

04/09/2021, 3:44 PM
do you know if you had both of those commits or was this commit the HEAD?
u

user

04/09/2021, 3:46 PM
i'll dig more if we think master is still busted.
u

user

04/09/2021, 3:46 PM
I think it’s still happening on master
u

user

04/09/2021, 3:47 PM
kk
u

user

04/09/2021, 3:47 PM
i will look this morning.
u

user

04/09/2021, 3:50 PM
defo still in master
2 Views