is anyone seeing slow image pull times when runnin...
# contributing-to-airbyted
Davin Chia (Airbyte)
04/09/2021, 9:09 AM
is anyone seeing slow image pull times when running the airbyte server on master? mine has gone from seconds -> minutes
u
user
04/09/2021, 10:59 AMseems like my temporal cluster is just spinning in place
u
user
04/09/2021, 11:00 AM Copy code
airbyte-webapp | 172.25.0.1 - - [09/Apr/2021:10:58:32 +0000] "GET /connections HTTP/1.1" 200 2858 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36" "-"
airbyte-webapp | 172.25.0.1 - - [09/Apr/2021:10:58:32 +0000] "GET /favicon.ico HTTP/1.1" 200 1150 "<http://localhost:8000/connections>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36" "-"
airbyte-server | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/workspaces/get
airbyte-server | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 POST 200 /api/v1/workspaces/get - {"workspaceId":"5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6"}
airbyte-server | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/web_backend/connections/list
airbyte-server | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/source_definitions/list
airbyte-server | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 POST 200 /api/v1/source_definitions/list - {"workspaceId":"5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6"}
airbyte-server | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/source_definitions/list_latest
airbyte-server | 2021-04-09 10:58:33 DEBUG i.a.s.c.SpecCachingSynchronousSchedulerClient(createGetSpecJob):81 - {workspace_app_root=/tmp/workspace/server/logs} - cache miss: airbyte/source-postgres:dev
airbyte-server | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 POST 200 /api/v1/source_definitions/list_latest - {"workspaceId":"5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6"}
airbyte-server | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/destination_definitions/list
airbyte-server | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 POST 200 /api/v1/destination_definitions/list - {"workspaceId":"5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6"}
airbyte-server | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/destination_definitions/list_latest
airbyte-server | 2021-04-09 10:58:33 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 POST 200 /api/v1/destination_definitions/list_latest - {"workspaceId":"5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6"}
airbyte-webapp | 172.25.0.1 - - [09/Apr/2021:10:58:33 +0000] "GET /fonts/inter/Inter-Regular.woff2 HTTP/1.1" 304 0 "<http://localhost:8000/index.css>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36" "-"
airbyte-webapp | 172.25.0.1 - - [09/Apr/2021:10:58:33 +0000] "GET /fonts/inter/Inter-Medium.woff2 HTTP/1.1" 304 0 "<http://localhost:8000/index.css>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36" "-"
airbyte-server | 2021-04-09 10:58:37 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/sources/list
airbyte-server | 2021-04-09 10:58:37 DEBUG i.a.s.c.SpecCachingSynchronousSchedulerClient(createGetSpecJob):81 - {workspace_app_root=/tmp/workspace/server/logs} - cache miss: airbyte/source-postgres:dev
airbyte-temporal | {"level":"info","ts":"2021-04-09T10:59:18.936Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Starting","wf-task-queue-name":"/_sys/temporal-sys-batcher-taskqueue/1","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:188"}
airbyte-temporal | {"level":"info","ts":"2021-04-09T10:59:18.945Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Started","wf-task-queue-name":"/_sys/temporal-sys-batcher-taskqueue/1","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:202"}
airbyte-webapp | 172.25.0.1 - - [09/Apr/2021:11:00:17 +0000] "GET /fonts/inter/Inter-Bold.woff2 HTTP/1.1" 304 0 "<http://localhost:8000/index.css>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36" "-"
airbyte-server | 2021-04-09 11:00:17 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/workspaces/get
airbyte-server | 2021-04-09 11:00:17 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 POST 200 /api/v1/workspaces/get - {"workspaceId":"5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6"}
airbyte-temporal | {"level":"info","ts":"2021-04-09T11:00:20.862Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Starting","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/1","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:188"}
airbyte-temporal | {"level":"info","ts":"2021-04-09T11:00:20.871Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Started","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/1","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:202"}
airbyte-server | 2021-04-09 11:00:21 INFO i.a.s.RequestLogger(filter):107 - {workspace_app_root=/tmp/workspace/server/logs} - REQ 172.25.0.1 OPTIONS 200 /api/v1/destinations/list
airbyte-server | 2021-04-09 11:00:21 DEBUG i.a.s.c.SpecCachingSynchronousSchedulerClient(createGetSpecJob):81 - {workspace_app_root=/tmp/workspace/server/logs} - cache miss: airbyte/destination-postgres:devu
user
04/09/2021, 11:00 AMdev image is on my machine so this should be quick
u
user
04/09/2021, 11:01 AMcould this be blocking on some other job?
u
user
04/09/2021, 11:11 AMah figured it out. somehow my
.envu
user
04/09/2021, 2:13 PMhmm this is still happening
u
user
04/09/2021, 2:14 PM Copy code
airbyte-scheduler | 2021-04-09 14:14:33 INFO i.a.c.EnvConfigs(getEnvOrDefault):176 - {workspace_app_root=/tmp/workspace/scheduler/logs} - WORKER_ENVIRONMENT not found, defaulting to DOCKER
airbyte-scheduler | Exception in thread "main" java.lang.IllegalArgumentException: Env variable not defined: LOCAL_ROOT
airbyte-scheduler | at io.airbyte.config.EnvConfigs.getPath(EnvConfigs.java:195)
airbyte-scheduler | at io.airbyte.config.EnvConfigs.getLocalRoot(EnvConfigs.java:99)
airbyte-scheduler | at io.airbyte.config.EnvConfigs.getLocalDockerMount(EnvConfigs.java:124)
airbyte-scheduler | at io.airbyte.scheduler.app.SchedulerApp.getProcessBuilderFactory(SchedulerApp.java:165)
airbyte-scheduler | at io.airbyte.scheduler.app.SchedulerApp.main(SchedulerApp.java:191)u
user
04/09/2021, 2:14 PMis the error
u
user
04/09/2021, 2:18 PMactually @charles I suspect this might be related to https://github.com/airbytehq/airbyte/pull/2675
u
user
04/09/2021, 2:19 PMI dropped the commit from master and this started working again
u
user
04/09/2021, 2:44 PMIs this related to the CI being super slow?
https://github.com/airbytehq/airbyte/runs/2304346313?check_suite_focus=true
(Acceptance tests running for multiple hours)
u
user
04/09/2021, 2:56 PMmmm probably stuck waiting for the scheduler. maybe kill for now?
u
user
04/09/2021, 3:32 PMwhat commit?
u
user
04/09/2021, 3:42 PMbelieve it’s this one: https://github.com/airbytehq/airbyte/commit/b5bf9df2eb58e6dba2d0b4e10359e67262344a82
u
user
04/09/2021, 3:42 PMI tried to look into it but brain too fried
u
user
04/09/2021, 3:43 PMokay.
u
user
04/09/2021, 3:43 PMso there was a bug in that commit
u
user
04/09/2021, 3:43 PMbut the next commit should have fixed it.
u
user
04/09/2021, 3:44 PMdo you know if you had both of those commits or was this commit the HEAD?
u
user
04/09/2021, 3:46 PMi'll dig more if we think master is still busted.
u
user
04/09/2021, 3:46 PMI think it’s still happening on master
u
user
04/09/2021, 3:47 PMkk
u
user
04/09/2021, 3:47 PMi will look this morning.
u
user
04/09/2021, 3:50 PMdefo still in master
2 Views