okasaka
11/04/2022, 11:17 AMairbyte-worker | 2022-11-04 10:46:08 INFO i.a.c.i.LineGobbler(voidCall):114 -
airbyte-worker | 2022-11-04 10:46:08 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK -----
airbyte-worker | 2022-11-04 10:46:08 INFO i.a.c.i.LineGobbler(voidCall):114 -
airbyte-worker | 2022-11-04 10:46:08 INFO i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/source-file:0.2.28 exists...
airbyte-worker | 2022-11-04 10:46:08 INFO i.a.c.i.LineGobbler(voidCall):114 - airbyte/source-file:0.2.28 was found locally.
・・・
airbyte-worker | 2022-11-04 10:46:11 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Checking access to <gs://source> csv about 200MB :)
and about 30s later,
It gives me a "Non-Json response" error in airbyte's gui.
but, not happened error in log. checking access continues.
・・・
airbyte-temporal | {"level":"info","ts":"2022-11-04T10:47:51.313Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/DISCOVER_SCHEMA/1","wf-task-queue-type":"Workflow","lifecycle":"Starting","logging-call-at":"taskQueueManager.go:238"}
airbyte-temporal | {"level":"info","ts":"2022-11-04T10:47:51.313Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/DISCOVER_SCHEMA/1","wf-task-queue-type":"Workflow","lifecycle":"Started","logging-call-at":"taskQueueManager.go:242"}
airbyte-temporal | {"level":"info","ts":"2022-11-04T10:47:51.342Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/DISCOVER_SCHEMA/3","wf-task-queue-type":"Activity","lifecycle":"Starting","logging-call-at":"taskQueueManager.go:238"}
airbyte-temporal | {"level":"info","ts":"2022-11-04T10:47:51.343Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/DISCOVER_SCHEMA/3","wf-task-queue-type":"Activity","lifecycle":"Started","logging-call-at":"taskQueueManager.go:242"}
airbyte-temporal | {"level":"info","ts":"2022-11-04T10:47:51.390Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/SYNC/3","wf-task-queue-type":"Workflow","lifecycle":"Starting","logging-call-at":"taskQueueManager.go:238"}
airbyte-temporal | {"level":"info","ts":"2022-11-04T10:47:51.390Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/SYNC/3","wf-task-queue-type":"Workflow","lifecycle":"Started","logging-call-at":"taskQueueManager.go:242"}
airbyte-temporal | {"level":"info","ts":"2022-11-04T10:47:52.350Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/CONNECTION_UPDATER/1","wf-task-queue-type":"Workflow","lifecycle":"Starting","logging-call-at":"taskQueueManager.go:238"}
airbyte-temporal | {"level":"info","ts":"2022-11-04T10:47:52.350Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/CONNECTION_UPDATER/1","wf-task-queue-type":"Workflow","lifecycle":"Started","logging-call-at":"taskQueueManager.go:242"}
airbyte-temporal | {"level":"info","ts":"2022-11-04T10:47:52.424Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/GET_SPEC/3","wf-task-queue-type":"Workflow","lifecycle":"Starting","logging-call-at":"taskQueueManager.go:238"}
airbyte-temporal | {"level":"info","ts":"2022-11-04T10:47:52.424Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/GET_SPEC/3","wf-task-queue-type":"Workflow","lifecycle":"Started","logging-call-at":"taskQueueManager.go:242"}
airbyte-cron | 2022-11-04 10:48:01 INFO i.a.c.s.DefinitionsUpdater(updateDefinitions):54 - Connector definitions update disabled.
airbyte-server | 2022-11-04 10:48:02 INFO i.a.s.RequestLogger(filter):112 - REQ 172.25.0.6 GET 200 /api/v1/health
・・・
finally, checking access was succeeded.
airbyte-worker | 2022-11-04 10:48:43 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Check succeeded
airbyte-worker | 2022-11-04 10:48:43 INFO i.a.w.t.TemporalAttemptExecution(get):162 - Stopping cancellation check scheduling...
airbyte-worker | 2022-11-04 10:48:43 INFO i.a.c.i.LineGobbler(voidCall):114 -
airbyte-worker | 2022-11-04 10:48:43 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK -----
airbyte-worker | 2022-11-04 10:48:43 INFO i.a.c.i.LineGobbler(voidCall):114 -
but, new source didn't register in Sources because Airbyte probably misunderstood by error ("Non-Json response" error.)Stuart Horgan
11/07/2022, 10:48 AMSaj Dider (Airbyte)
11/09/2022, 7:28 PMStuart Horgan
11/10/2022, 5:07 PMStuart Horgan
11/10/2022, 5:38 PM2022/11/10 17:28:34 [error] 13#13: *70 client intended to send too large body: 1051201 bytes, client: 172.18.0.1, server: , request: "POST /api/v1/web_backend/connections/create HTTP/1.1", host: "localhost:8000", referrer: "<http://localhost:8000/workspaces/fdee71e8-eff6-4056-85f9-d328465a71fa/connections/new-connection>"
1051201 bytes is just over 1MB. I did some searching and people are saying this error can be overcome by setting a client_max_body_size parameter to 200M in the nginx config.
My issue now is I have no idea what nginx is or how best to do this. I found some files called nginx-auth.conf.template and nginx-no-auth.conf.template in the airbyte-proxy folder and added it to them but it didn't seem to change anything. Is this where I should be putting this?Stuart Horgan
11/17/2022, 4:24 PMuser
12/06/2022, 8:21 PMokasaka
12/12/2022, 1:23 AMuser
12/12/2022, 8:18 PM