https://linen.dev logo
Join Slack
Powered by
Hi, Is File Source Connector bugged? I create new...
# ask-community-for-troubleshooting
o
Hi, Is File Source Connector bugged? I create new source and set up source, then start checking to access.
Copy code
airbyte-worker      | 2022-11-04 10:46:08 INFO i.a.c.i.LineGobbler(voidCall):114 -
airbyte-worker      | 2022-11-04 10:46:08 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK -----
airbyte-worker      | 2022-11-04 10:46:08 INFO i.a.c.i.LineGobbler(voidCall):114 -
airbyte-worker      | 2022-11-04 10:46:08 INFO i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/source-file:0.2.28 exists...
airbyte-worker      | 2022-11-04 10:46:08 INFO i.a.c.i.LineGobbler(voidCall):114 - airbyte/source-file:0.2.28 was found locally.
・・・
airbyte-worker      | 2022-11-04 10:46:11 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Checking access to　<gs://source> csv about 200MB :)
and about 30s later, It gives me a "Non-Json response" error in airbyte's gui. but, not happened error in log. checking access continues.
Copy code
・・・
airbyte-temporal    | {"level":"info","ts":"2022-11-04T10:47:51.313Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/DISCOVER_SCHEMA/1","wf-task-queue-type":"Workflow","lifecycle":"Starting","logging-call-at":"taskQueueManager.go:238"}
airbyte-temporal    | {"level":"info","ts":"2022-11-04T10:47:51.313Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/DISCOVER_SCHEMA/1","wf-task-queue-type":"Workflow","lifecycle":"Started","logging-call-at":"taskQueueManager.go:242"}
airbyte-temporal    | {"level":"info","ts":"2022-11-04T10:47:51.342Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/DISCOVER_SCHEMA/3","wf-task-queue-type":"Activity","lifecycle":"Starting","logging-call-at":"taskQueueManager.go:238"}
airbyte-temporal    | {"level":"info","ts":"2022-11-04T10:47:51.343Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/DISCOVER_SCHEMA/3","wf-task-queue-type":"Activity","lifecycle":"Started","logging-call-at":"taskQueueManager.go:242"}
airbyte-temporal    | {"level":"info","ts":"2022-11-04T10:47:51.390Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/SYNC/3","wf-task-queue-type":"Workflow","lifecycle":"Starting","logging-call-at":"taskQueueManager.go:238"}
airbyte-temporal    | {"level":"info","ts":"2022-11-04T10:47:51.390Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/SYNC/3","wf-task-queue-type":"Workflow","lifecycle":"Started","logging-call-at":"taskQueueManager.go:242"}
airbyte-temporal    | {"level":"info","ts":"2022-11-04T10:47:52.350Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/CONNECTION_UPDATER/1","wf-task-queue-type":"Workflow","lifecycle":"Starting","logging-call-at":"taskQueueManager.go:238"}
airbyte-temporal    | {"level":"info","ts":"2022-11-04T10:47:52.350Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/CONNECTION_UPDATER/1","wf-task-queue-type":"Workflow","lifecycle":"Started","logging-call-at":"taskQueueManager.go:242"}
airbyte-temporal    | {"level":"info","ts":"2022-11-04T10:47:52.424Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/GET_SPEC/3","wf-task-queue-type":"Workflow","lifecycle":"Starting","logging-call-at":"taskQueueManager.go:238"}
airbyte-temporal    | {"level":"info","ts":"2022-11-04T10:47:52.424Z","msg":"none","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/GET_SPEC/3","wf-task-queue-type":"Workflow","lifecycle":"Started","logging-call-at":"taskQueueManager.go:242"}
airbyte-cron        | 2022-11-04 10:48:01 INFO i.a.c.s.DefinitionsUpdater(updateDefinitions):54 - Connector definitions update disabled.
airbyte-server      | 2022-11-04 10:48:02 INFO i.a.s.RequestLogger(filter):112 - REQ 172.25.0.6 GET 200 /api/v1/health
・・・
finally, checking access was succeeded.
Copy code
airbyte-worker      | 2022-11-04 10:48:43 INFO i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Check succeeded
airbyte-worker      | 2022-11-04 10:48:43 INFO i.a.w.t.TemporalAttemptExecution(get):162 - Stopping cancellation check scheduling...
airbyte-worker      | 2022-11-04 10:48:43 INFO i.a.c.i.LineGobbler(voidCall):114 -
airbyte-worker      | 2022-11-04 10:48:43 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK -----
airbyte-worker      | 2022-11-04 10:48:43 INFO i.a.c.i.LineGobbler(voidCall):114 -
but, new source didn't register in Sources because Airbyte probably misunderstood by error ("Non-Json response" error.)
s
I am getting this same "non-json response" message when trying to make a salesforce -> bigquery (denormalised) connection, or make changes to my existing one. And also can find nothing in any logs when searching for that phrase. It just shows that message when i hit set up connection. I do think this may have started happening when I pulled the latest version from master a few days ago. (When i use the retest button on the source and destination individually, neither of them throw an error. I'm not sure what else I can do to check which connection test is throwing the non-json response)
s
@Stuart Horgan and @okasaka, if you downgrade Airbyte to version 0.40.17, do you still get the non json response error? Also can you share how you’ve deployed AIrbyte?
s
Hi @Saj Dider (Airbyte), thanks for responding. I have Airbyte open source deployed locally on my mac M1 and running on docker desktop / localhost. How would I go about downgrading to a previous version? I just followed the instructions in this guide and use git pull to update: https://docs.airbyte.com/deploying-airbyte/local-deployment
I have been trying to find some more info on the error. When trying to set up a connection with just a single table from salesforce I found that I was getting the following error in the logs of the airbyte-proxy container:
Copy code
2022/11/10 17:28:34 [error] 13#13: *70 client intended to send too large body: 1051201 bytes, client: 172.18.0.1, server: , request: "POST /api/v1/web_backend/connections/create HTTP/1.1", host: "localhost:8000", referrer: "<http://localhost:8000/workspaces/fdee71e8-eff6-4056-85f9-d328465a71fa/connections/new-connection>"
1051201 bytes is just over 1MB. I did some searching and people are saying this error can be overcome by setting a client_max_body_size parameter to 200M in the nginx config. My issue now is I have no idea what nginx is or how best to do this. I found some files called nginx-auth.conf.template and nginx-no-auth.conf.template in the airbyte-proxy folder and added it to them but it didn't seem to change anything. Is this where I should be putting this?
@Saj Dider (Airbyte) I did try downgrading to 0.40.17 and get the same error. Going to try and get more help on how to change this nginx thing and see if that fixes it
u
Hello okasaka, it's been a while without an update from us. Are you still having problems or did you find a solution?
o
@Marcos Marx (Airbyte) I attempted the same way, but it is not fixed... I think this is the cause of size of source file, I attempting to connect the csv file(200MB). so, I converted csv to parquet, the size of parquet became 30MB, and connection was successful.
u
Hey @okasaka, thanks for the update. I wasn't able to replicate the issue but thanks for providing additional context here and I'm glad to hear converting it to parquet made the connection successful. We will make an issue on our github repo once we can replicate it. Feel free to ask more questions in the channel!
12 Views
Open in Slack
PreviousNext
Powered by