I’m trying Airbyte for the first time and am seein...
# replication-troubleshooting
a
I’m trying Airbyte for the first time and am seeing a lot of errors when trying to set up sources, destinations and connections. I’ve been experimenting with MSSQL and Jira connections and it looks very unstable. Fields marked as optional aren’t as such. Source schemas appear to timeout when validating on large MSSQL databases. The Jira connection cannot sync due to “Invalid timezone offset” errors. Does anyone else see these sort of issues? Is this sort of instability typical or limited to the ALPHA connectors? Thanks.
m
I have a database of around 3000 tables on MSSQL and it is generally fine. Old versions would timeout, but after 0.40 it seems ok. Have not tried Jira. As for other connectors, it really depends. I have made my own connectors and they perform fine. Parallel streams would be a huge upgrade though
n
Hi @Marcelo Pio de Castro in our schema we have 300 tables in MSSQL server and we are trying to brings these tables to snowflake. and during connection setup we are getting "Failed to fetch schema, Please try again" error message in UI. one temporary solution I got was to create different users and limit tables that access in MSSQL. Could you please let us know how you handled this for 3000 tables
b
Yeah, anything labeled ALPHA barely works
n
Hey @Aaron Pritchard, sorry to hear you're running into issues. This Slack channel is here to support and foster the Airbyte community - if you give specific details on the problems you are running into, the community assistance team would be happy to help. As a rule the Generally Available connectors are the ones the team is focusing on most of the time as they are the connectors most-used and most asked for by the Airbyte community. The alpha connectors are usually contributed by the open source community. We have a rigorous development process that includes writing integration tests; as Airbyte is a product that is developing at an incredibly fast paced with versions being release anywhere from daily to weekly, it is difficult to make sure everything works smoothly on all possible platforms. If you have more questions, please let me know. As always, we are striving to do our best and keep the discussions here and on the forum constructive, useful, and thoughtful. You can find all the guidelines for our code of conduct here: https://docs.airbyte.com/project-overview/slack-code-of-conduct/
👍 1
m
Can you give details about your Airbyte deployment?
a
Hey @Marcelo Pio de Castro, like @Naren Kadiri I cannot fetch the schema from a large MSSQL database, citing a “non-json response” error. I would note in the console that I can see details about the schema being shown like field names. The deployment is a simple docker installation at the moment as a proof-of-concept.
m
I changed my configuration so that the jobs would have more memory to work. I am using an instance with 16gb of RAM
I changed JOB_MAIN_CONTAINER_MEMORY_LIMIT to 4G
n
Hi @Marcelo Pio de Castro in UI we are getting "Failed to Fetch Schema, Please Try Again". where as on docker container logs this is what error I'm getting "" airbyte-proxy | 2022/11/16 045512 [error] 13#13: *1 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.22.0.1, server: , request: "POST /api/v1/sources/discover_schema HTTP/1.1", upstream: "http://172.22.0.5:80/api/v1/sources/discover_schema", host: "localhost:8000", referrer: "http://localhost:8000/workspaces/dab48532-c867-45ab-a16c-5bb4596f9e11/source/new-connection" "" I tried JOB_MAIN_CONTAINER_MEMORY_LIMIT=4g but still running into same issue
as a temporary solution I created new user for airbyte and gave access to limited tables
m
Hmmm, what I would try is to get the docker command for the discover and run that on the command line to see how much time that would take
You can get the docker command from the logs probably from airbyte worker:
docker compose logs worker
Apparently Aaron problem is being cause by the recent addition of the airbyte proxy that implemented basic authentication
@Aaron Pritchard , you may try Airbyte below 0.40.15 so that you don't have that problem
👍 1