https://linen.dev logo
Join Slack
Powered by
# ask-community-for-troubleshooting
  • d
    Hi All, We are working with Airbyte OSS with about 30 active connections. As a first use case, we are trying to create an MSSQL connection to Snowflake and we’re having a lot of issues with it. The defined source passed all test and seems to be OK. Important info: • The source contains ~2000 tables • MSSQL connector version == 0.4.28 • Airbyte Version == 0.40.17 The issue starts when trying to creating a connection where it takes a lot of time and fails with the attached screenshot message. When fetching the response from the 
    discover_schema
    failed request (which appears twice for some reason), we get the following TO message. This appear after more than 3 minutes.
    Copy code
    <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
    When trying to fetch some info from the server logs we sometimes get the following:
    Copy code
    errors: $.ssl_method: must be a constant value encrypted_trust_server_certificate, $.ssl_method: does not have a value in the enumeration [encrypted_trust_server_certificate]
2023-02-14 10:09:15 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.ssl_method: must be a constant value encrypted_verify_certificate, $.ssl_method: does not have a value in the enumeration [encrypted_verify_certificate], $.trustStoreName: is missing but it is required, $.trustStorePassword: is missing but it is required
2023-02-14 10:09:15 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.tunnel_host: is missing but it is required, $.tunnel_port: is missing but it is required, $.tunnel_user: is missing but it is required, $.ssh_key: is missing but it is required, $.tunnel_method: must be a constant value SSH_KEY_AUTH
2023-02-14 10:09:15 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.tunnel_host: is missing but it is required, $.tunnel_port: is missing but it is required, $.tunnel_user: is missing but it is required, $.tunnel_user_password: is missing but it is required, $.tunnel_method: must be a constant value SSH_PASSWORD_AUTH
2023-02-14 10:09:15 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.method: must be a constant value CDC, $.method: does not have a value in the enumeration [CDC]
2023-02-14 10:09:15 INFO i.a.s.RequestLogger(filter):112 - REQ 10.130.4.155 POST 200 /api/v1/sources/update - {"name":"Microsoft SQL Server (MSSQL)","sourceId":"001893fd-02c0-4b9f-b4c5-1acddde3b591","connectionConfiguration":"REDACTED"}
    The above contradicts the Time out issue and the fact that the source seems to pass all tests contradicts that as well. Our main suspicions are: • Fetching 2000 tables schema takes more time than usual. • The connector is not very efficient with schema fetching. We currently don’t have any write access to that MSSQL. We currently don’t have any possibility to extend the timeout. Any help would be appreciated, Thanks!
    u
    • 2
    • 3
  • p
    Hi, I am looking to deploy Airbyte on multiple Git type runners hosted on on-premise VMs, accessed via a VPN connection. I can't get my head around how this could work. The runners host docker containers and various custom images. We have one runner dedicated to Prod and the other runner runs pipelines for multiple FBs, DEV and QA environments. Each environment queries the same SQL Server source but populates their own Snowflake DB. The pipelines are environment aware and share the docker images. I am not Docker savvy which does not help. The "On your Local machine" option appears to start up the Airbyte in a local terminal session. There is no internet access and the "docker compose up" option holds the session. I don't see any option to view the UI or enter further CLI commands. I assume this is the correct deployment option for on-prem VMs ? I have a AWS instance running to trial Airbyte, used the UI to build custom connections (yet to see how the target DB can be dynamically configured), and managed to orchestrate the asyc call to Airbyte via pre-existing docker based solution (using a "curl" call), but on-prem is route we need to take and all my trial work appears to go out the window. I am guessing, but do I need to build a docker container/set of images with some entry points and host on the runner? This would appear to result in "docker from docker" calls, good/bad/ugly ? I hope I have provided enough detail.
    n
    • 2
    • 1
  • f
    Anyone tried connecting to MongoDB’s new Atlas Data Federation https://www.mongodb.com/atlas/data-federation My connection fails at the schema discovery phase even tho it is working with the mongodb atlas cluster.
    g
    • 2
    • 10
  • f
    Working on Open Source Airbyte and hosting it in GCP. Working on Quickbooks source and Bigquery as destination. I have refresh token that's is supposed to be live for 100days from quickbooks but seems to expire after 24hrs. Am guessing that airbyte is getting a new refresh token which makes the one I have invalid according to quickbooks API docs. So the consequence of that is airbyte fails to pull data after 24hrs with the expired refresh token. Question: does airbyte store these refresh tokens somewhere I can look at?
    u
    • 2
    • 1
  • a
    Hi team airbyte here is a PR https://github.com/airbytehq/airbyte/pull/23064 with the changes : • Adding Max_Level so that the user can configure how much depth the user has to flatten the data for warehouses query. • For extensive data having nested documents the Mongo connector takes a lot of time and Memory. • Use the mongo cursor to find keys so that all collections will not be taken in RAM at once saving RAM. • Added step for also considering empty string key. maintainers have a look
    m
    • 2
    • 7
  • s
    Hi, i'm using the lastest airbyte/source-postgres and for every field of data type String that contains null, i get an error. for every field in every row.
    Copy code
    2023-02-15 12:24:23 INFO i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed.
errors: $: null found, string expected
    shouldn't null be a valid value for type string?
    n
    • 2
    • 8
  • a
    is it possible to use latest images of connectors with older versions of Airbyte? is it guaranteed to work?
    m
    • 2
    • 1
  • j
    for large tables with no defined primary key or bookmark ("index" or "cursor") fields, which locking semantics is Airbyte incompatible with? of the 
    TRUNCATE
    and refill, 
    DELETE
    without explicit 
    LOCK TABLES
    and refill, 
    DELETE
    with explicit 
    LOCK TABLES
    and refill, and 
    DROP TABLE
    and refill semantics
    🙏 1
  • e
    u
    • 2
    • 1
  • o
    I've been having a weird issue where where Airbyte says a sync is successful but it doesn't replicate one of the 10 streams some of the times. It'll work for a while when I first create the connection but then stop working and won't work again until I create a new connection. It is usually the same table that doesn't get synced (which is a small table of like 30 records right now). I can see from the logs it finds the table and deletes the existing data (right now we are doing a full overwrite), but never queues up the query for the table or do anything after that. Anyone encounter something similar or have any idea what could be going on? Airbyte version: 0.40.23 Source: Postgres 1.0.28 (I also tried with latest version and same issue) Destination: S3 0.3.17
    m
    u
    +2
    • 5
    • 17
  • k
    Hi, I just upgraded from 0.40.7 to 0.40.30 yesterday and now running into an issue with bootstrapping Airbyte environment, specifically it looks like the FlyWay migration is failing upgrading from 0.29.15.001 to 0.40.26.001. We host the Airbyte db on an RDS instance, could it be the user doesn't have permissions to do the migration? Anyone have any idea? Error is in the thread
    ✅ 1
    • 1
    • 1
  • a
    Hi, We are moving from Airbyte OS to Airbyte Cloud, and it seems we cannot use the Pipedrive connector there. Is it an oversight ? Can it be added there ? Thanks
    n
    • 2
    • 1
  • p
    Hi, I am using Airbyte OS to move data into CSV files. Everything is syncing and exporting fine but I am unable to find the actual csv file on my machine. Any thoughts? Thanks!
    m
    • 2
    • 1
  • m
    If I am using the S3 Staging method on my destinations (Redshift serverless in this case), I get the error message:
    Copy code
    errors: $.access_key_id: object found, string expected, $.secret_access_key: object found, string expected
    in the last dbt loading step of the job. What is this looking for? I set up the S3 information on the destination with access_key_id and secret_key and it validates. Does S3 loading work with Redshift when using S3 stages?
    n
    • 2
    • 4
  • w
    Trying to understand the division of responsibility with Airbyte Embed (also where are the docs?). Some questions: • Will Airbyte Embed provide a UI for credential entry? • Will Airbyte Embed do the OAuth dance to receive access and refresh tokens? What happens in the case where the refresh token expires? What party is responsible for renewing the refresh token.
    s
    • 2
    • 6
  • j
    @Saj Dider (Airbyte) thread here around 
    [error] 922#922: *167 client intended to send too large body
    s
    • 2
    • 4
  • m
    A complete beginner question: lets say I created a connection between a source and a sink with 30 min sync interval. Does the connection pulls and push all the data at every 30 min or does it only capture changed data?
    m
    • 2
    • 1
  • c
    Hi there, I’m currently using Airbyte Open Source to manage my data integration, but I’m interested in migrating my existing data to Airbyte Cloud. I’m not familiar with the migration process, and I was wondering if you could help me with it. Can you please provide me with some guidance on how to migrate my data from Airbyte Open Source to Airbyte Cloud? Any resources or documentation on this topic would be greatly appreciated. Thank you in advance for your help! regards,
    🙏 1
    u
    • 2
    • 1
  • m
    Hello! I'm attempting to source data from a custom JSON API via HTTP GET requests and am not sure if I'm in custom connector territory. The API requires a start date / end date parameter that I'd like to set dynamically (fetching a days worth of data every day) but it doesn't look like that's something I can accomplish via the Files source. Am I wrong, or is the another source connector that could do this so I don't have to self-host Airbyte?
    m
    u
    • 3
    • 3
  • d
    Hello. Has basic auth been added to the kubernetes versions of Airbyte yet?
    u
    • 2
    • 1
  • h
    Hi Team, I knew this issue has been asked different time in this group. But I think my case is a bit different. The same image is pushed to different repo - docker hub and aws private repo. I am able to set up the new connector by using docker hub. But the error “*Internal Server Error: Get Spec job failed*” pops up when using aws private repo. Would you take a look? Thanks in advance. image name is hhuggv/bamboohr-new-source and *XXXX.dkr.ecr.us-west-2.amazonaws.com/bamboo-new-source* respectively
  • k
    What to do when “Could not perform read with with error: Every message grouping should have at least one request and response”?
    u
    • 2
    • 1
  • p
    Hello Team , I have been trying to use Stripe Connector , Stripe Connector as Source is working fine , I am even able to pass the test but when i am trying to connect it to the Database it error is coming , I have no idea what my destination is already working with other sources too . Please check and let me know , what i can do to resolve this issue . Thank You
    m
    u
    u
    • 4
    • 12
  • d
    Our Iterable-to-Snowflake sync job keeps failing with the same error code, and always at ~850,000 records into the sync of the stream "users".
    Copy code
    File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 761, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
    A one-time sync was successfully completed in October 2022, going back to early 2021, and we've been in daily incremental sync mode ever since. This week we attempted to reset the "users" stream and do a full re-sync before settling into daily incrementals. The failure at around 850,000 records on the full sync is 100% reproducible. Why this error never happened on full or incrementals before this week, but is now 100% certain, is a mystery. Airbyte OSS: v0.40.15 Iterable source: v0.1.23 Snowflake destination: 0.4.47
    n
    • 2
    • 8
  • m
    Hi, I'm having problems with the env var substitution when using Octavia. env
    Copy code
    ❯ env | grep DWH
DWH_DATABASE=bi2
    yaml
    Copy code
    configuration:
  database: ${DWH_DATABASE}
    apply
    Copy code
    ❯ octavia apply -f destinations/dwh_v2/configuration.yaml --force
🐙 - Octavia is targetting your Airbyte instance running at <http://host.docker.internal:9999> on workspace 52ee3a28-e1bc-426e-b0d7-8f04803ee7c0.
🐙 - DWH-V2 exists on your Airbyte instance according to your state file, let's check if we need to update it!
🚨 - Running update because the force mode is activated.
🎉 - Successfully updated DWH-V2 on your Airbyte instance!
💾 - New state for DWH-V2 stored at destinations/dwh_v2/state_52ee3a28-e1bc-426e-b0d7-8f04803ee7c0.yaml.
    Result:
    m
    l
    • 3
    • 6
  • m
    Hello Everyone, I want only certain fields to be sync from source to destination. so i should able to choose the fields in UI where we choose the tables. I see many users were requested on this feature. When this feature will be released in Airbyte? Thanks.
    ✅ 1
    m
    • 2
    • 2
  • s
    Hello Awesome people, I am trying to connect zoom api via airbyte. I am getting this error Failure Origin: source, Message: Something went wrong in the connector. See the logs for more details. Its working for other tables but failing for few with this message. Attaching the log below. Can anyone tell me the way forward. Logs:
    Copy code
    File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 180, in _read_stream
    for record in record_iterator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 288, in _read_full_refresh
    for record_data_or_message in record_data_or_messages:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 115, in read_records
    for record in self.retriever.read_records(sync_mode, cursor_field, stream_slice, stream_state):
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 371, in read_records
    for record in records_generator:
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 429, in _read_pages
    yield from records_generator_fn(request, response, stream_state, stream_slice)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 412, in parse_records_and_emit_request_and_responses
    yield from self.parse_response(response, stream_slice=stream_slice, stream_state=stream_state)
  File "/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 323, in parse_response
    raise ReadException(error_message)
airbyte_cdk.sources.declarative.exceptions.ReadException: Request <PreparedRequest [GET]> failed with response <Response [404]>
,retryable=<null>,timestamp=1676527746108], metadata: {connector_repository=airbyte/source-zoom, workspace_url=<http://localhost:8000/workspaces/f0f9b8d1-8610-41ce-bcdb-21c02cd07fa8>, failure_type=system_error, connector_command=read, connector_name=Zoom, workspace_id=f0f9b8d1-8610-41ce-bcdb-21c02cd07fa8, airbyte_version=0.40.28, connection_url=<http://localhost:8000/workspaces/f0f9b8d1-8610-41ce-bcdb-21c02cd07fa8/connections/f77dbc83-f3a7-4faa-9609-5e15960ef46e>, connector_definition_id=cbfd9856-1322-44fb-bcf1-0b39b7a8e92e, failure_origin=source, connection_id=f77dbc83-f3a7-4faa-9609-5e15960ef46e, connector_release_stage=alpha, job_id=1518, deployment_mode=OSS}
2023-02-16 06:09:45 INFO i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed. 
errors: $.method: must be a constant value Standard
2023-02-16 06:09:45 INFO i.a.v.j.JsonSchemaValidator(test):130 - JSON schema validation failed. 
errors: $.access_key_id: object found, string expected, $.secret_access_key: object found, string expected
2023-02-16 06:09:45 INFO i.a.c.f.EnvVariableFeatureFlags(autoDisablesFailingConnections):27 - Auto Disable Failing Connections: false
    u
    • 2
    • 1
  • c
    Hello everybody, I'm working with sensitive data from my sources, and I was wondering if there was any technique to mask/hash PIIs during the sync.
    u
    • 2
    • 2
  • l
    Hey, I dont know where to raise this but I think it might be relevant: Yesterday night I updated 
    airbyte/source-postgres
    from version 1.0.44 -> 1.0.45. Right after that, all my jobs using that connector failed with:
    Copy code
    User 'airbyte' does not have enough privileges for CDC replication. Please read the docs and add required privileges.
    Only way I have been able to make my jobs work again is by downgrading to 1.0.44 again.
  • d
    Hey folks! I submitted this PR (https://github.com/airbytehq/airbyte/pull/22802) last week, not sure if any further action is needed from my end! Thanks a lot
    m
    • 2
    • 2
1...144145146...245Latest