https://linen.dev logo
Join Slack
Powered by
# ask-community-for-troubleshooting
  • l

    Leandro Bleda Cantos

    02/27/2023, 9:55 AM
    Hello, I have been testing Airbyte on an EC2 instance to CDC changes from an Aurora Postgres to Snowflake. (38 tables in total - Incremental | Deduped + history) The first runs are always successful but after the first run something weird happens: • Airbyte is unable to find the new records for several runs - I know there are new records. The job stales for 20 minutes (I set this timeout manually) trying to find the WAL resume position.
    Copy code
    Searching for WAL resume position
    No records were returned by Debezium in the timeout seconds 1200, closing the engine and iterator
    • Some synch jobs are able to find the the resume position but then it takes more than 5 hours to just load 11K records. HAve you seen this before? I have tried different configs but I never get an acceptable performance
    n
    • 2
    • 3
  • u

    Umar Hussain

    02/27/2023, 11:13 AM
    Hello has anyone had any luck deploying AB to an ec2 instance behind a security-group that has a load-balancer behind it and a private cert to be used to forward the UI ?
    u
    • 2
    • 4
  • s

    Slackbot

    02/27/2023, 12:05 PM
    This message was deleted.
    a
    l
    +4
    • 7
    • 10
  • j

    Jason Vondersmith

    02/27/2023, 1:19 PM
    Good Morning! I'm trying to use the
    Kustomer
    connector, currently in Alpha. I see there was a singer/user-agent issue with it originally. But then I found this PR that seems like it would correct the implementation. Is there a way to know when that PR might get merged?
    m
    u
    • 3
    • 2
  • m

    Mark Nuttall-Smith

    02/27/2023, 1:37 PM
    Hi, is it possible to configure the postgres destination to NOT truncate table names? (at least up to
    NAMEDATALEN
    ) eg. in the example below - the
    _airbyte_raw_...__user_course_performance_scd
    is truncated to
    ..._us___ourse_performance_scd
    m
    • 2
    • 2
  • l

    Layth Al-Ani

    02/27/2023, 1:55 PM
    Hi , When a new Helm chart will be released for the updated Airbyte v0.41.0 ?
    plus1 3
    m
    s
    k
    • 4
    • 27
  • l

    Lucas Abreu

    02/27/2023, 2:30 PM
    Hi there First time on AIrbyte and was following the quickstart docker on dploying it. I'm receiving this error, tho,
    io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline CallOptions will be exceeded in 9.996874017s
    Digging a bit, it seems to resources related, so I then edited all config in the
    .env
    file related to scaling to 1, so to at least give it a shot, but I can't get access to the UI. According to
    docker ps
    all containers are up and running, what seems misleading to me. Any tips would be much appreciated
    u
    • 2
    • 2
  • m

    Maciej Matuszewski

    02/27/2023, 3:00 PM
    Hello there 👋 Has anyone encountered a situation that multiple copies of all files are created without any clear reason? In my situation, I have Airbyte Open Source deployed on a local server, where I use File connector as a source and Postgres as a destination. Three weeks ago I updated a connection using Octavia CLI. I have not changed a thing since then and as I logged in today I noticed that every connection, source and destination are copied twice (so I have 3x more files now). The issue happened only in Airbyte API and from the sync logs it looks like it happaned on the same day I made the update. Nothing was changed in my Git Repo, and the files have not multiplied there. Unfortunately, I could not find anything here or in documentation in that regard, but maybe it is an issue that someone already had?
    n
    • 2
    • 4
  • r

    Richa Rochna

    02/27/2023, 3:43 PM
    Hi Team, I am trying to generate the template but I get this error. Can you please help.
    Copy code
    Running generator...
    While trying to generate a connector, an error occurred on line 39 of generate.sh and the process aborted early. This is probably a bug.
    u
    • 2
    • 2
  • c

    Carolina Buckler

    02/27/2023, 4:18 PM
    can Airbyte send SQS messages? or eventbridge?
    n
    • 2
    • 1
  • b

    Bradley Ming

    02/27/2023, 4:28 PM
    Hey there, we'd like to do CDC on a MySQL database, but we'd like the initial sync to be done on a separate instance of the database. Is that possible?
    • 1
    • 1
  • o

    Owais Javed

    02/27/2023, 5:24 PM
    hey all - I upgraded airbyte to version 0.41.0 and no longer see the option to export/import configurations in the UI under Settings>Configuration. Has this been moved somewhere else?
    u
    • 2
    • 3
  • s

    Shailendra Jaiswar

    02/27/2023, 5:33 PM
    Hey all, Is there a way to disable the sidecar injection in the pod creation by the worker job (source DB connection pod)?
    u
    u
    • 3
    • 4
  • a

    Adam Bloom

    02/27/2023, 6:22 PM
    I'm having an issue with postgres CDC that I don't quite understand. We have 20+ sources, most of which are successfully syncing. One source was syncing properly, but about a week ago started consistently encountering issues in debezium (logs in thread). Any ideas what could be causing this? I haven't been able to figure out what is different/unique about this source. I've tried resetting from airbyte, dropping and re-created replication slot + publication, etc - can't get it back to functioning normally.
    l
    • 2
    • 6
  • k

    Konstantin Lackner

    02/27/2023, 6:44 PM
    Where can I find the source code for the WooCommerce source connector? Is it possible to clone the connector and build on top of it myself?
    a
    • 2
    • 2
  • k

    Kevin Han

    02/27/2023, 7:57 PM
    If I turn off connection checking, are there any anticipated side effects? I don't want to launch pods to just verify the connection to source/destination on every X syncs
    u
    • 2
    • 1
  • b

    Bruno Carara Nandi

    02/27/2023, 9:11 PM
    Hi folks! Does anyone know what this error could be? Source: Zendesk Support Airbyte Version: 0.40.32
    c
    j
    • 3
    • 4
  • c

    Chen Lin

    02/27/2023, 11:18 PM
    Hi guys, is gzip the default compression for jsonl output format? I just recently upgraded the se destination to
    0.3.20
    and without specifying the compression(as shown in the screenshot below) and I noticed that the output files are already compressed, is this expected? our airbyte is on version
    0.39.36-alpha
    m
    u
    • 3
    • 3
  • g

    god830

    02/27/2023, 11:36 PM
    Got the same issue: https://discuss.airbyte.io/t/413-request-entity-too-large-nginx-1-23-2/3333
    l
    • 2
    • 1
  • t

    Tiger Sun

    02/28/2023, 2:28 AM
    • hello team, was able to deploy into our k8s (helm install with
    0.43.29
    ) environment but whenever I try to create a source or destination, it fails. from the k8s pod logs, i can see
    Copy code
    call-heartbeat-server Terminated
    main Using existing AIRBYTE_ENTRYPOINT: /airbyte/base.sh
    main Waiting on CHILD_PID 7
    main PARENT_PID: 1
    main EXIT_STATUS: 0
    relay-stderr 2023/02/28 01:24:32 socat[8] N reading from and writing to stdio
    relay-stderr 2023/02/28 01:24:32 socat[8] E TCP: wrong number of parameters (9 instead of 2)
    relay-stderr 2023/02/28 01:24:32 socat[8] N exit(1)
    init All files copied successfully, exiting with code 0...
    relay-stdout 2023/02/28 01:24:32 socat[8] N reading from and writing to stdio
    relay-stdout 2023/02/28 01:24:32 socat[8] E TCP: wrong number of parameters (9 instead of 2)
    relay-stdout 2023/02/28 01:24:32 socat[8] N exit(1)
    Stream closed EOF for airbyte2/n-local-json-check-57b98b6d-867d-4f38-878c-33b5cb1beabb-0-ymcce (main)
    Stream closed EOF for airbyte2/n-local-json-check-57b98b6d-867d-4f38-878c-33b5cb1beabb-0-ymcce (init)
    Stream closed EOF for airbyte2/n-local-json-check-57b98b6d-867d-4f38-878c-33b5cb1beabb-0-ymcce (relay-stderr)
    Stream closed EOF for airbyte2/n-local-json-check-57b98b6d-867d-4f38-878c-33b5cb1beabb-0-ymcce (call-heartbeat-server)
    Stream closed EOF for airbyte2/n-local-json-check-57b98b6d-867d-4f38-878c-33b5cb1beabb-0-ymcce (relay-stdout)
    but can’t figure out what exactly is going on (for example, this is an attempt with local json) and a related log (attached) with null pointer exceptions. has any seen this before?
    u
    m
    s
    • 4
    • 6
  • j

    James Salmon

    02/28/2023, 2:30 AM
    Quick simple question - I am running the dockerized version of Airbyte, but I do not see the options to export and import config? I’d like to export my config from one machine, and import it to another to put it into production. Any clues? I should mention that I looked in Setting -> Configuration, but the option does not appear - is there something additional I need to install?
    u
    • 2
    • 1
  • b

    Benen Cahill

    02/28/2023, 6:28 AM
    Hey Airbyte team, I noticed that release notes for 0.40.31 and 0.40.32 have been edited with the warning
    This version included a bug. Use 0.40.30 instead.
    Why is this the case? For those of us who had already migrated to
    0.40.32
    are we required to downgrade? I cannot find any further relevant documentation.
  • j

    Johannes Müller

    02/28/2023, 7:18 AM
    Could you point me to the (github) code that failed? The logs don't provide a stacktrace, so I don't see what "check" failed. I am trying to set up the Redshift destination.
    n
    • 2
    • 8
  • a

    Agrim Gupta

    02/28/2023, 7:35 AM
    Hi team , how to maintain the state of airbyte pipeline not locally
    u
    • 2
    • 1
  • m

    M. Farahmand

    02/28/2023, 8:09 AM
    Hi! Loving Airbyte so far, but facing an issue with custom HTTP source connector where I should send a POST request with two request body fields. I use this tutorial and everything works well, although with hard-coded request body fields. But I don’t know how to pass these currently hard-coded values to the container. Much appreciated if someone can explain how I should pass these two values to the container so that they can be used by MyClass’s
    request_body_json()
    method.
    u
    u
    • 3
    • 3
  • z

    ZouYong

    02/28/2023, 8:26 AM
    Hi, I'm building an airbyte destination connector for our db. Would anyone shed some lights on what I need to do to support basic normalization? I overrode
    supportsNormalization
    to return true in the acceptance test file. But this line in the DestinationAcceptanceTest.java always fails to assert.
  • z

    ZouYong

    02/28/2023, 8:27 AM
    I went through the docs and didn't found any specific guide.
  • u

    김건희

    02/28/2023, 9:44 AM
    Hi everyone! I'm using airbyte(Chart version : 0.43.33, App version : 0.40.32) on eks and i have a little problem Once airbyte-temporal pods are restart, there is an error because of below code update-and-start-temporal.sh
    Copy code
    update_postgres_schema() {
      { export SQL_PASSWORD=${POSTGRES_PWD}; } 2> /dev/null
    
      CONTAINER_ALREADY_STARTED="CONTAINER_ALREADY_STARTED_PLACEHOLDER"
      if [ ! -e $CONTAINER_ALREADY_STARTED ]; then
          touch $CONTAINER_ALREADY_STARTED
          temporal-sql-tool --plugin postgres --ep "${POSTGRES_SEEDS}" -u "${POSTGRES_USER}" -p "${DB_PORT}" create --db "${DBNAME}"
          temporal-sql-tool --plugin postgres --ep "${POSTGRES_SEEDS}" -u "${POSTGRES_USER}" -p "${DB_PORT}" --db "${DBNAME}" setup-schema -v 0.0
    
    
          temporal-sql-tool --plugin postgres --ep "${POSTGRES_SEEDS}" -u "${POSTGRES_USER}" -p "${DB_PORT}" create --db "${VISIBILITY_DBNAME}"
          temporal-sql-tool --plugin postgres --ep "${POSTGRES_SEEDS}" -u "${POSTGRES_USER}" -p "${DB_PORT}" --db "${VISIBILITY_DBNAME}" setup-schema -v 0.0
      fi
      echo "Starting to update the temporal DB"
      temporal-sql-tool --plugin postgres --ep "${POSTGRES_SEEDS}" -u "${POSTGRES_USER}" -p "${DB_PORT}" --db "${DBNAME}" update-schema -d "${SCHEMA_DIR}"
      echo "Update the temporal DB is done"
    
      echo "Starting to update the temporal visibility DB"
      temporal-sql-tool --plugin postgres --ep "${POSTGRES_SEEDS}" -u "${POSTGRES_USER}" -p "${DB_PORT}" --db "${VISIBILITY_DBNAME}" update-schema -d "${VISIBILITY_SCHEMA_DIR}"
      echo "Update the temporal visibility DB is done"
    
    }
    the problem is, i'm using database name ${DBNAME} but when pods restart, pods try to make ${SCHEMA_DIR} (named temporal) why this happen and if i want to use ${DBNAME} database after restart pods, how can i do that?
    n
    • 2
    • 2
  • c

    Chandrasekharan Mohan

    02/28/2023, 12:34 PM
    Hello all! I'm adding some custom changes to airbyte's (v0.40.30) python CDK such as • adding some logs • playing with state to pass filters .etc. I was able to locally test this by following these steps for a single connector. But now I wish to rebuild all the connectors to make use of my local python CDK changes. The steps described in the doc calls for extensive manual effort. 1. Is there any script/way to rebuild all connectors and make them use my custom python CDK? 2. If there are changes to airbyte_cdk, how does Airbyte take care of updating and pushing all connector images with this change? Love the existing work done, thanks in advance 😄
    n
    • 2
    • 3
  • h

    Hiren Patel_C

    02/28/2023, 12:42 PM
    Hello everyone! I'm using airbyte (App version: 0.40.32) I want to use MySQL source connecter (beta version) for my ETL which has around 1TB of data and 200 tables. so how much MySQL connecter is reliable to handle this Sync process in CDS and standard mode. can anyone help me out with this?
    s
    u
    • 3
    • 2
1...151152153...245Latest