https://linen.dev logo
Join Slack
Powered by
# ask-community-for-troubleshooting
  • r
    Hi there, I have four airbyte instances set up on Google Cloud. Running for a couple months now, but there are already three times I saw Airbyte stuck - in UI the sync processes just keep spinning, even after days. Server is fine, load is almost none. I can attach logs, but is this a common error? I'm attaching the biggest error message from airbyte-worker. BTW 
    docker restart airbyte-server airbyte-worker
    seems to fix it, but I do hope it's more stable
    m
    • 2
    • 19
  • s
    Hello, the link for the example of the values.yaml to use on helm deployment is broken on https://github.com/airbytehq/airbyte/blob/master/charts/airbyte/values.yaml
    m
    n
    • 3
    • 2
  • m
    Hi, what is the default time zone for the "_airbyte_emitted_at" timestamp field from basic normalization? and is it possible to change the default time zone to a different one?
    m
    • 2
    • 3
  • m
    Hi Airbyte Community. I have recently started working on Airbyte (replacing Singer API). Issue i am facing is that while trying to sync data from mongo db to AWS s3 location, few columns are being missed in tables. For instance, Consider a table users with columns user_id, name, location,phone_num While the sync is performed, column 'location' is missed, while it is present in mongo db. In Airbyte UI also column location is not being shown. I tried to figure out this issue via Octavia CLI, but could not find any way that columns can be defined. Just wanted to mention that this column has ~70% nan values. But going with the fact that even for 30% data the column should not be missed. P.S. I am re-posting this issue just to get support, as earlier posted it on a Friday evening.
    👀 1
    n
    s
    +2
    • 5
    • 15
  • j
    Hi everyone, Just installed Airbyte (my first time) on EC2 t3.medium instance running Ubuntu 20.04 docker version 23.0.1 docker compose version 2.16.0
    Copy code
    mkdir airbyte && cd airbyte
wget <https://raw.githubusercontent.com/airbytehq/airbyte-platform/main/{.env,flags.yml,docker-compose.yaml}>
docker compose up
    I'm getting 
    service bootloader" didn't complete successfully: exit 255
    Any help will be appreciated, thanks! [update...] successfully deployed using
    Copy code
    git clone <https://github.com/airbytehq/airbyte.git>
cd airbyte
./run-ab-platform.sh
    had to change 
    docker-compose
    in script to 
    docker compose
    👀 1
    j
    d
    • 3
    • 4
  • v
    Hi airbyte-community,
  • v
    I am running airbyte on my own docker-based infrastructure. We are using it to move data between data sources and databases. Now I can’t add a simple file with https access.
  • v
    Copy code
    Failed to load https://#####:11012/out_csv/telefondaten.csv: ConnectionError(MaxRetryError("HTTPSConnectionPool(host='node1688-igniti-test-98123z.public.inetsolutions.cloud', port=11012): Max retries exceeded with url: /out_csv/telefondaten.csv (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8c95630a00>: Failed to establish a new connection: [Errno 111] Connection refused'))")) Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn conn.connect() File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect self.sock = conn = self._new_conn() File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f8c95630a00>: Failed to establish a new connection: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 489, in send resp = conn.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen retries = retries.increment( File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='node1688-igniti-test-98123z.public.inetsolutions.cloud', port=11012): Max retries exceeded with url: /out_csv/telefondaten.csv (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8c95630a00>: Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/airbyte/integration_code/source_file/source.py", line 111, in check with client.reader.open(): File "/airbyte/integration_code/source_file/client.py", line 92, in open self._file = self._open() File "/airbyte/integration_code/source_file/client.py", line 138, in _open return smart_open.open(self.full_url, transport_params=transport_params, **self.args) File "/usr/local/lib/python3.9/site-packages/smart_open/smart_open_lib.py", line 224, in open binary = _open_binary_stream(uri, binary_mode, transport_params) File "/usr/local/lib/python3.9/site-packages/smart_open/smart_open_lib.py", line 400, in _open_binary_stream fobj = submodule.open_uri(uri, mode, transport_params) File "/usr/local/lib/python3.9/site-packages/smart_open/http.py", line 49, in open_uri return open(uri, mode, **kwargs) File "/usr/local/lib/python3.9/site-packages/smart_open/http.py", line 81, in open fobj = SeekableBufferedInputBase( File "/usr/local/lib/python3.9/site-packages/smart_open/http.py", line 237, in __init__ self.response = self._partial_request() File "/usr/local/lib/python3.9/site-packages/smart_open/http.py", line 324, in _partial_request response = requests.get( File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 73, in get return request("get", url, params=params, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 701, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 565, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='#####.public.inetsolutions.cloud', port=11012): Max retries exceeded with url: /out_csv/telefondaten.csv (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f8c95630a00>: Failed to establish a new connection: [Errno 111] Connection refused'))
  • v
    Do you have an idea where that could come from? The URL is definitely available and reacts pretty quickly. It is not a huge file… I currently have no idea how to proceed.
    r
    • 2
    • 15
  • q
    Hey guys, I'm wondering if airbyte can be used somehow to create an API that sends JSONs to Google Cloud Storage via POST request or is this a problem I have to solve completely on my own? What do you say?
    r
    m
    n
    • 4
    • 3
  • v
    hey guys, i am trying to deploy airbyte on kubernetes using helm . I am using version 0.44.1 I am getting this error from webapp pod
    Copy code
    Readiness probe failed: Get "<http://10.3.212.13:80/api/v1/health>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
    • 1
    • 2
  • f
    hello, should use helm 0.44.3, 0.44.1 version have a bug
    s
    • 2
    • 1
  • j
    When running shopify, the logs show 
    Creating docker container = source-shopify-check-6-0-gnovc with resources io.airbyte.config.ResourceRequirements@188a3aec[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] and allowedHosts null
    If I set 
    JOB_MAIN_CONTAINER_CPU_REQUEST=0.50
    as an ENV var, should the above log line reflect a value of .50?
  • b
    Greetings! I am currently evaluating airbyte. It appears to be working but I might be doing something wrong. I ran a job to consume data from salesforce to a pgsql db yesterday and it ran for hours but nothing ever appeared in the destination tables. How soon should data start appearing once the job starts moving objects and what settings should I be using on those SF objects to move the data over?
    c
    n
    • 3
    • 18
  • o
    Hey all - trying to tune airbyte running on EC2. Is there a recommendation for max number of workers per core?
    s
    • 2
    • 1
  • j
    Hi Everyone, I am having an issue using the Xero source connector, i can set it up fine, all looks to work, but when i create a new connection i get the following error. Failed to fetch schema. Please try again Error: non-json response Any one got this setup? Josh
    n
    • 2
    • 4
  • p
    Hi everyone! Quick question regarding Google Analytics 4 - I'm using Airbyte to get data from a custom report, I have listed a couple of dimensions and metrics. I see that my data is properly loaded into the destination table but I have one extra column "uuid" which was not listed in the source settings, its name also doesn't have "airbyte" prefix so I assume it's not a technical column generated by Airbyte. do you know if "uuid" comes from GA4 and if so, what is it exactly? example of "uuid" - f921c912-94ec-48db-b8fd-3fd9baa4727c
    a
    • 2
    • 1
  • i
    Hello everyone I am trying using a connector facebook Pages, but I receive a error below. But I no have options in connector to reduce data ...
  • b
    Is there a way to mass toggle all found streams to off instead of one by one? One of my data sources has thousand schemas and I would like to only enable few tables, instead of everything.
    j
    n
    • 3
    • 3
  • i
    does airbyte support upserts on any of the destinations?
    r
    m
    • 3
    • 11
  • s
    Hi! Bringing back this bug in case anyone knows how to fix it 😞
    s
    j
    • 3
    • 4
  • a
    Hello Everyone, So I am just starting using Airbyte I deployed the production version on GCP following the documentation then I had to sync data from snowflake to postgres. My Problem Is it that Snowflake Sync Source Failure after 10-12GB, which is the exact description for a post I found on the forms https://discuss.airbyte.io/t/snowflake-sync-source-failure-after-26gb-airbyte-cloud/757 could anyone help me ?
  • r
    Hi There, I am looking for installing airbyte in my linux virtual machine. Any installable available in tar.gz format for download?
    n
    • 2
    • 4
  • j
    Hello, are the connectors (sources and destinations) for unlimited time and unlimited (GB) of information using in EC2 or EKS?
  • a
    Hi I have seen a number of people having issues with the ui going timing out in v0.41.0 has anyone found an ability to resolve this . I need to use facebook marketing with the v16 versio of there api. Because of this 0.40.32 will not work. Thanks
    n
    • 2
    • 5
  • s
    Hi All, I have deployed Airbyte OSS on EKS via helm. I want to change the log location to a custom S3 bucket, question is how can I pass the 
    aws_access_key_id
    and 
    aws_secret_access_key
    ?
    n
    r
    e
    • 4
    • 8
  • j
    Hi all, we have been having trouble with our Postgres CDC to Snowflake connection. The job takes a very long time (several hours) and often causes an 
    idle-in-transaction
    timeout (10 minutes). Shortly before the timeout, we also are getting the message:
    Copy code
    2023-03-05 21:10:08 [44msource[0m > INFO i.a.i.d.i.DebeziumRecordIterator(computeNext):106 No records were returned by Debezium in the timeout seconds 60, closing the engine and iterator
2023-03-05 21:10:08 [44msource[0m > INFO i.d.e.EmbeddedEngine(stop):1048 Stopping the embedded engine
2023-03-05 21:10:08 [44msource[0m > WARN i.a.i.b.IntegrationRunner(watchForOrphanThreads):245 The main thread is exiting while children non-daemon threads from a connector are still active.
Ideally, this situation should not happen...
Please check with maintainers if the connector or library code should safely clean up its threads before quitting instead.
    When we previously used the full refresh sync, it only took 1-2 minutes. Does anyone have an idea what we could be doing wrong since we switched to CDC? Thanks!
  • g
    Hi folks! 😄 I was trying to understand if is possible to use a config parameter in 
    AirbyteTriggerSyncOperator
    , in order to define the values for the parameters defined in the 
    spec.yml
    of a custom source. Does anyone have an ideia if is it possible?
  • m
    Hi All, has anybody deployed successfully Airbyte via Helm charts into a k8s using Istio? - seems that this issue has been over a year and I can’t believe this hasn’t been fixed
    a
    • 2
    • 38
  • t
    Hey team👋, I was wondering why 
    TEMPORAL_WORKER_PORTS
    is defined in the 
    .env
    file as well as in the deployment.yaml of airbyte worker as container Ports. Can anyone tell, which one is actually contributing to actually to determine the temporal ports pool? _PS: I am using airbyte version 0.40.6 and I have disabled container orchestrator by setting CONTAINER_ORCHESTRATOR_ENABLED=false._
    m
    • 2
    • 5
1...156157158...245Latest