For those that have used Airbyte for some time, how happy are you with the ‘non-happy path’ experience? By that I mean when you have to troubleshoot things, having to reset syncs or backfill, etc. I’m concerned by what appears to be a ‘start it and it’s always perfect’ interface. Inside a source with multiple streams, I don’t see any methods to: • Reset a single stream • For incremental streams backfill from YYYY-MM-DD to current or restart from a specific id • see ingest metrics, write to the airbyte filesystem, write to destination metrics. Bottleneck identification in non-error scenario details • Any way to handle a non conforming piece of data. Like if a source returned garbage in an API call that didn’t map correctly a way to define how to handle or manually handle that resource

Hello, what's the good way to start using embedded Airbyte? we want to offer our customers a way to integrate their tools. I wasn't able to find a good way to start using it? is anyone using it already?

Please let me know if this is good enough: https://github.com/airbytehq/airbyte/issues/22037

Hello comunity, I have a issue with error "Additional Failure Information: scheduledEventId=48, startedEventId=49, activityType='RunWithJobOutput', activityId='5e061e7a-b6fe-36ec-b547-245364404b28', identity='', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED" How can i resolved this issue? GKE with 16+nodes and 29gb. Actually have 200 connections schedule 00 AM bd is as400 connector jdbc load method is: GCS worker configurations: - name: MAX_SYNC_WORKERS value: "400" - name: MAX_SPEC_WORKERS value: "200" - name: MAX_CHECK_WORKERS value: "200" - name: MAX_DISCOVER_WORKERS value: "200" - name: SYNC_JOB_MAX_ATTEMPTS value: "10" - name: SYNC_JOB_MAX_TIMEOUT_DAYS

Folks - is it possible to trigger “programmatically” a new Airbyte connector ? Maybe with a configuration file ? Thank you 🙏

Hi all, when deploying Airbyte (0.40.28) on AWS EKS, there is a problem that the DB cannot be connected. The detailed log is as follows:

I’m trying once again, Hello everyone! I have an S3 bucket with different folders that represent different tables (different schemas). For example: bucket: events folder_1: click_event folder_2: view_event Just to make sure I understand correctly, this is what “path patterns” solve? For example these 2 tables: events/click_event/…/…/… events/view_event/…/…/… events/next_event/…/…/.. I want to create a source for this S3 bucket (events) and have 1 connection for all different folders within it, is it possible? Or do I have to create sources as the number of subfolders I have and change the prefix each and every time? Thank you!

Hi. Question about octavia cli: How can I create a connection using the cli without that connection auto run (if possible) Thank you.

Hey! I'm currently using the self hosted version, building some custom connectors for the Heartland Point of Sale API. Heartland issues a custom subdomain to each of it's customers, so the base_url of the API changes with each storefront we have. We have a database we can pull each store's subdomain, ID, and apiKey from, but I'm not sure how to automate the ETL process in Airbyte. I need a way to request: 1. Foreach storefront a. Request a bunch of different streams from https://{subdomain}.retail.heartland.us/api b. Transform each subdomain data by adding a store database ID c. Load the streams' data into the DB. I'm brand new to Airbyte. I've built a working connector with 1 fixed subdomain, but I'm not sure how to do what I need to without creating 50+ nearly identical custom connectors for each store subdomain.

In the latest release i see references to launchdarkly and flags for it in a new file /configs/flags.yml? For a self hosted deployment, can i just remove all of these references without any negative impact?

I'm also getting an odd error in the browser console with my custom API source -> postgres. The custom source works fine when running

python main.py read --config...

, but when I attempt to load it into an Airbyte connection, I says it's obtaining my source schema, then fails with an unknown error. The browser console says

cannot convert undefined or null to object

. It's being called from object-assign. Has anyone experienced this before?

Hey Guys, Just stuck at one issue with Airbyte, Can someone help on how to disable the clustering when loading the data from MySQL to Snowflake as destination.

Hello. I am trying to submit my PR [https://github.com/airbytehq/airbyte/pull/20833/files] for Okta Connector. However, I keep getting failed source acceptance tests and wanted to test it locally. Does anyone know how can I add more time in the acceptance testing because it keeps timing out?

airbyte-integrations\bases\source-acceptance-test\source_acceptance_test\tests\test_core.py::TestDiscovery.test_backward_compatibility[inputs0] s                                                   81% ████████▏ {"type": "LOG", "log": {"level": "INFO", "message": "Using C:\\Users\\SampleData\\Documents\\airbyte\\airbyte-integrations\\connectors\\source-okta\\secrets\\config.json as configuration. It is the most recent version."}}
{"type": "LOG", "log": {"level": "INFO", "message": "The configured catalog is built from a fully discovered catalog."}}

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Timeout +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Stack of MainThread (5376) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
..
..
..

Hey Everyone. we’re a SaaS company considering migrating our shopify connectors to Airbyte (from fivetran). in the website I see it’s in alpha stage but I’m not sure what are the implications. has anyone tried to run it at scale (oauth authentication) and experienced significant issues?

This message was deleted.

Hi everyone ! Say I have an incremental migration that is base on the

end_date

column of a postgres table. For some reason there's been a bug and I have one row that was inserted with an

end_date

in the future and was migrated by airbyte. So airbyte considers that every row that's inserted in that table with a date anterior to that date in the future is already migrated and won't migrate it. I would like to fix the date that's in the future and remigrate the rows inserted in the meanwhile. I'm wondering if deleting the rows in the destination table until the date it want to start over the migration from will do the trick (or break everything instead 😱)?

Hey everyone, We're facing an issue while using Hashicorp Vault as a secret store. We are hitting an SSL exception on the airbyte-server when we try to configure a connection/source/destination. When airbyte tries to connect to our hashicorp vault to store a secret we provide in the connection UI, we hit the following exception:

Caused by: com.bettercloud.vault.rest.RestException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at com.bettercloud.vault.rest.Rest.postOrPutImpl(Rest.java:416) ~[vault-java-driver-5.1.0.jar:?]
	at <http://com.bettercloud.vault.rest.Rest.post|com.bettercloud.vault.rest.Rest.post>(Rest.java:306) ~[vault-java-driver-5.1.0.jar:?]
	at com.bettercloud.vault.api.Logical.write(Logical.java:264) ~[vault-java-driver-5.1.0.jar:?]
	... 82 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I have opened an issue here detailing the problem in full detail: https://github.com/airbytehq/airbyte/issues/22065 If anyone has any context on this please let me know thanku 🙂

Hey everyone, I am running airbyte in docker container in my mac . I am trying to add the transformation logic where I want to add new fields in the data and then load to destination can someone help me in configuring this where source and destination is postgres , is it possible to add those transformation using python since I have used hevo pipelines for transformation which was super easy to add ? does airbyte support such transformation. Thanks

Hello all, I'm in the process of deploying Airbyte to GCP (Compute Engine) following the documentation provided here. When initially launching Airbyte (

docker compose up -d

), I am being presented with the error

invalid mount config for type "bind": bind source path does not exist: /home/ad/airbyte/configs/flags.yml

as the individual containers are being created (Screenshot attached). Any idea why this is occurring?

this variable ACTIVITY_MAX_ATTEMPT shoud be set in the workers?

Hi Team, I am working on creating a custom connector for public API, I am able to complete the connection successfully, discover and read the data from API. But when I try to read the data using command

python main.py read --config secrets/config.json --catalog sample_files/configured_catalog.json

I am getting an data on command line but at the end of the response I am getting an error

"failure_type": "system_error"}}

I am not aware of the reason for this error, I have attached the screen shots for the response below, Please anyone help me understand the reason behind this error Thank you Slack Conversation

Hi, I am not sure if I am right here. I developed a custom connector and made now my first open source contribution to airbyte 🎉 . PR: https://github.com/airbytehq/airbyte/pull/21116. Can someone from Airbyte start a code review on it?

Hi! My name is León. I need help with the development of an Airbyte connector specifically with the type Incremental - Append Sync. Anyone who could be interested in helping me? It would be a small paid project. Please send your contact information at: leon.mycontact@gmail.com

I am trying to install Airbyte on an AWS EC2 instance following this guide. The command

sudo yum install -y docker-compose-plugin

is however failing with the error

No package docker-compose-plugin available.

. This has worked previously on the exact same machine (have been running Airbyte for several months). I have also tried using the docker-ce yum repository, but get a 404 error. Has anyone else run into similar issues before?

I have waited for almost 2 months for this PR https://github.com/airbytehq/airbyte/pull/21226 Can anyone help to review and share some feedback?

Hello, I'm running into an issue: Airbyte version: 0.40.23 Environment: standard self-hosted docker, on GC. Connection: Postgres -> BigQuery Logs attached below. I'm trying to sync a large table (> 300 GB). Smaller tables on this instance have already synced fine. When I click the "expand logs" arrow in the UI, I get the issue captured in the screengrab:

Hi! octavia wave We have been using Airbyte open source for several months now. Recently a new issue started plaguing our Postgres->Redshift CDC connector. Replication goes through, so does normalization, but the sync itself fails with a

Source cannot be stopped

error and then the next sync is a full one. Airbyte version: 40.28, Postgres version 1.0.36. Full stacktrace from logs in the thread.

I am trying to add a source (MongoDB, MySQL) in Airbyte but facing this issue attached, please help if possible.

hey! the only major problem we see from the experience with airbyte - is minio we’re running airbyte at scale and so we constantly reach minio max storage. our logs is configured to use S3 but Airbyte seem to be still using minio for something called “storage state”. how can we remove this coupling and use S3 only?

Hey, any chance to look into this PR https://github.com/airbytehq/airbyte/pull/19010 ?