Hi, I'm investigating if open source Airbyte can fit to our data aarchitecture. For now I need a load from Kafka to Postgres.

Hi, I'm investigating if open source Airbyte can fit to our data architecture. For now I need a data load from Kafka to Postgres. I've been able to prepare some basic connection, But how can I specify the data structure that would sync to Postgres table? I wonder how Airbyte recognize data structure from a specific Kafka topic. To be clear - I've got a destination table DDL defined and I want to create a Kafka topic that would sync and fill the table. Any help or source? Thanks.

Hello, I’m trying to convert .gz.parquet files (rds export to s3) to csv; so I setup both source and destination connector s3 with parquet file format and csv but when I deploy the connection it fails with “Failed to fetch schema. Please try again”

Hello folks, I am trying to configure a new source (Microsoft SQL Server); our airbyte is setup on EC2 instance, outbound traffic is full open, and we restrict the incoming traffic to VPN on 22 and 8000 port. We whitelist our public IP to the SQL server but when I try to connect I still have a timeout error (Could not connect with provided configuration. Error: HikariPool-1 - Connection is not available, request timed out after 60001ms.); I start to be out of ideas ... maybe someone had same issue before?

Hi 👋, could someone help me with this please - https://airbytehq.slack.com/archives/C038KM9NF7Z/p1657604594797989

Hi, I am trying to learn how to make an airbyte connector and i have issues while using zenoti API as example: • in order to iterate over diferent centers i am making an iterartor that updates the center_id parameter in the call inside of the method next_page_token, but i feel there is another way to do this

Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.

Hi Guys, not sure if this is the right place to ask , I am trying to use the airbyte api , specifically the endpoint POST /v1/sources/search ... and i am running into an issue where search only works for the first created source, any other source that is created is not searchable. Any ideas what could be causing this? It seems whatever indexing is being done on the sources for the search endpoint is not working

Hi All, I noticed that the bing ads connector has a lot of fields missing, anyone else is working on this? https://discuss.airbyte.io/t/bing-ads-missing-a-lot-of-fields-is-this-intentional/1824

Just testing the postgres connector (source) and logical replication. Not seeing anything other than Full Refresh | Append and Full Refresh | Overwrite though. Has anyone set this up recently? I've created: 1. unique index on table 2. replica identity using index on table 3. created publication on table 4. created replication slot using pgoutput 5. set wal_level to logical 6. set max_wal_senders to 2 7. set max_replication_slots to 10 8. no other publications, subscriptions, or replication slots created 9. restarted the pgsql instance 10. recreated the source using replication method (logical replication (cdc) ) 11. set the plugin to pgoutput 12. set the replication slot to name of replication slot in step 4 13. set the publication to name of publication in step 3 14. connected successfully 15. created a new connection 16. can see my table, but only have full refresh available

Hi ! I'm trying Airbyte in OSS version with MongoDB and AWS RDS MySQL sync into BigQuery : I'm wondering if it's possible to disable the creation of raw data ? This cost me a lot of (useless) data for no gain.

Another question about MongoDB : what is the best way to do if I have to sync data every 4h ? I have the feeling that the standard way import and overwrite all the data again and again, I'd hope there is something like CDC for MySQL : wdyt?

I'm running the local source (Slack) to destination (local csv & local json). For both, the syncs have generated the correct output. The syncs it show are "pending" for a while. I have set the sync to every 5 minutes. Any ideas as to how I can debug this? In the instructions:

Access the scheduler container using docker exec -it airbyte-scheduler bash

but airbyte-scheduler container isn't even running:

6c65bc474562   airbyte/server:0.38.2-alpha       "/bin/bash -c ${APPL…"   6 hours ago   Up 2 hours               8000/tcp, 0.0.0.0:8001->8001/tcp                                           airbyte-server
1cc732574ee0   airbyte/worker:0.38.2-alpha       "/bin/bash -c ${APPL…"   6 hours ago   Up 2 hours                                                                                          airbyte-worker
d0e4d1ca831d   airbyte/init:0.38.2-alpha         "/bin/sh -c './scrip…"   6 hours ago   Exited (0) 2 hours ago                                                                              init
dc2fd13d3fd7   airbyte/db:0.38.2-alpha           "docker-entrypoint.s…"   6 hours ago   Up 2 hours               5432/tcp                                                                   airbyte-db
0fcb790695b9   airbyte/webapp:0.38.2-alpha       "/docker-entrypoint.…"   6 hours ago   Up 2 hours               0.0.0.0:8000->80/tcp                                                       airbyte-webapp
ad950557824c   airbyte/temporal:0.38.2-alpha     "./update-and-start-…"   6 hours ago   Up 2 hours               6933-6935/tcp, 6939/tcp, 7234-7235/tcp, 7239/tcp, 0.0.0.0:7233->7233/tcp   airbyte-temporal

I'm just running what is in the getting started airbyte tutorial. Any tips on debugging?

We deployed Airbyte on an EC2 instance using docker-compose. It works well, but it seems like airbyte-server is crashing about daily (maybe when we add new tables). The web UI loads but shows an error message. ETL is still running. Is anyone else seeing this as well? We’re on the latest tag.

Hi I am copying here also just in case anyone has an idea https://airbytehq.slack.com/archives/C01AHCD885S/p1657706562537369

I have been looking around and cant seem to find a guide to code management and versioning. Is there a guide for best practices or how to work in a CI/CD framework to be able to deploy new/updated code without breaking existing code?

Hey there, is the required section in the source spec yml file for specific endpoints that we want to pull from the source ? I see there are authentication parameters ie client_secret included. Just wondering ifcredentials and endpoints are included in the required section

Hi. I m testing airbyte to ingest data into our S3 data lake, from some sources (mainly posture database). I noticed the parquet files has some extra fields and that updated rows are appended (which is good as it could provide time travel capabilities, like iceberg) How to I select the most updated version of the rows? Any python library to make it easier to access the file? Is there a way to periodically prune the parquet files so they have only the latest rows?

Hi all, we are evaluating Airbyte at the moment for some transfers and we really like it at start, but now having some setbacks and would like to ask for advice: 1. extending the raw tables with default Airbyte columns breaks our ETL, we can’t insert rows because column list doesn’t match (is there a workaround there or we would also need to map this new fields in every ETL script) - that doesn’t sound really user friendly 2. when trying to move large tables (it even started breaking with tables with 2M raws, so we didn’t even test with 2G or more) the job fails since after 3 attempts (trying in 50k batches and fail) - does using a larger machine in container solves this or we are missing some setup - we use GCS Stage for BigQuery upload as recommended any advice on these two things

Hello octavia wave I’m sending this message to help you know how this channel works. This channel is for basic questions related to Airbyte. Some examples are: Does the Salesforce connector support user region field in User endpoint? Does Airbyte deploys in AWS Container service? If you are facing a deployment or connection issue please use our Discourse forum to get support. Why we ask that? Because in Discourse your issue can be discovered by other users in the future; Discourse has a good integration with Github so you’re going to receive update when the issue is fixed in the project.

Hi there! Is it possible to install some additional Linux dependencies in Airbyte docker-compose definition? I'm trying to develop a new connector which needs some packages (in this case an Oracle Client), and I don't know which is the best way to achieve this. Thank you again for your support!

👋 howdy - I have a Q for folks using Airbyte to replicate Amazon Seller data. I would like to use airbyte to ingest the 'fulfilled shipments' stream. There is valuable customer id info in there for measuring new v repeat customers. However, I noticed an issue open that mentions this report is showing up empty when Airbyte runs. Anyone know if this has been resolved?

Is the airbyte helm chart published to a public helm repository anywhere?

Hi guys, I’m trying to add Kafka as a Source, though I have no clue what it mean “Subscription Method”. Anyone solved that ? I want it to sync to all partitions though I have no idea what is the pattern they are looking for.

Is embed API live? https://airbyte.com/embed-airbyte-connectors-with-api Has anyone used it? I couldn’t find much in documentation.

Hi Everybody, I´ve installed on linux cloud airbyte opensource locally. All seems works alright. Questions is.. Is there any setting for implement user authentication? I have checked entering to http://localhost:8000 / Settings / Account / Account Setting, and the unique option is typing an email as user..but Could I setting multiple accounts with password?

guys can someone give me a breakdown of airbyte CDK?

does anyone actually us it?

Hi all, new to Airbyte and I have built the container image locally but failed to create a connection to Hubspot source.. it seems to just hang..I also tried to connect to MySQL database and the same is happening..any ideas what the issue may be?

Hello! I am trying to setup this dashboard using these steps and I cannot get it to connect to the Postgres destination. https://airbyte.com/tutorials/build-a-github-activity-dashboard-for-your-project Can anyone help me troubleshoot?