Hi everyone, please I need someone who can assist me with how i can use dbt to transform data that is coming from airbyte before it gets into the destination database

Hi, I'm trying to setup a prefect task to schedule Airbyte runs. Is there a way to pass parameters to Airbyte? For example if I want to have a backfill flow, is there a way to programatically set the Airbyte sources start_date? Thanks!

Hi! I’m currently using JDBC connection (on Spark) to fetch a couple of Db2 tables for analytical use. I am considering using an ingestion tool. I wonder how Airbyte performs the db2 incremental append sync. The connector documentation mentions only

GRANT CONNECT

. Wouldn’t it need a lot more grants? For comparison, AWS DMS wants a user to have

SYSADM

(?!) and

DATAACCESS

. Debezium puts tables into a capture mode using ASN Capture/Apply agents, which sounds a bit scary on any production-related databases.

Hi can I do everything which I can do with airbyte UI using these API/s https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html Warm Regards.

Hello!!! I am creating a S3 destination for a POC but... i can see only Key id and Access Key as a authentication. Is there another method ? maybe secrets ? and.. where are the keys stored?

Hi there! I am new to Airbyte and I am trying to load 36 different sets of csv files from s3 to Snowflake. I am able to do this directly on Snowflake but wanted to test Airbyte. Is it possible to define a single S3 source that will understand the s3 "folder" structure and create separate tables in the DB? The structure is: s3://<bucket>/<table name>/yyyy/mm/dd/<table name>.csv.gz and I'd like to have a table for each <table name> created in the DB. Thanks!

Hi there! I’m new to Airbyte. Is there a way to embed the

workspaceId

somehow into the destination ?

Hello there - looking to setup a POC for syncing Stripe data to Snowflake and was reading through the docs and setup wizard that I would need to provide a lookback window to catch updates to records - I am sure each implementation is different, but is there a recommended value here or is there a way to check my source to see how much data is changing to influence this decision? I am fairly new to Stripe as a source so any help here would be appreciated. 🙂

Hi Airbyte Team, I am evaluating Airbyte. Source is CloudSQL (postgresSQL). Do we need SSH even when Airbyte is deployed in GKE in same VPC? Any help would be appreciated. FYI.. I did sign up for airbyte Cloud.

Morning, I put in a ticket for time based jobs (as our hubspot import was killing redshift during the import) into github and got a response suggesting that cron could be used to trigger jobs. I was wondering if anyone had written any docs on how to set this up?

Hello all! 🍀 I would like to create a Google Ads integration to Bigquery using Airbyte, but I’d like to be sure that Google Performance Max campaigns are included in the campaign reports. Do you know if it is the case? Thanks in advance for your help!

Hi all, new here to the community. Thank you for this! Does anybody know if Airbyte has a PostgreSQL issue where if there is too much time in between successful syncs that it will trigger a vacuum process that will fail the connection?

Hey all, Airbyte seems fantastic. At Greenly, we are doing carbon footprint reporting for companies. I wanted to try Airbyte Cloud this week end but got on the waiting list, any way to have a faster access? If that helps, I solemnly promise to give feedback once I did my tests 😉

Hey everyone 👋. I understand that right now connector secrets are to be entered in the UI, which I think is stored in a DB. I was just wondering if anybody has tried defining secrets loaded in as env vars in K8s, and have the secrets read from the environment instead?

Hi team, another question. I haven't set up Airbyte yet. How long has it taken others to get Airbyte set up and be able to start migrating data into data lake?

Hi team, Please I am using Quickbook as the source but I have always been getting this error. No successful migration so far. Is there anything I might have missed?

Hi Guys, It is stuck on connection creation

do I need to spin big on e

hey all, I wonder if airbyte fits my usecase: we want to enable our end user (users who register on our app) to manipulate their data from multiple sources and we are thinking to use airbyte to do the syncing for us, which means each end user has to allow us to access their data and we pool them into table per source, so it’s multi tenant. I see this: https://airbyte.com/embed-airbyte-connectors-with-api which i think is what we need but I am not able to find much information, it does link me to workspace? Is airbyte a fit for our need?

hello, Is there a source connector for neo4j?

Match for a specific use case

Hello there 👋, I wanted to ask you if Airbyte was a good option for a particular use-case: not as an ETL but as a RETL 🧐. I explain: we developed a lot of data-models via dbt in our DataWarehouse, Snowflake, and we unload them to a PostgreSQL instance in order to expose them to our end-users (Snowflake is OLAP oriented and not fast-enough for end-users queries, thus the need to unload them to Postgres) ⚡️. So our use-case is to keep a perfect sync between our Snowflake tables and our Postgres tables 🔄. By perfect sync, I mean having all the fields needed, a regular incremental replication and handling of deleted rows ⏰ We already set-up primary key in our Snowflake tables, as well as a "meta-field" with the last update date of the row 👍. Thanks a lot! 🙏

Hi! I created a S3 bucket and used a S3 parquet destination, but i have this message when I try to reed the parquet file

Would Airbyte be seen as a 1:1 replacement for Snowpipe (Snowflakes ingestion from S3)?

Hi 👋, Just playing around with airbyte and we set up a connector between an sql server db and bigqquery. When setting up which tables to sync in airbytes, some of the tables have the options for incremental sync and some only have full refresh. They are all coming from the same db and we enabled changed detection on all the tables as well, so there is no apparent different between tables on the sql side. Is there something I'm missing?

Hello everyone, Just getting started with Airbyte. I try to connect to Redshift server from local host. I was able to connect with a connector. However, I was not be able to connect to another Redshift server as a destination point. I keeps raising me with wrong configuration but I think I have looked for every on the internet for couple of days. But couln't find the answer. Is there anyone able to help me out.

Hi. I am trying to create a source for google sheets using the configuration API: https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/create The example schema looks like this: { "sourceDefinitionId": "17eb26bb-1d20-4000-8338-eab5bc180801", "connectionConfiguration": { "user": "charles" }, "workspaceId": "17eb26bb-1d20-4000-89b3-7a47513d8017", "name": "string" } How would that look like if I create a google sheets source using a service account?

Hey guys, hope everyone is doing great, testing out some sources and connectors, in postgress I have a JSONB column and I want to convert it to a variant column in snowflake, but i can’t find in the docs how to change the data type in the connections settings, can you help me figure it out please?

Hello. I have been stuck on an issue. How do I use airbytes API to build a connection between a preexisting source type(lets say MySQL) and a pre-existing destination(let's say Bigquery)? In both the API documentation and the python CDK I have found example to create a new source and all but I can't find an example as to how to build a source from one of the given types like MYSQL and all. Can someone please guide me. Thanks.

Hi Airbyte team, what access does the airbyte user need on an external database? Also, is there a sql script to generate this user? edit1: FYI i am using PG 13.4 on AWS RDS. edit2: I am talking about this btw

Running a fresh git clone of airbyte and getting this error on

docker-compose up

airbyte-server      | 2022-02-02 02:15:45 ERROR i.a.s.ServerApp(main):236 - Server failed
airbyte-server      | java.lang.NullPointerException: Cannot invoke "org.flywaydb.core.api.MigrationInfo.getVersion()" because the return value of "io.airbyte.db.instance.DatabaseMigrator.getLatestMigration()" is null
airbyte-server      |   at io.airbyte.db.instance.MinimumFlywayMigrationVersionCheck.assertMigrations(MinimumFlywayMigrationVersionCheck.java:75) ~[io.airbyte.airbyte-db-lib-0.35.15-alpha.jar:?]
airbyte-server      |   at io.airbyte.server.ServerApp.assertDatabasesReady(ServerApp.java:131) ~[io.airbyte-airbyte-server-0.35.15-alpha.jar:?]
airbyte-server      |   at io.airbyte.server.ServerApp.getServer(ServerApp.java:155) ~[io.airbyte-airbyte-server-0.35.15-alpha.jar:?]
airbyte-server      |   at io.airbyte.server.ServerApp.main(ServerApp.java:234) [io.airbyte-airbyte-server-0.35.15-alpha.jar:?]
airbyte-server exited with code 1