how long does it take to deploy completely as I am not able to use Web -UI ( http://localhost:8000)

question, how do we use the

source-quickbooks

oauth flow defined here: https://github.com/airbytehq/airbyte/blob/e4bb0b927415128a5d0519398b844955af6f3732[…]/src/main/java/io/airbyte/oauth/OAuthImplementationFactory.java? There is a

source-quickbooks-singer

, but no

source-quickbooks

that I can see to get the consent url for example

Need help in getting 👀 on this PR: https://github.com/airbytehq/airbyte/pull/18677

Hello. I saw on the documentation (https://docs.airbyte.com/operator-guides/configuring-airbyte/) a parameter to configure AWS Secret Manager but this is the only page I could find about that. I am looking for a way to store the credentials of the connectors I set on airbyte in AWS Secret Manager. Is it already available or not ?

Hi there. We’ve started using airbyte and we really love your app. We would like to store airbyte configuration (connections, sources, destination) in git (import/export configuration) or configure airbyte using tool like terraform. Do you plan to add such functionality to airbyte?

Hi, I wanna ask something about upgrading version. We have been using

0.39.1-alpha

and will upgrade into latest one,

0.40.28

(using docker deployment). I found that there are some cases which need to be considered like docker version, old airbyte configuration, and table migration. Do you guys have some tips for this case (I am afraid it will corrupt the existing pipelines and we have to start from scratch again) https://docs.airbyte.com/operator-guides/upgrading-airbyte/#resetting-your-configuration *NB, I am trying to simulate the upgrade mechanism in local machine (Mac OS BigSur), then I will do the production one (in GCP VM Ubuntu) when I am sure enough.

I have the following problem I am using full refresh with overwrite to copy a table from BigQuery to Postgresql however every time I do sync I am getting no permission to the table

SELECT relname, relacl FROM pg_class WHERE relname = 'my_table';
        relname        | relacl
-----------------------+--------
 my_table |

although my_user has read permission, once I manually give permission to my_user to read all tables

SELECT relname, relacl FROM pg_class WHERE relname = 'my_table';
        relname        |                                      relacl
-----------------------+-----------------------------------------------------------------------------------
 my_table | {airbyteuser=arwdDxt/airbyteuser,"\"my_user\"=r/airbyteuser"}
(1 row)

I can see the table, but ONCE I do

sync

again the story repeats. Is there way to avoid this?

this error is showing up

application version: v0.40.27 helm chart version : v0.43.18 i'm trying to use

mysql

connector as my source by establishing SSL connection None of the push buttons r working for me...where to start debugging this?

Has anyone come across this issue with MySQL destination and using custom transformation? using the normal transformation fails because it converts a json to string instead of text so I tried to set up the custom dbt but it fails to install the deps for some reason. I have followed the guides with setting the

packages-install-path: ../dbt

but it still fails. Is there a config somewhere where I need to define the profile My transformations doesn't contain any special arguments just deps followed by another transformation to run Any help would be much appreciated

Hi everyone, I am trying to setup Aitbyte on k8s in GCP. I tried the autopilot mode but randomly I was getting "Does not have minimum availability" error, so I turned to standard clusters. With a regional one it nicely worked, but with a zonal one again - for

airbyte-server

I am getting the same error. I've added one extra node, but it didn't help. I'm using

kubectl apply -k kube/overlays/stable

Any idea what might be the issue? 🙂

Hi guys!

I need help I'm new with this open source project and I don't have much knowledge in java. I managed to make the connector for Facebook, but for BigQuery it is having this error. How to configure this authentication?

image.png

I'm waiting for an answer, help to configure this connection variable to Google Cloud's BigQuery. Grateful friends.

Hello, I`m receiving this error while pulling images of sources (snowflake, mssql): failed to register layer: Error processing tar file(exit status 1): write /usr/lib/jvm/java-17-amazon-corretto/jmods/jdk.localedata.jmod: no space left on device info: Airbyte latest VM Instance on GCP e2-standard-2 (8GB)

There is still no solution to run dbt on kubernetes? It's a serious problem, we want to abandon GG from oracle for airbyte and we need to obfuscate data by policy tag with dbt @Nataly Merezhuk (Airbyte) @Marcos Marx (Airbyte)

I am looking to hire someone to help us complete the ZOHO integration with Airbyte. We were interested in using the cloud solution from Airbyte, but no one from the organization responded to our past emails. We prefer not to host ABT by ourselves and use airbyte.io for our use case. If anyone has done ZOHO integration or has some ideas, please reach out. Much appreciated. 🙏

Hey all 👋 I’m looking into how to get data from S3 properly, using the S3 source connector 👀 In S3, we have a full snapshot of a third party system dumped every day, and named after the day for which the dump is (so basically

<s3://my-bucket/some-path/dataset1-2023-01-17.csv>

for today, etc). As you probably know, I need to provide a regex to define which file(s) to pull. What I’d like to do, using the S3 connector, is to only read the file for the current day, rather than all the files present there for

dataset1

(i.e.

dataset1-*.csv

does not work and something like

dataset1-<some_date>.csv

would work). I thought about going incremental but since each day is a full snapshot, I think this is somewhat overkill and the “state” does not make sense in that situation. Is there a way to achieve this with Airbyte and the S3 connector ? Let me know if there’s any additional detail I can provide. Thanks!

Is it possible to set resource limits for

airbyte-connector-builder-server

?

For AirByte on Kubernetes, which components benefit from replication factor > 1? ie: do any of the components benefit from the ability to horizontally scale? Coming from deploying this via docker-compose there weren't many options, and I don't really see an operations guide for running AirByte on K8s -- so without knowing more about the internal architecture I'll be mostly guessing.

Hi team, How can I enable OAuth support in airbyte using helm chart? Can any one share some runbooks or documentation?

Hi guys. I'm using the airbyte cloud, and would like to access the metadata of each connection made in my workspace. In the open source version, I can access the data registered in PostgreSQL, how can I access the data registered in the parameter table using airbyte cloud?

Hi guys! I'm running airbyte local and I'm running

docker compose up

and at the end I have a problem, because I have other services running in the port 8000. I would like to know wich files I need to change port 8000 to another port. The error below

Hi Team, we want to source data from salesforce marketing cloud. At this link, it is mentioned that currently Airbyte does not support this connector. https://airbyte.com/connections/Salesforce-Marketing-Cloud-to-S3 . Whereas there is a connector listed for salesforce at https://airbyte.com/connectors/salesforce. So does this mean that this salesforce connector is different than the required one which connects SFMC?

Where can I find Airbyte normalization compiled SQL? I'm trying to understand this error better by trying to find this compiled SQL during a Redshift -> Postgres sync:

2023-01-18 03:35:27 normalization > Database Error in model monthly_bank_account_balance_by_workspace_stg (models/generated/airbyte_incremental/accounting_metrics/monthly_bank_account_balance_by_workspace_stg.sql)
2023-01-18 03:35:27 normalization >   cannot delete from table "monthly_bank_account_balance_by_workspace_stg" because it does not have a replica identity and publishes deletes
2023-01-18 03:35:27 normalization >   HINT:  To enable deleting from the table, set REPLICA IDENTITY using ALTER TABLE.
2023-01-18 03:35:27 normalization >   compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/accounting_metrics/monthly_bank_account_balance_by_workspace_stg.sql

Hello everyone. I'm having a strange failure with Airbyte deployed to Kubernetes. When trying to setup any connection or source, the

check_connection

calls are retuning a "upstream request timeout" error and I'm currently not able to understand where is it coming from because there are absolutely no errors in the logs, even with DEBUG enabled. I've logged an issue here, any help is appreciated: https://github.com/airbytehq/airbyte/issues/20963. Can anyone tell me how can I further debug this issue? I've exec'd into the server container and it can communicate and connect to all the other hosts (temporal and webapp, for example). I'm stuck and have no idea how to debug it further.

Hello everyone! Does somebody know that there any connectors right now at airbyte, which uses substreams like Shopify. For example there if you would like to fetch

transactions

, first it fetches

orders

and they are both incremental reads. What I would like to know if there are any connectors at airbyte right now which fetches the parent stream incrementally, but after that fetches the child stream with full refresh. Thank you in advance.

hey All, Sweeper Pod is not deleting the Pods and also when I check the logs, it’s not even generating any logs

Hi, Im not able to add custom dbt transformation in airbyte opensource, when fill form and click on save changes, it disappears. Any one?