Hi. Per my understanding, Airbyte supports 4 sync modes: 1. Full refresh - Overwrite 2. Full refresh - Append 3. Incremental Sync - Append 4. Incremental Sync - Deduped History Ref: https://docs.airbyte.com/understanding-airbyte/connections but according to this link https://docs.airbyte.com/quickstart/set-up-a-connection, it says that Airbyte is doing

Full refresh

on final destination table.

At the moment, Airbyte runs a full-refresh to recreate the final tables. This can cause more costs in some destinations like Snowflake, Redshift, and Bigquery. To understand better what sync mode and frequency you should select, read this doc. There is a FAQ topic on our Discourse that more extensively explains the cost issue here.

Can someone tell me if this is still true for Incremental Sync - Deduped History too?

Hi we would like to know if Airbyte can help us to take all the data changes in SQL server(we use web edition and CDC is not working in our version ) and in mondoDB ?

Hi, I just want to know if Airbyte has a Rest API that I can use to connect with the already built in connections and destinations, instead of using it’s UI ???

Just wanted to say hi--I'm looking to make a first contribution here at Airbyte but might need some guidance!

In which programming language you can contribute with ?

Is there any way I can receive notification that a job is done ???

Hello! I am trying to create a HTTP connector for consume a servicesnow API i am following the tutorial Speedrun but T do not know where put the credentials for the API

{
  "type": "LOG",
  "log": {
    "level": "FATAL",
    "message": "HttpStream.__init__() got an unexpected keyword argument 'auth'\nTraceback (most recent call last):\n  File \"/Users/daniel_edu/Projects/PERSONAL/airbyte/airbyte-integrations/connectors/source-servicesnow-api/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/Users/daniel_edu/Projects/PERSONAL/airbyte/airbyte-integrations/connectors/source-servicesnow-api/.env/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 127, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/Users/daniel_edu/Projects/PERSONAL/airbyte/airbyte-integrations/connectors/source-servicesnow-api/.env/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py\", line 118, in run\n    for message in generator:\n  File \"/Users/daniel_edu/Projects/PERSONAL/airbyte/airbyte-integrations/connectors/source-servicesnow-api/.env/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py\", line 92, in read\n    stream_instances = {s.name: s for s in self.streams(config)}\n  File \"/Users/daniel_edu/Projects/PERSONAL/airbyte/airbyte-integrations/connectors/source-servicesnow-api/source_servicesnow_api/source.py\", line 55, in streams\n    return [ServicesnowApi(auth=(config[\"psswrd\"], config[\"psswrd\"]), user=config[\"user\"], psswrd=config[\"psswrd\"])]\n  File \"/Users/daniel_edu/Projects/PERSONAL/airbyte/airbyte-integrations/connectors/source-servicesnow-api/source_servicesnow_api/source.py\", line 67, in __init__\n    super().__init__(**kwargs)\nTypeError: HttpStream.__init__() got an unexpected keyword argument 'auth'"
  }
}

I was trying this script

def streams(self, config: Mapping[str, Any]) -> List[Stream]:
        """
        TODO: Replace the streams below with your own streams.

        :param config: A Mapping of the user input configuration as defined in the connector spec.
        """
        # TODO remove the authenticator if not required.
        auth = TokenAuthenticator(token="api_key")  # Oauth2Authenticator is also available if you need oauth support
        return [ServicesnowApi(authenticator=(config["psswrd"], config["psswrd"]))]

I'm trying to replicate connections in the config files and then upload it.. but I am having troubles getting it to recognize the added connections. I have ensured they all have unique identifiers. Is there something else I am missing?

Hi, I just started looking at using Airbyte to replicate on-prem Sqlserver database tables to BigQuery in GCP. I used “Basic normalization” with “No custom transformation”. I am able to replicate the tables in Sqlserver to BigQuery. However, I notice the dataset created in BigQuery is automatically chosen as “dbo” which is the schema name of the Sqlserver database tables (btw, the database is “XYZ” and the tables are like dbo.tableABC in Sqlserver). I would like to force the destination dataset name in BigQuery to use the Sqlserver database name “XYZ” (and the table names in BigQuery should be the same as in Sqlserver with or without the dbo prefix). Is there a simple way to do this? Thanks a lot.

Hey, I want to know if there is any way to customize the UI, for example changing the logo etc.., and do I have the right to modify the UI if I want to use it internally ?

Hello, I'd like to know if it's possible to extract and load data directly from Postgres database hosted on AWS RDS to BigQuery?

Hi. Is there a Firestore source or destination please? There seems to be a menu item in the docs, but the page itself appears to be empty? Thanks https://docs.airbyte.com/integrations/destinations/google-firestore

or destination

Heya! We are slowely setting up Airbyte (deployed locally as Airbyte Cloud not availabe in EU yet). Is it possible already possible to use the Google Sheets adapter with OAuth? Having to download the json file with credentials doesn't seem too save! (And if not - any suggestions on where/how to store those credentials?)

Hi there! Curious about how the Airbyte Cloud offering is implemented (and I can’t trial as I’m in the EU). Say I were to use Cloud and want to transfer data from source A to destination B, does the data first flow to your cloud and gets staged there before it goes to my destination B? Thanks!

Hi everyone! When resetting data, my connection doesn't initiate a full sync, anyone encountered a problem like that? My understanding is that on resetting the data, the entire dataset in the destination gets deleted and re-synced, but in my case it's indeed get's deleted but only new data get's sync. This problem can be solved by creating a new connection.

Hi, I got one very simple question in AirByte architecutre. If Airbyte already has a scheduler and workers, why does it need Airflow? What's the point of triggering Airbyte job from Airflow?

Hi guys - one question - how can I move an instance of airbyte to a new host (i don't want to move the VM, just airbyte). It's important I want to keep the state, so no full resyncs, etc. I'm using docker. Thanks

Hi. I've followed the two steps for deployment shown at https://docs.airbyte.com/quickstart/deploy-airbyte but am getting a "_File not found_" issue. Log attached. Can anyone help please? macOS 12.0.1 (Intel) docker version 20.10.6, build 370c289 docker-compose version 1.29.1, build c34c88b2 Thanks

I'm attempting to deploy airbyte onto an EKS cluster but am hitting an error. All pods are displaying this error. Any advice? Solution: Was due to no node group being created on the cluster.

hey guys! im a total noobie. got a question, an I use airbyte to send data from a DWH to salesforce (like a reverse etl) ?

Hi guys! I am new to the platform. You guys have done an amazing jobs in cracking the APIs for lots of the sources connectors. It seems that most APIs are read-only API. what if I want to write my data to these apps. For example, I want to sync user data to intercom.

Heya! I have a question regarding Airbyte + dbt We have an important table in Google sheet with product information that get's updated once every week where new rows get added and others deleted/changed ( and the schema changes sometimes). And the goal is to use airbyte + dbt on a posgres database to save all historic data for downstream aggregation and analysis My first thought was to sync tables by

full refresh - append

but this already broke because someone added a new column and the sync schema didn't stay the same so we need to stick with the

overwrite

option. Does anyone have a suggestion on how to properly archive/store all data with dbt? My idea was to use the

snapshot

feature of dbt as it can also add new columns that don't fit with the created schema. Thank you!

and even retry get stucked

Hey everyone! I have a question regarding the integration of Prefect and Airbyte. I have a Prefect flow that is copying a couple of Google BigQuery tables between 2 different GCP projects. The flow is reading a config (json) file that I have in google storage. For each region and table in the config file a copy task is spawned using Prefect map option (I have multiple regions, each one has different tables - each table has its own unique source and destination). Now I was wondering if its possible to do the same thing using Airbyte? spawning a new connection by using Prefect if this connection doesn’t exists (lets say - for region A between BQ table a and table b) and if it does exist - just trigger the sync? More specifically - since my configuration is dynamic and regions/tables will get added I don’t want to configure each connection manually in the Airbyte UI. Thanks!!

Hi, Successful install of AirByte on AWS Lightsail, but is there a straightforward way of adding authentication / a login & password to secure my AirByte install? Thanks

What is the appropriate channel to ask for help if I am stuck with something?

Hello, I am trying to install airbyte from the docs here ->https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine but i am getting an error sudo apt-get install -y docker-ce docker-ce-cli containerd.io -- > E: Package 'docker-ce' has no installation candidate E: Unable to locate package docker-ce-cli E: Unable to locate package containerd.io E: Couldn't find any package by glob 'containerd.io' E: Couldn't find any package by regex 'containerd.io'

hey folks, I'm evaluating Airbyte vs. Fivetran for MySQL -> Redshift CDC data replication. Have a few questions about how this works: 1. How does the Redshift connector handle updates and deletes from the MySQL source data on the Redshift side? 2. How fast can it run? What is the latency between MySQL to Redshift at the maximum possible replication speed Airbyte currently allows?

Hi! Can someone help me understand the relationship, if any, between the list of connectors here on Github https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors and the connectors listed on the Airbyte website at https://airbyte.com/connectors ? Some seem to be in one place but not the other. eg; Firestore is on Github but not on the web, ebay is on the web but not on the Github list ^^. Apologies if I missed something obvious. :-) Thanks