Hello guys, wanted to know if there is a support for APIs for Airbyte cloud now?

Hi everyone. I'm quite new to airbyte. Can anyone confirm does Airbyte allows to do transformation on top of the connection that are made. I came to this query based on this thread https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-sql. Can someone please confirm.

Hi all! as the google ads and bing ads source connectors have predetermined cursor fields for the incremental + history deduped syncs, I think it doesn´t update the field. For example, keyword with id x cost may change retroactively, but as the cursor is the timestamp/ segments.date we won´t detect the change. How do people handle this problem? I thought of creating a process that once a month I full refresh those tables and rewrite existing rows and add the new ones, but I don´t see the possibility to set the source to refresh the last 30 days, as an expression. Any ideas?

A question I've never seen really answered is, can we use Airbyte to CDC from PostgreSQL (log based CDC) into Snowflake, in real time? (ie, <= 30 seconds from data being generated in postreSQL to it being available in Snowflake)

Seeing this error when I attempt to reset the data of a Postgres <-> Postgres connection

Additional Failure Information: message='java.lang.NullPointerException: Cannot invoke "java.lang.Boolean.booleanValue()" because the return value of "io.airbyte.persistence.job.models.IntegrationLauncherConfig.getIsCustomConnector()" is null', type='java.lang.RuntimeException', nonRetryable=false

Airbyte v.0.40.24 deployed via Helm Chart

I've tried to sign up at https://discuss.airbyte.io but the email never comes.

Hello, I'm having some trouble with the octavia-cli. I'm trying to run the command:

octavia --airbyte-url <http://localhost:8500> get connection <CONNECTION_ID>

but end up getting the following error:

TypeError: _from_openapi_data() missing 3 required positional arguments: 'schema_change', 'notify_schema_changes', and 'non_breaking_changes_preference'

This does not occur when trying to get a source or destination. Any help is greatly appreciated. Thanks!

Hey y’all! I’m writing a destination connector and I’m a little bit confused at what combination of sync modes are valid to support. I would ideally like to only support full refresh overwrite and incremental append, but it looks like this is not possible to specify, since the following spec implies support for full refresh append as well:

"supported_destination_sync_modes": ["overwrite", "append"],
  "supportsIncremental": true

Is my understanding correct?

Could you help me? I'm trying to connect two Oracle databases in Destinations and I'm getting the following error: The connection tests failed. Could not connect with provided configuration. ORA-00972: identifier is too long Has anyone ever experienced this?

Does anyone have experience using an http webhook as a source? I'm looking to ingest from a service that provides log events in the form of an (outbound) webhook, with no other API. I've looked at the connector catalog and I can imagine doing this using the SQS source (with my webhook hitting SNS HTTP or lambda + SNS) but I'd like to check for others' experiences, especially doing this with less middleware infrastructure.

Hey Airbyte team! I just got Airbyte OpenSource deployed to GKE kuberentes cluster, but for some reason I'm not seeing logs anywhere in Stack Driver or in the GCS bucket. Any ideas?

Hello all, I am currently seeing notifications error. Is it critical? Do I have to set up notifications? How do I fix this?

Hi everyone, I am totally new to airbyte and playing around with connectors. I currently have a bigquery source and keep getting this error whenever I try to start creating a destination. Failed to fetch schema. Please try again. Error: non-json response There is no log provided to enable digging deeper to establish the root cause.

is there an API to load a custom destination? right now I'm doing this in the settings > Destinations page but trying to automate this in an integration test

Edit: figured it out. So please ignore I am unable to configure clickhouse as a source. Here is what I did: Deploy clickhouse locally

docker run -d --name clickhouse-server -e CLICKHOUSE_DB=test \
    -e CLICKHOUSE_USER=airbyte -e CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 \
    -e CLICKHOUSE_PASSWORD=mypassword -p 9000:9000/tcp \
    -p 8123:8128/tcp clickhouse/clickhouse-server

Deploy airbyte locally by running:

docker-compose up

Configure clickkhouse source like in the screenshot however it will throw a "non-json response" error.

Hello! I was working on a new Destination connector for Timeplus. I created an Issue and a PR. Can somebody help me in the process of getting the connector to master? 🙂 Please let me know when I can expect a review 😉 It is my first Airbyte contribution so be patient with me thanku

Greetings, all. I’ve been attempting to run AirByte over the weekend but ran into some sync issues. Above is the log messages I’ve been getting from the AirByte Worker. There have been restarts on cron, the server, the worker, and the webapp. Does anyone have any pointers as to why this might be happening?

We are still having issues getting any connection to run on Amazon EKS. We think we have isolated the error to

Attempting to start pod = source-postgres-check-30-0-cbfqh

IS there someplace were the namespace is configured?

Hi Team! We have a sudden failure on Airbyte - during the night on Wednesday the Snowflake Connector failed and has been failing ever since. The credentials provided for it and AWS staging are tested separately. It failed middle of the night after some connections succeeded. There were no changes in infra, so it’s extra worrying. Could you please help point us in the right direction to fix this? 🙏 🥺 🙇‍♀️ Logs ⬇️

Hi team, i need some help understanding what middleware is used by airbyte to communicate between two connectors? Could anyone please help me out with this? Is there any documentation for this? I am using the open source verisosn of Airbyte

Hi Airbyte team! I've seen a few threads here and there about performance but nothing definitive. We would like to roll out airbyte, but are seeing some massive performance difference vs the current approach: Example dataset: full sync of 800MB / 6.5M records parquet in S3 -> Postgres Our current approach: load in pandas and push to postgres using https://github.com/d6t/d6tstack . Downside: high memory consumption due to reading data in pandas. We work around this by chunking. We're also looking at https://www.embulk.org as an alternative option to this custom code. But quite fast: full push to Postgres takes 15mins to read parquets, and only 5m30sec to push to postgres Tested the same dataset in airbyte: pull using S3 connector, record batch size set to 524288 -> push to postgres with full refresh takes 2h10mins. tested the same but push to Redshift with COPY strategy to see if that would improve, same performance I've read the optimization work is ongoing, and that 3-5m records / hour is within expected range. Is this still the case? Any way to further optimize? At this level of performance, the airbyte sync is too slow to be an option.

Hi, how can i run an airbyte connector locally without docker?

Hey, we're having a weird issue with airbyte

0.40.18

- bootloader keeps upgrading

mongodb-v2

connector to the latest version on each deployment (even though, we want to only increase sync workers for example). Which overloads our database when syncing a couple collections. Other connectors are fine. Is this expected behavior? Why? We're deploying via helm.

hey guys, is it possible to move from a docker compose hosted Airbyte on a VM to a K8S based deployment ( without losing connections, jobs, and sync schedules ), i am using an external Postgresql db, is that enough?

Hi, I am using facebook marketing connector and syncing ad creatives to s3. The sync mode is set to full refresh append, so ideally all of the ad creatives should have an entry for every day, but some of the adcreatives are missing in the latest pull. Can someone suggest a reason as to why this can happen.

Hi team! We had an issue yesterday upgrading the salesforce connector. • on connector version 1.0.2, everything runs as expected • on connector version 1.0.27, duplicates appear (around 30 for over 30k rows). It happens for different salesforce objects. We didn't change anything except the connector version. Sync mode is set to Full refresh | overwrite. Did I miss something? Should I open a github issue?

Hello there ! was wondering about submitting prs for some connectors I made, but struggling with the integration tests timing out because I have a lot of data in these sources, even if it works fine on airbyte platform. Was wonderiong how they were made when submitting connectors, since i won’t publish my own creds ? Do you create accounts on each source to run integration tests ?

Hey guys, we’re using helm to deploy airbyte open source on k8s and we’re trying to spin up and leverage external DBs for the webapp/UI. I see how to pass the SSL flag in the docs for a Docker deployment, but don’t see anything identical for kubernetes+helm. Anyone run into this before?

Unable to start server: sql schema version compatibility check failed: pq: no pg_hba.conf entry for host "10.13.34.11", user "airbyte", database "temporal", no encryption.

Hi everyone! I have some trouble with my PG <> Snowflake connection.

I share the log  --> Schema 'PUBLIC' already exists, but current role has no privileges on it. If this is unexpected and you cannot resolve this problem, contact your system administrator. ACCOUNTADMIN role may be required to manage the privileges on the object.

I set up a different schema but Airbyte insist on write in public. Btw, the role has privileges to public. Any ideas? Best, Ignacio.

Hello there! I am trying to add to my custom source (HTTP API source) support for different sync_modes but get kind of lost for where to configure this I tried per stream in my

configured_catalog.json

with following syntax

{
    "streams": [
        {
            "stream": {
                "name": "items",
                "supported_sync_modes": ["full_refresh", "incremental"],
                "default_cursor_field": ["id"],
                "source_defined_cursor": true,
                "json_schema": {...}
            },
            "sync_mode": "incremental",
            "destination_sync_mode": "overwrite",
            "cursor_field": ["id"],
            "primary_key": [["id"]]
        }]
}

This did not change anything if the sync options I try after that to add to my

spec.yaml

following

supported_sync_modes:
  - overwrite
  - append
  - append_dedup
connectionSpecification:

Non of that helped. How can I configure what supported_sync_modes are available for my source, that I can choose from the UI