AirbyteLocal task execution commands?

Complete rookie to airbyte here, I recently wanted to create a connector for airbyte to be able to get data from a bunch of API endpoints and I'm stuck in the first step of this process. Which option do you guys think I should pick from the Generator, the HTTP API Source or the configuration based source?

Airbyte team Please help me with this PR https://github.com/airbytehq/airbyte/pull/24887. Need a review

Airbyte team, Wanted to know how can we scale the airbyte?

Hey Team, We are trying to achieve a stream between postgres to snowflake, whenever we start the sync airbyte tries to fully refresh the source table which is taking lot of time, is there any way to avoid that, can't we configure it with a column to use which can let it know which row has been copied ?

Hello folks, I tried adding a connector request on this page, it is not working. It says it will take a few seconds to show up but it does not

This message was deleted.

This message was deleted.

Anyone know about this issue ?

How can you log in and have your bearer token that you get automatically entered, I need to make sure it takes into account the expiration date of the token how do you do that the token is open for 300 seconds before it is no longer valid

Hi 👋 We are currently have Airbyte version 0.40.32 deployed on EKS using kustomize. Is there a migration path for moving to helm in order to avail of the new airbyte versions? We have RDS setup for external database and S3 for logs, so I am wondered if we setup a new helm install of airbyte and point it at the existing airbyte RDS & S3 infra will it work?

Hello Airbyte Community airbyte I have a question regarding the Facebook Marketing App Review for Advanced Access I’ve been following the official documentation (Facebook Marketing | Airbyte Documentation) and this post (How to get your Facebook App reviewed to use with Facebook Marketing connector 1) to get my Facebook Marketing app verified for advanced access, in order to avoid rate-limit throttling. During the “App Review” process, it’s required to provide access to the working app, including credentials, for a Facebook employee to review. In our case, the “app” is just a Dockerized Airbyte instance running on an EC2 behind a VPN (and SSH). The thread suggested using the demo site (https://demo.airbyte.io/) for demonstration purposes, but I also don’t understand how that’d work. What’s the best way to grant Facebook access & get the app verified in this situation? Is it necessary to deploy a new public Airbyte instance with a unique username/password just for the Facebook team to test this standard use-case? I’m concerned I might be overcomplicating the process or misunderstanding its purpose, I believe what I have here really is an absolute standard use-case. I’d greatly appreciate any advice or success stories from anyone who has successfully navigated this process. Thanks in advance!

hello all, hope you are doing well. I set up airbyte OSS on a EC2 linux instance. we are using a VPN to connect to the machine that hosts airbyte. Before airbyte stats up, it requests credentials, we currently have the default creds. How can we update them?

Hello folks, I'm trying to sync data from SFTP here... that's not working though... Is this your first time deploying Airbyte: No Memory / Disk: 8Gb / 16GB Deployment: EKS Airbyte Version: 0.41.0 Source name/version: SFTP Bulk Destination name/version: S3 CSV/S3 Parquet Source Connector Version: 0.1.1 Step: Running SFTP Bulk connection Description: The extracted data has only "_airbyte_ab_id","_airbyte_emitted_at" columns I'm trying to extract all the csv files available in a SFTP folder... like /smartfiles/outbound. So we may have 50 files with distinct names, all csv files. I could create and test the source... with no issues... I could create the connection... When I run the connection, the destination files are generated without the source file data. I have only data from "_airbyte_ab_id","_airbyte_emitted_at" columns. I tried to find something here in this channel before sending this post.

How can I setup airbyte to make a simple GET API Request to an API which takes in an api key?

Hello folks, I have an issue with Postgres data extraction to BigQuery using CDC. Deployment: Helm Chart Airbyte Version: 0.40.32 Source name/version: Postgres / 1.0.42 Destination name/version: BigQuery (denormalized typed struct) / 1.2.11 The problem is we have a random behavior happens randomly and we couldn’t manage to find a valid reason for it. We have configured a connections using

postgres

and

bigquery

to extract our incremental data but randomly it decides to do a full-resync. • CDC replication • Incremental • Daily Sync every 10 minute triggered through API (Airflow Operator)

Hi airbyte team, I created a custom connector. So , when syncing api data it have lot of records. I used airbyte stream slicer, and incremental options. But stream slicer not increments dates because api response return an array. Following array object have IVDATE attribute. So how to set cursor_field to IVDATE value. I attached manifest.yaml file, AINVOICES.json and sample data set

If my Airbyte connections stopped syncing on a previous date for some reason, how can I troubleshoot to find out the reason? Which docker images logs should I be looking into?

Hey all! We are getting started with Octavia for managing Airbyte configuration as code and I wanted to know if there is anyway of storing state files remotely (e.g. in S3), as you would do with a remote backend on Terraform. Having to git our state files does not seem like an interesting approach

HI All, I'm running airbytes on EKS via helm. With new helm chart version 0.45.5, I get the following error when I try to upgrade -

Failed to pull image "airbyte/connector-atelier-server:0.42.1": rpc error: code = Unknown desc = Error response from daemon: manifest for airbyte/connector-atelier-server:0.42.1 not found: manifest unknown: manifest unknown

Anyone seeing this issue ?

Hi Team, we have started free trail today and 400 credits is shown only in one workspace?

Hello - what are the best practices for deploying dbt, airbyte, and prefect in Google Cloud Platform? Looking for something more up to date and robust than what I’ve found Googling so far.

What are the steps to fully re-install on windows? My playground is no longer booting and I'm willing to start over.

hello everyone, I just want to enquire one thing i.e. where does airbyte write the state of the tables in incremental deduped + history

what if state is not showing in postgress db but the processes are going on in background

can anyoune help me to sort this out?

Hello everyone, I have a question. When moving data from MySQL to Postgre airbyte automatically hash the column that contains emails. Is there a way to mitigate this? I am using the open source version deployed on GCP as described in the documentation. Many thanks

I need help setting up slack notification, I had generated incoming webhook token, which is valid (tested with curl). However, I get error when pasted the generated url into the field and click test button. I posted the details here https://discuss.airbyte.io/t/slack-webhook-failed-with-message-webhook-test-failed-please-verify-that-the-webhook-url-is-valid/4295 My Airbyte version is 0.43.0 The error is Webhook test failed. Please verify that the webhook URL is valid.”

Hi everyone. I've been wrestling with a series of Octavia CLI issues. I've submitted github issues, posted on discuss.airbyte.io, and posted on #C02RRUG9CP5 over the last month. Thus far I've received no responses, hoping this channel might be able to help. 1.

octavia import connection foo

fails Github issue: https://github.com/airbytehq/airbyte/issues/23953 Airbyte discussion topic: https://discuss.airbyte.io/t/octavia-get-import-connection-is-completely-broken/4199 Octavia will fail outright if you try to import/get a connection by name (even if it doesn't exist). Importing by UUID works, which is the workaround I've put in place. 2.

octavia apply

can't send certain fields that

octavia import

creates Github issue: https://github.com/airbytehq/airbyte/issues/19768 Airbyte discussion topic: https://discuss.airbyte.io/t/unable-to-apply-airbyte-configuration-using-octavia-cli/4138/2 There are three fields (

geography

,

notify_schema_changes

, and

non_breaking_changes_preference

) that are included when you do an import, but then fail if you try to apply them back. You can manually remove these fields and rely on the default values, but then apply will think there are changes to be applied every single time. 3. Octavia doesn't work with multi-line variables (e.g. secret keys) https://discuss.airbyte.io/t/unable-to-specify-snowflake-private-key-as-secret-through-octavia-cli/4139 Because Octavia relies on docker environment files for passing information into the container, it seems impossible to pass multi-line secrets, most notably Snowflake private keys. Is there any way around this short of creating a custom container with a different

ENTRYPOINT

? It would be nice if you could pass the secrets configuration via a more robust method (such as a YAML/JSON file).

Hello team, I have created custom connector on dev instance and pushed the docker images to aws ECR and pulled in production instance. I can see my docker image from ECR repo in prod instance as well. While adding the new connector on interface its giving me below error, any help would be appreciate !