Hi guys! I'm new to Airbyte and I would like to understand if it's right for our data pipeline. Currently we have different systems that push an HTTP request to our integration tools based on an event (ex: an order is created). Can we just set Airbyte to receive HTTP request? Is it feasible to have something like a queue, so that we wouldn't loose any ingoing requests? Thank you so much!

Hello! I’m new to Airbyte. I’m using the

HTTP Request

source but I’d like to have dynamic query parameters on it. Is that possible? For example, my url has

start_date

and

end_date

. I’m executing it once per day with

start_date

= d - 1 and

end_date

= current day. How can I have that in Airbyte?

Hello there, Quick question on local deployments. I've got the airbyte docker stack up and running but it appears to require airbyte account registration before accessing any features. Is this a hard requirement? If so, this would prevent my customer from using airbyte, so I hope there is an alternative. If it's not a hard requirement, how do I go about configuring the stack for an alternative setup? I don't see it mentioned in the docs anywhere I could find. Thanks!

Quick question! Can Airbyte see views from the underlying? Ours is Amazon RDS Postgres

Hi everyone, Quick introduction,  I am a developer currently evaluating Airbyte as an alternative to Fivetran in a personal project of migrating an in-house custom ETL pipeline to an updated / more managed approach. As a test case, I have developed a Lightspeed Retail HTTP connector to extract data from different Lightspeed shops. I am fairly new to data engineering and looking to get insights on multi-tenant ingestion, partitioning and best practices in an Airbyte context. I am aware of the different tenancy models, just unsure of which approach is best in the current context. Q: How should I approach data organisation to identify records to their tenants. In Lightspeed's response / records, there is no unique identifier that would allow me to identify the tenant from the raw data in further stages and eventually into normalization. My connector utilizes an AccountID parameter that would be a good fit to partition data. However, I currently don't see a way to pass such data to the records / streams when syncing. I am aware that my approach might be wrong. I would love to understand the correct approach before jumping into transformations with DBT. Love the project and looking forward to having the opportunity to contribute!

Hi everyone, I'm new using Airbyte and I want to extract data from Google Sheets but when I use the service account from Google and share the sheet with that account it's appear a warning because the company not permits to share the sheet with the external account. How can I solve this? When I connect with my personal account from google, the sharing works well.

This was asked by someone in one of the channels; “How can we use Airbyte with Authentication and secure it with SSL” I’m sure this is being worked upon in a future release but till that time I’ve created a solution using nginx, let’s encrypt and htpwd. My solution was build on OCI using 2 instances, the VM hosting Airbyte runs with a Private IP and the VM with Public IP runs nginx which acts as a reverse proxy to the private instance running Airbyte. Hope to blog about it soon. +1 if you would like to see this solution and get me motivated 🙂

Hey all! Followed the AWS EC2 instructions to get Airbyte running and run into the following issue when running

docker-compose up -d

.

ERROR: Couldn't connect to Docker daemon at <http+docker://localhost> - is it running?

If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

I spotted from searching slack that a previous user had forgotten to run

sudo usermod -aG docker $USER

but I’ve done that and it still doesn’t work. Anyone come across this before?

I am someone evaluating the product for data replication between heterogenous databases. My question: 1. If the primary keys are not signed as primary key in the DDL, so I cannot use incremental replication is it true and the rule "the primary keys in source should well be defined"? for replication method "CDC"

Hi everyone, I'm new using Airbyte and I want to extract data from Pipedrive to work on the deals, pipelines and stages tables but these tables are empty after the import (which was not the case with Fivetran import). I'm wondering if anyone of you has already worked with pipedrive and has some tips for me please.

Hi, is it possible to connect a private git repository (custom dbt transformation) using ssh keys? if not now, is it planned?

Hi Everyone I'm curious is it possible to change the frequency of replication after 1st job run without breaking anything or any form of data issue? As an instance for my 1st configuration and setup I did a 24hrs frequency for all my replication to monitor and see how Airbyte performs with our data , however with intention to change some tables with incremental append to frequency less than 24hrs to reduce latency after the first week of successful replication. I will love to get feedbacks so as to be able to communicate with the management to be aware of data latency as well as properly manage them. There was no mention of changing frequency possibility in the doc, so I want to be check if it's a one of setting or the frequency can be changed after Look forward your response. Thank you

since the airbyte hosted solution (Cloud pricing tier) is still in progress, would someone be able to give an idea of the most common cloud people are using to deploy production airbyte?

also, im scanning the docs, but may be looking in the wrong place. how does Airbyte handle failures for both ingestion and output, and are there any retry mechanisms in place? is there some notion of atomic operations? can a sync fail partially, fully, etc?

Hi, looking at the roadmap it looks like Airbyte Cloud beta will launch soon(ish). Will Cloud beta support integration with Airflow?

heduler/logs} - Waiting for database to become available... airbyte-scheduler  | 2021-08-17 213549 INFO i.a.d.i.j.JobsDatabaseInstance(lambda$static$2):45 - {workspace_app_root=/tmp/workspace/scheduler/logs} - Testing if jobs database is ready... airbyte-scheduler  | 2021-08-17 213549 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):60 - {workspace_app_root=/tmp/workspace/scheduler/logs} - Database is not ready yet. Please wait a moment, it might still be initializing... airbyte-server   | 2021-08-17 213550 WARN i.a.c.p.FileSystemConfigPersistence(createWithValidation):67 - {workspace_app_root=/tmp/workspace/server/logs} - Config volume is not ready yet (waiting time: 24 s) airbyte-server   | 2021-08-17 213553 WARN i.a.c.p.FileSystemConfigPersistence(createWithValidation):67 - {workspace_app_root=/tmp/workspace/server/logs} - Config volume is not ready yet (waiting time: 27 s)

how to change the port number 8000 to another port number? Which Config file(s) I have to chage to starting airbyte on different port

when using CDC, do i still need to select

incremental dedupe

? if it’s reading from the logs, wont it just modify the table in the destination accordingly? and thus remove the need for the

scd

table as well as the final table

Hello, i'm trying to do a migration from postgres to mysql for demo purposes but i can't use incremental sync mode, i have an error, does this mean it's not currently supported or is this a bug ?

Hi all, Kindly help me to start the airbyte in Ubuntu server.

Hi Everyone, I am trying to use airflow as my airbyte etl orchestrator but I am having challenges with it. Things I have done: -I have installed airflow and airbyte in docker containers and they are running and I can access them via their UI -I have setup source to destination connection in airbyte. -I have installed the airbyte operator for airflow using: pip install apache-airflow-providers-airbyte -I have also configured the connection in airflow and put my .py file in the Dags folder using the guide here:https://github.com/airbytehq/airbyte/tree/master/resources/examples/airflow The challenge I am having is that my dag shows import errors and the error was : ModuleNotFoundError: No module named 'airflow.providers.airbyte' How can I fix this?

is there a recommendation on how many tables per connector? i.e. if i have 20 tables of varying sizes from one source. should i create one connector with all tables or break it into seperate connectors from the one source (and run them at different times). additiionally, there would be a few sources with this situation. so many connectors with few tables? few connectors with all the tables from each source?

Hi

I would like ask if it is possible to use airbyte for this specific scenario if not what alternative can achieve this

Hi team, I am deploying my first Airbyte istance on an azure VM. I am stuck at last step : Create ssh tunnel for port 8000 ==> got error :

bind [127.0.0.1]:8000: Address already in use
channel_setup_fwd_listener_tcpip: cannot listen to port: 8000
Could not request local forwarding.

all Airbyte containers are running but I cant connect to ui Any thought on what I am doing wrong ?

hey everyone! I think i've just managed to deploy airbyte on a GCP VM connecting to Facebook with BigQuery as a destination. I am completely new to all of this. Just wondering if I need to do anything once I've finished e.g. with my connection in the mac terminal or on the VM I've created on GCP or do I just leave them?

also how long does a first sync of facebook take?

Guys, I have an extremely large MSSQL database with thousands of tables, I have been waiting for a day, but Airbyte couldn't load the schema. Is there any workaround for this? Thank you

Hey guys, has anyone made a screencast for the Facebook App Review to get past their rate limit?

Hi everyone! My team has deployed Airbyte on a GCP VM. Is there any way to export the syncs we have set up in the VM? It seems like if the VM goes away, or we switch to deploying using another method, all of our existing connections go away. Is there a way we can persist information about the sources, destinations, and connections we have set up on the VM?