Hello! We've written a custom incremental HTTP source and are using the S3 destination. We are running in incremental append mode. The current behavior is that, when we run a sync, all of the new records from that sync are written into a single new file in S3. Is there some way to instead write each record into its own file in S3? Thanks!

Hello! How can I set

table_ab2.sql

? I want that my Airbyte pipeline changes the date type from BigInt to Int, I access to the model and the sql, and modified but I want that it would be the default for the next normalization without using dbt and sql tool, only changing the sql. I'm using S3 source to Redshift.

Morning I am currently getting started with Airbyte and trying to get it to work with hubspot. When trying to get the connection setup it just sits on testing connection for ever and a day. I was wondering if this was expected behaviour or something that is odd about my setup. IT does seem like it can authenticate (as there is some logs of the connection on the hubspot side) but before I do any more digging I was wondering if anyone had seen this?

Hey all, I'm a total noobie to airbyte and I have really simple orchestration/ingestion needs for a client, any chance I can run airbyte on cloud run and scale to 0 or similar? I'd love to have it run for a few hours a day (and I'm happy to ping it at a specific time to wake up). Is this something that is supported currently?

Hello all, I am trying to use airbyte for a scenario where data needs to be anonymized before being pushed to S3 destination. I am planning to use faker python package to do the anonymization (first cut) . I looked at the airbyte documentation, but I don't find support for adding custom python code as a transformation step. We are kind of at this point constrained to use a Data Warehouse and dbt to run any custom transformations. Is there a plan to support custom , preferably python based , non-dbt (so that no requirement to have a warehouse setup) transformations in near future. Thanks

Hi everyone, am loading a table into snowflake, the table is about 1.4Gb and it has 160M records, and is taking about 15 hours, how can i optimized the process in airbyte?

Hi, I am having trouble replicating MongoDB to BigQuery. The nested JSON objects are omitted and the column contains NULL values across all rows.

Hi! I have tried searching the documentation but couldn't find an answer - what is the most optimal way to sync a lot of tables from a single database? Should I just setup a single connection, multiple connections based on table size and table priorities (e.g. have all the small tables in a single connection, have an important table separately) or have a connection for each table? Also, we are planning on orchestrating through Airflow so I could document which connection is which through there but what is the recommended best practice to documment connections in general?

Hi. Does anyone have a link to a public git with an example of DBT transformation for AirByte? Never worked with DBT before but I guess I can make it happen when working off an example

Hey, all. Thanks for the help in advance! Just joined a team that is using airbyte to replicate postgres data to snowflake. We are running into an issue with a BIGINT field in postgres that is being replicated as a FLOAT in snowflake. It’s an 18 digit number, and since it’s being replicated as a FLOAT, it is impossible to do precise math or reliably convert to an int in snowflake. (see https://community.snowflake.com/s/article/To-Float-or-Not-to-Float-Choosing-Correct-Numeric-Data-Type-in-Snowflake). Do I have any options to explicitly map this as INT?

Hi, does anyone know how I can point to a private registry when adding a connector? (deployment: K8s/ helm chart -- 0.33.7-alpha)

Is there a way I can setup my connection in such a way that a row doesn't get removed in the destination table even if the corresponding row in the source table has been removed? Our source tables are cleaned on regular basis which makes it hard to have persistent data.

Hi all! Is possible to set the cursor manually? My cursor is timestamp. I want to re-sync only one day. I'm using source Oracle DB to Redshift.

Hi all 👋 I am messing around with basic normalization and am running into some issues. I exported the dbt project airbyte created for my connector and threw it in a git repo so I could test custom transformations. However without changing any of the models yet I cant seem to get the connector to work with the same exact dbt files airbyte generated. This is the error I am getting

dbt found 1 package(s) specified in packages.yml, but only 0 package(s) installed in /tmp/dbt_modules. Run "dbt deps" to install package dependencies.

Hi! I saw 1 credit was equal to 1 hour of compute for APIs, but 1h on which instance size? I saw you recommended to use t2.large for production, does that mean that 1 credit = 1 hour on a t2.large?

hi all, can someone please explain what does this 

configured_catalog.json

does. we pass it while reading the data using  

python main.py read --config sample_files/config.json --catalog sample_files/configured_catalog.json

Hi everyone, does anyone know of any blogs or guides to installing locally on windows? Just getting to grips with it, but it seems Docker doesn't play nice with Windows.

Hi everyone, I want to be notified on the line when sync fails. Suggest or tell me a link where I can study further.

Hello! I'm completely new to Nginx, trying to deploy airbyte over http (at first) It works perfectly with configuration like this events {} http { server { listen 80 default_server; location / { auth_basic "airbyte"; auth_basic_user_file /etc/nginx/.htpasswd; proxy_pass http://127.0.0.1:8000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_cache_bypass $http_upgrade; } } } but when I try to change the URI to something like location /airbyte, it throws such an error. Could you help me to figure out, what's the problem is?

Hi, I'm working on replacing Stitch with Airbyte (self-hosted) in a Postgres -> Replicator -> Snowflake setup. Some data from Postgres has previously been deleted and retained in Snowflake, to keep the Postgres instance size down. Because of this, it would be helpful if I could use Airbyte with the existing Snowflake target DB instead of starting with a clean slate with Airbyte. I've searched online and in Slack for notes about a Stitch -> Airbyte changeover but have not been able to find any notes on whether others have been successful with this. Has anyone been able to do this or have thoughts on why this would be a bad idea, as opposed to starting over with Airbyte (e.g., the metadata that Stitch and Airbyte store in tables may conflict)? Thanks!

Hi everyone, its me again haha, quick question, is it possible to change the batch size in airbyte? and if so how can i change it?

Hi everyone, new to airbyte, I saw in the roadmap that a declarative CLI is in the works. Is there an existing, non declarative CLI? I tried searching the docs but couldn't find anything on it. If there is can anyone point me to where I can learn about it and get started?

hey team, why is

internal staging

recommended for the loading method (snowflake destination). is it simply because its the easiest to get going with? just want to confirm that S3 staging is definitely more efficient?

Hi My first time to setting up the Airbyte with BigQuery as the source and Postgresql as the destination. Instead of copying all rows in a table. Any way I can just copy, for example, certain rows inside table based on a filter like store_id = 1200?

We are self hosting airbyte and I am try to understand the current status of workspaces. Can you create them through the UI? If I create one through the API, can I see it in the UI?

Does anyone know how I can specify (in the helm chart) image URIs for

JOB_POD_SOCAT_IMAGE

,

JOB_POD_BUSYBOX_IMAGE

,

JOB_POD_CURL_IMAGE

? (deployment: K8s/ helm chart -- 0.33.11-alpha)

Hi All - I'm not sure if this is an appropriate channel to reach out but we're looking for an Airbyte specialist to help with a project. If this isn't the correct channel kindly let me know and my apologies if that's the case :)

Hi, we used k8s to deploy airbyte, and added databases, airbyte lunched several jobs in pods. How can we deal with these finished jobs, manual housekeeping by period? Or any other solutions, thanks. source-oracle-sync-1-0-gfcxc 0/4 Completed 0 56m source-oracle-sync-2-0-bmeyg 0/4 Completed 0 39m source-oracle-sync-3ca70f7d-43b6-4649-80e8-fae6a10829f2-0-hbfhe 0/4 Completed 0 68m source-oracle-sync-3ead570e-e275-4744-8c8e-78a7265077fb-0-oeuaj 0/4 Completed 0 8m54s source-oracle-sync-4-0-kfpnw 0/4 Completed 0 36m source-oracle-sync-478294bf-dcb7-4882-90ca-f44bfa74652d-0-pijdw 0/4 Completed 0 61m source-oracle-sync-5b019e80-2600-4845-a1f2-8fef58c3e238-0-zyewg 0/4 Completed 0 29m source-oracle-sync-6-0-dyncx 0/4 Completed 0 25m source-oracle-sync-7-0-kzdho 0/4 Completed 0 23m source-oracle-sync-75f53a4c-1e14-4ecd-9720-5621c5b9fef8-0-wmnil 0/4 Completed 0 20m source-oracle-sync-7f437202-e299-46cb-b951-6deaf47ef331-0-gqunx 0/4 Completed 0 77m source-oracle-sync-8-0-xfrwx 0/4 Completed 0 16m source-oracle-sync-8f421254-0ef6-4327-ae7a-4131166becff-0-zimsm 0/4 Completed 0 58m source-oracle-sync-d5fb1b3c-3cb9-49b1-ace5-1e9584c44688-0-jrztn 0/4 Completed 0 30m source-oracle-sync-efb165c8-3c9d-491e-be2b-2ae15276d268-0-uhqol 0/4 Completed 0 85s source-oracle-sync-f2a4bcad-3f70-442d-a416-06479eca20bb-0-onihl 0/4 Completed 0 27m source-oracle-sync-f4d71dbb-ca38-4e71-bb9e-73f1e26c992b-0-jcpga 0/4 Completed 0 68m tion-postgres-sync-2b66216e-7a84-454e-8d63-26b5ad047373-0-gfaxd 0/4 Completed 0 18m tion-postgres-sync-6089227a-34a9-4755-9828-a8b8e73d7c6b-0-syztc 0/4 Completed 0 66m tion-postgres-sync-8d83bdf6-6a66-4ad7-98de-41a73cd1b57c-0-qfptj 0/4 Completed 0 28m urce-postgres-sync-30cdc9cc-6d41-4ac6-9172-940484c75b8b-0-jcrja 0/4 Completed 0 6m34s urce-postgres-sync-4102fce2-378f-4aff-8ea8-7365788e3418-0-njyoj 0/4 Completed 0 72m urce-postgres-sync-4f151ac0-f0a5-4ffd-bced-534b7077ff46-0-rhjqh 0/4 Completed 0 7m28s urce-postgres-sync-59d205d9-934f-4769-95b3-1fd0204c8d2b-0-yszdg 0/4 Completed 0 5m41s

I'm considering Airbyte for a DWH ETL setup (Mongo -> BigQuery). Due to legal requirements I have to obfuscate some user IDs before the data hits the DWH. What are my options with Airbyte to do this transformation in-flight?

Hello Team. I'm just getting started with Airbyte. I have a couple of questions: 1. Where is the code for Airbyte ETL stored? Is it possible to export it and put it on GitHub? 2. I want to download files from a website through HTTP GET Requests. It would be in ZIP format and would need to be extracted and the file name would be dynamic - date for everyday. Does Airbyte support this use case? I would also like to provided a referrer header in the GET request. Is this easily doable with Airbyte?