postgresql to bigquery. tables with jsonb column. tried both bigquery and bigquery typed struct connector and in both instances the jsonb column is serialized as a string in the destination, do I have to do anything special here to get jsonb to go to a struct at destination?

Also strangely, the bigquery typed struct connector doesn't have Incremental and Dedup ?

Hello everyone. I’m new to using airbyte and see a lot of promise in it. I have the following two questions: 1. How does airbyte scale when it comes to ingesting multiple api andpoints? Is each connector an isolated compute environment? 2. In the https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine documentation, I noticed that the docker compose has the database hosted on the same virtual machine as the airbyte server. Can airbyte be configured with external databases hosted on another vm or something like cloud sql for resiliency? 3. Is it possible to programmatically add custom connectors when deploying airbyte? From this page it looks like you can only add new connectors through the UI: https://docs.airbyte.com/integrations/custom-connectors

Hi,  I’m trying to sync a mongoDB to Redshift but I always get this error 

ERROR i.a.w.DefaultReplicationWorker(run):141 - Sync worker failed.

Could you help?

Hey everyone! I'm a new user and I'm trying to sync a MS SQL to S3 but the server is very restrictive, the remote timeout is set to 10min (I cannot change it) and the process times out before the 10min mark. Is there a way to "force" the sync to stop after X rows or a period of time so the next schedule it will take the cursor?

can I connect to API that using OAuth 1.0? airbyte code

Hello everyone I would like to know how to backup all work on airbyte. Example 1. I have 2 jobs 2. I want to backup 2 job (All)

Hi folks! 🔥 I am looking for documentation about Security, I only can find this https://docs.airbyte.com/project-overview/roadmap#3.-parity-with-enterprise-features-data-quality[…]acy-compliance-customer-data-consolidation-featu but look a little As of today, Airbyte supports? • SSO • integration with Okta • SSL • encryption • GDPR • OAuth 1 authentication • Logging Capabilities • Is it able to integrate with gitlab/CICD

Is it possible to use Google BigQuery as a source in M1 macs right now? I saw in the issue here that some connectors do not work for M1

Hi team 👋🏻, I’m new to Airbyte and currently working on setting up a custom Postgres DB to store the internal state and metadata for Airbyte based on this guide. Could you provide me with a recommendation or estimation for how much size is usually needed for the db to store these files or what it depends on?

Hello, I have these requirements: Integrability • Has to have out-of-the-box connectors (main ones: REST APIs, GraphQL, Lambda) • Nice to have able to handle push vs pull events (webhooks, kafka, etc.) Airbyte complies with it?

Hi, does Airbyte support multiple cursor fields for Postgres? My table does not use sequential IDs and it uses UUID. Also there are 2-3 timestamp fields (created_at, updated_at and sometimes deleted_at). Will Airbyte work?

Hi, I’m looking for some docs on how to use oauth with airbyte. We intend to send links to our clients where they do oauth-login, and are redirected back to our website at the end of the flow. I see Airbyte’s API exposes a few endpoints to manage oauth, is there support for an end-to-end flow where the refresh token is stored safely and automatically, or do we need to build our own credential manager to handle that?

Hi, what do you guys think about connectors that call other connectors through airbyte's own api? Sort of a rather dumb implementation to mimic a simple workflow?

Good morning! I am evaluating Airbyte, Could you help me with this? Airbyte complies with it?👋 Detectability/Reliability/Recoverability • Should be able to handle various error states (retries, alternate paths, etc) • Guaranteed delivery, data will get to its target eventually, ability to pick up where it left off • Ability to get alerted / notified clearly or visually when an error occurs Developability • Has to take a reasonable amount of time to migrate existing connectors and implement new connectors • Must be able to migrate over existing connectors (Jira, SQL, MYSQL, Snowflake, S3) without losing any necessary functionality • Must be easy to maintain and fix errors (not too much hidden away from engineers) • Must require a minimal amount of writing custom code • Must not be substantially more work to implement things than what it takes today • Must provide the ability to easily track/audit/troubleshoot how data flows from source(s) to target(s) end-to-end within the pipeline • Must be able to share and collaborate with others on a shared set of jobs/processors, etc (realtime collab nice-to-have, ability to publish to a shared registry/database/repository) • Shouldn't require a lot of debugging to maintain over time • Shouldn't be too easy to build it out the wrong way

Hello there, I was wondering if the "Sync Frequency" in Connection settings can be configured to have custom values. We have a Use Case where we need to sync data weekly and that option is not available.

I have another question related to "Sync Mode" for tables from the source schema. After I synced once or twice manually, when I examined "Sync Mode" options, I found that certain tables have only "Full Refresh" as the option, while certain other tables (from the same source) have "Full Refresh" as well as "Incremental" as the options. Why is the "Incremental" option not available for certain tables?

I'm trying to make a connection between a MS SQL database and an S3 bucket. Both source and destination test perfect but when I try to make the connection it errors out. I think this is because there are too many tables in the source for it to parse the schema in the allowed time and so it is timing out... Any way around this?

Is there a way to clone a specific source or destination so that one can change a single setting without having to recreate the whole source or destination?

hi, everyone! i’m running Airbyte locally, connecting to a PostgreSQL database on AWS RDS and it worked now, i’m trying to run it using AWS Elastic Beanstalk using the same YAML and .env files, but without success….could somebody give me some tips?

When working with large data sets incremental loads will always be better than full table loads correct? Are there any downsides? And why not have that selected by default when it’s available for certain fields?

Hey guys, is there any documentation about Airbyte's scheduling behaviour? Let's say we have 10 jobs, scheduled to run every 15 minutes. Concurrency is set to 5 parallel jobs at max. Which 5 of the 10 jobs will start first? And what happens if there are 5 other jobs scheduled to run every 5 minutes? I am just making stuff up here... try to understand the scheduling behaviour, as it is not that granular with regards to configuration-

Hey everyone, I am trying to use Airbyte API and when I try using postman I get this error*<title>405 Not Allowed</title>*

My questions: (I have deployed Airbyte opensource on an EC2 instance) 1. From where do I get the token to post these requests?

Hi Folks, where can track upcoming releases ?

Hi , i am trying to run airbyte locally within my cluster and trying to setup source/destination/schedule incremental refresh thru API. Is there any documentation link with sample API links to establish connection between source salesforce/hubspot and destination redshift. I have gone thru https://petstore.swagger.io/?url=https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-api/src/main/openapi/config.yaml#/ but additional information is appreciated !

Hey, guys! Is it possible to extract data from GA Multi-Channel Funnels API V3 through the GA Universal Analytics connector?  I tried, but it couldn't find the schema and it appears that it uses a different API endpoint. Is a separate GA connector needed?

I see Intercom to Redshift sync is doing a ton of stuff, saying it’s already read 791k records but I only see one file in S3 for our companies, and when I open the file it has 295k companies which matches what we have on Intercom. What exactly is the sync doing? There are 296k log messages so I’ll just post the latest.

Hi, I'm triyng to setup a connection to my FTP server using the source Files. I check the docs and support SFTP. Could that cause me an error? I try to config the source but I got.

TimeoutError(110, 'Connection timed out') Traceback (most recent call last): File "/airbyte/integration_code/source_file/source.py"

If I set to sync from a certain date on and an update/deletion request comes in for a piece of data that was not synced does it just ignore it? Or will it insert the data and mark it as deleted or something?