Feature request: Not sure if it has already been requested (as I can't see it in slack); I'm running Airbyte using Helm on AKS and to make the pipeline a little more DevOps want an API to GET, POST, and DELETE connections (sources and destinations). That way when updating the Helm chart on the fly I can pull updated credentials from a key vault for any of the connections which are defined in templates. Love your work!

Any updates on this issue? https://github.com/airbytehq/airbyte/issues/6544

Feedback: When I am on the introduction documentation (https://docs.airbyte.io/) There's a paragraph what says :

If you want to schedule a 20-min call with our team to help you get set up, please select some time directly here.

But the link (https://calendly.com/nataliekwong/airbyte-onboarding) ends up on a page that reads

Natalie Kwong. This Calendly URL is not valid.

Airbyte team - thoughts on creating separate troubleshooting channels? Really appreciate all the participation there, but personally appreciate more specific channel specification to more easily search, identify, and raise issues. To begin, would think of #troubleshooting-setup, #troubleshooting-deployment, #troubleshooting-scaling

Hey Airbyte team, Any thoughts on when will OAuth connectors will come to major SaaS apps(Salesforce, Hubspot etc) to open source project just like there was it coming to Airbyte Cloud ? (I am still waiting list for Airbyte Cloud).

have logging defaults been changed? Seeing a lot less log entries, any way to turn those back on in config? Looks like the level has always been INFO, but im getting a lot less info 🙂

Any plans on supporting airbyte on the new Azure Container Apps service? https://techcommunity.microsoft.com/t5/apps-on-azure/introducing-azure-container-apps-a-serverless-container-service/ba-p/2867265

Hi Airbyte Folks, I had a quick question: Does your HubSpot integration pull in historical properties, historical events, and historical deal-stage change? For example if I am building an analytics app for HubSpot, I’d like to pull in a customer’s entire history of events and deal stage changes, up to the current date. And going forward I would want to update this history incrementally (as long as the API allows, otherwise it would be pulled from scratch each time). Thanks

Hi everyone, I have a question about the helm chart in the repo. Are you planning to publish it somewhere like https://artifacthub.io/? Or is it already out somewhere?

Not sure if this is suggested somewhere, but in my field, digital marketing, I'm sure there's a pareto distribution of many users using the same sources, with the same config. At the same time, many of these users will likely not be power users, but instead marketers and analysts lacking technical experience. For these sources, I think it's make sense to have a menu of "recipes" or "sensible defaults", which for example Google Ads, one recipe could mean something like: 1. OAuth login (on Airbyte Cloud) 2. Pulling Campaign Performance data with Campaign Name,Impressions, Impr. share,Clicks,Cost,Conversions,Conversion value etc 3. Pre-selects incremental stream with dedup history with pre-defined primary keys and cursor fields. 4. Pre-defined DBT transformations that renames fields from

segment.date

to

Date

,

metrics.cost_micros

to

Cost

and casts them to the proper type I can see how this would reduce the setup time for a source from 5+ hours for Google Ads (for an analyst like myself) to 5min This may not be the market you're primarily going after, but having sources like this would mean that using Airbyte would suddenly be within reach (from a UI standpoint) of the hundreds of thousands marketers our there who are getting ripped off by Funnel.io, Supermetrics etc (in Europe)

Requests for trello source connector: if the root of everything is boards i.e. “return all cards for a board” (same for users, lists etc.), then allow for a list of `board_id`’s in the source connector configuration. Also I'm not sure the actions stream pulls all historical actions or just the last # number of actions.

Hi there , does any know if there are any terraform operators to deploy connectors ?

Hi, is there anyway to get the transactions table into the Chargebee connector - I see it pulls through singer but not through the Python api client, is this a simple addition? https://docs.airbyte.io/integrations/sources/chargebee https://github.com/chargebee/chargebee-python/ https://github.com/singer-io/tap-chargebee

Hey Airbyte folks! I'm wondering, is there any plans to add a sync timeout in the nearish future? I found today one of our postgres->postgres connections was running for over 6 days, clearly hung on something, but I didn't know because I only have error slacks turned on. It would be nice to set a 2 hour timeout on these big syncs, so they don't get hung indefinitely.

Hi All, and @abhi, the community call was great! One suggestion would be to have the audience list visible.. and open the chat, so people can introduce themselves, and share what they are working on. That will promote the sense of community further. Thanks!

Hello! When are we getting clickhouse as a destination? 😄 is there a roadmap for this one?

Replied to the PR, but this did not solve any performance issues. I'm seeing the same performance in the latest version 0.1.5 - The previous Mongo connector and all other services I'm using to connect to Mongo do not have this issue, so I believe it is specific to the connector. https://github.com/airbytehq/airbyte/pull/7982

Hello! I was wondering if a Google My Business source is on the roadmap? Our marketing team is asking and I would like to give them some type of ballpark estimate

Hi, A couple questions for which I didn't find any answer here or in the doc : • In Deduped mode, if I delete raw and scd tables, will the following syncs fail or be any slower ? And what about tmp tables ? • When I have a long sync going on for several hours, I can see in the logs that the data from all tables is extracted before it starts writing in the destination. Does it mean that this data is temporarily stored in memory wherever Airbyte is hosted ? It sounds odd as I would expect airbyte to write it in the destination as soon as it's loaded, maybe in raw or tmp tables, to save resources. But I can see that these tables remain empty until the whole extraction phase is over Thanks

Hi, I am currently using Airbyte to move data from Postgres to Snowflake using CDC 🥳 I recently had an error that affected one of my syncs and after I fixed the issue it seemed to only process data from after the failure, rather than processing the failed data too. Is there a way to manually replay the missed data? P.S. wasn't too sure if this was the right channel so happy to post elsewhere if necessary

HI My team has deployed Airbyte on GCP VM, We are facing multiple issues with Airbyte UI when Source is connected and when trying to connect to destination as Big query , its going blank if more number of users increases.. Tried upgrading the VM , its still the same , memory utilization and CPU utilization is more than 80 % even if only 3 users .. Note : using e2-highmem-8 as the Compute Engine..

In your Roadmap , it mentions you are not ready for production yet, we are suggesting Airbyte to the client for production , Your statement seems contradictory , can you clarify on that.. Size of data is quite huge and table count is also on the higher side !

Hello! This request is in response to a constraint with the Okta connector - it’s quite mission critical as the current connector does not effectively allow us to account for user lifecycle. Problem statement The users stream hits the default ‘list users’ endpoint for Okta users the first run unfiltered. This endpoint only returns users not in a

"*DEPROVISIONED"*

status when not filtered. This corrupts the output, as de-provisioned users aren’t returned, and you will always overstate your active users in the dataset. The

"DEPROVISIONED"

status users need to be returned within the stream in order for a complete user picture out of okta. This heavily devalues the use of this connector for seeing a full user lifecycle. Proposed solution The

"DEPROVISIONED"

status users can be returned through a filtered call - I’m not sure the best way this can be resolved, but I believe if the stream can be filtered initially - it would return all de-provisioned users along with it. A couple ideas - not sure which one makes most sense: • The users stream makes a secondary call setting the filter to

status eq "DEPROVISIONED"

• An initial “Sync data from date” parameter is passed, facilitating an initial filter of:

lastUpdated gt "yyyy-MM-dd'T'HH:mm:ss.SSSZ"

(this may be most accessible to the existing stream) • A secondary users stream, called

deprovisioned_users

is copied from the initial with a fixed filter of

status eq "DEPROVISIONED"

Thank you for your consideration, Again this is extremely important to having us able to effectively account for user lifecycle out of the Okta tenant.

Is this your first time deploying Airbyte: No OS Version / Instance: EC2 t2.large Memory / Disk: 8g/30g Deployment: Docker Airbyte Version: 0.32.5-alpha Source name/version: pipedrive 0.1.8 Destination name/version: postgres 0.3.12 Description: Hi, I am running some DBT transformations after Airbyte syncs, but I see that it creates my tables in the default destination schema rather than the specified namespace of my connection. How to deal with this or in other words how to edit the dbt profile that must be created during the process ? Thanks

Hi all, I'm using the Google Analytics Connector. I have 1 question, maybe it is already possible. I want to use a dynamic start_date, but i can't get it working. Can anyone tell me if it's already possible or put it on a wishlist?

For when everyone is back - is there a way I can hack in another stream without resetting an entire connection?

Heya, where can i find the airbyte logo SVG to use in my architecture diagram?

Hi everyone !  Using Kubernetes deployment and having increased the number of worker, I noticed that, depending on the node where workers were started, performances could be affected.  For instance, I have currently 4 workers replicas distributed on 3 nodes (

node A : 1 worker

,

node B : 1 worker

and

node C : 2 workers

). When syncing, if my

source-worker

and

destination-worker

are correctly distributed, I’m able to process about 25M rows in one hour (with a well distributed CPU load). However, if by any chance

source-worker

and

destination-worker

are both started in

node C

, then the node’s CPU goes up to 190% (against 10% and 10% for the two others) and time processing is much more slower as I’m only able to process about 15M rows within an hour.  Not sure if it's an actual request as I don't know if there is an existing strategy to avoid this situation but is there a way to force the parallelization of workers on different nodes to maximize performance ?

Hi everyone. I am testing the connector of Google Workspace Admin Reports 0.1.6. Since we have billions of audit logs, it takes a long time for the first time job to fetch all the records. Moreover, it seems that it has to fetch all the records before storing the temporary or final destination, say about a BigQuery table. So, we would like to shorten the elapsed time to make sure data in destination. What do you think? • It would be nice to specify the start date to fetch records. The current version tries to fetch all logs for the maximum retention time. • It would be also nice to iterate to fetch and store records so that we can see stored ones in destination as soon as possible. Aside from that, it might be risky to have a quite long running process in a container. We potentially has to re-run the container to fetch data, if it is failed. • It would be also nice to select a strategy about how to do backfill the past from the latest logs. In my opinion, newer data tend to be more important. If we collect data from newer to older by doing backfilling, it would be awesome. https://support.google.com/a/answer/7061566

I think the Airbyte docs would benefit from increasing the contrast on hyperlinked text. Right they they look basically the same as other text, making it hard to spot them: