Hi Ari, actually we decided develop a reverse etl library to load information in our transactional system, the principal issue is load information into a data interface pre-concept model (exiting tables)

Hey all, I am looking into notifying our teams on Slack regarding metrics and KPIs calculated in BigQuery on a weekly basis. Since the bot would need to attach pictures (plots) that are generated from BigQuery, the question is how to produce them. Does anyone have an idea how to do this cost efficiently? I was already thinking about Google Cloud functions that take data from BQ and then send it to a slackbot but maybe someone has a better idea.

Hi folks, if Airbyte is the open-source EL(T) solution, what is an analogous reverse-ETL open-source solution? I’ve seen “castled” mentioned here, but wanted to see if people had informed opinions from their experience. The scenario I am thinking about is — say my analytics tool generates some scores from data pulled from a CRM like HubSpot or Marketo data, and we want to push those scores back into the CRM, so people can leverage the analytics within their existing workflow without leaving their CRM.

will Grouparoo be able to leverage existing Source integrations in Airbyte?

Hello! I'd like to ask for connector development advice 🙂 I made a custom Intercom destination connector that performs reverse ETL (it syncs data from our database to certain objects in Intercom, such as the Company object in this case). It works okay: we have 100k+ records in the table and it's able to sync them to Intercom. However, even if it successfully syncs to Intercom; the connection itself doesn't complete as a 'success'; it always returns some form of error, such as '`Sync worker failed`' or '`Deadline exceeded`' (logs attached below). Notably, what happens is: after the connection syncs the whole table, the logs kind of 'freeze' (there's no update for hours). Then afterwards, it either spontaneously errors out, or it only returns the sync worker/deadline exceeded error when we cancel the job. This only happens for relatively big jobs (as mentioned, this table has 100k+ rows); I've tried using the same connector on 20 rows and it succeeds. We've looked up the errors, and saw the Github Issues about similar deadline exceeded errors, where the recommendation was to increase the resources allocated to Airbyte (we run Airbyte on EKS). We've already increased the resources, but still run into this error. Is there something else that may be causing the error? What are other approaches we can try?

Maybe a simplistic question, but: why does reverse ETL require different implementations like Grouparoo than ETL/EL(T) solutions like Airbyte?

Hey gang, You have mentioned Reverse ETL as a feature arriving in 2022, are you able to provide any guidance on when that might land? The programmatic/API driven nature of a reverse ETL platform is key for us and most other platforms census/hightough don't provide API or multitenancy support. It looks like an AirBtye CDK would give us a lot of control over how we implement reverse ETL, which would be ideal :)

hi, I understand that Airbyte doesn't support reverse ETL. However, does anyone know why wouldn't I be able to use let's say Facebook connector that Airbyte has as a destination if it is all in API? All I am looking to do is consume an offline conversion API from facebook or other marketing channels and send data to it. @Marcos Marx (Airbyte)

Does the CDK (Java destination) have rate limiting, exception handling etc out of the box or is that something that needs to be built as well?

When you guys look for ETL/ELT products, how do you determine the pricing mechanisms to ensure that you don’t go overboard? How are some of these products priced?

Hi Moss, I was a founder of Grouparoo 🦘 and now at Airbyte 👋 . We switched everything (including the enterprise features) over to a MIT license, so you can use anything you want. We are not supporting it any longer, but to my knowledge it still works.

We have a destination for databricks lakehouse https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-databricks and we have sources for bigquery and snowflake, so if that’s your setup, you should be able to make those connections.

Hi Jaafar, I was a founder of Grouparoo 🦘 and now at Airbyte 👋 . To my knowledge Grouparoo still works, but we are not making any updates. I would look into Hightouch or Census or related. You can also feel free to use the Grouparoo code in any way including the Typescript Klaviyo code, to do something custom.

Hi Everyone 👋, I have a question on rETL between GCP and Amplitude via Airbyte: Can I update track() calls in Amplitude from my DWH?

What is the fastest regarding latency Airbyte can accomplish? Would some event-based (traditional streaming) also work?

Hi! I tried Aibryte cloud and checked Bigquery source connector, which is in alpha at the moment. Do you have a roadmap for improving the Bigquery source connector? It is not scalable in the alpha version, I guess, how do you see it? cc: @Christoffer Möckelind

Hi, wondering if there is any advice on on-prem deployment of reverse ETL over the next couple of months before the Airbyte solution becomes available. Folks are (understandably) a little hesitant about deploying Grouparoo given that it is no longer developed. Thanks in advance!

how do we use reverse etl in airbyte? I don't see the typical destination connectors that exist in Census/Hightouch i.e. Customer.io, Hubspot etc. Do we need to implement Grouparoo standalone?

This message was deleted.