We are currently looking to build an ELT architecture where we store raw data from various sources in BigQuery. At the moment, we use Fivetran for this purpose. However, we are considering if we still need Fivetran or if we can use RudderStack to consolidate all our sources and dump the raw data into our data warehouse (DW). An alternative could be creating an architecture where our sources connect to both Fivetran and RudderStack. In this scenario, we could build our event stream in RudderStack and keep that separate from the other various sources directly connected to Fivetran and our DW. We’re seeking your advice on what the best and least complex solution might be for achieving our goal. Could you please provide your recommendations?

Yes, you can use RudderStack to consolidate all your sources and dump the data to your warehouse. If you want to apply transformations after the data reaches warehouse, you can do that using various tools including Dataform. If you need to apply transformations before your data reaches the warehouse, you can do that using RudderStack Transformations.

Thanks alot for your input, do you think this is a good and scalable solution? Or do we miss something along the way .

RudderStack is a warehouse-first solution which makes it flexible and scalable. I recommend to connect your warehouse and send data from all the sources to this warehouse. And then send data from warehouse to your preferred destination, so your warehouse becomes the central source of truth.

You mean connect all my sources to rudderstack then feed the DW?

Looks good to me. Although, I don't have enough insights about your system to make a more informed decision. How do you create 360 profiles?

Thanks alot! What i mean was to bundle events from different source into one stream that we dump into DW

@gentle-petabyte-80785 but is profiles also available for BigQuery? Because what I read is that it’s exclusively available for Snowflake customers

Let me check with team when will that be available. #ask-sales have the best resources regarding profiles at the moment.

@gentle-petabyte-80785 Thanks! We want to proceed with this as fast as possible.

BigQuery support for profiles will take some time. It is not prioritized yet.