https://linen.dev logo
#good-reads-and-discussions
Title
# good-reads-and-discussions
j

John (Airbyte)

06/10/2021, 4:22 AM
an interesting article about building data platforms from the perspective of ETL https://medium.com/codex/building-data-platforms-the-etl-bias-d589733ce4cc
👍 1
🔥 1
g

gunu

06/10/2021, 10:47 AM
this hits deep
j

Jenny Brown

06/10/2021, 3:24 PM
I was at a place for a while that was deep into that process. ETL was a component, and it never disappeared, but we also started designing for data collection in the primary user-facing product. That changed everything. It made the data cataloguing use case even more important than the ETL process; we piggybacked data catalog on top of our ETL as a result. The central data warehouse became about normalizing and joining data to make it clean for reuse, rather that just about getting it in one spot. The analytics data warehouse became a production resource with similar high-availability requirements, which put a lot of pressure on data movement to be faster and more reliable. Microservices were catalogued the same as data.
👍 1
3 Views