goodread from Claire Carroll (from dbt, now hex) on the problems with CTEs https://hex.tech/blog/stop-using-so-many-ctes

new oreilly book out on making data stack choices https://www.amazon.com/Fundamentals-Data-Engineering-Robust-Systems/dp/1098108302?&linkCode=sl1&tag=dataeng-20&linkId=3588b2ff2124df343a079088fb7e6c4e&language=en_US&ref_=as_li_ss_tl

We have just published a new tutorial! Learn how to extract data from the Intercom API and replicate it into S3 where it can be analyzed by ML/NLP jobs for generating customer insights and sentiment analysis. Additionally, replicating Intercom data into S3 can provide you with backups, help you to meet regulatory compliance requirements, and/or provide you with access to historical data. See: https://airbyte.com/tutorials/intercom-api-s3

today i am marveling at this ~1yr old post that has coined a name for analytics/data engs that has really stuck with me https://www.getdbt.com/blog/we-the-purple-people/

suuper well written piece on the BI paradigms from Maxime Beauchemin https://preset.io/blog/dataset-centric-visualization/

good thread on reddit with people sharing their data stacks! https://reddit.com/r/dataengineering/comments/wcw0nt/_/iifb0i3/?context=1

Hi all, we have published a new tutorial that shows you how to extract data from Stripe’s REST API and send it into Snowflake. Once your Stripe data is in Snowflake tables it can be easily combined with data from other internal systems, which will provide you with enhanced insights into your business. https://airbyte.com/tutorials/stripe-rest-api-to-snowflake

oh wow, Airflow being attacked again on HN https://news.ycombinator.com/item?id=32317558

“the modern data stack is dead” - https://www.linkedin.com/posts/ethanaaron_snowflakesummit-data-analytics-activity-694[…]vGT/?utm_source=linkedin_share&utm_medium=member_desktop_web

The first tutorial in Airbyte’s series on synchronization modes is now live. This article gives an overview of Airbyte’s ELT implementation, explores the SQL code used under-the-hood for full refresh synchronizations, and shows how your replicated data will look. https://airbyte.com/tutorials/full-data-synchronization

new BI tool just got funding! https://twitter.com/medriscoll/status/1555303693938962433?s=21&t=FevPcBw2cGsGMDoCCgqxaQ

TIL of “Potemkin data science” - performative data work that is a lot of work but not actually useful https://mcorrell.medium.com/potemkin-data-science-fba2b5ba5cc6 kind of an old article now but great term for a definitely real phenomenon

A new tutorial has been published! Learn how to build a Data Ingestion Pipeline from HubSpot to Snowflake as part of your data integration strategy. https://airbyte.com/tutorials/copy-data-from-hubspot-to-snowflake

anyone else excited about Dagster 1.0 and Dagster Day? pretty high production quality

a 2012 post I reshared on HN did pretty well today https://news.ycombinator.com/item?id=32407873 - brings up some interesting discussion on schema design (star schema, entity-attribute-value, etc)

A good-listen from Ceora Ford -- How to Make Your First Open Source Contribution or read the transcript included.

We recently published a tutorial where you can learn how we created an ELT pipeline to sync data from Postgres to BigQuery at Airbyte, using Airbyte Cloud! You can follow these steps to create your own. https://airbyte.com/tutorials/postgres-to-bigquery

on the topic of Dagster and orchestration, Benn’s post goes into whats wrong with DAGs https://benn.substack.com/p/down-with-the-dag (i also just posted on HN)

Interesting read also with its used data stack (he calls it

ngods (new generation open-source data stack)

instead of Modern Data Stack). https://blog.devgenius.io/modern-data-stack-demo-5d75dcdfba50

starschema founder writing about us! https://twitter.com/tfoldi/status/1559223629522542592

interesting new BI tool of the day - Omni announcing seed + Series A to compete with Looker https://techcrunch.com/2022/08/16/omni-looks-to-take-on-looker-with-its-cloud-powered-bi-platform/

Omni is comparable to existing BI tools like the aforementioned Looker and Tableau, Zima says. But the platform can also take raw SQL — the language used to communicate with databases — and break it into modeled components. Omni’s built-in tools generate data models and components from SQL, creating a “sandbox” data model and allowing users to promote metrics to the official, shared model that the whole organization can use. Beyond this, Omni runs “automated aggregates” in-database to accelerate queries and manage costs for users (and their employers).

this is the first meme ive ever seen that in itself is a goodread https://twitter.com/largedatabank/status/1559651463919452161?s=21&t=zsoIYmluQHBReWSYaIKD1Q

Dremio wrote a blog post about consolidating all your data into a data lake with Airbyte, and how to turn the data lake into lakehouse with Dremio! Interesting how Dremio adds a SQL engine and data management to your data lake. Does anyone here have experience with Dremio? https://airbyte.com/tutorials/build-an-open-data-lakehouse-with-dremio

postgres in the browser, with tutorial! https://www.crunchydata.com/blog/learn-postgres-at-the-playground

Awesome write-up about trendy Table Formats (Delta Lake, #ApacheIceberg #ApacheHudi) and their features explained well with super illustrations: https://twitter.com/sspaeti/status/1560549706459414529

recently discovered https://pgstats.dev/ which visually explains postgres functionality + architecture BY VERSION which is just mindblowing

interesting Snowflake discussion on HN today: https://news.ycombinator.com/item?id=32551212

I really enjoyed this blog post by @Abi Noda (co-founder of DX--a tool we use at Airbyte). Great synthesis of how to think about waste in the development process.

Bill Inmon on Data Mesh vs Big Data (Cloudera) vs Data Warehousing is just a fantastic listen https://overcast.fm/+wcMo2qvqg/36:00