We have published a new article that discusses an EtLT (Extract, “tweak”, Load, Transform) approach to data integration – in which a very light “tweak” (small “t”) transformation is done on the data before it reaches the destination. This can be used to improve GDPR compliance. https://airbyte.com/blog/etlt-gdpr-compliance

We recently published a new article that discusses the evolution of the data engineering role! 👩🏻‍💻 https://airbyte.com/blog/data-engineering-past-present-and-future

just caught up on this fantastic read from @Anna Filippova on how even dbt are facing real issues with analytics complexity. 21 hour long weekly

dbt run

😱 https://roundup.getdbt.com/p/complexity-the-new-analytics-frontier

Learn how to easily replicate your data from IBM Db2 to Postgres, where it can be combined with data from other sources to get a holistic view of your business. - https://airbyte.com/tutorials/replicate-from-ibm-db2-to-postgres

Have others here tried to manage Airbyte configurations with code? I gave it a try and wrote a tutorial about how to use Octavia CLI to manage source, destination and connection configurations in YAML instead of using the web app UI. https://airbyte.com/tutorials/version-control-airbyte-configurations

https://airbyte.com/tutorials/replicate-data-from-mysql-to-bigquery Learn how to easily replicate your data from MySQL to BigQuery, where it can be combined with data from other sources to get a holistic view of your business.

Hey folks, I was wondering if you have any good-reads for me on how to valuate the benefits of the “modern data stack”. Are there any good publications that put a value on setting up a a data warehouse compared to not having one? For example for marketing one should have a benefit from unified and improved data, but are there any numbers/percentages from studies or testimonials?

Have you tried to set up Monitoring for Airbyte? I tried it and explored the possibilities with dbt and Metabase (and gave Rill Data a try). All the code is on GitHub. What can you expect? 🛫 Set Up Airbyte 🗃️: Sync Data to Postgres DB 🤹‍♂️: Create Transforming Views with dbt 📈: Create a Metabase Dashboard I hope you enjoy it. I'm looking forward to your feedback as there are lots of improvements (especially on the dashboard side), but I wanted to get it out and get some feedback. I will add more to the open-data-stack repo with orchestration, semantic layer, and all the good stuff. Make sure you follow on if that is what interests you 🙂.

#vendor-content

🙂 YAML and Python are obviously lingua franca in the data world, so here at Cube—a headless BI platform—we’re taking steps to speak YAML fluently. With today’s update, you can author Cube’s data models with the same experience you have when working with dbt models upstream: https://cube.dev/blog/introducing-cube-support-for-yaml-data-modeling

good recap of snowflake recent offerings https://commonstock.com/post/a87ec8b1-df05-41f2-83b5-7560a2cad9af

Congrats to @Thalia Barrera for getting a shoutout in the latest Analytics Engineering Roundup! “Thalia Barrera writes perhaps the best overview of the state of data engineering since Max B’s canonical post.” https://roundup.getdbt.com/p/the-data-diaspora

Good read! 👍 https://taleno.digital/3rd-way-to-setup-metabase-in-production/

this is an old piece but good read on data contracts https://dataproducts.substack.com/p/the-rise-of-data-contracts

Redshift

turns 10 today (can you believe it?) and we just published a new blog post about it 🎂 https://airbyte.com/blog/amazon-redshift-data-warehouse-evolution

Interesting read about how to transform Airbyte raw data with JSON functions in BigQuery. The author builds a Python bot to send Intercom data extracted with Airbyte to Telegram. https://medium.com/@alexnikdanilin/bigquery-airbyte-intercom-and-another-story-of-google-cloud-platform-cost-optimization-7160156958d9

biggest AWS reinvent announcement so far. data lake game keeps changing https://techcrunch.com/2022/11/29/awss-new-amazon-security-lake-is-a-standards-based-data-lake-for-security-data/

and more point to point transfers https://techcrunch.com/2022/11/29/amazon-takes-a-step-towards-a-zero-etl-future-with-two-announcements-at-reinvent/

Implementing, maintaining & scaling open source modern data stack doesn't have to be so hard 🫰 https://www.plural.sh/blog/joey-taleno/

This is a meta-“good reads” - publishing our first results from the State of Data Engineering survey, covering the top 10 newsletters in data engineering! https://airbyte.com/blog/best-newsletters-2022

this was an excellent conversation about Data Contracts from 4 key players https://overcast.fm/+w94UnP0f8

How do you test data replication pipelines? @Madison Mae shared how to find row differences between sources and destinations with data-diff! https://airbyte.com/tutorials/validate-data-replication-postgres-snowflake

some nice data hot takes in here https://twitter.com/matsonj/status/1599906050458943492

our CEO @Michel (Airbyte)’s latest appearance! https://www.moderndatastack.xyz/podcast/s01-e10-commoditizing-data-integration-with-airbyte-michel-tricot-co-founder-a-ayxh

this issue on dbt core is actually a good overview of how dbt sees the semantic layer from @Callum McCann https://github.com/dbt-labs/dbt-core/issues/6379

A Conversation with chatGPT on Data Engineering https://medium.com/towards-data-engineering/a-conversation-with-chatgpt-on-data-engineering-5e9434d0f4bc