A new Data Insight on DataLake and Lakehouse includes: • Differences between Lakehouse & Warehouse • Components of Data Lake ◦ Storage Layer (AWS S3, Azure Blob Storage, Google Cloud Storage) ◦ File Format (Apache Parquet, Avro, ORC) ◦ Table Format (Delta Lake, Apache Hudi, and Iceberg) • Trends in the Market • How to turn it into a Lakehouse https://airbyte.com/blog/data-lake-lakehouse-guide-powered-by-table-formats-delta-lake-iceberg-hudi

We just published a new blog post! Reverse ETL is the process of syncing data from a source like a data warehouse to a business application so it can be used by marketing, sales, support, and other teams in the tools they use. It is rapidly becoming a standard component of data stacks.

reading Barr’s OG post on data observability today - old but really gold https://www.montecarlodata.com/blog-what-is-data-observability/

Snowflake had a very good quarter https://softwarestackinvesting.com/snowflake-snow-q2-fy2023-earnings-report/

Hey! I shared some thoughts about why I feel data quality is more complex than code quality, and how to make data quality an easier problem to solve. Curious to hear how others think of a data architecture and organization that reduces the overall data quality issues you face? https://airbyte.com/blog/data-quality-issues

A new tutorial has been published! Learn how to build a Data Ingestion Pipeline from MSSQL Server to Snowflake as part of your data integration strategy. https://airbyte.com/tutorials/replicate-microsoft-sql-server-to-snowflake

The second tutorial in Airbyte’s series on synchronization modes is now live. This article gives an overview of Airbyte’s ELT implementation, explores the SQL code used under-the-hood for incremental data synchronizations, and shows how your replicated data will look. https://airbyte.com/tutorials/incremental-data-synchronization

Check out these data security insights from our Head of Data Policy here at Airbyte, Patsy Bailin: https://airbyte.com/blog/4-questions-data-security-experts-ask-before-moving-data

A new tutorial has been published! Learn how to build a Data Ingestion Pipeline to send CSV data from S3 to Snowflake as part of your data integration strategy. https://airbyte.com/tutorials/copy-s3-csv-to-snowflake

A new blog post is live. It's the start of a series where we bring you on a journey on how we build our internal data stack at Airbyte!

A new blog post that shows how Airbyte can be used to move data from Snowflake to Postgres is now live! https://airbyte.com/tutorials/replicate-data-from-snowflake-to-postgres

ooh, a Dagster vs Airflow article on HN! https://news.ycombinator.com/item?id=32839147

Hello everyone🙂. I'm preparing a presentation about Airbyte for my internal data team; anyone knows any good resource to help me with that? The idea is to be something simple. To complement, I prepared an Airbyte Demo replicating Postgres Data (CDC) to Snowflake. Thanks 🙂

https://cloud.google.com/datastream-for-bigquery That's a huge game changer

🎙️ A Semantic Layer is a topic that fascinated me from the beginning. As a BI engineer, I have used the initial concept since the early SAP BusinessObjects. Now with the open-source trend, there are new exciting tools out there. I explored the history, trends, tooling, and difference between existing concepts we all know. I am answering questions about whether it is just an OLAP cube, a new way of virtualization, another term for lakehouse, the difference between a Metrics Layer or Headless BI tool, and much more. Hopefully, this is helpful to you or sparked some new ideas. Please let me know what you think or if you have any questions—looking forward to it 🤗.

The third tutorial in Airbyte’s series on synchronization modes is now live. This article gives an overview of Airbyte’s Change Data Capture (CDC) data synchronization. https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication

https://seekingalpha.com/article/4538291-snowflake-inc-snow-deutsche-bank-technology-conference-transcript snowflake with some eye popping numbers - 25% of the largest 5000 companies in the world, avg spend 1.2m each. Capital One went from spending $29m/yr to $50m/yr in 2 years. “Apache Iceberg is emerging as the de facto standard for table formats for our largest customers” they bought Applica for 200m, despite their previous valuation being 600m. a lot of companies for sale at deep discounts

Hi everybody, I'm Dario from Fortisoft. I would like to share with the community an article we wrote on how to develop Airbyte connectors. It is intended for people who have completed the PokeAPI tutorial, and are looking to learn more about connector development through a harder challenge. How to create your own Airbyte connector and pull data from Discord API

big /r/dataengineering data warehouse poll going on right now with 2.2k votes https://twitter.com/felipehoffa/status/1577413199875387393?s=46&t=enuXKuJ_dYz27lwVk3Tldg

just heard about Canvas for BI: https://techcrunch.com/2022/01/28/canvas-gives-non-technical-teams-data-exploration-knowledge-without-needing-a-degree-in-sql/ Spreadsheets > SQL, and can easily reuse dbt logic

There's a series of posts here that are anti FiveTran, dbt, Airbyte. Good read for the alternative perspective, if not just for the laughs: https://medium.com/@laurengreerbalik/how-fivetran-dbt-actually-fail-3a20083b2506

The final article in Airbyte’s series on synchronization modes is now live. This article gives an overview and comparison of the different replication options that are available. https://airbyte.com/blog/understanding-data-replication-modes

Great read, shows how you can visualize data dependencies across tools! https://medium.com/starschema-blog/dagster-airbyte-dbt-how-software-defigned-assets-change-the-way-we-orchestrate-ac70bb29d640

this interview of George Fraser gets really interesting when he talks about the possible danger to the mpp analytical datawarehouse because of advancements in hardware

good new memo on dbt https://research.contrary.com/reports/dbt-labs

need help to sync data from mysql to postgres. anyone ?

good new metrics definition template from Mode, from the dbt conference today https://docs.google.com/document/d/1GOKQQRzC4bfj0-xviDD4L5YLmKHrVOVK8s9hHWRTp3A/edit

Learning Rust? I gave it the first try, compared it to Python, and checked it for data engineering tasks. Let me know if you disagree and what your experiences with Rust are so far.