Hello everyone, We're conducting a POC using the Airbyte Cloud trial version. I am trying to set up a connection between an MS SQL Server (source) and Azure Data Lake Storage (ADLS) Blob (target). The goal is to split files before writing them to the Blob by configuring the Azure Blob Storage file spill size with the desired MBs. Additionally, we configured the Azure Blob Storage output buffer size to match the spill size to ensure that files are written only once to the Blob. However, despite these efforts, Airbyte creates the file in the Blob as 0 bytes initially, and the file size keeps increasing as the process progresses until completion. This behavior creates an issue because we have an Event Grid that triggers a downstream task as soon as the file is created. We are looking for a solution to ensure that Airbyte writes the file simultaneously as it is created in the Blob and does not continue appending rows to the same file. Note: We cannot use an intermediate Blob storage before pushing the data to the target Blob (which is linked to the Event Grid). Is there any way to solve this issue?

📚 Just published a new blogpost Deploy a Self-service Business Intelligence Project With Whaly & Airbyte | Airbyte

Learn how to move your data to a data warehouse with Airbyte, model it, and build a self-service layer with Whaly’s BI platform.

Read the complete article here

📚 Just published a new blogpost Build a connector to extract data from the Webflow API | Airbyte

Learn how to create a custom Airbyte source connector – this tutorial shows you how to use Airbyte’s Python connector development kit (CDK) to create a source connector that extracts data from the Webflow API. You will learn about authentication, requesting data, and paginating through responses, as well as how to dynamically create streams and how to automatically extract schemas.

Read the complete article here

📚 Just published a new blogpost Building a Social Media Sentiment Analyzer Using Airbyte and Twitter API | Airbyte

Learn to build a social media sentiment analyzer using Airbyte and Twitter API. Simplify data integration and analyze trends effectively.

Read the complete article here

📚 Just published a new blogpost Financial Market Monitoring with Airbyte and Polygon.io Integration | Airbyte

Discover financial market monitoring using Airbyte and Polygon.io integration. Streamline data for actionable insights.

Read the complete article here

📚 Just published a new blogpost Healthcare Data Integration: FHIR API Connector with Airbyte's AI Assistant | Airbyte

Streamline healthcare data integration with Airbyte's AI Assistant and FHIR API connector. Simplify workflows and improve insights.

Read the complete article here

📚 Just published a new blogpost Creating a GitHub Documentation Chatbot Using PyAirbyte and pgvector | Airbyte

Learn how to build a GitHub documentation chatbot with PyAirbyte and PG Vector for seamless data retrieval and enhanced user experience.

Read the complete article here

🖥️ Evolving Data Models: Backbone of Rich User Experiences (UX) for Data Citizens - Hegel's Framework to Distill Value of Data Models underneath ANY and ALL User Interfaces, Good Traits of UX/UI in Data, and Addressing Fundamental User Emotions 📃 In this article, you'll explore how the relationship between users and products shapes user experience (UX), particularly for data users. We’ll dive into the role of evolving data models as key enablers, examine UX evolution through Hegel’s stages of consciousness, and uncover the traits of exceptional UX in data platforms. By the end, you’ll gain fresh insights into designing better experiences for data-driven tools. 💌 Read the complete article here: https://moderndata101.substack.com/p/evolving-data-models-backbone-of

Hi guys ! Just finished reading fundamentals of data engineering and wrote up a review in case anyone is interested! https://medium.com/@sergioramos3.sr/self-taught-reviews-fundamentals-of-data-engineering-by-joe-reis-and-matt-housley-36b66ec9cb23

📚 Just published a new blogpost Explore Airbyte's full refresh data synchronization | Airbyte

Step-by-step instructions that help you to understand how Airbyte’s full refresh overwrite and full refresh append synchronization modes function behind the scenes.

Read the complete article here

📚 Just published a new blogpost Incremental data synchronization between Postgres databases | Airbyte

Learn how Airbyte’s incremental synchronization replication modes work.

Read the complete article here

📚 Just published a new blogpost Automating Customer Support Analytics: Zendesk + Airbyte + OpenAI Integration | Airbyte

Automate customer support analytics with Zendesk, Airbyte, and OpenAI integration. Unlock insights and enhance support efficiency.

Read the complete article here

📚 Just published a new blogpost Building a Knowledge Management System with PyAirbyte and Vector Databases | Airbyte

Discover how to build efficient knowledge management systems using PyAirbyte and vector databases for streamlined data access.

Read the complete article here

🤖 How AI Agents & Data Products Work Together to Support Cross-Domain Queries & Decisions for Businesses: The Two Primary Gaps in AI's Business Enablement Capabilities and the Solution Framework Addressing Both Data and AI Stack Essentials 📃 The article explores how AI agents and data products collaborate to address cross-domain queries and decisions, highlighting gaps in LLMs like context isolation and task execution. Readers will learn about multi-agent workflows, reinforcement learning with RAG, semantic layers, knowledge graphs, and data governance to enable accurate, scalable AI-driven business insights. 💌 Read the complete article here: https://moderndata101.substack.com/p/how-ai-agents-and-data-products-work

🚅 Speed-to-Value Funnel: Data Products + Platform and Where to Close the Gaps - Fundamental approach to building products, platforms, and user-relevant data interfaces 📃 In this article, community author Travis Thompson discussed the Speed-to-Value Funnel for turning data into actionable insights fast. Learn how to define success metrics, prioritize impactful data strategies, build deployable data product MVPs, and boost adoption. Explore frameworks, metric modeling, and evolving data products to align speed, precision, and user-focused development for maximum ROI. ✉️ Read the complete article here: https://moderndata101.substack.com/p/speed-to-value-funnel-data-products

📚 Just published a new blogpost How to Create an LLM Application with ChromaDB & Airbyte | Airbyte

Learn how to build a robust Large Language Model application using ChromaDB for vector storage and Airbyte for data integration, simplifying your AI development workflow.

Read the complete article here

🏗️ Building Supply Chains From Within: Strategic Data Products (Part 1/2): Building Strong & Dynamic Supply Chain Models with a Strategic Approach to Data Products, Data Platforms, and AI Agents. > 📃 In this article by community authors Arielle Rolland and Alexandre Gontcharov talks about - > • The Evolution of the Supply Chain > • Why Your Supply Chain is NOT linear > • Building a Strong Data Foundation Means Moving Beyond Process Mining > • How to Embrace the Future of GenAI 📧 Read the complete article here: https://moderndata101.substack.com/p/strategic-data-products-building-supply-chains-from-within

🏗️ 𝐓𝐡𝐞 𝐂𝐮𝐫𝐫𝐞𝐧𝐭 𝐃𝐚𝐭𝐚 𝐒𝐭𝐚𝐜𝐤 𝐢𝐬 𝐓𝐨𝐨 𝐂𝐨𝐦𝐩𝐥𝐞𝐱: 70% 𝐃𝐚𝐭𝐚 𝐋𝐞𝐚𝐝𝐞𝐫𝐬 & 𝐏𝐫𝐚𝐜𝐭𝐢𝐭𝐢𝐨𝐧𝐞𝐫𝐬 𝐀𝐠𝐫𝐞𝐞 The modern data stack promised speed, scalability, and flexibility—but in reality, it has led to fragmentation, operational overhead, and rising costs. Every new tool claims to simplify, yet the stack only grows more complex, harder to manage, and increasingly expensive. A quick search on data stack complexity will surface common issues: tool sprawl, governance gaps, and interoperability struggles. But these are just symptoms. What’s the root cause? And what does it mean for the future of data infrastructure? 💠 Why has the data stack become so complicated? 💠 What are the real pain points practitioners face daily? 💠 Is simplification even possible at this stage? In this latest Modern Data 101 article, our community authors—Saurabh Gupta, Animesh Kumar, and Matt Lampe—cut through the noise to break down the core challenges of today’s data stack and explore whether the future lies in evolution or revolution. 𝐖𝐡𝐚𝐭 𝐲𝐨𝐮’𝐥𝐥 𝐥𝐞𝐚𝐫𝐧 𝐢𝐧 𝐭𝐡𝐢𝐬 𝐝𝐞𝐭𝐚𝐢𝐥𝐞𝐝 𝐬𝐭𝐮𝐝𝐲? 𝐇𝐞𝐫𝐞’𝐬 𝐚 𝐭𝐚𝐬𝐭𝐞 - 💠 Why there’s a disconnect between promise and reality in data tooling 💠 Why 70% of data leaders see complexity as a bottleneck 💠 The stack-wide impact & outcome of modular architectures 💠 Is consolidation or abstraction the right way forward? And much more! 📩 𝐑𝐞𝐚𝐝 𝐭𝐡𝐞 𝐜𝐨𝐦𝐩𝐥𝐞𝐭𝐞 𝐚𝐫𝐭𝐢𝐜𝐥𝐞 𝐡𝐞𝐫𝐞: https://moderndata101.substack.com/p/the-current-data-stack-is-too-complex

📚 Just published a new blogpost Creating Data Pipeline with dbt & DuckDB Using Airbyte | Airbyte

Learn to build efficient data pipelines using Airbyte, dbt, and DuckDB. A comprehensive guide for data engineers with practical implementation steps.

Read the complete article here

📘 The Data Product Testing Strategy: Handbook - Goals & Components of the Data Product Testing Strategy, Testing Integration in the Data Product Lifecycle, Testing Facilitation Strategies & Technologies, and more! 📃 In this article, community authors will take you through importance of a metrics-first approach in developing data products. It outlines a strategy that begins with identifying business opportunities and use cases, followed by defining a metric model to establish clear relationships between metrics. This approach ensures that data products are aligned with business goals and can be effectively validated and iterated upon. 📧 Read the complete article here: https://moderndata101.substack.com/p/the-data-product-testing-strategy

📚 Just published a new blogpost Orchestrate data ingestion and transformation pipelines with Dagster | Airbyte

Learn how to ingest and transform Github and Slack data into Postgres. Dagster can orchestrate data ingestion pipelines with Airbyte, SQL-based transformations with dbt, and any kind of Python transformation.

Read the complete article here

📚 Just published a new blogpost Building ETL Pipeline with Python, Docker, & Airbyte | Airbyte

Learn how to build robust ETL pipelines using Python, Docker, and Airbyte. A guide for data engineers covering setup, implementation, & best practices.

Read the complete article here

🛠️ How the Ontology Pipeline Powers Semantic Knowledge Systems: The Need for a Structured Approach, Elements of the Ontology Pipeline, the Pipeline as a Framework for Developing Knowledge Management Systems, and More! 📃 Ontologies and knowledge systems don’t have to be a black box. They’re the secret sauce behind *AI, search, and intelligent automation*—if structured right. Jessica Talisman’s latest guide breaks it all down: How the Ontology Pipeline transforms scattered data into machine-readable, scalable knowledge. What’s inside? • Why taxonomies, ontologies & vocabularies are key to AI-ready data • How the Ontology Pipeline eliminates ambiguity & boosts accuracy • The library science principles that power modern AI systems And more! If you’re building RAG pipelines, training LLMs, or structuring enterprise knowledge, this is a must-read. 💌 Read it here: https://moderndata101.substack.com/p/the-ontology-pipeline

⚔️ From Data Tyranny to Data Democracy: How Risk-Based Governance Frameworks and Data Product Owners can transform Data Tyranny into agile, scalable Data Democratization. 📃 In this article, community author Francesco dig into a shift every data leader should be thinking about: • From rigid, centralized control ("data tyranny") • To agile, risk-aware data democratization. But this shift only works if we change how we think about data: • Treat it like a long-term product, not just an asset. • Push ownership left—closer to where data is created. • Adopt risk-based governance that scales with context (not all data needs the same level of scrutiny). If your governance model slows down delivery or bottlenecks innovation, it’s time to rethink it. 📧 Read the complete article here: https://moderndata101.substack.com/p/from-data-tyranny-to-data-democratization

Hi @[DEPRECATED] Marcos Marx, I am a software engineer with experience in writing for tech blogs such as Baeldung and OpenReplay. Could you please guide me through the process of contributing as an author for the Airbyte blog?

📚 Just published a new blogpost A step-by-step guide to setting up and configuring Airbyte and Airflow to work together | Airbyte

Learn how to create an Airflow DAG (directed acyclic graph) that triggers Airbyte synchronizations.

Read the complete article here

📚 Just published a new blogpost Build a connector to extract data from the Webflow API | Airbyte

Learn how to create a custom Airbyte source connector – this tutorial shows you how to use Airbyte’s Python connector development kit (CDK) to create a source connector that extracts data from the Webflow API. You will learn about authentication, requesting data, and paginating through responses, as well as how to dynamically create streams and how to automatically extract schemas.

Read the complete article here

📚 Just published a new blogpost MySQL CDC: Build an ELT pipeline from MySQL Database | Airbyte

Easily set up MySQL CDC using Airbyte, harnessing the power of a robust tool like Debezium to construct a near real-time ELT pipeline.

Read the complete article here

📚 Just published a new blogpost Version control Airbyte configurations with Octavia CLI | Airbyte

Use Octavia CLI to import, edit, and apply Airbyte application configurations.

Read the complete article here

⛓️ Data Lineage is Strategy: Beyond Observability and Debugging - Gaps in passive lineage, how and why Data Products change and uplift lineage, and notes on stepping up to the AI-native era. 📃 Let’s be honest, most data teams only turn to lineage when things break. It’s reactive. Fragmented. Siloed. But in modern, modular data stacks (especially AI-native ones), that’s not enough. That’s where Strategic Lineage comes in. It’s not about watching what happened. It’s about designing for what should happen starting day one. In this article, community author makes the case for treating lineage as a product capability, not an ops tool. He argues that true lineage must be intentionally designed into how data products are built, governed, and consumed. ✉️ Read the complete article here: https://moderndata101.substack.com/p/data-lineage-is-strategy-beyond-observability