Excited to share our journey of building incremental flow and how it’s making a difference at Orthodox Union. Check out the case study and let us know your thoughts! 🔍 https://www.linkedin.com/feed/update/urn:li:activity:7266841072868151296

My take on Jaguar recent mishaps and what they could have done better 💠 Who is it for: Anyone leading a brand marketing strategy. 💠 Why is it relevant: Effective easy to use solution. https://www.linkedin.com/posts/samuel-momodu_avoid-making-business-decisions-solely-b[…]740304429058-LNvy?utm_source=share&utm_medium=member_desktop

🧑‍💻 Data Leadership Extends Beyond Core-Data Professionals: Core-Data Leaders, Business Data Leaders, and Enablement Strategies to Lead with Data In the face of Low Data Literacy or Analytics Training 📃 In this article, community authors James & Travis explains why data leadership is broader than what you know it to be. They discussed in detail why 'data leadership' needs to extend beyond the specialists. Why? Because, it only works when business leaders are fully engaged. Data = business, after all. The article also includes some great practical advice from Travis Thompson, on how to make it easier for business leaders to engage, via products and metrics. 📧 Read the complete article here: https://moderndata101.substack.com/p/data-leadership-extends-beyond-core

📚 Just published a new blogpost Export data from LinkedIn Ads to BigQuery | Airbyte

Learn how to easily move your LinkedIn Ads marketing data into BigQuery where it can be combined with data from other sources to get a more holistic view of your business. Gain valuable insights about customer acquisition and the value of your customer conversions from advertisements.

Read the complete article here

⚙️ Governance for AI Agents with Data Developer Platforms: Facets of Governance for AI Agents, Current Challenges/Risks and Data/AI Architectures that Address them. 📃 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞 continues to be the most critical aspect, irrespective of continuous innovations and technologies in the Data & AI Space. Now we have AI Agents to do our bidding, but to what extent? This piece contributed by Brij M., Travis Thompson, and Ritwika Chowdhury covers all these facets as well as approaches, governance models, and tech implementations to tackle them. What’s Inside: • Why governance is the backbone of AI, even with tech like AI Agents. • Tackling challenges: trust gaps, policy confusion, and faulty AI responses. • Governance frameworks and models to manage AI autonomy effectively. • Tools and strategies for risk mitigation and seamless implementation. 📧 Read the complete article here: https://moderndata101.substack.com/p/governance-for-ai-agents-with-ddp

Are you unsatisfied with your current data catalog or maybe curious just to learn what is out there today? After deep research, and spending time building with each catalog, I created a teardown with feature comparison matrices and I created a metastore to rank each data catalog across features ranging from access controls, data quality, data discovery, and much more: https://www.linkedin.com/posts/lakehouse_comprehensive-data-catalog-comparison-activi[…]770251563008-cgYP?utm_source=share&utm_medium=member_desktop

🎨 The Art of Discoverability and Reverse Engineering User Happiness: Core Challenges, Range of Discoverability Solutions, User Motivations, Metadata Fundamentals, and Infrastructures that Back This! 📃 In this article, the authors have discussed in details - • The Problem with Discoverability • Challenges with present-day Metadata Platforms/Technologies • The Solution: Consolidation at a Global Level • How is Data Disoverability Realised? • What it means to build for the User: Discoverability-as-a-Feature • Discoverability Solutions 💌 Read the complete article here: https://moderndata101.substack.com/p/the-art-of-discoverability-and-reverse

Clickhouse Use Case Guide: Digital Twins https://runportcullis.co/blog/clickhouse-use-case-guide-digital-twins/

📚 Just published a new blogpost AI Task Prioritizer: A Step-by-Step Guide to Data Pipelines with Airbyte, Milvus, and Next.js | Airbyte

This tutorial demonstrates an end-to-end AI Task Prioritizer, using semantic/similarity search to find and arrange results of tasks from Asana. Data is ingested through the Airbyte tool, using Asana as the source, and Milvus as the destination (vector database). The tech stack is primarily Next.js.

Read the complete article here

Let's face it, Clickhouse is already a super fast OLAP data warehouse, but with each release it feels like performance keeps improving in various key areas. With the 24.12 release, one of the standout additions is a new cache for primary indexes, which can dramatically improve query performance by reducing disk reads and network traffic in distributed setups. https://runportcullis.co/blog/primary-key-caching/

From performance optimizations to new features that will make this real-time warehouse even more powerful, 2025 is shaping up to be a transformative year for Clickhouse. https://www.runportcullis.co/blog/clickhouse-2025-roadmap/

Does anyone have any good readings or resources on Iceberg? We started exploring Snowflake open catalog (polaris) and I am wondering how it relates to glue/nessie - and whether the upcoming S3 datalake iceberg support would also allow ingestion for Iceberg through SF open catalog?

🛠️ Federated Modeling: When and Why to Adopt - The Most Straightforward Business Case You'll Find for Data Modeling 📃 Many have highlighted the need to adopt 𝗳𝗲𝗱𝗲𝗿𝗮𝘁𝗲𝗱 𝗺𝗼𝗱𝗲𝗹𝗶𝗻𝗴 practices within their data ecosystems. While this approach offers significant advantages, such as cost savings, it also introduces challenges like increased effort. The goal of this article is to 𝐈𝐥𝐥𝐮𝐬𝐭𝐫𝐚𝐭𝐞 𝐚𝐧𝐝 𝐪𝐮𝐚𝐧𝐭𝐢𝐟𝐲 both the benefits and challenges of federated modeling, helping you make informed decisions for your data ecosystem. 💌 Read the complete article here: https://moderndata101.substack.com/p/federated-modeling-when-and-why

I wrote some key principles to enhance data modeling, with DBT examples for technical understanding as well. https://www.linkedin.com/feed/update/urn:li:share:7282146373435559937/

Hello everyone, We're conducting a POC using the Airbyte Cloud trial version. I am trying to set up a connection between an MS SQL Server (source) and Azure Data Lake Storage (ADLS) Blob (target). The goal is to split files before writing them to the Blob by configuring the Azure Blob Storage file spill size with the desired MBs. Additionally, we configured the Azure Blob Storage output buffer size to match the spill size to ensure that files are written only once to the Blob. However, despite these efforts, Airbyte creates the file in the Blob as 0 bytes initially, and the file size keeps increasing as the process progresses until completion. This behavior creates an issue because we have an Event Grid that triggers a downstream task as soon as the file is created. We are looking for a solution to ensure that Airbyte writes the file simultaneously as it is created in the Blob and does not continue appending rows to the same file. Note: We cannot use an intermediate Blob storage before pushing the data to the target Blob (which is linked to the Event Grid). Is there any way to solve this issue?

📚 Just published a new blogpost Deploy a Self-service Business Intelligence Project With Whaly & Airbyte | Airbyte

Learn how to move your data to a data warehouse with Airbyte, model it, and build a self-service layer with Whaly’s BI platform.

Read the complete article here

📚 Just published a new blogpost Build a connector to extract data from the Webflow API | Airbyte

Learn how to create a custom Airbyte source connector – this tutorial shows you how to use Airbyte’s Python connector development kit (CDK) to create a source connector that extracts data from the Webflow API. You will learn about authentication, requesting data, and paginating through responses, as well as how to dynamically create streams and how to automatically extract schemas.

Read the complete article here

📚 Just published a new blogpost Building a Social Media Sentiment Analyzer Using Airbyte and Twitter API | Airbyte

Learn to build a social media sentiment analyzer using Airbyte and Twitter API. Simplify data integration and analyze trends effectively.

Read the complete article here

📚 Just published a new blogpost Financial Market Monitoring with Airbyte and Polygon.io Integration | Airbyte

Discover financial market monitoring using Airbyte and Polygon.io integration. Streamline data for actionable insights.

Read the complete article here

📚 Just published a new blogpost Healthcare Data Integration: FHIR API Connector with Airbyte's AI Assistant | Airbyte

Streamline healthcare data integration with Airbyte's AI Assistant and FHIR API connector. Simplify workflows and improve insights.

Read the complete article here

📚 Just published a new blogpost Creating a GitHub Documentation Chatbot Using PyAirbyte and pgvector | Airbyte

Learn how to build a GitHub documentation chatbot with PyAirbyte and PG Vector for seamless data retrieval and enhanced user experience.

Read the complete article here

🖥️ Evolving Data Models: Backbone of Rich User Experiences (UX) for Data Citizens - Hegel's Framework to Distill Value of Data Models underneath ANY and ALL User Interfaces, Good Traits of UX/UI in Data, and Addressing Fundamental User Emotions 📃 In this article, you'll explore how the relationship between users and products shapes user experience (UX), particularly for data users. We’ll dive into the role of evolving data models as key enablers, examine UX evolution through Hegel’s stages of consciousness, and uncover the traits of exceptional UX in data platforms. By the end, you’ll gain fresh insights into designing better experiences for data-driven tools. 💌 Read the complete article here: https://moderndata101.substack.com/p/evolving-data-models-backbone-of

Hi guys ! Just finished reading fundamentals of data engineering and wrote up a review in case anyone is interested! https://medium.com/@sergioramos3.sr/self-taught-reviews-fundamentals-of-data-engineering-by-joe-reis-and-matt-housley-36b66ec9cb23

📚 Just published a new blogpost Explore Airbyte's full refresh data synchronization | Airbyte

Step-by-step instructions that help you to understand how Airbyte’s full refresh overwrite and full refresh append synchronization modes function behind the scenes.

Read the complete article here

📚 Just published a new blogpost Incremental data synchronization between Postgres databases | Airbyte

Learn how Airbyte’s incremental synchronization replication modes work.

Read the complete article here

📚 Just published a new blogpost Automating Customer Support Analytics: Zendesk + Airbyte + OpenAI Integration | Airbyte

Automate customer support analytics with Zendesk, Airbyte, and OpenAI integration. Unlock insights and enhance support efficiency.

Read the complete article here

📚 Just published a new blogpost Building a Knowledge Management System with PyAirbyte and Vector Databases | Airbyte

Discover how to build efficient knowledge management systems using PyAirbyte and vector databases for streamlined data access.

Read the complete article here

🤖 How AI Agents & Data Products Work Together to Support Cross-Domain Queries & Decisions for Businesses: The Two Primary Gaps in AI's Business Enablement Capabilities and the Solution Framework Addressing Both Data and AI Stack Essentials 📃 The article explores how AI agents and data products collaborate to address cross-domain queries and decisions, highlighting gaps in LLMs like context isolation and task execution. Readers will learn about multi-agent workflows, reinforcement learning with RAG, semantic layers, knowledge graphs, and data governance to enable accurate, scalable AI-driven business insights. 💌 Read the complete article here: https://moderndata101.substack.com/p/how-ai-agents-and-data-products-work

🚅 Speed-to-Value Funnel: Data Products + Platform and Where to Close the Gaps - Fundamental approach to building products, platforms, and user-relevant data interfaces 📃 In this article, community author Travis Thompson discussed the Speed-to-Value Funnel for turning data into actionable insights fast. Learn how to define success metrics, prioritize impactful data strategies, build deployable data product MVPs, and boost adoption. Explore frameworks, metric modeling, and evolving data products to align speed, precision, and user-focused development for maximum ROI. ✉️ Read the complete article here: https://moderndata101.substack.com/p/speed-to-value-funnel-data-products

📚 Just published a new blogpost How to Create an LLM Application with ChromaDB & Airbyte | Airbyte

Learn how to build a robust Large Language Model application using ChromaDB for vector storage and Airbyte for data integration, simplifying your AI development workflow.

Read the complete article here