Hey, We are currently implementing Apache Pinot in our fintech company to enhance our analytics capabilities. Previously, we used MySQL as our OLTP system. As part of this transition, we’ve introduced a data pipeline that captures CDC (Change Data Capture) events from MySQL, pushes them to Kafka, and then loads them into Pinot’s real-time tables. We’ve also loaded historical data into offline tables. However, we’re encountering an issue: when updating a historical row in MySQL, the changes are not reflecting in Pinot, even after flushing the real-time data to disk and running the

RealtimeToOfflineSegmentsTask

. We’re still seeing the old data in queries. Steps Taken So Far: 1. Verified that CDC events are correctly propagated to Kafka. 2. Confirmed that real-time tables are receiving new updates. 3. Flushed real-time segments and executed the

RealtimeToOfflineSegmentsTask

to merge data. Questions: • Are there any known gaps in this workflow that could cause stale data in offline segments? • Could there be a misconfiguration in the real-time-to-offline task or retention settings? • Are there additional steps required to ensure updates to historical data are properly reflected? • How we can eliminate duplicate rows in offline table? • What are the correct bucket time to use when delaing with years of historical data for dedup task in Offline tables. We’d appreciate any insights or troubleshooting suggestions to resolve this issue. Thanks in advance for your help!

Are you running these as updates or upserts, @Chirag Varshney? Did you make any progress?