Hi, when ingesting large data into a realtime tabl...
# general
p
Hi, when ingesting large data into a realtime table (TPS around 50K), I perform sql count number record of that table and the returned result was not stable (for example: first time return 32M but the second time only return 30M, third time is 31M then again 30M for the forth time...), could you explain this behavior and what should to do with this
@Mayank @Xiang Fu
x
How many replicas do you have? For multiple replicas, pinot may return the latest segments from any replica, which could be the number of what you observed
p
@Xiang Fu I set replicas to 3. But I mean... is that Pinot does not follow common principle of replication that it need to make consistency in query result? Pinot is designed to support realtime analytics but can not handle consistency in querying data with realtime ingestion? Or having another more suitable configuration to solve this
x
Yes, Pinot is an eventual consistency system for the realtime data. Since each replica is consuming at its own speed, each query might hit different replica hence the different results. Realtime query results consistency is a requested feature. We could support the query to a specific partition to ensure the result consistency.