Hi Team. I am looking into the memory storage of segment files in Pinot.During upserts where are the metadata map files stored?Are they stored in-memory or on-disc.

Hi Sharon Upsert maintains metadata in memory containing which docIds are valid in a particular segment (ValidDocIndexes). This metadata gets lost during server restarts and needs to be recreated again. https://docs.pinot.apache.org/basics/data-import/upsert

If it has to be recreated again isn't there a chance of data mismatch?I read that the ValidDocIndexes are persisted periodically after a particular time interval. So considering at one point of time my pk to docId map is persisted and I get 50 more upserts which have not yet persisted the map… after which the process fails due to some technical issue on my end…. In that case what will happen to those 50 rows? Will it reread those from the Kafka topic …. Since the map for those is lost?or is there some other method?

ValidDocIds can be persisted to disk when the segment is completed. To get this behavior snapshots need to be enabled.

what about case of process failure during real time ingestion into a segment.How does Pinot handle data backup.Are all the ValidDocIds persisted to disk?Or does the the pinot consumer re-read from the point of process failure?

In the case of consuming segments, the ValidDocIds can only be retrieved from memory. If all the replicas for a partition fail then ValidDocIds must be reconstructed by re-ingesting from latest offset.