Hello, could someone take a look at this github issue : https://github.com/apache/pinot/issues/17233 I think there have multiple follow-ups for CRC mismatch issue which has happened due to Lucene indexes, using ingestion transformations like now(), non-deterministic metadata file ordering etc. And recently, we pushed a change to allow config based skip CRC check altogether during segment load here. This has surfaced more edge-cases mentioned here, where skipping a CRC flag would make a server not download a new segment which is needed in case of backfill jobs, or Upsert compaction. What do we think about computing CRC only for the data part of the segment and not the Indexes ? I understand that this still runs into issue of indeterminism during ingestion transformations where each replica may be running the transform function locally, but apart from that - do we see any issue in switching the CRC computation to be data only ? @Jackie @coco

It's a great idea. I can see 2 challenges for this: 1. We need to find a way to isolate the data only indices. Things become a little bit tricky when forward index is disabled 2. How to migrate To address these challenges, we can probably generate 2 CRCs, and fallback to the current check when the data only CRC is not available. To get more context, do you find Lucene index not deterministic even when data is the same?

To address these challenges, we can probably generate 2 CRCs, and fallback to the current check when the data only CRC is not available.

Yes, that's what I had in mind. We will not touch the existing CRC generation, but instead add a dataCRC and use it whenever available.

do you find Lucene index not deterministic even when data is the same

Yes, lucene index creation even for same data can be indeterministic. This may be because of different lucene commit time from IndexWriter, # of lucene segments, merge policies etc.

cc @Manish @Kartik Khare ^^

Things become a little bit tricky when forward index is disabled

@Jackie - If the data CRC be generated using the : forward index files, inverted index files, dictionary files and metadata file ... I think it will be reliable in that case against external index file ? Even if forward index is disabled for all columns, the inverted index + dict should be enough to detect data changes. Let me know if Im missing a case.