Hi team! I was wondering if anyone has seen lag in json index creation in realtime segments for Pinot 1.4.0. Currently our json_match filter seems to return partial results for our realtime table and have no way to track this. If so, is there a way to track json index creation lag?

@Raghvendra Yadav ^^

@Soon We have not observed any lag in json index creation in real time segments. Can you please share the query and sample records if possible to analyze the Json_match results ?

@Raghvendra Yadav @Mayank We found the root cause for this: https://github.com/apache/pinot/issues/18737 Mutable Json would fail to index strings with a split utf16 character (ie an emoji where only the first byte is present becasue the string was truncated). The json index would extend the

docIdMapping

but not increase the

_nextFlattenedDocId

because

Utf8.encodedLength(key);

throws earlier in MutableJsonIndexImpl.java I propose https://github.com/apache/pinot/pull/18738 to fix it.

@Samuel Papin I will take a look today.

This should take care of it! Thanks!