Hi 🙂 Little question comming with prod getting closer and realData 😄 A few things goes wrong. I’ve got two table reading the same kafka topic. Both of them are using a complexTypeConfig to unnest 30 days arrays. And I gettin an infinite loop error

java.lang.RuntimeException: shaded.com.fasterxml.jackson.databind.JsonMappingException: Infinite recursion (StackOverflowError) (through reference chain: org.apache.pinot.spi.data.readers.GenericRow["fieldToValueMap"]->java.util.Collections$UnmodifiableMap["$MULTIPLE_RECORDS_KEY$"]->java.util.ArrayList[0]->org.apache.pinot.spi.data.readers.GenericRow["fieldTo>
        at org.apache.pinot.spi.data.readers.GenericRow.toString(GenericRow.java:247) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at java.util.Formatter$FormatSpecifier.printString(Formatter.java:3031) ~[?:?]
        at java.util.Formatter$FormatSpecifier.print(Formatter.java:2908) ~[?:?]
        at java.util.Formatter.format(Formatter.java:2673) ~[?:?]
        at java.util.Formatter.format(Formatter.java:2609) ~[?:?]
        at java.lang.String.format(String.java:2897) ~[?:?]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:543) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:420) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:598) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at java.lang.Thread.run(Thread.java:829) [?:?]

TransformConfig as Folow ->

"complexTypeConfig": {
        "fieldsToUnnest": [
          "data.attributes.regularTimes"
        ],
        "delimiter": ".",
        "collectionNotUnnestedToJson": "NON_PRIMITIVE"
      }

The other table as the same complexTypeConfig but based on another field. Any idea ?

I will try to increase the Xss size on the server side to avoid that 😕 Getting 39k messages with at least 30days in each to unnest not sure he like it 😄

Hmm how big is each event and how many levels deep is the nesting

Big : 30days per event and 27 cols

What does 30days per event mean

an array with 30 days in a single event I’m Unnesting

Ok. What is the event rate?

For now it’s quite big because lot (39k) message in kafka queue. But expected to be 3 to 4 per seconds

Any reason of having 30 days worth of data in one event? That seems like an anti pattern

Yes 30 days is linked to a toplevel object summing a few things

Your ingestion rate is really low, not sure if that will help

Yes you are right

If so, it doesn’t seem terribly bad. The root cause of infinite might be something different, and worth filing an issue

I will look for a bad message an try to reproduce the issue on my local instance.

Sounds good, thanks

Made some further test with 15k row ingested multiple time 10 to 20 times locally no issue

Do you know what payload causes the issue?

Not the one in the first half -> loading a full 30K row to try to get the faulty one 😄

Ok

Start thinking about RAM... process on Local an preProd does not have the same RAM props 😕

I don’t think your problem is memory bound, it is more about either bad payload or a bug that triggers for a specific payload

ok i’ve managed to reproduce it on local -> bad messsage but different messages between 9.3.0 en 10.0.0

Cool, please file GH issue with the two bad event payloads and share the link here. We will pick it up

In fact was’nt payload issue but name in the configs that cause this issue. V10.0.0 deployed in prod working like a charm but as soon as I add a filter config get back to the jackson error 😕 Issue filled here -> https://github.com/apache/pinot/issues/8549

Thanks for filing, we will take a look shortly