https://pinot.apache.org/ logo
#general
Title
# general
f

francoisa

04/13/2022, 2:23 PM
Hi 🙂 Little question comming with prod getting closer and realData 😄 A few things goes wrong. I’ve got two table reading the same kafka topic. Both of them are using a complexTypeConfig to unnest 30 days arrays. And I gettin an infinite loop error
Copy code
java.lang.RuntimeException: shaded.com.fasterxml.jackson.databind.JsonMappingException: Infinite recursion (StackOverflowError) (through reference chain: org.apache.pinot.spi.data.readers.GenericRow["fieldToValueMap"]->java.util.Collections$UnmodifiableMap["$MULTIPLE_RECORDS_KEY$"]->java.util.ArrayList[0]->org.apache.pinot.spi.data.readers.GenericRow["fieldTo>
        at org.apache.pinot.spi.data.readers.GenericRow.toString(GenericRow.java:247) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at java.util.Formatter$FormatSpecifier.printString(Formatter.java:3031) ~[?:?]
        at java.util.Formatter$FormatSpecifier.print(Formatter.java:2908) ~[?:?]
        at java.util.Formatter.format(Formatter.java:2673) ~[?:?]
        at java.util.Formatter.format(Formatter.java:2609) ~[?:?]
        at java.lang.String.format(String.java:2897) ~[?:?]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:543) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:420) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:598) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at java.lang.Thread.run(Thread.java:829) [?:?]
TransformConfig as Folow ->
Copy code
"complexTypeConfig": {
        "fieldsToUnnest": [
          "data.attributes.regularTimes"
        ],
        "delimiter": ".",
        "collectionNotUnnestedToJson": "NON_PRIMITIVE"
      }
The other table as the same complexTypeConfig but based on another field. Any idea ?
m

Mayank

04/13/2022, 2:31 PM
@User
f

francoisa

04/13/2022, 2:41 PM
I will try to increase the Xss size on the server side to avoid that 😕 Getting 39k messages with at least 30days in each to unnest not sure he like it 😄
reducing reading rate made the tricks using “topic.consumption.rate.limit” : “2",
m

Mayank

04/13/2022, 3:18 PM
Hmm how big is each event and how many levels deep is the nesting
f

francoisa

04/13/2022, 3:20 PM
Big : 30days per event and 27 cols
m

Mayank

04/13/2022, 3:20 PM
What does 30days per event mean
f

francoisa

04/13/2022, 3:21 PM
an array with 30 days in a single event I’m Unnesting
m

Mayank

04/13/2022, 3:21 PM
Ok. What is the event rate?
f

francoisa

04/13/2022, 3:22 PM
For now it’s quite big because lot (39k) message in kafka queue. But expected to be 3 to 4 per seconds
keep failling even with slow message rate :(
m

Mayank

04/13/2022, 7:11 PM
Any reason of having 30 days worth of data in one event? That seems like an anti pattern
f

francoisa

04/13/2022, 7:30 PM
Yes 30 days is linked to a toplevel object summing a few things
It's a time report.
May using more partition help ? Only two partitions now :)
m

Mayank

04/13/2022, 7:43 PM
Your ingestion rate is really low, not sure if that will help
Can the upstream not flatten the events to be 1 row? Also, do I understand it right, one event has an array of 30 elements, and each element is a row of 27 columns?
f

francoisa

04/13/2022, 8:39 PM
Yes you are right
m

Mayank

04/13/2022, 8:39 PM
If so, it doesn’t seem terribly bad. The root cause of infinite might be something different, and worth filing an issue
If you can provide a sample payload with the issue that can help reproduce it, we should be able to identify the root cause, and hopefully fix it
Can you file an issue and paste a link here?
f

francoisa

04/13/2022, 8:41 PM
I will look for a bad message an try to reproduce the issue on my local instance.
m

Mayank

04/13/2022, 8:41 PM
Sounds good, thanks
f

francoisa

04/14/2022, 1:08 PM
Made some further test with 15k row ingested multiple time 10 to 20 times locally no issue
👍 1
m

Mayank

04/14/2022, 2:16 PM
Do you know what payload causes the issue?
f

francoisa

04/14/2022, 2:17 PM
Not the one in the first half -> loading a full 30K row to try to get the faulty one 😄
m

Mayank

04/14/2022, 2:33 PM
Ok
f

francoisa

04/14/2022, 2:39 PM
Start thinking about RAM... process on Local an preProd does not have the same RAM props 😕
m

Mayank

04/14/2022, 2:43 PM
I don’t think your problem is memory bound, it is more about either bad payload or a bug that triggers for a specific payload
f

francoisa

04/14/2022, 4:04 PM
ok i’ve managed to reproduce it on local -> bad messsage but different messages between 9.3.0 en 10.0.0
m

Mayank

04/14/2022, 4:38 PM
Cool, please file GH issue with the two bad event payloads and share the link here. We will pick it up
f

francoisa

04/15/2022, 12:51 PM
In fact was’nt payload issue but name in the configs that cause this issue. V10.0.0 deployed in prod working like a charm but as soon as I add a filter config get back to the jackson error 😕 Issue filled here -> https://github.com/apache/pinot/issues/8549
m

Mayank

04/15/2022, 1:47 PM
Thanks for filing, we will take a look shortly