hi i am trying to store a column with a serialized HLL i wro Apache Pinot #troubleshooting

hi! i am trying to store a column with a serialize...

Jacob M

02/16/2022, 7:45 PM

hi! i am trying to store a column with a serialized HLL. i wrote a small spark app for an offline table and i'm using

com.clearspring.analytics

and generating my HLL like this:

Copy code

val hll = new HyperLogLog(12)
hll.offer(rows.filter(p => p.customer.isDefined).map(p => p.customer.get))
val serializedHll = ObjectSerDeUtils.HYPER_LOG_LOG_SER_DE.serialize(hll)

it seems to ~work ok but when i generate segments i'm getting some errors that I'm not really sure how to debug. any thoughts?

Copy code

java.lang.RuntimeException: Caught exception while de-serializing HyperLogLog
Suppressed: java.lang.NullPointerException	atorg.apache.pinot.segment.local.startree.v2.builder.OffHeapSingleTreeBuilder.close(OffHeapSingleTreeBuilder.java:346)
at org.apache.pinot.segment.local.startree.v2.builder.MultipleTreesBuilder.build(MultipleTreesBuilder.java:143)

and when i look at where the null pointer is coming from, it seems to be from

_starTreeRecordBuffer.close()

which seems odd?

Jacob M

02/16/2022, 7:46 PM

if it's any help I have my schema configured as:

Copy code

name: "customers"
dataType: "BYTES"

and set a

functionColumnPairs

for

DISTINCTCOUNTHLL__customers

Mayank

02/17/2022, 1:52 AM

What version of the jar are you using from clearspring?

Mayank

02/17/2022, 1:53 AM

Pinot is using

2.7.0

Mayank

02/17/2022, 1:53 AM

Note, that you can also do HLL on the fly (during query time) on raw columns. Unless you want to preaggregate offline.

Jacob M

02/17/2022, 2:15 AM

Ooo this is a good catch I think I'm using 2.9.5. I didn't even think about this!

Jacob M

02/17/2022, 7:46 AM

hmmm, didn't end up being it. switched to 2.7.0 but still erroring. different error thought which makes me think it is further along:

Copy code

Caused by: java.lang.RuntimeException: Caught exception while de-serializing HyperLogLog
        at org.apache.pinot.segment.local.utils.CustomSerDeUtils$6.deserialize(CustomSerDeUtils.java:172)
        at org.apache.pinot.segment.local.utils.CustomSerDeUtils$6.deserialize(CustomSerDeUtils.java:156)
        at org.apache.pinot.segment.local.aggregator.DistinctCountHLLValueAggregator.deserializeAggregatedValue(DistinctCountHLLValueAggregator.java:103)

i wrote a test for my spark app that passes as well to check if deserializing was working and it passed.

Copy code

val hll = ObjectSerDeUtils.HYPER_LOG_LOG_SER_DE.deserialize(output.head.serializedHll)
hll.cardinality() shouldBe 2

Open in Slack

Previous Next