https://pinot.apache.org/ logo
Join Slack
Powered by
i am ingesting serialized apache datasketch state ...
# troubleshooting
p
i am ingesting serialized apache datasketch state (both hll and quantiles) into pinot via spark batch ingestion. lets call them distinct_count_hll and val_quantiles. how do i represent them in schema so i can query them over a subset of dimensions via "merge" to get approx distinct counts or percentiles?
x
in schema you can use 
bytes
as data type.
then use hll_raw functions to work directly on the bytes hll objects
p
perfect, thank you!
Copy code
select server_event_ts, distinctcounthll(unique_devices_hll) from lego_gc group by server_event_ts order by server_event_ts limit 500
is returning
Copy code
Error Code: 200

QueryExecutionError:
java.lang.RuntimeException: Caught exception while merging HyperLogLogs
	at org.apache.pinot.core.query.aggregation.function.DistinctCountHLLAggregationFunction.aggregateGroupBySV(DistinctCountHLLAggregationFunction.java:172)
	at org.apache.pinot.core.query.aggregation.groupby.DefaultGroupByExecutor.aggregate(DefaultGroupByExecutor.java:218)
	at org.apache.pinot.core.query.aggregation.groupby.DefaultGroupByExecutor.process(DefaultGroupByExecutor.java:206)
	at org.apache.pinot.core.operator.query.GroupByOperator.getNextBlock(GroupByOperator.java:112)
...
Caused by: java.lang.RuntimeException: Caught exception while deserializing HyperLogLog
	at org.apache.pinot.core.common.ObjectSerDeUtils$19.deserialize(ObjectSerDeUtils.java:764)
	at org.apache.pinot.core.common.ObjectSerDeUtils$19.deserialize(ObjectSerDeUtils.java:752)
	at org.apache.pinot.core.common.ObjectSerDeUtils$19.deserialize(ObjectSerDeUtils.java:746)
	at org.apache.pinot.core.common.ObjectSerDeUtils$19.deserialize(ObjectSerDeUtils.java:733)
...
Caused by: java.nio.BufferUnderflowException
	at java.base/java.nio.Buffer.nextGetIndex(Buffer.java:643)
	at java.base/java.nio.ByteBufferAsIntBufferB.get(ByteBufferAsIntBufferB.java:121)
	at org.apache.pinot.core.common.ObjectSerDeUtils$19.deserialize(ObjectSerDeUtils.java:758)
	... 21 more
i got similar error with 
distinctcountrawhll
Copy code
Error Code: 200

QueryExecutionError:
java.lang.RuntimeException: Caught exception while merging HyperLogLogs
	at org.apache.pinot.core.query.aggregation.function.DistinctCountHLLAggregationFunction.aggregateGroupBySV(DistinctCountHLLAggregationFunction.java:172)
	at org.apache.pinot.core.query.aggregation.function.DistinctCountRawHLLAggregationFunction.aggregateGroupBySV(DistinctCountRawHLLAggregationFunction.java:75)
	at org.apache.pinot.core.query.aggregation.groupby.DefaultGroupByExecutor.aggregate(DefaultGroupByExecutor.java:218)
	at org.apache.pinot.core.query.aggregation.groupby.DefaultGroupByExecutor.process(DefaultGroupByExecutor.java:206)
...
Caused by: java.lang.RuntimeException: Caught exception while deserializing HyperLogLog
	at org.apache.pinot.core.common.ObjectSerDeUtils$19.deserialize(ObjectSerDeUtils.java:764)
	at org.apache.pinot.core.common.ObjectSerDeUtils$19.deserialize(ObjectSerDeUtils.java:752)
	at org.apache.pinot.core.common.ObjectSerDeUtils$19.deserialize(ObjectSerDeUtils.java:746)
	at org.apache.pinot.core.common.ObjectSerDeUtils$19.deserialize(ObjectSerDeUtils.java:733)
...
Caused by: java.nio.BufferUnderflowException
	at java.base/java.nio.Buffer.nextGetIndex(Buffer.java:643)
	at java.base/java.nio.ByteBufferAsIntBufferB.get(ByteBufferAsIntBufferB.java:121)
	at org.apache.pinot.core.common.ObjectSerDeUtils$19.deserialize(ObjectSerDeUtils.java:758)
	... 22 more
We are using 
import org.apache.druid.hll.{HyperLogLogCollector, HyperLogLogHash}
to build base64 encoded strings for hll sketch state, and 
import com.yahoo.sketches.quantiles.{DoublesSketch, UpdateDoublesSketch, DoublesUnion, HeapCompactDoublesSketch}
to build serialized compact doubles sketch for quantile state.
Looks like I will have to re-generate input data with apache datasketch. we are using druid internal version.
Open in Slack
PreviousNext
Powered by