Hey 👋 We're trying to set-up

Ingestion-time HLL

for our real-time Pinot table using HLL-sketch, to improve the performance of

DISTINCTCOUNTMV

based backend queries on our

multi-value column

i.e.

ids

. Schema config:

{
		"name": "hll_unique_ids",
		"dataType": "BYTES"
 },
 {
		"name": "ids",
		"dataType": "STRING",
		"singleValueField": false
 }

Transform config:

{
  "columnName": "hll_unique_ids",
  "transformFunction": "HLL(murmurHash64(ids), 12)"
}

We added

murmurHash64

because - the HLL() transform config was failing and wanted to use a function more compatible with the multi-value column. But whether we use HLL(), or HLL() & murmurHash64() chained functions, we get the below error. We're currently on Pinot 1.0.

error: Invalid TableConfigs: real_time_pinot_table. Invalid transform function 'HLL(murmurHash64(ids), 12)' for column 'hll_unique_ids'

Appreciate any help to understand what might be causing the HLL() transform function to fail. cc: @Sai Tarun Tadakamalla @ZEBIN KANG @Jessica Stewart

I don't think you can build one HLL for an MV column. Also, it is not a good idea to build one HLL per row (ie without aggregating)

I see! Would you recommend pre-computing the HLL sketches externally before the data reaches Pinot eg. in the Kafka producer?

Few options: 1. Use ST index to do the pre-aggregation - need to check if this works on MV 2. Use query time HLL function (this works on raw data also) 3. Pre-aggregated externally

Thanks @Mayank! Few follow-ups 🙇‍♀️ 1.

DISTINCTCOUNTMV

doesn't seem to be a supported function for Star-Tree index, but probably

DISTINCTCOUNTHLLMV

is supported? Do you know? 2. We tried query-time HLL function eg.

DISTINCTCOUNTHLLMV(ids)

but we're still facing server timeout issue 3. Pre-aggregated externally - a. would you be able to provide any recommended options for real-time data ingestion to Pinot? cc: @Sai Tarun Tadakamalla

1, From code it doesn't seem so

Rather than having a separate row for each id, we tried reducing it by having them listed in an array column

ids

.

you mean one column or one row for each?

one row for each

Can you elaborate why one row for each?

Number of elements in the array is not a fixed number...

what's the max number of elements?

I guess maximum number of elements are expected to be ~1000 (roughly)