So we have data that contains an event_id that is obviously very high cardinality, a unique record per row. The thing is, we don't really have any use at all for the field other than for audit purposes so we're considering getting rid of it. But let's say we did keep it and we employee the pinot managed offline flows to roll things up. Can you ignore a field during rollup so that you keep the detail for a time but as you move to offline you can default it to something so that it basically gets rolled up?

What’s the concern that makes you want to drop or default that field? If it’s size, you can have a field with no index/dictionary, in which case it won’t take up much space…

it's just a high cardinality field that I'm arguing we don't need to have in pinot. But in our discussions some people feel we need it for auditing.

And the issue with high cardinality is?

space

I would suggest defining a schema/table where that field has no dictionary or index, and then see if you save a significant amount of space (versus removing it).

well what about the rollup component?

You are thinking about it the right way.. you can remove or make event if null in the roll up

ohhh so the rollup would allow me to null as I move the data?

I don’t know about that

ok

But should be easy to add a transform function

OK, got it - I took a quick walk through the code, and row equality during a rollup (reduce) is based on “sort fields.”

what my options are there

Yeah

I def will and thanks!

Design wise I don’t see any big change adding a transform on a column

@Jackie - what defined equality between rows when doing a rollup?

If this works as I think it might I could have the best of both worlds maybe. A certain amount of my data is in realtime table w/ the event_id for potential auditing, then as I move to offline table I default event_id to 0 and get good rollup.

@Stuart Millholland It is absolutely reasonable. We don't support it currently, but it is doable. Essentially we need to add a new task config to skip some columns when running the task in

ROLLUP

or

DEDUP

mode. Internally we will fill default values to these columns so that they won't be considered. Can you please help file a github issue describing the requirements?

yep, will do!