What is the performance implications of defining a dimension field in a schema as a string with a max length of Integer.MaxValue (2GB if all space is fully used)?

Will there be filtering on this column? If so, may be better to use JSON if it is indeed JSON.

There could potentially be some filtering, yes, though right now the use-case is simply to extract fields within this json field. What do you mean by "use JSON if it is indeed JSON"?

Use JSON indexing

I had it initially but by default Pinot's String are set to 512 chars in length. For the index to truly be useful I need to set this field's length to Integer.MaxValue (2^31-1 chars). I'm trying to understand if having fields this large should be a concern.

What's the max size?

2³¹ -1, realistically I think the largest I've seen was around ~20k chars

Yeah, it will have perf impact. Do you really need the entire 2GB JSON blob? Or is there specific fields you can extract out from it?

Can you quantify (very roughly) what the perf impact would be?

I don't have any concrete numbers for that. I'll have to see the overall data size, and query

My consumers do, yes.

I see

< 1 second for queries which access json blob. <100ms for top-level concrete fields.

Ok, may have to do some quick test for the largest json blob.

Preliminary results tell me the largest json blob is 40500 chars long.

I meant like actually querying

I am, it just takes a while to get long-term historical data.

I'd say just do one day may be? then we can extrapolate?

Largest in the past month was 40k

Of course, with actual data, it will be more accurate.

Unfortunately I can not share the actual data, it may contain PII

Oh I was suggesting for you to run it

Ah I see now what you meant, I'll try that tomorrow probably.