Is there any limitation on the size of a single re...
# general
Is there any limitation on the size of a single record written into
? Our average records are about 6KB when stored in
, but can reach up to ~50K in edge cases
Pinot is columnar. Is the size due to wide schema or columns that have large data?
If former, no issues If latter, what’s the data type of those columns?
A combination - we have about 90 fields, some are numeric, others are short strings - the rest are potentially large strings (e.g. HTTP Request/Response Headers) - that can reach several KB for a single field. Also, we keep nested records within each record - and each outer record can contain several nested records - which also increases the size of a single column
Also - for some of these fields, we do not need indexing (e.g. Request Headers) - I just need to be able to find them based on other dimensions
yes, you can apply snappy compression on such columns
indexing a column is optional in Pinot
👍 1
For Strings, there is a default max length (iirc 512), but can be overwritten:
👍 1