Is there any limitation on the size of a single re...
# generalr
Ron Kitay
03/16/2021, 5:19 PMIs there any limitation on the size of a single record written into
PinotAVROm
Mayank
03/16/2021, 6:55 PM
Pinot is columnar. Is the size due to wide schema or columns that have large data?
Mayank
03/16/2021, 6:55 PM
If former, no issues If latter, what’s the data type of those columns?
r
Ron Kitay
03/16/2021, 7:28 PMA combination - we have about 90 fields, some are numeric, others are short strings - the rest are potentially large strings (e.g. HTTP Request/Response Headers) - that can reach several KB for a single field.
Also, we keep nested records within each record - and each outer record can contain several nested records - which also increases the size of a single column
Ron Kitay
03/16/2021, 7:29 PMAlso - for some of these fields, we do not need indexing (e.g. Request Headers) - I just need to be able to find them based on other dimensions
k
Kishore G
03/16/2021, 8:29 PM
yes, you can apply snappy compression on such columns
Kishore G
03/16/2021, 8:29 PM
indexing a column is optional in Pinot
👍 1
m
Mayank
03/16/2021, 8:59 PM
For Strings, there is a default max length (iirc 512), but can be overwritten: https://docs.pinot.apache.org/configuration-reference/schema#advanced-fields
👍 1