Hi Team , as part of my poc , I am trying to a loa...
# generaln
Nikhil Thomas
11/06/2023, 10:44 AMHi Team , as part of my poc , I am trying to a load pinot table with data ( in which one of the i have considered as partition column)
Nikhil Thomas
11/06/2023, 10:46 AMAnd defined the table config like this -
Nikhil Thomas
11/06/2023, 10:47 AMBut still it is creating only on segement, as per my knowledge based on the routing column, it should create segement accordingly.
Nikhil Thomas
11/06/2023, 10:47 AMCan anyone help on what I have missed here
x
Xiang Fu
11/06/2023, 11:01 AM
Pinot creates one segment per file. If you have partition scheme, you need to prepartition the table into different files
n
Nikhil Thomas
11/06/2023, 11:02 AMMeans one source file will Create one segment ...?
Nikhil Thomas
11/06/2023, 11:05 AMIn that case , if my source files is not in proper way ( skewness , unequal distribution, lot of small file) then how pinot segment will be give good performance during query execution
x
Xiang Fu
11/06/2023, 11:08 AM
It depends on the partition key. Usually users can over partition, then pinot can locate multiple partitions on same server for serving.
Xiang Fu
11/06/2023, 11:09 AM
Each partition can have many segments for data splits if one partition is skewed
Xiang Fu
11/06/2023, 11:10 AM
Pinot itself has a minion job, which you can configure to merge small segments later on after ingestion.
Xiang Fu
11/06/2023, 11:11 AM
Those happened in background task
m
Mayank
11/06/2023, 10:08 PM
Side note, better to use Murmur and hashcode.
n
Nikhil Thomas
11/07/2023, 7:56 AMOk thanks