https://pinot.apache.org/ logo
n

Neha Pawar

05/07/2020, 4:02 PM
tried with a parquet file with data having similar characteristics to yours @kish. It worked fine for me 🤔
Copy code
2020/05/07 08:56:29.835 INFO [SegmentIndexCreationDriverImpl] [main] Finished building StatsCollector!
2020/05/07 08:56:29.836 INFO [SegmentIndexCreationDriverImpl] [main] Collected stats for 36000000 documents
2020/05/07 08:56:29.859 INFO [SegmentDictionaryCreator] [main] Created dictionary for INT column: m1 with cardinality: 49, range: 1 to 49
2020/05/07 08:56:29.875 INFO [SegmentDictionaryCreator] [main] Created dictionary for INT column: m2 with cardinality: 64000, range: 0 to 63999
2020/05/07 08:56:29.879 INFO [SegmentDictionaryCreator] [main] Created dictionary for LONG column: count with cardinality: 1, range: 1 to 1
2020/05/07 08:56:29.884 INFO [SegmentDictionaryCreator] [main] Using fixed bytes value dictionary for column: d1, size: 11
2020/05/07 08:56:29.884 INFO [SegmentDictionaryCreator] [main] Created dictionary for STRING column: d1 with cardinality: 1, max length in bytes: 11, range: ucdmJUDhdcv to ucdmJUDhdcv
2020/05/07 08:56:29.888 INFO [SegmentDictionaryCreator] [main] Using fixed bytes value dictionary for column: d2, size: 200000
2020/05/07 08:56:29.889 INFO [SegmentDictionaryCreator] [main] Created dictionary for STRING column: d2 with cardinality: 8000, max length in bytes: 25, range: A to zzltgSI
2020/05/07 08:56:29.897 INFO [SegmentDictionaryCreator] [main] Using fixed bytes value dictionary for column: d3, size: 570000
2020/05/07 08:56:29.898 INFO [SegmentDictionaryCreator] [main] Created dictionary for STRING column: d3 with cardinality: 38000, max length in bytes: 15, range: A to zzzqDAvrzsUvncL
2020/05/07 08:56:29.899 INFO [SegmentDictionaryCreator] [main] Using fixed bytes value dictionary for column: d4, size: 16
2020/05/07 08:56:29.899 INFO [SegmentDictionaryCreator] [main] Created dictionary for STRING column: d4 with cardinality: 4, max length in bytes: 4, range: Is to o
2020/05/07 08:56:29.900 INFO [SegmentDictionaryCreator] [main] Using fixed bytes value dictionary for column: d5, size: 910
2020/05/07 08:56:29.900 INFO [SegmentDictionaryCreator] [main] Created dictionary for STRING column: d5 with cardinality: 35, max length in bytes: 26, range: BHYTfkbxibsOoZYRInA to ysLZeqUdcMzWRYlWueR
2020/05/07 08:56:29.901 INFO [SegmentDictionaryCreator] [main] Using fixed bytes value dictionary for column: d6, size: 27
2020/05/07 08:56:29.901 INFO [SegmentDictionaryCreator] [main] Created dictionary for STRING column: d6 with cardinality: 3, max length in bytes: 9, range: GNhWQNwor to a
2020/05/07 08:56:29.902 INFO [SegmentDictionaryCreator] [main] Using fixed bytes value dictionary for column: d7, size: 35
2020/05/07 08:56:29.902 INFO [SegmentDictionaryCreator] [main] Created dictionary for STRING column: d7 with cardinality: 5, max length in bytes: 7, range: MlMtQ to z
2020/05/07 08:56:29.902 INFO [SegmentDictionaryCreator] [main] Using fixed bytes value dictionary for column: d8, size: 15
2020/05/07 08:56:29.903 INFO [SegmentDictionaryCreator] [main] Created dictionary for STRING column: d8 with cardinality: 5, max length in bytes: 3, range: cLf to x
2020/05/07 08:56:29.903 INFO [SegmentDictionaryCreator] [main] Using fixed bytes value dictionary for column: d9, size: 4553
2020/05/07 08:56:29.904 INFO [SegmentDictionaryCreator] [main] Created dictionary for STRING column: d9 with cardinality: 157, max length in bytes: 29, range: A to zwaGwwzPkH
2020/05/07 08:56:32.072 INFO [SegmentDictionaryCreator] [main] Created dictionary for LONG column: timestamp with cardinality: 29441824, range: 1587366000002 to 1587452399994
2020/05/07 08:56:32.148 INFO [SegmentIndexCreationDriverImpl] [main] Start building IndexCreator!
2020/05/07 08:59:05.704 INFO [SegmentIndexCreationDriverImpl] [main] Finished records indexing in IndexCreator!
2020/05/07 08:59:05.787 INFO [SegmentIndexCreationDriverImpl] [main] Finished segment seal!
have you set indexing/ special dictionary configs in the table config?
k

kish

05/07/2020, 4:25 PM
I was using same config as for CSV
n

Neha Pawar

05/07/2020, 4:27 PM
1. can you share the table config 2. can you try with https://github.com/apache/incubator-pinot/pull/5348 so that there's more log?