Hello slightly smiling face When ingesting Batch data + data

Question

Hello slightly smiling face When ingesting Batch data + data partitioning Parquet using a key that key is missing from the parquet file parts makes sense However from what I ve seen Pinot cannot find that key then and fails to generate the segments My current workaround is to duplicate the partition column Is that a known issue possible to adjusts settings

Xiang Fu · Answer

do you have stacktrace for the job The key should be a column in your data even in batch side

Jonathan Meyer · Answer

The schema contains several columns including `dateString` which it is partitionned on This creates parquet partitions without this key

Jonathan Meyer · Answer

Actually now that I look at it again I m seeing ``` file kpi data raw date=2020 11 30 ab331a05255849bf811a173a380aaf1d parquet``` Not `dateString=XXX` Curious but I ll check that

Xiang Fu · Answer

oic cause the default null string caused the parsing failure

Xiang Fu · Answer

this date has to be one column in your parquet file

Xiang Fu · Answer

if you generated this parquet from spark you can add the partitionkey as a column as well

Jonathan Meyer · Answer

If the parquet partitioning key was the one expected by Pinot `dateString` it would have worked right Pulling `dateString` values from the file paths

Xiang Fu · Answer

yes

Xiang Fu · Answer

the error says the job tries to generate the partitionkey but got null value

Xiang Fu · Answer

so it s failed

Jonathan Meyer · Answer

Makes sense thanks < Xiang Fu>

Jonathan Meyer · Answer

On an unrelated note I ve opened an issue on Python pinot db driver let me know what you think when you ve got the time

Xiang Fu · Answer

sounds good