Hi Team, I am trying to execute pinot ingestion job with segment type ‘fixed’ . Input data is on different directory - dir Batch1 - part-**.avro dir Batch2 - part-00**.avro etc . I would like to generate segment with fixed type segment name e.g. - segment name - Batch1 , segment name - Batch2 etc. Can any one please help with the same?

How are you doing ingestion? Through preconfigured batch jobs?

yes, but it will create single segment after taking all the avro file. My requirement is to create segment per directory. e.g. segment name should be batch1 , batch2 etc.

Will the directories be fixed? Or will them be dynamic, like, in the future you want a "Batch3" directory and want to automatically consume that as a segment? If they're fixed, it's just a matter of configuring new jobs; If they're dynamic, then I don't know how to solve that...

Do you need single segment for each directory? What if the directory has too much data that it doesn’t fit in one segment? Or are you suggesting batch1 as prefix?

Yes , I want single segment for each directory and each directory has single file.If two much data that it doesn’t fit in one segment in a particular directory , it will create multiple segment with prefix batch1 .

Currently we don’t have it but there is an ask to pick up time column value from folder path. Would you like to contribute? cc: @Xiaobing

we don’t have any time column value

Yeah, but the two asks might have a bit of overlap

I found there is a segment name generator type called ‘inputFile’, which seems fairly new and not integrated with all necessary places yet. But it might be interesting to your use case: Basically, it allows one to configure a regex to extract part(s) of the file path to generate the segment name. The current implementation looks pretty limited as it has to map 1 file to 1 segment. But for your use case, I think it may be extended to generate a segment prefix for segments from one directory and append a sequenceId to differentiate those segments.