Is there a way to configure the desired segment size and segment creation job? I've got some small avro files and the job seems to create a segment per file, is this how it works? I'd like to squash these small files in to one bigger segment. Do I need to pre-process them myself before running the job?

There is a meetup today on the segment merging and roll up which might help in your case https://www.meetup.com/apache-pinot/events/280616089/

I'm registered and I plan to attend, thank you