We created an offline table and are using a job spec to load or update segments, where all input files are in CSV format. Is it necessary to use JSON, Proto, or Avro as the input format in production? Will these formats consume more disk space or cause performance issues? I’ve noticed that CSV uploads segments faster, and we want to process around 3.5GB of data per hour.

Pinot is input format independent, you can pick what works best for you.

Thank you. This means Pinot handles internal compression. However, when using another format like Protobuf, I would need to serialize the data during the write process. If the message structure is dynamic, I would also need to update the Protobuf definition accordingly.Its not a easy task in production to provide proto file format. hence diskspace remain same what i understood