Is it possible and a good idea to use offline tables and create segments directly in flink, or better to write the data from flink to Kafka and use a real time table? We generate the data in flink but given the size writing directly to the segment store might have advantages. We could use a short-ish retention period as well.

Piping Kafka directly to Pinot would keep your data fresh. As opposed to batching to create segments and then pushing. However, if that is not a concern for you, you could write segment from flink (doing so frequently, like every few minute or so) might then be an anti pattern.

For high volume (100s of segments per second) is batch load an anti-pattern if the latency is acceptable? Or would that overly strain the system?

Yes that would be an anti pattern for sure

Ok. I thought so, but appreciate the confirmation.

You bet 👍