Apache Pinot

Hi!  Question: What would be the consequences to creating many small segments, say via the segment upload API, say one every 10 seconds, for many months?

Not much, as long as you have a job to merge them in the background

Thanks <@UDQU92KBK>.  Do you have docs on how to merging segment?  Is there anything built-in?

There’s a minion merge rollup task you can check in the docs page.

<https://docs.pinot.apache.org/operators/operating-pinot/minion-merge-rollup-task#:~:text=The%20Minion%20merge%2Frollup%20task,storage%20and%20the%20query%20performance.|Minion merge rollup task - Apache Pinot Docs>

But sounds like you might be better off going the streaming route

We'd like to allow our users to push data to Pinot without operating _their own_ kafka.  Hence, segment upload.

In that case you might want to evaluate this: <https://www.startree.ai/blog/introducing-writeapi-the-interface-of-direct-write-into-pinot>

Thanks.  And that's part of released OSS Pinot, not just StarTree SaaS?

I think the primitives are there in OSS. But the solution as a whole (which requires running Kafka) won’t be.