Hi! What is the reason for having Pinot queries implicitly default to

limit 10

? Is there a way to remove the limit in the query without specifying an arbitrary limit value?

Without a default, a simple looking query (to some) like

select *

could end up fetching all of Pinot's data?

Ok, so it's purely for safety. Is there a way to remove the limit without specifying an arbitrary limit value?

No. What's your use case? Is it purely for convenience, or your application does indeed want to pull out all data from Pinot?

Data is streamed into Pinot and over the course of the day, application instances want to get all the messages of interest starting from 0000 hours. Number of messages differ according to the query criteria. We can put in a "large enough" limit value, but that is an assumption it will suffice in the future.

There's the syntactic challenge there. If we make default unlimited then we have the problem above. Once we limit the default, then any non-default value has to be specified due to syntax

I think if anyone does it explicitly, they know what they are getting into.

Yes, the worry is about new users who don't know that they may end up fetching entire data from Pinot just by doing

select *

. Since most new folks won't specify

limit

I think it's fine to retain the implicit default limit. Then offer something to the effect of

limit unlimited

for users wanting to query with knives.

I see. What's the SQL way of doing it?

set a session parameter?

So, there isn't a cross query state

how about setting HTTP headers?

Typically, folks set a really large value like 1M, or 100M. But they also are aware that doing so means that if they run a really expensive query they might stress the cluster load.