Hi All, is there a possibility to soft delete Pino...
# generalr
Rohit Yadav
12/26/2022, 7:20 AMHi All, is there a possibility to soft delete Pinot tables and have an instant recovery if needed?
OR
Can I disable a table from only being queried but allow ingestions to happen?
The requirement arises from table management use cases where I do not want a prod table to be deleted immediately but have a grace period for recovery. I came across an API to disable Pinot table that stops querying and ingestion for a table but recovery can be tricky in the case of Realtime tables(stream data getting evicted and recovery is not instant on high qps stream).
n
Neha Pawar
12/26/2022, 5:51 PM
You could set maxQueriesPerSecond to 0, that disallows queries https://docs.pinot.apache.org/configuration-reference/table#quota
I've seen this hack being used at production setups to momentarily disallow queries without affecting anything else.
Not sure what behavior you wanted you clients to see in your case, but this suggestion will result in the clients receiving 429 error
r
Rohit Yadav
12/27/2022, 6:03 AMThis information is helpful. With some metadata, I think it will be easy to figure out if 429 is due to soft delete or it is actually rate limited. Thank you.
Rohit Yadav
12/27/2022, 9:09 AMSo the config validation won't allow me to set exact 0 value but it works if I put something like 0.0000001 which I can work with. I hope this is an expected behaviour.
Rohit Yadav
12/27/2022, 9:16 AM@Neha Pawar Could you also shed some light or point to a doc which explains how this rate limit is implemented? Wanted to understand how reliably does it rate limits?
➕ 1
n
Neha Pawar
12/27/2022, 5:06 PM
@Jack @Sajjad Moradi is there a user facing doc for this feature already?
j
Jack
12/27/2022, 10:50 PMI think the only explanation of this rate limiter in wiki doc is this: https://docs.pinot.apache.org/configuration-reference/table#quota
And yes, for now we don’t allow the total quota to be set to 0:
https://github.com/apache/pinot/blob/master/pinot-spi/src/main/java/org/apache/pinot/spi/config/table/QuotaConfig.java#L65c
Setting a very small number like 0.00000001 should be good for your use case.
👍 1
s
Sajjad Moradi
12/28/2022, 1:12 AMThe behavior & implementation of consumption rate limiter is described here:
https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion#throttling-stream-consumption
Sajjad Moradi
12/28/2022, 1:16 AMAlso if you'd like to stop stream consumption, please don't use a small number for topic.consumption.rate.limit parameter. Instead, use the "pause" endpoint on controller to effectively stop consuming from topic immediately.
Sajjad Moradi
12/28/2022, 1:16 AMMore info on pause/resume endpoints can be found here:
Sajjad Moradi
12/28/2022, 1:16 AMr
Rohit Yadav
01/02/2023, 6:20 AMThanks for the pointer @Sajjad Moradi, but my usecase is to not pause the ingestion. It is to disable the queries on the table. Do you see an issue with rate limiting queries(to achieve blocking) on a table with a very low value?
s
Sajjad Moradi
01/03/2023, 6:55 AM@Rohit Yadav I misread it as consumption rate limiter 😅
Sajjad Moradi
01/03/2023, 6:55 AMFor query rate limit implementation, Guava’s RateLimiter is used:
https://github.com/apache/pinot/blob/989df641b7d29e18773a668663b0be894835d24f/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java#L326
Each broker’s rate is simply calculated by dividing table’s QPS by number of active brokers:
https://github.com/apache/pinot/blob/989df641b7d29e18773a668663b0be894835d24f/pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/HelixExternalViewBasedQueryQuotaManager.java#L194
Guava’s rate limiter doesn’t take zero as the rate limit parameter. So if setting a very small number theoretically works for you, rate limiter will honor that.
r
Rohit Yadav
01/03/2023, 6:57 AMHi Sajjad, thanks a lot. Was going over the same. I will check if there are any drawbacks for setting extremely low QPS rate limit for tables with high QPS. Will share the findings here.
👍 1
Rohit Yadav
01/05/2023, 6:36 AMThe rate limiter is guava's SmoothBurty rate limiter which is a token bucket rate limiting algorithm. For me it does not look like a good solution as the broker restarts will reset the rate limiters. Moreover, since the ratelimit qps will be divided over x brokers, it means that first x calls will still go through and couple this with broker restarts, then it does not look like a good solution to block all queries on a table for a long duration. For blocking for small durations it is good solution.