Hi all. RocksDB supports ordered iteration and range scans natively. Has the Venice team ever considered exposing range queries (fetch all keys within a

[lo, hi]

range) as a first-class read API on top of it? What we understand about why it's not there today: - Stores are hash-partitioned on the full key, so logically-adjacent keys scatter across partitions. There's no bounded home for a range, unlike point/batch-get. - Keys are opaque (Avro-serialized) bytes, and Avro's encoding isn't bytewise-order-preserving, so even a within-partition RocksDB iterator wouldn't map to a logical range. - A general unbounded scan would break the predictable-tail-latency / RCU quota model. Curious whether it's ever been discussed or prototyped, and if there's a design doc or RFC. If it was ruled out, I'd love to understand the main reasons. Thanks! cc @Felix GV @Zac Policzer @Koorous Vargha @Hubert Puszklewicz @Amre Shakim fyi

It's actually there already, but only on the Da Vinci Client.

Yep. It's exposed through the DaVinciRecordTransformer, but it will iterate through every key in your subscribed partitions. https://github.com/linkedin/venice/blob/1a16109a140e9c6109ae738c2d919a33389636ff/clients/da-vinci-client/src/main/java/com/linkedin/davinci/client/DaVinciRecordTransformerUtility.java#L164-L165

Oh, is it in DVRT nowadays?

That's what I know of. Is it supported in regular DaVinci too?

It wasn’t this way previously, unless I’m getting confused

I don't see it on the interface. Maybe Zac knows

But in the time I've been here I haven't seen an ask to support this for remote queries, but I'm curious what's the use case for this?

Venice naturally overlaps with DynamoDB use-cases, where it allows to provide a sortKey and query data based on a given range. We internally use it to query values for some keys based on datetime ranges.

It's supported I think 'roughly' as part of the partial key lookup feature which is part of davinci. We were using it internally for the CDC at one point

The first CDC client or the DVRT one?

Yep, partial key lookup, exactly

The first CDC client or the DVRT one?

I.... seem to no longer remember 😨

Is the partial key thing related to read compute?

Yeah it's a read compute operation. It's kind of inspired by espresso's interface. In espresso keys are path based (member-id/mailboxid/messages) and you can list all messages under (member-id/*). So it's like prefix search.

I see, so it looks like we can have, for example, composite key with (key, sortKey), and use

computeWithKeyPrefixFilter

. But I don't see range query support in particular - so if we want, for example, fetch keys within a certain (datetime1, datetime2) range, the daVinci client would only be able to fetch all the records for that particular key, and the further filtering would need to be done one the client side.

I believe so

Hmm… it supports passing predicates including greater than, lower than (I believe)

Even though those predicates exist,

AvroComputeRequestBuilderV4

explicitly says that only AND and EQUALS are allowed (link) 😞

Dang