Apache Pinot

Hi Devs - previously I’d discussed w/<@U031K0TCB6D> the idea of leveraging <https://github.com/durner/AnyBlob|AnyBlob> to maintain all Pinot storage in the cloud, versus our current approach of local plus deep. See <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855|FLIP-423> and more specifically <https://issues.apache.org/jira/browse/FLINK-34975|FLINK-34975> for something similar that’s started over in Flink-land, re storing all state in the cloud, versus using RocksDB for local and then S3 (or DFS) for persistence.

Would this be a configurable option in addition to the present system?

Hi Tim - yes, but you might be able to implicitly change behavior by using a filesystem protocol for the destination directory of “local files”.

Plus you’d add the (e.g.) “ab://” filesystem protocol for AnyBlob.

I guess you’d also want some kind of support for “direct push”, where you’re just telling Pinot there’s a new or updated file at `<ab://blah>`, which triggers the controller to tell various brokers about the file(s).

The <https://www.vldb.org/pvldb/vol16/p2769-durner.pdf|AnyBlob research paper> looks very interesting.

Yes, I thought that was a really good deep dive, and useful for anyone working with cloud storage.

In particular, the summary:
&gt; Our DBMS Umbra, equipped with our download manager and caching disabled, achieves similar performance on a single instance as large configurations of state-of-the-art cloud database systems that cache data on local SSDs. Our fast and low overhead networking integration facilitates switching instances without performance clis, improving elasticity. As switching comes without performance clis, our approach is able to better utilize spot instances, available at huge discounts.
Leveraging spot instances is always appealing, when trying to lower costs.

If you read their paper, leaving aside low-level network tweaks, the interesting bits to me were:

1. How they use io_uring to manage multiple connections per thread asynchronously.
2. Dynamic, adaptive allocation of threads between downloading and processing tasks.
3. Zero user-space copies, by including space for HTTP headers etc in the receive buffer.
