This message was deleted.

Segments are stored on disk and memory-mapped, and are in a format that is "proprietary to Druid" in the sense that it is a format that is primarily used by Druid. It is, of course, an open format, since Druid is open source! Data is processed in-place whenever possible, minimizing deserialization or copies. For details check out this talk:

Hi @Gian Merlino just to clarify, mmaping a segment doesn't mean loading that file into main memory right? And druid will first identify the rows in the segment files based on meta-data and then loading only that particular portion of segment into memory and do the necessary calculations on-demand correct?

mmapping is https://en.wikipedia.org/wiki/Memory-mapped_file; it won't necessarily load the entire file into main memory, although it may, if you actually use the entire file and if you have free memory. the OS page cache (https://en.wikipedia.org/wiki/Page_cache) handles it