Hi, "I am setting up Apache Pinot 1.4.0 with a REALTIME table consuming from Kafka. My understanding is that as Pinot consumes messages from Kafka, the incoming row data is written to

.0

files (in the consumers directory) which act as a Write-Ahead Log (WAL) buffer for the currently consuming segment. This continues until a flush is triggered, at which point the

.0

file is sealed, converted into an immutable Pinot segment (columnar format with indexes), and a new

.0

file is created for the next consuming segment. With this understanding, I noticed that when the server starts, these

.0

files are created immediately before any data arrives — one per Kafka partition. For example for a table with 3 partitions I see them under:

data/server/index/table_REALTIME/consumers/

-rw-r--r--. 1 pinot pinot 536870912 Apr 21 16:20 table__0__0__20260421T1050Z.0
-rw-r--r--. 1 pinot pinot 536870912 Apr 21 16:20 table__1__0__20260421T1050Z.0
-rw-r--r--. 1 pinot pinot 536870912 Apr 21 16:20 table__2__0__20260421T1050Z.0

When I check with

ls -lh

they always show 512MB regardless of what I set for

realtime.segment.flush.desired.size

or

realtime.segment.flush.threshold.rows

. However when I check with

du -sh

they only show 128KB:

128K    table__0__0__20260421T1050Z.0
128K    table__1__0__20260421T1050Z.0
128K    table__2__0__20260421T1050Z.0

My questions are: 1. Is my understanding correct that

.0

files act as WAL buffers for incoming Kafka data before segment commit? 2. Is 512MB a hardcoded pre-allocation size in Pinot regardless of flush settings or can it be configured? 3. Does

flush.desired.size

only control WHEN the file is flushed/sealed, not the pre-declared file size? 4. As real data arrives from Kafka, will

du

grow until it hits

flush.desired.size

and then trigger a flush — meaning the maximum real disk consumed per

.0

file equals

flush.desired.size

? 5. After flush, a new

.0

file is created — does it again show 512MB in

ls

but 128KB in

du

? Also what exactly is stored in those initial 128KB — is it column metadata, index structures, or something else?" @Mayank, @Xiang Fu

Short answer: your WAL mental model is wrong. The

.0

files are the mmap'd off-heap backing storage for the mutable consuming segment's index buffers (dictionaries, forward/inverted indexes). They are not a Kafka WAL and are not replayed on crash — recovery happens via the committed Kafka offset in ZooKeeper. Q1 — Is

.0

a WAL for Kafka rows before commit? No. Rows are indexed into `MutableSegmentImpl`'s off-heap structures, which happen to be mmap'd onto

.0

. On crash, the

.0

file is discarded on restart (see — stale files matching the segment prefix are deleted) and consumption resumes from the committed Kafka offset in ZK. Q2 — Is 512 MiB hardcoded or configurable? Hardcoded.

MmapMemoryManager.DEFAULT_FILE_LENGTH = 512 * 1024 * 1024L

at . The file is mmap'd as a sparse file, which is exactly why

ls

shows 512M (apparent size) but

du

shows only the physically allocated blocks. No config flag changes it. If a single consuming segment needs more than 512 MiB, a second file

.1

, then

.2

, is allocated — the suffix is just a running counter. Q3 — Does

flush.desired.size

only control when to flush, not the pre-declared file size? Correct. Flush thresholds —

realtime.segment.flush.threshold.rows

,

.time

,

.segment.size

,

.segment.rows

(see ) — are evaluated in

endCriteriaReached()

inside . None of them look at the

.0

file's on-disk footprint.

flush.threshold.segment.size

targets the final immutable tar.gz segment size, not the mmap file. Q4 — Does

du

grow until it hits

flush.desired.size

and then trigger a flush? Partly.

du

grows as pages are touched (ls stays at 512M because the mmap is sparse), but flush is triggered by rows / time / index-capacity / target segment size, not by the

.0

file's physical size. In normal setups the segment commits well before 512 MiB is touched; if it doesn't, a

.1

file is added. The ceiling on physical

.0

size is whichever flush trigger fires first, capped at 512 MiB per file. Q5 — After flush, does a new

.0

appear (512M ls, small du)? Yes. On commit:

RealtimeSegmentConverter

builds the immutable segment, it's uploaded to the controller, the old

MmapMemoryManager

is closed and its `.0`/`.1`/… files deleted, and a fresh

MmapMemoryManager

for the next consuming segment (new segment-name sequence number) allocates a new 512 MiB sparse

.0

. What's in the initial ~128 KiB? Per-column off-heap initialization: initial dictionary pages, forward-index header pages, null-value vectors, min/max trackers, etc., allocated when

MutableSegmentImpl

is constructed. These are the only pages the kernel has actually written, so

du

reports ~128 KiB. The rest of the 512 MiB is virtual address space with no disk blocks backing it yet.

Thanks for the detailed explanation!