This message was deleted.

Hi Koustav, When the the real-time data is first read from the streaming buffer (e.g. Kafka/Kinesis) it is stored in row format in a memory buffer. Each row (or set of rows) that is pulled from the stream is parsed and placed in this buffer immediately, so if a query needs to read from this time chunk it can read it immediately directly from this memory buffer, although from another processing thread than the ingesting thread. The "persist" is an intermediate form of storage of the real-time data ... in this stage the content of the row buffer is periodically converted to columnar format and stored in a small file on local disk on the ingesting node. I don't know the exact format of this file but you can consider it as a "mini segment". These persist operations can happen very frequently compared to the number of times an actual segment is built. Dozens or even hundreds of persisted column store files can be persisted to disk. And again, any query that needs to look at this time chunk will (via separate thread) read through all of these locally persisted files to scan for query results. The final phase is to build a segment. This happens when you either exceed a time limit (intermediate handoff period or task duration) or exceed a size limit (# rows/bytes per segment). In this phase another thread in the ingestion tasks gathers up the persisted files, merges them into a real segment and pushes it to deep storage. So, to answer you question in short, the "real-time" data is stored in both row and columnar format, at different times.

Thank You for your response. Two more things I would like to know. • Is it documented somewhere ? • Is there any index on this data on memory buffer ?

I don't know if it's in the official documentation, you would have to search around on the Druid docs HERE or search through the Github repo HERE ... Otherwise I suggest you look at the learn.imply.io Basics and Ingestion courses (they're free) ... they may take a few hours each to go through but they are very informative and offer online lab exercises. I also suggest you poke around the Imply YouTube channel HERE ... there are lots of very informative videos there as well. Fyi @Sergio Ferragut is working on a more detailed presentation on streaming ingestion specifically ... I don't know if he has any material to share yet but I am cc'ing him here just in case. Thanks. John