This message was deleted.

Hi Dejan, I can't answer for specific directory paths, but I can respond to some extent to your questions: 1. The Kafka (streaming) ingestion engine runs the data through three stages: a. When the data first comes from Kafka it is parsed to understand the schema and then it is appended to an unindexed row buffer in JVM heap, here it can immediately service queries.

maxRowsInMemory

,

maxBytesInMemory

and

intermediatePersistPeriod

limit how much data can accumulate here before being "persisted" to local disk b. Data from the row buffer is periodically compiled into a full segment format (columnar, dictionary encoded, compress) and stored as a "mini-segment" or what we call "intermediate persist", file, which is still located on local disk for the MM that is ingesting it. Many of these files will likely be created before building and publishing a "real" segment to deep storage. The data in phases 1 and 2 above are both considered as "real-time segments". c. When the amount of data accumulated reaches a larger threshold (e.g.

maxRowsPerSegment

), all of the intermediate persist files will be compacted into a single "real" segment and pushed to deep storage. This process is generally called the "Handoff", transferring the segment from real-time to historical status. 2. Ingestion node failure. a. If you have ingestion replicas set up they both operate in tandem, syncing to the same Kafka offsets, so when one replica dies the other one can continue on. If they are both operating without error then when the first replica finishes publishing a segment the second one is told to abort that segment, but both replicas would continue ingesting the next segment. b. If you do not have ingestion replicas (i.e. only one task) then if it fails then at worst case it will have to go back to the Kafka offset related to the last published segment, and start over again from there. There may be an intermediate savepoint related to the intermediate persist files stored on the MM local disk, but I don't know enough about that part of the ingestion to say whether the recovery can pick up from that point ... someone else will have to chime in here 🙂 3. Someone else will have to clarify this one too ... I don't know for sure what happens if the cluster loses contact with Deep Storage. a. Ingestion -- if the ingestion tasks cannot publish a segment to deep storage I imagine you will see ingestion task failures resulting in Kafka lag and retries b. Historicals have deep storage segments loaded onto their local disk, so I don't know if they can continue to service queries even if they cannot contact deep storage anymore. Hoping someone else can fill in the gaps here. Thanks. John

Thank you very much for your reply and detailed info John! I really appreciate it and indeed hope someone would be able to expand on stuff you were not able to help with.

Hi Dejan, "segment-cache" can mean one of two things: • On the Historical side, when published segments are loaded from Deep Storage into the historicals, sometimes people call that "caching" the segments. But more likely it is ... • The Coordinator and Broker nodes both cache a list of metadata for all segments that are "active" in the cluster. This uses heap memory for both Coordinator and Broker processes and should be included in the memory sizing calcs on the Basic Cluster Tuning page. By "mini-segments" am referring to the ingestion-side, intermediate persisted files ... these are stored in columnar, indexed and compressed format, just like a real segment, but they have not been combined and published yet, so they are still considered to be part of the "real-time segment" category.