@Subbu Subramaniam @Sajjad Moradi Can I come at 12:00 for lunch?

sure, can u be here by 12? thanks

a little after may also be fine

Yes, I'll be at the receptionist by 12

drive safely.

🙂

a little late is ok

i am at 408-329-8893

Thanks!

Just arrived the lobby

Adding @Xiang Fu

Adding @Kartik Khare and @Aman Khanchandani here who are working on this feature

@Subbu Subramaniam @Sajjad Moradi Can we set up a meeting to go over the design of the pauseless consumers. Please let us know what time you are available. This would help us answer the queries faster and also figure out missing gaps quicker

Our goal with this design is to keep things straightforward by minimizing the changes needed in the ingestion and query layers.

Can we meet tomorrow morning between 930 and 11 (say, 30m). Meanwhile, can you please take a look at https://github.com/apache/pinot/pull/14439

I have a doctor appointment in the morning, should be available at 10:30

Let me setup meetup from 10-11:30 then does that work?

Synced up with jackie and he'll be available, I have sent the invite for 10-11:30AM. Please accept cc: @Manish @Aman Khanchandani

I need to stop a little before 11, but i can be available from 10 onwards.

Hi @Subbu Subramaniam @Sajjad Moradi, we thoroughly evaluated both approaches (early marking segment ONLINE vs create child consuming segment). Initially I'm leaning towards the child consuming segment approach, but realize it actually comes with a big limitation in happy path which is caused by broker not aware of the child consuming segment: During the transitioning period of the child consuming segment detached from the parent consuming segment (after the controller creates the new CONSUMING segment but before it is detached), there could be inconsistent windows across components. E.g. broker queries the new CONSUMING segment from serverA, and the committed segment from serverB which hasn’t finished the detachment, it can get duplicate data.

This is one example problem caused by this hidden segment, and I'm afraid there are other similar issues.

Regarding the marking segment ONLINE early approach, the contract around it could be: given the start/end offset, server is able to bring the segment online - by sealing locally, download from peer, or re-consumption. As for the download mode, it is also doable if we ask server to catch up but not keep.

Complexity wise, we do need to make more changes, but we should not compromise on happy path

We can change the server code to count the new segment only once right? It should be an easy change in the segment selection code.

Broker might query server A for the new consuming segment, and server B for the previous committed segment. Server B doesn’t know broker already queried server A for the new consuming segment, thus will also serve the child consuming segment

Got it. I can only think of some hacky solutions at this time, but let me discuss this with Sajjad and see if there is a good way we can avoid this.

It seems that we prefer the approach in Sajjad's PR, but it has a race condition in the happy path. In order to fix it, I would suggest to change the broker segment selection. Assuming segments S7 is getting completed, and S8 has started in pauseless consumption mode. In the broker, we choose between selecting S7 only, and selecting S7 and S8 based on examining the last two segments ev on the instances. If the EV does not contain S8, then it is clear that a call to select S7 will include both (or, in the worst case, include S7 only if by the time the request goes thorugh the server has already transitioned S8 to CONSUMING state). This will take a bit of time to change/implement correctly, but I think we should be able to come up with a set of combinations that we should pay attention to, and validate that there will not be duoplication (not including the new segment is ok) Meanwhile, if it is urgent that you move forward with a solution, I suggest that you not change the existing protocol to add a flag for pauseless consumption. Instead, introduce a new end point in

LLCSegmentCompletionHandlers

class that accepts the completion request. This code path should bypass the protocol completely, change the idealstate as well a metadata as per your proposal. Of course, you need to handle segment download calls for the completed segment (maybe make them wait on the server until the upload is complete, or support download from in-memory rows. I suggest the former to keep it simple for now). You also need to handle restart of a server when there are two consuming sergments (I suggest holding the state transition for the second segment until the first one reaches online state).

This can indeed be fixed by modifying the broker logic, but this eliminates the main idea of hiding the child consuming segment as server local knowledge, and this solution is no longer simple. Similarly, another problem could be a rebalance without consuming segment, the new consuming segment could be assigned to a different set of servers. In this scenario controller needs to notify server to drop the child consuming segment. To solve the above problems, the child consuming segment needs to be a shared knowledge to all components, which is what Helix is designed for. We should stick with the state transition model to maintain shared state, or there will be endless of corner cases to handle.

Another point is that we should try to avoid complicating the happy path. If we can figure out a way to recover the segments not persisted but marked as ONLINE, the solution is much more robust

Hi folks, I was wondering how pause-less consumption deal with the failure case: winning server and follower servers are restarted during segment commit. As the previous consuming segment were marked as Online in IdealState, the StateTransition will be OfflineToOnline and further invoke the addNewOnlineSegment function. Since the deepurl is empty, that segment will run into Error State and not recoverable.