you can just send us the urls right?

mcvsubbu@gmail.com works for me

better, ssubrama@linkedin.com

Please check if you can access the url.

yeah i have the access

i'll try to join as well, neha.pawar18@gmail.com

thanks. Neha, it seems that the url is enough -- i removed any domain restriction on the meeting. So email is not needed. Can you just access the link to see if you can access it?

i havent gotten a chance to look through the PR yet. Please meet without me. i'll sync up with Ting if i have questions later

Is there a PR? I thought it was a design discussion

PR 4914 is the PR.

Sorry folks. I have a Production emergency issue right now. Can we talk around 10:30am?

Sorry for the short notice.

ok

I have added a note in the design doc with a question

can we meet now?

@User @User

I just joined the ZOOM. feel free to join when you are ready.

ah ok, joining

@User the segment fetcher based on ExternalView is here in the PR: https://github.com/apache/incubator-pinot/pull/4914/files#diff-39b1ee00ad2903f060a124b6fd49b5d5

Just want to put a quick note from my side after today's discussion about the various smaller PRs to work on for this feature. Thanks for the inputs.

1. New Server Rest End Point to download segments. 2. A new or refactored SegmentCommitter class to encapsulate commit policy and segment upload   * A new segment uploader class could be defined here too.   * This is probably the biggest change needed over the current PR and I will also need to look into the codes again. 3. PeerServerSegmentFetcher to download segment from peer servers if not found in deep store   * Use retries to handle EV update lag   4. Controller side changes 5. RealtimeValidationManager change to let servers to upload missing segments in deep store for last C (say 5) segments per partition.

@User I have updated your PR 4914 with a comment on what I think should be the steps. Please take a look and let me know. If done in small increments, i can certainly help you get the feature in quickly. Be sure to get the architectural changes out in a discussion so that we dont have to discuss those in PRs (e.g. new uris, protocol changes, placement and naming of new config items, etc.). These are harder to change once implemented, so it is good to have a discussion on these items independent of a PR (your design doc is a great place)

@User Thanks. Definitely small PRs will make things easier for review. I am going through your proposal now and they make sense. I will reply and work on breaking the POC into smaller ones.

thanks , and i will promise timely reviews.

@User I updated the https://github.com/apache/incubator-pinot/pull/4914 to clarify the steps you proposed earlier. I have some question about Step 5 on why it is needed and alternatives. The rest looks good to me. Working on step 1 right now.

PR of the 1st change is ready for review: https://github.com/apache/incubator-pinot/pull/5221

API changes proposal also sent out in mailing list and slack general channel.