@User @User @User @User @User

@User I suppose you are co-ordinating the efforts of the release certifier? Wanted to bring to your notice a ticket I had created long time ago, and also had a design doc. https://github.com/apache/incubator-pinot/issues/4854

@User is currently working on this.

@User we need to unblock @User. Will you be available for a quick call sometime?

I will drive this

thanks

just talked to jackie also, we can touch base later.

@User I don't believe any work has started on the performance side. So, it is best that we get going with the functional and compat testing. I suggest you NOT use go language, but let us stick to Java and Shell. Let me know when you are available, and we can do a quick conf on how to start merging code.

<!here> It would be nice to define the following. If already defined somewhere, perhaps pin the doc in this channel?

1. Scope of the release certification effort (eg should include minion, deployment specific scenarios: eg S3 pinotFs)
2. Success criteria

MVP scope is query set with all possible data type

Ok. We should still list down what all are needed for a release to be certified in its end state, and phase it out. We can use this as a starting point:

1. Functional testing
   - Query types x data types (including mv/sv)
   - Data Ingestion (offline/rt, seg generation etc)
   - Deep storage
   - Control plane
   - Operational endpoints

2. Backward Compatibility and deployment ordering

3. Performance testing

4. Other components  (minion)

5. Chaos and Stress testing

6. Code coverage

It will also help to categorize issues seen in the past in the above areas, so we can focus on the most vulnerable areas first.

agreed

@User I understand. I'll begin working on the scripts and am available for conf on weekdays after 4:00 pm PST to sync up.

Adding Hara

Hi Sailesh, lets setup some time to meet and discuss the plan

We can always build it a little at a time and get the functionality we desire as we go along.

have created the binitial framework and the lib with cluster creation method and also have logging enabled

@User what does the framework do?

I prefer a shell + java framework. The shell part can be used to just spawn the java elements that can be used either for performance or functionality check. Doing it in python. may be fine for functionality and compat check, but can be very slow for sending queries and processing responses for performance check. I have asked @User to put a PR out for a few promitives that we know we will need for both perf and functionality checks.

Hey guys, sorry if I am missing something obvious, is there a plan explaining the overall goal, with definition of 'done', along with short/medium/long-term milestones.

We do have real issues that keep popping up in production today as new code gets checked in. It would be really helpful to ensure that at least our immediate/short term goals will indeed help address the issues we are actually seeing.

@User do you have anything to share

Hi, I have some changes but they are not quite ready yet https://github.com/apache/incubator-pinot/compare/master...saileshnankani:add-compatability-verifier?expand=1.

Thanks, let’s do a zoom call when you have something working?

Sure 👍, will let you know

Thanks

PR is up https://github.com/apache/incubator-pinot/pull/5911. I'm available for zoom call on weekdays after 4:00 pm PST to sync up.

@User has left the channel