Hi All- Thank you or attending today's session and read out. We're planning to send through the follow-up materials, recording, and presentation. Mentioned at the start of our discussion was a customer utilizing Antithesis in a design phase: A brief write-up on that engagement can be found here: https://medium.com/@thinkx_/torturing-postgres-extreme-autonomous-testing-for-distributed-architectures-e1c4139ed72e

Hi <!here> Thank you again for attending today's session for the read out of Antithesis Testing of Venice for the last couple of weeks. Safe to say we're just scratching the surface, looking forward to the potential. Items we covered off on today: Antithesis refresher: Components, Architecture, Workflow, Set-up, Findings. POC covered/validated items: ✅ Configured Image Builds ✅ Antithesis Set-up configured docker-compose.yml ✅ Implemented and Configured Workload ✅ Antithesis Simulated set-up executed (testing framework, fault injector, etc.) ✅ Identified and reproduced potential data durability issues ✅ Review and Readout Recommended next session: Walk through of debugging with Antithesis. Open Items: Security considerations/review of Antithesis (What is needed here) Cadenced session with Amit, timing Today's read out recording: https://us06web.zoom.us/rec/share/P9SyKn9Xfr_4GWo1hjRMe8LLJqcri9dFr3MqbtiOfIaAMlzOfK0X0GRi6TT2zmQw.xGFaCMC2pSHF8qNA Passcode: fQ$m6ZAz Today's presentation deck .PDF:

Hi Craig, thanks as well for the great work so far. We are thinking about which use cases we can focus on next so that we can effectively guide the POC. One quick thing - Can you share the yml files and container definition files used so we can see exactly what the setup is? We would like to make sure that kafka , helix and zookeeper are in a different container than venice components.

Hey team, Sorry had to bail for an important meeting. Did we come to an understanding on the next steps?

Hi <!here> Wanted to thank everyone again for their time yesterday. The recording can be accessed here: https://us06web.zoom.us/rec/share/qsQD2fTY_yBQKWT8P2MNsFqV5DkN-rvhnmqaFVKYZ1MZD9wMMsXSjCULNkoKpmPw.t4aZmo2_WNMMkSUa We enjoyed the discussion and would recommend the following two items to dive deeper on in our next session: 1. Antithesis to Demo with repro of the bug found (with the session we ran excluding faults on controller/Kafka) 2. Antithesis to show how our fault injection works and the state of Venice nodes and share data visualizations along the way This will help support the two capabilities of high value identified on our last discussion: 1. Fault Injection Capabilities and 2. Reproducibility of issues This will also provide a practical workflow for the teams to better understand how to remediate once a bug/bad state is found. We can also address top of mind questions/open discussion that have come up in the meantime. Open to additional to explore, we'll defer to the team on what those are. Best-

Yeh that sounds like a plan. We would also like to understand - how reproduction will work , how much control will we have over the process and how debugging will work .

thanks

Hi team, questions we have in our reproduction of the messages: 1. At the point of when we see

com.linkedin.venice.exceptions.validation.MissingDataException

from

<http://service_venice-server-1.dc-1.venicedb.io|service_venice-server-1.dc-1.venicedb.io>

DC1 errors out and we no longer can query it for data. 2. We are still able to query DC0 for the dataset E.g.

service_venice-client info
2023-07-24 21:49:23 INFO [VenicePushJob] [main] Specific status: {dc-0=END_OF_PUSH_RECEIVED, dc-1=STARTED}
333.087
service_venice-client info
2023-07-24 21:49:24 INFO [VenicePushJob] [main] Specific status: {dc-0=END_OF_PUSH_RECEIVED, dc-1=STARTED}
334.193
service_venice-client info
2023-07-24 21:49:25 INFO [VenicePushJob] [main] Specific status: {dc-0=END_OF_PUSH_RECEIVED, dc-1=ERROR}

a. Would we still be able to to validate that all data exists just from DC0? b. How does the data push in general works between the the data centers? In the

batch-push-job.properties

we have the venice.discover.urls set as the parent controller

If missing data is detected during a push, then the push errors out and the dataset version associated with the push never comes online (at least in the DC where the error occurred), as you’ve seen. If missing data is detected from nearline writes (i.e. after a push has already ended, or if there was no push at all…) then we only record a metric, but don’t error out, since the dataset version is already online (the show must go on).

So the follow up question is that in our case, if DC1 errors out, if we only fetch data from DC0, can that be used to validate all the data are present?

Hi Venice team - Would it be possible to look towards early next week (8/7 or 8/8) to re-gather on de-bugging / reproducing workflow? I would have suggested this week but we have some team members out of the office for PTO.

Sure. Monday is better than Tuesdays . 9-11 or 1-3 should work.

Thanks Amit! Ill run these avails by our team and see which window lands best.

Hi Amit- I just checked calendars on our side and it looks like we have the opposite situation on our side, (Tuesday is better than Monday). Any chance there is an opening on Wednesday (8/9)?

Wednesdays are no meeting days for us. Thursday after 12 can work as well.

Hi Craig Thursdays 1-2 pm is also good on 24th as well. I saw updated meeting invite for 24th. Not sure if its a mistake. These are the times that will work for us. Mon 9-11, 1-3:30 . Tue 1030 1130 , 2-3 . Wed not avail. Thu 10:30 to 11:30 , 2- 4 pm , fri 9-11:30 , 1-4 pm. Also Friday 18th is a no meeting day

Thank you Amit- Not sure how/why the invite drifted to the 24th. I moved this to Friday 8/11 from 1-2 est. 10-11 pst. Thank you for patience while we got that squared away. Looking forward to reconnecting then.

Hi everyone we are starting today's session if you can still make it, here is the VC details:

Hi 👋 Venice Team, Thank you again for taking the time to meet on Friday afternoon with our team. Hoping everyone came away with a better understanding of the Network Faults our platform can apply to an application under test and how these can be manipulated. These faults include: Network Disruptions (Latency and Packet Dropping), Service Interruptions, CPU Modulation, File System, Thread Pausing, User-Defined Faults with more on the way. The slides presented can be accessed here. In the second half of the session Jeremy walked the team through the Antithesis Notebook, capturing the power of our deterministic environment allowing for the ability to step through points in time in the test runs to debug the discovered availability issue. The heap dumps can all be accessed via this link, please let us know if anyone has any issues with accessing. Lastly we also covered briefly our Reporting; which provides the initial alerting and logging where various test have passed/failed and is the initial point of test inspection for application test runs kicked off. Once the team has had an opportunity to review these materials and particularly the heap dumps, let's regroup at the earliest convenience to ensure we've provided the necessary materials to help resolve the open availability issue. Best- Antithesis Team

@Zac Policzer Craig got the green light for us to re-engage on the issue you've been investigating. We are getting started on it first thing next week! Could you confirm which version would be best to run with for the reproduction? Would the latest tag version 0.4.139 be a good candidate?

Hi Venice Team- <!here> Hope you are all doing well! We recently released V2 of Antithesis and wanted to share some highlights. If there are any questions on these release notes or areas of interest we want to delve deeper into please do not hesitate. Thank you again and take great care. Antithesis Version 2 Summary 1. Antithesis Version 2 (V2) is now being rolled out to our customers. 2. This new version of Antithesis enables us to offer more to customers—and offer it sooner. This is because V2 empowers Antithesis to build new features in response to customer demand far quicker than we could on our old platform; our timeline has moved from months to days. Upsides for our Customers Reports • All reports now give access to log and core dumps, rather than these features having to be built specifically for each customer report. • Further, old reports only generated logs at crashes; we now generate logs and core dumps at any point in time, as well as providing the state of each test property at said points in time. • All reports contain examples and counter-examples of the selected test properties. • These new report features can be used to reduce the massive amount of logs our customers need to store. • This can be done by running most or all tests within Antithesis; since we capture such detailed info for each experiment, we can reconstitute a particular test within minutes. Performance • Each test run will be more effective because: ◦ Breadth of search increases. ◦ Latency decreases. • "SnapshotGC" will keep us from losing test runs due to running into RAM limits. New Features These are the new features that will immediately enable our Professional Services team to provide greater value to customers. • Ability to build powerful command line extensions. • We have already used this to create features which give greater detail of activity leading to crashes, or provide detailed system performance graphs. • "Multiverse Analysis". • We can explore the multiverse of system states created by Antithesis in two powerful ways: ◦ We can use test properties or particular combinations of properties to analyze particular system states, ◦ or explore further from said states. • Our new eventset query language enables members of Antithesis Professional Services to craft better test properties and reports for customers, by making their workflow far more intuitive. • We now have the ability to observe an ongoing test and interact with it in real time, running any command. • Before, we could only interact with a test after it had completed.

Interacting with a test is kind of interesting. I wonder if we could rig it up to view the state of zk through the steps

We could log the entire EV/CV every time it changes I guess, right? We wouldn't want to run that in prod, but it could be an ok mode to enable in this kind of setup.

We could use the Zookeeper-cli at points during the test reproduction to inspect. Would that be able to get to the state information you’d be interested in?

Hi team, it was very nice meeting @Saxena Amit @Min Huang and @Zac Policzer at the Linkedin campus as well as catching up with @Gaojie Liu at the Qcon conference venue. Just to summarize some of the next steps we discussed: 1. The Venice team will use some "bug basher" time to fix the MissingDataException issue we found as part of the POC. Once this fix is in place will mark the conclusion of the POC and move towards discussion on an pilot engagement. 2. During the process of the fix, the Venice team may need to add more debugging, etc to the code. We are happy to help build off a debugging branch as needed to help this process and rerun our tests that finds the issue with the additional debugging. 3. Once a fix is in place, we will rerun and confirm with high confidence that this issue is resolved. Attached is the quick presentation I gave onsite, you can find additional debugging information including jmap core files in the analysis. Again please reach out if you need any additional help!

Hi Venice Team- Wanted to share some materials <!here> , we'd like to walk the team through these at the team's convenience. The purpose of these slides is to help illustrate where Antithesis would theoretically work among LinkedIn Venice's release workflows. https://drive.google.com/file/d/13Hi2F1od9D_5CP6MXr009_HFCroKrzO1/view?usp=drive_link Starting on Slide 6, and building through 14, these slides depict a nightly release and nightly test scenario of (6-12 hours) with Github Actions/Antithesis. Not all customers test with Antithesis nightly, some may test for longer test runs (24-48 hours) and less frequently; really depends on release behaviors and nature of tests. Also we can choose to gate releases based on failed tests, or not gate releases and test alongside. Slides 15 and on, are deeper dives on the concepts that are called out from Slides 6-14 (i.e. Webhooks and Reports mentioned). I don't think these slides stand on their own; but when the team is ready we'd like to review these with the group. @Zac Policzer @Min Huang let us know if any of the concepts here can help with triaging the open bug. Happy to answer any questions, thoughts, or feedback in the meantime. 🙏 -Antithesis Team

@Zac Policzer Welcome back! I wanted to check in to see if you have had time to work on the MissingDataException issue. Is there a new commit or branch for us to put under test?

<!here> Hi Venice Team - Hoping all are well. Wanted to extend an invitation to the team for a Private Dinner Antithesis is hosting on December 12th in Redwood City at Selby's. Our Founders are in town for the Cassandra conference in San Jose; so we decided to put together a dinner for the Database/Distributed Systems community. The idea is to come together for a casual evening to share ideas, discuss trends, network with other area folks focused on Infrastructure/Distributed applications. So far we have a great group of diverse-area companies attending. We'd be excited to host the Venice team if schedules allow. Dinner is at Selby's , starting at 6PM, more details can be found on the invitation here The private dinner will have ~30-35 attendees. This is filling up quickly, so if there is interest please let us know so we can reserve spots for the team.

Hi Venice Team! - Update from our side - 2 items of note: 1. We’ve continued to test for the Missing Data Exception and have no longer found this on our testing. Rather than assume the fix has been submitted, is it possible to confirm the fix? @Zac Policzer @Min Huang 2. Additional testing has uncovered two new additional interesting (maybe?) findings: i. Potential Leader Election ii. Store Ingestion Task Error An error component and source report can be found here, to use the report just match the Err Component (Left Column) with the corresponding Source (Right Column). Happy to discuss or explore these further with the team if these are indeed of interest. -Craig + @Chang Xiao @Jeremy