I have 2 PR blocked by what it appears as flaky Pi...
# pinot-devj
Jack Luo
10/17/2024, 7:21 PMI have 2 PR blocked by what it appears as flaky Pinot-controller unit test under
Pinot Unit Test Set 2 (temurin-21) with skip bytebuffers:truea
Abhishek Sharma
10/17/2024, 7:24 PMRunning them again.
j
Jack Luo
10/17/2024, 7:25 PMty
e
Evan Galpin
10/21/2024, 6:34 PM+1, I’ve been experiencing a lot of flaky tests (different failures each run) and it’s a source of frustration. Is Pinot using Apache-run test infrastructure? If so, do we need to do any maintenance on the servers? Do we need more resources?
I’m getting failures like this within different suites on each run, which seem clearly unrelated to the outcome of tests but instead point to the test environment:
Copy code
Runtime java.net.BindException: Address already in usej
Jack Luo
10/21/2024, 7:15 PMI think there's some effort in mitigating this already
🙌 1
Jack Luo
10/21/2024, 7:16 PMimage.png
Jack Luo
10/21/2024, 7:16 PMI saw one PR from @Xiang Fu which reduces test workload of a unit test
x
Xiang Fu
10/21/2024, 7:19 PM
Which test fails for the Address already in use
e
Evan Galpin
10/21/2024, 9:35 PM@Xiang Fu at the moment it’s
QueryRoutingTestEvan Galpin
10/21/2024, 9:47 PMmaybe I should rebase / merge master to bring that IdealStateGroupCommitTest update into my branch, thanks @Jack Luo for pointing that out 🙏
👍 1
x
Xiang Fu
10/21/2024, 10:05 PM
yeah, I looked at the current master branch CI, it’s getting better
🌟 1
e
Evan Galpin
10/21/2024, 10:06 PMawesome, thanks for your efforts to improve that @Xiang Fu! Pulled those changes into my branch as well 🤞
x
Xiang Fu
10/21/2024, 10:07 PM
🤞
j
Jack Luo
10/21/2024, 10:07 PMJack Luo
10/21/2024, 10:07 PM Copy code
[INFO] org.apache.pinot.integration.tests.BasicAuthBatchIntegrationTest.testIngestionBatch -- Time elapsed: 8.014 s
[INFO]
[INFO] Results:
[INFO]
Error: Failures:
Error: HybridClusterIntegrationTest.testReload:248->BaseClusterIntegrationTestSet.testReload:769->BaseClusterIntegrationTestSet.lambda$testReload$5:773 expected [0] but found [1]
[INFO]
Error: Tests run: 425, Failures: 1, Errors: 0, Skipped: 12
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 35:18 min
[INFO] Finished at: 2024-10-21T20:40:45ZJack Luo
10/21/2024, 10:08 PMAny chance this is also an environment issue? My change has nothing to do with the failed integration test.
x
Xiang Fu
10/21/2024, 10:16 PM
I don’t know what’s going on, let’s add more logs to check: https://github.com/apache/pinot/pull/14269
e
Evan Galpin
10/21/2024, 10:46 PMIs there a way to re-run just failed tasks? Via Github UI or similar?
x
Xiang Fu
10/21/2024, 10:47 PM
commiter can run failed test
Xiang Fu
10/21/2024, 10:48 PM
image.png,image.png
j
Jack Luo
10/21/2024, 10:48 PMCan I apply to become a "commiter" so I can run-run tests?
x
Xiang Fu
10/21/2024, 10:50 PM
Definitely, just apache commitership requires a process and may take certain amount of time
🚀 1
e
Evan Galpin
10/21/2024, 10:54 PMI’ve seen other OSS projects with bots that PR authors can message to help work around this issue. It can be helpful for allowing momentum for authors that are not yet committers. I’ve seen them operate by taking action on the author sending PR comments like “Run Pinot Integration Test Set 2” and then re-executing a test suite by the matching name. It could be something to explore as a middle ground
Evan Galpin
10/21/2024, 10:54 PMbut of course, takes time and effort haha