https://pinot.apache.org/ logo
Join Slack
Powered by
On <Creating an Apache Release>, there's a step to...
# pinot-dev
r
On Creating an Apache Release, there's a step to run 
mvn clean test
after the release branch is cut. Is this step still valid? On my end (using the master branch) I see that maven skips all of the tests because “clean tests” is an unknown lifecycle.
x
clean test
not 
clean tests
r
Thanks, but I didn't have a typo. I was working on Trino recently and forgot to switch back from jdk 23 to 17. I'm running tests now and will report any errors.
x
👍
r
Copy code
[ERROR] Failures:
[ERROR]   AccessControlTest.testAccessDenied:108 » Processing java.net.ConnectException: Operation timed out
[ERROR]   HealthCheckResourceTest.checkHealthProbes:43 » Processing java.net.ConnectException: Operation timed out
[ERROR]   PinotServerAppConfigsTest.testAppConfigs:58 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TableSizeResourceTest.testTableSizeDetailed:38->verifyTableSizeDetailedImpl:44 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TableSizeResourceTest.testTableSizeNoDetails:56->verifyTableSizeNoDetailsImpl:64 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TableSizeResourceTest.testTableSizeNotFound:32 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TableSizeResourceTest.testTableSizeOld:73->verifyTableSizeOldImpl:79 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TableTierResourceTest.testSegmentNotFound:42 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TableTierResourceTest.testTableNotFound:34 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TableTierResourceTest.testTableSegmentTierInfo:62->verifyTableTierInfo:71 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TableTierResourceTest.testTableTierInfo:50->verifyTableTierInfo:71 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TablesResourceTest.getSegments:100 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TablesResourceTest.getTableIndexes:129 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TablesResourceTest.getTableMetadata:163 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TablesResourceTest.getTables:68 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TablesResourceTest.testDownloadSegments:265->downLoadAndVerifySegmentContent:358 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TablesResourceTest.testDownloadValidDocIdsSnapshot:285->downLoadAndVerifyValidDocIdsSnapshot:404 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TablesResourceTest.testOfflineTableSegmentMetadata:558 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TablesResourceTest.testSegmentCrcMetadata:250 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TablesResourceTest.testSegmentMetadata:199 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TablesResourceTest.testUploadSegments:522 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TablesResourceTest.testValidDocIdMetadata:307->downLoadAndVerifyValidDocIdsSnapshot:404 » Processing java.net.ConnectException: Operation timed out
[ERROR]   TablesResourceTest.testValidDocIdsMetadataPost:331->downLoadAndVerifyValidDocIdsSnapshot:404 » Processing java.net.ConnectException: Operation timed out
[INFO]
[ERROR] Tests run: 36, Failures: 23, Errors: 0, Skipped: 0
These tests are for Pinot Server
I got the 1st 2 tests to pass by replacing 
NetUtils.getHostAddress()
with 
"0.0.0.0"
when initializing 
_webTarget
x
Can you try to just run test for one Test class?
r
I’m re-running all of the tests now with the change I mentioned earlier
I was able to reproduce a couple of the errors in IntelliJ before I made any changes
All of the tests for Pinot Server passed. But now a Pinot Controller test is failing:
Copy code
[ERROR] Failures:
[ERROR]   PinotResourceManagerTest.testBasicAndConcurrentAddingAndDeletingSegments:170 expected [30] but found [29]
[INFO]
[ERROR] Tests run: 1451, Failures: 1, Errors: 0, Skipped: 0
x
hmm, could you share the machine spec running these tests?
r
Apple M2 running Sonoma 14.6.1
x
jdk?
and maven
r
Copy code
java -version
openjdk version "17.0.13" 2024-10-15
OpenJDK Runtime Environment Temurin-17.0.13+11 (build 17.0.13+11)
OpenJDK 64-Bit Server VM Temurin-17.0.13+11 (build 17.0.13+11, mixed mode, sharing)
Copy code
mvn --version
Apache Maven 3.9.4 (dfbb324ad4a7c8fb0bf182e6d91b0ae20e3d2dd9)
Maven home: /opt/homebrew/Cellar/maven/3.9.4/libexec
Java version: 17.0.13, vendor: Eclipse Adoptium, runtime: /Users/robertzych/.sdkman/candidates/java/17.0.13-tem
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "14.6.1", arch: "aarch64", family: "mac"
x
hmm
do you run tests in sequential or parallel?
r
I'm just using 
mvn clean test
and testing 2 different versions of pinot simultaneously
x
hmm, what do you mean by 
simultaneously
? running two 
mvn clean test
commands at same time?
r
no, more like 2 console sessions
x
two VMs?
r
no
x
this might cause race conditions I think
some tests might bring up zk or controller or something components using same ports
r
I initially started testing only 1 version of pinot
then I starting the latest version in parallel
I've upgraded jdk to 21:
Copy code
java -version
openjdk version "21.0.5" 2024-10-15 LTS
OpenJDK Runtime Environment Temurin-21.0.5+11 (build 21.0.5+11-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.5+11 (build 21.0.5+11-LTS, mixed mode, sharing)
re-running 
mvn clean test
on the latest version now
x
i see. for the test, just run it against one version
you cannot test two versions simultaneously under one environment.
or you need to have two VMs for this
r
the tests are still running but I already see the same error in the logs
the Pinot Server tests failed again. the same 23 tests failed with the same 
Operation timed out
error
x
hmm, is it same build and test ?
r
Yes, only difference is the newer jdk and without any other running tests
I’m running a similar set of tests on my other machine
I suspect that security software may be the root cause of the 
Operation timed out
errors.
I tested version 1.2 on my other machine and it got much further but still failed on the integration tests:
Copy code
[ERROR] Failures:
[ERROR]   ExactlyOnceKafkaRealtimeClusterIntegrationTest>BaseRealtimeClusterIntegrationTest.setUp:50->ClusterTest.startBroker:183->ClusterTest.startBrokers:189->ClusterTest.startOneBroker:199 » Runtime java.io.IOException: Failed to bind to address 0.0.0.0/0.0.0.0:50879
[ERROR]   OfflineClusterMemBasedServerQueryKillingTest.setUp:118->startBrokers:151->ClusterTest.startBrokers:189->ClusterTest.startOneBroker:199 » Runtime java.io.IOException: Failed to bind to address 0.0.0.0/0.0.0.0:56072
[INFO]
[ERROR] Tests run: 794, Failures: 2, Errors: 0, Skipped: 33
I also just tested version 1.3 on my other machine which also failed on a broker test:
Copy code
[ERROR] Failures:
[ERROR]   HelixExternalViewBasedQueryQuotaManagerTest.testWhenOnlyDefaultAppQuotaIsSetItAffectsAllApplications:333->runQueries:695 Expected failure with qps: 201.0 and app :someApp expected [true] but found [false]
[INFO]
[ERROR] Tests run: 150, Failures: 1, Errors: 0, Skipped: 0
Switched to a fresh VM and got a bit further. Now the next error is in Pinot Protocol Buffers:
Copy code
[ERROR] Failures:
[ERROR]   ProtoBufConfluentSchemaTest.setup:58 » IllegalState Could not find a valid Docker environment. Please see logs and check configuration
[INFO]
[ERROR] Tests run: 198, Failures: 1, Errors: 0, Skipped: 2
Fixed that VM config issue and got further. Now the next error is in the Integration tests:
Copy code
[ERROR] Failures:
[ERROR]   QueryQuotaClusterIntegrationTest.testApplicationAndDatabaseQueryQuotaWithTableQueryQuotaWithExtraBroker:224->testQueryRateOnBroker:250->runQueriesOnBroker:338 Expected >0 failures for qps: 50.0 expected [true] but found [false]
[ERROR]   QueryQuotaClusterIntegrationTest.testApplicationQueryQuotaWithTableQueryQuota:173->testQueryRate:242->runQueries:272 Expected >0 failures for qps: 50.0 expected [true] but found [false]
[ERROR]   QueryQuotaClusterIntegrationTest.testDatabaseQueryQuotaWithTableQueryQuotaWithExtraBroker:191->testQueryRateOnBroker:250->runQueriesOnBroker:338 Expected >0 failures for qps: 25.0 expected [true] but found [false]
[ERROR]   QueryQuotaClusterIntegrationTest.testDefaultApplicationQueryQuota:111->testQueryRate:242->runQueries:272 Expected >0 failures for qps: 100.0 expected [true] but found [false]
[INFO]
[ERROR] Tests run: 911, Failures: 4, Errors: 0, Skipped: 18
@Xiang Fu Do these look like legitimate failures?
@Gonzalo Ortiz These errors appear to be related to https://github.com/apache/pinot/pull/14226
x
Is this reproducible every time when you just run this test individually
r
I wasn't able to reproduce the error with 
testApplicationAndDatabaseQueryQuotaWithTableQueryQuotaWithExtraBroker
when I ran it individually on my other machine. So it could still be some new type of VM config issue.
x
Yes, it might be previous test causing the issue/failure for the next one
r
testDefaultApplicationQueryQuota
is reproducible when I run it individually on my other machine.
And 
testDefaultApplicationQueryQuota
passes when I run all of the tests in 
QueryQuotaClusterIntegrationTest
x
Other machine means?
r
My other machine is my personal laptop where I’m only testing Apache Pinot
x
also mac or linux
r
Mac
It appears that my other machine is unable to generate the 50 qps needed for this test to fail
x
facepalm too slow?
r
I added actual qps logging to the test and see that the test fails even though my other machine is generating 63 qps
The test passes when I increase the qps
My other machine only has 6 cores and my VM currently only has 8 cores. Since Ankit recommends using a 16 core machine, on Monday I'll upgrade my VM to use 16 cores and will run the tests again.
x
Sounds good, where is your VM deployed
EC2 or k8s
r
My VM is running on-prem. Neither EC2, k8s, or any other public cloud.
After upgrading my VM to 16 cores, 2 out the 4 previously failing tests in 
QueryQuotaClusterIntegrationTest
are still failing:
Copy code
[ERROR] Failures:
[ERROR]   QueryQuotaClusterIntegrationTest.testApplicationAndDatabaseQueryQuotaWithTableQueryQuotaWithExtraBroker:224->testQueryRateOnBroker:250->runQueriesOnBroker:338 Expected >0 failures for qps: 50.0 expected [true] but found [false]
[ERROR]   QueryQuotaClusterIntegrationTest.testDatabaseQueryQuotaWithTableQueryQuotaWithExtraBroker:191->testQueryRateOnBroker:250->runQueriesOnBroker:338 Expected >0 failures for qps: 25.0 expected [true] but found [false]
[INFO]
[ERROR] Tests run: 911, Failures: 2, Errors: 0, Skipped: 18
I added actualQps logging to one of the tests and I'm finding that when the actualQps is lower than expected, the expected quota errors aren't generated. For example, when the expected qps is 100 but the actual qps is < 70 the 
testDefaultApplicationQueryQuota
fails
Copy code
[ERROR] Failures:
[ERROR]   QueryQuotaClusterIntegrationTest.testApplicationAndDatabaseQueryQuotaWithTableQueryQuotaWithExtraBroker:224->testQueryRateOnBroker:250->runQueriesOnBroker:344 Expected >0 failures for qps: 50.0 expected [true] but found [false]
[ERROR]   QueryQuotaClusterIntegrationTest.testDatabaseQueryQuotaWithTableQueryQuotaWithExtraBroker:191->testQueryRateOnBroker:250->runQueriesOnBroker:344 Expected >0 failures for qps: 25.0 expected [true] but found [false]
[ERROR]   QueryQuotaClusterIntegrationTest.testDefaultApplicationQueryQuota:111->testQueryRate:242->runQueries:278 Expected >0 failures for qps: 100.0 (actual qps: 60.45949214026602) expected [true] but found [false]
[INFO]
[ERROR] Tests run: 911, Failures: 3, Errors: 0, Skipped: 18
@Ankit Sultana ☝️
x
I feel there are not enough cpu resources to bombard the server endpoint
r
Assuming these tests passed in GitHub, how much CPU does the GitHub runner have? Only 4 cores, right?
If I disable the 4 tests that failed, the rest of the tests passed on my VM
@Bolek Ziobrowski I believe that you wrote the quota integrations tests that are failing. Could you please take a look at the `QueryQuotaClusterIntegrationTest`tests results above?
When trying to increase the difference between the actualQps and the quota I lowered the quota in the test, but this caused a timeout error because the lower quota couldn’t be applied.
x
image.png
r
My VM has more than that. 16 cores, 64 GB RAM, and 120 GB SSD
x
hmm, that’s weird
r
Are you able to run 
mvc clean test
with the latest from master on your machine?
x
hmm. it also failed on my side, will let @Bolek Ziobrowski chime in as well.
this command failed for me:
Copy code
mvn test -Dtest=org.apache.pinot.integration.tests.QueryQuotaClusterIntegrationTest
b
Sorry for the delay. I've been busy with other stuff. I'll have a look at those tests later today . From what I remember all of them depend on event timing and are quite fragile . An unexpected delay (e.g. GC STW) can make them not reach the desired qps and thus fail.
r
@Shounak Kulkarni this is the thread I was referring to during today's year in review meeting
b
@robert zych This PR should fix quota tests -> https://github.com/apache/pinot/pull/14602
r
@Bolek Ziobrowski I've confirmed that your changes cause the tests to pass on my VM
Open in Slack
PreviousNext
Powered by