https://pinot.apache.org/ logo
Join Slack
Powered by
I have been looking into what would be needed to r...
# general
s
I have been looking into what would be needed to run pinot tests in parallel - it's mostly collisions in networking ports,and test files. Concurrent tests may use same ports or write same files. It runs the entire suite in ~30m on my machine. Is there any interest making PRs for those changes? It can get pretty involved.
Copy code
[INFO] Reactor Summary for Pinot 1.6.0-SNAPSHOT:
[INFO]
[INFO] Pinot .............................................. SUCCESS [  6.209 s]
[INFO] Pinot Dependency Verifier .......................... SUCCESS [  2.303 s]
[INFO] Pinot Service Provider Interface ................... SUCCESS [ 19.421 s]
[INFO] Pinot Segment Service Provider Interface ........... SUCCESS [ 21.050 s]
[INFO] Pinot Query Planner SPI ............................ SUCCESS [  3.963 s]
[INFO] pinot-timeseries ................................... SUCCESS [  0.155 s]
[INFO] pinot-timeseries-spi ............................... SUCCESS [  7.176 s]
[INFO] Pinot Common ....................................... SUCCESS [ 43.725 s]
[INFO] Pinot Plugins ...................................... SUCCESS [  0.306 s]
[INFO] Pinot Input Format ................................. SUCCESS [  0.214 s]
[INFO] Pinot Avro Base .................................... SUCCESS [ 15.195 s]
[INFO] Pinot Avro ......................................... SUCCESS [  4.294 s]
[INFO] Pinot Csv .......................................... SUCCESS [ 18.868 s]
[INFO] Pinot JSON ......................................... SUCCESS [ 19.834 s]
[INFO] Pinot Metrics ...................................... SUCCESS [  0.162 s]
[INFO] Pinot Yammer Metrics ............................... SUCCESS [ 19.776 s]
[INFO] Pinot local segment implementations ................ SUCCESS [01:13 min]
[INFO] Pinot Core ......................................... SUCCESS [02:31 min]
[INFO] Pinot Query Planner ................................ SUCCESS [ 16.531 s]
[INFO] pinot-timeseries-planner ........................... SUCCESS [  2.031 s]
[INFO] Pinot Query Runtime ................................ SUCCESS [ 25.290 s]
[INFO] Pinot Materialized View ............................ SUCCESS [  5.896 s]
[INFO] Pinot Segment Uploader ............................. SUCCESS [  0.225 s]
[INFO] Pinot Segment Uploader Default ..................... SUCCESS [  1.216 s]
[INFO] Pinot SQL DDL ...................................... SUCCESS [  5.158 s]
[INFO] Pinot Server ....................................... SUCCESS [ 11.238 s]
[INFO] Pinot Clients ...................................... SUCCESS [  0.280 s]
[INFO] Pinot Java Client .................................. SUCCESS [ 17.587 s]
[INFO] Pinot Controller ................................... SUCCESS [08:16 min]
[INFO] Pinot Broker ....................................... SUCCESS [ 35.576 s]
[INFO] Pinot JDBC Client .................................. SUCCESS [ 17.375 s]
[INFO] Pinot CLI .......................................... SUCCESS [ 18.620 s]
[INFO] Pinot Batch Ingestion .............................. SUCCESS [  0.151 s]
[INFO] Pinot Batch Ingestion Common ....................... SUCCESS [  2.749 s]
[INFO] Pinot Minion ....................................... SUCCESS [  8.339 s]
[INFO] Pinot Arrow ........................................ SUCCESS [ 18.295 s]
[INFO] Pinot CLP Log ...................................... SUCCESS [  2.287 s]
[INFO] Pinot Confluent Avro ............................... SUCCESS [  5.756 s]
[INFO] Pinot Confluent Json ............................... SUCCESS [ 14.438 s]
[INFO] Pinot Protocol Buffers ............................. SUCCESS [ 17.388 s]
[INFO] Pinot Confluent Protocol Buffers ................... SUCCESS [ 16.396 s]
[INFO] Pinot ORC .......................................... SUCCESS [ 24.759 s]
[INFO] Pinot Parquet ...................................... SUCCESS [ 27.553 s]
[INFO] Pinot Thrift ....................................... SUCCESS [ 14.236 s]
[INFO] Pluggable Pinot file system ........................ SUCCESS [  0.206 s]
[INFO] Pinot Azure Data Lake Storage ...................... SUCCESS [ 19.733 s]
[INFO] Pinot Hadoop Filesystem ............................ SUCCESS [ 18.042 s]
[INFO] Pinot Google Cloud Storage ......................... SUCCESS [ 21.775 s]
[INFO] Pinot Amazon S3 .................................... SUCCESS [ 24.091 s]
[INFO] Pinot Batch Ingestion Spark Base ................... SUCCESS [  1.916 s]
[INFO] Pinot Batch Ingestion for Spark 3 .................. SUCCESS [  8.753 s]
[INFO] Pinot Batch Ingestion for Hadoop ................... SUCCESS [ 12.990 s]
[INFO] Pinot Batch Ingestion Standalone ................... SUCCESS [  6.362 s]
[INFO] Pinot Stream Ingestion ............................. SUCCESS [  0.224 s]
[INFO] Pinot Kafka Base ................................... SUCCESS [  5.141 s]
[INFO] Pinot Kafka 3.x .................................... SUCCESS [ 47.677 s]
[INFO] Pinot Kafka 4.x .................................... SUCCESS [ 47.060 s]
[INFO] Pinot Kinesis ...................................... SUCCESS [ 17.788 s]
[INFO] Pinot Pulsar ....................................... SUCCESS [ 39.304 s]
[INFO] Pinot Minion Tasks ................................. SUCCESS [  0.167 s]
[INFO] Pinot Minion Built-In Tasks ........................ SUCCESS [ 10.143 s]
[INFO] Pinot Plugin Assembly Descriptor ................... SUCCESS [  7.288 s]
[INFO] Pinot Dropwizard Metrics ........................... SUCCESS [  4.372 s]
[INFO] Pinot Compound Metrics ............................. SUCCESS [  4.162 s]
[INFO] Pinot Segment Writer ............................... SUCCESS [  0.205 s]
[INFO] Pinot Segment Writer File Based .................... SUCCESS [  4.741 s]
[INFO] Pluggable Pinot Environment Provider ............... SUCCESS [  0.200 s]
[INFO] Pinot Azure Environment ............................ SUCCESS [  4.610 s]
[INFO] Pluggable Pinot Time Series Languages .............. SUCCESS [  0.201 s]
[INFO] pinot-timeseries-m3ql .............................. SUCCESS [ 11.550 s]
[INFO] Pinot Tools ........................................ SUCCESS [ 22.718 s]
[INFO] Pinot Test Utils ................................... SUCCESS [  2.801 s]
[INFO] Pinot UDF test ..................................... SUCCESS [  1.904 s]
[INFO] Pinot Integration Tests ............................ SUCCESS [14:31 min]
[INFO] Pinot Perf ......................................... SUCCESS [ 28.645 s]
[INFO] Pinot Distribution ................................. SUCCESS [ 45.690 s]
[INFO] Pinot Connectors ................................... SUCCESS [  0.282 s]
[INFO] Pinot Spark Common ................................. SUCCESS [ 16.953 s]
[INFO] Pinot Spark 3 Connector ............................ SUCCESS [ 14.888 s]
[INFO] Pinot Flink Connector .............................. SUCCESS [ 17.043 s]
[INFO] Pinot Compatibility Verifier ....................... SUCCESS [  2.866 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  30:49 min (Wall Clock)
[INFO] Finished at: 2026-06-25T18:10:58-04:00
[INFO] ------------------------------------------------------------------------
[INFO] 1537 goals, 1523 executed, 14 from cache
x
Definitely! I used to make one change for dynamic ports for tests, but some devs complaining about that it's hard for them to debug. So if we could make the port assignment to be preferable the default port only use dynamic when there is a port conflict.
s
👍 What kind of tests were they making? Running a local cluster and then testing in a UI?
x
Mainly test locally, e.g. the controller default port is 9000, if we make it some random port then they need to check test logs for the controller port to perform extract tests if they want
This often happens when running integration tests locally
s
OK, presumably that's never run in parallel?
x
Yes, locally people won't run it in parallel, mainly for running integration tests in CI
t
@Samuel Papin do you have a plan for this? I'd love to have parallel unit tests. Even if we can't get all of them running in parallel, we could split it into two jobs: one parallel, and one non-parallel, so we can start taking advantage of the speed up earlier
s
Yes I have a plan for this and a working version but I need to go over it and split up the different fixes into separate PRs. One caveat is parallel integration tests get quite memory hungly. they'd run less parallel than the others 🙂
❤️ 1
2 Views
Open in Slack
PreviousNext
Powered by