ok

We seem to have a dead-link in the landing page https://pinot.apache.org/ . Scroll to the bottom of the page and under

Docs

click

Administration

. Note, this is not from https://docs.pinot.apache.org/. cc @User

The dead-link is: https://docs.pinot.apache.org/operating-pinot

@User Thanks for pointing that out. I've temporarily fixed the dead link. We'll make a more permanent fix soon.

Morning team! I'm struggling in building local SNAPSHOT version of Pinot that I can refer from other project. the fact that the version are not explicitely defined but made configurable with properties

${revision}

and

${sha1}

(this is breaking maven conventions btw) makes it very hard. The maven release plugin jobs does a great job of updating versions numbers between releases, any specific reason we historically use this pattern? Maven doesn't recommend placeholders in version since you can't guarantee what was released vs what is in your current VCS. Cheers!

@User we had to do this to get a nightly build going. @User has more information. Also, why do we need to release a snapshot of pinot for other projects? If you are doing manual builds, can you provide some values for these variables and get a snapshot build

Having a SNAPSHOT release channel allows a CI pipeline to quickly identify potential breaking changes and new features before the release is published. Most OSS project I have seen have snapshot releases channel. What I've seen work well is to always have maven version explicit in the poms and manage version upgrade with the mvn release plugin as part of the release process. When a release is cut, snapshot is moved to a final version, tag is done on git, build is published to maven, then main branch is updated with the new version snapshot.

I feel that after a git clone,

mvn package

should just work without specific instructions.

Also, when working locally, sub modules of Pinot can't be built individually with

-rf :sub-module

because of that. The whole project needs to be rebuilt.

@User you don’t have to specify anything for revision and sha1 if you want to make a snapshot build;

sha1

has its default value, which is

-SNAPSHOT

. Plus, there is an alternative to make a release instead of using release plugin: https://techluminary.com/discard-maven-release-plugin-with-a-new-approach/

Hi @User default values will only apply if you build from the parent pom. I'll go through that interesting reading. Thank you!

Is there any ongoing project to add "EXPLAIN" functionality to Pinot query?

with more index types added, EXPLAIN can show how the query would be executed. We got a few requests for our internal customers too.

There is none, will be a good feature to have but don’t see it as top priority for this quarter

<!here> we have too many PR's waiting for reviews. will be great to get some help

<!here> I am planning to merge the cluster-manager-ui PR

i tested the query console and it works fine

rest admin is still pointing to swagger ui

hey folks, whats your thought on a rank function like in sql server: https://www.sqlservertutorial.net/sql-server-window-functions/sql-server-rank-function/. I wonder if it can be an intermediate solution to the upsert problem: rank by the timestamp and filter by the top

will be too expensive

it’s like an agg function?

no, window functions need partition + sort

we wont be able to push down the computation to the servers

hmm, i’m thinking of top k here

not a full window function

if it’s top 1, we may have some optimization and turn it into an aggregation problem

mvn clean install -DskipTests -Pbin-dist

takes awfully long time, it seems to have creeped up quite a bit over the years. Any ideas on how to make the build faster?

~ 10 minutes on my own CI server.

[INFO] Pinot .............................................. SUCCESS [ 22.428 s]
[INFO] Pinot Service Provider Interface ................... SUCCESS [  7.931 s]
[INFO] Pinot Common ....................................... SUCCESS [ 32.668 s]
[INFO] Pinot Plugins ...................................... SUCCESS [  0.563 s]
[INFO] Pinot Input Format ................................. SUCCESS [  0.510 s]
[INFO] Pinot Avro Base .................................... SUCCESS [  1.694 s]
[INFO] Pinot Avro ......................................... SUCCESS [  5.284 s]
[INFO] Pinot Csv .......................................... SUCCESS [  0.844 s]
[INFO] Pinot JSON ......................................... SUCCESS [  0.738 s]
[INFO] Pinot Core ......................................... SUCCESS [ 22.112 s]
[INFO] Pinot Server ....................................... SUCCESS [  9.760 s]
[INFO] Pinot Controller ................................... SUCCESS [01:25 min]
[INFO] Pinot Broker ....................................... SUCCESS [  4.878 s]
[INFO] Pinot Clients ...................................... SUCCESS [  0.230 s]
[INFO] Pinot Java Client .................................. SUCCESS [  0.750 s]
[INFO] Pinot JDBC Client .................................. SUCCESS [  0.975 s]
[INFO] Pinot Minion ....................................... SUCCESS [  2.538 s]
[INFO] Pinot Confluent Avro ............................... SUCCESS [  3.746 s]
[INFO] Pinot ORC .......................................... SUCCESS [  4.028 s]
[INFO] Pinot Parquet ...................................... SUCCESS [  6.670 s]
[INFO] Pinot Thrift ....................................... SUCCESS [  1.268 s]
[INFO] Pinot Protocol Buffers ............................. SUCCESS [  1.329 s]
[INFO] Pluggable Pinot file system ........................ SUCCESS [  0.354 s]
[INFO] Pinot Azure Data Lake Storage ...................... SUCCESS [  5.128 s]
[INFO] Pinot Hadoop Filesystem ............................ SUCCESS [  1.115 s]
[INFO] Pinot Google Cloud Storage ......................... SUCCESS [  5.645 s]
[INFO] Pinot Amazon S3 .................................... SUCCESS [ 11.483 s]
[INFO] Pinot Batch Ingestion .............................. SUCCESS [  0.568 s]
[INFO] Pinot Batch Ingestion Common ....................... SUCCESS [  0.785 s]
[INFO] Pinot Batch Ingestion for Spark .................... SUCCESS [ 11.263 s]
[INFO] Pinot Batch Ingestion for Hadoop ................... SUCCESS [  3.663 s]
[INFO] Pinot Batch Ingestion Standalone ................... SUCCESS [  0.985 s]
[INFO] Pinot Batch Ingestion .............................. SUCCESS [  0.448 s]
[INFO] Pinot Ingestion Common ............................. SUCCESS [  1.763 s]
[INFO] Pinot Hadoop ....................................... SUCCESS [ 34.624 s]
[INFO] Pinot Spark ........................................ SUCCESS [ 35.108 s]
[INFO] Pinot Stream Ingestion ............................. SUCCESS [  0.581 s]
[INFO] Pinot Kafka Base ................................... SUCCESS [  1.323 s]
[INFO] Pinot Kafka 0.9 .................................... SUCCESS [  9.815 s]
[INFO] Pinot Kafka 2.0 .................................... SUCCESS [  5.919 s]
[INFO] Pinot Tools ........................................ SUCCESS [01:47 min]
[INFO] Pinot Integration Tests ............................ SUCCESS [ 18.378 s]
[INFO] Pinot Perf ......................................... SUCCESS [  7.437 s]
[INFO] Pinot Distribution ................................. SUCCESS [ 38.707 s]

Overall Pinot Controller and Pinot tools are the suspects

Oh wait, don't mind that, it's a

mvn deploy

 command to my personal artifactory.

shaded jars take time