is there a way to get PR's not updated in last 6 months?

@User I think that we can also close all PRs that are older than 6 months. We can leave the message to recreate one if needed. Github keeps the closed PRs so code will be safe.

i will look into your bitpacking PR

In the meantime, I need edit access to the release wiki. https://cwiki.apache.org/confluence/display/PINOT/Creating+an+Apache+Release Some instructions need to be updated. Thanks @User for helping with some issues during the release. @User tried to give me access, but no success yet. Any other admin can help?

I prepared a release candidate for 0.4.0 (with cherry-picking @User’s 5470 and 5472). Please check the pinot dev email and help to verify. Also added some notes in the release

A different segment writing logic was introduced in #5256. Although this is backward compatible in a sense that the old segments can be read by the new code, rollback would be tricky since new segments after the upgrade would have been written in the new format, and the old code cannot read those new segments.

Neha, are we pushing FLATTEN_KEY fix ?

Wanna wait for CI on this one - or force the merge right away?

@User thanks for seeing the release through. I take it you updated any part of the release doc that was not complete or did not work as documented. Sometimes, things change over time, and we need to update our docs and processes to keep current.

It works, I basically only need to configure my IDE with proper formatting and PR is ready to go (any reference on what settings the pinot team uses for eclipse is welcome, couldn’t find anything on gh).

When Spring Boot 2.0 was released, there were a lot of breaking changes that millions of users had to deal with. How they handled the deprecation and removal of legacy features was to bundle the removal with many other benefits of migration. I saw this with Neo4j as well. Another example today of a big OSS change would be the removal of Zookeeper from Apache Kafka. We should look at that as a kind of case study on operational costs of these kinds of decisions.

@User and @User first of all, my statement was not that backward compat has to be avoided at all costs. I had specifically said that it should be avoided unless absolutely necessary. That being said, in this case, it is absolutely necessary. Controller port used is used as helix instance ID. This is then parsed by the servers when locating lead controllers for tables. Pinot is a distributed system, so a change in one end can affect something else.

We at LinkedIn need to encrypt segments in Controllers for some use cases. To do so, we’d like to store the boolean encryption flag cryptr name in the table config as the source of truth for this. That means that for each segment upload to Controller, table config needs to be fetched. The concern is that by doing so, we might put pressure on ZK. What’s your opinion on this?

I suppose the table config would be a crypter name, and not a flag?

Can someone help review the PR? And please leave comments if you have concern on removing the support of HAVING

normally these "message" nodes should be deleted once they are delivered and acted on?

Hi @User, I have a PR to fix the issue of incorrectly fetching the value of multi-value column. Could you review it? https://github.com/apache/incubator-pinot/pull/5746

@User, is it possible to fix the avro files? such that the field you’re interested in is not in a GenericRecord?

Hi everyone, https://github.com/apache/incubator-pinot/pull/5787 UnitTests - OracleJDK8 and IntegrationTests - OracleJDK8 in travis are failed due to scalatest maven plugin. it says some tests are failed, but all tests working as expected in local env when i build it. What can be problem? local maven version 3.6.1

I’m removing travis from the CI test: https://github.com/apache/incubator-pinot/pull/5834

just curious, why the pool size for HelixTaskExecutor is 1 for your cluster?

Hi folks, I'm new to Pinot and looking to help out with this ticket here: https://github.com/apache/incubator-pinot/issues/5951 What would be the easiest way to test out this command once I've got it implemented? Are there some example configs somewhere that I can use? Is there a PinotFS implementation that I can easily test out with my commands without needing to setup s3 creds and stuff?

@User can we remove the sha1 from pom?

@User Can you review this?

FastHLL

has been deprecated for a while, the replacement for that is

DistinctCountHLL

with

BYTES

column for both performance and compaction

I’m trying to get the BaseClusterIntegrationTest code working inside one of my projects. I’ve solved a few issues, but am now running into the FunctionRegistry’s static{} init code trying to treat org/apache/pinot/core/operator/transform/function/zone-index.properties file as a class file, which obviously fails. No idea why this doesn’t cause problems normally, but seems like the

filterInputsBy()

filter should be set up to only include class files, so this .properties file isn’t picked up. Thoughts?

I see that pinot-spi’s pom has a dependency on log4j-slf4j-impl. I don’t think this is right - it should only depend on slf4j-api. Because of this dependency, the pinot-java-client’s dependency on pinot-spi pulls in the logging implementation, which is not what you want because it means an external project using the client often needs to exclude those logging jars.

I’m working on https://github.com/apache/incubator-pinot/issues/6293, and had some questions about what the final jar should contain. The pinot-integration-tests-0.7.0-SNAPSHOT.jar is empty, and the pinot-integration-tests-0.7.0-SNAPSHOT-tests.jar has both the code that would be useful for 3rd party projects wanting to test against Pinot (BaseClusterIntegrationTest and associated classes) AND all of the integration test code and resources used by Pinot internally. I’m wondering if it makes sense to try to include only the externally useful code in the -tests jar, or move that code into

src/main/java

so that it winds up in the pinot-integration-tests-0.7.0-SNAPSHOT.jar. Thoughts?

@User and @User any yaml file that you could share with me regarding schema registry integration