Hi all, I would like to use this thread for discussing about 0.8.0 release. Some points that we need to discuss: • Should we cut the release before or after the graduation? • Let’s call the timeline so that people can check-in the features to be included to the release I think that we should make the call first on the before/after graduation and then decide the timeline.

Hi, My use case is to query pinot, and data fetched should be stored in S3 in CSV format. Is there any support which pinot provides to store query result in S3?

By default you can map top level columns to Pinot schema. Or as you mentioned either use transform functions or json objects to store them in Pinot.

We at LinkedIn saw production issues with a recent commit in upload segment endpoint. I described it in this issue, but wanted to share it here as well. (cc @User @User @User @User @User - ppl involved in the PR)

When running

mvn clean install -DskipTests -Pbin-dist

on master, I got a failure:

on project pinot-jdbc-client: *Some files do not have the expected license header

.*

👋 I'm curious to know if there any recent conversation around supporting the window functions in Pinot 🙇 https://github.com/apache/incubator-pinot/issues/2797

Please take a look this: Upgrade pinot to use java11 compiler and allowed fallback to build and run with JDK 8: https://github.com/apache/incubator-pinot/pull/6424

Hi team, we have

"realtime.max.parallel.segment.builds"

that controls the number of parallel segment builds on realtime servers. I notice that the default value is 0 (which I think implies unconstrained, which is inherently capped by number of partitions). I am curious to know if folks have found this default to be good? If not, should we have a more conservative default (say 4)? cc @User @User @User

I’m going to merge java 11 upgrade PR(#6424) today, for devs still on java8, please use

mvn clean install -DskipTests -Pbin-dist -T 4  -Djdk.version=8

to build pinot source code.

Any Developments in progress to support in pinot reg sorting and partitioning offline data(orc) present in Hdfs ?

In

SegmentDeletionManager.removeAgedDeletedSegments()

, there’s this bit of code:

try {
        // Check that the directory for deleted segments exists.
        if (!pinotFS.isDirectory(deletedDirURI)) {
          LOGGER.warn("Deleted segment directory {} does not exist or it is not directory.", deletedDirURI.toString());
          return;
        }

But if we’ve never deleted any segments, then this directory won’t exist yet. So shouldn’t this check be “if it exists && it’s not a directory”? Otherwise we get regular warnings in our log files, which trigger an alert, etc, etc.

Is it possible for provide snapshot tar with current code? above 7.1

Can we have multiple zookeeper for pinot?

@User @User I am looking to support nulls in aggregates (a common use case for us). Is there a place where I can get prior thoughts and research, and potential starting ideas?

I just updated my Pinot fork from master, and tried a top-level build with

mvn clean install -DskipTests -Pbin-dist -T 4 -Djdk.version=8

. It failed with

[*ERROR*] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M2:enforce *(enforce-dependency-convergence)* on project pinot-integration-tests: *Some Enforcer rules have failed.*

In Eclipse I’ve pulled in all of the pinot projects from

master

, and

pinot-segment-local

has an error. The ReplicationUtils class isn’t in the correct package (file says

org.apache.pinot.core.util

, but location in project is

org.apache.pinot.segment.local.utils

. See https://github.com/apache/incubator-pinot/blob/master/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/ReplicationUtils.java in GitHub

https://docs.pinot.apache.org/basics/getting-started/running-pinot-locally#setting-up-a-pinot-cluster Under the section

setting up pinot cluster

manual cluster setup link is not accessible

<https://app.gitbook.com/@apache-pinot/s/apache-pinot-docs/~/drafts/-MYGXF71U0yIiqM1EDPS/basics/getting-started/advanced-pinot-setup|Manual cluster setup>

@User @User please review the distinct PR and let me know if it looks ok

hello everyone, currently we are using latest kinesis kcl library for streaming ingestion. why don't we have support for kcl fanout which will reduce the kinesis limitations ??

Folks, I wrote a one pager on subquery support in Pinot. Please see and comment: https://docs.google.com/document/d/1_dQkNLmadOQUq6wtTykgJgNQ3Y_plzSZYg_kd4QSYcE/edit?usp=sharing

Hi I'm looking at the use case of reading the pinot segments from a realtime table in an Apache spark/ Apache Flink job . The objective of the spark job is to 1. Get/download the segments of a realtime table 2. Read the segments and convert to JSON I was wondering if there's a plugin or library(jar) of sorts that I can use to achieve this ? Any pointers would be helpful. Thanks

Congratulations to the community on a successful graduation!!!

Does our integration tests pipeline have some issues? Kafka leader election is continously failing for a test

Hi devs - I started working on defining a solution for https://github.com/apache/pinot/issues/7206, but ran into a bunch of confusing code and undocumented options that seem related.

I've been looking at the new segment writer/segment uploader interfaces and implementations for use in a spark uploader and came across something I think could be improved. Seems like the FileBasedSegmentWriter pulls in pinot-core for exactly one class 'SegmentProcessorAvroUtils' ... which is only used currently by that class. I can't think of any good reason to pull in pinot core as a dependency for that plugin.

Is the source to the docs in a public git repo that I can fork? I've seen a number of small issues I could submit merge requests for....

@User The BOOLEAN datatype earlier was based on STRING, but now is based on INTEGER. I am wondering if the old tables that use BOOLEAN columns will continue working with STRING or do they need to be modified and upgraded? I am looking at a somewhat related issue (on the client side), but wondering how the backward compatibility works on the server side?

Re Sensitive config obfuscation in API responses and logs https://github.com/apache/pinot/issues/6823 https://github.com/apache/pinot/issues/7232 @User @User

https://github.com/apache/pinot/issues/7271 we encountered a server start issue in Pinot 0.7. We proposed to rollback the Reflection lib to 0.9.9 instead of 0.9.11. @User @User pls let us know what you think.

to find annotations?