I think I've got a unit test that asserts the issue, I'm going to push it to a branch named after the issue number and raise a draft PR for further discussion

This is fantastic.. thanks for finding this.. we will try to merge this before .11

no problem Kishore, here it is https://github.com/apache/pinot/pull/9093 fix not implemented yet

would be possible to have this patched in 0.10?

I think 0.11 is in process.. I might be wrong cc @Mayank

Folks, while running the tests on 0.11.0 rc branch, I see failures like: [

INFO] Running org.apache.pinot.segment.spi.creator.IndexingOverridesTest

21:59:54.364 [main] ERROR org.apache.pinot.segment.spi.index.IndexingOverrides - could not construct MethodHandle for org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider

java.lang.ClassNotFoundException: org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider

at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?]

at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]

at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]

at java.lang.Class.forName0(Native Method) ~[?:?]

at java.lang.Class.forName(Class.java:398) ~[?:?]

at org.apache.pinot.segment.spi.index.IndexingOverrides.invokeDefaultConstructor(IndexingOverrides.java:131) [classes/:?]

at org.apache.pinot.segment.spi.index.IndexingOverrides.createDefaultCreatorProvider(IndexingOverrides.java:116) [classes/:?]

at org.apache.pinot.segment.spi.index.IndexingOverrides.<clinit>(IndexingOverrides.java:67) [classes/:?]

at org.apache.pinot.segment.spi.creator.IndexingOverridesTest.indexingOverridesLoadableWithoutDefaultImplementation(IndexingOverridesTest.java:40) [test-classes/:?]

at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]

at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]

at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]

at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]

at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108) [testng-6.11.jar:?]

at org.testng.internal.Invoker.invokeMethod(Invoker.java:661) [testng-6.11.jar:?]

at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:869) [testng-6.11.jar:?]

at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1193) [testng-6.11.jar:?]

at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:126) [testng-6.11.jar:?]

at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109) [testng-6.11.jar:?]

at org.testng.TestRunner.privateRun(TestRunner.java:744) [testng-6.11.jar:?]

at org.testng.TestRunner.run(TestRunner.java:602) [testng-6.11.jar:?]

at org.testng.SuiteRunner.runTest(SuiteRunner.java:380) [testng-6.11.jar:?]

at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375) [testng-6.11.jar:?]

at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340) [testng-6.11.jar:?]

at org.testng.SuiteRunner.run(SuiteRunner.java:289) [testng-6.11.jar:?]

at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52) [testng-6.11.jar:?]

at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86) [testng-6.11.jar:?]

at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301) [testng-6.11.jar:?]

at org.testng.TestNG.runSuitesLocally(TestNG.java:1226) [testng-6.11.jar:?]

at org.testng.TestNG.runSuites(TestNG.java:1144) [testng-6.11.jar:?]

at org.testng.TestNG.run(TestNG.java:1115) [testng-6.11.jar:?]

at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:136) [surefire-testng-3.0.0-M5.jar:3.0.0-M5]

at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112) [surefire-testng-3.0.0-M5.jar:3.0.0-M5]

at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99) [surefire-testng-3.0.0-M5.jar:3.0.0-M5]

at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:145) [surefire-testng-3.0.0-M5.jar:3.0.0-M5]

at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428) [surefire-booter-3.0.0-M5.jar:3.0.0-M5]

at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162) [surefire-booter-3.0.0-M5.jar:3.0.0-M5]

at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562) [surefire-booter-3.0.0-M5.jar:3.0.0-M5]

at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548) [surefire-booter-3.0.0-M5.jar:3.0.0-M5]

Is this reproducible, or specific to my M1 machine? I am using Java 11 for the build

Hi Team, Can someone help me on how we can run compatibility test in local ? I have cloned the repo and have a feature branch which is in sync with upstream master now I have made some changes and have one commit which need to be merged. Now how do we run the compatibility test for the new commit I am basically trying to fix the failures by making the changes to the required json hence wanted to know if I can somehow run the test in local and then commit the changes. Reference PR: https://github.com/apache/pinot/pull/9092

Incremental pinot build using maven are pretty slow - 10mins+. Are others facing the same issue? Even switching maven daemon, the incremental pinot build take 5-7 minutes. Are there any tricks/fixes to make the builds faster?

This is a continuation PR from the issue previously discussed https://github.com/apache/pinot/pull/9163

I'm still on time to add some documentation for the 0.11 release?

Hey team, I just landed the last PR of a series of PRs to support proper null handling in Apache Pinot (PR1, PR2, PR3, PR4, and PR5). It would be great if all of these PRs make it to Pinot 0.11 release for the following reasons: 1. More than half of the PRs are already part of 0.11 release. It makes sense to include the remaining PRs to have the e2e feature available in 0.11. 2. Including these PRs will unblock a large percentage of queries that were not onboarded before because of the absence of proper null handling. Our team plans to immediately leverage this feature in our production shadow testing in order to hit our Q3 deliverable deadlines. Thanks

@Atri Sharma ^^

Sure, I can re cut the branch

Thanks @Atri Sharma and @Kishore G

I have a question on documentation: we do not seem to include the version of Pinot where a given function/feature was introduced. As new features and functions are introduced this may be confusing for users as they will have a hard time correlating what version has what. Is this not considered a limitation? Other projects I have worked on had text in docs indicating the first version where the function or feature was first available: do we have the possibility of doing something similar?

Hi team, Issue: In one of our use case to query pinot we were using pinot-java-client-0.6.0 where protocol scheme '*http*' is hardcoded, this setup used to work with our inhouse pinot queried via svc(k8) endpoint. Now, After migration of brokers to a different VPC, it was exposed via this url https://broker.xyz.cloud (using https), which the existing cient

pinot-java-client-0.6.0

couldnt connect. What didnt work: Updating pinot-java-client dependency to 0.7.1 or later in our usecase didnt work for us, as it was breaking other parts of the implementation Fix: To find a workable fix we updated the url to "https://" in JsonAsyncHttpPinotClientTransport of pinot java client 0.6.0 and repacked it as a new artifact

pinot-java-client-0.6.0-tls.jar

, As of today we are using this jar in build gradle as file(..) import, which still is a dirtyway to use a dependency, Would like to know from the team if we can have this change pushed somwehere in mvnreposiroty or any other Repository Manager, or any cleaner approach because all we needed was https instead of http The above fix works perfectly!

Hi guys! Watch @Mark Needham (Developer Relations, StarTree) shows us how to batch import JSON files from Amazon S3 into Apache Pinot.

Hey Pinot Community, Is there any RFC proposal page for Pinot? (Similar to the one for Hudi)

Hey hey dear pinot community little question about segment upload and editability. I’ve been working few month ago on purge process and the purge can only happened on OFFLINE segment because REALTIME segments aren’t editable. Few weeks ago I’ve seen a PR which seems to introduce upload of segment to realtime table. So does this lock on real time segment which are not editable is over ? If yes purge can be run on realtime table so ? PR I’m talking about (https://github.com/apache/pinot/pull/9063)

Hi there! Two months ago we changed Pinot GitHub Actions to add a new action that uses trivy to analyze the Pinot docker image looking for vulnerabilities (PR). This check is evaluated on all PRs, but it may have false positives in the sense that it may consider that a PR has vulnerabilities even when the included code doesn't have them. For example, let's say that a new vulnerability is discovered in one of our dependencies. Given that the job in master wasn't executed, trivy may consider that it is clean, but once a new PR is created, the pipeline is fired and the new vulnerability is detected. When this job really matters is when the PR adds a new dependency or change the version used by some dependency. In that case, if the PR adds a known vulnerability, it will be detected. I know it is not ideal to have false positives in tests, but AFAIK there is no better way to detect this vulnerabilities. The other alternative is periodically check our docker images in master, but in my experience that is problematic. At the end of the day people just forget to check the reports and the vulnerability stays there until some user complains about that, which is very bad news.

Hi ! I had a question regarding the Pulsar plugin. Adding a realtime table to Pinot requires a time column. Natively, a Pulsar message contains 2 temporal metadata fields: the publishTime and the eventTime (but which are not part of the payload). I think it would be interesting that the Pulsar plugin could directly integrate the publishTime and the eventTime as available fields inside the GenericRow in addition to the payload fields, this would allow the direct use of Pulsar messages without adding an additional redundant field in the schema. The evolution should be relatively simple by modifying the "public GenericRow next(GenericRow destination)" method of the PulsarStreamLevelConsumer class (unless i forget something). What do you think? Thank you !

Hi team! I’m looking to understand the package structure in pinot. I have read the Code Modules and Organization but I cannot found the info re some modules below • pinot-spi • pinot-segment-spi • pinot-segment-local • some others… Anyone knows where I can find this info? If necessary, I can collect these info from here and create a PR to update the Gitbook doc for Code Modules and Organization Thanks!

Hello, can we add support for

DATE

data type support in pinot? We have use case where we need to store date not datetime. Please let me know if i can help in anyway Thanks

Hey folks, is there a way to fetch the nth value in a multi-valued column?

Hi, created a PR for fixing pinot-realtime-quickstart.yml and README.md. https://github.com/apache/pinot/pull/9627. I noticed that documentation on the website is located in https://github.com/pinot-contrib/pinot-docs/tree/latest/basics/getting-started repo. How does that get updated?

Hi everyone, has anyone seen or developed a JDBC Connector to AWS Athena as a datasource using Athena Federated Queries (https://github.com/awslabs/aws-athena-query-federation)?

I’m trying to develop one but am having problems getting the partition info from tables. I tried the internal API REST call but it doesn’t seem to work:

I need to get the partition name info about the tables. Normally you can do this by querying the information_schema table in a rdbms: GET_PARTITIONS_QUERY = “SELECT DISTINCT partition_name FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_NAME = ? AND TABLE_SCHEMA = ? ” + “AND partition_name IS NOT NULL”;

Is there a way to do this in Pinot?