Who knows the most about the CI build system? Trying to figure out why my build fails with various issues (dependency convergence, rat check on new .fst files) while (I hope) the regular builds from master are fine.

Hello, Is there an upper limit to how big bitmap inverted index file can be? Seems to me like it can't exceed 2GB for now due to integer offsets. I am wondering if this is accidental or intentional or if we can consider larger size index files? One potential fix may be to change BitmapInvertedIndexWriter._bytesWritten to a

long

?

Error: java.lang.RuntimeException: java.lang.IllegalArgumentException: Negative position at org.apache.pinot.hadoop.job.mappers.SegmentCreationMapper.map(SegmentCreationMapper.java:310) at org.apache.pinot.hadoop.job.mappers.SegmentCreationMapper.map(SegmentCreationMapper.java:66) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171) Caused by: java.lang.IllegalArgumentException: Negative position at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:863) at org.apache.pinot.segment.local.segment.creator.impl.inv.BitmapInvertedIndexWriter.mapBitmapBuffer(BitmapInvertedIndexWriter.java:102) at org.apache.pinot.segment.local.segment.creator.impl.inv.BitmapInvertedIndexWriter.resizeIfNecessary(BitmapInvertedIndexWriter.java:95) at org.apache.pinot.segment.local.segment.creator.impl.inv.BitmapInvertedIndexWriter.add(BitmapInvertedIndexWriter.java:73) at org.apache.pinot.segment.local.segment.creator.impl.inv.json.OnHeapJsonIndexCreator.seal(OnHeapJsonIndexCreator.java:57) at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.seal(SegmentColumnarIndexCreator.java:560) at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:266) at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:238) at org.apache.pinot.hadoop.job.mappers.SegmentCreationMapper.map(SegmentCreationMapper.java:277) ... 9 more Suppressed: java.lang.IllegalArgumentException: Negative size at sun.nio.ch.FileChannelImpl.truncate(FileChannelImpl.java:324) at org.apache.pinot.segment.local.segment.creator.impl.inv.BitmapInvertedIndexWriter.close(BitmapInvertedIndexWriter.java:118) at org.apache.pinot.segment.local.segment.creator.impl.inv.json.OnHeapJsonIndexCreator.seal(OnHeapJsonIndexCreator.java:50) ... 13 more

Hello, when is the next release expected to be made?

I am targeting to cut the branch on Monday next week, so by month end

Awesome, thanks Atri

Hi, I am facing one issue while deploying pinot on K8s. Can you please help me out? I am trying to install pinot on k8's cluster which is the latest helm released chart:

0.9.3

- https://github.com/apache/pinot/blob/master/kubernetes/helm/README.md The only change I have made is the

nodeSelector

in values.yaml file. I am getting errors while using this command for installing helm in K8's cluster

helm install pinot . -f values.yaml -n pinot --set cluster.name=pinot --set server.replicaCount=2

Here are the zookeeper logs:

+ /config-scripts/run
mkdir: cannot create directory '/data/log': No space left on device
/config-scripts/run: line 44: echo: write error: No space left on device
+ exec java -cp '/apache-zookeeper-3.5.5-bin/lib/*:/apache-zookeeper-3.5.5-bin/*jar:/conf:' -Xmx256M -Xms256M org.apache.zookeeper.server.quorum.QuorumPeerMain /conf/zoo.cfg
2022-06-23 09:56:06,224 [myid:] - INFO  [main:QuorumPeerConfig@133] - Reading configuration from: /conf/zoo.cfg
2022-06-23 09:56:06,244 [myid:] - INFO  [main:QuorumPeerConfig@385] - clientPortAddress is 0.0.0.0/0.0.0.0:2181
2022-06-23 09:56:06,244 [myid:] - INFO  [main:QuorumPeerConfig@389] - secureClientPort is not set
2022-06-23 09:56:06,294 [myid:] - ERROR [main:QuorumPeerConfig@645] - Invalid configuration, only one server specified (ignoring)
2022-06-23 09:56:06,307 [myid:] - ERROR [main:QuorumPeerMain@89] - Invalid config, exiting abnormally
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing /conf/zoo.cfg
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:154)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:113)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Caused by: java.lang.IllegalArgumentException: serverid null is not a number
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.setupMyId(QuorumPeerConfig.java:690)
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.setupQuorumPeerConfig(QuorumPeerConfig.java:602)
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:420)
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:150)
        ... 2 more
Invalid config, exiting abnormally

How to query to get difference between two tables using apache pinot PQL . can anyone please help on this?

What would you mean by difference?

I think we have an integration test regression. Seeing this failure in my PR and another independent PR. @Rong R wondering if you are familiar with GrpcBrokerClusterIntegrationTest.java. Seems like the file was last modified by you?

Error:  Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project pinot-integration-tests: There are test failures.
Error: 
Error:  Please refer to /home/runner/work/pinot/pinot/pinot-integration-tests/target/surefire-reports for the individual test results.
Error:  Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
Error:  ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error:  Command was /bin/sh -c cd /home/runner/work/pinot/pinot/pinot-integration-tests && /opt/hostedtoolcache/jdk/11.0.15/x64/bin/java '-javaagent:/home/runner/.m2/repository/org/jacoco/org.jacoco.agent/0.8.6/org.jacoco.agent-0.8.6-runtime.jar=destfile=/home/runner/work/pinot/pinot/pinot-integration-tests/target/jacoco.exec,includes=org/apache/pinot/**/*' -Xms8g -Xmx8g -Dlog4j2.configurationFile=log4j2.xml -jar /home/runner/work/pinot/pinot/pinot-integration-tests/target/surefire/surefirebooter17272138790391647176.jar /home/runner/work/pinot/pinot/pinot-integration-tests/target/surefire 2022-06-27T21-02-39_952-jvmRun1 surefire16409506125147645868tmp surefire_489742404707542412tmp
Error:  Error occurred in starting fork, check output in log
Error:  Process Exit Code: 137
Error:  Crashed tests:
Error:  org.apache.pinot.integration.tests.GrpcBrokerClusterIntegrationTest
Error:  org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error:  Command was /bin/sh -c cd /home/runner/work/pinot/pinot/pinot-integration-tests && /opt/hostedtoolcache/jdk/11.0.15/x64/bin/java '-javaagent:/home/runner/.m2/repository/org/jacoco/org.jacoco.agent/0.8.6/org.jacoco.agent-0.8.6-runtime.jar=destfile=/home/runner/work/pinot/pinot/pinot-integration-tests/target/jacoco.exec,includes=org/apache/pinot/**/*' -Xms8g -Xmx8g -Dlog4j2.configurationFile=log4j2.xml -jar /home/runner/work/pinot/pinot/pinot-integration-tests/target/surefire/surefirebooter17272138790391647176.jar /home/runner/work/pinot/pinot/pinot-integration-tests/target/surefire 2022-06-27T21-02-39_952-jvmRun1 surefire16409506125147645868tmp surefire_489742404707542412tmp
Error:  Error occurred in starting fork, check output in log
Error:  Process Exit Code: 137
Error:  Crashed tests:
Error:  org.apache.pinot.integration.tests.GrpcBrokerClusterIntegrationTest
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:479)
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:322)
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
Error:  	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:370)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:351)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:215)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:171)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:163)
Error:  	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
Error:  	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
Error:  	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
Error:  	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
Error:  	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:294)
Error:  	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
Error:  	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
Error:  	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:960)
Error:  	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:293)
Error:  	at org.apache.maven.cli.MavenCli.main(MavenCli.java:196)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Error:  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Error:  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
Error:  Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
Error:  Command was /bin/sh -c cd /home/runner/work/pinot/pinot/pinot-integration-tests && /opt/hostedtoolcache/jdk/11.0.15/x64/bin/java '-javaagent:/home/runner/.m2/repository/org/jacoco/org.jacoco.agent/0.8.6/org.jacoco.agent-0.8.6-runtime.jar=destfile=/home/runner/work/pinot/pinot/pinot-integration-tests/target/jacoco.exec,includes=org/apache/pinot/**/*' -Xms8g -Xmx8g -Dlog4j2.configurationFile=log4j2.xml -jar /home/runner/work/pinot/pinot/pinot-integration-tests/target/surefire/surefirebooter17272138790391647176.jar /home/runner/work/pinot/pinot/pinot-integration-tests/target/surefire 2022-06-27T21-02-39_952-jvmRun1 surefire16409506125147645868tmp surefire_489742404707542412tmp
Error:  Error occurred in starting fork, check output in log
Error:  Process Exit Code: 137
Error:  Crashed tests:
Error:  org.apache.pinot.integration.tests.GrpcBrokerClusterIntegrationTest
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:748)
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$700(ForkStarter.java:121)
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:465)
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:442)
Error:  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Error:  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Error:  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Error:  	at java.base/java.lang.Thread.run(Thread.java:829)
Error:  -> [Help 1]
Error: 
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error: 
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] <http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException>
Error: 
Error:  After correcting the problems, you can resume the build with the command
Error:    mvn <args> -rf :pinot-integration-tests
+ exit 1
Error: Process completed with exit code 1.

Yeah it is a Linux runner OOM issue https://github.com/apache/pinot/issues/8968

ok thanks, so should we just wait for this fix or any workaround to merge PR?

Run it again should fix the issue

Unfortunately I havent been able to reproduce locally. It seems like the GHA runner were so slow that some of these bugs are surfacing

👍

hi, I'm trying to understand how to index geospatial field which is linestring. In the example you use stPoint, is there stLineString?

{
  "dataType": "BYTES",
  "name": "location_st_point",
  "transformFunction": "toSphericalGeography(stPoint(lon,lat))"
}

someting like that?

{
  "dataType": "BYTES",
  "name": "location_linestring",
  "transformFunction": "toSphericalGeography(stLinestring(arr))"
}

Hi all, the avro decoder bundled with pinot doesn't handle poison pills and I was wondering if there was any reason for this. I need to handle common deserialization exceptions and it be nice to get this out of the box instead of I having to write and bundle a plugin

There seems to be some confusion around this PR https://github.com/apache/pinot/pull/9051 please let me know whether it's better to discuss over slack

thanks Dan. I am trying to understand Subbu's comment

IMO, what he is suggesting does not address the issue you are facing

Yup, exactly

I do have some questions around the change • will returning null be handled gracefully by Pinot?

It should, null rows are counted as errors by the LLC data manager https://github.com/apache/pinot/blob/master/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java#L523

I'd like to add a test but I can't find an integration test using avro and kafka

So the kafka avro serde that ships confluent writes a binary payload where the first byte is a magic byte (0) and the next 4 bytes are the schema ID, the rest is the avro payload

If the first byte isn't 0 the message will no be decoded and an exception with that message thrown

and the offset moves to the next payload right?

It won't move because the consumer will fail

sorry, I was referring to case where returned decodedRow is null

Oh yes, sorry I misunderstood your question, the stream should move on if the row is null

Is a similar behaviour to the json decoder