Hi All. I enabled

lucene

based

text index

on one column and ran into the following error:

2023/06/02 13:03:25.464 WARN [viewName_REALTIME-RealtimeTableDataManager] [HelixTaskExecutor-message_handle_thread_25] Failed to download segment viewName__1__9294__20230602T1151Z from deep store:
java.lang.RuntimeException: java.lang.RuntimeException: Caught exception while instantiating the LuceneTextIndexCreator for column: columnName
        at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.replaceLLSegment(RealtimeTableDataManager.java:552) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.untarAndMoveSegment(RealtimeTableDataManager.java:493) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadSegmentFromDeepStore(RealtimeTableDataManager.java:471) [pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadAndReplaceSegment(RealtimeTableDataManager.java:440) [pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.downloadSegmentAndReplace(LLRealtimeSegmentDataManager.java:1201) [pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.goOnlineFromConsuming(LLRealtimeSegmentDataManager.java:1188) [pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromConsuming(SegmentOnlineOfflineStateModelFactory.java:112) [pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at jdk.internal.reflect.GeneratedMethodAccessor424.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) [pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) [pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.RuntimeException: Caught exception while instantiating the LuceneTextIndexCreator for column: columnName
        at org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.<init>(LuceneTextIndexCreator.java:113) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider.newTextIndexCreator(DefaultIndexCreatorProvider.java:164) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.spi.index.IndexingOverrides$Default.newTextIndexCreator(IndexingOverrides.java:191) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.segment.index.loader.invertedindex.TextIndexHandler.createTextIndexForColumn(TextIndexHandler.java:188) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.segment.index.loader.invertedindex.TextIndexHandler.updateIndices(TextIndexHandler.java:140) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessor.process(SegmentPreProcessor.java:112) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.preprocessSegment(ImmutableSegmentLoader.java:267) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.preprocess(ImmutableSegmentLoader.java:143) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:110) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:92) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.core.data.manager.BaseTableDataManager.addSegment(BaseTableDataManager.java:222) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.replaceLLSegment(RealtimeTableDataManager.java:550) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        ... 17 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /var/pinot/server/data/index/viewName_REALTIME/viewName__1__9294__20230602T1151Z/v3/columnName.lucene.index/write.lock
        at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:139) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:729) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.<init>(LuceneTextIndexCreator.java:111) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider.newTextIndexCreator(DefaultIndexCreatorProvider.java:164) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.spi.index.IndexingOverrides$Default.newTextIndexCreator(IndexingOverrides.java:191) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.segment.index.loader.invertedindex.TextIndexHandler.createTextIndexForColumn(TextIndexHandler.java:188) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.segment.index.loader.invertedindex.TextIndexHandler.updateIndices(TextIndexHandler.java:140) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.segment.index.loader.SegmentPreProcessor.process(SegmentPreProcessor.java:112) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.preprocessSegment(ImmutableSegmentLoader.java:267) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.preprocess(ImmutableSegmentLoader.java:143) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:110) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:92) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.core.data.manager.BaseTableDataManager.addSegment(BaseTableDataManager.java:222) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.replaceLLSegment(RealtimeTableDataManager.java:550) ~[pinot-all-hypertrace-0.12.0-5-shaded.jar:0.12.0-06acc7c10dc7a30a35e713da60fd9516e7efd1be]
        ... 17 more

Has anyone seen this issue? Is it a known issue?

hey team! I am trying to upload a csv to a table on our servers -- I wrote a job_spec.yaml file, cp'd it to the pod, and then tried to run

bin/pinot-admin.sh LaunchDataIngestionJob \ -jobSpecFile configs/job_spec.yaml

to kick off the job ingestion. I got this error:

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.net.BindException: Address already in use
        at java.base/sun.nio.ch.Net.bind0(Native Method)
        at java.base/sun.nio.ch.Net.bind(Net.java:459)
        at java.base/sun.nio.ch.Net.bind(Net.java:448)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
        at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
        at jdk.httpserver/sun.net.httpserver.ServerImpl.bind(ServerImpl.java:134)
        at jdk.httpserver/sun.net.httpserver.HttpServerImpl.bind(HttpServerImpl.java:54)
        at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:145)
        at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:31)
        ... 6 more
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed
Aborted (core dumped)

I'm working on k8s deployment of

pinot

. I'm having issues with bootstrapping

server

pods where

Server

pods log message that they have successfully started, yet

Broker

pods are getting `ConnectionTimeoutException`s trying to reach them. Any tips to help troubleshoot why a new node is not successfully connecting to the cluster?

Hello team - I am getting this error when I am parsing tsv file with CSV config modified. The exception is happening after segments are created. Below is the exception stackTrace. can someone provide pointers Caused by: java.lang.RuntimeException: Failed to generate Pinot segment for file - file:/<file path>/hits_data_smallaa.tsv at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.lambda$submitSegmentGenTask$1(SegmentGenerationJobRunner.java:286) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1623) Caused by: java.io.IOException: Failed to retrieve file descriptor of /var/folders/qn/606hgfg55fj7b3y47g7lrhzh0000gp/T/pinot-5a95541b-19a4-4759-955f-b5bd805cbcea/output/hitsTable_OFFLINE_0/v3/star_tree_tmp/star-tree.record: Unable to make field private int java.io.FileDescriptor.fd accessible: module java.base does not "opens java.io" to unnamed module @3f3e6f71 at xerial.larray.mmap.MMapBuffer.<init>(MMapBuffer.java:73) at org.apache.pinot.segment.spi.memory.PinotNativeOrderLBuffer.mapFile(PinotNativeOrderLBuffer.java:49) at org.apache.pinot.segment.spi.memory.PinotDataBuffer.mapFile(PinotDataBuffer.java:194) at org.apache.pinot.segment.local.startree.v2.builder.OffHeapSingleTreeBuilder.ensureBufferReadable(OffHeapSingleTreeBuilder.java:186) at org.apache.pinot.segment.local.startree.v2.builder.OffHeapSingleTreeBuilder.getDimensionValue(OffHeapSingleTreeBuilder.java:174) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructNonStarNodes(BaseSingleTreeBuilder.java:368) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:348) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.constructStarTree(BaseSingleTreeBuilder.java:359) at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.build(BaseSingleTreeBuilder.java:311) at org.apache.pinot.segment.local.startree.v2.builder.OffHeapSingleTreeBuilder.build(OffHeapSingleTreeBuilder.java:42) at org.apache.pinot.segment.local.startree.v2.builder.MultipleTreesBuilder.build(MultipleTreesBuilder.java:141) at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.buildStarTreeV2IfNecessary(SegmentIndexCreationDriverImpl.java:348) at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:312) at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:260) at org.apache.pinot.plugin.ingestion.batch.common.SegmentGenerationTaskRunner.run(SegmentGenerationTaskRunner.java:113) at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.lambda$submitSegmentGenTask$1(SegmentGenerationJobRunner.java:264) ... 5 more

👋 Hello, team! There is a sentence in the Pinot Docs #HDFS as Deep Storage : "To use HDFS as deep storage you need to include HDFS dependency jars and plugins." I'd like to know what exact the HDFS dependency jars and plugins are? A full Hadoop cluster with HDFS is needed or just some jar packages? If jar packages, what packages are they? If I can get an entire sample for using HDFS as Deep Storage , that would be really helpful to me.

hey guys! @Mayank @Mark Needham Is there a table which shows me query logs in pinot from which I can see actual query execution time without network overhead? for example in ClickHouse there is a table

system.query_log

and i can query in it like below:

select avg(query_duration_ms) from system.query_log 
where match(query, 'from reels.view2')
and `type` = 'QueryFinish'

Hello team! We are seeing pinot servers crashing with

SIGSEGV

soon after we enabled lucene based

text index

on one column. Here is one such instance:

2023/06/05 10:57:24.157 INFO [MutableSegmentImpl_viewName__1__9363__20230605T0929Z_view-name] [HelixTaskExecutor-message_handle_thread_30] Trying to close RealtimeSegmentImpl : viewName__1__9363__20230605T0929Z
2023/06/05 10:57:24.157 INFO [MutableSegmentImpl_viewName__1__9363__20230605T0929Z_view-name] [HelixTaskExecutor-message_handle_thread_30] Segment used 4547528251 bytes of memory for 2525493 rows consumed in 5275 seconds
2023/06/05 10:57:24.219 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_30] Closing buffer org.apache.pinot.segment.spi.memory.PinotByteBuffer@51be6026
2023/06/05 10:57:24.237 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_30] Closing buffer org.apache.pinot.segment.spi.memory.PinotByteBuffer@243b600e
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f5aeb45d676, pid=1, tid=5080
#
# JRE version: OpenJDK Runtime Environment Temurin-11.0.18+10 (11.0.18+10) (build 11.0.18+10)
# Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.18+10 (11.0.18+10, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# J 8576 c2 java.nio.DirectByteBuffer.getInt(I)I java.base@11.0.18 (28 bytes) @ 0x00007f5aeb45d676 [0x00007f5aeb45d620+0x0000000000000056]
#
# Core dump will be written. Default location: /core.%e.1.%t
#
# An error report file with more information is saved as:
# /opt/pinot/hs_err_pid1.log
2023/06/05 10:57:24.247 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_30] Closing buffer org.apache.pinot.segment.spi.memory.PinotByteBuffer@5b5aed94
2023/06/05 10:57:24.255 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_30] Closing buffer org.apache.pinot.segment.spi.memory.PinotByteBuffer@79c70989
2023/06/05 10:57:24.260 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_30] Closing buffer org.apache.pinot.segment.spi.memory.PinotByteBuffer@15e05ae4
2023/06/05 10:57:24.263 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_30] Closing buffer org.apache.pinot.segment.spi.memory.PinotByteBuffer@fbfbe40
2023/06/05 10:57:24.281 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_30] Closing buffer org.apache.pinot.segment.spi.memory.PinotByteBuffer@68f9d3aa
2023/06/05 10:57:24.288 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_30] Closing buffer org.apache.pinot.segment.spi.memory.PinotByteBuffer@74c98f35
2023/06/05 10:57:24.289 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_30] Closing buffer org.apache.pinot.segment.spi.memory.PinotByteBuffer@283a45ea
2023/06/05 10:57:24.331 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_30] Deleted file /var/pinot/server/data/index/viewName_REALTIME/consumers/viewName__1__9363__20230605T0929Z.0
2023/06/05 10:57:24.390 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_30] Deleted file /var/pinot/server/data/index/viewName_REALTIME/consumers/viewName__1__9363__20230605T0929Z.1
2023/06/05 10:57:24.426 INFO [MmapMemoryManager] [HelixTaskExecutor-message_handle_thread_30] Deleted file /var/pinot/server/data/index/viewName_REALTIME/consumers/viewName__1__9363__20230605T0929Z.2
Compiled method (c2) 7597812 28400       4       org.apache.pinot.segment.local.realtime.impl.dictionary.StringOffHeapMutableDictionary::get (6 bytes)
 total in heap  [0x00007f5aec681090,0x00007f5aec682058] = 4040
 relocation     [0x00007f5aec681208,0x00007f5aec681290] = 136
 main code      [0x00007f5aec6812a0,0x00007f5aec6819c0] = 1824
 stub code      [0x00007f5aec6819c0,0x00007f5aec6819f8] = 56
 oops           [0x00007f5aec6819f8,0x00007f5aec681a00] = 8
 metadata       [0x00007f5aec681a00,0x00007f5aec681a90] = 144
 scopes data    [0x00007f5aec681a90,0x00007f5aec681d40] = 688
 scopes pcs     [0x00007f5aec681d40,0x00007f5aec681f20] = 480
 dependencies   [0x00007f5aec681f20,0x00007f5aec681f30] = 16
 handler table  [0x00007f5aec681f30,0x00007f5aec681fc0] = 144
 nul chk table  [0x00007f5aec681fc0,0x00007f5aec682058] = 152
Compiled method (c2) 7597812 35340       4       org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator::indexRow (2005 bytes)
 total in heap  [0x00007f5aecefd910,0x00007f5aecf0fab0] = 74144
 relocation     [0x00007f5aecefda88,0x00007f5aecefe318] = 2192
 main code      [0x00007f5aecefe320,0x00007f5aecf09e80] = 47968
 stub code      [0x00007f5aecf09e80,0x00007f5aecf0a228] = 936
 oops           [0x00007f5aecf0a228,0x00007f5aecf0a278] = 80
 metadata       [0x00007f5aecf0a278,0x00007f5aecf0a5b0] = 824
 scopes data    [0x00007f5aecf0a5b0,0x00007f5aecf0d6d8] = 12584
 scopes pcs     [0x00007f5aecf0d6d8,0x00007f5aecf0edb8] = 5856
 dependencies   [0x00007f5aecf0edb8,0x00007f5aecf0edf8] = 64
 handler table  [0x00007f5aecf0edf8,0x00007f5aecf0f818] = 2592
 nul chk table  [0x00007f5aecf0f818,0x00007f5aecf0fab0] = 664
Compiled method (c2) 7597813 28753       4       org.apache.pinot.segment.spi.index.mutable.MutableForwardIndex::getDictId (8 bytes)
 total in heap  [0x00007f5aec081510,0x00007f5aec081c78] = 1896
 relocation     [0x00007f5aec081688,0x00007f5aec0816c8] = 64
 main code      [0x00007f5aec0816e0,0x00007f5aec081940] = 608
 stub code      [0x00007f5aec081940,0x00007f5aec081958] = 24
 oops           [0x00007f5aec081958,0x00007f5aec081960] = 8
 metadata       [0x00007f5aec081960,0x00007f5aec0819d0] = 112
 scopes data    [0x00007f5aec0819d0,0x00007f5aec081af8] = 296
 scopes pcs     [0x00007f5aec081af8,0x00007f5aec081c18] = 288
 dependencies   [0x00007f5aec081c18,0x00007f5aec081c28] = 16
 handler table  [0x00007f5aec081c28,0x00007f5aec081c40] = 24
 nul chk table  [0x00007f5aec081c40,0x00007f5aec081c78] = 56

Has anyone faced this issue earlier? Any thoughts regarding potential root cause and resolution is greatly appreciated. cc @Mayank @Atri Sharma

Hi Team. Our production QPS averaging about 60 for pinot, but we are getting alot of exception and mostly due to Broker CPU is maxed out. We have already done one scale up operation, but it has not solved the problem. May I know if there is any query optimisation that we can apply for the reduce phase?

Hi Team, I was just wanted to check how consuming segments are converted to completed segment in case of Upsert table. If a record is coming and its older record was present on the completed segment how this is updated ? Why star tree index is not supported in Upsert tables ?

I added an Inverted Index to a very simple 3 column table. Queries execute fine on v1, but not on v2. This is the query:

SET timeoutMs=30000;
SET useMultistageEngine=true;
select * from $table limit 10

This is the exception from the server logs

SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable@78073e1f
io.grpc.StatusRuntimeException: INTERNAL: Invalid protobuf byte sequence
	at io.grpc.Status.asRuntimeException(Status.java:530)
	at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:218)
	at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:118)
	at io.grpc.MethodDescriptor.parseRequest(MethodDescriptor.java:307)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:333)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:316)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:835)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag.
	at com.google.protobuf.InvalidProtocolBufferException.invalidEndTag(InvalidProtocolBufferException.java:137)
	at com.google.protobuf.CodedInputStream$ArrayDecoder.checkLastTagWas(CodedInputStream.java:638)
	at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:868)
	at org.apache.pinot.common.proto.Worker$StagePlan$Builder.mergeFrom(Worker.java:3135)
	at org.apache.pinot.common.proto.Worker$StagePlan$1.parsePartialFrom(Worker.java:3469)
	at org.apache.pinot.common.proto.Worker$StagePlan$1.parsePartialFrom(Worker.java:3461)
	at com.google.protobuf.CodedInputStream$ArrayDecoder.readMessage(CodedInputStream.java:883)
	at org.apache.pinot.common.proto.Worker$QueryRequest$Builder.mergeFrom(Worker.java:1433)
	at org.apache.pinot.common.proto.Worker$QueryRequest$1.parsePartialFrom(Worker.java:1871)
	at org.apache.pinot.common.proto.Worker$QueryRequest$1.parsePartialFrom(Worker.java:1863)
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:86)
	at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
	at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parseFrom(ProtoLiteUtils.java:223)
	at io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:215)
	... 10 more

And the output from the query console is above

[
  {
    "errorCode": 200,
    "message": "QueryExecutionError:\njava.lang.RuntimeException: Error executing query: [0]@pinot-broker-0.pinot-broker-headless.pinot-dev-02.svc.cluster.local:40289 MAIL_RECEIVE(RANDOM_DISTRIBUTED)\n├── [1]@10.193.4.169:39021 MAIL_SEND(RANDOM_DISTRIBUTED)->{[0]@pinot-broker-0.pinot-broker-headless.pinot-dev-02.svc.cluster.local@{40289,40289}|[0]} (Subtree Omitted)\n├── [1]@10.193.1.220:42431 MAIL_SEND(RANDOM_DISTRIBUTED)->{[0]@pinot-broker-0.pinot-broker-headless.pinot-dev-02.svc.cluster.local@{40289,40289}|[0]} (Subtree Omitted)\n├── [1]@10.193.7.222:38803 MAIL_SEND(RANDOM_DISTRIBUTED)->{[0]@pinot-broker-0.pinot-broker-headless.pinot-dev-02.svc.cluster.local@{40289,40289}|[0]} (Subtree Omitted)\n├── [1]@10.193.4.40:42057 MAIL_SEND(RANDOM_DISTRIBUTED)->{[0]@pinot-broker-0.pinot-broker-headless.pinot-dev-02.svc.cluster.local@{40289,40289}|[0]} (Subtree Omitted)\n...\nCaused by: java.lang.RuntimeException: Error dispatching query to server=10.193.3.243@{38573,38853} stage=1\n\tat org.apache.pinot.query.service.dispatch.QueryDispatcher.submit(QueryDispatcher.java:170)\n\tat org.apache.pinot.query.service.dispatch.QueryDispatcher.submitAndReduce(QueryDispatcher.java:92)\n\t... 29 more\nCaused by: io.grpc.StatusRuntimeException: UNKNOWN\n\tat io.grpc.Status.asRuntimeException(Status.java:539)\n\tat io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)\n\tat io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:576)\n\tat io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)"
  }
]

Hello everybody, how do you monitor minion batch ingestion on schedule in production? What’s the best way to do it?

Hi Team

The number of partitions in input streams determines the partition numbers of the Pinot table. The more partitions you have in input topic/stream, more Pinot servers you can distribute the Pinot table to and therefore more you can scale the table horizontally. Do note that you can't increase the partitions in future for upsert enabled tables so you need to start with good enough partitions (atleast 2-3X the number of pinot servers)

What does this statement means 1. Can we increase our server count in future or not 2. Can we increase kafka partitions or not

Hi team，

select * FROM limbus_monitor
WHERE first_time < 1683275400
  AND company_id != ''
  AND company_id != 'null'
GROUP BY 1 OPTION(timeoutMs=80000)

When broker timeout = 60s, Query timeout = 80s, it will timeout after 60s. But I set Query timeout = 10ms, it will timeout after 10ms. Is the behavior of timeout like what I say?

Hi all, I have a table with around 8 billion. I want to be able to do the following - 1. count based on 2 columns (date and a name (around 100k distinct names)) 2. sum based on date 3. use cte's/joins (this would need the v2 multi stage engine I would think) 4. appy logarthmic methods to normalize skew to be able to create better charts/dashboards. Can I please get any suggestions based on the above as to what all index should be used (target sum and counts) and any configurations to optimize considering we want to achieve around 500ms-1s results.

Hi all, any help on this - https://apache-pinot.slack.com/archives/C011C9JHN7R/p1680511376204529 we are seeing the same issue for one of our tables having lucene text index.

Hi Team I was just going through Dedup documentation I just wanted to check if we can define some kind of watermark policy that define we will consider event for deduplication for a defined cycle based on the some time column( If set to 1 hour primary keys older than 1 hour will be removed form the in-memory map) If not memory required to maintain the primary key will keep on increasing

Hey can please someone help me debug this scheduling problem? Why is the cron schedule not being updated even if

"schedule"

in the task config was? Do I need to perform additional steps to do that?

Hello here, We are benchmarking pinot with our use cases. so far so good. We are using the the latest snapshot of the 0.13 release to try the V2 engine. We 2 tables, one small and one bigger table. the small one is about 300Mb with 1 segment and the big one is about 900Gb for 1500 segments. The data has been ingested use the spark batch from Pinot but we change the input by doing a repartition to get something homogeneous in term of segment size (which might be the issue). We are running 12 instances of the server with a xmx to 40Gb and a lot of CPU. We use S3 as deep storage and running on EKS. Our query is the following:

set timeoutMs=200000;
SELECT 
	col1,
	col2,
	col3 
FROM big_table 
INNER JOIN small_table ON small_table.ids in ('ABC','DEF','IJK') and small_table.codes = big_table.codes
WHERE (low_cadilanity_col in ('XYZ') or low_cadilanity_col is null)

This works well between 15 to 20 seconds but we would like to see if we could improve that speed. (We tried different indexes but nothing that made it better.) Any idea how we could improve that?

Hello team - quick question on GCP json key for service accounts as mentioned here . Due to data control issues, service account can not be used for access . This is not only for deep store but for input/output store as well. How can a normal user account can be configured - like apiKey or something like that. Any pointers will be helpful.

Hi team, I am trying to use Stream Ingestion with Upsert for our realtime metric. I have changed all configs required in the doc, but it seems not working as expected.

schema.yml

"primaryKeyColumns": ["order_id"],

and

table.yml

"upsertConfig": {
    "mode": "FULL",
    "comparisonColumn": "updated_at",
    "hashFunction": "NONE",
    "enableSnapshot": true
  },
  "routing": {
    "instanceSelectorType": "strictReplicaGroup",
    "segmentPrunerTypes": [
      "time"
    ]
  },

Hi team, not sure if I encountered a bug or I missed something… I have an upsert real-time table working as expected with the single comparison column config

comparisonColumn

. However, when I try the multiple comparison columns config

comparisonColumns

, the upsert behavior wasn’t happening, then i figured out the

comparisonColumns

wasn’t showing up in the table configs in the Pinot UI, though when I created the table no error reported as expected. What could have the

comparisonColumns

missing the in the table configs?

hi all! we are planning on deploying pinot to prod. will keeping trino and pinot on same machine/node group perform better than if they reside on different machines ? due to locality? Thanks!

Hi all, I’m having problems with zookeeper. While performing a minion batch ingestion a single server started to throw:

Session 0x20070c6a80f003d for sever pinot-prod-zookeeper/172.22.65.152:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.io.IOException: Packet len 1201140 is out of range! at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:121)

I’ve tried to increase

-Djute.maxbuffer=1000000

on zookeeper but it didn’t help. I’m running pinot and zookeeper on a GKE cluster on GCP. Did anybody ever faced this?

Hi Team In case on Upsert table when a segment is converted from Realtime to offline segment and if query is fired on the offline segment then only latest row is returned or all rows without upsert are returned ?

HI Team, curious to know how indexing works for offline tables. Does it store the indexing such as star-tree index in memory or with the segments or brokers.

Hi team , I have done ingestion to the table but a server was down on that time from a total replication of 3 , While Ingestion Job segment pushed successfully , But When I query I get alternative results 0 and exact one I have done reload all segments , It reamains the same Will share u some screen shots For the clarification of my doubt

Hi team, continuing from this discussion...applied the changes to have a ST index to get a counts by slug. The Explain plan shows that its taking the ST index, yet its not able to complete the query and throws a 504 timeout..the table has around 6b records with 100k+ slugs..Details in the thread..

I have deleted a Pinot table using the API, I dont see it on controller console, still i see msg. Any pointers much appreciated

{
  "code": 409,
  "error": "Table config for <tablename> already exists. If this is unexpected, try deleting the table to remove all metadata associated with it."
}

Hi guys I've just setup the rbac and now I'm wondering if is it possible to allow a users to query through controller but not allow them to view the the table definition/configuration (I dont want them to see the

"sasl.jaas.config"

info), or to achieve only by using the broker endpoint... Thanks!

Hi team, Sometimes this problem occurs in the query, and it disappears after checking twice, and the table structure has not changed. What kind of problem is this?