Hey Team, We are observing increased query latency after increase in GC count and GC time. W/o GC issue queries are much faster (<100 ms), with increase in GC count and GC time the query latency is >10 secs. We are using Java 11. Below are GC logs

[85563.074s][info][gc             ] GC(5548) Pause Full (G1 Evacuation Pause) 32705M->32631M(32768M) 3647.713ms
[85563.074s][info][gc,cpu         ] GC(5548) User=45.55s Sys=0.00s Real=3.65s
[85563.111s][info][gc,start       ] GC(5549) Pause Young (Concurrent Start) (G1 Evacuation Pause)
[85563.111s][info][gc,task        ] GC(5549) Using 13 workers of 13 for evacuation
[85563.132s][info][gc,mmu         ] GC(5549) MMU target violated: 20.6ms (20.0ms/21.0ms)
[85563.132s][info][gc             ] GC(5549) To-space exhausted
[85563.132s][info][gc,phases      ] GC(5549)   Pre Evacuate Collection Set: 0.4ms
[85563.132s][info][gc,phases      ] GC(5549)   Evacuate Collection Set: 15.2ms
[85563.132s][info][gc,phases      ] GC(5549)   Post Evacuate Collection Set: 4.7ms
[85563.132s][info][gc,phases      ] GC(5549)   Other: 0.5ms
[85563.132s][info][gc,heap        ] GC(5549) Eden regions: 12->0(204)
[85563.132s][info][gc,heap        ] GC(5549) Survivor regions: 0->0(26)
[85563.132s][info][gc,heap        ] GC(5549) Old regions: 3311->3323
[85563.132s][info][gc,heap        ] GC(5549) Humongous regions: 773->773
[85563.132s][info][gc,metaspace   ] GC(5549) Metaspace: 144441K(152832K)->144441K(152832K)

GC setting

-Xms24G -Xmx32G -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:G1ReservePercent=15 -XX:+ParallelRefProcEnabled -XX:+DisableExplicitGC -XX:InitiatingHeapOccupancyPercent=35

Is there any recommended setting for GC?

Hi folks, I am seeing the OSS PR merge blocked by many flaky tests these day. E.g.

05:42:50.343 WARN [IdealStateGroupCommitTest] [pool-30-thread-100] IdealState updater 91 failed to commit.
java.lang.RuntimeException: Caught exception while updating ideal state for resource: potato_5_OFFLINE
	at org.apache.pinot.common.utils.helix.IdealStateGroupCommit.updateIdealState(IdealStateGroupCommit.java:312) ~[pinot-common-1.3.0-SNAPSHOT.jar:1.3.0-SNAPSHOT-e782d0f645ff1c8a32bb0b9c5a0e07f991d50ec5]
	at org.apache.pinot.common.utils.helix.IdealStateGroupCommit.commit(IdealStateGroupCommit.java:124) ~[pinot-common-1.3.0-SNAPSHOT.jar:1.3.0-SNAPSHOT-e782d0f645ff1c8a32bb0b9c5a0e07f991d50ec5]
	at org.apache.pinot.controller.helix.IdealStateUpdater.run(IdealStateGroupCommitTest.java:149) ~[test-classes/:?]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed afte

this error is printing everywhere in UT set 2 in many unrelated PRs. e.g. 14229 14244 A few days back, we just saw and fixed one case due to a PR whose UT is failing got merged. Could anyone help take a look? Thanks

Hi, Im trying to batch ingest some tab-separated csv files, but I can figure out how to specify that the files are tab-separated in the job-spec-file. a simple ' \t' doesnt work. Anyone got a suggestion for me?

Hey Team, We have a use-case where we are using ingestion aggregation with COUNT(*) function. We have 2 dimension column, 1 metric column and 1 timestamp column. One of the dimension column has high cardinality and we need to disable dictionary for that. But when we disable dictionary for that column, ingestion aggregation stops working and all values from ingestion aggregation are zero. Is this expected?

hi We have accidentally configured pinot retension period to -1 thinking it will keep the data forever but the data is deleted from Realtime table, we can only get 1 week data from kafka, but we see deleted segments in server folders, we tried to copy them back from deleted segments to table folder in server, but that did not work, wee don't have deep store configured as of now. we need solution without deepstore @Subbu Subramaniam @Mayank can you help us with this

Did you try uploading the segments back to the table (after re-configuring the retention period)? @Sajjad Moradi

we tried to take the segments from deleted segments in server node and copied them back to the $table_name_Realtime folder where generally segments recide @Subbu Subramaniam

If you still have the segments on the servers, tar them and upload the tarred segments to controller using

/segments

endpoint:

$ curl -X POST -F segName=@segName.tar.gz controllerHost:controllerPort/segments

Yes, this is what I also suggested in a DM ping. Also, to clarify @samirun shaik deepstore is not for preventing against such deletes, it is more for data durability and disaster recovery.

Hey team, I hope we're fine I want to import CSV or JSON to PINOT, how do I do it?

I'm ingesting log-files into my Pinot-Cluster. My problem is that some of the data looks corrupt in the database, but I cant see anything diffrent in the log-files them self. They look consistent, but some rows end up corrupt in the database. What is the most effective way to debug the ingestion?

Hi Team, I have a quick question if server changed the IP address how do we reassign the segments ?

Hi team, we are running into some issues with Multi-Stage Query engine with our pinot cluster. > Error Code: 200 > QueryExecutionError: > Error dispatching query to server=pinot-server-dev-1.example.com@{-1,-1} stage=1 > org.apache.pinot.query.service.dispatch.QueryDispatcher.submit(QueryDispatcher.java:144) > org.apache.pinot.query.service.dispatch.QueryDispatcher.submitAndReduce(QueryDispatcher.java:93) > org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:179) > org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:263) > UNAVAILABLE: io exception > io.grpc.Status.asRuntimeException(Status.java:539) > io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487) > io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:576) > io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) > finishConnect(..) failed: Connection refused: pinot-server-dev-1.example.com/{ip obfuscated}:80 > finishConnect(..) failed: Connection refused > io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) > io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) > io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:321) > io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) > finishConnect(..) failed: Connection refused > io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) > io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) > io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:321) > io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) Any idea what might be causing a redirection to port

80

(

finishConnect(..) failed: Connection refused: <http://pinot-server-dev-1.example.com/{ip|pinot-server-dev-1.example.com/{ip> obfuscated}:80

)? Nothing is listening on that

server

port. This configuration is using an unspecified `pinot.query.server.port`/`pinot.query.runner.port`.

hi everyone. I'm working on configuring Pinot to store segments in MinIO, and I've followed the configuration instructions in the documentation, adding the necessary settings to the

values.yaml

file in my Helm deployment. However, after ingesting a real-time table, I don't see any segments being saved to MinIO. Below is my configuration: Pinot Controller configuration: # ------------------------------------------------------------------------- # Pinot Controller: # ------------------------------------------------------------------------- ... data: dir: s3://pinot-segments # Extra configs will be appended to pinot-controller.conf file extra: configs: |- controller.disable.ingestion.groovy=false pinot.set.instance.id.to.hostname=true controller.task.scheduler.enabled=true controller.enable.split.commit=true pinot.controller.storage.factory.s3.accessKey=minioadmin pinot.controller.storage.factory.s3.secretKey=minioadmin pinot.controller.segment.fetcher.protocols=file,http,s3 pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.controller.storage.factory.s3.endpoint=http://10.168.6.104:9000 pinot.controller.storage.factory.s3.region=us-west-1 pinot.controller.storage.factory.s3.disableAcl=false controller.local.temp.dir=/var/pinot/controller/data ... Pinot Server configuration: # ------------------------------------------------------------------------- # Pinot Server: # ------------------------------------------------------------------------- ... dataDir: s3://pinot-segments segmentTarDir: /var/pinot/server/data/segment # Extra configs will be appended to pinot-server.conf file extra: configs: |- pinot.set.instance.id.to.hostname=true pinot.server.instance.realtime.alloc.offheap=true pinot.query.server.port=7321 pinot.query.runner.port=7732 pinot.broker.disable.query.groovy=false pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.server.storage.factory.s3.region=us-east-1 pinot.server.storage.factory.s3.accessKey=minioadmin pinot.server.storage.factory.s3.secretKey=minioadmin pinot.server.storage.factory.s3.disableAcl=false pinot.server.storage.factory.s3.endpoint=http://10.168.6.104:9000 pinot.server.segment.fetcher.protocols=file,http,s3 pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher ...

Looking at these docs for using HDFS as deep storage (https://docs.pinot.apache.org/basics/getting-started/hdfs-as-deepstorage) The server configs has this property for its own storage: pinot.server.instance.dataDir=/path/in/local/filesystem/for/pinot/data/server/index Can this path be in HDFS too? As is the dataDir is not on local filesystem but in HDFS.

Hi pinot friends! We are updating from .12 to 1.2 in our gke environment and all things went well except for the following error being sent around every 10s ([test__0__5__20241022T1906Z] ERROR org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager_test__0__5__20241022T1906Z - Message loss detected in stream partition: 0 for table: test_REALTIME startOffset: 51465 batchFirstOffset: 55568). I have searched and a few other people mentioned this and I have made sure to follow any advice there. I believe we are consuming all events from kafka (will keep confirming that), so not sure why we're getting this error message. Any advice is appreciated! Here are what I believe to be the relevant configs: "ingestionConfig": { "continueOnError": true, "rowTimeValueCheck": true, "streamIngestionConfig": { "streamConfigMaps": [ { "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.segment.size": "500M", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder", "streamType": "kafka", "stream.kafka.consumer.type": "LOWLEVEL", "realtime.segment.flush.threshold.segment.rows": "$REALTIME_SEGMENT_FLUSH_THRESHOLD_ROWS", "stream.kafka.broker.list": "$BOOTSTRAP_SERVERS", "realtime.segment.flush.threshold.time": "$REALTIME_IMMUTABLE_SEGMENT_FLUSH_THRESHOLD_TIME", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.consumer.prop.auto.offset.reset": "smallest", "stream.kafka.isolation.level": "read_committed", "stream.kafka.topic.name": "pinot-test", "stream.kafka.decoder.prop.schema.registry.rest.url": "$SCHEMA_REGISTRY_URL", "schema.registry.url": "$SCHEMA_REGISTRY_URL" } ], "columnMajorSegmentBuilderEnabled": true, "trackFilteredMessageOffsets": false }, "transformConfigs": [], "segmentTimeValueCheck": false },

What should be the correct order of restarting pinot services to minimize impact ? Pinot controller -> Server -> broker

Hey Team, We have observed an issue multiple times recently. For a segment, one of the host returns wrong results while the other server which keeps the same segment returns correct result. After restarting the server which is returning wrong results, data looks correct. Can someone help to figure why this might be happening? Pinot version: 1.0.0

🧵 Dear Pinot Technical Support Team, I have 3 questions about using Minion on MergeAndRollUp Task. Please kindly help me clarify my doubts and observations. Thank you. 1. Can Minion support maxSizePerSegment ? (instead of maxRowsPerSegment) 2. After merge will the source segment get deleted ? 3. Can we control the minion segment name ? Thank you

We've got some empty real time tables setup (data's not yet flowing for them) reporting 5gb of off heap memory usage per server. Is this some kind of default allocation for real time segments that doesn't get adjusted until data is actually ingested, a metric reporting issue?

Logs are not are not generating it is failed due to location is not found

Can you please help me here ?

Hey Team, Can we use Java 21 with Pinot?

Hi Team , I am trying to upgrade to Pinot 1.2 and facing issue with HDFS deepstore integration . We have working integration for 1.1 , following https://docs.pinot.apache.org/release-1.2.0/basics/getting-started/hdfs-as-deepstorage#executable-1 We are seeing below error

2024/10/25 12:17:22.939 INFO [PinotFSFactory] [main] Got scheme hdfs, initializing class org.apache.pinot.plugin.filesystem.HadoopPinotFS
2024/10/25 12:17:22.939 INFO [PinotFSFactory] [main] Initializing PinotFS for scheme hdfs, classname org.apache.pinot.plugin.filesystem.HadoopPinotFS
2024/10/25 12:17:23.125 WARN [NativeCodeLoader] [main] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024/10/25 12:17:23.198 ERROR [StartServiceManagerCommand] [main] Failed to start a Pinot [CONTROLLER] at 8.227 since launch
java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
        at org.apache.hadoop.fs.FsTracer.get(FsTracer.java:42) ~[hadoop-common-3.2.3.3.2.2.0-1.jar:?]
        at org.apache.hadoop.fs.FileSystem.java:3370) ~[hadoop-common-3.2.3.3.2.2.0-1.jar:?]
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:125) ~[hadoop-common-3.2.3.3.2.2.0-1.jar:?]
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3424) ~[hadoop-common-3.2.3.3.2.2.0-1.jar:?]
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3392) ~[hadoop-common-3.2.3.3.2.2.0-1.jar:?]
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:485) ~[hadoop-common-3.2.3.3.2.2.0-1.jar:?]
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:233) ~[hadoop-common-3.2.3.3.2.2.0-1.jar:?]
        at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:67) ~[pinot-hdfs-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
        at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:52) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.

In the executable file classpath is set as

export HADOOP_HOME=/usr/odp/3.2.2.0-1

export CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-${HADOOP_GSON_VERSION}.jar"

Further debugging understood that it due to mismatch box has

hadoop-common-3.2.3.3.2.2.0-1

but pinot-1.2 uses

hadoop-common-3.3.6

Tracer class has been moved from to

org.apache.htrace.core.Tracer

->

org.apache.hadoop.tracing.Tracer

Since pinot

fat/shaded

jars

apache-pinot-1.2.0-bin/lib/pinot-all-1.2.0-jar-with-dependencies.jar , apache-pinot-1.2.0-bin/plugins/pinot-file-system/pin

ot-hdfs/pinot-hdfs-1.2.0-shaded.jar

are being added to the classpath anyway later on .. I removed my boxes hadoop lib from classpath and controller startup succeeded . Query 1. If the fat jar's have all needed libraries .. do we still need to add hadoop libraries in the classpath in executible file as per documentation ? 2. We will not be able to update hadoop lib version .. so will there be any issues by removing conflicting versions and relying on pinot fat jars. Please help with above query ?

Hello, For multistage engine we are observing a bug. Please suggest if there is a workaround or what is causing this issue. For the below query we get 0 records in multistage engine. But the same query in single stage returns correct records satisfying the intersection of AND clause.

SELECT count(*)
FROM usersample_main
WHERE arraytomv(dmp_segments) IN (1132) AND arraytomv(dmp_segments)IN (1135)

cc: @Mayank @Jackie @Xiang Fu

Hello , For realtime table we have the configuration replication = 2 replicationPerPartition = 2 number of servers = 2 segment count in server 1 = 308 segment count in server 2 = 308 Deepstore segment count = 171 Since we have mentioned the replication as 2 , total number of segments across the server should be 171+171=242 , But here the total count is 616, how ? What is the difference between replication and replicaPerPartition ?

Hi Team, Regarding Query Explain plan, how to check if inverted index is being used or not ? Usage: select .... from fact_table a left join dim_table b ON a.index_column = b.join_column where a.time_column between ... group by ... Also, we have observed that queries with multi-stage engine is running slower than single stage. 15-20seconds in Multi-Stage 2-4 seconds in Single Stage cc: @Jackie @Xiang Fu

Hello, We are migrating from druid to pinot, we have a use case where we want to consume data from kafka topic but there is no schema in kafka topic as it is json and addition of fields is very common as multiple teams are writing to same topic. we have this support in druid where addition of new fields is auto detected and added in druid table how to achieve this in pinot as we need to pre define schema in pinot? cc: @Mayank

Hello, we have been using json_extract_scalar to pull values out of some json columns. We have found when extracting very large LONG numbers, it is losing some precision, even though the numbers should be able to fit into a LONG size just fine. For example, if I have a json column called

properties

and the json looks like

{
  "num_clicks": "5514400327644543899"
}

and we use

json_extract_scalar(properties, '$.num_clicks', 'LONG', 0)

, the actual resulting value is

5514400327644544000

Has anybody else run into this? I have also reproduced a failing test here that shows the problem: https://github.com/apache/pinot/pull/14332/files

Before I go implementing something from scratch, are there any existing examples of minion tasks that delete rows by particular column values in all segments for a table, then replace each segment with the new segment that has the rows removed?