Hi everyone, is there a way to see more logs for controller or server? I have a short error response from the api when trying to create a table but there's not much details. I get

{
  "code": 400,
  "error": "Could not create StreamConfig using the streamConfig map"
}

I tried changing the log level by swagger and log4j configs but i got no results. I'm running pinot using docker compose

Hi @Xiang Fu , Hope UPSERT feature is available in pinot for dimension tables now. Do we have documentation on how to setup and also which version of pinot should we refer ?

Still In my old 0.12 version and restart one of my server ... While rebuilding segments for some index crashed on a weird OOM memory buffer error. Is there a way to keep to a resonable size or number of segment rebuilded or reset in // per server ? I’ve found out

pinot.server.instance.realtime.max.parallel.segment.builds

but not sure if my already closed (in a real-time table) segment are concerned about that ?

.

Date Time functions not working in 1.4 version According to https://docs.pinot.apache.org/functions/datetime-functions dateDiff and timestampDiff --> getting exception 'hours' or 'seconds' not a valid Timestamp field I did read one of Mayank's post suggesting for dateDiff had to pass in quotes but same exception for both the functions. Query failed with exceptions. Stats may be partial (SQLParsingError) ror: Caught exception while invoking method: timestampDiff with arguments: [' hours, 1700000000000, 1700010800000]: "hours" is not a valid Timestamp field Please refer the screenshot in the reply, if you notice I am not passing quotes to timestampDiff but in the error message I can still see @Xiang Fu @Mayank

@Xiang Fu Hi, where can I download the spark connector jar for pinot, I have use case to read from pinot and write to s3. I'm exploring possibilities using spark. Can someone pls assist with the configuration details for spark,pinot.. I tried with spark 3.5.8 and pinot 1.4 , I built jar from spark-connector directory using -

pbuild-shaded-jar

command. However I'm facing following error when I try to submit my pyspark code via spark submit utility

Screenshot

@Xiang Fu Im facing below error when trying to read pinot data using spark 3 connector, I'm able to successfully print scheme of table, however I'm facing below issue when printing df.show() or df.count().. can someone please help?

Traceback (most recent call last):
  File "/Users/nitheeshkumar.prabakaran/frameworks/pinot4/pinot/pinot-connectors/pinot-spark-3-connector/target/test.py", line 56, in <module>
    print(f"Total rows:{df.count()}")
  File "/Users/nitheeshkumar.prabakaran/frameworks/spark/spark-3.5.8-bin-hadoop3/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 1239, in count
  File "/Users/nitheeshkumar.prabakaran/frameworks/spark/spark-3.5.8-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/Users/nitheeshkumar.prabakaran/frameworks/spark/spark-3.5.8-bin-hadoop3/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 179, in deco
  File "/Users/nitheeshkumar.prabakaran/frameworks/spark/spark-3.5.8-bin-hadoop3/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o32.count.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 0.0 failed 1 times, most recent failure: Lost task 4.0 in stage 0.0 (TID 4) (10.191.22.199 executor driver): org.apache.pinot.connector.spark.common.PinotException: 10.250.19.128:7050(OFFLINE) could not respond the query
	at org.apache.pinot.connector.spark.common.reader.PinotServerDataFetcher.fetchData(PinotServerDataFetcher.scala:146)
	at org.apache.pinot.connector.spark.common.reader.PinotAbstractPartitionReader.getIteratorAndSource(PinotAbstractPartitionReader.scala:65)
	at org.apache.pinot.connector.spark.common.reader.PinotAbstractPartitionReader.$init$(PinotAbstractPartitionReader.scala:39)
	at org.apache.pinot.connector.spark.v3.datasource.PinotScan$$anon$1.<init>(PinotScan.scala:85)
	at org.apache.pinot.connector.spark.v3.datasource.PinotScan.$anonfun$createReaderFactory$1(PinotScan.scala:85)
	at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.advanceToNextIter(DataSourceRDD.scala:92)
	at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.hasNext(DataSourceRDD.scala:71)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.hashAgg_doAggregateWithoutKey_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenEvaluatorFactory$WholeStageCodegenPartitionEvaluator$$anon$1.hasNext(WholeStageCodegenEvaluatorFactory.scala:43)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
	at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
	at org.apache.spark.scheduler.Task.run(Task.scala:141)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
	at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Hi, Is there a way to get rid of constraint over timeColumn that it must always be between

1971-01-01 UTC

to

2071-01-01 UTC

? We do have usecases where we only have one timeColumn in the data and its values are outside this range.

Hello, for the consistent data load is there any requirement that data load must happen daily. The table load was working fine when i implemented it, but today when i am reloading the same table again after a week its throwing me error as attached.

"ingestionConfig": {
      "continueOnError": false,
      "retryOnSegmentBuildPrecheckFailure": false,
      "rowTimeValueCheck": false,
      "segmentTimeValueCheck": true,
      "batchIngestionConfig": {
        "segmentIngestionType": "REFRESH",
        "segmentIngestionFrequency": "DAILY",
        "consistentDataPush": true
      }
    },

Hi Team. When querying a Pinot data source from Trino, segments from the Pinot server are streamed via gRPC, bypassing the broker. I'm curious whether Pinot's index is used in this case. When I checked the Start-Tree, it didn't use the Start-Tree index. I'm also curious about whether other indexes are also used.

Hi team I'm exploring the spark connector to read from pinot, I have pinot service self hosted on AWS cloud and it has enabled basic auth (user and password) to login..is there an option available in spark connector to pass credentials while reading from spark? Pinot version - 1.3.0 Spark version - 3.5.2 @Xiang Fu please help me with this

Team, Just created a test table consuming a test kafka topic.. it works in other cluster, but in on one cluster it points to

Specified EXTERNALVIEW pinot_test_REALTIME is not found!

There are other tables in prod that works from same kafka cluster, no new tables are working now. All ended up in this error or consuming_segment_info shows all servers as not responding, but they are working for others and are up. Tried, rebalance, reload, pause and resume.. seems externalview i dont see this table at all. Should we have to go for restarts? am worried what if currently working table runs into similar issue and unrecoverable.. any thoughts here?

Hi folks, Can anyone please tell me the place where I can download the shaded jar for spark -3 connector 1.4.0 or 1.30.. I don't find it the maven repository.

Hi everyone, I’m trying to clean up empty segments on a REALTIME table in Pinot 1.4.0 and wanted to check if I’m using PurgeTask correctly (no ref in the documentation). Context: • REALTIME table • no upsert • MergeRollupTask is already working fine and merging small segments • but I still have many segments with totalDocs = 0 • I want to automatically delete only those empty segments I found PR #16368 (feat: Enhance PurgeTask to automatically delete empty segments), so I tried adding PurgeTask like this:

"task": {
  "taskTypeConfigsMap": {
    "MergeRollupTask": {
      "1day.mergeType": "concat",
      "1day.bucketTimePeriod": "1d",
      "1day.bufferTimePeriod": "1d",
      "1day.maxNumRecordsPerSegment": "5000000",
      "1day.maxNumRecordsPerTask": "50000000",
      "1day.maxNumParallelBuckets": "5"
    },
    "PurgeTask": {
      "schedule": "0 */10 * ? * *"
    }
  }
}

But when it runs, I get this error in the controller logs:

ERROR [PinotTaskManager] Failed to generate tasks for task type PurgeTask for table analyse_test_REALTIME
java.lang.NumberFormatException: Cannot parse null string
    at java.base/java.lang.Long.parseLong(Long.java:674)
    at java.base/java.lang.Long.valueOf(Long.java:1163)
    at org.apache.pinot.plugin.minion.tasks.purge.PurgeTaskGenerator.generateTasks(PurgeTaskGenerator.java:121)

A few questions: 1. Is PurgeTask in Pinot 1.4.0 expected to support the “delete empty segments only” use case ? 2. Does it require additional config besides schedule? 3. Is this a known bug in PurgeTaskGenerator? 4. What is the recommended way today to automatically clean up totalDocs = 0 segments on a REALTIME non-upsert table? Thanks !

Hi Team, How to achieve Baremetal Awarness. (Consider Rack Awarness is Baremetal Awarness in my case) My architecture: 100 Baremetal(BM) 1BM = 2VM Pinotserver will be on each VM. Requirement is : primary(data1) , secondary(data1) can't be on same BM. i.e: IF BM1 has VM1 and VM2 ; tThen VM1 has p(data1) , VM2 shouldn't have p(data2). Same should follow across BMs and VMs. For that I wana check before I execute, would like to verify the steps to perform for the same with you. ON INSTANCE_VM1: POST http//&lt;CONTROLLER HOST&gt;<CONTROLLER_PORT>/instances/INSTANCE_VM1/updateInstanceConfig Request Body: JSON { "rack": "BM_1", "tags": ["BM_AWARE_REALTIME"] } ON INSTANCE_VM2: POST http//&lt;CONTROLLER HOST&gt;<CONTROLLER_PORT>/instances/INSTANCE_VM2/updateInstanceConfig Request Body: JSON { "rack": "BM_1", "tags": ["BM_AWARE_REALTIME"] } ON INSTANCE_VM3: POST http//&lt;CONTROLLER HOST&gt;<CONTROLLER_PORT>/instances/INSTANCE_VM3/updateInstanceConfig Request Body: JSON { "rack": "BM_2", "tags": ["BM_AWARE_REALTIME"] } And on Table:

{
  "tableName": "your_table_name_REALTIME",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "replication": "2",
    "schemaName": "your_schema_name",
    "replicasPerPartition": "2"
  },
  "tableIndexConfig": {
    "loadMode": "MMAP"
  },
  "instanceAssignmentConfigMap": {
    "REALTIME": {
      "tagPoolConfig": {
        "tag": "BM_AWARE_REALTIME",
        "poolBased": false
      },
      "partitionSelector": "FD_AWARE_INSTANCE_PARTITION_SELECTOR",
      "replicaGroupPartitionConfig": {
        "replicaGroupBased": true,
        "numReplicaGroups": 2,
        "numPartitions": 1,
        "numInstancesPerPartition": 100
      }
    }
  },
  "routing": {
    "instanceSelectorType": "replicaGroup"
  },
  "upsertConfig": {
    "mode": "NONE"
  },
  "metadata": {
    "customConfigs": {}
  }
}

Hi team! I was wondering if anyone has seen lag in json index creation in realtime segments for Pinot 1.4.0. Currently our json_match filter seems to return partial results for our realtime table and have no way to track this. If so, is there a way to track json index creation lag?

I am trying to use the helm 0.3.5 to implement TLS. It seems that it is not possible whyle it has no options to do that Is there a proper document how to harden Pinot?

I find that pinot cannot do

WHERE json_extract_scalar(some_column, "$.some_field", 'String', 'null') = some_other_string_col

But I am not sure why. Is that a pinot limitation? the error messages are not descriptive

Caught exception while doing operator: class org.apache.pinot.core.operator.query.SelectionOnlyOperator on segment some_segment: For input string: "null"

Quick question on multi-controller behavior in Pinot: From the /leader/tables API, it looks like tables are distributed across controllers, and schema changes (e.g., adding a column) go through the table’s leader. How frequently do controllers sync metadata? Scenario: I added a column to a table whose leader is c2, but when querying via c1 UI, I temporarily don’t see the column (especially in multi-stage engine setups where different tables have different leaders). Restarting c1 resolves it. Is this expected behavior, and are there configs or best practices to avoid this lag/glitch? @Mayank @Xiang Fu

Hello! We’re running Apache Pinot 1.3.0 consuming from Kafka topics produced by Flink 1.19 KafkaSink with EXACTLY_ONCE delivery guarantee. Pinot tables use Kafka isolation level

read_committed

. On low-volume topics (can be idle for ~30 minutes), during server restart Pinot’s startup readiness using the

FreshnessBasedConsumptionStatusChecker

gets stuck forever and never turns

GOOD

. Freshness-based check does the fallback to offset-based check, but it fails as well. The log repeats with a stable “+1” offset gap, log example in thread. Manually consuming last hour of user messages shows last user record offset is 18895 (no record at 18896 visible to a normal consumer) This blocks server startup readiness and cause restart loops. It looks like it possibly caused by kafka transactional markers/control records at the tail that aren’t visible for consumer, but still advance the offset. We fixed this issue using

realtimeFreshnessIdleTimeoutMs

, but as I understand it's not recommended to enable it in production environment. Is there any better options to let Pinot skip transactional markers instead of enabling

realtimeFreshnessIdleTimeoutMs

? Using

enableRealtimeOffsetBasedConsumptionStatusChecker

produced the same results

Hi team, I am running into an issue on Pinot 1.4.0 with a REALTIME upsert table when trying to use UpsertCompactMergeTask. Context I have a REALTIME upsert table with enableSnapshot=true, and I configured UpsertCompactMergeTask. However, the task does not generate any work for a subset of segments. These segments appear in the UI with: • segment.realtime.status = "IN_PROGRESS" • Helix/server state = OFFLINE • no segment.download.url • no segment.realtime.endOffset • no segment.total.docs Example segment metadata:

{
  "segment.realtime.numReplicas": "1",
  "segment.creation.time": "1717253407093",
  "segment.flush.threshold.size": "10000",
  "segment.realtime.startOffset": "4317",
  "segment.realtime.status": "IN_PROGRESS"
}

In the UI, the segment replica is shown as OFFLINE. Example segment name:

news__4__1__20240601T1450Z

These segments are also not present in deep store. Problem When I trigger UpsertCompactMergeTask, Pinot skips these segments because their download URL is empty. Controller logs:

WARN [UpsertCompactMergeTaskGenerator] Skipping segment news__4__1__20240601T1450Z for task as download url is empty
...
INFO [ServerSegmentMetadataReader] Retrieved validDocIds metadata for 0 segments from 1 server requests.
INFO [UpsertCompactMergeTaskGenerator] Finished generating 0 tasks configs for table: news_REALTIME

So the task cannot proceed because these segments are not considered mergeable. My understanding These look like old zombie consuming segments: • still marked as IN_PROGRESS • server state is OFFLINE • no deep store artifact • no download URL • apparently never committed as completed segments Because of that: • UpsertCompactMergeTask skips them • they cannot be reloaded from deep store • they look like stale metadata / orphaned realtime segments Questions 1. Is this a known issue in Pinot 1.4.0? 2. What is the recommended way to repair or clean up these old IN_PROGRESS + OFFLINE realtime segments when they no longer exist in deep store? 3. Should they be removed from metadata manually, or is there a supported repair workflow? 4. Is there any controller/server operation that should convert or clean up these stale segments automatically? Thanks !

Hi, quick question about moving data across environments — looking for best practices. I have a Pinot setup with 36 REALTIME tables (all Kafka-ingested) spanning roughly 2k customers. Some customers are small (100–1k rows per table), some are significantly larger. I’ve built an isolated DEV environment with all my tooling (MCP, Trino, etc.) connected to it, and I deliberately don’t want any of those tools to touch prod directly. What I’m trying to solve: selectively copy a single customer’s data from prod to dev, across all tables. Constraints: - Kafka retention in prod is only 7 days, so replaying from the topic isn’t an option for historical data. - Several tables have large blob columns (noDictionary, multi-MB raw JSON stored in attributes/relationships) that are never used in queries — they’re only there for backfill purposes. I don’t need to copy those. - My main concern is not breaking prod with expensive, large-volume exports — even with pagination, I’m wary of putting too much read pressure on the Pinot cluster. Current approach I’m considering: - Query prod via Trino (which sits in front of Pinot), filtering by meta.customer, selecting only the lightweight indexed columns (skipping the heavy blob columns). - Paginate with LIMIT/OFFSET in manageable batches. - Reconstruct the original Kafka JSON message format by inverting the transformConfigs (since I only have the derived columns, not the raw source JSON). - Push the reconstructed messages into the dev Kafka topic using kcat or a Kafka producer, and let dev Pinot ingest them normally. Questions for the community: 1. Is querying through Trino a reasonable way to extract per-customer data without hammering Pinot directly? Any tips on throttling or batching? 2. Are there better patterns for cross-env data sync in Pinot that I’m not seeing? (Segment-level copy seems too coarse since segments aren’t partitioned by customer.) Thanks in advance!

Is there any way to update hdfs uri (segment.download.url) for all segments in a pinot table at a time? I would like to change hdfs hardcoded namenode uri to NN HA nameservice. cc: @Mayank @Xiang Fu @Jackie

Hi Pinot team, I’m seeing another UpsertCompactMergeTask failure on Pinot 1.4.0 The task fails during the upload step with this stacktrace:

org.apache.pinot.spi.utils.retry.RetriableOperationException:
org.apache.pinot.common.exception.HttpErrorStatusException:
Got error status code: 400 (Bad Request) with reason: ""
while sending request:
/segments?enableParallelPushProtection=true&tableName=success_board&tableType=REALTIME
	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:63)
	at org.apache.pinot.plugin.minion.tasks.SegmentConversionUtils.uploadSegment(SegmentConversionUtils.java:140)
	at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.pushSegment(BaseMultipleSegmentsConversionExecutor.java:538)
	at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:316)
	at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:84)

This looks similar to another UpsertCompactMergeTask failure I had before: • task generation succeeds • source segments seem valid • failure happens when uploading the merged segment back to the controller • the request includes enableParallelPushProtection=true Questions: 1. What are the common reasons for the controller to return 400 Bad Request on /segments?enableParallelPushProtection=true... for UpsertCompactMergeTask ? 2. Does this usually indicate: • parallel push conflict, • segment lineage changed while the task was running, • or another controller-side validation failure ? 3. Is there a recommended way to debug the exact controller-side reason when the response body is empty ? 4. Are there recommended limits for maxNumSegmentsPerTask / maxNumRecordsPerTask to reduce the chance of this happening? Thanks!

@Mayank @Xiang Fu do we have any tool or documentation for migrating Druid to Pinot?

In Pinot 1.4.0 , It does support multi topic ingestion from same kafka cluster right ...? When created table config , with

streamConfigMaps

Pinot table is only consuming from the first config in the list

"streamConfigMaps" : [
    { "stream.kafka.topic.name": "topic_1", ... },
    { "stream.kafka.topic.name": "topic_2", ... }
]

This result in consuming only from

topic_1

When looking at log , i can see that, the

topic_partition

that is trying to consume is going with

Segment Partition ID

(In this case the second topic segment is created with largest partition ID - 10000, https://github.com/apache/pinot/issues/17260) Log for correctly consuming topic

2026/04/15 02:19:58.163 INFO [KafkaConsumer] [multi_topic_test_1__0__0__20260414T2017Z] [Consumer clientId=multi_topic_test_1_REALTIME-mulit_topic_test_2-0, groupId=null] Seeking to offset 6 for partition mulit_topic_test_2-0

Seeking to offset 6 for partition mulit_topic_test_2-0

Correct Log for not consuming topic

2026/04/15 02:19:56.000 INFO [KafkaConsumer] [multi_topic_test_1__10000__0__20260414T2017Z] [Consumer clientId=multi_topic_test_1_REALTIME-mulit_topic_test_1-0, groupId=null] Seeking to offset 0 for partition mulit_topic_test_1-10000

Seeking to offset 0 for partition mulit_topic_test_1-10000

???????? Is this is a bug ..? Partition part of

topic_partition

is picking from

Segment Partition ID

...?

cc: @Mayank, @Xiang Fu Can you please help here .

Hi Pinot Team , We are migrating from Elasticsearch to Pinot . We use cursor pagination (POST /query/sql?getCursor=true) on V1 to fetch large result sets including the requestId column (primary key). We originally used V2 (/query) for cursor, but V2's multi-stage engine fails when requestId is in the SELECT clause — the query is cancelled during plan setup (EMPTY_MAILBOX_SEND, error 503, "CANCELLED: client cancelled") because requestId conflicts with V2's internal gRPC stage-tracking identifier. Alias, CAST, and CONCAT wrapping all failed. Per official docs, cursor is a V1 feature, so we switched to /query/sql?getCursor=true. The initial cursor query works correctly and returns the first page. However, subsequent pagination GETs to /responseStore/{requestId}/results fail intermittently with "Query results for {requestId} not found". Out of 12 concurrent Lambda invocations (1 hour split into 5-min windows), some succeed fully, some fail immediately, and some partially succeed (e.g., 600K of 1.4M records fetched before failure). @Mayank @Xiang Fu

@Jinesh Parakh @Mayank @Xiang Fu Hi Pinot team, We're running into a blocking issue with the V2 multi-stage engine and need guidance. Our Setup: • Pinot cluster with 18 brokers behind a Network Load Balancer (NLB) • Table: urdmCore with requestId as the primary key column • Query volume: ~1.4M+ records per query, using cursor-based pagination • Client: AWS Lambda (Java) The Problem: When we include requestId in the SELECT clause on the V2 multi-stage engine (/query), the query gets cancelled immediately with EMPTY_MAILBOX_SEND / error 503 CANCELLED errors. Omitting requestId from SELECT works perfectly — cursor, pagination, everything functions as expected. We believe this is a namespace collision between V2's internal gRPC stage-tracking (which uses requestId for inter-stage communication) and our column named requestId. requestId is our primary key — we cannot omit it from SELECT. We need it for record-level validation between data sources. We are thinking of creating new column with changed name since pinot columns are immutable , was considering using transform function to get old data of requestId column in new column. Just needed an opinion / heads-up on this approach.