Hi team, I am creating an autoscaler for minion-based batch ingestions. To scale up and down, I need the number of tasks that are waiting and the number of tasks that are running. I checked the Pinot metrics and found these two:

pinot_controller_numMinionSubtasksWaiting_Value

and

pinot_controller_numMinionSubtasksRunning_Value

. However, for each task type, they always show a value of 0 even when tasks are running. Am I using the wrong metrics? Which metrics should I use to build a custom autoscaler for minions?

Hi team 😉 Quick question about some messages I see in my monitoring : “Recreating stream consumer for topic partition *, reason: Total idle time: 183647 ms exceeded idle timeout: 180000 ms” What is the behaviour behind that ? Reset the consumer to last commited offset and reingest things ? Or just re-pop the consumer to his last consumed offset ?

Hello Team, We are encountering an issue where our Pinot servers are timing out when attempting to establish a session with Zookeeper. This is causing the Pinot Servers to crash (or go down). Although the server attempts to iteratively establish a new connection, the process continues to time out until we manually restart the server instance. A similar scenario can be found in the following GitHub issue: https://github.com/apache/pinot/issues/4686. 1. The initial issue between the Pinot Server and Zookeeper was session expiration. 2. Regardless of the underlying issue (e.g., Zookeeper latency, GC pauses blocking the main thread), Pinot should be capable of automatically re-establishing the connection once the problem is resolved. Instead, we are forced to manually restart the server to restore a healthy Zookeeper session. As a result , the server is being removed from the LIVE_INSTANCE metadata and registered as DEAD.

Hello, I have a question about how the controller handles rebalancing segments. I am more interested in the following aspect: is there any downtime while moving a segment from a server to another ? I see that in the manual rebalance job you can specify downtime=false only if you have replication. Is the mechanism behind controller rebalancing the same ?

Hi Team, I was checking if pinot really logs auditing as in

who did what, how, from which source and at what time

? Seems like the code base logs only the response and the type and not the request. It will be great if request is also being logged, so audit info is fully available. In code base : ControllerResponseFilter.java > LOGGER.info("Handled request from {} {} {}, content-type {} status code {} {}", srcIpAddr, method, uri, contentType, > respStatus, reasonPhrase); If this has requestContext is also added , I believe it should add request details with payload that is initially sent by the user, or if its disabled on purpose, do you mind giving that control to log4j that enduser can choose to enable it or not. I'm no developer 🥺, I'm trying the make sense of the code and see if it can be added . Where I'm coming from is: I just added a user via controller to have read,write permissions of a particular user on all tables. All I get is below.

2025/11/03 20:30:59.922 INFO [ControllerResponseFilter] [grizzly-http-server-15] Handled request from 192.168.13.1 PUT <http://test-phaseroundtoaudit11.ori.com:9000/users/dedactid_rw?component=BROKER&passwordChanged=false|http://test-phaseroundtoaudit11.ori.com:9000/users/dedactid_rw?component=BROKER&passwordChanged=false>, content-type text/plain;charset=UTF-8 status code 200 OK
2025/11/03 20:30:59.957 INFO [ControllerResponseFilter] [grizzly-http-server-14] Handled request from 192.168.13.1 GET <http://test-phaseroundtoaudit11.ori.com:9000/tables|http://test-phaseroundtoaudit11.ori.com:9000/tables>, content-type null status code 200 OK
2025/11/03 20:30:59.980 INFO [ControllerResponseFilter] [grizzly-http-server-12] Handled request from 192.168.13.1 GET <http://test-phaseroundtoaudit11.ori.com:9000/users|http://test-phaseroundtoaudit11.ori.com:9000/users>, content-type null status code 200 OK

But its missing read,write has been given my admin user to ALL/particular tables. There is further granularity missing which is crucial I believe. Let me know your views. Thanks!!

QQ: Is there a certain task we can run to force a server to re-upload it's segment to the deep store (in our case S3)? We have a situation where a realtime server failed to upload to S3, and then the segment was offloaded to offline servers, The offline servers were able to fetch the segment from their online peers to successfully load the segment, but the segment is still in a weird state where it is missing from the deep-store/S3. Should some periodic task be running to check on this, or can we run some manual controller task to "heal" the situation?

Hi Team, Context: We want to use Apache Pinot for real-time analytics query use cases in our microservices. Since realtime Pinot tables ingest directly from Kafka, ingestion delays/lag can occur. Our requirement is: whenever a document (row) in Pinot is updated, we want to push an event to Kafka with the primary key that changed. This would allow downstream microservices to consume that event, know that a specific record has been updated in Pinot, then trigger real-time analytics queries and perform required downstream actions. Question: Is there any existing feature or recommended workaround in Pinot to detect when a row is updated in a realtime table and trigger an event (e.g., send a Kafka message) so downstream services can be notified?

Hi Team, Recently I was trying to enable OOM (https://docs.pinot.apache.org/operators/operating-pinot/oom-protection-using-automatic-query-killing). I have added below configurations in both broker and server config files.

pinot.broker.instance.enableThreadCpuTimeMeasurement=true
pinot.broker.instance.enableThreadAllocatedBytesMeasurement=true
pinot.server.instance.enableThreadAllocatedBytesMeasurement=true
pinot.server.instance.enableThreadCpuTimeMeasurement=true
pinot.query.scheduler.accounting.enable.thread.memory.sampling=true
pinot.query.scheduler.accounting.enable.thread.cpu.sampling=true


pinot.query.scheduler.accounting.oom.enable.killing.query=true
pinot.query.scheduler.accounting.query.killed.metric.enabled=true

pinot.query.scheduler.accounting.oom.critical.heap.usage.ratio=0.3
pinot.query.scheduler.accounting.oom.panic.heap.usage.ratio=0.3
<http://pinot.query.scheduler.accounting.sleep.ms|pinot.query.scheduler.accounting.sleep.ms>=30
pinot.query.scheduler.accounting.oom.alarming.usage.ratio=0.3
pinot.query.scheduler.accounting.sleep.time.denominator=3
pinot.query.scheduler.accounting.min.memory.footprint.to.kill.ratio=0.01

pinot.query.scheduler.accounting.factory.name=org.apache.pinot.core.accounting.PerQueryCPUMemAccountantFactory
pinot.query.scheduler.accounting.cpu.time.based.killing.enabled=true
pinot.query.scheduler.accounting.publishing.jvm.heap.usage=true
<http://pinot.query.scheduler.accounting.cpu.time.based.killing.threshold.ms|pinot.query.scheduler.accounting.cpu.time.based.killing.threshold.ms>=1000

I have run some heavy queries to test the OOM killing feature, but I don't see any killed queries in the broker/server metrics.

SELECT accountId,countryCode,direction,day,hour,msgType,currency,topic,finalStatus,year,month,
  SUM(CASE WHEN finalStatus = 'Failed' THEN 1 ELSE 0 END) AS failed_count,
  SUM(CASE WHEN finalStatus = 'Delivered' THEN 1 ELSE 0 END) AS success_count,
  COUNT(*) AS total_records,
  COUNT(DISTINCT udrId) AS unique_udrs,
  SUM(price) AS total_revenue,
  AVG(price) AS avg_price,
  MAX(price) AS max_price,
  MIN(price) AS min_price,
  SUM(CASE WHEN errorCode > 0 THEN 1 ELSE 0 END) AS error_count,
  SUM(price * (CASE WHEN direction = 'Unknown' THEN 1 ELSE -1 END)) AS net_revenue
FROM
  dummy_table
GROUP BY
  accountId,countryCode,direction,msgType,currency,topic,finalStatus,year,month,day,hour
ORDER BY
  total_revenue DESC,
  avg_price DESC
LIMIT 1000000

Whenever I run this query, the server goes down, but no queries are terminated automatically. Can you please help me to understand if I am missing any configurations or steps to enable this feature? I did test on

apachepinot/pinot:1.5.0-SNAPSHOT-9d32f376d8-20251016, size of Heap -Xms2G -Xmx2G for server and broker.

Hi Team, Im getting this error continously even my servers are running properly and status of the tables are in good state. help me to resolve the issue kubectl get pod -n dp-1-346 NAME READY STATUS RESTARTS AGE pinot-broker-0 1/1 Running 0 13h pinot-controller-0 1/1 Running 0 6m26s pinot-minion-stateless-84fc6899f9-2shqp 1/1 Running 0 13h pinot-server-0 1/1 Running 0 6m32s pinot-server-1 1/1 Running 0 6m39s presto-coordinator-0 1/1 Running 0 25h presto-worker-0 1/1 Running 0 25h zookeeper-0 1/1 Running 0 13h

Hi, In my Pinot cluster, I initially had 4 servers (A, B, C, D) with segments distributed across them. I wanted to consolidate all segments onto a single server, so I removed the tags from servers B, C, and D, and then ran a rebalance operation to allocate all segments to the remaining server (A). After rebalancing, all segments were assigned to the single server. However, the segments that were originally on the other servers appeared in ERROR state in the external view, even though their ideal state in ZooKeeper showed them as ONLINE. For example: • Ideal State:

"load_chat_messages_core_1756318894786_1758914214102_1758919671601": {
    "Server_172.18.0.6_8098": "ONLINE"
}

• External View:

"load_chat_messages_core_1756318894786_1758914214102_1758919671601": {
    "Server_172.18.0.6_8098": "ERROR"
}

To resolve this, I performed a reload and reset operation on the affected segments. After the reset, the segment state transitioned from ERROR to OFFLINE, allowing it to be properly reloaded. Setup details: • Running Pinot in Docker • Using local storage for segment files • Segment data is volume-mounted

Hi questions on strange load accross VMs. Does anyone have faced this kind of issues before ? We’ve got 4 servers 2 of them shows lot of CPU/ RAM usages where the two others looks like normal/chill. Data is properly balanced (32 tables / 2 partitions (non spooled) with replication of 2 ) so each server is used correctly when queried. Can be seen on the network traffic close to equal on each server. Tried to rebalance tables but all already balanced. I’ve rework the JMX to grab only relevant data and I do not see anything. Same queries rate and segment processed / server. Any clues ?

I am trying to debug a strange issue regarding segment generation from a realtime table. Its config is set up like this:

"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.segment.size": "500M",
"realtime.segment.flush.threshold.time": "4h"

However, when inspecting the metadata of any of the realtime segments we can see for example:

"segment.realtime.endOffset": "67399447",
"segment.start.time": "1762424217000",
"segment.time.unit": "MILLISECONDS",
"segment.flush.threshold.size": "100000",
"segment.realtime.startOffset": "66512835",
"segment.size.in.bytes": "14018213",  <====== 14MB instead of 500M    
"segment.end.time": "1762426143000",  <====== subtracting segment.start.time from this we get roughly 35 min 
"segment.total.docs": "100000",
"segment.realtime.numReplicas": "1",
"segment.creation.time": "1762511599197",
"segment.index.version": "v3",
"segment.crc": "3704033136",
"segment.realtime.status": "DONE",

Hello Everyone, I am facing some issues in production with the Pinot setup. Could anyone help me?🙂

Where should I add the following configuration for enabling or disabling multi-tenant instance isolation in Apache Pinot?

cluster.tenant.isolation.enable=false

Should it go inside

controller.conf

or

pinot-controller.conf

? I’m running Pinot in Docker, and for the controller service, my command looks like this:

command: "StartController -zkAddress pinot-zookeeper:2181 -configFileName /opt/pinot/conf/pinot-controller.conf"

I added the configuration in

pinot-controller.conf

, but the controller container is failing to start.

Can u pls help me resolve a doubt? in azure, does pinot support the (wasbs) format? when i try to ingest the files using wasbs, pinot is not picking it up.. In our dev setup, I'm using abfss.. it's working fine, pinot is picking up the files.

Hi team, pls help and tell the solution guys. this is the blocker to my work and let me know production level suggestions. I added the upsert configuration to the existing setup, but it didn’t work initially. After deleting the old configuration and re-adding the same setup with upserts, it started working. However, after some time, the segments began failing. { "REALTIME": { "tableName": "views_REALTIME", "tableType": "REALTIME", "segmentsConfig": { "schemaName": "views", "replication": "1", "retentionTimeUnit": "DAYS", "retentionTimeValue": "90", "replicasPerPartition": "1", "timeColumnName": "view_end", "minimizeDataMovement": false }, "tenants": { "broker": "DefaultTenant", "server": "DefaultTenant", "tagOverrideConfig": {} }, "tableIndexConfig": { "aggregateMetrics": false, "starTreeIndexConfigs": [], "enableDefaultStarTree": false, "nullHandlingEnabled": false, "noDictionaryColumns": [ "events" ], "invertedIndexColumns": [ "workspace_id", "country_code", "fp_playback_id", "browser_name", "is_final" ], "bloomFilterColumns": [], "onHeapDictionaryColumns": [], "rangeIndexColumns": [ "view_end", "view_start", "created_at" ], "sortedColumn": [ "view_end", "quality_of_experience_score", "playback_score", "render_quality_score", "stability_score", "startup_score" ], "varLengthDictionaryColumns": [], "rangeIndexVersion": 2, "optimizeDictionaryForMetrics": false, "optimizeDictionary": false, "autoGeneratedInvertedIndex": false, "createInvertedIndexDuringSegmentGeneration": false, "loadMode": "MMAP", "enableDynamicStarTreeCreation": true, "columnMajorSegmentBuilderEnabled": true, "noDictionarySizeRatioThreshold": 0.85 }, "metadata": {}, "quota": {}, "task": { "taskTypeConfigsMap": { "UpsertCompactionTask": { "schedule": "0 0 * ? * *", "bufferTimePeriod": "1h", "invalidRecordsThresholdPercent": "30", "invalidRecordsThresholdCount": "100000", "tableMaxNumTasks": "10", "validDocIdsType": "SNAPSHOT" } } }, "routing": { "segmentPrunerTypes": [ "partition" ], "instanceSelectorType": "strictReplicaGroup" }, "query": {}, "upsertConfig": { "enableSnapshot": true, "deletedKeysTTL": 0, "mode": "FULL", "comparisonColumns": [ "view_end" ], "metadataTTL": 0, "dropOutOfOrderRecord": false, "hashFunction": "NONE", "defaultPartialUpsertStrategy": "OVERWRITE", "enablePreload": true, "consistencyMode": "NONE", "upsertViewRefreshIntervalMs": 3000, "allowPartialUpsertConsumptionDuringCommit": false }, "ingestionConfig": { "transformConfigs": [ { "columnName": "created_at", "transformFunction": "Now()" } ], "streamIngestionConfig": { "streamConfigMaps": [ { "streamType": "kafka", "stream.kafka.topic.name": "fp-data-processed-views-v1", "stream.kafka.consumer.prop.group.id": "pinot-views", "stream.kafka.broker.list": "kafka-cluster-broker-0.kafka-cluster-kafka-brokers.prod-kafka.svc.cluster.local9092,kafka cluster broker 1.kafka cluster kafka brokers.prod kafka.svc.cluster.local9092,kafka-cluster-broker-2.kafka-cluster-kafka-brokers.prod-kafka.svc.cluster.local:9092", "stream.kafka.consumer.type": "lowlevel", "stream.kafka.consumer.prop.auto.offset.reset": "largest", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "sasl.mechanism": "SCRAM-SHA-512", "security.protocol": "SASL_PLAINTEXT", "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"hfjcnfrjnrc\" password=\"njffhirjfkriviruuir\";", "realtime.segment.flush.threshold.rows": "0", "realtime.segment.flush.threshold.segment.size": "200M", "realtime.segment.flush.threshold.time": "24h" } ], "columnMajorSegmentBuilderEnabled": true, "trackFilteredMessageOffsets": false }, "continueOnError": false, "rowTimeValueCheck": false, "segmentTimeValueCheck": true }, "isDimTable": false } }

Hi Team, I am checking out release 1.4 code base and building it. https://github.com/apache/pinot/releases/tag/release-1.4.0 I have a doubt regarding below PR: https://github.com/apache/pinot/pull/16624 Why above PR is not included in 1.4 code base?

Hi Team, we are seeing a major performance issue on a spring boot application which queries pinot through pinot jdbc connectivity. The api RPS is expected to ~150 with less than 1 sec response time. Do we need to implement Hikari CP to mitigate the performance issues ? From the pinot brokers we see that response times are less than 1 sec. Can you please help ? CC: @Mayank

Hi team, what is the steps for adding index in running hybrid table??

Hi Team Need help in debugging this issue. The cpu for all servers in a tenant keeps spiking. There is no queries running, just kafka ingestion. Tried pausing ingestion for the main table then the cpu went down, But after resuming it spiked back again. Things I have tried. • Restarting servers • Reloading segments • Pausing ingestion None of my attempts worked for stabilising the cpu load. Can someone please help here?

hi, i have enabled pinot auth in helm values, for controller and broker, but the server is showing error 401 unauthorized. can anybody guide the solution.

pinotAuth:
  enabled: true
  controllerFactoryClass: org.apache.pinot.controller.api.access.BasicAuthAccessControlFactory
  brokerFactoryClass: org.apache.pinot.broker.broker.BasicAuthAccessControlFactory
  configs:
    - access.control.principals=admin
    - access.control.principals.admin.password=<password>

error

org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 401 (Unauthorized) with reason: "HTTP 401 Unauthorized" while sending request: /segmentConsumed?reason=forceCommitMessageReceived&streamPartitionMsgOffset=214683&instance=Server_prod-pinot-server-0.prod-pinot-server-headless.prod-pinot.svc.cluster.local_8098&name=views__0__0__20251113T0646Z&rowCount=21&memoryUsedBytes=84567768 to controller: prod-pinot-controller-2.prod-pinot-controller-headless.prod-pinot.svc.cluster.local, version: Unknown

Hi Team, We initially had 3 controller nodes and recently added 3 more. Following this, we stopped the controller service on the 3 old nodes. However, after about a week, the ingestion process started failing because the Pinot server was attempting to connect to one of the stopped controller nodes. Interestingly, we did not encounter this issue in another cluster where the same approach was implemented. While I understand that we need to remove the stopped instances from Zookeeper using the Swagger API, ideally, the Pinot server should fail over and connect to an active/leader controller automatically. Am I missing something ?

2025-11-12 16:11:54.373 ERROR [sample__121__287__20251105T1324Z] LLRealtimeSegmentDataManager_sample__66__339__20251105T1654Z - Could not send request <http://pinot-controller-01.local:9000/segmentUpload?segmentSizeBytes=1073117101&buildTimeMillis=91591&streamPartitionMsgOffset=10967889843&instance=pinot-server-01.local_8098&offset=-1&name=sample__121__287__20251105T1324Z&rowCount=73431535&memoryUsedBytes=2015467107>
org.apache.pinot.shaded.org.apache.http.conn.HttpHostConnectException: Connect to pinot-controller-01.local:9000 [pinot-controller-01.local/10.10.10.56] failed: Connection refused (Connection refused)
  at org.apache.pinot.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]

Hi team, I have a question regarding the realtime to offline minion tasks. Our data must keep a certain column sorted throughout segments. In our current flow we do that by waiting for a consuming segment to reach a large enough size (hundreds of MB) and then seal it and let the minion task run nightly and move data to offline. Re-reading the documentation it seems that the

RealtimeToOfflineSegmentsTask

(as opposed to

MergeRollupTask

where this is not mentioned) keeps the data sorted in the segment it builds. If this were the case I would change the configs in order to seal smaller segments more frequently and let the minion bunch all these small segments up and create the larger, sorted segment to the offline table. My question is if there is any way of validating that the data inside of the offline segments does indeed keep the data sorted by the column

Hello, do we know when we might have consistent data load for spark execution. https://github.com/apache/pinot/issues/12941#event-12498082012 Currently we a table that is pretty huge and sometimes it takes time for all segments to be refreshed and if a user queries the table during this time we might show incorrect results if only some segments are refreshed. cc: @Jackie

❓Looks like partition-based segment pruning in Pinot can be configured for multiple columns for

offline

tables. Appreciate any help in confirming if the same applies to

real-time

tables as well 🙇‍♀️ Here is the associated reference for Kafka stream, but it doesn't explicitly mention if partition-based segment pruning can be set up for multiple columns in the corresponding Pinot table config: https://docs.pinot.apache.org/basics/getting-started/frequent-questions/ingestion-faq#how-do-i-enable-partitioning-in-pinot-when-using-kafka-stream I guess the pre-requisite is: Input Kafka stream needs to be configured with a custom partitioner to match the partition column(s), partition function and number of partitions set up in the Pinot table config.

Hi team, Currently in pinot, when performing sum or avg on a BIG_DECIMAL column, the result is returned as double, even though the source data is of BIG_DECIMAL type. This leads to the output being displayed in exponential format after a certain precision. While there is a sumprecision function available for sum to maintain big decimal output, there is no equivalent function for avg to return the result as big decimal instead of double. Why was the sumprecision function introduced rather than supporting big_decimal as a result type for aggregations on big_decimal columns? In Trino/Presto the output of aggregate functions is as expected without exponential format. Example: input: 1234567890.1234567 output: 1.2345678901234567E9 expected output: 1234567890.1234567

Hello, could someone take a look at this github issue : https://github.com/apache/pinot/issues/17233 I think there have multiple follow-ups for CRC mismatch issue which has happened due to Lucene indexes, using ingestion transformations like now(), non-deterministic metadata file ordering etc. And recently, we pushed a change to allow config based skip CRC check altogether during segment load here. This has surfaced more edge-cases mentioned here, where skipping a CRC flag would make a server not download a new segment which is needed in case of backfill jobs, or Upsert compaction. What do we think about computing CRC only for the data part of the segment and not the Indexes ? I understand that this still runs into issue of indeterminism during ingestion transformations where each replica may be running the transform function locally, but apart from that - do we see any issue in switching the CRC computation to be data only ? @Jackie @coco

Hi team, I have a table in production, in which i have deleted a few consuming segments as the servers were not able to locate it. After triggering resumeConsumption, consuming segments are not getting created. please help

guys need help

Hi There, I have the following query

select a.id, b.id, sum(a.amt) amt, sum(b.amt) as amt_b

from tbla a

join (select id, ccy, sum(amt) from tblb group by id, ccy) b on b.id = a.id and b.ccy - a.ccy

group by a.id, b.id

I cant join these tables directly, and need to aggregate tblb as shown in the query above. It appears that Pinot is not returning all the id's from tblb (subquery), only a subset. May be the servers just respond partial results? Has anyone faced this, and appreciate any alternatives to get the complete result from the subquery, and execute the join.