https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • b
    Hey @Xiang Fu I am facing a weird issue when filtering on a multivalued field. where clause if not working correctly.
    x
    • 2
    • 39
  • m
    Hi @Kartik Khare Unable to publish report using start tree connector in tableau. Even if report is published. Passwords does not embed if we have added basic authentication and it keeps asking for passwords and does not accept it even provided. Always errors out
    • 1
    • 1
  • p
    Hi team, I have a query. we observe s3 Connection Timeout issues on customer environment based on what we see in the log it seems default no of retries is 3 and timeout is evey 2 sec. we are testing the configurations to increase this defaults. My concern is do we document somewhere in pinot docs SLA that the network should adhere to for pinot to function correctly?
    x
    • 2
    • 3
  • z
    Hey team 👋 we are trying to optimize pinot's index on the time series data: we currently have a column request_ts with timestamp in seconds, so we define it as
    Copy code
    {
		"name": "request_ts",
		"dataType": "TIMESTAMP",
		"format": "1:SECONDS:EPOCH",
		"granularity": "1:SECONDS"
	  },
    to improve the index, we also doing 1. "timeColumnName": "request_ts", 2. "rangeIndexColumns": ["request_ts"], 3. "routing": {"segmentPrunerTypes": ["time"]} 4. add request_ts to timestampConfig with granularities like ["DAY","WEEK","MONTH"] Could you please share if the operation is helpful or some of them does not improve the performance too much 🙇 cc: @Neeraja Sridharan @Sai Tarun Tadakamalla
    m
    n
    s
    • 4
    • 51
  • m
    Hi @Mayank I am not able to use queryOptions=useMultiStageEngine=true in tableau server using apache startree pinot. Can some on guide me please. This works in tableau desktop but errors out in server and gives authentication error.
  • m
    Hi @Mayank Couple of question 1. Can we use dedup for full upsert or partial upsert secnario https://docs.pinot.apache.org/manage-data/data-import/upsert-and-dedup/upsert ? 2. Can we use a query like select * from table1 where column1 in (id1, id2, id3....id10k) if column1 has inverted index?
    m
    • 2
    • 14
  • s
    Hello Pinot team! Just had a question on inverted index. Reading the doc on inverted index and referencing the following line from the doc:
    Copy code
    As explained in the forward index section, a column that is both sorted and equipped with a dictionary is encoded in a specialized manner that serves the purpose of implementing both forward and inverted indexes. Consequently, when these conditions are met, an inverted index is effectively created without additional configuration, even if the configuration suggests otherwise.
    We have a column that is dictionary enabled and configured as sorted in realtime table. We have also set 
    autoGeneratedInvertedIndex
    and 
    createInvertedIndexDuringSegmentGeneration
    as 
    true
    in the table config. However we are not seeing inverted index being used in the explain plan query. Would inverted index also be configured in the table config to see it in effect?
    m
    • 2
    • 19
  • r
    Hey Team, We’re running Apache Pinot 1.4.0 with 24 servers. On 3 servers, we consistently see errors like: • 
    RuntimeException: Caught exception while running BloomFilterSegmentPruner
    (caused by 
    TimeoutException
    in 
    QueryMultiThreadingUtils.runTasksWithDeadline
    ) • 
    RuntimeException: Caught exception while running CombinePlanNode
    (also 
    TimeoutException
    ) These errors appear all the time, not just under peak load. We recently increased server RAM, but otherwise no config changes. Unfortunately, I don’t have older logs to check if this was happening before. Has anyone seen similar behavior, and what could cause it to affect only a subset of servers?
  • m
    Hi @Mayank Is it possible pinot single table can consume from two different region, we can pass 2 urls or 2 brokers?
    m
    • 2
    • 11
  • v
    Hi Team, We are facing an issue with MergeRollupTask in our Pinot cluster. After the task runs, the original segments are not getting deleted, and we end up with both the original and the merged segments in the table. Retention properties: left as default. Any guidance on what we might be missing would be super helpful. Adding task, table and segments related details in the thread.
    • 1
    • 2
  • d
    #C011C9JHN7R I am having issues with star-tree tableau connector where the additional properties does not work consistently with respect to query options when trying to set multistage option to true
  • d
    In tableau server it is failing to sign in where it at least connects in desktop
  • r
    Hey all, Very new to Pinot, and been trying to get a realtime table working from Confluent Kafka, Pinot doesnt like something in my config, and it times out before the table is created, the id/api keys do have access to kafka, so I am expecting something wrong with my config below, any reference/pointers is much appreciated.
    m
    • 2
    • 1
  • r
    {
    "tableName": "kafka_test_1",
    "tableType": "REALTIME",
    "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant",
    "tagOverrideConfig": {}
    },
    "segmentsConfig": {
    "timeColumnName": "time",
    "replication": "1",
    "replicasPerPartition": "1",
    "retentionTimeUnit": null,
    "retentionTimeValue": null,
    "completionConfig": null,
    "crypterClassName": null,
    "peerSegmentDownloadScheme": null,
    "schemaName": "kafka_test"
    },
    "tableIndexConfig": {
    "loadMode": "MMAP",
    "invertedIndexColumns": [],
    "createInvertedIndexDuringSegmentGeneration": false,
    "rangeIndexColumns": [],
    "sortedColumn": [],
    "bloomFilterColumns": [],
    "bloomFilterConfigs": null,
    "noDictionaryColumns": [],
    "onHeapDictionaryColumns": [],
    "varLengthDictionaryColumns": [],
    "enableDefaultStarTree": false,
    "starTreeIndexConfigs": null,
    "enableDynamicStarTreeCreation": false,
    "segmentPartitionConfig": null,
    "columnMinMaxValueGeneratorMode": null,
    "aggregateMetrics": false,
    "nullHandlingEnabled": false,
    "streamConfigs": {
    "streamType": "kafka",
    "stream.kafka.topic.name": "PINOT.TEST",
    "stream.kafka.consumer.type": "lowlevel",
    "stream.kafka.broker.list": "{}",
    "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory",
    "stream.kafka.security.protocol": "SASL_SSL",
    "stream.kafka.sasl.mechanism": "OAUTHBEARER",
    "stream.kafka.sasl.login.callback.handler.class": "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler",
    "stream.kafka.sasl.oauthbearer.token.endpoint.url": "{url}",
    "stream.kafka.sasl.jaas.config": "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required clientId='{}' clientSecret='{}' scope='' extension_logicalCluster='{}' extension_identityPoolId='{}';",
    "stream.kafka.ssl.endpoint.identification.algorithm": "https",
    "stream.kafka.consumer.prop.group.id": "{}",
    "stream.kafka.consumer.prop.auto.offset.reset": "earliest",
    "<http://stream.kafka.consumer.prop.request.timeout.ms|stream.kafka.consumer.prop.request.timeout.ms>": "60000",
    "<http://stream.kafka.consumer.prop.metadata.max.age.ms|stream.kafka.consumer.prop.metadata.max.age.ms>": "60000",
    "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaAvroMessageDecoder",
    "stream.kafka.decoder.prop.schema.registry.url": "https://{}.westeurope.azure.confluent.cloud",
    "stream.kafka.decoder.prop.schema.registry.basic.auth.credentials.source": "USER_INFO",
    "<http://stream.kafka.decoder.prop.schema.registry.basic.auth.user.info|stream.kafka.decoder.prop.schema.registry.basic.auth.user.info>": "{key}:{secret}"
    }
    },
    "metadata": {},
    "ingestionConfig": {
    "filterConfig": null,
    "transformConfigs": null
    },
    "quota": {
    "storage": null,
    "maxQueriesPerSecond": null
    },
    "task": null,
    "routing": {
    "segmentPrunerTypes": null,
    "instanceSelectorType": null
    },
    "query": {
    "timeoutMs": null
    },
    "fieldConfigList": null,
    "upsertConfig": null,
    "tierConfigs": null
    }
  • r
    Just to give an update, this was resolved with the below config
  • r
    "streamType": "kafka",
    "stream.kafka.topic.name": "asdas",
    "stream.kafka.consumer.type": "lowlevel",
    "stream.kafka.broker.list": "asasds.westeurope.azure.confluent.cloud:9092",
    "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "PLAIN",
    "sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\" password=\"\";",
    "ssl.endpoint.identification.algorithm": "https",
    "auto.offset.reset": "earliest",
    "<http://stream.kafka.consumer.prop.request.timeout.ms|stream.kafka.consumer.prop.request.timeout.ms>": "60000",
    "<http://stream.kafka.consumer.prop.metadata.max.age.ms|stream.kafka.consumer.prop.metadata.max.age.ms>": "60000",
    "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
    "stream.kafka.decoder.prop.schema.registry.rest.url": "<https://dasdsa.westeurope.azure.confluent.cloud>",
    "stream.kafka.decoder.prop.schema.registry.basic.auth.credentials.source": "USER_INFO",
    "<http://stream.kafka.decoder.prop.schema.registry.basic.auth.user.info|stream.kafka.decoder.prop.schema.registry.basic.auth.user.info>": ":",
    "stream.kafka.decoder.prop.schema.registry.schema.name": "KsqlDataSourceSchema",
    "stream.kafka.decoder.prop.format": "AVRO"
  • m
    Thanks @Rajkumar for confirming
    👍 1
  • n
    "task": { "taskTypeConfigsMap": { "MergeRollupTask": { "1hour.mergeType": "rollup", "1hour.bucketTimePeriod": "1h", "1hour.bufferTimePeriod": "3h", "1hour.maxNumRecordsPerSegment": "1000000", "1hour.maxNumRecordsPerTask": "5000000", "1hour.maxNumParallelBuckets": "5", "1day.mergeType": "rollup", "1day.bucketTimePeriod": "1d", "1day.bufferTimePeriod": "1d", "1day.roundBucketTimePeriod": "1d", "1day.maxNumRecordsPerSegment": "1000000", "1day.maxNumRecordsPerTask": "5000000", "metric2.aggregationType": "avg", "metric.aggregationType": "avg", "metric3.aggregationType": "avg", "metric4.aggregationType": "avg", "scores_sc.aggregationType": "avg" } } } based on the docs, I understood avg is not supported during aggregation is my understand is correct or we can do average as well. is my above job is correct or not.
  • r
    Hi All, what's a neat way of extracting strings from a concatenated sting. Example 'Raj|25|Male' should go to three fields Name, Age, Gender.
    m
    • 2
    • 4
  • r
    I tried to do the below, but doesnt work 
    split(PEXP_DEAL_KEY, '|', 1)
  • m
    Hi @Mayank Can I used balanced segment assignment strategy with strict replica group for routing?
    m
    • 2
    • 12
  • p
    Hello Good morning. Our servers intermittently stop responding and return 502 Gateway Timeout errors. The issue resolves temporarily after a restart, but it recurs once the system is under load. Given that our dataset is extremely large (around 300 billion records across all tables), my suspicion is that one or more expensive queries are causing the system to hang. Observations & Context: • I want to identify which queries are running for a long time or consuming excessive resources. • Running 
    EXPLAIN "query"
    manually in the UI isn’t feasible since we have hundreds of queries. • After a restart, the servers function normally until the issue reappears. • My main doubt is that certain queries are significantly expensive and triggering these outages. Request: If anyone has experience or additional context on how to efficiently identify long-running or expensive queries (without manually running 
    EXPLAIN
    for each one), please share your insights.
    m
    m
    • 3
    • 9
  • t
    Hi, I've got a question regarding multiple tenants on Kubernetes deployments of Pinot. Given, • the clients of our pinot cluster need to be able to reach the appropriate broker for the table that they are querying from outside the kubernetes cluster • brokers can be rebalanced, and we might have multiple brokers for any one tenant • as far as I can tell, the controller will only give you the cluster-local URL of the appropriate broker when asked about where to find a table How can I set up a single URL for the whole cluster, or a single URL for each tenant (either case is fine) for clients outside our kubernetes cluster to access the correct broker? I do not see anything detailing a way to allow the controller to advertise the cluster-external broker URL instead of the cluster-local one. If it were able to do this, it would likely be sufficient for our use case.
    m
    • 2
    • 3
  • n
    Hi, To use rollup segments effectively is there any condition we need to follow like filtering on raw timestamp column without using
    Copy code
    (DATE_TRUNC('DAY', "eventsgroups".servertime)) >= CAST('2025-08-25' AS timestamp)
    date_trunc functions?
    m
    m
    • 3
    • 9
  • k
    Hi everyone, We’ve hit a strange bug. We have a Realtime table with upserts and a relatively large primary key, where one of the columns is a string of up to 2048 characters. When an 
    UpsertCompactionTask
    runs, that column gets truncated to 512 characters, which breaks the application since we’re losing data. Am I missing some configuration here, or is this a bug/intentional behavior?
    m
    r
    c
    • 4
    • 66
  • s
    table config
    table config
  • s
    table config
    Untitled
  • s
    Hi Team, I am facing a 
    java.lang.OutOfMemoryError: Java heap space
     issue while running a realtime upsert table in Pinot. Setup details: • Table type: 
    REALTIME
    (upsert = FULL mode) • Primary keys: ~2B (hashFunction = MURMUR3) • Cluster: 4 servers • Memory: 128 GB per server Problem: Whenever I start the servers, memory usage grows continuously until ~100–110 GB, and then the server eventually hits: 
    java.lang.OutOfMemoryError: Java heap space
    attaching heap memory use trend, server config, table config, server logs and gc logs
    gc.logserver logstable config.rtfserver config.rtf
    m
    • 2
    • 1
  • t
    We're using Pinot in Kubernetes, and we would like to start using multiple tenants in order to support physical data segregation. We need to not only physically separate the data, but also encrypt each tenant's data at-rest using a different encryption key for their bound volume. The feature that would enable all of this is the ability to set specific servers to be assigned to specific tenants. I have not seen that this is possible in any configuration. Does anyone know of a way to do it?
    x
    • 2
    • 7
  • c
    Hi. Pinot team. I'm receiving a "Message loss detected" error on a server in a Pino 1.2 cluster. Does this error mean that the segment failed to start consuming from startOffset and instead started consuming from batchFirstOffset? Does this mean data loss occurred between startOffset: 769410318 and batchFirstOffset: 769525930? However, when I query the data corresponding to the offset in Pino, the data corresponding to the offset was not lost. What is this error log? 2025/09/12 150935.996 ERROR [debug_tiara_core__1__1__20250912T0608Z] org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager_debug_tiara_core__1__1__20250912T0608ZreportDataLoss958 Message loss detected in stream partition: 1 for table: debug_tiara_core_REALTIME startOffset: 769410318 batchFirstOffset: 769525930 select $segmentName, * from debug_tiara_core where __metadata$partition='1' and __metadata$offset >= '769410318' and __metadata$offset <= '769525930 ' -- Data exists, no loss. -- debug_tiara_core__1__1__20250912T0608Z { "segment.realtime.startOffset": "769410318", "segment.realtime.endOffset": "769571100", "segment.creation.time": "1757657338493", "segment.start.time": "1757635200000", "segment.end.time": "1757635200000", "segment.total.docs": "150,000", }
    h
    • 2
    • 2