Apache Pinot #troubleshooting

Ralph Debusmann

12/22/2022, 1:42 PM

Hi Pinot team! We've just found a very weird behavior of the "strpos" function...

Ralph Debusmann

12/22/2022, 1:43 PM

Assume that some of our "label" columns reads "Copper"... now we want to match this with a string containing "Copper" as a substring - using the following SQL statement: select * from sentiments where strpos('Copper Price LME', label) > 0

Ralph Debusmann

12/22/2022, 1:45 PM

The strange thing is now that we get no results, even though there are a bunch of rows where the "label" column is set to "Copper".

Ralph Debusmann

12/22/2022, 1:46 PM

It does work however if we add any set of characters at the beginning of the string, like a whitespace character: select * from sentiments where strpos(' Copper Price LME', label) > 0

Ralph Debusmann

12/22/2022, 1:46 PM

Is this a bug or somehow intentional?

Ralph Debusmann

12/22/2022, 1:56 PM

(Using Pinot 0.11)

Larry Meadors

12/22/2022, 2:00 PM

that is position 0

Larry Meadors

12/22/2022, 2:01 PM

you want

>=0

because it'll return

-1

if it is not there

Larry Meadors

12/22/2022, 2:01 PM

https://docs.pinot.apache.org/configuration-reference/functions/strpos

Larry Meadors

12/22/2022, 2:01 PM

see the last example?

Larry Meadors

12/22/2022, 2:02 PM

so in the string

" Copper..."

, it's

Shakti Singh

12/22/2022, 3:44 PM

<!here> I am unable to create realtime table in managed service kafka in AWS. It results in 500 error message without saying anything.. Can someone help on this please

Abhijeet Kushe

12/22/2022, 11:14 PM

<!here> We are getting the following exception in our prod environment - Pinot server Sleep for 10000ms as service status has not turned GOOD: MultipleCallbackServiceStatusCallbackIdealStateAndCurrentStateMatchServiceStatusCallbackpartition=workflowEvents_ │ │ at shaded.software.amazon.awssdk.core.exception.AbortedException.create(AbortedException.java:38) ~[pinot-all-0.9.1-jar-with-dependencies.jar:0.9.1-f8ec6f6f8eead03488d3 │ │ at shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.handleInterruptedException(ApiCallAttemptTimeoutTrackingStage.jav │ │ at shaded.software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.translatePipelineException(ApiCallAttemptTimeoutTrackingStage.jav │ └──

Mahesh babu

12/23/2022, 5:41 AM

Hi Team we are running performance benchmark with druid where i found out that there is huge difference between running query in PINOT-UI and PINOT-JDBC

Ralph Debusmann

12/24/2022, 3:45 PM

Hi we have a strange problem seemingly with pinot-server... at some point, it stops consuming messages from Kafka and continuously shows the following error messages/exception:

Millis since last controller cache value invalidate 27102 is less than allowed frequency 30000. Skipping invalidate.

Holding after response from Controller: {"offset":-1,"streamPartitionMsgOffset":null,"status":"NOT_SENT","isSplitCommitType":false,"buildTimeSec":-1}

Sending request: <http://pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local:80/segmentConsumed?reason=timeLimit&streamPartitionMsgOffset=97526&instance=Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098&offset=-1&name=sentiments__0__0__20221216T1149Z&rowCount=97526&memoryUsedBytes=18746772> to controller: pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local, version: Unknown

Could not send request <http://pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local:80/segmentConsumed?reason=timeLimit&streamPartitionMsgOffset=97526&instance=Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098&offset=-1&name=sentiments__0__0__20221216T1149Z&rowCount=97526&memoryUsedBytes=18746772>

_org.apache.pinot.common.exception.HttpErrorStatusException_: Got error status code: 403 (Forbidden) with reason: "Permission is denied for READ '/segmentConsumed'" while sending request: <http://pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local:80/segmentConsumed?reason=timeLimit&streamPartitionMsgOffset=97526&instance=Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098&offset=-1&name=sentiments__0__0__20221216T1149Z&rowCount=97526&memoryUsedBytes=18746772> to controller: pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local, version: Unknown

at org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:442) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-36c82b66b9845e8f1be7c8f3070656099d10ecfa]
at org.apache.pinot.common.utils.FileUploadDownloadClient.sendSegmentCompletionProtocolRequest(FileUploadDownloadClient.java:1039) ~[pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-36c82b66b9845e8f1be7c8f3070656099d10ecfa]
at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.sendRequest(ServerSegmentCompletionProtocolHandler.java:217) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-36c82b66b9845e8f1be7c8f3070656099d10ecfa]
at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.segmentConsumed(ServerSegmentCompletionProtocolHandler.java:184) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-36c82b66b9845e8f1be7c8f3070656099d10ecfa]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.postSegmentConsumedMsg(LLRealtimeSegmentDataManager.java:1110) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-36c82b66b9845e8f1be7c8f3070656099d10ecfa]
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:650) [pinot-all-0.12.0-SNAPSHOT-jar-with-dependencies.jar:0.12.0-SNAPSHOT-36c82b66b9845e8f1be7c8f3070656099d10ecfa]
at java.lang.Thread.run(Thread.java:829) [?:?]

Invalidating cached controller leader value

Ralph Debusmann

12/24/2022, 3:45 PM

Any ideas?

Mayank

12/24/2022, 5:15 PM

Any reason to use port 80 instead of much larger port number?

Mayank

12/24/2022, 5:18 PM

I think your yaml file has misconfigured ports. This call should have gone to controller port 9000, but log says port 80

Ralph Debusmann

12/24/2022, 5:38 PM

Ah that might be it, I'll try that next!

Ralph Debusmann

12/24/2022, 5:38 PM

...if I find the config item for the port...

Alice

12/24/2022, 12:29 AM

Hi team, I’ve a question about rebalance. When I add a new server and applied rebalance with includeConsuming as false according to the doc, I found one consuming segment hasn’t been moved to the right instance after its status became online, while other online segments of the same partition had moved to the right server instance. So my question is how long will a consuming segment be moved to the right instance after it’s completed? Or do I have to do a rebalance again to make this segment change the server instance?

piby

12/27/2022, 3:17 PM

Hi, Is there a way to set base url in pinot controller UI? I have EKS cluster with single ingress controller and single alb. I want to host pinot controller UI at

CLUSTER_URL/pinot/controller

. I can see the main UI window but nothing is working as all the inbuilt links are redirected to

CLUSTER_URL

. For example, the swagger link is being redirected to

CLUSTER_URL/help

instead of

CLUSTER_URL/pinot/controller/help

. Here is my ingress resource Here is my ingress resource

Copy code

apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  annotations:
    <http://nginx.ingress.kubernetes.io/force-ssl-redirect|nginx.ingress.kubernetes.io/force-ssl-redirect>: "false"
    <http://nginx.ingress.kubernetes.io/rewrite-target|nginx.ingress.kubernetes.io/rewrite-target>: /$2
    <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "false"
    <http://nginx.ingress.kubernetes.io/use-regex|nginx.ingress.kubernetes.io/use-regex>: "true"
    <http://nginx.ingress.kubernetes.io/app-root|nginx.ingress.kubernetes.io/app-root>: "/pinot/controller"
    <http://nginx.ingress.kubernetes.io/x-forwarded-prefix|nginx.ingress.kubernetes.io/x-forwarded-prefix>: "/pinot/controller"
  name: pinot-controller
  namespace: pinot-quickstart
spec:
  ingressClassName: nginx-v2
  rules:
  - http:
      paths:
        backend:
          service:
            name: pinot-controller
            port:
              number: 9000
        path: /pinot/controller(/|$)(.*)
        pathType: ImplementationSpecific

Anish Nair

12/29/2022, 7:25 AM

Hey @Mayank @Neha Pawar and team, Needed help with something. Can we change/update the time boundary of the _OFFLINE table? I am checking TimeBoundary from this endpoint: http://host:8099/debug/timeBoundary/pinot_test_table_name

Shreeram Goyal

12/29/2022, 11:50 AM

Hi @User @User and team, I am using pinot release v0.11 and have set up a cluster with 2 controllers, 3 servers and 2 brokers. I am trying to use presto for querying the data and have observed the following: 1. When I do a normal query (eg. select * from table), I get results. 2. when I try to limit the query or try count(*), I get the following error on the broker logs:

Copy code

ERROR [PinotClientRequest] [jersey-server-managed-async-executor-3] Caught exception while processing POST request
java.lang.IllegalStateException: Null handling cannot be enabled for data table version smaller than 4
	at org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:444) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.core.util.QueryOptionsUtils.isNullHandlingEnabled(QueryOptionsUtils.java:95) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.core.query.request.context.QueryContext$Builder.build(QueryContext.java:502) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils.getQueryContext(QueryContextConverterUtils.java:163) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.core.query.reduce.BrokerReduceService.reduceOnDataTable(BrokerReduceService.java:104) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.broker.requesthandler.SingleConnectionBrokerRequestHandler.processBrokerRequest(SingleConnectionBrokerRequestHandler.java:139) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:684) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:280) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.broker.requesthandler.BrokerRequestHandlerDelegate.handleRequest(BrokerRequestHandlerDelegate.java:105) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.broker.api.resources.PinotClientRequest.executeSqlQuery(PinotClientRequest.java:224) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.broker.api.resources.PinotClientRequest.processSqlQueryPost(PinotClientRequest.java:146) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.lambda$apply$0(ResourceMethodInvoker.java:387) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2$1.run(ServerRuntime.java:816) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.glassfish.jersey.server.ServerRuntime$AsyncResponder$2.run(ServerRuntime.java:811) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2022/12/29 17:04:41.273 ERROR [PinotClientRequest] [jersey-server-managed-async-executor-3] Caught exception while processing POST request
java.lang.IllegalStateException: Null handling cannot be enabled for data table version smaller than 4
	at org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:444) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.core.util.QueryOptionsUtils.isNullHandlingEnabled(QueryOptionsUtils.java:95) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.core.query.request.context.QueryContext$Builder.build(QueryContext.java:502) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.core.query.request.context.utils.QueryContextConverterUtils.getQueryContext(QueryContextConverterUtils.java:163) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.core.query.reduce.BrokerReduceService.reduceOnDataTable(BrokerReduceService.java:104) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.broker.requesthandler.SingleConnectionBrokerRequestHandler.processBrokerRequest(SingleConnectionBrokerRequestHandler.java:139) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:684) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:280) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.broker.requesthand

What could be the issue? From the error msg I understand that table version 4 is related to v2 engine. I could run the query after disabling null handling, but we would require null handling on production.

Mostafa Ghadimi

12/31/2022, 2:03 PM

Hi guys, we are experiencing the server consuming much memory. Can someone helping me trying to figure out why this problem happens? https://stackoverflow.com/questions/74969909/apache-pinot-server-component-consumes-unexpected-amount-of-memory

vishal

01/02/2023, 11:40 AM

Can somebody help me with this thread? https://apache-pinot.slack.com/archives/C03R3U53XAA/p1672659588880209 https://apache-pinot.slack.com/archives/C03R3U53XAA/p1672662383478469

Mostafa Ghadimi

01/02/2023, 2:13 PM

Is it possible to query to Pinot in order to set alert in Grafana? I have searched over internet and find just this link in Grafana official repository issues.

Shreeram Goyal

01/02/2023, 4:02 PM

Hi, I am using presto over pinot to query data from tables. My query requires scanning rows ~100M and I observed that even if I abort the query on presto, the memory isn't released by pinot servers and when I re-run the query, the server eventually goes dead due to low memory and I have to restart the servers. Can anybody help with this?

Rohit Anilkumar

01/03/2023, 9:18 AM

hey, I am trying to batch ingest ~160 GB data to a pinot cluster hosted on EC2 instances. Instance used is m5.xlarge and the server has 200GB memory. When running the ingestion job, after a while it crashes with the following error

Caused by: java.io.IOException: No space left on device

. When looking at the logs in controller, I see that the s3 files are copied to tmp/ (Eg:

Copy <s3://bucket-name/combined/2022/test/part-00296-892dfe08-a398-4cfb-97cd-e7fd9ac9a001-c000.json> to local /tmp/pinot-82d605a7-8a5b-4fce-b6dd-c19b3cab6983/input/67860cff-4669-4d9d-a8c0-9e2756cec8f3/part-00296-892dfe08-a398-4cfb-97cd-e7fd9ac9a001-c000.json

) but I dont see any such data in the /tmp folder in controller. Also when I run

df -h

on controller and server, there is enough space left. Controller space

Copy code

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.7G     0  7.7G   0% /dev
tmpfs           7.7G     0  7.7G   0% /dev/shm
tmpfs           7.7G  508K  7.7G   1% /run
tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup
/dev/nvme0n1p1   10G  3.0G  7.1G  30% /
tmpfs           1.6G     0  1.6G   0% /run/user/1000

Server space

Copy code

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.7G     0  7.7G   0% /dev
tmpfs           7.7G     0  7.7G   0% /dev/shm
tmpfs           7.7G  444K  7.7G   1% /run
tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup
/dev/nvme0n1p1  200G  3.1G  197G   2% /
tmpfs           1.6G     0  1.6G   0% /run/user/1000

Can someone help me out with this

Mostafa Ghadimi

01/03/2023, 10:50 AM

Another issue and maybe the final issue I am facing to in deploying Apache Pinot is setting HDFS as deep store for it. I have read the https://docs.pinot.apache.org/basics/getting-started/hdfs-as-deepstorage article about how configuring HDFS as deep store for Pinot. But I have a question about

pinot.server.storage.factory.hdfs.hadoop.conf.path

. Since there isn't any real-world example about how to configure a remote deep store (in this case HDFS) I am asking this question. In the example described in the above link it talked about the situation where we have HDFS installed on the local nodes! How can I set a remote HDFS deep store in a Dockerized solution?