Hey, Pinot server got crashed unable to see /query data

on server logs i found these exceptions

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "nioEventLoopGroup-3-1" AsyncLogger error handling event seq=AsyncLogger error handling event seq=AsyncLogger error handling event seq=AsyncLogger error handling event seq=AsyncLogger error handling event seq=AsyncLogger error handling event seq=AsyncLogger error handling event seq=AsyncLogger error handling event seq=10, value='Exception in thread "pqr-0" null': java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space

Hey, After running a query in dashboard , zookeeper got crashed

im trying to restart it, but somehow its always getting crashed

in logs i can see storage issues

after debugging i found /data directory in zookeeper pod is getting full

can anyone please help in this ?

@Daniel Lavoie ^^ do we have a values. Yaml for prod

@beerus you can config  environment variables : 

ZK_SNAP_RETAIN_COUNT=3

  and 

ZK_PURGE_INTERVAL=1

  to zk statefulset

Hello team !

Actually when i am doing setup of kafka and pinot on AWS facing some issues. Like i have installed pinot using docker container and in same network i have launched kafka container but kafka container is shutting down automatically i check the logs please have a look into this and help me out. Thanks.

Hi team, The query response time is approx 5sec all the time on Pinot. Is there any ideal configs suggested for prod environment?Also, currently we see server is taking up 91% server requests. We are going to double up CPU for Pinot server.But just wanted to check, what is the ideal configs for broker,controller and zk/server

Cc @Mohamed Sultan @Mohamed Kashifuddin @Sadim Nadeem

This is of little high priority, if you can be of assistance ,it would be great!

Hello, I'm thinking of a way of deploying pinot on k8s suitable for my internal systems. Is the following an option: deploying servers, brokers and controllers without persistent storage and counting on S3 deepstorage for data persistence? PS: I'm only interested in streaming data. From my understanding this would only really affect the server, since it stores completed segment both local disk and S3 (and probably prefers reading from local rather than downloading from S3). Do controllers and brokers make use of persistent storage in any?

Brokers don’t need persistent storage..

Thanks! Do you think I can get away with using ephemeral local storage for the server? Documentation on how deepstorage is used is sparse. Do servers save backups of segments right after they are created/completed?

Yes. Servers backup immediately as part of segment commit

That a good point, didn't think of that, ty

how long does it take for config update to propagate?

$ curl -X POST "http://.../cluster/configs" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{\"controller.dimTable.maxSize\": \"500M\"}"
{"status":"Updated cluster config."}

$ curl -X GET "http://.../cluster/configs" -H  "accept: application/json"
{... "controller.dimTable.maxSize":"500M", ...}

$ curl -i -X POST -H 'Content-Type: application/json'  -d @dimension-table.json http://.../tables
{"_code":400,"_error":"Invalid table config for table demographics-2021-03_OFFLINE: java.lang.IllegalStateException: Invalid storage quota: 210763776, max allowed size: 209715200"}

Hey hey! 👋 🔈 🔈 🔈 We're looking for presenters for the Apache Pinot 🍷 meetup!!!! 😃 🧠 Anyone have any topics they're interested in presenting or have ideas for topics you'd like to see, please DM me! ✉️

Hello, I want to create a new table on Pinot, on my data source I don't have a column for the time I only have STRING type columns. I created the schema, I made the config table file but when running I got this error do you have an idea or a solution

Hello guys, Any ideas how to troubleshoot this issue. My kafka topic has 70M+ messages but pinot only consume 23M+

I got this from my zookeeper log

2021-09-28 06:26:58,746 - WARN  [NIOWorkerThread-3:NIOServerCnxn@371] - Unexpected exception
zookeeper_1     | EndOfStreamException: Unable to read additional data from client, it probably closed the socket: address = /172.31.5.244:36258, session = 0x1060148907c0003
zookeeper_1     |       at org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:170)
zookeeper_1     |       at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:333)
zookeeper_1     |       at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:508)
zookeeper_1     |       at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
zookeeper_1     |       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
zookeeper_1     |       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
zookeeper_1     |       at java.base/java.lang.Thread.run(Thread.java:829)
zookeeper_1     | 2021-09-28 23:20:06,750 - WARN  [NIOWorkerThread-2:NIOServerCnxn@371] - Unexpected exception
zookeeper_1     | EndOfStreamException: Unable to read additional data from client, it probably closed the socket: address = /172.31.12.84:51724, session = 0x1060148907c000c
zookeeper_1     |       at org.apache.zookeeper.server.NIOServerCnxn.handleFailedRead(NIOServerCnxn.java:170)
zookeeper_1     |       at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:333)
zookeeper_1     |       at org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:508)
zookeeper_1     |       at org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
zookeeper_1     |       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
zookeeper_1     |       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
zookeeper_1     |       at java.base/java.lang.Thread.run(Thread.java:829)

Hi, I get below error when starting a pinot broker, any idea what could be causing it. The key thing I want to achieve is to set the broker to use hostname instead of the IP which changes on restart:

Executing command: StartBroker -zkAddress pinot-zookeeper:2181 -configFileName /tmp/config/broker.conf
Caught exception while starting broker, exiting
java.lang.NullPointerException: null
        at java.util.HashMap.putMapEntries(HashMap.java:496) ~[?:?]
        at java.util.HashMap.putAll(HashMap.java:780) ~[?:?]
        at org.apache.pinot.tools.admin.command.StartBrokerCommand.getBrokerConf(StartBrokerCommand.java:140) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at org.apache.pinot.tools.admin.command.StartBrokerCommand.execute(StartBrokerCommand.java:121) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:166) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:186) [pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]

Hello, is there an optimal table config for upserts?. I'm able to consume all my kafka topic without the upsert config (93M+ messsages). However, when I put the upsert config on my table it stops consuming at a certain offset around (23M+ messages)

When i use the pinot api to check that the tables are all ok , using ('ingestionState') == "UNHEALTHY": I get the following error messages Error Message for stream4_metric: Did not get any response from servers for segment: stream4_metric__0__22__20210930T1201Z Segment Error for:-> stream4_metric__0__22__20210930T1201Z: {"code":404,​"error":"[] not found."} and Error Message for stream5_events: Did not get any response from servers for segment: stream5_events__0__0__20210819T1031Z Segment Error for:-> stream5_events__0__0__20210819T1031Z: {"code":404,​"error":"[] not found."} So when i go to the server(x2) the steam4 file is totally missing and stream5 is in a _tmp directory ls -l stream4_metric__0__22__20210930T1201Z ls: cannot access stream4_metric__0__22__20210930T1201Z: No such file or directory There are no files in _tmp and for the stream4 file it is in the _tmp directory , multiple times and as you can see has data .. ls stream5_events__0__0__20210819T1031Z tmp-stream5_events__0__0__20210819T1031Z-1629822465115: tmp-c1054878-57f2-47f7-b7c2-085b5dfec02f tmp-stream5_events__0__0__20210819T1031Z-1630594394292: tmp-fdb518aa-929b-4eb3-9ecf-a4386ae9d084 tmp-stream5_events__0__0__20210819T1031Z-1630661411117: tmp-5f8ed825-d323-4344-b433-5287a7a32d4e tmp-stream5_events__0__0__20210819T1031Z-1632927825055: tmp-0f94cfe9-f024-4deb-9f72-8435b5315ee5 tmp-stream5_events__0__0__20210819T1031Z-1633016079847: tmp-df43885d-83c5-41fb-9eb1-0cfd1bfc6be9 /data/pinot/data/index/stream5_events_REALTIME/_tmp/tmp-stream5_events__0__0__20210819T1031Z-1629822465115/tmp-c1054878-57f2-47f7-b7c2-085b5dfec02f tmp-c1054878-57f2-47f7-b7c2-085b5dfec02f]$ ls -l total 24176 -rw-r--r--. 1 pinot pinot   20 Aug 24 17:27 device_id.dict -rw-r--r--. 1 pinot pinot   541 Aug 24 17:27 device_id.sv.unsorted.fwd -rw-r--r--. 1 pinot pinot 24638218 Aug 24 17:27 event.dict drwxr-xr-x. 2 pinot pinot  4096 Aug 24 17:27 event.json.idx.tmp -rw-r--r--. 1 pinot pinot  1982 Aug 24 17:27 event.sv.unsorted.fwd -rw-r--r--. 1 pinot pinot  9384 Aug 24 17:27 event_timestamp.dict -rw-r--r--. 1 pinot pinot  1982 Aug 24 17:27 event_timestamp.sv.unsorted.fwd -rw-r--r--. 1 pinot pinot    0 Aug 24 17:27 h3_index.sv.raw.fwd -rw-r--r--. 1 pinot pinot    2 Aug 24 17:27 is_private.dict -rw-r--r--. 1 pinot pinot   181 Aug 24 17:27 is_private.sv.unsorted.fwd -rw-r--r--. 1 pinot pinot    2 Aug 24 17:27 is_working_time.dict -rw-r--r--. 1 pinot pinot   181 Aug 24 17:27 is_working_time.sv.unsorted.fwd -rw-r--r--. 1 pinot pinot  1920 Aug 24 17:27 namespace.dict -rw-r--r--. 1 pinot pinot  1081 Aug 24 17:27 namespace.sv.unsorted.fwd -rw-r--r--. 1 pinot pinot  11528 Aug 24 17:27 rcvd_timestamp.dict -rw-r--r--. 1 pinot pinot  11528 Aug 24 17:27 rcvd_timestamp.sv.sorted.fwd -rw-r--r--. 1 pinot pinot    0 Aug 24 17:27 uuid.sv.raw.fwd So the question is 1. how to do remove phantom files from reporting when you check the tables health and look at ingestionStatus .. 2. why would the files for stream5 still be in _tmp and also not being seen when you do a table health check and look at the ingestionStatus , and how do you fix this issue.

@Mayank @Xiang Fu @Jackie @Subbu Subramaniam The timeseries databases like influxdb or cortex can be connected with grafana and charts can be loaded with aggregations .. is there support for connecting grafana and build visualizations on top of data in pinot tables .. or we need to go with something like superset