Data stays

But it is very consistent. @User if you want we can do a zoom and I will show you

can you grep the segment name from the server logs and post the results?

oh ok, you are saying the data stays, That is good.

We are experiencing somethning similar, and have narrowed it down to SimpleMarioClient used by kafka consumer. It creates hashed wheel timers causing the server to freeze and die, and we have to restart it. I wonder if this is the same problem you are seeing

@User how mmany realtime servers do you have? How many partitions are there in your kafka topic? What is the rate of ingestion? (events per second, on each partition), What is the amount of memory on the hosts? Have you turned on offheap memory for consuming segments? Reading through https://engineering.linkedin.com/blog/2019/auto-tuning-pinot? may help

3 servers, 100 partitions. No ingestion right now. Offheap is enabled

what is the replication factor? So, realtime servers are cracking up with zero ingestion and one query?>

Kafka - 3

Same for table

Yep, nothing on Kafka

1 query

Is there a to uppercase string function in pinot?

should be simple to add

image.png

Can’t think of a valid reason for that.

We are trying to use HadoopPinotFS for deep storage in the controller but get the following classpath error. The HadoopPinotFS is not in the lib directory of the pinot-distribution:

java.lang.ClassNotFoundException: org.apache.pinot.filesystem.HadoopPinotFS
	at <http://java.net|java.net>.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_232]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_232]
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) ~[?:1.8.0_232]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_232]
	at java.lang.Class.forName0(Native Method) ~[?:1.8.0_232]
	at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_232]
	at org.apache.pinot.filesystem.PinotFSFactory.init(PinotFSFactory.java:57) ~[pinot-common-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-eb45b438c5053f5caaf289614f386706a472947e]
	at org.apache.pinot.controller.ControllerStarter.initPinotFSFactory(ControllerStarter.java:422) ~[pinot-controller-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-eb45b438c5053f5caaf289614f386706a472947e]
	at org.apache.pinot.controller.ControllerStarter.setUpPinotController(ControllerStarter.java:267) ~[pinot-controller-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-eb45b438c5053f5caaf289614f386706a472947e]
	at org.apache.pinot.controller.ControllerStarter.start(ControllerStarter.java:229) ~[pinot-controller-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-eb45b438c5053f5caaf289614f386706a472947e]
	at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:163) ~[pinot-tools-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-eb45b438c5053f5caaf289614f386706a472947e]
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:145) ~[pinot-tools-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-eb45b438c5053f5caaf289614f386706a472947e]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:156) ~[pinot-tools-0.2.0-SNAPSHOT.jar:0.2.0-SNAPSHOT-eb45b438c5053f5caaf289614f386706a472947e]

Is anyone familiar with using HadoopPinotFS?

You need to add Pinot Hadoop filesystem jar to the classpath

Yep, I see that the admin.sh script is generated in the pinot-tools module. Where would I add it? If this works I can submit a pr with the fix.

Set EXTRA_CLASSPATH environment variable to include Hadoop filesystem jars.

Hello everyone, how can we work with many date fields in a pinot table?

You could have multiple date fields, but will need to pick at most one of them as a time column.

Pinot documentation doesn't mention about date type in dimensions

There is a datetime spec

You can have multiple columns datetime fields.

https://cwiki.apache.org/confluence/display/PINOT/Schema+TimeSpec+Refactoring This might help

INT, FLOAT, LONG, DOUBLE, STRING, BYTES and also DATETIME field? I think that we need update the section "Pinot schema" then.