Hi team, Can you pls help on how a downloaded segment can be decompressed ?

📣 Interested in being a conference speaker? 📣 Current Summit is looking for speakers!!! (formerly Kafka Summit) Submit here: https://sessionize.com/current-2022/ Need help with sharpening abstract/talk description or title? Send me a blurb and I will see if I can spice it up for you! 😃

When making the connections to pinot brokers via vip, the connection can be stale if the broker is changed. How can you solve this problem?

Hi team , I came up checking the difference between Groovy Function Time difference with best case scenario Topic's . When I came up with the segment.start.time and end.time and creation.time , I am confused with the start and end time . I have given the time , segments , and conf for my tables , Kindly resolve my issue.

Thanks to @Kam Leung! We’ve added linen.dev to Pinot community to provide searchable(Google also indexed this!) slack public channel. You can search for past discussions and join the threads from here: https://www.linen.dev/s/apache-pinot

Hey team, deployed two different 0.11.0 nightly builds and found that an equality filter predicate on a field with a sorted column index isn’t working as expected if the segment is in CONSUMING state. Ex:

select distinct (company) from role_with_company limit 1000000 -- answer: 51

Queries w/ less than or greater than predicates returns always the correct results:

select count(distinct company) from role_with_company where company < '6269223774083d800011fd95' limit 1000000 -- answer: 36
select count(distinct company) from role_with_company where company > '6269223774083d800011fd95' limit 1000000 -- answer: 14

On the other hand, equality predicates when the segment is in CONSUMING state does not return the correct results:

select count(distinct company) from role_with_company where company = '6269223774083d800011fd95' limit 1000000 -- answer: 0, when segment is in CONSUMING state

When the segment is COMMITTED, the query returns the correct results:

select count(distinct company) from role_with_company where company = '6269223774083d800011fd95' limit 1000000 -- answer: 1, when segment is COMMITTED

Anyone aware of a change in behaviour that was introduced recently? @Richard Startin @Jackie

@Nizar Hejazi Thanks for reporting the issue. Can you share the commit hash for the 2 builds?

I don't recall any recent change on the equals predicate. You may also query the virtual column

$segmentName

to identify which segments contain the value

Let's move the conversation to #C011C9JHN7R

One basic question about data preparation (for ingestion into Pinot) - how do you combine e.g. multiple Kafka topics into one table in Pinot so that you can query them as one - without having JOINs? Is there any way to do it without heavy upfront stream processing using e.g. Kafka Streams/ksqlDB/Flink/Materialize etc.?

Hi team, I have a question about using Pinot JDBC to connect Pinot controller deployed in K8s.

DriverManager.registerDriver(new PinotDriver());
//Connection conn = DriverManager.getConnection(DB_URL);

// will query DefaultTenant if not specified tenant here
Properties info = new Properties();
info.putIfAbsent("tenant", "TestBroker");
Connection conn = DriverManager.getConnection(DB_URL,info);
Statement statement = conn.createStatement();

but the following error returned:

Caused by: java.net.UnknownHostException: pinot-broker-8.pinot-broker-headless.pinot.svc.cluster.local: nodename nor servname provided, or not known
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
	at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1515)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
	at java.base/java.net.InetAddress.getByName(InetAddress.java:1248)
	at com.ning.http.client.NameResolver$JdkNameResolver.resolve(NameResolver.java:28)
	at com.ning.http.client.providers.netty.request.NettyRequestSender.remoteAddress(NettyRequestSender.java:359)
	at com.ning.http.client.providers.netty.request.NettyRequestSender.connect(NettyRequestSender.java:370)
	at com.ning.http.client.providers.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:282)
	... 12 more

Is there any configuration to make sure controller can get the right accessible broker url?

I tried locally to execute query via broker through the command below. curl -H “Content-Type: application/json” -X POST -d ‘{“sql”:“select * from action limit 1”}’ http://pinot.broker.eks0.auw:80/query/sql This pinot cluster is deployed in K8s cluster and we have 4 brokers in this pinot cluster. http://pinot.broker.eks0.auw:80 is the exposed broker gateway address. This table named action here is configured to tenant named tenanta which has only one broker. When I run the above command, I can get the right results sometimes. But most of the time, the following error returned. org.apache.pinot.client.PinotClientException: Query had processing exceptions: [{“message”“BrokerResourceMissingError”,“errorCode”410}] at org.apache.pinot.client.Connection.execute(Connection.java:127) at com.bigdata.PinotJava.main(PinotJava.java:53) Is there some configuration I’m missing here for this issue? Any idea how to fix it?

Hello Team, Can someone help me with the below error

Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "Ingress" in version "extensions/v1beta1"

Hey, I was reading about the data centric design, where the proposed architecture suggests that application code to be less, and more things happening from database systems only. The idea is database will evolve to serve greater needs which essentially means, applications will be heavy on business logic rather than doing mundane (generic) work. My mind immediately went to Pinot. Pinot is serving a greater need out of the box. Is this the trend folks here observe too? What do you think about the future?

Hello Team, Can someone explain why the controller pod keeps crashing in the EKS cluster wondering if is about the resource allocation

👋 Hi everyone! Fernando from Brazil here!

Hi.Team. Some pino-servers fail to connect to Zookeeper and do nothing. But the daemon process is still running. I built a cluster with 6 pino-servers on premises. Zookeeper is 3 generations old. The cluster has 1 offline table (1 TB) and 2 upsert realtime tables(30 GB ~). This is the last log written by the pinot-server. The pino-server is dead and there are no logs for several hours.

2022-05-27 01:32:43.487 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.manager.zk.ZKHelixManager:checkConnected:376 zkClient is not connected after waiting 10000ms., clusterName: PinotPoC_2, zkAddress: <http://pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181|pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181>
2022-05-27 01:32:43.487 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.manager.zk.ZKHelixManager:checkConnected:376 zkClient is not connected after waiting 10000ms., clusterName: PinotPoC_2, zkAddress: <http://pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181|pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181>
2022-05-27 01:32:43.487 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.manager.zk.ZKHelixManager:checkConnected:376 zkClient is not connected after waiting 10000ms., clusterName: PinotPoC_2, zkAddress: <http://pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181|pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181>
2022-05-27 01:32:43.487 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.messaging.handling.HelixTask:finalCleanup:390 Error to final clean up for message : 723f5e30-1ffa-447b-a61a-72482e5d6ab4
2022-05-27 01:32:43.487 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.messaging.handling.HelixTask:finalCleanup:390 Error to final clean up for message : 47521252-6e9f-49d5-aaf0-1e1169b65cda
2022-05-27 01:32:43.487 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.messaging.handling.HelixTask:finalCleanup:390 Error to final clean up for message : aa0ae67b-6b92-46dd-b19d-b6bda195af18
2022-05-27 01:32:43.487 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.messaging.handling.HelixTask:call:194 Exception after executing a message, msgId: 723f5e30-1ffa-447b-a61a-72482e5d6ab4org.apache.helix.HelixException: HelixManager is not connected within retry timeout for cluster PinotPoC_2
org.apache.helix.HelixException: HelixManager is not connected within retry timeout for cluster PinotPoC_2
        at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:378) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.manager.zk.ZKHelixManager.getHelixDataAccessor(ZKHelixManager.java:593) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:172) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
2022-05-27 01:32:43.487 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.messaging.handling.HelixTask:call:194 Exception after executing a message, msgId: aa0ae67b-6b92-46dd-b19d-b6bda195af18org.apache.helix.HelixException: HelixManager is not connected within retry timeout for cluster PinotPoC_2
org.apache.helix.HelixException: HelixManager is not connected within retry timeout for cluster PinotPoC_2
        at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:378) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.manager.zk.ZKHelixManager.getHelixDataAccessor(ZKHelixManager.java:593) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:172) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
2022-05-27 01:32:43.487 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.messaging.handling.HelixTask:call:194 Exception after executing a message, msgId: 47521252-6e9f-49d5-aaf0-1e1169b65cdaorg.apache.helix.HelixException: HelixManager is not connected within retry timeout for cluster PinotPoC_2
org.apache.helix.HelixException: HelixManager is not connected within retry timeout for cluster PinotPoC_2
        at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:378) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.manager.zk.ZKHelixManager.getHelixDataAccessor(ZKHelixManager.java:593) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:172) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]
2022-05-27 01:32:43.494 WARN  [HelixTaskExecutor-message_handle_thread] org.apache.helix.manager.zk.ZKHelixManager:checkConnected:369 zkClient to <http://pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181|pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181> is not connected, wait for 10000ms.
2022-05-27 01:32:43.494 WARN  [HelixTaskExecutor-message_handle_thread] org.apache.helix.manager.zk.ZKHelixManager:checkConnected:369 zkClient to <http://pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181|pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181> is not connected, wait for 10000ms.
2022-05-27 01:32:43.494 WARN  [HelixTaskExecutor-message_handle_thread] org.apache.helix.manager.zk.ZKHelixManager:checkConnected:369 zkClient to <http://pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181|pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181> is not connected, wait for 10000ms.
2022-05-27 01:32:57.146 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.manager.zk.ZKHelixManager:checkConnected:376 zkClient is not connected after waiting 10000ms., clusterName: PinotPoC_2, zkAddress: <http://pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181|pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181>
2022-05-27 01:32:57.146 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.manager.zk.ZKHelixManager:checkConnected:376 zkClient is not connected after waiting 10000ms., clusterName: PinotPoC_2, zkAddress: <http://pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181|pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181>
2022-05-27 01:32:57.146 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.manager.zk.ZKHelixManager:checkConnected:376 zkClient is not connected after waiting 10000ms., clusterName: PinotPoC_2, zkAddress: <http://pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181|pinot-poc-zk1.company.com:2181,pinot-poc-zk2.company.com:2181,pinot-poc-zk3.company.com:2181>
2022-05-27 01:32:57.146 ERROR [HelixTaskExecutor-message_handle_thread] org.apache.helix.util.StatusUpdateUtil:logMessageStatusUpdateRecord:352 Exception while logging status update
org.apache.helix.HelixException: HelixManager is not connected within retry timeout for cluster PinotPoC_2
        at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:378) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.manager.zk.ZKHelixManager.getHelixDataAccessor(ZKHelixManager.java:593) ~[pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.util.StatusUpdateUtil.logMessageStatusUpdateRecord(StatusUpdateUtil.java:348) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.util.StatusUpdateUtil.logError(StatusUpdateUtil.java:392) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:195) [pinot-all-0.10.0-jar-with-dependencies.jar:0.10.0-30c4635bfeee88f88aa9c9f63b93bcd4a650607f]
        at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.10.0-jar-with-depe

Hi folks! I'm a Pinot user (not a developer) and have been amazed at the support I can get from the Pinot folks in this Slack workspace, they're awesome! I'd like to share some suggestions to other users though, to optimize things for them, but which also might help everyone else: 1. I noticed that usually to try to fix issues the #C011C9JHN7R channel is better, I noticed the Pinot devs more active there, so that's where I suggest users ask for help, not here; 2. One good practice I learned actually here in this workspace (can't remember with whom) whenever I have an issue, was to first start with a short question, and then in the thread for that same question, I put more lengthy information, like config files, stacktraces etc. This is super cool to avoid taking massive amounts of space of the channel wall - it reduces visual pollution, while maintaining the content for the people who want to get involved with that particular discussion.

has anyone tried integrating pinot with looker?

Is there a doc available which discusses differences between Pinot & Druid from a Cloud cost persspective

Hey guys, we have a usecase of doing following query a lot:

select distinct(<columnName>) from <tableName> limit 1000;

Is there an index that might reduce the latency? Assuming the cardinality of the column will rarely cross 1000

Hey guys, how should we be handling the scenario if there are multiple kafka topics that need to be ingested to pinot and joined to have the final result? Should there be a pre-aggregate/lookup streaming job that consolidates multiple topics data into one topic that gets ingested by pinot, or should we use Presto to do the joins?

Hi Pinot team, our security team flagged our pinot deployment in labs for security vulnerabilities with majority coming from

com.fasterxml.jackson

and all of them are addressed in newer versions of dependencies. Any thoughts on how should we go about addressing these. Can share with you the list if interested,

Hi all, doing some testing with Pinot lately. Just wondering, is there a "Kibana"-like tool for Pinot that can make it a little bit easier to visualize data, without having to write an application that does so?

Another question I've been wondering about is this idea of both realtime and offline tables being queried at once, via the same table name. Does anyone have an interesting use case for when they've used this? I'm trying to wrap my head around one

just quick question: does pinot support other partition logics other than time based partition?

quick question: From which pinot version star-tree index GA is available (production ready)?

Hi folks, is it possible to change the primary time column on existing (offline) tables without reloading segments?

Hey folks, I know I said this before, but what a fantastic piece of software Apache Pinot is! I keep finding more and more interesting things about it, smart decisions, great OS support...

Hi in the pinot trino connector, is there a way to change the schema name

default

to something else?