HI, Is the cloud pubsub driver available with apache pinot?

Can anyone explain me what is the issue here.i have started Pinot controller broker and server.since last 2 weeks it is working fine but today suddenly that UI part is not loading.

I have attached the video also.its not stable loading for milliseconds and then again showing blank

We have seen both models in deployment. It depends on your use case. If you are going to just server dashboards and are ok with sub-second latencies, Pinot+Presto might work (note that complex joins/nested queries may make the latency higher than expected). If you are looking for high-throughput at low latency, perhaps using Pinot would be better option.

In config file I have added like so i.am.only able to see the rows whose value is not null.

Hi, I have a question about StarTree indices: are there any recommendations about ordering in

tableIndexConfig.starTreeIndexConfigs.dimensionsSplitOrder

Thanks in advance :)

Hi All, My server is crashing every time due of OOM, My servers are r5.4xlarge and heap space i have provided is 110GB. I have 2 servers.

Hello -- I created a default real time table . After consuming some 300k events i wanted to add a sorted inverted so i edited the table to add the sorted col. How can I check if my query is using the index and whether the index is successfully created or not.?

is there any way to get the explain plan ?

Any idea why we convert division of long/long to double by default? for example, in presto

select (1618900560000 - 1618437600000)/604800000*604800000

returns 0, while

SELECT (1618900560000 - 1618437600000)/604800000

returns 0.76547 ?

🍷 Welcome new Pinot community members! 👋 We're so happy that you've joined us! If you haven't already, please take a moment and introduce yourself, let us know how you found Pinot and what you're working on! 😃 @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User

Hey, do we have any example querying/working on the multi-value column? I cannot find one in the document. e.g. get the first element in the multi-val column.

Hi, i'm streaming from Kafka and would like to have a way to uniquely identify messages. In Kafka consumer, I can do that with offset. Is it possible to expose Kafka metadata such as offset/timestamp to Pinot clients?

How Pinot loads the segment.i.e. If I am consuming data from Kafka and if any new events come in Kafka so pinot will load the entire segment once again or only the new events.?

Hello, I'm new to Pinot and I'm trying to run some tests on the tool using a GCP Deployment. I have deployed GCP o GKE using the the official documentation and it seems to work well. The next step is to import batch data from Google Cloud Storage in order to run some queries. The data on Cloud Storage is 300 Gb of csv files. First thing : I couldn't figure out how to link Cloud Storage to Apache Pinot via the pinot plugin. Second thing : How can I transfrom the csv files into Pinot Tables ? Can I have some guidance on these two subjects ? Thanks in advance 😄

What’s the use case for running with multiple controllers? These are stateless, and don’t have a lot of load (if using something like HDFS for deep storage), right? So is it just zero downtime (assuming you have a LB in front of them) in case one goes down?

If I have one controller and two brokers, does the controller distribute the query load across the two brokers? I thought it would, but the Pinot in production page recommends “HTTP load balancers for spraying queries across brokers (or other mechanism to balance queries)“. Or is that to automatically route traffic away from a broker which has gone down?

Hi All, everyone my pinot server keep getting lost on restart, have kept JVM 16GB and table [“loadMode”: “MMAP”] . Screenshot

Hi all! For anyone unfamiliar with Zookeeper operations (which we were), you will see that Zookeeper will keep increasing its disk usage. We found this odd since it only allows 1MB worth of data in its leaf nodes. Looking at our data directory we saw its disk usage mainly in the logs folder and the snapshots. After searching a bit, we found that there are two configurations that will automatically purge these files (

autopurge.snapRetainCount

and

autopurge.purgeInterval

). The logs are transaction logs, not application level logs, and they relate to the snapshots such that Zookeeper can recover from a failure by using the latest snapshot and the transaction logs. The

purgeInterval

is 0 by default so it does not purge anything. and

snapRetainCount

is 3 - but again this is disabled. Depending on the docker image and helm chart you are using you already have an env var to change the `purgeInterval`: •

ZOO_AUTOPURGE_PURGEINTERVAL

for the official docker image •

ZK_PURGE_INTERVAL

for the zookeeper incubator helm chart Hope this helps!

Hi All, Expecting that all the finished segments will be rebalanced among all 3 pinot servers, instead of just the consuming server post addition of 1 new server from 2 to 3 nodes now.

Can I use an aggregation result as argument to a UDF during query?

Someone kindly guide where can I find these 2 details.I am trying to use HDFS as deepatorage and need to set these 2 properties as per the pinot docs, but unable to find which path I should use. I am loading streaming data from Kafka topic and pinot is running in cluster-mode. Controller , server and brokers are running on diff diff nodes.

Is there any limit in Pinot to execute queries simentenoulsly. I.e. I have created a pipeline through pinot where we are ingesting data from Kafka topic and creating Pinot table.After that connecting from prestro using pinot-connector and in the last step connecting from superset and from there querying pinot table. Here I have generated 4 graphs using diff SQL queries and added all 4 in one dashboard.After sometime when I am trying refresh the dashboard page so it's firing 4 queries to Pinot and all are getting failed it's showing internal error couldn't reach host, and if I am refreshing one slice from the dashboard then it's working fine.i.e. when firing one query at a time. P.S. I am facing this issue when I have started pinot in cluster mode with 3 servers 2 controller and 2 broker. When I am doing the same using quick-start it's working fine and entire dashboard is loading/refreshing at same time.

are there any systemd scripts for starting / stopping the pinot components?

Wow, great content. Excellent half hour spent. Great job @User @User

Hi! What is the reason for having Pinot queries implicitly default to

limit 10

? Is there a way to remove the limit in the query without specifying an arbitrary limit value?

Hi, in Pinot docs, “e.g. 365 DAYS in the example means that segments containing data older than 365 days will be deleted periodically.” Does Pinot delete the segment containing ALL data or ANY data older than the retentionperiod?

Welcome new Apache Pinot community members! 👋 We're so happy you're here and joining the movement of speeeeeed for user-facing analytics. 🍷 Please take a moment, if you haven't already - to introduce yourself and tell us what brought you here! 😃 @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User

Hi team, we’re considering Pinot for our customer-facing analytics (mainly dashboards). We also want to use the same data store (Pinot table) to export denormalized report (CSV file of 10k - 100k rows filtered from the table, with ~20 selected columns) to our clients. Is Pinot also a good fit in the report generating use case. For example, our backend service could query the Pinot table and generate the CSV report from the query result. Thanks!

Hello everyone 😄 , is there a tutorial or some doc in which a multi-node setup is explained. i have 5 VMs and want to set it (Pinot) up on them. What are the best practices regarding infrastructure here?