Hi! I'm trying to follow the instructions to deploy quickstart on AWS but I'm hitting issues. The pods are crashing regularly. I added some reproduction instructions here. I'm getting

TLS handshake timeout

error when trying to output the logs. Any help would be appreciated.

regarding mentioned "view for last point" - more precisely querying metric values for given set of dimensions that have most recent timestamp: how to achieve it? can it be created in PQL or schema or maybe requires some C++ plugin?

Unrelated, I was able to get AWS Lambda to send Kafka events to a Kafka running in AWS EKS. If someone else needs to do this, I'm happy to help. The main challenges were (1) I thought updating values in Kafka's Helm chart and then running

helm upgrade

would apply changes to pods.  Some of the values cannot be updated this way (and silently fail). Debugging was challenging until I understood this. (2) Understanding how Kafka's

advertised.listeners

worked.

Hello -- does pinot support using confluent schema registry today ? I saw this PR (https://github.com/apache/incubator-pinot/pull/4973) which is open ...

<!here> we made a video tutorial for How to setup a Pinot cluster:

<!here> Thanks to @User we have a new website https://pinot.apache.org/. Don't forget to try the dark mode, it looks cool 😎

Is it possible to connect to Pinot using existing SQL clients? No worries if not. Thanks!

@User Which version of Pinot are you running? The connection pool implementation has been replaced with the new multiplexing implementation, where connection is reused

@User You should be able to use JMX to instrument the JVM process. https://github.com/prometheus/jmx_exporter

I am guessing it shouldn't be hard to add a Grafana adapter

let us know if you end up building one. Note that presto-pinot has a lot of optimizations to push down aggregation and group by to Pinot and this is only available in PrestoDB

This needs upsert feature , Uber is working on this feature.

Before I build out an offline ingestion solution, is it possible to get Pinot to take the events from the realtime table to populate segments in the offline table?

Hey all, I have deployed pinot in kubernetes. When I start ingesting data in kafka, the server RAM keeps on piling up even after segment creation and eventually pod crashes due to OOM.  I am keeping the segment threshold size as 1000000 and ingesting about 10 million entries in kafka, memory limit is 4GB for server. Why is the memory not released? Thanks!

As I sent earlier in the dev mail list, I proposed adding a segment download api to Pinot server. It is part of the PR to by-pass deep-store requirement for segment completion in low level realtime consumer (design doc). The new segment download rest API in Pinot server would be in the TablesResource class with URI: • /tables/{tableName}/segments/{segmentName} The API allows download of a segment as a zipped tar file. Its primary usage is for segment download when a deep store is unavailable. Please let me know if you have any question.

Remove the tenant tag from server instance

Checking. What was the command you ran?

Was doing a PT on the pinot setup ( 3 brokers, 3 servers, 3 controllers). Currently there are 20,000,000+ entries (64 segments over all 3 servers 35MB as segment size ). The topic in kafka has 3 partitions and replicationPerPartition is 2 in pinot. With a minimal load of 100 rps the brokers are going down. logs given below

Schema?

• stop the producers • wait for pinot to consume all the messages in Kafka • force pinot to flush the segments (not sure if we have an API for this) • change the table config in Pinot to use new decoder • restart the producers

how to enable

enable.parallel.push=true

in new spark data ingestion library ?

Folks, we are currently using simple kafka stream consumer. Can we migrate to HighLevel consumer? I see some documentation on this. Just wanted to check if any tried and succeeded.

Is it possible to connect powerbi direct query mode with backend as pinot via presto or will it support only powerbi import mode? Please confirm.

restarting the server will be the simplest solution. @User, is there any api to drop/reset all consuming segments

@User: we tried restarting controllers & servers. But that didn’t work.

@User There's one here: https://docs.google.com/document/d/1GiB8zoiJ4Qs10A1LdIr6lJGWTfk-0ElU8aJbejomstY/edit?usp=sharing

hmm, do you mean that the offline table schema should match table name ? We will take a look, thanks for reporting !

For monitoring?

Hi All, I am Cinto and currently work at Apple. How are you guys doing ? My team has a real time analytics problem and were trying to evaluate Pinot. However, our data is mutable i.e. older data can come again and will need to be updated. I saw on https://pinot.apache.org/ that pinot works only on immutable data. Can you correct me if I am wrong. I can provide more details if needed. Thanks

Hi All, We are using Pinot. While uploading the segment getting below error. Your help will be much appreciated.