Is it possible to query Pinot from a database IDE like Datagrip?

Hi folks! Let me ask a question about SQL queries in Pinot: is it possible to either truncate a table, or at least delete all of its rows? I'm asking this because I want to be able to do that for integration-testing my application. Thanks!

Hi, I am seeing 2 segments in consuming state for my table and rest are online. Is that expected behavior? Also the consumption is very slow after a certain number of records 3M. What could be the reason for this?

Hi Team, We are planning to build Pinot cluster, and the data from Kafka will be 1000 TPS, So what would be the best cluster spec will look like? like how many nodes we required, memory, cores?

Hi @all can any one please add me to the Thirdeye slack workspace, I am trying to do the setup and the docs look bit old. Facing issues with installation

Hey guys, just to confirm: this is the official Python library to access Pinot, correct? https://github.com/python-pinot-dbapi/pinot-dbapi - I'm asking because of the organization name mismatch

If the Server doesn’t have enough memory for all its segments, does it offload some to disk? What’s the offload directory? Or should I turn on swap + mmap, and let the OS deal with it? (Sorry I couldn’t find any documentation for that.)

Hi folks! Simple question: is it OK to use the same Kafka topic to process events that generate rows for multiple tables and schemas, in a project? Suppose that I have a project, "MyProject", and then I have a "users" table, a "pets" table and a "cars" table, supposing they're all REALTIME, and then I want to create and use a "my_project" topic to publish all these events. Would this be fine?

Hi, I have a STRING column product_str with json blob stored. I have created an index on the column as mentioned in the doc. Sample data from column looks like this: {"fieldA":{"key1":"val1"},"fieldB":"something"} My query is Select ••• from mytable where json_match(product_str, '"$.fieldB"=''somevalue''') I am not getting any result back with above query. Any ideas ?

Hi all, I am looking for thirdeye to be used with pinot, but not getting any good resource. Whatever resource I have found out is not useful. Any suggestion would be helpful.

Hi Team, I have few queries regarding Pinot hybrid table. 1) Lets say, we have a primary key pk1, which is both available in realtime and offline table, on query which table is be preferred by pinot i.e from which table data will be shown in output? 2) Can i append "1 record" into existing offline table? if yes , how soon it will be available to query? Thanks

👋 Welcome to all the new Apache Pinot slack members who joined us this month! 🍷 Would love to know what brought you here, who you are, and how you found out about Apache Pinot! ❤️ @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User

Hi team , I am facing issue with consuming data in my pinot table from kafka streams. I am not getting any errors in logs, can you please help me in letting me know what is the issue? Also there is nothing coming in replica set and the status of table is showing as Bad.

What strategies are commonly used for multi-tenant tables in pinot? Partitioning segments by tenant and using Partitioned Replica-Group Segment Assignment seems like a good strategy to avoid excessive fan out. How do you avoid hot spotting in this scenario? Is Pinot smart enough to balance the number of segments when assigning partitions to groups?

Hi folks! I have a few questions: 1. Is there any estimation for the 0.9.0 release date? 2. What features will be included with it? 3. Will a Docker image be released as soon as the main program itself gets released? Or is this something you're planning to work on later?

How to use GraphQL API endpoint with Pinot? We are storing data in Pinot DB. GraphQL endpoint should query the Pinot DB. The need of GraphQL endpoint is to fulfill dynamic info requirements from front end.

Hi, I have a question regarding fields with with low cardinality. (fields like transaction_status). Needed to ask if we should be keeping those as integers in pinot and maintain enums on application end. or Can we keep the field values as string in pinot and pinot's forward index already optimizes for low cardinality string fields. Will the lookup and aggregation performance for string values be nearly same as with integer?

Hi folks! I'd like to ask some questions about the next release (0.9.0): • Is it going to include any sort of table truncation, even if rudimentary? • Is there a rough estimation for when it will be released?

Hey Everyone 🖐️ I just started using Pinot and loving it already! I am using stream ingestion with Apache Kafka to import data. I just can't get a grasp on deep storage concept. I don't have HDFS or cloud storage setup currently. •Do segments still get flushed to local disk periodically in the absence of a deep storage? •If i restart my Pinot Servers, would it recover old segments in the absence of a deep storage? •What is the main purpose of a deep storage in a Pinot cluster setup? •Is deep storage a must in a production Pinot cluster setup? I would appreciate if you could share your knowledge with me.

Coming from a relational database perspective, I've had some difficulty conceptualizing what my data might look like in Pinot. Is the standard to have multiple tables like in a RDBMS and query them relationally using something like Presto, or should I strive to have less tables with more columns that remove the need for relational querying? If the latter is preferable, is that still the case when the table would have to contain many columns to replace the relational structure and many of those columns would need to contain things like array or JSON object

Just a random comment/praise: the Pinot open source community support is amazing! Thanks for that, guys! I'm looking forward for my next steps in using it in production ❤️

Pinot does not support “NOT” operator and there is no regexp_not_like. So is there any way to do the equivalent of “NOT regexp_like(…,…) at all in Pinot?

Hi team, is there a Pinot query to find out when is the last ingest time of a offline table?

How do I do “array_agg” (in Presto’s sense) in Pinot? (The agg function should just return an array of all the elements.) Should I write my own custom agg function, or run it in Presto? Thanks!

Hi folks, I have some questions about migrating data from a previous database into Pinot: In my project, we'll start publishing data to a Pinot realtime table, but we also need to port historical data. For historical data, do you recommend using an OFFLINE table to be used in conjunction with the REALTIME table, or is it fine to port the historical data to the REALTIME table directly? What are the pros and cons for each approach? Thanks!

Hi, I am running a cluster with HDFS as deep store. I can see that segment are pushed to hdfs and there are no error in debug endpoint but when I am running a simple query , it hangs and there is exception on server side. Attached is the stack trace.could you help ?

Hello 👋 Where can we find Pinot server protobuf definitions files ? I'd like to try and connect to its gRPC streaming endpoint manually

Is regex_like function supported on map column? e.g query

SELECT mapValue(mapDim1__KEYS, 'k1', mapDim1__VALUES) FROM FeatureTest1 WHERE  REGEXP_LIKE(mapValue(mapDim1__KEYS, 'k1', mapDim1__VALUES), '^p.*')  LIMIT 5

Tried similar query on (pinot 0.7.1) and getting exception.

In case you are using MacOs Monterey. They introduce a new feature of airplay receiver which uses port 5000 and 7000. However 7000 is used in pinot server query port by default for Quickstart. The resolution is to disable this feature in System Preference -> Sharing -> uncheck AirPlay Receiver