According to the pinot document, it seems that pinot only supports

Pagination on Selection

. Can we support

Pagination on Aggregation

?

Hi Pinot community, Is there any way we can download the query results in the excel or csv format through api or cmd? Any input would be welcome!

Hi , new to using apache pinot... i want to know which config is actually used to flush segments to deep store , is it?

realtime.segment.flush.threshold.size

or

realtime.segment.flush.threshold.segment.size

?

Also does pinot server writes to deep store (object store) or pinot controller?

If i want optimal performance in pinot which is having n servers , is it recommended to have n partition too of a kafka topic?

What is the frequency or interval at which pinot writes its segment to object store? Is there any config which decides that, what is the size of segment which gets written to object store?

why do i see size as 0 , i do see data in query console, i have set ""realtime.segment.flush.threshold.size": "16M"" but still i dont see segments if i query

curl "<http://ip:9000/tables/table_name/size?detailed=true>" 2> /dev/null | jq -c
{"tableName":"table_name","reportedSizeInBytes":0,"estimatedSizeInBytes":0,"offlineSegments":null,"realtimeSegments":{"reportedSizeInBytes":0,"estimatedSizeInBytes":0,"missingSegments":0,"segments":{}}}

kafka topic has around GB's of data but no segments in pinot... When i run producer to add records in pinot , i do see increase in number of records via pinot query console (ran count(*))... what could be the issue here, why segments are not getting created?

Is there any limit on record size which pinot can consume from kafka topic? I observed if kafka topic has message size of 512 bytes , it is getting 512 bytes in pinot , if message has 1024 bytes then also i see 512 bytes in pinot. (string with 512 chars in pinot ui is displayed)

Do we have direct mapping of number of partition in kafka table to number of segments in pinot table? I mean if kafka table has 4 partition , pinot will also have 4 segments?

Do we know if we need to tag a minion host with some tenant or not? I see our minion host as

minion_untagged

but i find no documentation around how tagging it will be used.

Is there a limit on the Number of columns we can have in pinot for a realtime upsert table ? Also is there a recommended way to do this for eg using Json object instead of multiple column which is better ?

Hi All, I am completely new to Apache Pinot and have some basic questions about whether this is the right fit for our use-case: 1. We have a network flow time-series data which we want to store 2. The data needs to be partitioned on a daily basis and if we see the same flow again within a day (imagine a flow is defined by a unique key probably), then we would just upsert the flow count for that in the same day. 3. There are applications which use this data to run adhoc queries for real-time analysis. Can someone with good background let me know if Pinot is a right fit for the above use-cases?

Hi @Xiang Fu ! The Compatibility Regression Tests are failing due to unexpected compilation errors. I've tried merging and rebasing but my changes to pinot-core appear to be missing. Any suggestions?

Is it true that by default, Pinot creates a single Kafka consumer per server instance for each partition?

Is there any kafka consumer configuration in Pinot to ensure efficient data fetching and processing ?

Hello Is there any way to configure snappy for compression for the deep store files. If so, is there a way to configure Pinot to use native libs. Just want to speed the compression process...

Hi guys I have an append-offline table with time column format YYYY-MM-DD-HH and when I do time range queries of the form: WHERE ((\"time_stamp\" >= 2023-08-02-00 AND \"time_stamp\" <= 2023-08-31-23)) Pinot fails to find the proper entries, if I keep the low bound then the proper records are produced: WHERE (\"time_stamp\" >= 2023-08-02-00): Is this a known issue? any workaround? any other formats for the time column that work properly?

@Mayank @Jackie Can anyone please share the link or info related to helm charts for pinot version 1.0

Hi everyone, I’m trying to implement a new query aggregation function for Apache Datasketches CPC sketches. Looking at the implementation for HLL, I came across a dictionary encoded expression. Can anybody give an example of what this is and how it is used? I’d like to ensure that I test it properly, if I need to support it. Thanks

Hi @Xiang Fu, I'm working on the PR Add GENERATE_ARRAY function #11786 . Could you help to point out to where I could find the code for the existing function

ARRAY[x, y, z]

. I'm trying to find it in both the code and in the documentation but could not find 🤔

Pinot supports ARRAY[x, y, z] syntax to manually craft an array literal. This is good for small scale, and in real prod, we need to add an ARRAY function to generate an ARRAY sequence for a given range.

Similar to the GENERATE_ARRAY function in Google BQ

Question. Is there a official set of legal characters and restrictions (i.e. length, can't begin with certain characters) for the table column name? We are trying to encode arbitrary data into the table column name and want to ensure we escaped all illegal characters and work around the restrictions.

Hi @Jackie I was working on this issue Add API to update ZK metadata of a segment #10458 Can you or someone please help with my understanding here: https://github.com/apache/pinot/issues/10458#issuecomment-1766907480

Hey folks, is there a locally runnable integration test for testing StarTree index?

Question, is

dataTypeTransformer

supposed to be enabled by implicitly for all tables? Is this the expected default behavior?

Can I get help in installing Pinot, Zookeeper and Kafka on Ubuntu Linux machine. I have made several attempts but having a lot of issues. My requirement calls for straight installation on a Linux machine for proof of concept. I will appreciate your help. Thanks!

Hi guys! Since we are already exporting prometheus metrics from controller, broker and server, it makes sense to provide the ability to install service monitors for it as well. Right now we are installing service monitors on our own. It would be nice if we can do this within the helm chart. We can have enable flag in helm chart so installation of service monitor can be enabled or disabled. Just like Argo does it here https://github.com/argoproj/argo-helm/blob/main/charts/argo-workflows/templates/controller/workflow-controller-servicemonitor.yaml There is one request opened in github in June as well https://github.com/apache/pinot/issues/10912 Is something already planned for it?

Hi, we are using StreamNative for our Pulsar clusters, and I have a PR to add OAuth2 support to Pinot so that we can connect and ingest from those clusters. It is a really small set of changes to add the 3 additional config params to setup the oauth2 Authentication and send it to the Pulsar client. I would appreciate a review of the PR. https://github.com/apache/pinot/pull/12068

Issue here: https://github.com/apache/pinot/issues/12067

Hey folks, do we have database driver implementation for Pinot for using database from Go? Like trino-go-client

Hi Pinot dev team, Is there anyone I can chat to regarding the Pinot Gitbook documentation and how it is hosted on Apache webservers? I’m trying to help another Apache project (Datasketches) do the same thing and would appreciate some help with this, thanks! @Charlie Dickens