Hi, Any thoughts on this OLAP benchmark. Not sure how reliable/verified this benchmark is. https://benchmark.clickhouse.com/ Pinot is in the middle of the pack

Hi team, how to 1) limit user access to pinot rest apis? Only selected userid can query via api. 2) setup api token by userid - users who can access api but have dedicated api token. 3) (optional) acl at column / row level?

Hi Team, Is it possible to recover deleted realtime segments back into REALTIME table? Context: We had a REALTIME table DRIVER_METRIC, we accidentally deleted the table from pinot console which used 7d default retention of segments. So we can still see the deleted segments in S3 but table is not available for query anymore. Now, we have created DRIVER_METRIC REALTIME table again which is available for query and consuming data from Kafka but it only has last 1day of data (kafka topic retention). Now, we want to push old data into this table from deleted segments folder in S3. Is it possible? If possible how? cc: @Lee Wei Hern Jason

Apache Pinot Roadmap 2023 meetup | March 23 For the Community- by the Community 🍷 • In this meetup we’ll be featuring the Apache Pinot roadmap for 2023 ; • Get to hear from Linkedin, Uber, StarTree and more, what they have in store this year for Pinot. • Explore what other community members are working on and Hear what the community wants to see in Pinot. RSVP here >> https://www.meetup.com/apache-pinot/events/291954166/?isFirstPublish=true

Quick question: How is`EXPLAIN PLAN FOR` generated for Hybrid table? My testing result is that

EXPLAIN PLAN FOR

for hybrid table is same as realtime table which is different from offline table. Is it expected?

Hi community, I am trying to use the lucene text index for a hybrid table setup. I was able to set it up for realtime table without much effort. For offline table part, we rely on a spark job to generate and upload the segment URIs. Do I need to create segments with text index and then upload or does Pinot create the text indexes automatically after the segments without indexes are uploaded?

Hi community, Is there any way to specify table and column description in the schema json? We ideally want to store some table metadata right within the schema and not use an external solution for it.

Hellos, Are there docs available on how to make Udf's work with helm setup.

Hi Community, I am exploring the RealtimeToOffline Minion Task. I wanted to confirm if the upload of converted offline segments and update of

watermark metadata

is atomic? I do see the segment replacement protocol but I think it doesn't come into picture for RealtimeToOfflineTask but for Merge task.

can we download segments and put data into csv? @saurabh dubey

Hi Team, what's the difference between

LaunchSparkDataIngestionJobCommand

&

LaunchDataIngestionJobCommand

? When using batch ingestion job (https://docs.pinot.apache.org/basics/data-import/batch-ingestion/spark) which one should be the main class?

I have a Kafka topic with AVRO encoding. The time column is of type (long). The logical type is (timestamp-micros). Any way to convert it to milliseconds and defined a datetime field spec like the following (without having to create a new column):

{
  "name": "event_time_ms",
  "dataType": "TIMESTAMP",
  "format": "1:MILLISECONDS:TIMESTAMP",
  "granularity": "1:MILLISECONDS"
}

Hi Guys where is the best place to ask about Pinot SQL querying engine? and how it behaves specifically using a very long array expression for the

IN

operator?

Hi can anyone make and share me a complete doc for how to start pinot with docker with hdfs setup ?

Hey, Recently we saw very high disk usage for some of our hosts. On investigating, we found in our servers, directories something like this for a table:

_tmp/tmp-<segment_name>-<timestamp>/tmp-<uuid>

The segment name in this path^ does not exist anymore for that table (deleted by retention). The contents of the directory are of this manner:

0	col1.sv.sorted.fwd
0	col2.mv.fwd
0	col3.sv.sorted.fwd
0	col4.sv.sorted.fwd
0	col5.sv.sorted.fwd
0	col6.sv.sorted.fwd
4.0K	col1.dict
4.0K	col2.dict
4.0K	col3.dict
26G	    col4.dict
132G	col5.dict
148G	col6.dict

Any idea what this

_tmp

folder signifies and why are they getting created?

is there a recommended way to apply rollups to an offline data import that's come in with

SegmentGenerationAndPushTask

? do I need to trigger a

MergeRollupTask

and hope it does it?

hellos, Any ideas on how folks are providing

ssl.truststore.location

as mentioned here part of the pinot Deployment using helm. Is it local

jks

file being packaged as part of the docker or being added post cluster deployment. Any ideas/best practices around this ? Thanks !

ls there any way to backfill segments in a realtime only table?

Hello team, I'm looking for some suggestions on how to setup graceful shutdown properly, on brokers and servers, similar as how trino/presto works. Out Pinot cluster is deployed in K8S, to avoid downtime when a pod is restarted, • a server node need to commit any consuming segments, and mark as inactive to avoid new queries coming; • a broker is marked as inactive to avoid new queries, and wait until active queries are finished. cc @Grace Lu

If you haven’t seen the silly parody videos my team has been making, today is your day: https://www.linkedin.com/posts/startreedata_pinot-s-kafka-hes-a-friend-from-work-activity-7043973647237042177-tzKb/

All of this madness is remind you of rtasummit.com. Go there and check out the details, look at the program, and register. There’s a lot of Pinot content, and I’d love to see this community there in force.

PM me if you want a discount code. 💥

Hi team, want to consult about one of our high cardinality use case and see how to set it up properly with pinot or whether it is a proper use case for pinot. We have a metrics table that contains hundreds of daily level metrics columns that associated with uuids, the data updates daily to add more than 50 millions unique rows every day (add one row for each uuid everyday, and there is millions uuids). A simplified table schema looks like:

uuid    date    group    metrics_1,    metrics_2.     … metrics_xxxx

And a typical simplified query we want to run on this table is selecting a bunch of metrics aggregation for certain groups of uuids across days and then aggregate them again by group, eg:

select 
   group,
   avg(m1),
   sum(m2),
   ...
   avg(mxxx)
from 
(
    select 
        uuid,
        group,
        avg(metrics_1) as m1,
        sum(metrics_2) as m2,
        …
        avg(metrics_xxx) as mxxx
    from metrics_table where group in (xxx) and date between aa and bb
    group by 1, 2
) group by 1

When we did preliminary testing previously, we ran into issues of simple aggregation query on uuid takes very long to return, or query return inaccurate approximations due to high cardinality, we want to get some suggestions about whether it is a good use case with pinot, and if it is how to model this with proper cluster config and index config, thank you! cc @Mingmin Xu

Hi, 1. what's the difference between building pinot-0.12.0 from source code with

-Pbuild-shaded-jar

and without it? 2. Is it possible to shade

org.apache.hadoop

being used in main pom.xml in pinot-0.12.0, seems like it's using different version then hadoop being used in our team's cluster. I believe, if we can shade it and build pinot from source code, then it should be fine.

Join us TOMORROW- Apache Pinot Roadmap 2023 meetup 💥 🍷 In this meetup we’ll be featuring the Apache Pinot roadmap for 2023 ; Get to hear from Linkedin, Uber, StarTree and more, what they have in store this year for Pinot. Explore what other community members are working on and Hear what the community wants to see in Pinot. Meet, chat, and deepen your knowledge in Real-Time Analytics with Pinot. RSVP here: https://www.meetup.com/apache-pinot/events/291954166/?isFirstPublish=true

https://www.linkedin.com/posts/startreedata_pinot-s-kafka-hes-a-friend-from-work-acti[…]647237042177-tzKb?utm_source=share&utm_medium=member_desktop Just making sure you all saw the latest release… 🤣 Lord of Pinot is in the house @Tim Berglund

So, way less fun than a Thor remix - https://www.thenile.dev/blog/things-dbs-dont-do. Interesting input for the Pinot roadmap…

Speaking of less interesting than a Thor parody, I thought I'd point y'all to my first blog post for StarTree... Enjoy!

I also wrote a thing for DZone on Pinot and IoT, if you're at all interested. 🙂 https://dzone.com/articles/real-time-analytics-for-iot

Time for the Apache Pinot Roadmap Meetup! https://www.meetup.com/apache-pinot/events/291954166/