Hi team, I have a question regarding

tableIndexConfig

for

dimension

tables. 1. If we don’t specify the

loadMode

in

tableIndexConfig

, is

HEAP

the default load mode? 2. If we specify the

loadMode

in

tableIndexConfig

, will it overwrite the default load mode?

"tableIndexConfig": {
    "invertedIndexColumns": [],
    "loadMode": "MMAP"
  },

Hi, How can i provide zookeeper configuration (zoo.cfg) to

bin/pinot-admin.sh

script while starting the zookeeper process? i don’t see any command options here - https://github.com/apache/pinot/blob/master/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/StartZookeeperCommand.java

hi guys, incase of a cluster break down, is there a way to restore and map the existing segments i have deep stored in s3 to the new cluster? any docs available on this?

@Chinmay Soman and @Neha Pawar published a magnum opus of a blog post this morning. I strongly commend it to your reading. 🙂 https://startree.ai/blog/a-tale-of-three-real-time-olap-databases

In fact, this seems like a great place for discussion. A technical comparison of competitive technologies (written by engineers, not product marketers or even my own DevRel people) is an incredibly useful thing, but only if it’s fair and accurate. Slack is a great place to discuss things like this.

Hello Team, We are going to set up Pinot multinode cluster soon. What should be the Number of Controllers, Brokers, and Servers in the case of 3 and 5 node clusters. Do we have any general guidelines? Also, we are planning to set up cluster using Pinot 0.12.0 binaries is fine or only k8s cluster is recommended? Note: Data size will be 1-10 GB's as we are setting this for Pinot cluster setup understanding and some basic performance testing.

Hello, I am new with pinot, I try to model my data into few meta data tables, and one major table. One of my metadata table should be large, maybe millions of records.. I read that there is a problem with the join of large table(more than 200MB), I'm lookin for ideas how to handle it, or if there are another solutions

hello team, I am wondering how I can transition my prod cluster to use deep storage. currently, i have a prod pinot cluster running on aws eks and i want to transition it to use s3 backed deep storage without losing data from current cluster. is that possible? what are the options?

#RTASummit- is ON⚡ StarTrees’s CEO and Co-Founder, @Kishore G, just took the stage, kicking off the first day of presentations. In his keynote, he will discuss the rise of Real-Time Analytics. Tune in live to watch the keynotes at https://rtasummit.com/

IMG_2520.png

Hi Team, do we have any puppet automation available for pinot deployment?

Hi team, I am new in Pinot. I have a question about modeling tables. I have one real-time table and one dimension table, the dimension table is larger than 200MB.. Is there a solution/best practice how to handle lookup\joins with dimension tables that are larger than 200MB? thanks 🙏

Hey team 👋 We are currently using time & partition based segment pruning in Pinot (murmur function & one partitioning column). Just wanted to check if partition pruning can be done on more than one column. Appreciate any inputs! cc: @Sai Tarun Tadakamalla

Hi Team, Wanted to know if pinot support any zookeeper authentication mechanism like sasl or plain digest

Heyyy everyone! 🌻 I’m excited to share May events, and to have 2 in-person ones! check them out>> • Intro to Pinot with St Louis Data group- Online event with @Barkha Herman May 3rd 🔴 • Confluent x StarTree at Optum meetup with @Lakshmi Rao @Will Gan @Tim Berglund May 9th- IN-PERSON 🎤 • Stirpe and DoorDash x StarTree in SFO, with @Tim Berglund May 10th _IN-PERSON_: Register in the form <a must> 🎤 • Streaming with Kafka workshop : moving forward with the 2/5 workshop with @Barkha Herman Online event 🔴

My goodness, that’s a lot of Tim Berglund for one month.

Hi, G'day everyone. Do we have documentation for upgrading pinot in kubernetes? Can anyone share it please.

Hi I wanted to increase parallelism (I am playing with

segmentCreationJobParallelism

and

pushJobSpec/pushParallelism

) in my batch ingestion, but even though my machine has 32 cores, speedup above parallelism of 2 is not significant. This log message gives me a hint that my configuration is not entirely correct

Creating an executor service with 2 threads (Job parallelism: 4, available cores: 2.)

How to tell the executor in the controller it has more cores available?

Hi is there an endpoint where I can get the table state like shown in the UI

May Newsletter is OUT! Here are the latest news: 📢 Major News • Real-Time Analytics Podcast with Tim Berglund - check out the first six episodes on YouTube or your favorite podcast player. • Real-Time Analytics Summit is in the books - check out the keynotes and panel at rtasummit.com - sessions will be added in the next few weeks! 📖 To Read • A Tale of Three Real-Time OLAP Databases: Apache Pinot, Apache Druid, and ClickHouse by Neha Pawar and Chinmay Soman • Introduction to Apache Pinot™: A Beginner's Guide by By Barkha Herman • How to Stream Sensor Data to Apache Pinot for Real-Time Analysis by David G. Simmons • What's Next in the Apache Pinot™ Roadmap for 2023? by Mark Needham ▶️ To Watch • • • • 🗓️ Events • San Francisco - TONIGHT! - Real-Time Analytics with Apache Pinot Meetup with Stripe &amp; DoorDash • Virtual Meetup on May 18 - Stream Ingestion with Apache Pinot [Hands-on Workshop] • Chicago - May 23 - GOTO Chicago 2023 - Building a Real-Time Analytics Database: A 'Choose Your Own Adventure' Journey w/ Tim Berglund • Krakow - May 31-June 2 - Devoxx Poland - Pinot, why are you so fast? w/ Tim Berglund View the full newsletter on LinkedIn here! In case you’re not signed up for our Newsletter send me your email and you’ll be added 💛

Hi Team, We are working in upsert and nullHandlingEnabled parameter by to update null value with new value it is working perfectly with string datatype but how can we do same for integer value because we can not push null value to integer, we have to push some fix value like -1 but it will not handle by nullHandlingEnabled. is there anyway around this to fix it?

hey all, wanna share a talk that we gave in March at Data Council on Uber's journey migrating from Elasticsearch to Pinot for our business metrics platform for better reliability and scalability. Now the video is available .

Hi Team, I had few questions around data quality regarding 2 aspects and wanted to get suggestions or how others are doing it: Pinot related issues: How can we guarantee Pinot actioned/persisted all the rows that were provided during ingestion. For offline tables it is easier since the complete segment gets pushed and we can check for segments presence in the table. But for realtime, how do we guarantee data completeness wrt upstream? For non Pinot related issues: We have third eye for anomaly detection and other anomaly detection tools can be integrated but can we do a reconciliation between some source truth data and Pinot table without putting too much query load on Pinot table.

Hi everyone, We are running Pinot with 3 servers on separate nodes and each realtime table segment has 3 replicas. Each night we shut down the VMs in a dev cluster to look for stability issues. This sometimes causes segments to bad. In the docs the instructions are to reset the erroneous segments for a table using the API. This seems to work. Is there a job/task that does this automatically that we have failed to configure or do we need to handle it "manually" through the API?

From Yoav:Due to popular demand I will be giving a lightning talk on "Anomaly Detection with Apache Pinot and Thirdeye" at a meetup organized by @ZipRecruiter . This is next week on Wednesday, May 17 at 9:30 AM. Come check it out! https://eventbrite.com/e/data-lightning-talks-tickets-634671348087… @startreedata @ApachePinot

Does Pinot support Jaeger distributed tracing or other flavours of one? Don't see anything in the code apart from

Tracer

interface.

I'm attempting to run the free trial through the AWS Marketplace. After completing the

Set up your account

step, it's not clear what I should do next to launch Pinot.

Hi, We want to connect to AWS MSK from pinot using IAM authentication. For this to work, this library must be added to classpath. Is there an easy way to do so? https://github.com/aws/aws-msk-iam-auth

Is there something like pg_trgm https://www.postgresql.org/docs/current/pgtrgm.html in Pinot?

Hi Team. https://docs.pinot.apache.org/operators/tutorials/authentication-authorization-and-acls - controller.segment.fetcher.auth.token - pinot.server.segment.fetcher.auth.token - pinot.server.segment.uploader.auth.token - pinot.server.instance.auth.token - segment.fetcher.auth.token - task.auth.token What are these settings that this article is talking about? When I check Pino's source code, these settings are not there. What processing of Pinot do these settings affect? In my tests, these settings seem to be independent of table and schema creation and deletion and segment ingestion.