Hi team, one question about highly available pinot setup. Assuming we deploy one pinot cluster to 3 availability zone, and we have table that has 3 replicas per segment, is there anyway to enforce pinot distribute the table segment replicas to 3 servers in different availability zone? Want to see if there are some example to achieve this with server tagging or something.

hello team, I'm trying to enable logging for all API calls in controller, currently there're entities like below(from

org.apache.pinot.controller.api.resources.ControllerResponseFilter.java

?)

Handled request from 127.0.0.1 PUT http://localhost:9000/tables/test_table, content-type application/json; charset=UTF-8 status code 400 Bad Request

Is there a way to log the 'application/json' content? Besides adding a line in

PinotTableRestletResource.updateTableConfig

to log before/after schema

hi. team. I want to monitor query response time. However, the pinot_broker_queryTotalTimeMs_* metrics seem to expose continuous cumulative values since the broker started up. Is there any metric to get the average response time of queries over a period of time (eg 1 minute)?

HI Team, Can we query data from particular segment? @saurabh dubey

Hi Team, can we use multiple kafka topics for one realtime table? @saurabh dubey

https://www.meetup.com/real-time-analytics-london/events/291332514/ Starting in 15 min! Join us

Hi, Is it possible to use Apache Pinot to build CDP like Segment or rudderstack?

Hi #there! What is the best approach to enable multi-stage query engine when using docker-compose?

Hey 👋 what’s the easiest way to load the

Hive partition columns

as

Pinot data columns

Hi, quick question why segments naming are differents when pushed via JobSpec and via API (controller) ? Even if the 2 files are archives JobSpec -->

segmentName.tar.gz

controller -->

segmentName

Hi all, this is my first project and we are looking at

ingestion aggregation

feature. I know it is a new feature. I read the doc and example https://docs.pinot.apache.org/developers/advanced/ingestion-level-aggregations but it is hard to find examples from other places. In the example, I dont see any primary keys in the schema, then how can Pinot know which fields to group by?

Hi team, Is it possible to write a spark dataframe into pinot directly?

I was exploring the text search index (text_match) : https://docs.pinot.apache.org/basics/indexing/text-search-support#text-parsing-and-tokenization As part of this, Is there a way to use a custom tokenizer or a way to configure it? As an example, if I have a text

<http://abc.pqr.xyz|abc.pqr.xyz>

and would like to use

.

as also my split term token along with

space

,

tab

. So, in this example, I have three terms -

abc

,

pqr

and

xyz

Hi everyone! March is here, and so as our monthly update, fasten your seatbelt, here we go! 💺🚀 🦸‍♀️ Major News • Real-Time Analytics Summit is coming this April! - watch to learn why you should register and join us in San Francisco, April 24-26 📖To Read • StarTree ThirdEye: Preventing Fraudulent Transactions in E-Commerce with Anomaly Detection by Madhumita Mantri, Suvodeep Pyne, Tushar Thole • Next Generation Data Architecture for Advanced Analytics by Sandeep Dabade ▶️To Watch • • • • 🗓️March Events • Virtual Meetup on March 16 - Real-Time Analytics Using Apache Pinot • Virtual Meetup on March 22 - Real-Time Anomaly Detection for Seamless Ride-Share Experience • Virtual Meetup on March 23 - Apache Pinot™ Roadmap 2023 meetup You can always view the full newsletter on LinkedIn here Have a lovely March, and see you at the upcoming community events! 💥 *I’ll start posting this monthly post on #C01444V4DQC so please be sure to join the channel ❤️ thx!

Hey folks, is there currently any transform (or other operator) that can be used to “fan-out” from 1 doc/row to many at query time? Effectively, query-time multi-value generation. An example use case would be user sessions which span multiple intervals of time, where the desired query outcome would be to compute average concurrent user sessions per time interval. Can this be computed if the rows contain session start time, end time, and duration, where a user session could exist in many time intervals due to its duration?

Hey folks, we are upgrading our clusters from 0.10 -> 0.11. This required upgrading Helix version from 0.9.8 -> 1.0.4. But after upgrading, we saw all Helix related metrics are not getting pushed anymore. Any idea if there is any specific change we need to keep in mind to get back the metrics.

Hey Team, I wanted to check once if I can use Amazon MSK to write directly to pinot? And we also need to do a cost comparison of Pinot. Do you have any specific documents of Pinot that I can refer too?

hey folks! We’ve created a poll like the previous two years, listing some features we think would be interesting to add to Apache Pinot this year. We’d love for you to vote on features you think are interesting, and also add suggestions of other things you’d like to see 🗳️ Here’s the link: https://poll.ly/mHPulIEPyRKzjsy9SHXA We’ll take these inputs and present the 2023 roadmap on March 23rd in this community meetup: https://www.meetup.com/apache-pinot/events/291954166/?isFirstPublish=true. So don’t forget to register if you’re interested to see where this goes 🙂 This vote will stay open all week. Please cast your votes soon and help us shape the direction of Apache Pinot 🍷

just fixed the link (Thanks @francoisa for the catch) Please re-vote if you already did so in the last 30 mins 🙈

Hi all, this might not be the correct channel to discuss this, in which case, please point me in the right direction. I was just wondering if anyone knows why the

traceInfo

value returned when querying Pinot via the ReST API is an escaped string of JSON, as opposed to an actual JSON value?

Is there a reason for this?

hellos, can someone share a sample query/blog on how to join 2 Pinot tables in Presto/Trino. Thanks ?

Hello, I added a new column to my pinot schema and now i am. getting this -

There are 8566 invalid segment/s. This usually means that they were created with an older schema. Please reload the table in order to refresh these segments to the new schema.

have tried reloading segments as well as rebalancing servers still getting the same warning messaghe.

Does pinot allows the users claim a query id together with the query? More general question is how to set up query context like Druid did? I cannot find the relevant doc. It will be convenient for tracing among different systems.

Hi, Is it possible to use SASL_IAM to connect realtime tables with AWS MSK? https://github.com/aws/aws-msk-iam-auth

Hi, can we customise segment creation on the basis of incoming data? So right now I have different partitions for different brand Ids but the number of partitions have gone to 1000 in a topic and will keep increasing and I think there is an upper cap in kafka for partitions per broker. Instead of having a partition and segment one to one mapping, if I can configure somehow to send data to different segments based on brand id then it would be great

Hellos, Is it possible to configure a Pinot Table to read from Multiple Kafka Topics if the underlying schema is same. Thanks !

Hi, do range indices not work with metric columns? I have two copies of the same table. One has a column as a metric-column and the other has it as a dim-column. Both have range indices on the column in question, and the table with the metric column does:

FILTER_FULL_SCAN(operator:RANGE,predicate:colname < '10')

Whereas the table with the dimension column uses the range index like expected:

FILTER_RANGE_INDEX(indexLookUp:range_index,operator:RANGE,predicate:col < '10')

Hello Team Can Anyone give inputs on clickstream data Datamodel designs in pinot

Hello, I noticed Apache Pinot container images have a target CPU instruction set encoded in image versions. Why is that? I usually see multi-platform images being used. Is there anything special about Pinot images? Or it’s just the way the Apache release infrastructure is set?