In case you missed yesterday's online meetup on Apache Pinot 2020 Year-in-Review and 2021 Roadmap, here it is:

Thanks to everyone who voted on the 2021 roadmap poll (50+ participants) ! We're keeping it open for 1 more week. You can still vote if you haven't here: https://poll.ly/#/P7J1obwA

📢 If anyone is using Kafka alongside Apache Pinot and wants to become a conference speaker: https://twitter.com/confluentinc/status/1336612838983471104?s=21

I have a boneheaded question

Let’s say I want to query a table based on a datetime column, and I want to aggregate all records that have a timestamp within the last 10 minutes

in a more traditional SQL DB, I would write something like “SELECT dim1, dim2, DISTINCTCOUNTHLL(dim3) FROM myTable WHERE eventTimestamp >= NOW() - INTERVAL 10 MINUTES”

That should work in Pinot as well

presumably it doesn’t like the interval subtraction, but I can’t find the correct syntax in the docs

Oh, sorry, I just read until now().

You will need to specify the actual interval value I think.

Hi team, I am using official docker image of Apache pinot referring to this doc https://github.com/apache/incubator-pinot/tree/master/docker/images/pinot My broker and server container is going down without any exception. can someone help me to look out this issue Error log at zk side is:

EndOfStreamException: Unable to read additional data from client sessionid 0x17685043a2e001d, likely client has closed socket

For realtime ingestion, does

tableIndexConfig.sortedColumn

actually sort the segments, or is it intended to tell Pinot what column the data is already sorted by when ingesting from Kafka?

Thanks, Kishore. I have another question - is the murmur hash used for broker-side partition pruning the same as the default murmur3 hash Kafka uses for partition assignment, or will I need to do anything special with my keys when producing to Kafka for partitioned real-time ingestion in Pinot?

I believe we use murmur2 hash

If you've ever wanted to get into the conference speaking circuit, now is your chance (if you're using Kafka 😏 ) 🎉*The Kafka Summit Europe 2021 CFP is open*! 🎉 The deadline is in 9 days. Go ahead and submit a talk. You have nothing to lose -- only to gain the possibility of being a thought leader in the innovations around real time analytics. 🍷 ❤️ If you need some feedback on your submission, I am happy to help. Also, the Kafka Summit people are available for real-time feedback on submissions in one of their slack channels. More details in link below. https://kafkasummiteurope2021-speakers.splashthat.com/logout

We also have a bunch of data-centric meetups groups globally that are looking for speakers. If you're interested in presenting in one of our future meetups (Pinot or others), please send me a DM 💃

We have a lot of new 🍷 community members in 2021! 👋 Welcome!!! We're curious to know what brought you here! 😃 Please introduce yourself in this thread! Also, if you have any technical questions, you can ask in #C01H1S9J5BJ or #C011C9JHN7R @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User

Welcome Pinot community! pinot I come from the Apache Superset community, invited by @User I’m a huge viz nerd but also have a stats / ML background. Happy to answer any Pinot <> Superset questions! I also spent the last ~5 years building Dataquest (online learning platform for learning data science) and am always open to discussing careers in data science!

Quick question - when i modify a table to add inverted index to the config and then do a reload, i can see that the segment has the inverted index added and affect on queries. But the reload does not work for star index? Is this expected?

it does work on star-tree as well, I think it needs a flag to turn on auto-reload for star-tree (@User, can you please confirm)

should this be made default true @User ?

not very intuitive that we enable a special flag just for star-tree?

Agree. We have it disabled by default to protect the servers, but I think most users want it enabled

BTW - on another topic related to reload, i saw that there is a proposal to show the status of reload - https://github.com/apache/incubator-pinot/issues/5390. Just wanted to say that i think that will be super useful. Right now i did not find a way to know whether the segments were reloaded or not. Might want to have some sort of desired vs realized state API for same.

Hi team, does this config

controller.offline.segment.interval.checker.frequencyInSeconds

control when added/updated offline segments are actually used? Is there a cron schedule to control when new segments take effect?

@here

Hi Pinot Folks, I am coming from Apache Helix project. We would like to hold an Apache Helix meet up. Would like to invite one presenter to talk about Apache Pinot how to use Helix. Anyone interest to give a talk about that?

This is starting in a minute. @User and Tim Berglund (Confluent) will be presenting "Using Kafka and Pinot for Real-Time, User-Facing Analytics" https://www.meetup.com/Data-Science-DC/events/275683778/

That was a great presentation, thanks to everyone for organizing and giving it - I’m excited for @User’s into intro next week.