is there a way to control skipUpsert through the connection string. trying to use superset to get history but it doesn't allow us to edit generated pinot queries to add the option.

Hi Is there any anomaly detection algorithms that can be used in pinot?

I had a question about enabling Upsert on a hybrid table. We currently have separate REALTIME-only and OFFLINE-only tables that we use to maintain two dashboards with different refresh rates. But we oftentimes find them diverging. To address this I’d like to set these up as a single hybrid table (with upsert on the REALTIME subtable). is that possible? I identify the following considerations: 1. We will call the tables with _OFFLINE and _REALTIME to avoid the broker providing duplicate results. 2. Will the offline table creation ignore the primaryKeyColumns in the schema? 3. Is there a mechanism to set common parameters in the tablespec, like indexes? Otherwise it looks like the star indexes, streamsconfig, routing and upsertConfig rules could easily be set only on the REALTIME segments. Will this work?

Hello Everyone, Thank you for being a part of Pinot community and congratulations on the graduation! Last year, we started with 100 members and today we are a 1600+ vibrant community.  There are awesome things happening beyond the slack channel and we didn't have a way to communicate important things to everyone in the community. We've created an Apache Pinot newsletter with Pinot releases and events ONLY! We recently added a check box to receive news when you join our slack and many of you have opted in already! However, we only added the opt-in recently, so members who joined earlier did not have the opportunity to sign up. We will be sending an email to everyone in the slack channel with the opportunity to opt in, or you can register here. If you don't, you won't receive any Apache Pinot newsletters in the future.  Congratulations again on the graduation, we can't wait to continue to grow our community!

Getting this while starting pinot/third eye for the first time org.h2.jdbc.JdbcSQLException: Database may be already in use: null. Possible solutions: close all other connection(s); use the server mode Anyone knows how to close the connections?

Hi. I have question regarding using DISTINCTCOUNTHLL 1. What's the deviation rate for this aggregation since it says "approximate distinct count" ? It seems on smaller data size, I don't see any difference from DISTINCTCOUNT 2. What should be the use case of it ? 3. If I want to use star-tree index, it seems DISTINCTCOUNTHLL is the closest thing to DISTINCTCOUNT. What could be issue from using DISTINCTCOUNTHLL with star-tree index ? Because I do care if the result is accurate or not

What is the recommended way to migrate a pinot cluster to a different s3 bucket?

hi!! I have a question about querying the database, We are working with time series and we want to define different kind of time windows in our queries. Are windowing options implemented out of the box or do we need to implement them by ourselves?

Hi Team, We are adding a TIMETAMP col in our schema and our use case is to populate it with the current timestamp value when the record is getting ingested to Pinot. How can we set this inside Pinot? Something like CURRENT_TIMESTAMP() what we have in other Dbs like postgres?

how I can see the query plan in pinot, I want to verify it only hits a certain set of segments based on specific time-range filter in my query

Hi Team, I am trying to create hour based segments in pinot but it's creating more than one folder into segments for the same hour, I guess this is due to some default row/data size, can I modify these default configurations and how what it preferable size of the data segment in pinot, what is the philosophy here too many files with a small size or minimum file with a decent size any reference on above

I had a general question about Upsert. Are the resource required expected to be “significantly” higher than a normal Realtime table? I ask because our Upsert table seems to take significantly more resources. Our upsert table is a considerably wider table, but I’d like to understand if it’s that width that’s contributing a bulk of that load, or if it could be Upsert itself.

One question, in the official java client (Not the JDBC one) is it possible to configure the basic auth?

Thanks to @User and @User for the great presentation - this was so informative!

Hi all I found pinot 0.8.0 has been released, Do we have released package for download? thx

Hi Team, I found a issue when using Partial mode Upsert. When getting a previous record before the merge, we always reuse the GenericRow object but not clean or modify _nullValueFields, see MutableSegmentImpl#getRecord. And PartialUpsertHandler will use the _nullValueFields which may expire to check column value is null. This will cause Partial Upsert to miss previous column data and not work.

An important question, can we use pinot for this usecase. I want to store users and attributes. The attributes will keep increasing. But user base can remain constant or incrementaly increase. The writes and reads are very high on this data store. I am interested to use query routing feature to replica groups for reads and support writes at high scale.

hi!! one quick question about ingestion transformation configuration, is this this syntax valid?

fromDateTime(jsonPathString(json.path, '$.timestamp, ''), 'EEE MMM dd HH:mm:ss ZZZ yyyy')

I mean, is it supported to nest transformation functions? I’m trying to convert a json string date-time into a Long timestamp

Hi All, Need we add index (like inverted index) for time column or time column already have index itself? thx

Hi All, anyone know what’s the buildQueryPlan_Count metric mean in pinot server?

Hi all is this still the right place for doc updates https://github.com/pinot-contrib/pinot-docs ? I’m checking as I can see inconsistencies between what is published and what’s in the source

How are people handling timeseries analytics with pinot? Looking for things like zero filing with last value (step interpolation), summing across interpolated series to get a total by time bucket, get last and first value in a time bucket, moving averages etc. Guess we can go over to python etc and implement but it would be nice for some native analytic functions.

@User where can we find the docker image for release-0.8.0 ? I can't seem to pull

apachepinot/pinot:release-0.8.0

Can the pinot-admin MergeSegments command be used on real time segments which are online. I have a lot of 10mb segments and am tweaking the segment flush threshold time. Else guess I can try hybrid table to take the older ones into offline tables.

This question is sort of off topic, but I was wondering if anyone here has details on the tools that the Pinot project uses to generate daily digests of this slack workspace?

Hey there, I have a question about the range index. Are Pinot allowed to have multiple range index?

Hi @User - Jackie suggested I ask you whether there’s a way to query the build version (and/or git hash), to confirm what’s actually running on a cluster & handling requests.

Hi Pinot team, what do you recommend to optimize a range search on a “date” column? Specifically, we have a column

created_at

which records the epoch time in seconds, we have a use case needs to search for entries created in the last 30 days, 60 days, 90 days, etc. I wonder if there is a best practice I could follow to optimize this, for example: 1. What type is best for this

created_at

column? Timestamp, Long, etc.? 2. What kind of indexing would help? range index? 3. anything else could help? Thanks in advance!

Hi Team, for our use cases, we were trying out Apache Pinot. Since the use cases involved nested queries we thought of integrating it with some connector like Presto. As per Pinot doc, looks like the recommended connector is PrestoDB. Can anyone confirm if PrestoSQL (Trino) is recommended or not? P.S: We are reaching out to other communities (Trino) also to check their view.

👋 Welcome to all the newest 🍷 Pinot community members! ❤️ 👋 Please tell us who you are and what brought you here. @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User