Is it possible to batch ingest via Apache Flink to the HDFS as deep store in Apache Pinot?

Hi Group, We want to compute rolling window percentile on top of group by results. Is there an option to expose serialized MergingDigest which is used by PercentileTDigest, similar to DistinctCountRawHLL ?

If I am not mistaken, Pinot's integration with kafka schema registry is only for avro decoder? Can pinot read table schemas directly from Kafka schema registry?

Hi @User! I'm a Data Engineer and interested in real-time data processing, analytics, and anomaly detection. Some the nice folks from the Kafka community recommended StarTree/Pinot. It's great to such great performance, easy Kafka integration, and usability. Can't wait to get started and learn more about Pinot!

Hi everyone, I am an engineer working on Presto development. I was interested when reading the tutorial using Pinot and Presto posted in medium. I am very exited about data analytics in general so definitely want to learn more about Pinot!

Howdy! I’m a senior data scientist at Atlassian. Formerly worked at LinkedIn where I used Pinot to deliver metrics to downstream apps. I’m here for the community and to learn from you all how you use pinot in your applications.

Debugging batch ingestion from S3 🧵 The job got submitted correctly to my cluster, but I suspect my input file format is to blame for the failure

java.lang.IllegalStateException: Cannot flatten value node: null

@User gave a very interesting Kafka Summit APAC talk recently entitled "Real-time Analytics with Upsert Using Apache Kafka and Apache Pinot"

Is apache pinot support connection pool mechanism, if we use JDBC client.

Hi everyone! I am an engineer working on implementing Pinot + Presto to modernise our current stack for realtime analytics. Our current stack is not well suited for realtime analytics, and this is when we came across Pinot. I’ve been looking into the stack for a while now, ran various performance tests and it looks really promising so far!

if i am reading correctly, when

segmentPushType

set to

REFRESH

, no segment build will be triggered? Once a segment is flushed, it will be dropped? However, in my test, it seems segments still get built

Hello folks. We are evaluating deploying Pinot. Do most folks here deploy Pinot on Kubernetes or directly on EC2 (or equivalent) nodes?

Hi All, Do we have some tools to migrate data from ClickHouse to Pinot? thx

I've heard the Pinot is for "User-Facing, Real-Time Analytics" and was wondering who the "User" is in this context? Is it referring to external end-users? If so, is Pinot overkill for internal analytics?

Hi, we are ingesting a timestamp data into a offline Pinot offline table as a long data type and 1millisecondsepoch date time specs. But for all null records Pinot ingested as min long value. How should we define the schema so that null timestamp can be also ingested as a null value in Pinot?

Hi, does anyone know why I might be getting a

java.net.BindException: Address already in use

when I use

bin/pinot-admin.sh

? For context I'm using Kubernetes and running the command from the Controller Pod

hello, i’m trying to figure out how pinot can support the queries i’ll like to make. my schema has the following fields:

userId | time (15 minutes) | cell
a        0                      1
a        1                      2
a        2                      -1
a        3                      -1
a        4                      -1
a        5                      3
a        6                      3
..
z        0                      1

i’ll like to interpolate the missing values (-1), then query the result of that transformation. my understanding is that pinot doesn’t support subqueries but does so in conjunction with presto, is this the right channel to ask?

# example of values after interpolation
userId | time (15 minutes) | cell
a        0                      1
a        1                      2
a        2                      2
a        3                      2 or 3
a        4                      3
a        5                      3
a        6                      3

# example of values after forwardFill
userId | time (15 minutes) | cell
a        0                      1
a        1                      2
a        2                      2
a        3                      2
a        4                      2
a        5                      3
a        6                      3

Hi @User and everyone 👋, I am a data engineer at expediagroup and currently POC ing Pinot for one of our use case. Performance is very promising but we struggled a bit with setup, one of the reason was low kubernetes support in our environment. Another hurdle we encountered was lack of subquery/join support, guess we might have to use it in conjunction with presto for that. Still trialling.

Hello, does Pinot support initDB scripts when deploying in kubernetes? Akin to cassandra: https://artifacthub.io/packages/helm/bitnami/cassandra#cassandra-parameters

Hey all!!!! 🎉 TODAY IS A REALLY BIG DAY!!! 🎉 Pinot has officially graduated as a top-level Apache Software Foundation project! 🎓 🍷 🥳 One of the co-authors of the Pinot project (and now, founder of StarTree.ai) @User had published a blog about our journey here (URL below) 📢 Please help us spread the word on this monumental day by sharing this news!!!! 🔊 Big shoutout to all the contributors who helped bring this project to this place! Congrats to you all!!! ❤️ @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User https://www.startree.ai/blogs/apache-pinots-graduation-a-celebration-of-community-and-a-look-ahead/

Congrats!!!

We have a offline table field truncating texts as it exceed the default 512 length. Is it possible to change the default maxlength without having to reingest all the records to Pinot?

Great news on Pinot being selected as Apache TL project

I unfortunately had to miss the meetup today. Is it going to be made available on youtube/etc at some point?

With respect to complex data types[1] (not yet available, not until 0.8 AFAIU), what would happen in the given example if there were an additional complex data type in the input object. Say that the example input object also had a

friends

array (arbitrary example) at the same level as

group

. I noted that the docs say this about the unnesting example (emphasis mine):

For instance, the example in the beginning will transform into two rows with this configuration example.

Are these then distinct rows in the table, or some kind of special nested rows associated with a top-level row? And if there were another nested/complex field, would that result in a combinatorial explosion in terms of number of rows? A related question: how does this approach work with conjunctions? In the example given where the topics are

paddling

and

hiking

, can a query be written to answer the question “which meetup events had talks about both paddling and hiking?” [1] https://docs.pinot.apache.org/basics/data-import/complex-type#handle-the-complex-type-with-ingestion-configurations

Hi All, We are evaluating Apache Pinot around the security posture and wanted some info around the same. For encrypting Personally Identifiable Information (PII) data at rest, we need to handle this at two places. 1. Completed segments sitting on Pinot servers. 2. Segments sitting in Deep store say HDFS (looks like Pinot supports this) Is there any support from Pinot for segments sitting on the server's disk? On the Github issue, we see https://github.com/apache/pinot/issues/5762 that talks about this and is still open.

Hi all - I am really interested in Pinot and tooling like it which is enabling simpler development for building customer-facing analytical applications, especially cases where you are moving beyond a dashboard and want to really embed data into the application workflow. I have spent a few months surveying a lot of the different tools and approaches that are coming together to make this easier, including Pinot. I wrote up some of what I have seen here, in case it is of interest to anyone: https://towardsdatascience.com/the-analytical-application-stack-eead8ce6b70

Hi guys, I am looking for a way to delete a record in a pinot table. I am aware that Pinot doesn't support this type of operation, but in our use case we have deletes once in a while so I am wondering if there is a Hack on how to delete a record without rebuilding the whole Table ?

Hi all can we add more than 500 columns in one table for some special scenarios ?

Hi All do we have some example to extract attributes from json objects in transformFunction