Hi Team, while using Kafka with pinot , below error I am getting.

Happy Diwali to those who celebrate! ❤️

Hi team, I have a quick question on Upsert tables with default

BalanceNumSegmentAssignmentStrategy

migrating to

ReplicaGroupSegmentAssignmentStrategy

. Since all segments within a partition will only locate on the same server for Upsert table, does that mean if I change to

ReplicaGroupSegmentAssignmentStrategy

, I will need to run a rebalance first to move COMPLETED and CONSUMING segments to the corresponding servers, so that Upsert functionality is not broken? @Jackie @Qiaochu Liu

@Si Lao You can simply ignore the

segmentAssignmentStrategy

field. For upsert table (real-time) table, segments are automatically assigned with replica-group

Hello, can anyone suggest how to query string that has apostrophe in it like

McDonald's - Nadd Al Hamar

?

Going to re:Invent? Come say hi! @Kishore G, @Tim Berglund , @Vibhuti Bhushan, @Sandeep Dabade, and other experts Will present at booth #M6B to answer all your questions about Apache Pinot, StarTree, and real-time analytics!

Hi Team, Why my timestamp coln has different value when I query from pinot query

Hi Team, is there any way to get explode function lity in pinot . Consider if I have source data like this -

Hey folks 👋🏼 we're planning on upgrading from

0.11.0

to

1.0.0

. Are there any concerns around doing that directly?

Hi, We have three pinot servers. The replication factor of our table is 3. When upgrading servers on a rolling basis, one server goes down but the remaining two are up all the time. But our queries start throwing error

unable to resolve host pinot-server-1.pinot-server-headless.pinot.svc.cluster.local

What is the purpose of replication if it can’t handle queries if one server goes down? Or are we doing something wrong?

Hey all 👋 I have a question related to the consistent data push mechanisms. I’ll first explain our setup: We have a hybrid table 1. Realtime table that ingests from Kafka 2. Offline table that is pushed to from standalone Hadoop jobs. The data is batched by day so we generate segments like

{tableName}_{date}_{partition}_{index}

. So data is appended. We sometimes (rarely) want to refresh data in the offline table and consistent data push sounds like the solution but according to the documentation it only seems to work for REFRESH tables. Is that the case or does it also work for APPEND tables?

Nominations for StarTree All Stars 2024 are now open! ⭐ Submissions will be closed end of November, tag someone who needs to know! Apply now and check the class of 2023 ✨

Hi Team, how can bring the address1.type and addrss2.type to address_type in pinot table. So will be having 2 record .

Hello 🙂 Is it possible to ingest real data from the pub/sub? So, is pinot support the pub/sub? Also is there any documentation about use of Apache Superset with Apache Pinot?

Hi, I have a column containing json objects with arbitrary schema For example in raw data, for some rows, column X would be { a: 1, b : 4} and for other rows column X would be like { c: 9} If I set JSON as the type of column X, all values in column X would be like { a: 1, b:4, c: null}. Each value in column X would contain all the keys. Is JSON the preferred data type for column X, or should we store column X as string and convert string to json within SQL? And how to convert string json to json format so I can use json_extract_scalar function? Thanks

Happy Monday ya’ll ! Check out the Real-Time Analytics Horror Stories and How to Fight Back blog and meetup recording ! with @User @Minesh Patel Here>

You have a chance to get a FREE TICKET TO #RTASummit 💥 The raffle will be held end of month! Check how>

Is it possible to add a custom authentication method in pinot? (like there are plugins to extend stuff). Or something like this on roadmap? I was looking to get certificate/trusted-ca auth in pinot for controller, broker and even for intra cluster interactions (like broker->server).

Hi Team, asper the digram

Hi Team, we looking for details on whether there is a way to track record/row level metadata in Pinot - say, when the record was ingested or when the original kafka message for the record was created. For the second part we are trying to write a custom avro decoder that we can use to capture this data. But not sure if we can access the message level metadata from kafka as we only have access to the payload in the decoder.

Hi! I'm planning to use Pinot with a single real-time upsert table that will have around 8b rows and almost all of them will be updated daily with upserts every 30 days. No offline table here as I need to be able to upsert any record at any time, is this an intended usecase for Pinot? I'm asking because the docs always mention a low retention for real-time table, then move segments to offline table (and segment storage). Are there any issues with using Pinot this way? Thank you!

StarTree All Stars Class of 2024- Apply Today Here>

Hello everybody, does anyone know the ETA for the next release? thanks 🙂

Hi everyone! TL:DR : Fill out the Community survey and get a chance to win 50 Swag coins! 🕶️ ---- Our community has grown immensely over the past year, and it’s truly exciting to be part of this spectacular group of humans! 🥲 As we wrap up the year, I kindly ask for your participation in a short survey. Your insights will help us understand what you like, what you need, what works well, and what doesn’t, all to make this a valuable place for all of us to rise and shine! Since the holidays are just around the corner (and we genuinely want your feedback! 😉🎅), we’ve decided that 5 lucky participants will strike gold in our December 7th raffle, each winning 50 swag coins! We’ll announce the winners here on #general and via DM

I think the rebalance API does not work as expected for a very large table(34TB). I ran into a dead host situation, where I removed the tenant tag from it and disabled it. I then tried to rebalance the servers with reassign instances flag. The response does not list the dead instance but the Ideal state is not removing the dead instance in the segment assignment config. What are my options to rebalance the table? I do not want to try the rebalance with downtime as I suspect there will be missing deep store copy for some segments.

Hello 👋🏻, I am new to using Apache Pinot, in conjunction with Debezium and Kafka. I have a question, is there a way to store the data that Pinot returns? That is, in json format or a table in my database. Please if anyone can help me I would appreciate it

dancingcharmander Olaaaa everyone! I just wanted to thank all of you who filled out the community survey. 🖊️ We received lots of thorough and super-helpful responses. If you’d still like to fill it out, today is the last day to do so before we close it. 5 out of the repliers will win 50$ for swag! winners will be announced on Thursday, stay tuned! 💰

[Help]: I have a task where I want to rewrite offline segments. I know we can do it using Minion. Is it possible to achieve this using Spark?

Congrats to the 5 winners of the EOY community survey 🤝 You just received an email with your 50$ to our swag store⭐

Hi i want to ingest data from a kafka topic that is not on my localhost to a realtime table. i have tried ip:19092 in y broker list but it doesnt work. also everthing is ok when my kafka topic in on my local host in a same network. ant tips?