hopefully 😉

Screen Shot 2020-08-11 at 10.57.56 AM.png

haha thanks kishore. lucidchart?

@User is deleting rows (or more in general segments) something that Pinot supports? I found this config parameter for retention of deleting segments, but I can’t operatively understand how do you go about doing it In other words is there a way in which I can delete a set of rows between 2 points in time?

We don’t support that

You can delete at segment granularity

Anthony pointed out that the offline rows win over the live, my question on delete was really to address that at serving and it seems that there is no need of deleting

thanks for your response!

ah got it

@User as an FYI ^

@User I have a question around the input connector for Realtime tables. Kafka has a contract of

at least once

how does Pinot preserves reporting integrity in case Kafka outputs the same element twice into the datastore? Thanks in advance

Kafka does not output into Pinot

Pinot pulls from Kafka and we ensure that every event is consumed only once

@User talk at Kafka Summit will talk about this in detail

I understand, great to hear, so we have one less thing to worry about 🙂

yes. we ensure consistency across replicas as well.

@User sorry to bother you again, but I would like to ask one more clarification around the duality between real-time and offline. As we mentioned with an hybrid table setup at query time if an entry appears in both stores the offline ones take precedence • How is the “collision” determined? Is there some sort of identifier column in the table schema that governs that? • When setting up real-time and offline, my understanding is that real-time has some sort of window associated with it: 1. What happens when that period passes? Are entires purged by the real time datastore? 2. If an instance exists in which entries are consolidated and moved from real time to offline, how the same key aspect is dealt with? Are real time entries dismissed in favor of the existing offline (if any)? I apologize if some of these questions are already addressed in the documentation, happy to read relevant sections in case I missed them Thanks,

Regarding hybrid table and which data takes precedence, this might help: https://docs.pinot.apache.org/basics/components/broker (time boundary section)

I’m not sure what you mean by “window” of a realtime table. We have a concept of retention. This can be configured for both realtime and offline tables. If the data in the table becomes older than the retention, it is deleted

Thanks Neha, I am going to read that and reply after

That page perfectly explains the first question, it’s done on the time series dimension, there’s no key on single object involved, makes sense. As for the retention, yes I was referring to that seems like that is a different concept than the behavior I was worring about. I guess the only question left is if there is instances in which Pinot independently consolidate Realtime segments into the offline ones, I have a vague memory of reading something about it, but not 100% sure

If that is the case I am basically trying to understand if the Offline data is replaced by the newly “consolidated” realtime segments

as of now, the offline data needs to be populated by you, with your own offline jobs setup.

but,

we have a project ongoing, which will move segments from realtime to offline table - https://docs.google.com/document/d/1-e_9aHQB4HXS38ONtofdxNvMsGmAoYfSnc2LP88MbIc/edit#

though i dont know if that’ll help in your case, since you need the accurate data merged with the realtime data

Thanks Neha, in my case it would actually hurt us

I was more watching out for that happening under the cover

so in that regard we are good, thank you very much