https://pinot.apache.org/ logo
#general
Title
# general
k

Kishore G

06/11/2020, 11:00 PM
curious, does pinot inherently handle duplicate rows based on some column? ā€¢ Not as of now.
k

Kenny Bastani

06/11/2020, 11:08 PM
@Pradeep In many cases, de-duplication can be done at query time by using DISTINCT. Let me know if this is not an option for you. Maybe we can find a solution.
p

Pradeep

06/11/2020, 11:12 PM
yeah I was aware of that, was just curious if we were to do upload of data from from multiple places and was wondering if there is a way to handle overlaps. not very important at the moment, something that just came to my mind, so thought would ask, thanks.
šŸ‘ 1
k

Kishore G

06/11/2020, 11:17 PM
This is part of the upsert solution
we dont have it yet.
@Kenny Bastani fyi, kafka producers can generate duplicate messages
dedupe during ingestion will make sure that the data is accurate
šŸ‘ 1
p

Pradeep

06/11/2020, 11:18 PM
nice
k

Kenny Bastani

06/11/2020, 11:18 PM
@Pradeep Thanks for asking. I really appreciate you putting together these questions. šŸ‘
šŸ‘ 1