Hi <@UDQU92KBK> Couple of question 1. Can we use d...
# troubleshooting
m
Hi @Mayank Couple of question 1. Can we use dedup for full upsert or partial upsert secnario https://docs.pinot.apache.org/manage-data/data-import/upsert-and-dedup/upsert ? 2. Can we use a query like select * from table1 where column1 in (id1, id2, id3....id10k) if column1 has inverted index?
m
1. Not sure what you mean by dedup for upsert scenario. In dedup, the newer occurance of the PK is discarded. 2. You can, yes. What's the use case?
m
ok understood for point 1. For point 2 I have a order table and certain PoNumber I need to filter the data (run the query using IN clause) @Mayank
m
But why you have 10k of them, who’s generating them
m
During some issue window we need to analyse the data for specific Po numbers
m
Yeah but what generates this specific Po
m
This is some other system like OMS which generates this PO and we store this order relavant data in pinot. For some issue window we figure out that these many number of PO are impacted and once issue is fixed in duration of fix we keep on monitoring if these order or Po getting fixed or not
This is our usecase @Mayank
m
Ok, got it. Asking to understand if there's a better way to query. 10k in one query may (or may not) be inefficient.
m
Okay understood
@Mayank And do you recommend adding one or two inverted indexes for these type of use cases?
and which one to suggest? Inverted index or idSet
m
IdSet is not an index
Both can be used together
m
Ok