Hi, I'm new to Pinot and have a possibly dumb ques...
# generalj
John Knapp
04/20/2021, 3:46 AMHi, I'm new to Pinot and have a possibly dumb question - with so many features and horizontal scalability, when should one not use Pinot as a general relational database solution?
m
Mayank
04/20/2021, 3:48 AM
Pinot is optimized for performance, but is not a general relational database. For example, it doesn't provide transactions, or support for complex joins/nested queries.
j
John Knapp
04/20/2021, 3:54 AMThanks. Do you have any links that talk about the limitations? I know that joins are not supported but presto is supposed to enable that.
m
Mayank
04/20/2021, 3:55 AM
Yes, we close those gaps with Presto
Mayank
04/20/2021, 3:58 AM
We don't have an exhaustive list of limitations but here are the major ones:
Copy code
1. Non transactional, eventually consistent
2. Does not support full SQL (used with Presto to fill that gap).
3. Not a source of truthj
John Knapp
04/20/2021, 4:01 AMI see. Well, it sounds like it should pair well (to use wine terminology) with any other transactional database. Use the transactional DB for read-modify-write and use Pinot for reads so long as eventually consistent is acceptable.
John Knapp
04/20/2021, 4:02 AMCan it be paired with a no-sql DB like Mongo DB?
d
Daniel Lavoie
04/20/2021, 4:05 AMIf you can leverage a CDC infrastructure to get your changeset to pinot, it does work nicely together
m
Mayank
04/20/2021, 4:07 AM
So the pattern there is to emit the changeset to a messaging system (eg Kafka) which Pinot can consume from
d
Daniel Lavoie
04/20/2021, 4:09 AMAnd since Pinot scales well for reads and write, it can be fed by different transactional sources from your system.
j
John Knapp
04/20/2021, 4:12 AMCDC = change data analysis?
d
Daniel Lavoie
04/20/2021, 4:12 AMChange Data Capture
j
John Knapp
04/20/2021, 4:14 AMAny recommended CDC architectures? debezium any good?
m
Mayank
04/20/2021, 4:18 AM
Yeah I have heard folks using that
c
Chinmay Soman
04/20/2021, 5:02 AMChinmay Soman
04/20/2021, 5:03 AMDebezium is definitely the way to go !
Chinmay Soman
04/20/2021, 5:08 AMKenny has another blog coming up with Debezium and Pinot upserts which is even better
j
John Knapp
04/21/2021, 4:20 AMthanks!