https://pinot.apache.org/ logo
#troubleshooting
Title
# troubleshooting
d

Diogo Baeder

06/07/2022, 4:07 PM
Hey guys, what sort of architecture would you choose to go for if you wanted to have separated handling of queries according to the client needs - whether "user interface" or "reporting"? More on this thread.
I was thinking about the possibility of having two Pinot clusters for a project I'm developing - one cluster to serve data for the UI of an application, for end-users, and another cluster to run reporting stuff, so that one cluster suffering high load would never impact the other. But maybe this could be achieved with replica groups, I don't know. What do you guys think about this? How would you approach this from a systems architecture perspective, considering Pinot's capabilities?
m

Mayank

06/07/2022, 4:21 PM
You can have single Pinot cluster but provide isolation via Pinot tenants (hosting different workloads on different tenant tags)
d

Diogo Baeder

06/07/2022, 4:25 PM
Got it. That's interesting. But then the mechanics of ingesting data would have to be duplicated as well, right? Like, if I create segments with SegmentTarPushTask for one tenant, I would also have to do the same for the other tenant?
m

Mayank

06/07/2022, 4:26 PM
Are the two tables going to mirror the data? I think you can push same data to multiple tables
Using same pipeline
d

Diogo Baeder

06/07/2022, 4:30 PM
But then I would end up having Controllers and Servers sharing the duty to respond to queries for both tables, no? Or is there a way to separate this? What I'm thinking of is to have a set of Controllers and Servers for each "department" (UI and reporting), so that there's absolutely no way of one set interfering into the other - for example, if a server gets OOM-killed because of some very large reporting task, I want this to not even be felt by the UI at all. Or like too many reporting tasks taking too much RAM and CPU from all the servers and the end-users starting to see the UI responding too slow.
m

Mayank

06/08/2022, 12:00 AM
Controllers can serve thousands of tables. Brokers also have tags, so you can do separate tags for brokers as well. Servers are the ones that do most of the work, so you definitely want to isolate them.
d

Diogo Baeder

06/08/2022, 12:06 AM
Ah, I didn't know about tags. I'll search more about that. And about isolating Servers, how would you do that?
m

Mayank

06/08/2022, 12:07 AM
Same - tenant tags
d

Diogo Baeder

06/08/2022, 12:07 AM
Ah, nice! OK then, sounds like a good plan 🙂
Thanks again, man!
👍 1