https://pinot.apache.org/ logo
#general
Title
# general
j

Jonathan Locke

01/11/2022, 6:33 PM
can someone point me to the documentation for java connection pooling with pinot?
m

Mayank

01/11/2022, 6:35 PM
Do you mean making concurrent requests to Pinot? If not, could you elaborate what you are looking for?
j

Jonathan Locke

01/11/2022, 6:38 PM
We don't want the connection to Pinot to be dropped and restarted for each query. In JDBC, you need a connection pooler. Does Pinot take care of that issue inside its client then?
By connection, I mean the underlying HTTP(S) connection
k

Kenny Bastani

01/11/2022, 7:35 PM
We do not have connection pooling for client-side HTTP requests. The JDBC client sends HTTP requests to the available broker, which means each request blocks on a different thread, as you would expect. The thread executor for the client does not have a maximum amount of threads.
Because Pinot is a read-only datastore through the JDBC client, there is no transaction guarantees that would require connection pooling. But I'm interested to hear your comments on any limitations you are experiencing.
j

Jonathan Locke

01/12/2022, 8:07 PM
We're not experiencing any problems. I just wanted to understand how to minimize HTTP connection setup overhead. If Pinot is using JDBC, maybe something like c3p0 could keep a pool of open connections? This really has nothing to do with transactions or threads. All that's being pooled are a set of HTTP connections (to a broker). This optimization is really only useful because establishing new HTTP connections is expensive and there's no reason not to reuse old ones that are already opened. To make a query, a connection is retrieved from the pool, the request/response cycle occurs and the connection is returned to the pool. After a period of not being used, connections in the pool may be closed and removed from the pool.
It looks like c3p0 is dead, instead there's https://commons.apache.org/proper/commons-dbcp/
m

Mayank

01/12/2022, 8:22 PM
The pinot client currently doesn’t do that (mostly because we haven’t experienced any issues). If you can open a GH issue, we can follow up
j

Jonathan Locke

01/12/2022, 8:26 PM
with large numbers of queries, HTTP setup overhead can be a substantial effect on latency. I will open an issue to record our discussion.
not urgent, but it could help heavy duty users
thanks for your help!
m

Mayank

01/12/2022, 9:10 PM
Thanks @User
k

Kenny Bastani

01/14/2022, 3:24 PM
This might be possible to implement using Keep-Alive HTTP header. Let me look into it.
👍 1
f

Fizza Abid

04/26/2022, 6:56 AM
@User
m

Mayank

04/27/2022, 2:22 PM
@User
s

saurabh dubey

04/27/2022, 2:28 PM
I had done a POC with a netty based client, keeping a pool of client -> broker active keep-alive HTTP channels, such that we don't need to open a new one for each query. https://github.com/saurabhd336/pinot/pull/3/files If the community feels the need of such a client, I can pick this up and build on top of my changes.
cc: @User ^