Hello, I’m very interested in querying Pinot via the GRPC service and returning Apache Arrow tables out: https://docs.pinot.apache.org/users/api/broker-grpc-api The trouble is the relevant settings don’t seem to be settable on

1.3.0

:

> Add gRPC port config in pinot broker to enable the `BrokerGrpcServer`:

Copy code

pinot.broker.grpc.port=8010

Is this something that’s available in

1.3.0

? Or is this a planned feature that isn’t in a stable version of Pinot yet?

Could you elaborate a bit on what problem you are trying to solve? Pinot is not a system where you want to stream out raw data from as a primary use case, you want to serve analytical queries using Pinot.

In some cases we may be running an aggregation on a relatively high cardinality column,. Imagine that

SELECT agg_col_1, agg_col_2, AGG_FUNCS_HERE(...) FROM my_table WHERE timestamp > ... GROUP BY agg_col

…were to return 500k rows. Converting the output from JSON to an in memory format (we use Arrow) could become a bottleneck. If we could get bytes of arrow tables as a response, we’d be able to skip that step

@Gonzalo Ortiz for the broker gRPC endpoint question.

To add more color, I’m particularly concerned about end to end latency, including data transfer/ deserialization costs. The benchmarks here look promising: https://docs.pinot.apache.org/users/api/broker-grpc-api Thanks for looking into this

You could also break down the query into multiple queries, for example getting data per day at a time

This GRPC endpoint is used for transferring query results between broker and client. Note that the query results is already cached in broker for the heap memory

My two cents here, the broker GRPC endpoint may also have higher memory consumption and latency costs than you may expect, at least in MSE, because right now the broker doesn't pipeline results but instead keep them in memory until the query is finished. That is something we would like to improve but there are other features with higher priority right now

right, this just save the response serde cost between client and broker