https://pinot.apache.org/ logo
#general
Title
# general
r

Ryan Ruane

05/05/2022, 11:16 AM
Pinot Client Rust Hi there. I wrote in the other day about multi-value column ingestion jobs, and at the request of @Mayank, I created the issue: https://github.com/apache/pinot/issues/8635. The reason I was trying to create a table with ingestion of all possible types is because I am writing a rustlang client modelled after https://github.com/startreedata/pinot-client-go. Here is the repo, if anyone is interested: https://github.com/yougov/pinot-client-rust
Given https://apache-pinot.slack.com/archives/CDRCA57FC/p1651693832915089 I will have to remove the PQL support, however.
k

Kartik Khare

05/05/2022, 11:17 AM
Hi Ryan, I am looking into this issue. Can you also send me the link to discussion with Mayank. Or just high level points.
r

Ryan Ruane

05/05/2022, 11:18 AM
Everything and more about that issue is on the issue itself, including schema, ingestion job, and data
I more meant to let the community know about the rust client here
k

Kartik Khare

05/05/2022, 11:19 AM
cool. I'll update you in some time. The rust-client will be amazing! Thanks a lot for the contribution!
r

Ryan Ruane

05/05/2022, 11:23 AM
The client provides a DataRow type for SQL responses, which is strongly typed. It does this by first deserializing the result table to json, and then rationalizing the types based on the schema. Alternatively, a row can be deserialized to any struct implementing
serde::Deserialize
, which it does by converting each row to a json map of column name to value, and then passing that to the final deserializer. Ultimately there's a question of whether result table deserialization might be better supported (faster/lower memory) by implementing a custom serde deserializer, but for now I just inserted myself as a middle man and manipulated the row structure into one already supported for arbitrary deserialization.
k

Kartik Khare

05/05/2022, 11:25 AM
We are also planning to introduce binary responses so that serialization/deserialization time can be reduced. Sort of like proto/thrift response instead of json from broker.
r

Ryan Ruane

05/05/2022, 11:25 AM
Interesting. Good to know.
I would welcome any feedback and contributions. Apart from removing PQL, and some mild uncertainty about edge cases for array columns of the issue referenced, the repo is usable. Examples and documentation are provided, and everything is tested.
2 Views