https://pinot.apache.org/ logo
#general
Title
# general
a

Aaron Wishnick

04/09/2021, 3:54 PM
Is pinot-admin.sh the preferred way to upload batch data or is there a REST API for that too?
m

Mayank

04/09/2021, 3:54 PM
There's rest-api as well.
You can use either one that works best in your deployments
a

Aaron Wishnick

04/09/2021, 3:54 PM
Cool. Is there any documentation on how to use that REST API?
m

Mayank

04/09/2021, 3:55 PM
Let me find
For data import ^^
Getting the rest api in a sec
k

Kishore G

04/09/2021, 3:56 PM
Most commands in Pinot admin use rest api under the hood
You can see swagger UI
m

Mayank

04/09/2021, 3:56 PM
Yes, swagger UI lists all the rest apis including segment upload
message has been deleted
From swagger UI ^^
j

Jonathan Meyer

04/09/2021, 4:00 PM
Just curious, what's the difference with this API ? This new API directly uploads to the server (via the controller) without going through the deep store, right ?
m

Mayank

04/09/2021, 4:02 PM
@User The PR description states:
These are meant for quick testing/trials, and not intended for production usage.
j

Jonathan Meyer

04/09/2021, 4:03 PM
Yes, so I wonder how they are differing under the hood ?
k

Kishore G

04/09/2021, 4:05 PM
The segment generation is happening on the controller node.. running this on a controller is not scalable/fault tolerant for production setting
✔️ 1
m

Mayank

04/09/2021, 4:05 PM
Correct
But if you are looking for prod supported data push using uri (ie do not push payload to controller), the ones listed in swagger should already do that @User
👍 1
j

Jonathan Meyer

04/09/2021, 4:07 PM
With the
/v2/segments
API (also called by the CLI
LaunchDataIngestionJob
?) - the minion is responsible for generating segments, right ? 🙂
m

Mayank

04/09/2021, 4:10 PM
These upload api's are independent of where/how the segments are generated
j

Jonathan Meyer

04/09/2021, 4:16 PM
Ok, thanks @User @User (and sorry for "hijacking" this thread 😄)
m

Mayank

04/09/2021, 4:16 PM
No worries, we are all knowledge sharing here, so everyone benefits 🙂
1
a

Aaron Wishnick

04/09/2021, 4:22 PM
Thanks all!
Ok so the Swagger UI is super cool, I had no idea that existed at all 🙂
So if I understand right, pinot-admin.sh is largely a wrapper around that API, and for example in the case of LaunchDataIngestionJob, the machine that I run pinot-admin.sh on isn't doing much work at all, it's just telling the Pinot cluster to begin data ingestion?
k

Kishore G

04/09/2021, 6:40 PM
launchDataIngestion is an exception
but if you configure minion, you can convert that into an API call
a

Aaron Wishnick

04/09/2021, 6:42 PM
I'm sorry -- I'm pretty new at this and don't understand. How is it an exception?
k

Kishore G

04/09/2021, 6:42 PM
because it uses the pinot library and generates the segment on the machine you ran the command and uploads the generated segment to Pinot
a

Aaron Wishnick

04/09/2021, 6:45 PM
I gotcha, so it is doing some work on that machine I run the command on. Is this work essentially re-encoding the data from whatever the incoming format is to Pinot's segment format?
m

Mayank

04/09/2021, 6:45 PM
Yes
a

Aaron Wishnick

04/09/2021, 6:45 PM
Gotcha, thanks
Are there any docs about how to do this with minion?