Is pinot-admin.sh the preferred way to upload batc...
# generala
Aaron Wishnick
04/09/2021, 3:54 PMIs pinot-admin.sh the preferred way to upload batch data or is there a REST API for that too?
m
Mayank
04/09/2021, 3:54 PM
There's rest-api as well.
Mayank
04/09/2021, 3:54 PM
You can use either one that works best in your deployments
a
Aaron Wishnick
04/09/2021, 3:54 PMCool. Is there any documentation on how to use that REST API?
m
Mayank
04/09/2021, 3:55 PM
Let me find
Mayank
04/09/2021, 3:55 PM
Mayank
04/09/2021, 3:55 PM
For data import ^^
Mayank
04/09/2021, 3:55 PM
Getting the rest api in a sec
k
Kishore G
04/09/2021, 3:56 PM
Most commands in Pinot admin use rest api under the hood
Kishore G
04/09/2021, 3:56 PM
You can see swagger UI
m
Mayank
04/09/2021, 3:56 PM
Yes, swagger UI lists all the rest apis including segment upload
Mayank
04/09/2021, 3:59 PM
message has been deleted
Mayank
04/09/2021, 3:59 PM
From swagger UI ^^
j
Jonathan Meyer
04/09/2021, 4:00 PMJust curious, what's the difference with this API ?
This new API directly uploads to the server (via the controller) without going through the deep store, right ?
m
Mayank
04/09/2021, 4:02 PM
@User The PR description states:
These are meant for quick testing/trials, and not intended for production usage.j
Jonathan Meyer
04/09/2021, 4:03 PMYes, so I wonder how they are differing under the hood ?
k
Kishore G
04/09/2021, 4:05 PM
The segment generation is happening on the controller node.. running this on a controller is not scalable/fault tolerant for production setting
✔️ 1
m
Mayank
04/09/2021, 4:05 PM
Correct
Mayank
04/09/2021, 4:06 PM
But if you are looking for prod supported data push using uri (ie do not push payload to controller), the ones listed in swagger should already do that @User
👍 1
j
Jonathan Meyer
04/09/2021, 4:07 PMWith the
/v2/segmentsLaunchDataIngestionJobm
Mayank
04/09/2021, 4:10 PM
These upload api's are independent of where/how the segments are generated
j
Jonathan Meyer
04/09/2021, 4:16 PMOk, thanks @User @User
(and sorry for "hijacking" this thread 😄)
m
Mayank
04/09/2021, 4:16 PM
No worries, we are all knowledge sharing here, so everyone benefits 🙂
➕ 1
a
Aaron Wishnick
04/09/2021, 4:22 PMThanks all!
Aaron Wishnick
04/09/2021, 6:33 PMOk so the Swagger UI is super cool, I had no idea that existed at all 🙂
Aaron Wishnick
04/09/2021, 6:36 PMSo if I understand right, pinot-admin.sh is largely a wrapper around that API, and for example in the case of LaunchDataIngestionJob, the machine that I run pinot-admin.sh on isn't doing much work at all, it's just telling the Pinot cluster to begin data ingestion?
k
Kishore G
04/09/2021, 6:40 PM
launchDataIngestion is an exception
Kishore G
04/09/2021, 6:41 PM
but if you configure minion, you can convert that into an API call
a
Aaron Wishnick
04/09/2021, 6:42 PMI'm sorry -- I'm pretty new at this and don't understand. How is it an exception?
k
Kishore G
04/09/2021, 6:42 PM
because it uses the pinot library and generates the segment on the machine you ran the command and uploads the generated segment to Pinot
a
Aaron Wishnick
04/09/2021, 6:45 PMI gotcha, so it is doing some work on that machine I run the command on. Is this work essentially re-encoding the data from whatever the incoming format is to Pinot's segment format?
m
Mayank
04/09/2021, 6:45 PM
Yes
a
Aaron Wishnick
04/09/2021, 6:45 PMGotcha, thanks
Aaron Wishnick
04/09/2021, 6:48 PMAre there any docs about how to do this with minion?