https://pinot.apache.org/ logo
#general
Title
# general
m

Mohamed Kashifuddin

02/15/2022, 6:43 PM
Hi Team, Does anyone know any bash or python script to create schema and table in pinot ?
@User @User
j

Jackie

02/15/2022, 6:49 PM
You may use
pinot-admin.sh
within the distribution to achieve that
m

Mohamed Kashifuddin

02/15/2022, 7:01 PM
Can you please point me to the reference pinot-admin.sh file ?
@User
j

Jackie

02/15/2022, 7:08 PM
You should be able to find it under
pinot-tools/target/pinot-tools-pkg
If you are using the distribution, this page should provide some guidance: https://docs.pinot.apache.org/basics/getting-started/running-pinot-locally#getting-pinot
m

Mohamed Kashifuddin

02/15/2022, 7:15 PM
The pinot is running in K8s schema and table are being created via web-ui, so if there is any bash or python script which I can run and let the script create it @User?
j

Jackie

02/15/2022, 7:23 PM
pinot-admin.sh
should be able to serve the purpose
m

Mohamed Kashifuddin

02/15/2022, 7:34 PM
Okay let me try but I see it's generally used as bin/pinot-admin.sh to start the pinot in local but I don't see any eg sample table and schema inside pinot-admin.sh file which I can use it as reference @User
j

Jackie

02/15/2022, 7:45 PM
You can check all the sub-command using
pinot-admin.sh -h
m

Mohamed Kashifuddin

02/16/2022, 10:35 AM
Yes I can understand using this we can create schema and tables inside the container eg (inside controller pod) but how do I run outside the container ? because In my case its not ideal to copy schema and table json inside a container then run it. it does makes more complex then creating it from WebUI. @User @User
Copy code
bin/pinot-admin.sh AddTable \
    -schemaFile examples/stream/airlineStats/airlineStats_schema.json \
    -tableConfigFile examples/stream/airlineStats/airlineStats_realtime_table_config.json \
    -exec
d

Diogo Baeder

02/16/2022, 12:18 PM
In YouGov we ended up using the Swagger UI to help understanding what endpoints in the Controller we could use to create the schemas and tables, via a Python script.
m

Mohamed Kashifuddin

02/16/2022, 12:23 PM
I didn't understand about what your saying @User
Can you elaborate it more ?
d

Diogo Baeder

02/16/2022, 1:41 PM
When you get Pinot running, you can reach a Swagger UI which is a UI that documents the API endpoints available in the controller, where you can also do some experimentation with endpoints and see which ones are available. So it's just a matter of sending requests to some of those endpoints, if you want to create schemas and tables.
j

Jackie

02/16/2022, 5:34 PM
As @User suggested, if you find using WebUI is easier, you may just send the same HTTP/HTTPS request using the script to achieve the same effect. The pinot-admin is essentially a helper to create the request and send it to the controller.
x

Xiang Fu

02/16/2022, 7:24 PM
You can run
Copy code
bin/pinot-admin.sh AddTable -controllerProtocol http -controllerHost 127.0.0.1 -controllerPort 9000 ....
specify with your local reachable controller url
m

Mohamed Kashifuddin

02/17/2022, 4:49 AM
Yes agree its kind of need to do some experimentation with it @User
Is it possible to point a folder instead of a pointing a single json file eg .
Copy code
bin/pinot-admin.sh AddTable \
    -schemaFile examples/stream/airlineStats \
    -tableConfigFile examples/stream/airlineStats
In my case even I can copy the tables and schema folder then at once then if I can point each folder of table and schema folder using
bin/pinot-admin.sh AddTable \
like above then too its fine . but the question is whether
bin/pinot-admin.sh AddTable
can take the whole folder and apply each json file inside the folder ?
@User @User ??
j

Jackie

02/17/2022, 5:04 AM
No, but you may loop over each json file and invoke pinot-admin in your script
m

Mohamed Kashifuddin

02/17/2022, 5:07 AM
yaa that is also a way but it would have been simple if we can simple point a folder and rest is done any reason why
bin/pinot-admin.sh AddTable
can't take the whole folder ?
x

Xiang Fu

02/17/2022, 5:23 AM
No, you can only create a table per command
Just write a wrapper script to loop the folder and create the table
m

Mohamed Kashifuddin

02/17/2022, 5:31 AM
okay