This message was deleted.
# generals
Slackbot
03/10/2023, 4:14 PMThis message was deleted.
j
John Kowtko
03/10/2023, 4:30 PMThere is an API for the new MSQ SQL-based ingestion operations: https://druid.apache.org/docs/latest/multi-stage-query/api.html ...
m
Madhav Gunampalli
03/10/2023, 4:33 PM@John Kowtko We are not using SQL based ingestion. My use case is submitting ingestion task to druid on new file arrival. I am generating ingestionSpec and making an API call to
/druid/indexer/v1/taskMadhav Gunampalli
03/10/2023, 4:34 PMTo generate ingestionSpec i was looking for if there is any client library druid has to use these classes
j
John Kowtko
03/10/2023, 4:35 PMOk sorry I don't know the answer to that one ...
👍 1
m
Madhav Gunampalli
03/10/2023, 4:35 PMThank You
t
Tish Johnson
03/10/2023, 5:31 PM@Madhav Gunampalli - I am not sure if you have seen this but, the API documentation for the Overlord is here: https://druid.apache.org/docs/latest/operations/api-reference.html#overlord
Tish Johnson
03/10/2023, 5:32 PMAlso, it is possible to use Apache Airflow to complete the same operation of taking a new file and creating a Druid ingestion task.
Airflow's filesensor:
https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/file.html
https://stackoverflow.com/questions/54791596/any-example-of-airflow-filesensor
Airflow has a Druid operator that can be used to generate the task for the ingestion https://airflow.apache.org/docs/apache-airflow-providers-apache-druid/stable/operators.html
m
Madhav Gunampalli
03/10/2023, 6:01 PM@Tish Johnson Thanks sharing the information on the API. The info i was looking around client libraries in order to send ingestion request to Druid from Java application. For example to send ingestion task we need to send JSON spec can be generated from Java POJOs
Madhav Gunampalli
03/10/2023, 6:01 PMIf there is any client library that has all these supported classes to use from Java client applicaiton
t
Tish Johnson
03/10/2023, 6:29 PM@Madhav Gunampalli - Most libraries and other 3rd party apps for Druid are here: https://druid.apache.org/libraries, but most focus more on query than ingestion.
Still, there maybe parts of the source code that is useable for your use case.
m
Madhav Gunampalli
03/10/2023, 6:33 PMGot it. Thanks @Tish Johnson I will look into it
👍🏾 1