Hi, I was looking at the state of ingestion in Apa...
# general
e
Hi, I was looking at the state of ingestion in Apache Druid. https://druid.apache.org/docs/latest/ingestion/ I noticed there seems to be quite a lot of options, which is normal for open-source, but was wondering if we are thinking of some convergence. Here's some duplicated/parallel options in different tracks: • Ingestion methods: 4-5 ◦ Streaming (Kafka and Kinesis) ◦ Native batch ◦ SQL (MSQ) ◦ Hadoop-based (deprecated) • task orchestrators/managers: - 2-3 (at least) ◦ middleManager ◦ Kubernetes-based (MM-less). ◦ To some extent, YARN (In case of Hadoop). Although this have been deprecated. • Schedulers: 2 (ok, actually 1 + Null) ◦ Supervisors for streaming. ◦ No scheduler for batch (user relies on external scheduler). I was wondering if there is a roadmap or any thoughts where it goes long-term and perhaps converges: • Should native batch be replaced with MSQ? • Should streaming use SQL (INSERT) as well, instead of native spec? • Should a batch scheduler be introduces and should it share API similarity with streaming schedule (Supervisor).
b
Hi Eyal, good to "see" you! I don't know official ideas, but I get the impression that the future direction is towards SQL, whether MSQ or the newer "dart" that might expand on MSQ. But I don't know whether there are any ideas or plans around a scheduler.
e
Thank you for the insight! On a side note, what got me thinking on all of that is the Hadoop deprecation notice in 32.0 invites to use SQL (MSQ), not Native:
Copy code
Hadoop-based ingestion is now deprecated. We recommend that you migrate to SQL-based ingestion.
This couples SQL interface and MSQ engine. That got me thinking about how streaming doesn't have SQL interface for ingestion, so anyone implementing Lambda architecture cannot easily share logic between streaming and batch.
c
there is actually an experimental simple batch scheduler supervisor that was introduced in druid 33, https://github.com/apache/druid/pull/17353
❕ 1
i think we are heading towards using SQL for ingestions, though we don’t have any hard plans for streaming quite yet
i think it is currently unlikely that we will change how streaming ingestion works to do all of what MSQ can do since doing shuffles on the streaming data adds a lot of complexity, so it is much more likely that we will introduce some syntax sugar to allow using an SQL statement as a transform of the stream instead of the native json spec, but still otherwise operating the way it currently does
💯 1