Hey guys! is there any reason why Druid lookups ca...
# general
p
Hey guys! is there any reason why Druid lookups can't be fetched from web resources (eg. API) with URI? Right now I can only fetch a file from S3 where I'd be looking for updating my lookup thru HTTP endpoint
g
i think no reason other than it hasn't been implemented yet
if you're interested in contributing, it would be an implementation of
org.apache.druid.data.SearchableVersionedDataFinder
the interface contemplates some form of searching for the latest file at a location, but i think for http (since there isn't a standard "list files" operation) it should skip that and just fetch the provided
uri
, assuming it matches the
pattern
.
or another approach— you can load your data into a regular table with the
http
input source periodically (using some external scheduler) for an example check out "REPLACE an entire datasource" here: https://druid.apache.org/docs/latest/multi-stage-query/examples#replace-an-entire-datasource then use standard SQL
JOIN
at query time. the table will not be broadcast to all servers ahead of time (broadcast will need to be done at query time). but perhaps this is OK for your use case. you could try it out and see how well it performs
p
thank you Gian! I Will consider contributing! Thanks for pointing out directly which piece of the code does it!