This message was deleted.

I don't know of one that applies to Superset, but at Imply, with Pivot (our analytical UI for Druid) we periodically fetch the max timestamp from Druid, and if that max timestamp has increased, we refresh the view. Obviously this doesn't catch any update, but it catches updates that add new data with newer timestamps Fetching the max timestamp is a cheap query if you use native

timeBoundary

, or if you use

SELECT MAX(__time) FROM tbl

in SQL and set

enableTimeBoundaryPlanning: true

. It's metadata only in those cases

Wow, fantastic to know about the dynamic refresh, that was something we could have done in my last company with Druid! 🤨

Very interesting @Gian Merlino! I assume the MAX(__time) would also work with filters? If so, we could do a simple packaging of a set of filter objects to periodically check for a newer __time

MAX(__time)

does work with filters, but it won't be metadata-only in that case. It will need to actually apply the filter. But it should still be fast if it can use an index. Just not -as- fast as if it could use

timeBoundary

Ah yes, when selecting the max(__time) one already knows that it is AT LEAST as much as X