This message was deleted.

Hi Michael, the data is always partitioned by time as the primary partitioning strategy. Under what use cases would you like to utilize Druid for ?

I have a large database of files like assets and like for users to have the ability to do analytics over them. They have creation and update columns but rarely would query using them

Druid works really well with streaming data with event time stamps occurring as they get ingested … I am not sure Druid would be appropriate for your use case if you want to

fake

the time column

I see it was suggested by one of the imply founders

Give it a spin and see what you think about it

It is always an option but if someone did something like that I appreciate a reference.

Check out lots of use cases in https://druidsummit.org/

how does the data change over time? How frequently do you add new data?

If memory serves @Michael Tsibelman this video contains some stuff about this… But my memory is awful… 😄

@Abhishek Agarwal it has some additions and updates over time but it is small compared to all existing data maybe 1% of data is updated in any given day

If someone would like to update that blog, we have a link pinned for updates/changes over at #marketing :)

@Michael Tsibelman here's another thread that might be worth a read: https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1644937752232759 @James Kelleher might be able to share some follow-up.

@Michael Tsibelman a couple of thoughts : • If you create a dummy time by hashing something else, you could distribute the data along a fake timeline (like the Zeotap case describes), but you will need to query it by filtering on a specific value for the __time column on most, if not all queries. You would otherwise need to access the whole dataset to resolve the query. • Updating the data. Druid uses immutable segment files to store the data. Update operations can be done, but require that you update a whole time interval. Even if you only change 1% of the data, the REPLACE operation works by time interval, given that this would be a hash, you'll need to test the

REPLACE OVERWRITE WHERE __time = <hash value>...

. While unlikely, one possible concern here is hash collisions such that you would need to include all data for any matching hash values when running the REPLACE.

@Sergio Ferragut very helpful comments, it looks tricky to make it right, hashing needs to be clever to locate items with high probability of changing together, but the upside is that we could use Druid for all of our data events and snapshots, one helpful thing is that dataset is 250 gb in Postgres so even full scan of data in druid can be fast enough.

At 250GB, you might be able to just load it every day...just a thought. If this is the case you could forego the hashed _time strategy and just use secondary partitioning with

REPLACE .... CLUSTERED BY

and cluster on the most commonly used filtering dimension(s). If you can get 500MB segment sizes (or so) that would give you ~500 segments which are plenty to take advantage of parallelism and when filtering on the clustered dimensions, you'll also get segment pruning. Very doable.

Thanks again I need to wrap my head around all of this.