https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
o
This is for an automotive application to track data about cars that will go into predictive model training, ideally in ways that are easy to slice and dice so that the models can be trained on edge cases.
Scale of data set is around 1 TB. Right now we're looking at putting it in a postgres instance until we find something better
r
it's possible to query with radius if you use native query but it's possible to use basic lat/lng filters with rectangular (using lat>X and lat < y and long>...) over the SQL engine
g
you can also use geohashing (druid has no builtin support for this but the approach works without DB support)
i would look into those two things and see if one meets your needs as far as geo constraints
for time constraints, druid has a sense of primary timestamp, so probably you'll put the start of the range as primary timestamp (always called 
__time
), and the end as secondary (let's call it 
end_time
). filters on the primary timestamp are super fast (there is a special index for this) so you can leverage this if you have a sense of what the max duration is. something like this for a point filter on `target_time`:
Copy code
WHERE [target time] >= __time AND [target_time] < end_time AND [target_time - max_duration] <= __time
the 
[target_time - max_duration] <= __time
clause enables more aggressive use of the primary timestamp index
2 Views
Open in Slack
PreviousNext
Powered by