https://linen.dev logo
Join Slack
Powered by
Is there any mechanism to allow a dynamic segment ...
# general
h
Is there any mechanism to allow a dynamic segment granularity yet? I.e specify the beginning and the end of a segment, rather than letting druid decide based on the 
segmentGranularity
parameter? Since the meta table already has 
start
and 
end
as timestamp, such segments should just work ™️ but I don't see how I could create one.
j
In theory it seems that it could be done, although I can see some complications in working through normal operations like a compaction over multiple time intervals. Just curious, what is the use case for doing this? Do you have a specific interval type that is needed?
h
cohorts in realtime would be my use case. and no, forget compacting that 🙂 so each event has a 
ts,
which is more or less the current time, and also a 
source_ts,
which is more or less a random timestamp within the last year. I would like to configure the ingestion such that: • 
__time
maps to 
source_ts
• 
start
is [today - 180.days] • 
end
is [end of current day] • if the event is outside the segment, drop it w/o an error
each day, we start a new segment. this way I can have a rolling window of 180 days
right now, I am doing a realtime intake on 
ts
and after 24h I do a 
index_parallel
job that takes 
source_ts
as 
__time
. With the right tuning, this works out quite ok to never need a compaction.
but now they like it so much, they want realtime intake
actually compaction would be nice, but only if it allows for partial, e.g. "only compact segments that are older than x days into 1"
but the write amplification is quite high, so right now I am avoiding compacting by tuning the jobs
I guess I would need to configure the code that selects and/or creates the segment to observe 
ts
and then use 
source_ts
inside the segment... shouldn't be too hard to do, but I thought I ask if there is any prio work
j
If you ingest directly using source_ts as __time, and if you set your segment granularity to, say, MONTH, and set your taskDuration to something larger like 4 or 8 hours, then in theory you should be able to ingest directly in real-time without too much fragmentation. There is also concurrent append/replace, which would allow you to compact the months while streaming ingestion is still sending in data. Would you still have an issue if you did it this way, based on your use case?
h
Copy code
druid=> select
  start,
  "end",
  count(1) as segments,
  min(seg_size) as min_size,
  sum(seg_size) as total_size,
  max(seg_size) as max_size
from (
  select *, ((convert_from(payload,'UTF8')::json->>'size')::bigint)/1024/1024 as seg_size
  from druid_segments
  where
  datasource = 'cohorted'
  and created_date like '2025-03-13%'
  ) as segments
group by 1,2;
          start           |           end            | segments | min_size | total_size | max_size
--------------------------+--------------------------+----------+----------+------------+----------
 2024-07-01T00:00:00.000Z | 2024-10-01T00:00:00.000Z |        1 |       41 |         41 |       41
 2024-10-01T00:00:00.000Z | 2025-01-01T00:00:00.000Z |        1 |       42 |         42 |       42
 2025-01-01T00:00:00.000Z | 2025-04-01T00:00:00.000Z |      458 |       84 |     195166 |      623
(3 rows)
my problem is that if any segment is full, all get flushed. I.e. I would get 458 micro segments for the previous two quarters
j
Do you need segment granularity of any sort for query performance? If not, you could use YEAR, or even ALL granularity (although I tend to stay away from ALL unless it's a smaller datasource).
h
I want a 6 month window, so quarter was the way to go... but yeah, year granularity would avoid the problem... but then I have would need to grow to 2 years before I can unload data... that's 4x from what I actually need.
right now the overhead is 1 quarter, so only 50% rather than 300%
j
Looking at your size query ... if 40mb is your minimum segment size, that's not bad at all ... you might be fine as-is from an ingestion perspective. Then my attentions turns towards queries ... if your queries are against much smaller time intervals than quarters, and if you are concerned about query performance, then you will get more segment pruning if you increase the granularity from quarter down to MONTH. Granted those fragment segments will get even smaller, but you could then start running compactions on specific months as needed to reduce fragmentation if it makes a different on query performance.
h
I don't see how I can run compaction w/o massive write amplification
above are the segments of 1 day. now imagine what it looks like for 6 months... I mean yeah, given how I wrote my code, it makes sense to drop to month in the batch import...
in the meantime, I was thinking about building the segment with 
all
and then manually changing the 
start
and 
end
but I didn't test how druid likes that
I guess I could build it into a datasource that's not actually loaded and only then do the move in one transaction, that way druid would never see me changing 
start
and 
end
or I add a cohort mode to the code
or better: fix in code that only full segments get flushed, not all open
j
on that last point ... if you increase taskDuration, intermediateHandoffPeriod, and maxRowsPerSegment then it will publish segments less frequently, which will reduce overall fragmentation. Your max segment size is only 623mb ... if your query filters are reasonably selective you could probably increase that segment size several-fold without noticable increase in scan times.
h
isn't half a gig the optiomal segment size?
j
That's the ballpark rule-of-thumb. in reality it all depends on your query accesses. If the query filters are highly selective and can use indexed lookups on the column indexes, then you could have much larger segment sizes with virtually no additional latency. But if your queries are scan/compute heavy on the segments then it will go the other way. Have your monitored your segments scans to know how fast they are?
h
I am logging all metrics via the prometheus logger, but tbh a lot of them don't make any sense to me... so what's the best metric to check this?
j
I usually start with Broker query times, then move to Historical/Peon query times and segment scan times. If the segment scan times are very fast (e.g. < 10ms) that usually means you can increase the segment size without slowing down the queries. Here is the Metrics doc page ... use this as a reference to help you determine which metrics to monitor. https://druid.apache.org/docs/latest/operations/metrics
Open in Slack
PreviousNext
Powered by