https://linen.dev logo
Join Slack
Powered by
I have noticed in some cases far more segment file...
# general
j
I have noticed in some cases far more segment files than expected (before compaction) and I'm curious why this might be the case. This isn't a major issue causing us trouble; but I am curious why we have these additional segment files. Picking a random kafka_index log file, I see that the StreamAppenderator decided to open a new segment file but doesn't really state why. All of the data belongs to the same period (segmentGranularity=DAY) and the number of rows persisted was well below the tuningConfig (maxRowsPerSegment=5000000, actual rows persisted was 667,150 ~ 173.58 MB). Other than __time and maxRowsPerSegment, why would the Kafka Indexer open a new segment file?
b
If you have late arriving (or early arriving) data, it will open new segments for each day that has data during the ingestion period.
j
We don't have any late or early arriving data. The indexer closed the old segment file and put all new data into the new segment file. The data all falls into the same window.
b
There are three counters that can trigger segment handoff. 1. maxRowsPerSegment - as you mentioned before 2. maxTotalRows - unlikely you're hitting this if you don't have late-arriving data 3. intermediatePersistPeriod - unless you have overwritten the default value, this should not be used. Finally, the segment handoff will occur at the end of the task duration, which by default is 1 hour. Are the segments being created prior to the end of the task duration?
j
Sorry to be so late looping back... maxRowsPerSegment - not being reached maxTotalRows - not being reached intermediatePersistPeriod - using the default task duration - using the default of 1 hour
Are the segments being created prior to the end of the task duration?
Yes, that's exactly what I'm saying. I'm curious why this would happen.
Ooooo, I wonder if 
maxRowsPerSegment
is counting all events including filtered events. We filter a lot of events in this datasource so even though we only persist < 1M per hour we are filtering >5M rows out.
b
It should not include those records.
How many segments is it creating during a single task execution?
j
Typically two
I reanalyzed the log files and I realized I'm mistaken. I misunderstood this log line:
Copy code
2024-09-19T18:00:20,500 INFO [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Persisted rows[565,122] and (estimated) bytes[357,901,652]
This is unrelated to the push. When I dug up the segment that was pushed I found it had 4.9M rows; so it's actually opening the segment upon reaching 
maxRowsPerSegment
as would be expected. Sorry for the confusion.
I ran a test: boolean flag over 260 million rows, 138 segment files and 266 GB of data. I saw no discernible difference between the queries on the long column with values 0 and 1 and the string column with values true and false.
b
That would be what I expect. Since a string 
true
/ 
false
field is going to be dictionary encoded the data is going to be mapped to a value, so under the hood essentially 
true/false
is going to be stored as 
0/1
.
j
That was also my impression. The part I wasn't sure about was the indexing strategy.
I should note that this was the first dimension in my dimension list. Now I would like to test the impact of putting it elsewhere in the dimension list (e.g. last).
Open in Slack
PreviousNext
Powered by