This message was deleted.

maybe the problems is way too many small segments, so your historicals have too many metadata structures to handle? Do you know how the average amount of rows per segment you have? for basic cluster tuning, check here

The memory mapping that Historicals do for their local cache of segments is OS driven. It uses that memory left over on the OS after the historical takes its Heap and max Direct Memory allocations. What error are you getting ? Perhaps the Historical settings are too high overcommitting the resources. What nodes are you running on? What are your historical JVM and config settings. Are the Historicals running on their own on the node?

historical run on i3en.3xlarge ec2 node as docker container histrocial config (druid version 0.19.0)

- Name: AWS_REGION
  Value: us-east-1
- Name: druid_historical_cache_useCache
  Value: true
- Name: druid_historical_cache_populateCache
  Value: true
- Name: druid_segmentCache_lazyLoadOnStart
  Value: true
  # set to 6.5 TB = 7146825580544 bytes (it should be smaller then /opt/data total disk space )
- Name: druid_cache_sizeInBytes
  Value: 7146825580544
- Name: druid_server_maxSize
  Value: 7146825580544
- Name: druid_segmentCache_locations
  Value: '[{"path": "/opt/data", "maxSize": "7146825580544"}]'
- Name: druid_segmentCache_locationSelector.strategy
  Value: mostAvailableSize
- Name: DRUID_XMX
  Value: '35g'
- Name: DRUID_XMS
  Value: '35g'
- Name: DRUID_MAXDIRECTMEMORYSIZE
  Value: 65536m
  #Value: 13312m
- Name: druid_processing_buffer_sizeBytes
  Value: 536870912
- Name: druid_processing_tmpDir
  Value: '/opt/data/processing/tmp'

segment size avg - from router - 54.27 MB segment number of rows - dont know Error is taking place after succed to load 98% of segments ERROR -

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f9b06001000, 65536, 1) failed; error='Not enough space' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.

Did that work? Here's an old thread by Vijay, reducing segment count (eg compacting) is ideal. Otherwise, might have to increase vm.max_map_count (eg, see here). VDid that work? Here's an old thread from

it still on loading so wait to see if the delete old works ..

historicals will load whatever segments are assigned to them by coordinator. Best to compact to fewer segments, or maybe add historicals. I'm not sure on what to set max_map_count to, just that it's currently too low for the number of segments you're loading.

• Druid UI provides a lot of statistics as to how many segments are generated per partition date and how many rows in segment file. This info will help you tune how many segment files are ideal. • Historical data itself is a cache. It will load everything you tell it to load in the retention rules.

@Ben Krug - after fix retention and delete older segments historical succed to fully load segments from disk ..

The target segment row is 5 M or 300-700M per segment

WOW .. this is my ingestion spec ..(config part) "tuningConfig": { "type": "index_parallel", "maxRowsPerSegment": 5000000, "maxRowsInMemory": 150000, "maxBytesInMemory": -1, "maxTotalRows": null, "numShards": null, "splitHintSpec": null, "partitionsSpec": { "type": "dynamic", "maxRowsPerSegment": 5000000, "maxTotalRows": null }, "indexSpec": { "bitmap": { "type": "roaring", "compressRunOnSerialization": true }, "dimensionCompression": "lz4", "metricCompression": "lz4", "longEncoding": "longs", "segmentLoader": null }, "indexSpecForIntermediatePersists": { "bitmap": { "type": "roaring", "compressRunOnSerialization": true }, "dimensionCompression": "lz4", "metricCompression": "lz4", "longEncoding": "longs", "segmentLoader": null }, "maxPendingPersists": 0, "forceGuaranteedRollup": false, "reportParseExceptions": false, "pushTimeout": 0, "segmentWriteOutMediumFactory": null, "maxNumConcurrentSubTasks": 2, "maxRetry": 1, "taskStatusCheckPeriodMs": 1000, "chatHandlerTimeout": "PT10S", "chatHandlerNumRetries": 5, "maxNumSegmentsToMerge": 100, "totalNumMergeTasks": 10, "logParseExceptions": false, "maxParseExceptions": 2147483647, "maxSavedParseExceptions": 0, "buildV9Directly": true, "partitionDimensions": [] } }, "context": { "forceTimeChunkLock": true },

you may need to enable auto-compation just check with the data people how often the late data arrives, to delay the auto-compation jobs by that time, so the compation jobs don't fail often

ok , we have a schedule hourly job that execute every hour on the 20 minutes .. 1:20 / 2:20 / 3:20 it take about 25 minutes to finish and create new segments .,.

what version of druid are you running?

currently 0.19.0 , and working to upgrade to 25.0.0

oh, I don't remember if the auto-compation exists at 0.19

it is .. I set something .. but I dont know how to explicit controll the time it start

usually, I use the web-ui to configure the auto-compation

view on 19

sometimes, when the number of very small, and very old segments it too high , druid chooses to compact more bytes instead (from the recent ones) so a manual compaction job can help to reduce the number of segments (just don't do it for the full period in one go, do for each day)

it is executing every 30 min .. by default ..

in the job log, you can read to check how many segments it reduces (and what intervals it is operating on)

I see , and the merged segments are flag as "unused" ?

the old one yes, but there's a period in which the segments exists simultaneously, the coordinator waits for the historicals to be ready before swapping and removing the old ones

Thanks!!

I don't think so, they are doing they job right!

so how I set the execution time for the auto-compact ..?

Not sure what version introduced it, but with autocompaction you can control the delay with

skipOffsetFromLatest

expressed as a period (e.g. PT1H )

I will check this .. "offset" meaning time offset ..

it stands for time, I think... it is used for hour, minute, seconds periods. P1D is a day, P1M is month, PT1M is a minute.

cool , thanks!!

You can use Request Logging. There is an option to use SLF4J to output to logs.

Very good !!