https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
b
If it stays steady at 57%, could there be segments missing from deep storage? The coordinator log might have more information.
j
In addition to Ben's theory, two scenarios where I experienced this happening were (a) not enough disk space on the Historical nodes, and (b) not enough Historical cpu power, too many segments getting created by ingestion and compaction having a hard time keeping up.
s
Great. Thanks for replies. It helps. Ill continue to debug based on this.
@Ben Krug @John Kowtko may i know, how historical and coordinator connected in druid ? (we had some issues with coordinator being connected to mysql). do you think, does this causing this issue ? After waiting couple of days, i have restarted the historical by clearing the disks, Now the data availability in the historical is moving slow, After 2 days it loaded only 30 percent of the data. do you have any recommendations where to look at ?
j
If you have a large number of segments it could take a while to reload segments. First to confirm you don't have excessive fragmentation, can you tell us the size and number of segments in the datasources that have the largest number of segments?
s
we have two bigger datasources, first one - 1233935 segments - 41TB - loaded 89% second - 1469184 segments - 34 TB - loaded 35% we are worried about the second one
j
okay that seems like too much fragmentation ... rule of thumb is 500MB/segment ... = 2000 segments / TB ... so ideally your datasource should have 70-80k segments in them. What is the retention for these datasources? • If short (e.g. 7 days) then you may need to focus on generating fewer segments up front • If long (e.g. 90 days or longer) then you may just need to run compaction
s
yes i understand, we have retention of 30 days. Thinking, why first one is faster and second is slower and not loading
j
I don't know the loading algorithms in detail ... however there has been recent emphasis on 
smartSegmentLoading
, ref doc page: https://druid.apache.org/docs/latest/configuration/#dynamic-configuration ... so if you are not using that you might want to try it. I also recommend checking out this THREAD if you haven't seen it already.
If you are purposely not running compaction due to relatively short data retention, then I suggest looking at generating fewer, larger segments up front ... that is if your segment scan times aren't too long.
s
yea. w e have enabled the compaction. Its shows in the datasource page that compaction is running. does the compaction causing the issue to load the segment files in to historical (I mean. is it compacting while loading the in to historical from deep storage ) ?
j
Hi Sai, Auto-compaction is reindexing (i.e. completely replacing with a new set of segments) one time chunk at a time. What you are trying to achieve with compaction will have a significant impact on the datasource segments you end up ... including the number of segments per time chunk. Can you share the following info: • your auto-compaction spec • roughly how many segments there are in a time chunk before and after compaction? • your segment timeline chart showing segment count (see screenshot example) • how long it takes each compaction job to run • how often these compaction jobs are running Thanks. John
s
here is the compaction spec:
Copy code
{
  "dataSource": "voice_data",
  "taskPriority": 25,
  "inputSegmentSizeBytes": 100000000000000,
  "maxRowsPerSegment": null,
  "skipOffsetFromLatest": "P1D",
  "tuningConfig": {
    "maxRowsInMemory": null,
    "appendableIndexSpec": null,
    "maxBytesInMemory": null,
    "maxTotalRows": null,
    "splitHintSpec": null,
    "partitionsSpec": {
      "type": "range",
      "targetRowsPerSegment": 1000000,
      "maxRowsPerSegment": null,
      "partitionDimensions": [
        "site_id"
      ],
      "assumeGrouped": false
    },
    "indexSpec": null,
    "indexSpecForIntermediatePersists": null,
    "maxPendingPersists": null,
    "pushTimeout": null,
    "segmentWriteOutMediumFactory": null,
    "maxNumConcurrentSubTasks": 10,
    "maxRetry": null,
    "taskStatusCheckPeriodMs": null,
    "chatHandlerTimeout": null,
    "chatHandlerNumRetries": null,
    "maxNumSegmentsToMerge": null,
    "totalNumMergeTasks": null,
    "maxColumnsToMerge": null,
    "forceGuaranteedRollup": true,
    "type": "index_parallel"
  },
  "granularitySpec": null,
  "dimensionsSpec": null,
  "metricsSpec": null,
  "transformSpec": null,
  "ioConfig": null,
  "taskContext": null
}
j
Hi Sai, okay it looks like you may have very small rows, in which case you can up your segment target row count quite a bit. I am working with another cluster which has the target set at 20m ... and the segments are still well under 100mb in size. Not sure why your target row count is set to 1m ... 5m is generally the product default ... so maybe start by setting it to 10m, and see if you can get the row count down by a factor of 10 ... You can do this for both ingestion and compaction.
s
Okay thanks for the suggestion. Do you think, is this what causing the data availability issues? Since 3-6 days it’s still at 45 percentage
j
I'm not an expert on this, but you might want to look at ingestion as well ... If you are generating a large volume of segments continually, this may put a strain on the overall segment loading process, because new segments are constantly being created that have to be loaded. So if ingestion is creating lots of tiny segments, increase those size/row limits to allow ingestion to create fewer, larger segments ...
Open in Slack
PreviousNext
Powered by