This message was deleted.

Segments stored in the Historical local segment cache are stored decompressed.

Thanks @Sergio Ferragut Appreciate your quick response. I was inspecting the segment metadata JSON, looks like the size information is misleading.

{
  "dataSource": "BL_AUTH_LOGS_FIXED",
  "interval": "2023-06-25T00:00:00.000Z/2023-06-26T00:00:00.000Z",
  "version": "2023-08-09T09:55:41.301Z",
  "loadSpec": {
    "type": "s3_zip",
    "bucket": "log-analytics",
    "key": "druid/segments/BL_AUTH_LOGS_FIXED/2023-06-25T00:00:00.000Z_2023-06-26T00:00:00.000Z/2023-08-09T09:55:41.301Z/0/index.zip",
    "S3Schema": "s3n"
  },
  "dimensions": "command_hostname,process_name,authentication_module,process_id,user_id,operation_message,event_type,user_name,session_open_time,session_close_time,source_ip,sudo_command",
  "metrics": "",
  "shardSpec": {
    "type": "numbered",
    "partitionNum": 0,
    "partitions": 0
  },
  "binaryVersion": 9,
  "size": 273567,
  "identifier": "BL_AUTH_LOGS_FIXED_2023-06-25T00:00:00.000Z_2023-06-26T00:00:00.000Z_2023-08-09T09:55:41.301Z"
}

I assumed the

"size": 273567

refers to the object

"key": "druid/segments/BL_AUTH_LOGS_FIXED/2023-06-25T00:00:00.000Z_2023-06-26T00:00:00.000Z/2023-08-09T09:55:41.301Z/0/index.zip"

in s3. But actually it is not.

Does that match the size of the decompressed files on the historical? I assume that the decompressed size is the important value for calculating space on Historical local storage which is a limited resource.

@Sergio Ferragut, yes the size looks similar in

/var/druid/segment-cache/

in Historical.

280K    2023-06-25T00:00:00.000Z_2023-06-26T00:00:00.000Z

So it is the Historical local segment cache. Not s3. But the metadata JSON says the s3 path. That's kinda confusing 🙂