https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
p
j
that repository is archived
up to date chart can be found here
for the logging issue, if you want those logging files to push to stdout/stderr you'll want to set 
configVars.DRUID_LOG4J: '<?xml version="1.0" encoding="UTF-8" ?><Configuration status="WARN"><Appenders><Console name="Console" target="SYSTEM_OUT"><PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/></Console></Appenders><Loggers><Root level="info"><AppenderRef ref="Console"/></Root><Logger name="org.apache.druid.jetty.RequestLog" additivity="false" level="DEBUG"><AppenderRef ref="Console"/></Logger></Loggers></Configuration>'
p
Hi Jason, thanks for your reply. I will refer the latest one. However, the issue that I am facing is that the files are not getting cleaned from this folder 
druid.indexer.task.baseTaskDir
j
do you want the files to remain in the container?
like... at all... for any amount of time?
typically (but not always) in a k8s based setup that is not desirable
p
we want to delete files older than 10 days
j
ok, yea my solution above will not write any logs to disk at all, so that won't work for you
p
is there any way we can rotate the logs under 
druid.indexer.task.baseTaskDir
j
Copy code
By default, Druid uses RollingRandomAccessFile for rollover daily, and keeps log files up to 7 days. If that's not suitable in your case, modify the log4j2.xml accordingly.
seems like by default it should be rotating once a day for 7 days. does the log4j config inside your container match that example ?
or something near it at least
p
ohh okay, just noticed, however, the files are there since january 1st week.
j
Copy code
druid_indexer_logs_kill_durationToRetain=5400000
p
yes this is set to 1.5 hours
we tried with various settings, none of them worked
j
so... I'm not 100% on this
p
ideally this 
druid_indexer_logs_kill_durationToRetain
should delete the files under 
/task/
folder right?
j
but taht setting seems to be about 
task logs and entries in task-related metadata storage tables to be retained
Did you try 
druid.request.logging.durationToRetain
?
druid.request.logging.durationToRetain=P10D
for example
p
no haven't tried this yet
j
Copy code
druid.request.logging.durationToRetain	Period to retain the request logs on disk. The period should be at least longer than P1D.
p
I will setup this and check, one thing am not clear, is this not the task log directory 
druid.indexer.task.baseTaskDir
? According to https://druid.apache.org/docs/latest/ingestion/tasks.html#task-logs, the files under this directory should get pushed druid.indexer.log.directory? correct me if my understanding is wrong please.
j
in the logging link I pasted above, I see
Copy code
If you want to change the log directory, set the environment variable DRUID_LOG_DIR to the right directory before you start Druid.
but I don't know the specific answer to your question
it also might depend on your druid version.
p
Ok
v
@Prabir Choudhury did this get sorted out? The druid.indexer.task.baseTaskDir is not the place were logs are stored. So all the kill sittings don’t apply. This is a temp working directory for the task and the task should clean this up when it finishes
are you seeing any errors in the task log?
also please paste the task spec here
p
Hi Vijay, thanks for the information, here is the task spec
Copy code
{
  "type": "index_kafka",
  "id": "index_kafka_LTE_DIST_MEAS_810930d44d61e40_poojjhdo",
  "resource": {
    "availabilityGroup": "index_kafka_LTE_DIST_MEAS_810930d44d61e40",
    "requiredCapacity": 1
  },
  "dataSchema": {
    "dataSource": "LTE_DIST_MEAS",
    "timestampSpec": {
      "column": "measTime",
      "format": "iso",
      "missingValue": null
    },
    "dimensionsSpec": {
      "dimensions": [
        {
          "type": "double",
          "name": "binMax",
          "multiValueHandling": "SORTED_ARRAY",
          "createBitmapIndex": false
        },
        {
          "type": "double",
          "name": "binMin",
          "multiValueHandling": "SORTED_ARRAY",
          "createBitmapIndex": false
        },
        {
          "type": "double",
          "name": "count",
          "multiValueHandling": "SORTED_ARRAY",
          "createBitmapIndex": false
        },
        {
          "type": "double",
          "name": "cumCount",
          "multiValueHandling": "SORTED_ARRAY",
          "createBitmapIndex": false
        },
        {
          "type": "string",
          "name": "distName",
          "multiValueHandling": "SORTED_ARRAY",
          "createBitmapIndex": true
        },
        {
          "type": "string",
          "name": "extendedDistName",
          "multiValueHandling": "SORTED_ARRAY",
          "createBitmapIndex": true
        },
        {
          "type": "long",
          "name": "index",
          "multiValueHandling": "SORTED_ARRAY",
          "createBitmapIndex": false
        },
        {
          "type": "string",
          "name": "measEndTime",
          "multiValueHandling": "SORTED_ARRAY",
          "createBitmapIndex": true
        },
        {
          "type": "long",
          "name": "measInterval",
          "multiValueHandling": "SORTED_ARRAY",
          "createBitmapIndex": false
        },
        {
          "type": "string",
          "name": "measurement",
          "multiValueHandling": "SORTED_ARRAY",
          "createBitmapIndex": true
        },
        {
          "type": "long",
          "name": "totCount",
          "multiValueHandling": "SORTED_ARRAY",
          "createBitmapIndex": false
        }
      ],
      "dimensionExclusions": [
        "measTime",
        "__time"
      ]
    },
    "metricsSpec": [],
    "granularitySpec": {
      "type": "uniform",
      "segmentGranularity": "DAY",
      "queryGranularity": {
        "type": "none"
      },
      "rollup": false,
      "intervals": []
    },
    "transformSpec": {
      "filter": null,
      "transforms": []
    }
  },
  "tuningConfig": {
    "type": "KafkaTuningConfig",
    "appendableIndexSpec": {
      "type": "onheap"
    },
    "maxRowsInMemory": 1000000,
    "maxBytesInMemory": 1073741824,
    "skipBytesInMemoryOverheadCheck": false,
    "maxRowsPerSegment": 5000000,
    "maxTotalRows": null,
    "intermediatePersistPeriod": "PT2M",
    "basePersistDirectory": "/mnt/tmp/druid-realtime-persist6287816053949577185",
    "maxPendingPersists": 0,
    "indexSpec": {
      "bitmap": {
        "type": "roaring",
        "compressRunOnSerialization": true
      },
      "dimensionCompression": "lz4",
      "metricCompression": "lz4",
      "longEncoding": "longs",
      "segmentLoader": null
    },
    "indexSpecForIntermediatePersists": {
      "bitmap": {
        "type": "roaring",
        "compressRunOnSerialization": true
      },
      "dimensionCompression": "lz4",
      "metricCompression": "lz4",
      "longEncoding": "longs",
      "segmentLoader": null
    },
    "reportParseExceptions": false,
    "handoffConditionTimeout": 0,
    "resetOffsetAutomatically": true,
    "segmentWriteOutMediumFactory": null,
    "intermediateHandoffPeriod": "P2147483647D",
    "logParseExceptions": false,
    "maxParseExceptions": 2147483647,
    "maxSavedParseExceptions": 0,
    "skipSequenceNumberAvailabilityCheck": false
  },
  "ioConfig": {
    "type": "kafka",
    "taskGroupId": 0,
    "baseSequenceName": "index_kafka_LTE_DIST_MEAS_810930d44d61e40",
    "startPartitions": {
      "type": "end",
      "stream": "LTE_DIST_MEAS",
      "topic": "LTE_DIST_MEAS",
      "partitionSequenceNumberMap": {
        "0": 77728440232,
        "1": 3591020335
      },
      "partitionOffsetMap": {
        "0": 77728440232,
        "1": 3591020335
      }
    },
    "endPartitions": {
      "type": "end",
      "stream": "LTE_DIST_MEAS",
      "topic": "LTE_DIST_MEAS",
      "partitionSequenceNumberMap": {
        "0": 9223372036854776000,
        "1": 9223372036854776000
      },
      "partitionOffsetMap": {
        "0": 9223372036854776000,
        "1": 9223372036854776000
      }
    },
    "startSequenceNumbers": {
      "type": "start",
      "stream": "LTE_DIST_MEAS",
      "topic": "LTE_DIST_MEAS",
      "partitionSequenceNumberMap": {
        "0": 77728440232,
        "1": 3591020335
      },
      "partitionOffsetMap": {
        "0": 77728440232,
        "1": 3591020335
      },
      "exclusivePartitions": []
    },
    "endSequenceNumbers": {
      "type": "end",
      "stream": "LTE_DIST_MEAS",
      "topic": "LTE_DIST_MEAS",
      "partitionSequenceNumberMap": {
        "0": 9223372036854776000,
        "1": 9223372036854776000
      },
      "partitionOffsetMap": {
        "0": 9223372036854776000,
        "1": 9223372036854776000
      }
    },
    "consumerProperties": {
      "bootstrap.servers": "kafka:9092"
    },
    "pollTimeout": 100,
    "useTransaction": true,
    "minimumMessageTime": null,
    "maximumMessageTime": null,
    "inputFormat": {
      "type": "json",
      "flattenSpec": {
        "useFieldDiscovery": true,
        "fields": []
      },
      "featureSpec": {}
    }
  },
  "context": {
    "checkpoints": "{\"0\":{\"0\":77728440232,\"1\":3591020335}}",
    "useLineageBasedSegmentAllocation": true,
    "IS_INCREMENTAL_HANDOFF_SUPPORTED": true,
    "forceTimeChunkLock": true
  },
  "groupId": "index_kafka_LTE_DIST_MEAS",
  "dataSource": "LTE_DIST_MEAS"
}
v
Can you also paste one of the task logs here?
j
@Prabir Choudhury so did the modifications from yesterday not work ?
Open in Slack
PreviousNext
Powered by