https://linen.dev logo
Join Slack
Powered by
Hi Team, we are using druid 29.0.0. We suddenly s...
# general
s
Hi Team, we are using druid 29.0.0. We suddenly started seeing issues related to druid where all the tasks are being failed with in 5 minutes of its creation saying Incremental Persist Failed and its not able to delete the file. Its impacting 60% of all the ingestion, compaction tasks
Copy code
2024-08-30T13:39:59,947 ERROR [[index_kafka_vz_aug_masked_c363b3a66c73358_dhlfeoag]-appenderator-persist] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Incremental persist failed: 

java.lang.RuntimeException: java.lang.RuntimeException: org.apache.commons.io.IOExceptionList: /opt/druid/var/druid/task/slot0/index_kafka_vz_aug_masked_c363b3a66c73358_dhlfeoag/work/persist/vz_aug_masked_2024-08-30T00:00:00.000Z_2024-08-31T00:00:00.000Z_2024-08-30T03:25:35.441Z_73/0/tmpOutputFiles

Caused by: java.lang.RuntimeException: org.apache.commons.io.IOExceptionList: /opt/druid/var/druid/task/slot0/index_kafka_vz_aug_masked_c363b3a66c73358_dhlfeoag/work/persist/vz_aug_masked_2024-08-30T00:00:00.000Z_2024-08-31T00:00:00.000Z_2024-08-30T03:25:35.441Z_73/0/tmpOutputFiles

Caused by: org.apache.commons.io.IOExceptionList: /opt/druid/var/druid/task/slot0/index_kafka_vz_aug_masked_c363b3a66c73358_dhlfeoag/work/persist/vz_aug_masked_2024-08-30T00:00:00.000Z_2024-08-31T00:00:00.000Z_2024-08-30T03:25:35.441Z_73/0/tmpOutputFiles

Caused by: java.io.IOException: Cannot delete file: /opt/druid/var/druid/task/slot0/index_kafka_vz_aug_masked_c363b3a66c73358_dhlfeoag/work/persist/vz_aug_masked_2024-08-30T00:00:00.000Z_2024-08-31T00:00:00.000Z_2024-08-30T03:25:35.441Z_73/0/tmpOutputFiles/.nfs00000000000000e80008e0c0

Caused by: java.nio.file.NoSuchFileException: /opt/druid/var/druid/task/slot0/index_kafka_vz_aug_masked_c363b3a66c73358_dhlfeoag/work/persist/vz_aug_masked_2024-08-30T00:00:00.000Z_2024-08-31T00:00:00.000Z_2024-08-30T03:25:35.441Z_73/0/tmpOutputFiles/.nfs00000000000000e80008e0c0
does any one have any idea on this ?
➕ 1
j
Poking around at past issues with the "incremental persist failed" error, seems to be centered around the size and frequency of persists happening in the ingestion tasks. What are your parameters for: • maxRowsInMemory • maxBytesInMemory • intermediatePersistPeriod • numPersistThreads it might be easier to just post your supervisor spec. If the problem happens around 5 min after starting the ingestion tasks, then possibly the volumes are high and the persist buffer is large, making it difficult to persist. Fyi the default value for maxRowsInMemory for streaming ingestion has been dropped from 1m down to 150k ... so check that value first, you may want to adjust it if it is at the old default. rollup ingestion (rollup=true) is also more cpu intensive than non-rollup, if have that set. Also, because ioexceptions are mentioned, I assume you have already checked for IO and/or disk space issues on your servers. Thanks. John
s
Copy code
"tuningConfig": {
      "type": "kafka",
      "appendableIndexSpec": {
        "type": "onheap",
        "preserveExistingMetrics": false
      },
      "maxRowsInMemory": 1000000,
      "maxBytesInMemory": 0,
      "skipBytesInMemoryOverheadCheck": false,
      "maxRowsPerSegment": 1000000,
      "maxTotalRows": null,
      "intermediatePersistPeriod": "PT120M",
      "maxPendingPersists": 0,
      "indexSpec": {
        "bitmap": {
          "type": "roaring"
        },
        "dimensionCompression": "lz4",
        "stringDictionaryEncoding": {
          "type": "utf8"
        },
        "metricCompression": "lz4",
        "longEncoding": "longs"
      },
      "indexSpecForIntermediatePersists": {
        "bitmap": {
          "type": "roaring"
        },
        "dimensionCompression": "lz4",
        "stringDictionaryEncoding": {
          "type": "utf8"
        },
        "metricCompression": "lz4",
        "longEncoding": "longs"
      },
      "reportParseExceptions": false,
      "handoffConditionTimeout": 0,
      "resetOffsetAutomatically": true,
      "segmentWriteOutMediumFactory": null,
      "workerThreads": null,
      "chatRetries": 20,
      "httpTimeout": "PT30S",
      "shutdownTimeout": "PT80S",
      "offsetFetchPeriod": "PT30S",
      "intermediateHandoffPeriod": "P2147483647D",
      "logParseExceptions": false,
      "maxParseExceptions": 2147483647,
      "maxSavedParseExceptions": 0,
      "skipSequenceNumberAvailabilityCheck": false,
      "repartitionTransitionDuration": "PT120S"
    }
here is the full config. 1. we observed some tasks ran for 1 hour and failed (may be no data coming from that partition for an hour and data recieved after that) 2. seeing in 60% of the middlemanagers for all the tasks are failing (restarted and recreated the volumes)
supervisor.json
No space issues that i observed + able to create + and delete the files in that location
j
Okay then my first thought is to drop maxRowsInMemory down to 150k ... you also have it set to the same size as maxRowsPerSegment, which puts you in a sitation to build a segment after a single persist ... not 100% sure but seems like an area where the persist and build threads could ended up in some form of resource contention. So, first step, reduce MRIM and see if the problem goes away.
s
yes. we tried decreasing to 150k, its still showing the same issue unfortunately.
j
Hi Sivasai, can you post one of the ingestion logs? That will help to see the activity before and around the error.
s
issue is still there
@John Kowtko ^
We found the issue starts happening after upgraded the underlying hardware to rhel 8 and we also using nfs as storage. Does any one have any idea on how it’s affecting the Druid ? All other systems internally are working fine
Except Druid
n
Unfortunately we are also getting the same exception suddenly in our prod env. Most of the tasks are failing which causes loss of data in kafka ingestion.
s
Same in my case too !!
Got to know, Multiple people are facing this issue . Does any one have any idea on this ?
j
Can you try reducing intermediatePersistPeriod to PT1M (one minute) to force the persist earlier, to see if the error is actually occuring at the time of the persist?
s
yes. i have modified it, its causing during the same time.
j
Do you mean it is still failing after 5 minutes, in which some persists complete successfully in the first 4 minutes? Or is it now failing on the first persist, after one minute?
s
It’s failing After the first minute. Some times it’s able to persist for twice or thrice and fails while doing 4th one
j
Okay, then I don't know enough about Druid in this area to be of great help 😕 ... if you are using a non-standard deployment (e.g. K8s), can you try running it on standard linux with local disk to see if the problem still exists?
s
I’m sure that will work. We observed that out of 10 nodes, 5 nodes upgraded to rhel 8, so the pods on that nodes are failing always. Is there any operating system level requirement that Druid requires to run ? Or can you show some light on which specific resource or file system limits that I can compare between working and not working nodes ?
n
Hey @sriramdas sivasai not sure if you could bit set this value higher. I mean greater than taskDuration
I have done this change and just waiting for the outcome
So far I noticed that maximum tasks are successful
s
Mine is 2x of the duration
Open in Slack
PreviousNext
Powered by