hadoop ingestion job on Druid 25 failing at the en...
# general
k
hadoop ingestion job on Druid 25 failing at the end with below error . anyone faced similar issue ?
Copy code
{
  "ingestionStatsAndErrors": {
    "type": "ingestionStatsAndErrors",
    "taskId": "index_hadoop_datsource_dadkjpdk_2024-04-09T05:48:36.827Z",
    "payload": {
      "ingestionState": "BUILD_SEGMENTS",
      "unparseableEvents": null,
      "rowStats": {
        "determinePartitions": {
          "rowsProcessedWithErrors": 0,
          "rowsProcessed": 2554582,
          "rowsUnparseable": 0,
          "rowsThrownAway": 0
        },
        "buildSegments": {
          "rowsProcessedWithErrors": 0,
          "rowsProcessed": 2554582,
          "rowsUnparseable": 0,
          "rowsThrownAway": 0
        }
      },
      "errorMsg": "java.lang.RuntimeException: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException\n\tat org.apache.druid.indexing.common.task.HadoopIndexTask.runInternal(HadoopIndexTask.java:507)\n\tat org.apache.druid.indexing.common.task.HadoopIndexTask.runTask(HadoopIndexTask.java:297)\n\tat org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169)\n\tat org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477)\n\tat org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\nCaused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException\n\tat org.apache.druid.indexing.common.task.HadoopIndexTask.renameSegmentIndexFilesJob(HadoopIndexTask.java:594)\n\tat org.apache.druid.indexing.common.task.HadoopIndexTask.runInternal(HadoopIndexTask.java:472)\n\t... 8 more\nCaused by: java.lang.reflect.InvocationTargetException\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.apache.druid.indexing.common.task.HadoopIndexTask.renameSegmentIndexFilesJob(HadoopIndexTask.java:588)\n\t... 9 more\nCaused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found\n\tat org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2369)\n\tat org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2793)\n\tat org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)\n\tat org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)\n\tat org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)\n\tat org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)\n\tat org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)\n\tat org.apache.druid.indexer.JobHelper.renameIndexFilesForSegments(JobHelper.java:649)\n\tat org.apache.druid.indexing.common.task.HadoopIndexTask$HadoopRenameSegmentIndexFilesRunner.runTask(HadoopIndexTask.java:958)\n\t... 14 more\nCaused by: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found\n\tat org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2273)\n\tat org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2367)\n\t... 22 more\n",
      "segmentAvailabilityConfirmed": false,
      "segmentAvailabilityWaitTimeMs": 0
    }
  }
}