hi, We have moved today some of our indexing batch ingestion from hadoop to native, and are experiencing some sporadic errors (i.e they come and go, not consistent, so can't reproduce). In indexing logs we see for example

Unable to grant lock to inactive Task

- it seems to retry till

59

and eventually it fails and says

Check overlord logs for details.

and

Server error [400 Bad Request]; body: {"error":"Task [partial_range_index_generate_...] already exists."}

In overlord logs it seems like all errors are of type

Ignoring notification for already-complete task:...

When trying to follow all logs of failed indexing task - it looks like the task gets assigned to 2 indexers (?): on first one (by log time) it fails with

Server error [400 Bad Request]; body: {"error":"Task [partial_range_index_generate_..."}

and on second one it says

started

and even

Pushing .. to bucket ..

, but then errors

Process exited with code[1] for

Any idea on how to start checking it? we run version

27.0.0

Based on the information extracted from the RAG collection, it appears that you're encountering issues related to task locking and execution when transitioning from Hadoop to native indexing in Druid. Here are some potential causes and steps you can take to troubleshoot the errors you're facing: 1. Lock Conflicts: The message "Unable to grant lock to inactive Task" suggests that there might be lock conflicts in your indexing tasks. When two tasks attempt to acquire locks for overlapping time chunks, conflicts can occur. This can happen if multiple tasks are assigned to the same time chunk and the locking mechanism is unable to grant the requested locks. 2. Task Management: The error "Task [partial_range_index_generate_...] already exists" indicates that a task you're attempting to start is already running or has completed. Druid prevents duplicate task submissions for the same time chunk, which is why you are seeing this error. 3. Review Overlord Logs: Since your logs mention checking overlord logs for details, it's important to review them. Look for messages related to task state transitions and any conflicts in task execution. This might help you identify if tasks are being incorrectly flagged as complete or if there’s an issue with task retries. 4. Understand Indexing Context: The messages about "Ignoring notification for already-complete task" further suggest an issue with how your tasks are being managed by the Druid overlord. Ensure that your tasks have unique identifiers and that they are correctly configured to prevent overlap. 5. Configuration Settings: As a rule of thumb, review your Druid configuration, particularly those related to task locks and time chunks. Ensure you are using the correct settings for your new native ingestion tasks. 6. Experimental Features: If you are using experimental features like segment locking, be aware that these can have unknown bugs, as mentioned in your data source. Testing these features in a controlled environment might help determine if they are causing instability. 7. Version Considerations: Since you are running version 27.0.0, verify that your Druid version is compatible with the configurations you are using, specifically related to native ingestion. For further diagnostics, you could consider enabling additional logging to capture more detailed insights into your task executions and failures. Sources: - Task Locking and Configuration: tasks.md (internal document) - Indexing Task Management: upgrade-notes.md (internal document)