Hi Team, Issue: Data Length Mismatch During S3 Ingestion in Apache Druid

Overview

We are facing an issue while ingesting a large dataset from S3 into Apache Druid. The ingestion process fails during the segment building phase with a data length mismatch error.

Error Message

java.lang.IllegalStateException: java.io.IOException: com.amazonaws.SdkClientException: Data read has a different length than the expected: dataLength=9404416; expectedLength=1020242891; includeSkipped=true; in.getClass()=class com.amazonaws.services.s3.AmazonS3Client$2; markedSupported=false; marked=0; resetSinceLastMarked=false; markCount=0; resetCount=0
at org.apache.commons.io.LineIterator.hasNext(LineIterator.java:108)
at org.apache.druid.data.input.TextReader$1.hasNext(TextReader.java:73)
at org.apache.druid.data.input.IntermediateRowParsingReader$1.hasNext(IntermediateRowParsingReader.java:60)
at org.apache.druid.java.util.common.parsers.CloseableIterator$2.findNextIteratorIfNecessary(CloseableIterator.java:74)
at org.apache.druid.java.util.common.parsers.CloseableIterator$2.next(CloseableIterator.java:108)
at org.apache.druid.java.util.common.parsers.CloseableIterator$1.next(CloseableIterator.java:52)
at org.apache.druid.indexing.common.task.FilteringCloseableInputRowIterator.hasNext(FilteringCloseableInputRowIterator.java:68)
at org.apache.druid.data.input.HandlingInputRowIterator.hasNext(HandlingInputRowIterator.java:63)
at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:95)
at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:891)
at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:500)

Context

• The error occurs while ingesting a large JSON file from S3. • Data read has a length of 9,404,416 bytes, while the expected length is 1,020,242,891 bytes. • The error happens in the BUILD_SEGMENTS phase. • The same large dataset is ingested successfully from our local Druid setup without any issues. • Other datasets of smaller sizes are being ingested successfully.

Questions / Request for Support

We are looking for guidance and support on the following points: 1. Is this a known issue when ingesting large files from S3 into Druid? 2. Are there recommended configurations or best practices to handle such issues? 3. Should we consider splitting files, adjusting timeouts, or configuring retries to better handle large file ingestion? 4. Are there troubleshooting steps, patches, or workarounds that can help resolve this problem?

Additional Information

• Druid version, ingestion spec, and sample files can be provided upon request. • We are happy to share more logs and configuration details as needed. Thank you for your support!

This does not look like a druid issue since you are able to ingestion the same file locally.

Thanks, this helps