Hello, question for S3 streaming ingestion: - Job ...
# troubleshooting
m
Hello, question for S3 streaming ingestion: • Job Manager will S3 ls and provide task managers list of files [a, b, c, d] • Task Managers will split list • There's a race between when the task manager actually goes to open the file and if it exists - if the file has be deleted it'll put the job in a "crash loop" Currently we're clearing state to get "rid" of the missing file, but is there an easier way to swallow FileNotFoundExceptions?
Copy code
private static TrackingFsDataInputStream openStream(
            final Path file, final Configuration config, final long seekPosition)
            throws IOException {

        final FileSystem fs = file.getFileSystem();
        final long fileLength = fs.getFileStatus(file).getLen(); <---

        final int fetchSize =
                MathUtils.checkedDownCast(config.get(StreamFormat.FETCH_IO_SIZE).getBytes());
        if (fetchSize <= 0) {
            throw new IllegalConfigurationException(
                    String.format(
                            "The fetch size (%s) must be > 0, but is %d",
                            StreamFormat.FETCH_IO_SIZE.key(), fetchSize));
        }