We have a peculiar problem with our pipeline reading from Kafka, doing aggregation, writing them back to Kafka. These parts work fine. (Ok, sort of fine, it's not particularly efficient, but it doesn't break.) What does break is making checkpoints. Currently the instances only have shared S3 storage in the internal data center. It's currently failing, and it's giving the error

2023-06-16 11:13:05,718 WARN  org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory [] - Could not close the state stream for <s3p://some-bucket/flink.internal.checkpoints/18c1d780196f5badf2a4a83d7452d300/chk-1/24f92907-26eb-409d-bb73-da00443b4602>.
java.io.IOException: No space left on device

The partition where Flink is installed does not have much space space, and neither does

/tmp

, but we've tried moving everything to a partition that has a lot more storage, like this:

io.tmp.dirs: {{ flink_tmp_dir }}
process.working-dir: {{ flink_tmp_dir }}
web.tmpdir: {{ flink_tmp_dir }}
historyserver.web.tmpdir: {{ flink_tmp_dir }}
jobmanager.archive.fs.dir: {{ flink_tmp_dir }}
jobmanager.web.tmpdir: {{ flink_tmp_dir }}
jobmanager.web.upload.dir: {{ flink_tmp_dir }}
web.upload.dir: {{ flink_tmp_dir }}
env.log.dir: {{ flink_log_dir }}

S3 is not full. Nothing gets full locally as far as I can see with

df -h

. Any suggestions? Edit: I also tried installing Flink to a partition that has lots of space, but it didn't help. That partition is meant for data, so I moved it back.

One thing I could think of is that the account that you use doesn’t have all the necessary permissions. I actually can’t find in the docs which permissions you should have…

Do you mean S3 permissions?

Yes

Hmm. Both the Dell ECS S3 (managed by others) and the Minio (set up by me) behave the same way. I noticed I had misconfigured the user in minio; I'd forgotten to create the access key / secret key, and gave the user's name and password to Flink as those.

Looking at https://www.mail-archive.com/user@flink.apache.org/msg35825.html, it says

```As far as I know when uploading a file to S3, the writer needs to first

create some temporary files on the local disks. I would suggest to double

check all of the partitions on the local machine and monitor available disk

space continuously while the job is running.```

Could that be the case here as well?

I'll see if this help. Two snippets I copy-pasted: I did some analysis and realised that the temp files for s3 checkpoints are staged in /tmp although the *io.tmp.dirs *is set to a different directory. Could you try adding this to your flink-conf.yaml? s3.staging-directory:/usr/mware/flink/tmp