A backend engineering question: Suppose a user has been allotted 1 GB of storage space. He consumed 900 MB and then tries to upload 1GB of file. What would normally happen is, until the multi-part POST request is completed, the server will continue accepting the POST data and store it. And when the request is completed, it would check the file size and if it's exhausts subscription, it will deny the request and delete the stored file. What I really want is real time checking of storage limit and not wait for upload request to complete. How can I do that?

Depends on implementation. Require size before upload and reject using content length. This would prevent some usecases, which can be solved by cutting off the stream after receiving to the limit.

But can't content length header be spoofed by client?

It can be. What are you trying to protect against? Early abort ordinary use cases? Content length check is fine along with check at end. Malicious actor? You need to be more defensive

Cutting off the stream after receiving to the limit, how can I do this? In web framework like django, the request object is available only when, it request data is obtained fully.

https://www.aspsnippets.com/Articles/Check-Validate-File-Size-before-upload-using-HTML5.aspx Apparently it is quite trivial to pull off with HTML & JS Assuming you were asking for a Web based application. I would assume this is for better UX. So that I don't upload a 1 GB file for 15m just to be rejected. If the user is monkeying around, then it is their problem.

Thanks. However, we have apis for uploading data and they are public and can easily be known. So any malicious user can directly use the apis skipping the JS and HTML validation checks

how about this do a client side validation before hitting endpoint to upload with a separate endpoint which always gives the right balance available

But that separate endpoint is not private, it can easily be known. So a malicious user can directly call that endpoint, avoiding the validation checks on client side.

Ouch 😬 If your API is public then you should not be allowing such large file uploads to begin with. If you are passing the data through your server to S3, you'll end up paying for network costs as well. Very easy to DOS your servers as well. I think having some auth on it would be the first step.

Not super familiar with django, but they seem to expose a file-like interface for reading body in chunks: https://docs.djangoproject.com/en/4.0/ref/request-response/#django.http.HttpRequest.body