https://www.getdaft.io logo
Join Slack
Powered by
I'm getting a connection reset error periodically ...
# general
b
I'm getting a connection reset error periodically reading parquet from ADLS:
Copy code
daft.exceptions.DaftCoreException: DaftError::External Cached error: Unable to open file <abfs://path/to/my.parquet>: Error { context: Full(Custom { kind: Io, error: reqwest::Error { kind: Decode, source: hyper::Error(Body, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }) } }, "error converting `reqwest` request into a byte stream") }
With the job I mentioned above with 50k files it's happening often enough that the job always fails. Has anyone ever seen this?
k
Do you know if there is a specific file/set of files that this is occurring on, or if it happens on any file?
b
I've seen it happen on more than one file at least. These files are uniform in size / schema - it's a table stored as parquet files. I'm wondering if it's a transient error that needs to be retried.
k
It could be a retry-able error, Main thing that would be helpful from you is some sort of reproducible example if that's possible. But if that's not possible, I can add retries to a dev branch and publish it for you to try.
b
hmm that might be challenging to provide a reproducible example. I could also try developing against a local build, if that's not too difficult to get set up
k
It should be straightforward as long as you are able to install Python packages! I'll work on a build with retries, will likely get something you can try on Monday
b
thank you!
I'm almost there with running a ray job using a local build, but I'm getting this warning and then later an error related to request size limit:
Copy code
2025-10-20 13:20:28,479	WARNING packaging.py:405 -- File /Users/ben.cornelis/repos/Daft/wd/daft/daft.abi3.so is very large (165.49MiB). Consider adding this file to the 'excludes' list to skip uploading it: `ray.init(..., runtime_env={'excludes': ['/Users/ben.cornelis/repos/Daft/wd/daft/daft.abi3.so']})`
I ran 
make build-release
but the file is still large
k
@Ben Cornelis apologies for the delay. I'm trying to debug this issue and it looks like the Azure library that we use should already do retries on this error. However, if a retry occurred we would see a message like 
retry policy expired and the request will no longer be retried
in the error. Do you see that at all?
As for the build with 
make build-release
, you could try building without debug symbols to reduce the binary size.
👍 1
b
I just double checked and I don't see a log about a retry
here's the full stack trace in case it's helpful:
connection reset error stack trace.txt
connection reset error stack trace.txt
k
Since we vendor our own azure SDK library, I've modified it to panic on that error and I'm building a Daft dev version with those changes. That way we can get a Rust stack trace going as well which should make it easier to understand where this error is coming from
@Ben Cornelis you can install the dev build by running this command
Copy code
pip install daft --pre --extra-index-url <https://d1p3klp2t5517h.cloudfront.net/builds/dev/dceb27cb26e73bc73b432fb22586936203b127e1>
If you set the environment variable 
RUST_BACKTRACE=1
, you'll be able to see the Rust backtrace
b
thanks! I'll give that a try
using the dev build I get a new error immediately upon submitting the ray job:
Untitled.txt
Untitled.txt
daft.exceptions.DaftCoreException: DaftError::ComputeError Caught panic when spawning blocking task in the IO runtime: Context error called with kind: Io, error: environment variable not found, message: environment variable AZURE_FEDERATED_TOKEN not set)
It's odd because I have been able to have it download files - the error normally appears later in the job
k
Oh this errored earlier than I needed it to. Let me fix that
👍 1
New install instructions: 
pip install daft --pre --extra-index-url <https://d1p3klp2t5517h.cloudfront.net/builds/dev/2fc3d2e452f4f73ad7f66bc70b907f80f0bfec6a>
👍 1
b
I'm back to getting the original error with the python stack trace
k
No rust stacktrace at all?
b
not that I can see - same stack trace as last time
k
Hm, I am probably checking the wrong string. Let me fix that
🙏 1
Try this
Copy code
pip install daft --pre --extra-index-url <https://d1p3klp2t5517h.cloudfront.net/builds/dev/7c9f92fdf8a16fa0b22ba1d3d14d36d9f7423cda>
Thanks for sticking with me here
b
I really appreciate your help - that one worked
Untitled.txt
Untitled.txt
I'm not sure if it's relevant but that swordfish actor has logs for 4 threads with the same error
k
Hm ok, it's not going through the built-in retry flow which is weird. But what I'll do is add a retry on Daft's side at this point in the code
👍 1
Added retries to this build. Try it out and let me know if it succeeds
Copy code
pip install daft --pre --extra-index-url <https://d1p3klp2t5517h.cloudfront.net/builds/dev/6d25bb2c32bfbec515ee44689e031be35db4e44f>
b
thanks! I'll give it a try now
I see the errors are being retried and the first retry appears to succeed each time!
k
Great, if this solves your issue, I'll merge this in and we'll have this fix in the next official Daft release!
b
yes it does! thanks so much for your help
Open in Slack
PreviousNext
Powered by