https://discord.cloudflare.com logo
Join Discord
Powered by
# r2
  • v

    Vitali

    05/01/2022, 3:43 PM
    Then whatever it turns out to be, I’m right
  • e

    Erisa | Support Engineer

    05/01/2022, 4:07 PM
    Are there any tips for retrieving very large files with a worker? (To make them publicly accessible) Currently getting an expected ```json "exceptions": [ { "name": "TypeError", "message": "Memory limit would be exceeded before EOF.", "timestamp": 1651421012992 } ],```and curious if theres anything that can be done to make it work better, because I remember with Workers in the past I was able to stream very large files from origin without issue
  • i

    Isaac McFadyen | YYZ01

    05/01/2022, 4:08 PM
    Is that streaming?
  • i

    Isaac McFadyen | YYZ01

    05/01/2022, 4:08 PM
    Streaming files shouldn't touch RAM.
  • i

    Isaac McFadyen | YYZ01

    05/01/2022, 4:08 PM
    With streaming you should just be able to return the
    await env.BINDING.get("key")
    directly, I think.
  • e

    Erisa | Support Engineer

    05/01/2022, 4:08 PM
    I'll be honest I stole most of the code from James
  • e

    Erisa | Support Engineer

    05/01/2022, 4:08 PM
    Lemme check what I'm doing
  • i

    Isaac McFadyen | YYZ01

    05/01/2022, 4:08 PM
    Oh 😄
  • i

    Isaac McFadyen | YYZ01

    05/01/2022, 4:08 PM
    K.
  • i

    Isaac McFadyen | YYZ01

    05/01/2022, 4:09 PM
    James probably assumed it would be used for like 1mb images, not 1gb files.
  • e

    Erisa | Support Engineer

    05/01/2022, 4:09 PM
    file = await env.R2_BUCKET.get(id);
    and then later
    Copy code
    js
            response = new Response(await file.arrayBuffer(), {
                headers: {
                    'cache-control': 'public, max-age=604800',
                    'content-type': file.httpMetadata?.contentType,
                    'etag': file.httpEtag,
                },
            });
  • i

    Isaac McFadyen | YYZ01

    05/01/2022, 4:09 PM
    Yeah, that's the problem.
  • e

    Erisa | Support Engineer

    05/01/2022, 4:09 PM
    yeah and thats fair, im just asking what i should be doing in this situation
  • i

    Isaac McFadyen | YYZ01

    05/01/2022, 4:09 PM
    If you do an
    arrayBuffer
    it buffer it in memory.
  • j

    James

    05/01/2022, 4:09 PM
    In theory, you should be able to pass the
    R2Object.body
    directly to a
    new Response()
    since it's a readable stream, but that never actually worked for me. So in my code I do an inline
    await R2Object.arrayBuffer()
    , which will buffer it all.
  • i

    Isaac McFadyen | YYZ01

    05/01/2022, 4:09 PM
    Oh, hmm...
  • e

    Erisa | Support Engineer

    05/01/2022, 4:09 PM
    oh interesting
  • j

    James

    05/01/2022, 4:10 PM
    Which will work for ~128MB, but anything beyond that will fail, yeah
  • i

    Isaac McFadyen | YYZ01

    05/01/2022, 4:10 PM
    ^ Or less, Workers share memory if there's a couple invocations.
  • i

    Isaac McFadyen | YYZ01

    05/01/2022, 4:10 PM
    From what I've heard it can be as little as a few kb if it's under really heavy load.
  • e

    Erisa | Support Engineer

    05/01/2022, 4:10 PM
    using the body seems to work for me, at least on
    wrangler dev
    with wrangler1
  • e

    Erisa | Support Engineer

    05/01/2022, 4:11 PM
    its streaming a multi GB file currently
  • j

    James

    05/01/2022, 4:11 PM
    just
    new Response(file.body, ...)
    ?
  • e

    Erisa | Support Engineer

    05/01/2022, 4:11 PM
    Copy code
    js
            response = new Response(file.body, {
                headers: {
                    'cache-control': 'public, max-age=604800',
                    'content-type': file.httpMetadata?.contentType,
                    'etag': file.httpEtag,
                },
            });
  • j

    James

    05/01/2022, 4:11 PM
    fascinating. That didn't work at all for me, at least on smaller files when I tested 🤔
  • e

    Erisa | Support Engineer

    05/01/2022, 4:11 PM
    just replaced the arraybuffer in the above which was taken from you
  • e

    Erisa | Support Engineer

    05/01/2022, 4:11 PM
    the smaller files work too
  • e

    Erisa | Support Engineer

    05/01/2022, 4:11 PM
    i can try publishing and see if it breaks there
  • i

    Isaac McFadyen | YYZ01

    05/01/2022, 4:12 PM
    Were you accidently awaiting? Or was that pretty much the exact code you were trying?
  • j

    James

    05/01/2022, 4:12 PM
    yeah lemme know if it works in prod and I can update my examples
1...979899...1050Latest