Since we just recently fully migrated to Lucee, I wanted to see if there was anything built-in goodness that might help with a problem I'm working to solve. We have a reporting engine that can end up producing really large amounts of data. Today we're buffering the output into memory, but that can lead to JVM memory issues, so I was going to refactoring things to buffer the output to file system instead. I know I can just append to a file or use Java to create a file stream, but I wasnt' sure if there was anything more baked into Lucee which I should be looking at using.

what you describe is pretty much the standard solution

@zackster Thanks. That's what I figured, but wasn't sure if there was something backed in to address this, like some way to just send the output buffer Lucee builds to the FS instead of using RAM and then have it push the buffer from the FS upstream. I could see something like that useful for

<cfprocessingdirective />

. That would be super useful for pages which memory consumption could be extremely high.

That's a compile-time statement, isn't it?

something like savecontent which streams to a file?

I believe so, but that might still be correct tag. However, maybe it calls for something different.

i'd still probably just write directly to a file, given whitespace annoyances

@zackster Not really. I see this as like a request-based thing. Basically anything that's going into the content/output buffer stream would be streamed to disk and not to RAM and then when time to push to be sent upstream would stream from the FS.

could play with https://github.com/lucee/Lucee/blob/6.0/core/src/main/java/lucee/runtime/writer/CFMLWriterImpl.java#L318

I was just thinking this could be super handy in some cases.

Trying to make sense of the

fileWrite

docs. Is it all-or-nothing, or can one write repeatedly to the same file handle?

@zackster, it looks like the CFMLWriterImpl would be the place the functionality would go. Basically instead of building a StringBuffer, content would go to the FS (but probably only if it reach some threshold, like 1MB-10MB).

But that could just be the usual imprecision of Adobe's doc-writing skillz

The idea would be to try and help preserve JVM OutOfMemory issues on requests which you know could produce large amounts of output.

yeah, still savecontent seems a better fit tho

@zackster, not IMO. I'd like to see it request based. Because then you could even just do something globally in a application, like "if the output buffer is > 5MBs, use the the FS"

so we could make the

config.getCFMLWriter

configarable per pageContext, at the moment it's basically per context

It's just a thought, but I could see that being useful and give developers an easy way to protect their apps from OOM issues (at least those caused by the output being generated).

any reason you can't just flush along the way?

In this particular case, yes, because it renders the content in the background. Because the reports can take minutes to run, they run background and then served up once they finish running.

ok, can u go and aṡk on the mailing list and I'll get micha to have a squizz there

@zackster, Here's my post: https://dev.lucee.org/t/streaming-cfs-output-buffer-to-filestream-to-reduce-outofmemory-issues/11496

ta, i have pinged micha

OK, so now I see in your original you said you didn't want to append to a file - which CFML already does - but you want some other mechanism to arrive at the same result? Um... why?

@Adam Cameron I never said I didn't want to. I just asked if there was some baked in goodness that I might be missing. I didn't want to re-invent a wheel.

I honestly don't mean to seem obtuse, but I can't see how "use the function that is specifically for what you want to do" isn't the most baked-in option one could have? Does Lucee hold the whole file in memory as it's being appended to or something?

@Adam Cameron I'm not talking about file functions. I'm talking about buffer output that Lucee builds.

(I'm seriously not trying to be awkward or contrary. For once ;⁠-⁠)

Today when a process runs, everything that's generated as output to send to the browser gets build in memory. If a page is generating output that's extremely large, this can lead to OOM issues. I was asking if there was mechansim under the hoods, where Lucee would buffer that to the FS instead of buffer all to memory.

fileOpen() i suspect

So that instead of trying to store 100s of MBs in a StringBuilder, it would start streaming that to the file system instead. And then when it was time to send the output buffer, it reads directly from the FS.

sure. not clear how another BIF would compose the various file streaming operations though. afaik, that's about as ground level you can get with writing to files

@websolete I'm not asking about how to manage files directly.

ok, clearly i'm being annoying then. i wasn't implying either of those things. bowing out

@websolete I'm talking about how Lucee handles the output buffer that's generated in a request.

Gotcha. So you just blatter yer report content to the output buffer, and you want Lucee to handle that somehow for you.

I can't use CFFlush because this is running in a background job.

If it's run the background, where's the output going then? Why are you putting the stuff in the output buffer in the first place? At some point - for any solution to work - you are going to need to emit the data in a streamed fashion. Sounds like that's what yer problem it. It almost sounds like you are asking "I am generating this huge string that causes OOM issues. How can Lucee help me?". It can't. You have to not generate the huge string [etc] in the first place.

anyone else feel the urge to abuse

repeatString()

after reading this thread for shits n giggles?