http://coldfusion.com logo
Join Slack
Powered by
Urk. We're being bitten by Lucee's approach to onl...
# lucee
a
Urk. We're being bitten by Lucee's approach to only installing packages as they're needed, rather than having them there @ the outset. I'll reiterate that this is completely not viable / appropriate / workable in a production environment. But... well... [sigh]. Someone posted what's needed to handle all this when building the docker image, rather than at runtime? I think it might have been @Mark Drew (he/him)? or @jamiejackson? Can't find it it here now. Can someone pls remind me? Cheers.
m
I do a warmup before hand. this can be done via: 
RUN LUCEE_ENABLE_WARMUP=true ${CATALINA_HOME}/bin/catalina.sh run
⭐ 1
It then gets all the things it needs, creates all the folders, deploys all extensions etc etc.
And then shuts down
a
Cheers man.
👍 1
Hey, just to be clear (I'm reading https://github.com/lucee/lucee-dockerfiles/issues/68, having googled 
LUCEE_ENABLE_WARMUP
), I don't need to tell Lucee what to warm? I'm not adding any external modules or anything, I'm just wanting frickin 
encodeForHtml
to frickin work when we call it (etc).
What you suggested above is Lucee-ese for "fuck sake Lucee, will you install yourself properly"?
m
Well, lucee "installs" itself when it starts. i.e. expands and gets what it needs.
a
I'm getting the error when it encounters code in our codebase. After start-up.
ie: it's not erroring out until it hits some code that is trying to run 
encodeForHtml
Because the frickin ESAPI module isn't installed.
Because the server is not exposed to the outside world.
m
Oh interesting. so yeah, add that module... is it an extension or what?
a
encodeForHtml
So... native CFML.
The log entry says:
Copy code
In the OSGi Bundle with the name [esapi.extension] and the version [2.2.4.7] was no class with name [org.lucee.extension.esapi.functions.EncodeForHTML] found
Not sure what that might mean if it was presented in coherent English. Also this gets logged:
Copy code
org.osgi.framework.BundleException: Unable to resolve esapi.extension [70](R 70.0): missing requirement [esapi.extension [70](R 70.0)]
Hrm... also reading this... https://github.com/isapir/lucee-docker/blob/master/README.md
omcat is launched with the environment variables 
$LUCEE_ENABLE_WARMUP
and 
$LUCEE_EXTENSIONS
, so that Lucee does an initial run, creates all required directories and files, and downloads extensions if specified.
So I need to have 
$LUCEE_EXTENSIONS
specified for it to do anything? I hasten to add, I'm not trying to load an "extension" (something third-party), I'm trying to get native CFML functionality to work. Stuff that worked fine without all this shite in 5.3, but is now broke in 5.4...
m
You dont need extensions var . It will look in there if it needs it
a
I should not need to do this with a full Lucee image. With lucee-light? Sure. Not with the stock image. I should all be done already.
m
I dont think I have anything in there, checking
a
thanks man
So it kinda knows what stuff it hasn't installed already..?
m
Essentially what I do to install (an internal non web viewable version before the security maevns jump all over me)
Copy code
ADD  <https://cdn.lucee.org/lucee-${LUCEE_VERSION}.jar> ${CATALINA_BASE}/lib/ext/lucee-${LUCEE_VERSION}.jar

RUN echo "Setting password to ${LUCEE_PASSWORD}"
RUN mkdir -p /www/lucee/lucee-server/context && echo "${LUCEE_PASSWORD}" > /www/lucee/lucee-server/context/password.txt

COPY ./lucee_extensions /www/lucee/lucee-server/deploy
# Warmup to install extensions
RUN LUCEE_ENABLE_WARMUP=true ${CATALINA_HOME}/bin/catalina.sh run
This is on top of : 
FROM tomcat:9.0.68-jre11-temurin-jammy
a
(I'm using 
5.4.1.8-nginx
from Lucee) OK, but you've got that 
COPY ./lucee_extensions /www/lucee/lucee-server/deploy
step too.
m
I mean, you dont need it if you dont have em 🙂
a
yeah cool, I'm still trying to clarify if I need to tell Lucee "install your own ESAPI functionality" (and whatever else it's decided to not load between 5.3 and 5.4), by putting [something] in a dir like that. Or whether that's for additional extensions, and the extensions for native CFML stuff will be handled automagically provided I do the pre-warm thing just by itself.
The docs are not clear. It just talks about "extensions", and clearly there's divergence here as to what I think is an extension, and what Lucee thinks is an extension.
OK so... whilst trying to work around this (but before changing anything)... I rebuilt the containers in question. And... problem gone away. !!!! Not sure what to make of that. But for now... I'm gonna back away slowly like... after a a bit of distance I'm gonna turn and run... and never speak of this again.
* reserving right to come back to it if it shows up again...
m
LOL... so it might be that the new containers are not caching some layers that were cached before?
a
maybe?
m
there is a great utility called 
dive
that shows you the layers of an image you have built. So you can see what goes in at each stage https://dev.to/cloudx/analyzing-the-docker-layers-with-dive-5e7o
a
oooh
j
i'll reiterate that i'm not satisfied with the way that lucee defers installation of some extensions/resources until runtime. we're well into the age of containers and we need to be able to easily build at build time. runtime must be bulletproof--it's not a time of "let's see what else lucee can lazy install." yes, i can jump through hoops to create/run/destroy a mini-application in my dockerfile that calls extension/resource functions, but it's messy and shouldn't be necessary. before i had one of these messy arrangements for ORM, i would have intermittent (and undetectable) issues with builds (so much for repeatability). like maybe 1-5% of the time, i'd have a "dud" image. i wonder if you're running into the same thing with ESAPI. anyway, this: https://cfml.slack.com/archives/C06TA0A9W/p1688629006097909?thread_ts=1688577016.525639&amp;cid=C06TA0A9W
a
100% with you on that @jamiejackson.
j
it just dawned on me that we got bitten by the ESAPI issue, too, today. hah!
a
In my case it could have been pebcak or some transient think though, as a container rebuild cleared it. BUt this in itself maybe demonstrates the fragility of the approach. It's just not an appropriate way of managing modules, IMO.
j
we had three dud images on dev over the last few days because of encodeforhtml
a
It really has a feel of "Lucee is designed for devs' machines" or something, to me
j
In my case it could have been pebcak or some transient think though,
i don't think so, i think you'll see the same issue for a small percentage of your builds.
that's the major problem. you can get dud images and it's not your fault, and it's very intermittent
i suspect i know the workaround because it's the same workaround i do for ORM already
a
It was also a shit change to introduce in a jump between x.3 -> x.4
j
i only started using the light image very recently, where ESAPI would start to matter more because it's not stock in the light image.
so if the solution is like that of orm, what i'll need to do is: in dockerfile: • specify the plugin in the LUCEE_EXTENSIONS var • do 
RUN LUCEE_ENABLE_WARMUP=true ${CATALINA_HOME}/bin/catalina.sh run
to half 😕 install it • copy a mini application to the web root which calls 
encodeForHtml()
• loop a curl on the above application (e.g., localhost:8888/myLittleGoofyApp.cfm) until it stops throwing errors • remove the mini-application • continue with the docker build
only bullets 1, 2, and 6 are acceptable. the rest are not. (if this turns out to be like ORM)
oh, and hello, @richard.herbert
👋 1
It really has a feel of "Lucee is designed for devs' machines" or something, to me
to me, it has the feel of an installer that was born in the era before IaC and isn't quite there yet.
the ability to set the password in IaC , the `LUCEE_ENABLE_WARMUP`mechanism (and the upcoming 
cfconfig.json
configs) help a lot, but the deferred installation is still a headache.
a
OMG that bulleted list is tragic.
(thanks for the info though)
j
i've just implemented it in my Dockerfile. i don't want to risk it cropping up in prod.
warmup_esapi_extension.sh
warmup_esapi_extension.sh
Dockerfile
Dockerfile
i think we also got bitten by the same thing with the image extension:
Copy code
No matching function [ISIMAGEFILE] found;lucee.runtime.exp.ExpressionException: No matching function [ISIMAGEFILE] found
@zackster, do you have any insight into determining which extensions might have intermittent landmines if you don't call their functions within the build? so far, i know ORM, ESAPI (and now i think Image) need special treatment to prevent intermittent issues with the extension/resource fully getting installed. i believe we're talking about very intermittent issues, here (like 1 out of 100 deployments might become problematic, through no fault of the build's setoup), so i need to understand the extension installation/resource theory. (since it takes a long time to prove empirically.)
btw, i only used to have to deal with this for ORM but now that i've moved to the light image, other landmines seem to be appearing.
otherwise, i think the only way to reliably install extensions via IaC will be to jump through hoops with every extension: https://cfml.slack.com/archives/C06TA0A9W/p1691186877631149?thread_ts=1691140244.277319&amp;cid=C06TA0A9W
@bdw429s, might i have to worry about this for the Ortus Redis extension, for instance? (tl;dr: is the extension installation complete complete when it's installed or is there more "installation" that happens when you call it from code?)
i'm using the following extensions and I might have to retrofit my Dockerfile to jump through the hoops of calling extension functions in order to prevent runtime issues, but I"m unsure whether they all have potential issues or not: • S3 Resource Extension- Unknown installation "completeness" at installation time. • Ajax Extension- Unknown installation "completeness" at installation time. • Compress Tags- Unknown installation "completeness" at installation time. • ESAPI extension - Only half-installs when you "install" it. Need to call a function to fully install it. • Hibernate ORM Engine - Only half-installs when you "install" it. Need to call a function to fully install it. • Image extension - Only half-installs when you "install" it. Need to call a function to fully install it. • PDF Extension - Unknown installation "completeness" at installation time. • Ortus Redis Extension- Unknown installation "completeness" at installation time.
b
might i have to worry about this for the Ortus Redis extension,
@jamiejackson I'm not quite sure what "this" is (just got back from holiday and only skimmed the thread), but if "this" is extensions not actually bundling the OSGI bundles they require and expecting Lucee to automatically download what it needs later, then no you shouldn't need to worry about it. We bundle all the jars needed in the extension.
👍 1
I'm not aware of why any Lucee extension wouldn't be full installed otherwise, unless Lucee made some sort of recent changes to how it installs extensions.
j
@bdw429s,let's take the orm extension, for instance. you can "install" it with the following in your dockerfile:
Copy code
ENV LUCEE_EXTENSIONS ${LUCEE_EXTENSIONS},FAD1E8CB-4F45-4184-86359145767C29DE;name=Hibernate ORM Engine;version=3.5.5.89
RUN LUCEE_ENABLE_WARMUP=true /usr/local/tomcat/bin/catalina.sh start
but that doesn't really completely install it. you have to actually call an orm function before it truly completes the installation. in other words, it defers (complete) installation until runtime.
i don't understand the rationale for that behavior but that seems to be what happens with certain extensions.
good to know that i don't have to do anything special with the Ortus Redis extension, tho. so far, i've been bitten by the (stock) ORM extension, ESAPI, and Image extensions, so i have to do extra stuff in the build to really, truly install them.
(i would love to be proven wrong on this topic, by the way, but experience leads me to my conclusion.)
this old thread hints at the behavior (i think?): https://dev.lucee.org/t/install-extensions-on-demand/277/2?u=jamie_jackson
b
Right, I mean-- I get the overall general concept that something somewhere somehow is not "installed", but I don't know what that means.
Like, specifically on a technical level what is not happening
I assumed the actual lex archive did not bundle the OSGI jar/version that the extension tried to load, forcing Lucee to download it on demand, but that was just a guess
If it's something else, then it's possible it could affect our extension, but I'd need to know exactly just what Lucee is waiting to do until a BIF is called
j
i'd like to know what exactly goes on too, but the symptom, for say ORM is that you get a "No ORM Engine installed!" for maybe 1% of your deployments if you don't jump through hoops in your build to call an orm function.
and i'm finding similar issues for other extensions (e.g., ESAPI and Image) because i just started running the "light" docker image within the last few weeks.
(those latter extensions had been stock in the fat version)
b
What's also puzzling to me is that, from what I've been told, even when Lucee does install an extension or download a bundle, it's supposed to be on the fly and ready right away for that first request. But who knows if that works correctly.
j
it doesn't seem to work correctly 100% of the time. just 99% of the time.
b
Well, another interesting thing to note, is Lucee "fat" doesn't come with the extensions "installed" per se, just with • the lex files bundled in the jar • the extension IDs in the core manifest file saying they are required To my understanding, even Lucee "Fat" still installs them all on first run.
On first start, the entire Lucee home is essentially built and filled out from scratch regardless of Lucee lite or not
j
somehow or another, i never had a problem with ESAPI and Image when they were stock/bundled, whatever in the fat version.
b
Perhaps the download is failing. I'm unclear if you're allowing Lucee to download the lex's for you, or if you are sticking them into the available folder yourself prior to start.
I would recommend turning up the log level on the deploy log, and capturing the contents of that file (server and web context) when Lucee fails to look for clues.
j
for installation, i'm literally doing this. with an ENV line per extension:
Copy code
ENV LUCEE_EXTENSIONS ${LUCEE_EXTENSIONS},FAD1E8CB-4F45-4184-86359145767C29DE;name=Hibernate ORM Engine;version=3.5.5.89
RUN LUCEE_ENABLE_WARMUP=true /usr/local/tomcat/bin/catalina.sh start
no manual downloads or anything. however, in the case of orm (and now esapi and image), i'm creating/running little apps in Dockerfile after that installation which call functions from those extensions.
I would recommend turning up the log level on the deploy log, and capturing the contents of that file (server and web context) when Lucee fails to look for clues.
i could use a hint on how to do that.
found a ticket that might give me clues: https://luceeserver.atlassian.net/browse/LDEV-3289
nah, it didn't help me figure out how to set the log level
b
for installation, i'm literally doing this.
So that means you're relying on Lucee to download the lex file over the internet. Which, is at the mercy of the Lucee update server to be online at that moment. I would assume you're just having a lex that fails to download every once in a great while.
Lucee will check the lex's in the 
extensions/available
folder before downloading, but you'd need to also ensure you manually downloaded the correct versions for Lucee to use it.
To adjust the logging settings, you'd need to use a tool like CFConfig since the Lucee config file won't exist before the first start and you obviously can't use the web admin prior to the first start!
On a CommandBox-based container, an env var along the lines of
Copy code
cfconfig_loggers_deploy_level=trace
sould do the trick
j
thanks. i'm not running commandbox but i realized that there are log configurations in the lucee xml config files. e.g., 
<logger appender="resource" appender-arguments="path:{lucee-config}/logs/deploy.log" layout="classic" level="error" name="deploy"/>
b
Yep, if you're not using CommandBox/CFConfig, then you'll need to manually muck around in the XML files yourself 🙂 In Lucee 6, the XML has become a JSON file that seems to mostly match CFConfig's format which is at least easier and allows you to tap into tools like 
jq
to do it for you.
j
maybe i'm not correctly setting the log level or something at build time. i dunno. but this is what i got.
Untitled
(side note: cfconfig.json is supposed to work in 5.4.x, too, but i couldn't get it to work. it's not really documented in the context of non-commandbox lucee and i failed to get it picked up through trial and error.)
fwiw, when i go to the administrator, the extensions are there, and i believe that they are actually installed (or halfway-installed, that is) even at build time.
i don't know what to do with this information, but i need to deal with these flaky dependencies the only way i know how, which is to call extension functions from cfml within the build, which is all wrong. but until lucee is capable of truly installing dependencies at build-time, i think i'm stuck with the messy workaround.
b
I'm really curious what the heck this means
Copy code
unsuccessfully installed extensions
but I think your trace log levels aren't being used for the deploy log, or we'd know.
j
my interpretation (which is probably wrong) is that it can't install those extensions because they were already "installed" during the build. i think maybe it will try to reinstall them every time tomcat is started.
but I think your trace log levels aren't being used for the deploy log, or we'd know.
what would be the telltale of a successful level-setting? i don't know what to look for (since i don't know what detailed logs there might be).
b
Well I assume there would be a lot more messages, lol
j
yeah, me too, but i don't know how verbose deploy.log can be
i'll double check the xml file configs at build time
b
You'd need to seed that XML file in the correct directory BEFORE Lucee ever started for the very first time
In other words, you're creating that XML file from scratch in a folder structure you create in the middle of nowhere on your hard drive
In the location where the Lucee home will someday exist in
j
right, but i'm doing it before i ever start tomcat in the dockerfile (as part of this experiment, that is).
i'm double-checking now that the xml files look good during the build
b
I just ran
Copy code
set cfconfig_loggers_deploy_level=trace
set cfconfig_web_loggers_deploy_level=trace
server start
in an new folder in CommandBox and the web context deploy log had nothing but headers, but the server context was 81KB!
Lucee 5.3.10.120
deploy.log
deploy.log
So the logging levels certainly work
j
okay, cool, thanks. yeah, i'm getting squat in the web context, just stuff in the server context.
b
So that means there was an exception of some kind caught and logged above
👍 1
j
i spot a problem with my xml hack. i'm working on it....
b
However, seeing from my logs that an exception is thrown every time an extension is already installed eye roll that may or may not be useful
j
okay, that part, i suspected. i think i have the xml sorted so i'm waiting for the build/deploy to complete so i can get some logs.
i wasn't expecting this. (i have some inline comments that were added programmatically at different points of the build.)
Untitled
snippet from the dockerfile that generated the output
Dockerfile.dockerfile
hmm, i changed my warmups from: 
LUCEE_ENABLE_WARMUP=true /usr/local/tomcat/bin/catalina.sh start
to: 
prewarm.sh
and i'm getting a lot more activity from that stage. i had thought those were equivalent, but maybe they're not
still poking around...
unless i find some other gotcha, i think i've had a revelation here, and i'll probably have to retract my gripes about lucee extension installation.
@Mark Drew (he/him), my tentative takeaway is that my previous gripes were based on a faulty foundation--it seems that, 
LUCEE_ENABLE_WARMUP=true /usr/local/tomcat/bin/catalina.sh start
had once been equivalent to 
prewarm.sh
, but last september, that seems to have changed: https://github.com/lucee/lucee-dockerfiles/commit/518b15940bb56153ae75f220514922291f4ba1c7 so it looks like it's best to use the canonical 
prewarm.sh
. for now, it's looking like that does what we'd expect it to. i know at one point, i advised you to use 
LUCEE_ENABLE_WARMUP=true /usr/local/tomcat/bin/catalina.sh start
so i want to make a correction.
maybe your gripe still stands, though, mark, and maybe i conflated the two issues. (if so, sorry about that.)
@bdw429s, thanks for recommending the deploy.log tweaks. i'll probably keep them on 
trace
forever since they're useful, and not very verbose.
b
I assume if you exported your env var, your original method would have worked. That is, after all, all the prewarm script is doing for you
j
right, that's my takeaway, too. the old version of 
prewarm.sh
didn't export it, either, but something must have changed. i'm going to switch to 
prewarm.sh
, though, so if something changes again in the future, it should be more reliable.
b
The old versions of prewarm prolly didn't work 😆
😄 1
a
@jamiejackson just encountered the transient erraticness (is that a word? Is now) you alluded to above. Was seeing odd behaviour in the app... then was getting "OSGi it's not installed" errors... then... odd behaviour goes way, and so do OSGi errors. This is ridiculous.
j
just to clarify my current take on things. • 
LUCEE_ENABLE_WARMUP=true /usr/local/tomcat/bin/catalina.sh
is worse than useless, since it's misleading. it plain doesn't work (and maybe never did). besides, it's not canonical. • 
prewarm.sh
works. it does what i expect under the covers and it hasn't let me down since i switched to it a couple weeks ago. it's canonical, to boot. • if you're NOT using the OFFICIAL docker image, you may have to do this instead (untested):
Copy code
# the `export` seems to be key.
export LUCEE_ENABLE_WARMUP=true
# now tomcat will actually wait for installation to complete before shutting down
/usr/local/tomcat/bin/catalina.sh start
Or the Dockerfile equivalent:
Copy code
RUN export LUCEE_ENABLE_WARMUP=true \
  && /usr/local/tomcat/bin/catalina.sh start
a
cheers for the summary of that lot.
8 Views
Open in Slack
PreviousNext
Powered by