Hi All, I have a NEW Amazon Lightsail Ubuntu server (Ubuntu 22.04.4 LTS - GNU/Linux 6.5.0-1020-aws x86_64) running Lucee 6.0.1.83 on Nginx (nginx/1.18.0). It was created using this - https://github.com/foundeo/ubuntu-nginx-lucee Everything seemed to be working perfectly at first until attempting to write to the local disk which threw an

Invalid file: Read-only file system

error. My initial reaction is it was an obvious permissions issue, and every thread I have found on the subject emphatically insist the same thing - "It is a permissions issue, NOT a Lucee issue". But after 3 days of exhaustive testing, including a full-day ChatGPT step-by-step elimination process, everything points to Lucee. I first checked the web folder permissions:

0775 root:www-data

Then the www-data group members:

www-data:x:33:tomcat,ftpuser,ubuntu,daemon

That all seemed correct to me, so I created the following TEST script to see if it could write to the Lucee TEMP directory or not:

<!--- Lucee TEMP directory - WORKS --->

<cffile action="write" file="#getTempDirectory()#write-test.txt" output="Testing">

<!--- Relative directory - ERROR --->

<cffile action="write" file="write-test.txt" output="Testing">

The test script was able to successfully write to the Lucy temp directory, but still failed writing to the script directory with the same error, which suggests a permissions issue as originally anticipated. Here is the Lucee error output in the browser:

Lucee 6.0.1.83 Error (application)
Message	Invalid file [/web/litgate.dev/wwwroot/writetest/write-test.txt]
Detail	/web/litgate.dev/wwwroot/writetest/write-test.txt: Read-only file system
Stacktrace	The Error Occurred in
/web/litgate.dev/wwwroot/writetest/index.cfm: line 6
4:
5: <!--- Relative directory - ERROR --->
6: <cffile action="write" file="write-test.txt" output="Testing">
7:
8:

Java Stacktrace	lucee.runtime.exp.ApplicationException: Invalid file [/web/litgate.dev/wwwroot/writetest/write-test.txt]
  at lucee.runtime.tag.FileTag.checkFile(FileTag.java:1230)
  at lucee.runtime.tag.FileTag.actionWrite(FileTag.java:726)
  at lucee.runtime.tag.FileTag.doEndTag(FileTag.java:480)
  at writetest.index_cfm$cf.call(/writetest/index.cfm:6)
  at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1028)
  at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:951)
  at lucee.runtime.listener.ClassicAppListener._onRequest(ClassicAppListener.java:65)
  at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:45)
  at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2715)
  at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2701)
  at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2672)
  at lucee.runtime.engine.Request.exe(Request.java:45)
  at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1259)
  at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1205)
  at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97)
  at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
  at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359)
  at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
  at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1735)
  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
  at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
  at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
  at java.base/java.lang.Thread.run(Thread.java:829)
 
Timestamp	6/4/24 9:09:39 AM CDT

And this is what is logged to exception.log (/opt/lucee/config/server/lucee-server/context/logs):

"ERROR","http-nio-8080-exec-3","06/04/2024","14:09:39","","Invalid file [/web/litgate.dev/wwwroot/writetest/write-test.txt];Invalid file [/web/litgate.dev/wwwroot/writetest/write-test.txt]
/web/litgate.dev/wwwroot/writetest/write-test.txt: Read-only file system;lucee.runtime.exp.ApplicationException: Invalid file [/web/litgate.dev/wwwroot/writetest/write-test.txt]
	at lucee.runtime.tag.FileTag.checkFile(FileTag.java:1230)
	at lucee.runtime.tag.FileTag.actionWrite(FileTag.java:726)
	at lucee.runtime.tag.FileTag.doEndTag(FileTag.java:480)
	at writetest.index_cfm$cf.call(/writetest/index.cfm:6)
	at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1028)
	at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:951)
	at lucee.runtime.listener.ClassicAppListener._onRequest(ClassicAppListener.java:65)
	at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:45)
	at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2715)
	at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2701)
	at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2672)
	at lucee.runtime.engine.Request.exe(Request.java:45)
	at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1259)
	at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1205)
	at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97)
	at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
..."

I checked the Lucee temp folder permissions:

sudo ls -la /opt/lucee/config/server/lucee-server/context/temp

total 16

drwxr-x---  3 tomcat tomcat 4096 Jun  4 14:08 .

drwxr-x--- 20 tomcat tomcat 4096 Jun  4 13:30 ..

drwxr-x---  4 tomcat tomcat 4096 Jun  4 13:30 compress

-rw-r-----  1 tomcat tomcat    8 Jun  4 14:09 write-test.txt

Then updated the test script's Owner:Group to match (tomcat:tomcat) and enabled ALL permissions just in case:

sudo chown -R tomcat:tomcat /web/litgate.dev/wwwroot/writetest

sudo chmod -R 777 /web/litgate.dev/wwwroot/writetest

sudo service nginx restart

After restarting Lucee and hitting the test script, I STILL get the same error. I even rebooted the entire LightSail server but still no luck. During my ChatGPT session, one of its tests was to "determine if it is a local user/permissions issue, or a Lucee issue" by switching to the tomcat user and attempting to write a file to the same folder via the terminal which WORKED:

sudo touch /web/litgate.dev/wwwroot/writetest/test_output.txt

This is what now leads me to believe that it may in fact be a Lucee issue, and not a permissions issue. WHY can the tomcat user successfully write to the folder, but Lucee running as the tomcat user CAN'T? Is there some crazy setting hidden somewhere inside of Lucee that is restricting all write permissions to ONLY the TEMP folder? I am now on my 4th day with this, and have completely run out of ideas. I have set up numerous Lucee servers in the past, and have never run into this before, though this is my first attempt FROM SCRATCH on an FRESH Ubuntu/Nginx server. ANY help would be greatly appreciated! The TEST script is LIVE here - https://dev.litgate.com/writetest THANKS!!

Your error leads me to believe it is not a permissions error with Lucee or Tomcat but something else. You might want to check out https://stackoverflow.com/questions/55477337/why-i-am-getting-read-only-file-system-error-from-nginx

If you deployed this right onto linux, Amazon's images have SELinux enabled by default (including the Ubuntu version they distribute). Do some research on if SELinux is enabled, and if it is, if that is causing your issues. SELinux is, for a lack of a better description, a firewall for the file system. It very much will cause write permission errors even though the filesystem says things are fine.

Thanks for the ideas. @quetwo - I tried running

sudo sestatus

to see if SELinux was enabled, but got

command not found

. So I listed all installed packages and SELinux was not among them, so I don't think that is the case. @Rodney - I have a bad feeling this is the issue. Unfortunately that thread is way over my head, as they are mostly discussing Docker containers which as far as I know I am not using. I may end up having to completely redeploy a new server using Apache instead. 😓 I honestly can't believe how difficult it is getting a simple Lucee server running on Linux. I figured it would just like every other piece of Linux software -

sudo apt install lucee

, but ended up spending over 2 days getting one up and running, and it can't even write to the disk.

@PwrSrg the Ubuntu specific command is

aa-status

that should tell you if you are running it

AH, thanks! Looks like it is running after all?

apparmor module is loaded.
35 profiles are loaded.
33 profiles are in enforce mode.
   /snap/snapd/21465/usr/lib/snapd/snap-confine
   /snap/snapd/21465/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /snap/snapd/21759/usr/lib/snapd/snap-confine
   /snap/snapd/21759/usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/bin/man
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/sbin/chronyd
   /{,usr/}sbin/dhclient
   lsb_release
   man_filter
   man_groff
   nvidia_modprobe
   nvidia_modprobe//kmod
   snap-update-ns.amazon-ssm-agent
   snap-update-ns.lxd
   snap.lxd.activate
   snap.lxd.benchmark
   snap.lxd.buginfo
   snap.lxd.check-kernel
   snap.lxd.daemon
   snap.lxd.hook.configure
   snap.lxd.hook.install
   snap.lxd.hook.remove
   snap.lxd.lxc
   snap.lxd.lxc-to-lxd
   snap.lxd.lxd
   snap.lxd.migrate
   snap.lxd.user-daemon
   tcpdump
2 profiles are in complain mode.
   snap.amazon-ssm-agent.amazon-ssm-agent
   snap.amazon-ssm-agent.ssm-cli
0 profiles are in kill mode.
0 profiles are in unconfined mode.
3 processes have profiles defined.
2 processes are in enforce mode.
   /usr/sbin/chronyd (472)
   /usr/sbin/chronyd (484)
1 processes are in complain mode.
   /snap/amazon-ssm-agent/7993/amazon-ssm-agent (402) snap.amazon-ssm-agent.amazon-ssm-agent
0 processes are unconfined but have a profile defined.
0 processes are in mixed mode.
0 processes are in kill mode.

Yup. It's running in enforce mode. So, you can either dive in and learn how to allow Tomcat to edit those files, or you can completely disable it.

I like the second option. 😁 Unless of course I am opening the server up to some huge security risks.

You should be able to temporarily disable it and see if you can then write that file as a test. The fine folks at Ubuntu do things their own way, so I'm not sure how that is done (enterprise linux like RHEL, CentOS, Rocky and others all use the same commands for SELinux... Ubuntu forked it a few years back to their own project called AppArmour which is not compatible, but does the same thing)

Just out of curiosity, which profile are we looking at?

@PwrSrg No idea -- I don't know the specifics of your server, and which profile would cause the file system to be locked out of tomcat... but there should be a way to shut down the protection until next reboot to test. If that works, you may need to ask some Ubuntu folks on how to correctly remove or disable it across reboots

OK, THANK you for the help! I will start down this road.

BUT it's meant as a last line of defense so that even if bad guys get in through an exploit, they can't overwrite files or change configs without an interactive user being logged in

Good morning @quetwo, so it looks like it might not be AppArmor after all. Disabling it didn't work (

sudo systemctl disable apparmor.service

) and stopping it didn't work (

sudo systemctl stop apparmor.service

). Then I found a command to completely remove all profiles (

sudo aa-teardown

), and that didn't work. When I run

sudo aa-status

now all it returns is

apparmor module is loaded

, but I still get the same error. I had also restarted both Nginx (

sudo service nginx restart

) and Lucee (

sudo service tomcat9 restart

). When I reboot the server all AppArmor profiles are running again.

disable appArmor : sudo systemctl disable AppArmor Selinux maybe installed, so run : sudo setenforce 0 if that returns an error of command not found, do not worry about it. if it just works, then disable SELinux. edit the /etc/selinux/config and change the line that says SELINUX to disabled. Then restart lucee

Hey @Evil Ware, thanks for the suggestions. As stated in my last comment, disabling AppArmor (

sudo systemctl disable apparmor

) didn't work, nor did completely tearing down all AppArmor profiles (

sudo aa-teardown

).

sudo setenforce 0

returns

sudo: setenforce: command not found

.

/etc/selinux/config

doesn't exist, but

/etc/selinux/semanage.conf

does. Actually,

semanage.conf

is the only file in the

/etc/selinux

directory. Here is it's contents:

# Authors: Jason Tang <jtang@tresys.com>
#
# Copyright (C) 2004-2005 Tresys Technology, LLC
#
#  This library is free software; you can redistribute it and/or
#  modify it under the terms of the GNU Lesser General Public
#  License as published by the Free Software Foundation; either
#  version 2.1 of the License, or (at your option) any later version.
#
#  This library is distributed in the hope that it will be useful,
#  but WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
#  Lesser General Public License for more details.
#
#  You should have received a copy of the GNU Lesser General Public
#  License along with this library; if not, write to the Free Software
#  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
#
# Specify how libsemanage will interact with a SELinux policy manager.
# The four options are:
#
#  "source"     - libsemanage manipulates a source SELinux policy
#  "direct"     - libsemanage will write directly to a module store.
#  /foo/bar     - Write by way of a policy management server, whose
#                 named socket is at /foo/bar.  The path must begin
#                 with a '/'.
#  <http://foo.com:4242|foo.com:4242> - Establish a TCP connection to a remote policy
#                 management server at <http://foo.com|foo.com>.  If there is a colon
#                 then the remainder is interpreted as a port number;
#                 otherwise default to port 4242.
module-store = direct

# When generating the final linked and expanded policy, by default
# semanage will set the policy version to POLICYDB_VERSION_MAX, as
# given in <sepol/policydb.h>.  Change this setting if a different
# version is necessary.
#policy-version = 19

# expand-check check neverallow rules when executing all semanage commands.
# Large penalty in time if you turn this on.
expand-check=0

# By default, semanage will generate policies for the SELinux target.
# To build policies for Xen, uncomment the following line.
#target-platform = xen

Its the lowest form of linux, Ubuntu does not do anything normal.

I did them in that exact order again and no luck.

Just wait until you are forced to update, and it breaks. Fun times ahead.