wicla came originally on July 23rd with a permission issue that revolved around UIDs being switched around. This is the original message:

"Hi. I'm using openvox helmchart and updated to puppetserver from 8.14.1-main to 8.15.1-main and it fails with permission denied errors: <https://paste.gentoo.zip/Dg3ofDc9>. It appears to change the owner/group of mounted directory to 64604:64604. These files looks like they should be part of the container image itself though"

Entrypoint PID 7
Running /container-entrypoint.d/20-use-templates-initially.sh
/usr/local/share/openvox/config_lib.sh: line 14: /run/openvox/config-lib-cache: Permission denied
sed: can't read /run/openvox/config-lib-cache: No such file or directory
/container-entrypoint.d/20-use-templates-initially.sh: line 12: cd: null directory

Indeed, the UID of puppet was changed from 1001 to 64604 between those versions, which explains the switch. Later they said

"Tried to overwrite global.securityContext in values.yaml for the helm chart but didn't seem to have any effect"

which makes sense if the runAsNonRoot variable in the values.yaml was still set to false, which it probably was according to their values.yaml which they posted at a later point. Briefly, why changing runAsGroup and runAsUser are not effective if runAsNonRoot remains false: In templates/puppetserver-deployment-masters.yaml we can see that the UID and GID of .Values.global.securityContext is only consumed if .Values.global.runAsNonRoot is true.

spec:
  template:
    spec:
      containers:
        - name: {{ template "puppetserver.fullname" . }}
          securityContext:
            {{- if .Values.global.runAsNonRoot }}
            runAsUser: {{ .Values.global.securityContext.runAsUser }}
            runAsGroup: {{ .Values.global.securityContext.runAsGroup }}
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - all
            {{- else }}
            {{- toYaml .Values.puppetserver.securityContext | nindent 12 }}
            {{- end }}

The default UID and GID in values is 999:

securityContext:
    runAsUser: 999  # "puppet" UID
    runAsGroup: 999  # "puppet" GID
    fsGroup: 999  # "puppet" GID

Originally they were set to 999 as at the time (in 2023) puppets UID/GID was 999. So, not changing runAsNonRoot to true will not consume runAsUser and runAsGroup. fsGroup is actually hardcoded in templates/puppetserver-deployment-masters.yaml:

spec:
  template:
    spec:
      securityContext:
        fsGroup: 999   # "puppet" GID

This is also the reason why when wicla posted the output of

id

there was a supplemental GID of 999.

puppet@puppet:/$ id
id: cannot find name for group ID 999[7:17 AM]uid=1001(puppet) gid=1001(puppet) groups=1001(puppet),999(999)

wicla had at this point changed the runAsUser and runAsGroup to 1001 believing it would have an effect, but as established earlier, when runAsNonRoot: false they don't change anything, which is what they had: paste.gentoo.zip/pAN2AmbL Why do the UID and GID match the 1001 from wiclas values.yaml then? The UID is simply the UID that comes from the USER puppet:0 directive from the image, kubernetes ignores the GID though and checks the UID and GID inside /etc/passwd according to the runtime user name. Miranda correctly assessed the latter and mentioned it in container-openvoxserver#179 which is the PR that chowned /run/openvox to the puppet user in order to fix the original issue wicla had. This is a pretty important part which I will revisit later. But first, what happened afterwards is also interesting, and is another issue. wicla came back reporting that the container still failed due to /opt/puppetlabs/server/data/puppetserver/.puppetlabs/etc/puppet missing. After creating it manually the container booted all the way to the JVM crashing with

"(LoadError) cannot load such file -- puppet".

The fix Miranda implemented worked on a containers without the serverdata PVC. The serverdata PVC mount is in `templates/puppetserver-deployment-masters.yaml`:

spec:
  template:
    spec:
      containers:
        - name: {{ template "puppetserver.fullname" . }}
          volumeMounts:
            {{- if eq .Values.puppetserver.persistence.data.enabled true }}
            - name: puppet-serverdata-storage
              mountPath: /opt/puppetlabs/server/data/puppetserver/
            {{- end }}

Since the PR that I authored to make the image OpenShift compatible container-openvoxserver#92 in November 2025 the users $HOME is explicitly set to /opt/puppetlabs/server/data/puppetserver/ and symlinked the root users puppet config dirs to $HOME/.puppetlabs/ to make sure they agree. Worth mentioning at this point is that the $HOME directory of the puppet user is already set to that directory, I only made sure that arbitrary UIDs without preset $HOME also use the same directory.

There was an issue with how puppet discovers the correct paths, somehow puppet config print gives the right paths, but they are not respected via library calls from the server.

Thus I just opted to additionally symlink the non-root directories and the system directories.

RUN ln -sf /etc/puppetlabs ${HOME}/.puppetlabs/etc
RUN ln -sf /opt/puppetlabs ${HOME}/.puppetlabs/opt
RUN ln -sf /var/log/puppetlabs ${HOME}/.puppetlabs/var/log
RUN ln -sf /var/run/puppetlabs ${HOME}/.puppetlabs/var/run

and additionally initialised the puppet.conf with some directories set.

helm used the Ubuntu image and OpenShift compatibility originally only was implemented for the Alpine images. Since container-openvoxserver#141 they became unionized and it was discovered to be a problem in helm since openvox-helm-chart#44. Since container-openvoxserver#141 the puppet ruby lib is no longer installed as a .deb, instead the openvox gem is used and installed at build time into the server's gem-home /opt/puppetlabs/server/data/puppetserver/jruby-gems i.e. inside $HOME, under the serverdata PVC mount. It exists nowhere else in the image (not in vendored-jruby-gems, not in vendor_gems). This explains the second error wicla got after trying container-openvoxserver#179: (LoadError) cannot load such file -- puppet Conclusion So, right now, there are actually two issues. 1) The images symlinks and helms serverdata mount contradict each other resulting in the missing config dir when used with helm. Simply creating the directories manually does not fix the gem directory being shadowed, which is created at buildtime inside $HOME. This is what container-openvoxserver#181 is discussing. I will write more on that in the discussion on github in a moment. By the way, here it is interesting to see that 20-use-templates-initially.sh actually includes another solution, building the gem dir outside of $HOME and then copying it over to $HOME at runtime, this comes from puppet 7.x times and actually is no longer functional as the gem dir there in /var/tmp never gets produced. 2) The helm chart does not utilize the primary access model properly. Right now fsGroup is hardcoded to 999, this should be 0. Another quick solution would be to set the runAsGroup for the server to 0 (the root group). Neither of these options contradicts

runAsNonRoot: true

btw, only the UID is checked for this. I will be happy to bring out a fix for this. Another measure we should take here is a clearly communicated decision on the access model for containers. Right now the server image supports arbitrary UIDs, but there is a lot of evidence that suggests it is supposed to be run with the puppet UID instead. This is confusing and will create more problems down the road. One example is evident in the openvoxdb containers, after the unionization effort there the image declares the OpenShift convention by using the root GID in the USER directive, but it never actually changed the group of relevant files. Additionally then, there is a PR that recommends users to chown volumes to the new UID AND GID 64604 container-openvoxdb#95 but the executing users GID is 0, not 64604. So regarding the GID and UID confusion, I am happy to submit a series of PRs to clean up a bit in code and documentation for both images and helm. Links: container-openvoxserver#92: github.com/OpenVoxProject/container-openvoxserver/pull/92 container-openvoxserver#141: github.com/OpenVoxProject/container-openvoxserver/pull/141 container-openvoxserver#179: github.com/OpenVoxProject/container-openvoxserver/pull/179 container-openvoxserver#181: github.com/OpenVoxProject/container-openvoxserver/issues/181 openvox-helm-chart#44: github.com/OpenVoxProject/openvox-helm-chart/issues/44 container-openvoxdb#95: github.com/OpenVoxProject/container-openvoxdb/pull/95