https://linen.dev logo
Join Slack
Powered by
So thanks to everyone from the thread above. I am ...
# fluent-operator
d
So thanks to everyone from the thread above. I am deploying the operator to a cluster running Bottlerocket which uses a read-only root filesystem. When I deploy it I am getting the following error in the fluent-bit pods.
Copy code
a6871107bc42e9265df04a4" spec: failed to apply OCI options: failed to mkdir "/containers": mkdir /containers: read-only file system
 Warning  Failed     90s                   kubelet            Error: failed to generate container "a31459cc345ec00ee278673c2767f0e956c411482
p
Is that just an opensearch config? I'm wondering if the plugin uses an output folder for some reason - I know some of the other AWS plugins do like S3 which means they cannot run in a read-only root filesystem. You could mount an ephemeral volume or similar potentially but not sure how to do this exactly with the operator.
@Benjamin Huo does the operator CRD include support for custom mounts or extra volumes?
d
Yeah here is the weird thing. I'll drop the full output but that is supposed to be a RO mount
Copy code
ubectl describe pod -n fluent fluent-bit-qs6qc
Name:             fluent-bit-qs6qc
Namespace:        fluent
Priority:         0
Service Account:  fluent-bit
Node:             ip-192-168-27-255.us-west-2.compute.internal/192.168.27.255
Start Time:       Tue, 31 Oct 2023 11:05:58 -0400
Labels:           <http://app.kubernetes.io/name=fluent-bit|app.kubernetes.io/name=fluent-bit>
                  controller-revision-hash=5d7b4d5f86
                  pod-template-generation=1
Annotations:      <none>
Status:           Pending
IP:               192.168.21.118
IPs:
  IP:           192.168.21.118
Controlled By:  DaemonSet/fluent-bit
Containers:
  fluent-bit:
    Container ID:   
    Image:          kubesphere/fluent-bit:v2.1.9
    Image ID:       
    Port:           2020/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  200Mi
    Requests:
      cpu:     10m
      memory:  25Mi
    Environment:
      NODE_NAME:   (v1:spec.nodeName)
      HOST_IP:     (v1:status.hostIP)
    Mounts:
      /containers from varlibcontainers (ro)
      /fluent-bit/config from config (ro)
      /fluent-bit/tail from positions (rw)
      /var/log/ from varlogs (ro)
      /var/log/journal from systemd (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p2lmt (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  varlibcontainers:
    Type:          HostPath (bare host directory volume)
    Path:          /containers
    HostPathType:  
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  fluent-bit-config
    Optional:    false
  varlogs:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:  
  systemd:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/journal
    HostPathType:  
  positions:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/fluent-bit/
    HostPathType:  
  kube-api-access-p2lmt:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 op=Exists
                             <http://node.kubernetes.io/disk-pressure:NoSchedule|node.kubernetes.io/disk-pressure:NoSchedule> op=Exists
                             <http://node.kubernetes.io/memory-pressure:NoSchedule|node.kubernetes.io/memory-pressure:NoSchedule> op=Exists
                             <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists
                             <http://node.kubernetes.io/pid-pressure:NoSchedule|node.kubernetes.io/pid-pressure:NoSchedule> op=Exists
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists
                             <http://node.kubernetes.io/unschedulable:NoSchedule|node.kubernetes.io/unschedulable:NoSchedule> op=Exists
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  2m51s                 default-scheduler  Successfully assigned fluent/fluent-bit-qs6qc to ip-192-168-27-255.us-west-2.compute.internal
  Warning  Failed     2m51s                 kubelet            Error: failed to generate container "496b1793018e493a469060bcf059e375cc7b07731254ca2984a4fa510029c8d2" spec: failed to apply OCI options: failed to mkdir "/containers": mkdir /containers: read-only file system
  Warning  Failed     2m50s                 kubelet            Error: failed to generate container "986782b16258f1384bbef56e0aebb5c3bd771cc8a694bda0ebf05e2eb34706f9" spec: failed to apply OCI options: failed to mkdir "/containers": mkdir /containers: read-only file system
  Warning  Failed     2m40s                 kubelet            Error: failed to generate container "29cb60523b3bb724bc05946b58edd004244d3c7eab3f95257ee65de68b7cee0e" spec: failed to apply OCI options: failed to mkdir "/containers": mkdir /containers: read-only file system
  Warning  Failed     2m27s                 kubelet            Error: failed to generate container "f70f7bc6b7610928363a5cf46e105006933d6f4bc09f7592df9cacbbc50854ea" spec: failed to apply OCI options: failed to mkdir "/containers": mkdir /containers: read-only file system
  Warning  Failed     2m14s                 kubelet            Error: failed to generate container "d7a7b25eeaf95d22d63e2aae904ca79c0a16d020585e5869d56fba413050ac0f" spec: failed to apply OCI options: failed to mkdir "/containers": mkdir /containers: read-only file system
  Warning  Failed     2m                    kubelet            Error: failed to generate container "2196d56881564dea3841962c8b633ad8775566ea03da4a9b2058889faf097954" spec: failed to apply OCI options: failed to mkdir "/containers": mkdir /containers: read-only file system
  Warning  Failed     105s                  kubelet            Error: failed to generate container "f377e77f8fece1e03c6d8af54420107b7651d2337a6871107bc42e9265df04a4" spec: failed to apply OCI options: failed to mkdir "/containers": mkdir /containers: read-only file system
  Warning  Failed     90s                   kubelet            Error: failed to generate container "a31459cc345ec00ee278673c2767f0e956c41148297c784c332bb4af2d1232ba" spec: failed to apply OCI options: failed to mkdir "/containers": mkdir /containers: read-only file system
  Warning  Failed     77s                   kubelet            Error: failed to generate container "87539220889ce3beaee5af0ed35caa8486c6fac691c311725ab6116a6015a318" spec: failed to apply OCI options: failed to mkdir "/containers": mkdir /containers: read-only file system
  Warning  Failed     35s (x3 over 62s)     kubelet            (combined from similar events): Error: failed to generate container "6cd8caf4859ef9433809ff1a7f894c8827797e009a5ed5d85614f3b99adc7366" spec: failed to apply OCI options: failed to mkdir "/containers": mkdir /containers: read-only file system
  Normal   Pulled     20s (x13 over 2m51s)  kubelet            Container image "kubesphere/fluent-bit:v2.1.9" already present on machine
p
If you're using filesystem buffering then you obviously need a filesystem to write to as well, plus some of the plugins use it
d
The important parts here:
Copy code
Mounts:
      /containers from varlibcontainers (ro)

Volumes:
  varlibcontainers:
    Type:          HostPath (bare host directory volume)
    Path:          /containers
    HostPathType:
p
What's your config? Are you using a DB file for example?
Copy code
config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  fluent-bit-config
    Optional:    false
That should be the actual generated FB config. I'd be curious to see if it is using a 
DB
file with a 
tail
input for example - that requires writing an sqlite file and it may try to do it in the same directory
d
That may be it, here is the config: https://github.com/dtaivpp/bottlerocket-opensearch/blob/a6464ad0e0263b7b095a884f62c5df24e0ff990c/fluentbit.yaml
Copy code
apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterInput
metadata:
  name: docker
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/component: logging
spec:
  systemd:
    tag: service.*
    path: /var/log/journal
    db: /fluent-bit/tail/systemd.db
    dbSync: Normal
    systemdFilter:
      - _SYSTEMD_UNIT=docker.service
      - _SYSTEMD_UNIT=kubelet.service
p
If you can dump the secret it'll give you the actual config that is generated - just to rule out any CRD conversion changes. Looking at the CRD it doesn't seem to be writing to 
/containers
for the DB file:
Copy code
tail:
    tag: kube.*
    path: /var/log/containers/*.log
    parser: docker
    refreshIntervalSeconds: 10
    memBufLimit: 100MB
    skipLongLines: true
    db: /fluent-bit/tail/pos.db
    dbSync: Normal
Something else must be though. I'd be tempted to remove the opensearch output and use stdout to see if it goes away - suggesting it is then something inside opensearch.
d
I think I may have found it. Bottlerocket is relocating the containers path -_-
p
I think the only K8S standard is var/log/pods with everyone else then symlinking or otherwise 🙂 I could be wrong though
d
That makes a ton of sense. This is working now. Thanks again for all your help Pat!
p
no probs, worth adding the details for others
d
For sure, will have a spec file and writeup here shortly 😄
e
Open in Slack
PreviousNext
Powered by