>kubectl get pods NAME ...
# all-things-deployment
l
kubectl get pods
NAME READY STATUS RESTARTS AGE datahub-elasticsearch-setup-job-4zh9g 0/1 Error 0 28m datahub-elasticsearch-setup-job-7d58l 0/1 Error 0 34m datahub-elasticsearch-setup-job-d9698 0/1 Error 0 36m datahub-elasticsearch-setup-job-g58tr 0/1 Error 0 31m datahub-elasticsearch-setup-job-hnj9n 0/1 Error 0 38m datahub-elasticsearch-setup-job-r8r9d 0/1 Error 0 25m datahub-elasticsearch-setup-job-t657v 0/1 Error 0 20m elasticsearch-master-0 0/1 Pending 0 10d elasticsearch-master-1 0/1 Pending 0 10d elasticsearch-master-2 0/1 Pending 0 10d prerequisites-cp-schema-registry-5f89dd4974-sff65 1/2 CrashLoopBackOff 2691 (4m37s ago) 10d prerequisites-kafka-0 0/1 Pending 0 10d prerequisites-mysql-0 0/1 Pending 0 10d prerequisites-zookeeper-0 0/1 Pending 0 10d
b
nobody will be able to troubleshoot this without more details, you should look at
kubectl describe pod <pod>
to see what kind of errors is encountered
l
kubectl describe pod prerequisites-cp-schema-registry-5f89dd4974-sff65
Name: prerequisites-cp-schema-registry-5f89dd4974-sff65 Namespace: default Priority: 0 Service Account: default Node: ip-192-168-67-117.eu-west-1.compute.internal/192.168.67.117 Start Time: Thu, 30 Mar 2023 190107 +0200 Labels: app=cp-schema-registry pod-template-hash=5f89dd4974 release=prerequisites Annotations: prometheus.io/port: 5556 prometheus.io/scrape: true Status: Running IP: 192.168.69.136 IPs: IP: 192.168.69.136 Controlled By: ReplicaSet/prerequisites-cp-schema-registry-5f89dd4974 Containers: prometheus-jmx-exporter: Container ID: containerd://d16c08a288715cc5326646fac786e9fbb346397674eb65631717702ad6115848 Image: solsson/kafka-prometheus-jmx-exporter@sha256:6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143 Image ID: docker.io/solsson/kafka-prometheus-jmx-exporter@sha256:6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143 Port: 5556/TCP Host Port: 0/TCP Command: java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 -XshowSettings:vm -jar jmx_prometheus_httpserver.jar 5556 /etc/jmx-schema-registry/jmx-schema-registry-prometheus.yml State: Running Started: Thu, 30 Mar 2023 190113 +0200 Ready: True Restart Count: 0 Environment: <none> Mounts: /etc/jmx-schema-registry from jmx-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-g5mrk (ro) cp-schema-registry-server: Container ID: containerd://c33f5bd6700ef4fe76677bd4b14be4f22d92ba389bcae1c045a90bedcffa5881 Image: confluentinc/cp-schema-registry:6.0.1 Image ID: docker.io/confluentinc/cp-schema-registry@sha256:b52e16cf232e3c9acd677ae8944de813e16fa541a367d9f805b300c5d2be1a1f Ports: 8081/TCP, 5555/TCP Host Ports: 0/TCP, 0/TCP State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Tue, 11 Apr 2023 103532 +0200 Finished: Tue, 11 Apr 2023 103616 +0200 Ready: False Restart Count: 2871 Environment: SCHEMA_REGISTRY_HOST_NAME: (v1:status.podIP) SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081 SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: prerequisites-kafka:9092 SCHEMA_REGISTRY_KAFKASTORE_GROUP_ID: prerequisites SCHEMA_REGISTRY_MASTER_ELIGIBILITY: true SCHEMA_REGISTRY_HEAP_OPTS: -Xms512M -Xmx512M JMX_PORT: 5555 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-g5mrk (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: jmx-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: prerequisites-cp-schema-registry-jmx-configmap Optional: false kube-api-access-g5mrk: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning BackOff 86s (x67512 over 11d) kubelet Back-off restarting failed container
b
iirc schema-registry requires kafka to be up to actually run, does
describe
on kafka say anything also need to know why ES is not starting up