Overview
Domain 5 tests whether you can find and fix faults on a running cluster. You need to bring a NotReady node back by repairing the kubelet or container runtime, restore a failed control plane component such as the API server, scheduler, controller manager or etcd, measure CPU and memory use with kubectl top, read container output streams with kubectl logs and on-node log files, and trace broken Service, DNS and NetworkPolicy paths. The skill is method rather than memory: move from symptom to layer to cause, using kubectl describe, events, logs, systemctl, journalctl and crictl.
Every other domain feeds this one. Domain 1 teaches where static Pod manifests and kubelet files live, Domain 2 explains Pending and restarting Pods, Domain 3 supplies endpoint and DNS checks, and Domain 4 explains unbound claims. Here those pieces are combined under time pressure, often with the API itself unavailable.
How it's tested
Troubleshooting tasks describe a symptom, such as a node NotReady, kubectl refusing connections, Pods stuck Pending, or an application unreachable, and ask you to fix it so the cluster or workload works again. Some ask you to report a finding, like the Pod using the most CPU, to a file. Grading checks the repaired state.
-
Node down
A worker is NotReady; you SSH in, check the kubelet and container runtime with systemctl and journalctl, fix the configuration or service, and confirm the node returns to Ready.
-
Control plane down
kubectl fails or Pods are never scheduled; you inspect static Pod manifests and container logs with crictl to find a bad flag, path or certificate reference.
-
Find and report
You use kubectl top, kubectl logs or events to identify a Pod, node or error line and write the answer to a given file in the requested format.
kubectl returns 'connection to the server was refused' on port 6443. Because the API server is down, kubectl cannot help, so SSH to the control plane and run crictl ps -a to see the kube-apiserver container exiting. crictl logs on that container, or the files under /var/log/pods, shows the error, such as a mistyped certificate path, which you fix in /etc/kubernetes/manifests/kube-apiserver.yaml.
Competencies 5
Official competencies from the Linux Foundation outline, in outline order.
5.1 Troubleshoot clusters and nodesBringing NotReady nodes back and checking overall cluster health.
- What it means
- Node troubleshooting means finding why a node reports NotReady or is not accepting Pods, which usually comes down to the kubelet, the container runtime, networking, or node pressure conditions.
- What to understand
- Start with kubectl get nodes and kubectl describe node for conditions (Ready, MemoryPressure, DiskPressure, PIDPressure) and taints. On the node, systemctl status kubelet and journalctl -u kubelet show whether the service is stopped, disabled or failing on a bad setting in /var/lib/kubelet/config.yaml, the kubeadm drop-in or kubeadm-flags.env. Check containerd the same way, and kubeadm certs check-expiration for expired certificates.
- How it shows up
- A worker shows NotReady; the kubelet is stopped and disabled, or points at the wrong runtime socket or CA file.
- Common confusion
- Restarting the kubelet without enabling it, so the node fails again on the next reboot, or without reading the log that names the real misconfiguration.
5.2 Troubleshoot cluster componentsDiagnosing the API server, scheduler, controller manager and etcd.
- What it means
- On kubeadm clusters the control plane runs as static Pods defined in /etc/kubernetes/manifests, so a typo, wrong path or bad flag in one of those files stops that component.
- What to understand
- Map the symptom to the component: kubectl refused means kube-apiserver or etcd; new Pods Pending with no node and no scheduling events means kube-scheduler; Deployments that never create ReplicaSets or Pods mean kube-controller-manager. If the API is up, kubectl get pods -n kube-system and kubectl logs work; if not, use crictl ps -a, crictl logs and /var/log/pods on the control plane node.
- How it shows up
- A deliberate fault, such as a misspelled command or wrong certificate path, is planted in a manifest and you must locate and correct it.
- Common confusion
- Editing the mirror Pod with kubectl edit instead of the manifest file, which the kubelet treats as the source of truth.
5.3 Monitor cluster and application resource usageMeasuring CPU and memory with metrics-server and kubectl top.
- What it means
- Resource monitoring on the exam uses the Metrics API served by metrics-server, read through kubectl top nodes and kubectl top pods.
- What to understand
- kubectl top pods -A --sort-by=cpu (or memory) ranks Pods, --containers breaks a Pod down by container, and -l filters by label. kubectl describe node shows allocated requests and limits, which is different from live usage. Without metrics-server, kubectl top returns an error that the Metrics API is not available.
- How it shows up
- You find the Pod with the highest memory use among those with a given label and write its name to a file.
- Common confusion
- Answering from requests in kubectl describe when the task asks for actual usage.
5.4 Manage and evaluate container output streamsReading stdout and stderr through kubectl and on the node.
- What it means
- Containers write logs to stdout and stderr; the container runtime stores them on the node under /var/log/pods (linked from /var/log/containers), the kubelet rotates them, and kubectl logs reads them through the kubelet.
- What to understand
- Use -c for a specific container, --previous for the last terminated instance, --since and --tail to narrow, -f to follow, -l for a label selector and --all-containers for every container. An application that writes only to a file needs a sidecar container that streams the file to its own stdout, typically through a shared emptyDir.
- How it shows up
- A container is in CrashLoopBackOff and you must read the crashed instance's output, or add a sidecar so file-based logs become visible to kubectl logs.
- Common confusion
- Running kubectl logs without --previous on a restarting container and reading only the short output of the new instance.
5.5 Troubleshoot services and networkingTracing failures between clients, Services, Pods and DNS.
- What it means
- Service and network troubleshooting follows traffic layer by layer: DNS name resolution, the Service and its EndpointSlices, kube-proxy rules on the nodes, NetworkPolicies, and finally the Pod listening on the right port.
- What to understand
- Check that kubectl get endpointslices for the Service lists Pod IPs; if empty, compare the selector with Pod labels and readiness. Test from a temporary Pod by Service name, by ClusterIP and by Pod IP to isolate DNS, Service or Pod faults. Confirm CoreDNS and kube-proxy Pods are running in kube-system, and look for NetworkPolicies selecting the target.
- How it shows up
- An application cannot reach its backend by name; the cause is a selector mismatch, wrong targetPort, failing CoreDNS or a restrictive policy.
- Common confusion
- Restarting CoreDNS when name resolution works but the Service has no endpoints.
High-yield concepts
The terms, rules and formulas this domain turns into questions most often.
-
Diagnose by layer
Most faults sit in one layer: node, control plane, scheduling, container start, application, Service, DNS or policy.
- Remember
- Identify the failing layer from the symptom before changing anything.
- Tested as
- Open-ended tasks reward finding the layer quickly.
- Trap
- Changing several things at once and losing track of what fixed it.
-
kubelet service checks
The kubelet is a systemd service on every node, and a NotReady node frequently traces back to the kubelet or the container runtime.
- Remember
- systemctl status kubelet, journalctl -u kubelet, then systemctl enable --now kubelet after fixing the cause.
- Tested as
- Node tasks plant a stopped service or a bad config value.
- Trap
- Restarting without enabling, or without reading the log.
-
Static Pod manifests
Control plane components are defined by files in /etc/kubernetes/manifests that the kubelet watches.
- Remember
- Fix faults in the file; the kubelet recreates the Pod automatically after a short delay.
- Tested as
- Component tasks plant a typo or wrong path in these files.
- Trap
- Running kubectl delete pod on a control plane Pod and expecting a fixed version to return.
-
crictl when the API is down
crictl talks straight to the container runtime, so it works when kubectl cannot.
- Remember
- crictl ps -a lists exited containers; crictl logs ID shows why they failed.
- Tested as
- API server failure tasks need this to see the error.
- Trap
- Trying docker commands on a containerd node.
-
Component symptom map
Each control plane component failure leaves a distinct symptom.
- Remember
- API server: kubectl refused. Scheduler: Pods Pending with no node. Controller manager: Deployments create no ReplicaSets or Pods. etcd: API errors and timeouts.
- Tested as
- Lets you go straight to the right manifest.
- Trap
- Blaming the kubelet for Pods that were never assigned a node.
-
Pod status meanings
A Pod's status narrows the cause before you read details.
- Remember
- Pending: scheduling or unbound claim. ContainerCreating: volume, network or image pull in progress. ImagePullBackOff: image name, tag or credentials. CrashLoopBackOff: application exit. CreateContainerConfigError: missing ConfigMap or Secret key.
- Tested as
- Status columns in kubectl get pods are the first clue in most tasks.
- Trap
- Reading CrashLoopBackOff as an image problem.
-
Events
Events record scheduler decisions, pull failures, probe failures and kills for a short retention period.
- Remember
- kubectl describe shows an object's events; kubectl get events -n NS --sort-by=.lastTimestamp or kubectl events --types=Warning lists them across a namespace.
- Tested as
- Events usually name the cause directly.
- Trap
- Missing them because the default output is not sorted by time.
-
OOMKilled and exit code 137
A container that exceeds its memory limit is killed by the kernel and shows reason OOMKilled.
- Remember
- Exit code 137 means terminated by SIGKILL; raise the limit or fix the memory use.
- Tested as
- Restarting Pods with this reason point to limits from Domain 2.
- Trap
- Raising the CPU limit for a memory kill.
-
kubectl logs flags
kubectl logs can target containers, past instances, time windows and label selections.
- Remember
- -c NAME, --previous, --since=1h, --tail=50, -l app=x, --all-containers.
- Tested as
- Log tasks ask for a specific line or error to be saved to a file.
- Trap
- Omitting -c on a multi-container Pod and reading the wrong container.
-
Node log locations
Container logs live on the node, independently of the API server.
- Remember
- /var/log/pods/NAMESPACE_POD_UID/CONTAINER/ holds the files, with symlinks in /var/log/containers.
- Tested as
- Useful when the API server itself is broken.
- Trap
- Looking for kubeadm control plane logs in /var/log/syslog only.
-
kubectl top
kubectl top reads live CPU and memory usage from metrics-server.
- Remember
- kubectl top pods -A --sort-by=memory, kubectl top nodes, and --containers for per-container figures.
- Tested as
- Find-and-report tasks use it to pick the heaviest Pod or node.
- Trap
- Using requests from describe as if they were usage.
-
Service path isolation test
Testing by name, ClusterIP and Pod IP separates DNS, Service and Pod faults.
- Remember
- If Pod IP works but ClusterIP fails, check endpoints and kube-proxy; if ClusterIP works but the name fails, check CoreDNS.
- Tested as
- The fastest route to the broken layer in networking tasks.
- Trap
- Testing only by name and blaming DNS for everything.
-
Certificate expiry
kubeadm certificates last one year by default, and expired ones stop components authenticating.
- Remember
- kubeadm certs check-expiration lists dates; kubeadm certs renew all renews them, followed by restarting the control plane static Pods.
- Tested as
- A cluster that suddenly refuses connections after a long time may have expired certificates.
- Trap
- Regenerating the CA when only leaf certificates expired.
Side by side
Concepts candidates mix up, compared on the point that decides the answer.
API server vs. scheduler vs. controller manager failure
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| kube-apiserver down | No component or user can reach the API | kubectl fails with connection refused on port 6443 | Trying to debug with kubectl get pods -n kube-system |
| kube-scheduler down | No new Pods are assigned to nodes | Pods sit Pending with no node and no FailedScheduling events | Looking for resource shortages that do not exist |
| kube-controller-manager down | Controllers stop reconciling | New Deployments get no ReplicaSet, scaling does nothing, but existing Pods keep running | Blaming the scheduler because no new Pods appear |
Pending vs. CrashLoopBackOff vs. ImagePullBackOff
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| Pending | Pod accepted but not scheduled or not started | Look at scheduling events, taints, requests and claims | Reading container logs, which do not exist yet |
| CrashLoopBackOff | Container starts and exits repeatedly | Look at kubectl logs --previous and the command or probes | Looking at the image name instead of the application output |
| ImagePullBackOff | Image cannot be pulled | Look at the image name, tag, registry and imagePullSecrets | Restarting the Pod repeatedly without fixing the reference |
kubectl logs vs. crictl logs vs. journalctl
| Concept | Meaning | Key difference | Common trap |
|---|---|---|---|
| kubectl logs | Container output read through the API server and kubelet | Needs a working API server | Using it when kubectl cannot connect |
| crictl logs | Container output read directly from the runtime on the node | Works without the API server | Running it on a node other than where the container runs |
| journalctl -u kubelet | Logs of the kubelet systemd service itself | Explains why a node or static Pod is not running | Searching it for application output |
Exam traps
Each trap is an answer that looks right. Learn the clue that should change your mind.
-
kubectl against a dead API
- What candidates pick
- Running kubectl commands repeatedly when the connection is refused.
- Why it looks right
- kubectl is the normal tool for everything else on the exam.
- Why it's wrong
- It needs the API server; the evidence is in container logs on the control plane node.
Clue to watch for 'The connection to the server ... was refused' on port 6443.
-
Fixing the mirror Pod
- What candidates pick
- Using kubectl edit or delete on a kube-system control plane Pod.
- Why it looks right
- It looks like any other Pod in kubectl get pods.
- Why it's wrong
- Static Pods are owned by the kubelet and recreated from the manifest file, so changes through the API do not persist.
Clue to watch for Pod names ending in the node name, such as kube-scheduler-controlplane.
-
Temporary kubelet fix
- What candidates pick
- Running systemctl start kubelet and moving on.
- Why it looks right
- The node becomes Ready again at once.
- Why it's wrong
- If the service was disabled it will not start after a reboot, and if a configuration error remains it may fail again.
Clue to watch for systemctl status showing disabled, or journalctl showing a configuration error.
-
Pending blamed on resources
- What candidates pick
- Lowering resource requests when new Pods stay Pending.
- Why it looks right
- Insufficient resources is a frequent Pending cause in day-to-day work.
- Why it's wrong
- When the scheduler is down there are no FailedScheduling events at all, and every new Pod across the cluster waits.
Clue to watch for describe shows Node: <none> and no events, for Pods in every namespace.
-
Restarted container, lost evidence
- What candidates pick
- Reading kubectl logs on a CrashLoopBackOff Pod without --previous.
- Why it looks right
- The command returns output, so it seems to have worked.
- Why it's wrong
- The current instance may have just started; the error is in the previous one.
Clue to watch for A restart count above zero in kubectl get pods.
-
DNS blamed for empty endpoints
- What candidates pick
- Restarting CoreDNS when a client cannot reach a Service by name.
- Why it looks right
- Name-based failures feel like DNS problems.
- Why it's wrong
- If the name resolves to the ClusterIP but connections fail, the Service has no endpoints or a wrong targetPort.
Clue to watch for nslookup returning an IP while wget times out.
-
Requests reported as usage
- What candidates pick
- Writing the Pod with the highest memory request to the answer file.
- Why it looks right
- kubectl describe node lists per-Pod numbers that look like usage.
- Why it's wrong
- Those are requests and limits; usage comes from kubectl top.
Clue to watch for Task wording such as 'consuming' or 'using the most'.
Answer framework
Follow this order on any broken-cluster task so you find the failing layer before changing anything.
- Reproduce the symptom and note exactly what fails, such as kubectl connection, node status, Pod status or a client request.
- Choose your tools by API availability: kubectl describe, events and logs if the API works; ssh, crictl, systemctl and journalctl if it does not.
- Narrow to one layer using the symptom map: node, control plane component, scheduling, container start, application, Service, DNS or policy.
- Fix the root cause at its source, such as the manifest file, kubelet configuration, Pod spec or selector, changing one thing at a time.
- Confirm recovery the way the grader will, and make the fix persist, for example by enabling services and checking that static Pods stay Running.
Practice question
Want more like this? The CKA practice test has questions for every domain.
Checklist
0 of 14
Tick each one you can do without notes. Your ticks stay in this browser.
Study strategy
Start by learning a healthy cluster in detail: where each static Pod manifest lives, what kubectl get pods -n kube-system shows, what the kubelet's systemd unit and configuration look like, and how crictl ps lists the control plane. You cannot spot a broken file without knowing the working one.
Then practise breaking things on purpose, one fault at a time: stop and disable the kubelet, misspell a flag in kube-apiserver.yaml, point the scheduler at a missing kubeconfig, remove a Service selector label, add a default-deny policy, scale CoreDNS to zero. For each, write down the first symptom you saw and the command that revealed the cause. That symptom map is what you carry into the exam.
Candidates most often underestimate working without the API server, and the find-and-report tasks with kubectl top and kubectl logs that need exact output in a file. Drill crictl and the logs flags until they are automatic.
Afterwards, take full timed mock exams. Troubleshooting tasks can absorb as much time as you give them, so set a personal cap per task, flag it, and return once the quicker tasks are done.
FAQ
How much of the CKA is Troubleshooting?
The Linux Foundation weights it at 30%, the largest domain. Practice on deliberately broken clusters pays back directly here.
What do I do when kubectl cannot connect?
SSH to the control plane node and use crictl ps -a and crictl logs, or the files in /var/log/pods, to see why kube-apiserver or etcd is failing. Then fix the static Pod manifest in /etc/kubernetes/manifests.
Is metrics-server installed on the exam clusters?
Tasks that ask you to use kubectl top expect the Metrics API to be available. If kubectl top reports that the Metrics API is unavailable, check whether metrics-server is running in kube-system.
How do I see why a container keeps restarting?
Check kubectl describe pod for the last state and reason, such as OOMKilled or Error with an exit code, then read kubectl logs --previous for the output of the crashed instance.
Why are my Pods Pending with no events?
If no Pod is being assigned a node and there are no scheduling events, the scheduler itself is likely down. Pending with FailedScheduling events instead points to resources, taints, affinity or claims.
How should I manage time on troubleshooting tasks?
Set a cap per task, use the symptom map to reach the right layer quickly, and skip to another task if you are stuck, since tasks can be answered in any order.
How does Troubleshooting relate to the other domains?
It reuses all of them: control plane files from Domain 1, scheduling and probes from Domain 2, Services and DNS from Domain 3, and claims from Domain 4.
Test your Domain 5 knowledge
Practise diagnosing broken nodes, control plane components and Services against the clock, using the layer-by-layer method from this page.