kubeadm, upgrades, HA control planes, RBAC, Helm/Kustomize, CRDs and operators
kubeadm — Bootstraps a cluster: kubeadm init on the first node, kubeadm join on the rest. `kubeadm init` generates the cluster CA and certificates, writes static Pod manifests for the control plane and prints a join command. `kubeadm join` adds nodes using a bootstrap token. It expects a container runtime, kubelet and swap/kernel settings to be prepared first. Docs
kubeadm upgrade — Upgrades the control plane one minor version at a time. Upgrade the kubeadm package, run `kubeadm upgrade plan` then `kubeadm upgrade apply v1.x.y` on the first control-plane node (`kubeadm upgrade node` on the others), then drain each node, upgrade kubelet and kubectl, restart kubelet and uncordon. Skipping minor versions is not supported. Docs
etcdctl snapshot — Backs up (and restores) the whole cluster state stored in etcd. `etcdctl snapshot save` with --endpoints, --cacert, --cert and --key writes a point-in-time backup of etcd. Restore with `etcdutl snapshot restore --data-dir <new dir>` (etcdctl's restore is deprecated), then point the etcd static Pod's hostPath at the new data directory. Docs
HA control plane — Several control-plane nodes behind a load balancer, with stacked or external etcd. kubeadm supports two HA topologies: stacked (etcd runs on each control-plane node) and external etcd (a separate etcd cluster). Either way the API servers sit behind a load balancer set with --control-plane-endpoint, and you need an odd number of etcd members for quorum. Docs
RBAC — Roles grant verbs on resources; bindings grant roles to users, groups or ServiceAccounts. Role and RoleBinding are namespaced; ClusterRole and ClusterRoleBinding are cluster-wide. A RoleBinding can also reference a ClusterRole to grant it inside one namespace. Check access with `kubectl auth can-i <verb> <resource> --as <user>`. Docs
Helm — The package manager that installs cluster components from charts. `helm repo add`, `helm install <release> <chart> -n <ns> --create-namespace`, `helm upgrade` and `helm rollback` manage versioned releases. Values files override a chart’s defaults. Docs
Kustomize — Template-free overlays built into kubectl (kubectl apply -k). A kustomization.yaml lists resources and layers patches, name prefixes, labels, and generated ConfigMaps/Secrets on top of a base. `kubectl kustomize` prints the result; `kubectl apply -k` applies it. Docs
CustomResourceDefinition (CRD) — Adds new resource types to the API — the base every operator is built on. A CRD teaches the API server a new kind (e.g. a Certificate). An operator is a controller that watches those custom resources and reconciles the real world to match them. Installing an operator usually means applying its CRDs, RBAC and a controller Deployment. Docs
Workloads & Scheduling (15%)
Rollouts, config, autoscaling and where Pods are allowed to land
kubectl rollout — Watches, pauses and undoes Deployment rolling updates. Changing a Deployment’s Pod template creates a new ReplicaSet and shifts Pods over within maxSurge/maxUnavailable. `kubectl rollout status`, `history` and `undo [--to-revision=N]` let you follow it and roll back. Docs
ConfigMap — Non-secret configuration, injected as env vars or mounted files. Create one with `kubectl create configmap --from-literal` or `--from-file`, then consume it via env, envFrom or a volume. Mounted files update in place (env vars do not until the Pod restarts). Docs
Secret — Sensitive values, base64-encoded in the API and mounted into Pods. Base64 is encoding, not encryption — protect Secrets with RBAC and encryption at rest. Types include Opaque, kubernetes.io/tls and kubernetes.io/dockerconfigjson (image pull secrets). Docs
HorizontalPodAutoscaler (HPA) — Scales replicas to hit a target metric such as CPU utilization. `kubectl autoscale deployment web --cpu-percent=50 --min=2 --max=10`. Needs metrics-server and CPU requests on the containers, since utilization is measured against the request. Docs
Node affinity — Hard (required) or soft (preferred) rules for which nodes a Pod may use. requiredDuringSchedulingIgnoredDuringExecution must match or the Pod stays Pending; preferredDuringScheduling… only scores nodes. nodeSelector is the simple equality-only version. Docs
Taints & tolerations — Nodes repel Pods with taints; Pods opt back in with tolerations. `kubectl taint nodes n1 key=value:NoSchedule`. Effects are NoSchedule, PreferNoSchedule and NoExecute (which also evicts running Pods). A toleration only allows a node — it does not attract the Pod to it. Docs
Requests & limits — Requests drive scheduling; limits cap usage at runtime. The scheduler places Pods by the sum of requests. Going over a memory limit gets the container OOMKilled; going over a CPU limit throttles it. A LimitRange can set namespace defaults. Docs
Services & Networking (20%)
Pod connectivity, Services, Network Policies, Gateway API, Ingress and CoreDNS
Service types — ClusterIP (internal), NodePort (every node’s port) and LoadBalancer (external LB). ClusterIP is the default virtual IP. NodePort opens 30000–32767 on every node. LoadBalancer asks the cloud for an external load balancer (and gets a NodePort too). `kubectl expose` creates any of them. Docs
EndpointSlice — The list of ready Pod IPs behind a Service. Built automatically from the Service selector and Pod readiness. No endpoints usually means the selector does not match the Pod labels, or no Pod is Ready. Docs
NetworkPolicy — Pod-level firewall rules by label, namespace and port. Once any policy selects a Pod for a direction (ingress/egress), only traffic explicitly allowed gets through. Enforcement needs a CNI plugin that supports it (Calico, Cilium…). Remember DNS (port 53) when restricting egress. Docs
Gateway API — GatewayClass → Gateway → HTTPRoute: the role-oriented successor to Ingress. Infrastructure providers define a GatewayClass, cluster operators create Gateways (listeners, ports, TLS), and app teams attach HTTPRoutes with matches, header rules and weighted backends. Docs
Ingress — HTTP host/path routing to Services, implemented by an Ingress controller. An Ingress resource does nothing on its own — an Ingress controller (Traefik, HAProxy, Contour…) selected via ingressClassName implements it. Supports TLS via a kubernetes.io/tls Secret. Docs
CoreDNS — Cluster DNS: my-svc.my-ns.svc.cluster.local resolves to the Service IP. Runs as a Deployment in kube-system behind the kube-dns Service; its config is the coredns ConfigMap (Corefile). Pods get it as their nameserver through kubelet’s clusterDNS setting. Docs
Pod network model — Every Pod gets its own IP and can reach every other Pod without NAT. The CNI plugin implements this flat network across nodes. Containers in one Pod share the network namespace and talk over localhost. Docs
Storage (10%)
StorageClasses, dynamic provisioning, access modes and reclaim policies
StorageClass — Names a provisioner and parameters — the "flavor" of storage a claim asks for. One StorageClass can be marked default (annotation storageclass.kubernetes.io/is-default-class). volumeBindingMode: WaitForFirstConsumer delays provisioning until a Pod is scheduled, so the volume lands in the right zone. Docs
Dynamic provisioning — A PVC that names a StorageClass gets a PV created for it on demand. No admin pre-creates PersistentVolumes: the CSI driver behind the StorageClass creates the disk and the PV, then binds it to the claim. Docs
Access modes — ReadWriteOnce, ReadOnlyMany, ReadWriteMany, ReadWriteOncePod. RWO = read-write by one node; ROX = read-only by many nodes; RWX = read-write by many nodes; RWOP = read-write by a single Pod. A claim only binds to a volume that supports the requested mode. Docs
Reclaim policy — What happens to a PV once its claim is deleted: Retain or Delete. Retain keeps the volume and its data (the PV becomes Released and needs manual cleanup). Delete removes the backing storage — the default for dynamically provisioned volumes. Docs
PersistentVolumeClaim (PVC) — A Pod’s request for storage of a size, access mode and class. The claim binds to a matching PV (or triggers dynamic provisioning), and the Pod mounts the claim by name. A PVC stuck Pending usually means no matching PV, no default StorageClass, or WaitForFirstConsumer waiting for a Pod. Docs
Troubleshooting (30%)
Broken nodes, broken control planes, broken apps and broken networking
kubectl describe — Status, conditions and the Events at the bottom — the first stop for a broken Pod. Events tell you why a Pod is Pending (FailedScheduling), why it cannot pull (ErrImagePull) or why probes fail. `kubectl get events -A --sort-by=.lastTimestamp` shows them cluster-wide. Docs
kubectl logs — Container stdout/stderr — add --previous for the instance that just crashed. `kubectl logs pod -c container`, `-f` to follow, `--previous` for the last terminated container (essential in CrashLoopBackOff), `-l app=web` for many Pods. Docs
kubectl top — Live CPU/memory of nodes and Pods, served by metrics-server. `kubectl top nodes` and `kubectl top pods --sort-by=memory` read the resource metrics API, which metrics-server fills from each kubelet. No metrics-server, no numbers (and no HPA). Docs
journalctl -u kubelet — Why a node is NotReady: the kubelet’s own logs on the node. SSH to the node, `systemctl status kubelet`, then `journalctl -u kubelet`. Typical culprits: kubelet stopped, wrong config path or flag, expired certificates, the container runtime being down. Docs
crictl — Talks to the container runtime directly — works even when the API server is down. `crictl ps -a`, `crictl logs <id>`, `crictl pods` and `crictl inspect` show what the runtime is actually running on a node, including crashed control-plane containers. Docs
Static Pod manifests — Control-plane Pods live as files in /etc/kubernetes/manifests. The kubelet runs whatever is in its staticPodPath, no API server needed. A typo in kube-apiserver.yaml there takes the API down; fix the file and the kubelet recreates the Pod. Docs
kubectl debug — Attaches an ephemeral debug container to a running Pod (or a node). `kubectl debug -it pod --image=busybox --target=app` shares the target’s process namespace — handy for distroless images with no shell. `kubectl debug node/n1 -it --image=ubuntu` gives a shell on a node. Docs
Debugging Services — DNS → Service → endpoints → Pod port, checked one hop at a time. From a test Pod: does the name resolve (nslookup)? Does the Service have endpoints? Does targetPort match the container port? Is kube-proxy running? Is a NetworkPolicy blocking it? Docs
Practice questions
Which command prints the join command for a new worker node on a kubeadm cluster?
Answer: kubeadm token create --print-join-command. Bootstrap tokens expire (24 h by default); this creates a fresh one with the full join command.
Before kubeadm init, what must every node have?
Answer: A container runtime (e.g. containerd); kubelet installed. kubeadm also checks kernel settings such as IP forwarding; etcd is set up on control-plane nodes by kubeadm itself.
How do you see which kubeadm-managed certificates expire soon?
Where are a kubeadm cluster’s CA and component certificates stored?
Answer: /etc/kubernetes/pki. The etcd certificates are under /etc/kubernetes/pki/etcd.
Which RBAC objects let a ServiceAccount read Pods in every namespace?
Answer: A ClusterRole with get/list/watch on pods and a ClusterRoleBinding. Cluster-wide access needs ClusterRole + ClusterRoleBinding.
Which command creates a Role that can get and list pods in namespace dev?
Answer: kubectl create role pod-reader --verb=get,list --resource=pods -n dev. Then bind it with kubectl create rolebinding … --role=pod-reader --serviceaccount=dev:app.
How do you reference a ServiceAccount when creating a RoleBinding imperatively?
Answer: --serviceaccount=<namespace>:<name>. The format is namespace:name, e.g. --serviceaccount=dev:ci.
Which Helm command shows the values a release was installed with?
Answer: helm get values <release>. helm show values <chart> shows a chart’s defaults instead.
Which command renders a Kustomize directory without applying it?
Answer: kubectl kustomize <dir>. kubectl apply -k <dir> applies the same output.
How do you see which custom resource types an operator installed?
Answer: kubectl get crd. Each CRD adds a kind; kubectl api-resources lists them too.
Which interface does the kubelet use to talk to containerd?
Answer: CRI (Container Runtime Interface). CNI is networking, CSI is storage.
Which command creates a Deployment named web with 3 replicas of nginx:1.27?
Answer: kubectl create deployment web --image=nginx:1.27 --replicas=3. kubectl run creates a single Pod; --replicas is a create deployment flag.
How do you change the image of container app in Deployment web?
Answer: kubectl set image deployment/web app=registry/app:2.0. set image updates the Pod template and triggers a rollout.
How do you scale Deployment web to 5 replicas?
Answer: kubectl scale deployment web --replicas=5. If an HPA manages it, the HPA may change the count again.
Which command creates an HPA for Deployment web between 2 and 10 replicas at 70% CPU?
Answer: kubectl autoscale deployment web --min=2 --max=10 --cpu-percent=70. Needs metrics-server and CPU requests on the containers.
A ConfigMap is mounted as a volume. What happens when you update the ConfigMap?
Answer: The mounted files update after a short delay (unless mounted with subPath). Environment variables from a ConfigMap never update in a running container.
Which resource gives Pods a scheduling priority so more important Pods can preempt others?
Answer: PriorityClass. Pods reference it with priorityClassName.
Which command exposes Deployment web on port 80, forwarding to container port 8080, as a NodePort?
Answer: kubectl expose deployment web --port=80 --target-port=8080 --type=NodePort. expose copies the Deployment’s selector into the new Service.
Which NetworkPolicy field selects Pods in other namespaces as traffic sources?
Answer: namespaceSelector. Use namespaceSelector with a label such as kubernetes.io/metadata.name.
In one `from` entry, namespaceSelector and podSelector are both set. What does it mean?
Answer: Pods matching podSelector inside namespaces matching namespaceSelector. As separate list items (two dashes) they would be ORed instead — a classic mistake.
A default-deny egress policy is in place and Pods can no longer resolve DNS. What fixes it?
Answer: Allow egress to kube-dns on UDP and TCP port 53. Egress rules must explicitly allow DNS.
Which Gateway API resource is created by the cluster operator to define listeners (ports, hostnames, TLS)?
Answer: Gateway. GatewayClass is the implementation; Gateway the listeners; routes attach to it.
How does an HTTPRoute split traffic 80/20 between two Services?
Answer: Two backendRefs with weight 80 and weight 20. Weights are relative within a rule’s backendRefs.
Which Ingress field selects the Ingress controller that should implement it?
Answer: spec.ingressClassName. One IngressClass can be marked as the cluster default.
Where is the CoreDNS configuration (Corefile) stored?
Answer: The coredns ConfigMap in kube-system. Edit it to add stub domains or forwarders; CoreDNS reloads it.
A StorageClass uses volumeBindingMode: WaitForFirstConsumer. A new PVC shows Pending. Is that a problem?
Answer: No — it binds once a Pod using it is scheduled. Delaying binding lets the volume be created in the Pod’s zone.
How do you make a StorageClass the cluster default?
Answer: Annotate it storageclass.kubernetes.io/is-default-class=true. Only one StorageClass should carry the annotation.
Which field allows a bound PVC to be resized (when the StorageClass supports it)?
Answer: allowVolumeExpansion: true on the StorageClass. Then edit the PVC’s spec.resources.requests.storage.
A PV with reclaim policy Retain shows status Released after its PVC was deleted. Can a new PVC bind it?
Answer: Not until an admin cleans it up (e.g. removes spec.claimRef). Retain keeps data safe from automatic reuse.
A Pod is Pending with "0/3 nodes are available: 3 Insufficient cpu". What do you do?
Answer: Lower the Pod’s CPU request or add capacity. Scheduling uses requests; limits do not affect placement.
A Pod is Pending: "1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }". What does it mean?
Answer: The only free node is tainted and the Pod has no matching toleration. Add a toleration only if the Pod really should run there.
A Pod shows ErrImagePull, then ImagePullBackOff. Which causes are likely?
Answer: A typo in the image name or tag; A private registry without imagePullSecrets. kubectl describe pod shows the exact pull error.
A container exits immediately with code 1 and restarts repeatedly. Where do you look first?
Answer: kubectl logs <pod> --previous. The previous container’s output usually names the error.
Exit code 137 on a container usually means…
Answer: It was killed with SIGKILL, often OOMKilled. 137 = 128 + 9 (SIGKILL). describe shows Reason: OOMKilled when memory was the cause.
How do you list events in a namespace sorted by time?
Answer: kubectl get events --sort-by=.lastTimestamp. Events expire after an hour by default.
kubectl get nodes fails with "connection refused" on port 6443. What is most likely?
Answer: The kube-apiserver is not running. Check the static Pod manifest and crictl ps -a on the control-plane node.
The kube-scheduler static Pod is crashing. Where is its manifest on a kubeadm cluster?
Answer: /etc/kubernetes/manifests/kube-scheduler.yaml. Fix the file; the kubelet restarts the Pod automatically.
New Pods stay Pending with no events from the scheduler at all. What is the likely cause?
Answer: kube-scheduler is not running. Without the scheduler, Pods are never assigned and no FailedScheduling events appear.
Deployments stop creating Pods after edits, and ReplicaSets do not change. Which component is likely down?
Answer: kube-controller-manager. Controllers reconcile Deployments into ReplicaSets and Pods.
Which command shows the kubelet’s configuration file path and flags on a node?
Answer: systemctl cat kubelet (and the drop-in files it lists). kubeadm clusters keep the config in /var/lib/kubelet/config.yaml.
Which command shows the highest memory consumers among Pods across all namespaces?
Answer: kubectl top pods -A --sort-by=memory. kubectl top reads the metrics API, so it needs metrics-server; -A covers every namespace.
From a debug Pod, nslookup my-svc fails but nslookup my-svc.other-ns works. Why?
Answer: The Service is in another namespace; short names resolve in the Pod’s own namespace. Use <service>.<namespace> or the full FQDN across namespaces.
A Service has endpoints but connections time out. What should you check next?
Answer: That targetPort matches the port the container listens on; Whether a NetworkPolicy blocks the traffic. Endpoints exist, so selectors are fine — look at ports and policies.
Which crictl command lists all containers including exited ones?
Answer: crictl ps -a. Then crictl logs <container-id> to read its output.
You need a backup of etcd on a kubeadm control-plane node. Which command produces one?
Answer: etcdctl snapshot save /backup/etcd.db with --endpoints, --cacert, --cert and --key. etcd holds every object in the cluster. etcdctl snapshot save, authenticated with the etcd client certs from /etc/kubernetes/pki/etcd, writes a restorable snapshot. `get all` misses most resource types and every Secret detail you would need.
Upgrading the first control-plane node of a kubeadm cluster by one minor version — what is the right order?
Answer: Upgrade kubeadm → kubeadm upgrade apply → drain → upgrade kubelet & kubectl → restart kubelet → uncordon. kubeadm itself goes first (it performs the upgrade), the control plane is upgraded with `kubeadm upgrade apply` (other control planes use `kubeadm upgrade node`), then the node is drained before kubelet changes. Minor versions cannot be skipped.
ServiceAccount "ci" in namespace dev must be able to list Pods in dev — and nothing else. What do you create?
Answer: A Role allowing get/list on pods in dev, and a RoleBinding to the ci ServiceAccount. Namespaced permission = Role + RoleBinding. A ClusterRoleBinding would grant it in every namespace, and ServiceAccounts start with no API permissions beyond discovery.
How do you check whether user jane is allowed to delete Deployments in namespace prod?
Answer: kubectl auth can-i delete deployments -n prod --as jane. `kubectl auth can-i` asks the API server to evaluate RBAC for you, and `--as` impersonates the user.
Deployment "web" just rolled out a broken image. How do you return to the previous version?
Answer: kubectl rollout undo deployment/web. Deployments keep old ReplicaSets as revision history. `rollout undo` (optionally `--to-revision=N`) scales the previous ReplicaSet back up.
A Pod must only ever run on nodes labelled disktype=ssd. Which spec guarantees it?
Answer: nodeSelector: disktype: ssd; requiredDuringSchedulingIgnoredDuringExecution node affinity on disktype=ssd. Both nodeSelector and required node affinity are hard constraints. Preferred affinity only scores nodes, and a toleration only allows tainted nodes — it never attracts the Pod.
Node gpu-1 has the taint dedicated=gpu:NoSchedule. What lets a new Pod be scheduled there?
Answer: A toleration with key dedicated, value gpu and effect NoSchedule. Taints repel every Pod that does not tolerate them. A matching toleration is the only way past a NoSchedule taint.
Pods in namespace db must accept traffic only from Pods labelled app=api (in the same namespace). What do you create?
Answer: A NetworkPolicy selecting the db Pods with an ingress rule from podSelector app=api. NetworkPolicies are the Pod firewall. Once a policy selects the db Pods for ingress, only the listed sources are allowed (provided the CNI enforces policies).
A Service exists but `kubectl get endpointslices` shows no endpoints for it. What is the most likely cause?
Answer: The Service selector does not match the Pods’ labels (or no matching Pod is Ready). Endpoints are computed from the selector and readiness. DNS problems would not empty the endpoint list.
Which Gateway API resource holds the HTTP routing rules (paths, headers, backend weights) that attach to a Gateway?
Answer: HTTPRoute. GatewayClass names the implementation, Gateway defines listeners, HTTPRoute (attached via parentRefs) carries the routing rules.
What is the fully qualified DNS name of Service "api" in namespace "shop" (default cluster domain)?
Answer: api.shop.svc.cluster.local. Services resolve as <service>.<namespace>.svc.<cluster-domain>, served by CoreDNS.
A PVC with no storageClassName stays Pending, and the cluster has no default StorageClass. What fixes it?
Answer: Mark a StorageClass as default, or set storageClassName on the claim (or create a matching PV). Without a class (explicit or default) nothing provisions a volume, so the claim can only bind to a pre-created matching PV.
Which reclaim policy keeps a volume’s data after its PersistentVolumeClaim is deleted?
Answer: Retain. Retain leaves the PV Released with its data intact. Recycle is deprecated, and WaitForFirstConsumer is a binding mode, not a reclaim policy.
A worker node shows NotReady. What do you check first, on the node itself?
Answer: systemctl status kubelet and journalctl -u kubelet. Ready is reported by the kubelet. If it is stopped or crashing (bad config, certs, runtime down), the node goes NotReady.
A container is in CrashLoopBackOff. How do you see the output of the instance that just crashed?
Answer: kubectl logs <pod> --previous. The current container may have just restarted with an empty log; --previous shows the last terminated one.
After an edit, kubectl cannot reach the API server on a kubeadm cluster. Where do you look?
Answer: /etc/kubernetes/manifests/kube-apiserver.yaml, plus crictl ps -a and crictl logs on the control-plane node. With the API server down, kubectl is useless. It is a static Pod: fix its manifest file and inspect the container with crictl.