Networking, storage and compute best practices, cost management, multi-tenancy
OpenCost — Allocates Kubernetes cost per namespace, workload and label. Shows what each team spends so right-sizing and showback are possible. Docs
Right-sizing — Requests that match real usage — VPA recommendations, then HPA for scale. Over-requested Pods waste nodes; under-requested ones get throttled or OOMKilled. Docs
Multi-tenancy — Namespaces per team with quotas, RBAC and NetworkPolicies. Soft multi-tenancy shares a cluster safely; ResourceQuota and LimitRange keep one tenant from starving others. Docs
StorageClass tiers — Offer storage as named tiers (fast-ssd, standard) with dynamic provisioning. Platform users ask for a class; the CSI driver and reclaim policy handle the rest. Docs
Gateway API — Platform owns the Gateway; app teams attach their own HTTPRoutes. Role separation built into the API — a natural fit for platform networking. Docs
GitOps and Continuous Delivery (25%)
GitOps for apps and infrastructure, CI/CD pipelines on Kubernetes, progressive delivery
Argo CD — Applications and ApplicationSets syncing apps and infra from Git. Automated sync with prune and selfHeal keeps clusters matching Git; ApplicationSets fan out to many clusters. Docs
Flux — GitRepository + Kustomization / HelmRelease controllers. `flux bootstrap github` installs Flux and commits its own config to the repo it then reconciles. Docs
Tekton — Kubernetes-native CI: Task, Pipeline, PipelineRun. Each step runs as a container in a Pod; workspaces share data between tasks. Docs
Argo Rollouts — Canary and blue-green with automated analysis. Steps like setWeight and pause, with AnalysisTemplates checking metrics before promotion. Docs
Flagger — Progressive delivery operator for Flux and service meshes / ingress. A Canary resource wraps a Deployment; Flagger shifts traffic and rolls back on failed metric checks. Docs
Platform APIs and Self-Service Capabilities (25%)
CRDs for platform services, self-service workflows, operators, automation frameworks
CRD design — An OpenAPI schema, versions, status subresource and validation rules. Good platform APIs are small, validated (x-kubernetes-validations CEL rules) and report status conditions. Docs
Crossplane — Compositions turn cloud resources into one simple platform API. A CompositeResourceDefinition (XRD) defines the API; a Composition maps it to managed resources from providers. Docs
Operators — Controllers that automate day-2 operations for a service. Built with frameworks like Kubebuilder or Operator SDK; they reconcile custom resources into running software. Docs
Self-service provisioning — A PR or portal form creates a claim; the platform provisions it automatically. Git (GitOps) or a portal (Backstage templates) creates the custom resource; controllers do the rest. Docs
Prometheus — Metrics and alerting rules, routed by Alertmanager. ServiceMonitors (Prometheus Operator) select what to scrape; alert rules page on symptoms. Docs
Grafana — Dashboards over metrics, logs and traces. One pane for platform and app teams, with data sources like Prometheus, Loki and Tempo/Jaeger. Docs
Jaeger — Distributed tracing backend, fed by OpenTelemetry. Follow one request across services to find where latency or errors come from. Docs
Deployment metrics — DORA’s five: lead time, deployment frequency, recovery time, change fail rate, rework rate. Measure delivery performance from pipeline and GitOps events to find and fix bottlenecks. Docs
Incident remediation — Find the failing layer with events, logs and metrics, then fix it declaratively. Typical exam-style issues: Pending Pods, failed syncs, broken probes, exhausted quotas. Docs
Service mesh mTLS — Istio or Linkerd encrypt and authenticate service-to-service traffic. Istio PeerAuthentication STRICT; Linkerd enables mTLS automatically for meshed Pods. Docs
OPA Gatekeeper — ConstraintTemplates (Rego) and Constraints enforced at admission. Kyverno does the same with YAML-native policies; both also audit existing resources. Docs
Platform RBAC — Tenants manage their namespaces; only the platform team manages the platform. Aggregate ClusterRoles for custom resources so tenants can use platform APIs without broader rights. Docs
SBOM & scanning in pipelines — Generate SBOMs and scan images before deploy; fail on critical CVEs. e.g. Trivy in the Tekton pipeline, results attached as attestations, verified at admission. Docs
Audit trails — Kubernetes audit logs plus Git history and policy reports. Who changed what, when — and whether it complied — for compliance reporting. Docs
Practice questions
Which Kubernetes object gives each tenant team a default request/limit per container?
Answer: LimitRange. ResourceQuota caps the namespace total.
Which component recommends right-sized CPU/memory requests from observed usage?
Answer: Vertical Pod Autoscaler (in recommendation mode). Recommendation-only mode avoids restarting Pods.
How does OpenCost attribute shared node cost to workloads?
Answer: By each workload’s resource requests/usage share of node prices. It also accounts for idle and shared costs.
Which practice reduces wasted capacity in a multi-tenant cluster most directly?
Answer: Setting realistic requests and autoscaling nodes and workloads. Over-requested Pods block capacity that sits idle.
What is a hierarchical way to give teams self-managed sub-namespaces?
Answer: A multi-tenancy controller (e.g. HNC or Capsule) on top of namespaces. Policies and quotas propagate from the parent.
Why separate system and tenant workloads onto different node pools?
Answer: Isolation and capacity guarantees for platform components. Taints and tolerations keep tenant Pods off system nodes.
Which storage practice suits a platform offering databases on demand?
Answer: StorageClasses with dynamic provisioning, expansion and snapshot support. VolumeSnapshotClasses enable backups via snapshots.
Which Argo CD feature lets one Application deploy the same app to every cluster with a label env=prod?
Answer: An ApplicationSet with a cluster generator and a label selector. Clusters are registered in Argo CD with labels.
In Flux, which resource reconciles a Helm chart into the cluster?
Answer: HelmRelease. HelmRepository/OCIRepository provides the chart source.
In Flux, how do you make one Kustomization wait for another (e.g. CRDs before apps)?
Answer: spec.dependsOn. Flux applies dependencies first and waits for readiness.
How do you suspend Flux reconciliation during an incident?
Answer: flux suspend kustomization <name>. flux resume restarts it; remember to commit the fix to Git.
Which Tekton object defines a reusable series of steps (containers)?
Which Argo Rollouts feature promotes a canary only when Prometheus metrics look healthy?
Answer: An AnalysisTemplate referenced in the canary steps. Failed analysis aborts and rolls back.
What does Flagger need to shift traffic for a canary?
Answer: A supported service mesh or ingress/gateway provider. Flagger supports Istio, Linkerd, Gateway API, Contour and others.
How should CI hand off to GitOps CD?
Answer: CI pushes the image and commits the new tag to the environment’s desired state. The GitOps agent deploys from Git.
How can a platform deploy infrastructure (not just apps) with GitOps?
Answer: Commit Crossplane (or other operator) custom resources to the repo the agent reconciles. Infrastructure becomes Kubernetes resources reconciled like apps.
What does a blue/green deployment need to switch traffic instantly?
Answer: Two full versions running and a single routing switch (Service or route). Rollback is switching back.
Which CRD field makes a custom resource’s status writable only through the status subresource?
How do you add a column like "READY" to kubectl get for a custom resource?
Answer: additionalPrinterColumns in the CRD version. Columns are JSONPath expressions.
Which CRD validation feature expresses cross-field rules like "min <= max"?
Answer: x-kubernetes-validations with CEL rules. Rules run in the API server, no webhook needed.
Which field defines whether a custom resource is namespaced or cluster-wide?
Answer: spec.scope (Namespaced or Cluster). Platform APIs for teams are usually namespaced.
Which frameworks are common for building operators in Go?
Answer: Kubebuilder; Operator SDK. Both use controller-runtime.
What is a Crossplane Composite Resource Definition (XRD)?
Answer: The schema of a new platform API (the composite resource type). A Composition implements it with managed resources.
What does a Crossplane Composition do?
Answer: Maps a composite resource to a set of managed resources (e.g. network + database). Composition functions allow logic in that mapping.
Which Crossplane object holds the credentials a provider uses?
Answer: ProviderConfig (referencing a Secret or workload identity). Managed resources reference a ProviderConfig.
Why should a platform API report status conditions?
Answer: So users and GitOps tools know when the requested resource is ready or failing. Argo CD and Flux health checks read conditions.
What is a finalizer used for in an operator?
Answer: Running cleanup (e.g. deleting a cloud resource) before the custom resource is removed. The object stays until the controller removes the finalizer.
How should breaking changes to a CRD be introduced?
Answer: A new API version with conversion, keeping the old version served during migration. Mark one version as the storage version.
What does a Backstage Software Template typically create for self-service provisioning?
Answer: A repository and/or a pull request with the custom resource the platform reconciles. GitOps then provisions the resource.
What is the advantage of self-service via pull requests over direct API calls?
Answer: Review, history and policy checks on every request. GitOps gives auditability for free.
Which Prometheus Operator resource selects Services to scrape?
Which Prometheus Operator resource defines alerting rules?
Answer: PrometheusRule. The operator loads matching rules into Prometheus.
Which OpenTelemetry Collector deployment fits collecting node-level logs and metrics?
Answer: An agent DaemonSet on every node. A gateway Deployment then processes centrally.
How do you measure deployment frequency from GitOps tools?
Answer: Count successful syncs/reconciliations of production per period from events or metrics. Argo CD and Flux expose sync events and metrics.
Which metric reveals that a GitOps agent is failing to apply changes?
Answer: Reconciliation/sync failure counts or Ready=False conditions on its resources. Alert on resources stuck not Ready.
A Deployment rollout is stuck. Which command shows why?
Answer: kubectl rollout status deployment/<name> and kubectl describe on the new Pods. Look for failing probes, image pulls or quota errors.
Pods are Pending with "exceeded quota". What is the fix?
Answer: Raise the namespace ResourceQuota or lower the Pods’ requests. Quota violations are rejected at admission or block scheduling.
An Argo CD app is OutOfSync forever because an HPA changes replicas. Fix?
Answer: Ignore /spec/replicas in ignoreDifferences (or remove replicas from Git). The HPA owns replicas, not Git.
Which signal best shows user-facing latency of a platform service?
Answer: Request duration histograms (e.g. p95/p99). Use SLOs on latency percentiles.
What is mean time to restore (failed deployment recovery time) improved by?
Answer: Fast detection, easy rollbacks and good runbooks. GitOps rollbacks and progressive delivery help directly.
Which Linkerd feature encrypts traffic between meshed Pods without configuration?
Answer: Automatic mTLS. Meshed Pods get identities and mTLS by default.
Which Gatekeeper object defines a reusable policy written in Rego?
Answer: ConstraintTemplate. A Constraint instantiates it with parameters and a match scope.
How does Gatekeeper report violations by existing resources?
Answer: Its audit function records violations in the Constraint’s status. Kyverno uses PolicyReports for the same purpose.
How should platform RBAC let tenants use a custom platform API?
Answer: Grant create/get/list on the custom resource in their namespaces via a Role or aggregated ClusterRole. Aggregated ClusterRoles can extend the built-in edit/view roles.
Where should image vulnerability scanning gate deployments?
Answer: In CI before publishing; At admission, verifying scan attestations. Scan early and verify at the door.
What do Kubernetes audit logs provide for compliance?
Answer: A record of who changed what and when in the cluster. Combine with Git history for a full audit trail.
Which tool signs and attests Tekton pipeline outputs automatically?
Answer: Tekton Chains. It produces signatures and SLSA provenance.
Finance wants each team’s monthly Kubernetes spend. Which tool fits?
Answer: OpenCost. OpenCost allocates cluster costs to namespaces, workloads and labels.
One team’s namespace keeps starving others of CPU. What do you configure?
Answer: A ResourceQuota (and LimitRange defaults) on that namespace. Quotas cap total requests/limits per namespace; LimitRanges give defaults per Pod.
A platform must deploy the same add-ons to 30 clusters from Git. Which approach?
Answer: An Argo CD ApplicationSet with a cluster generator (or Flux Kustomizations per cluster). GitOps fan-out keeps every cluster continuously reconciled from one source.
Release v2 to 10% of users and roll back automatically if the error rate rises. Which tools fit?
Answer: Argo Rollouts with an AnalysisTemplate; Flagger with a Canary resource. Both implement metric-gated progressive delivery.
Which Tekton resource executes a Pipeline once with concrete parameters?
Answer: PipelineRun. Task and Pipeline are definitions; TaskRun/PipelineRun are executions.
Teams should request a Postgres database with five lines of YAML. What do you build?
Answer: A Crossplane XRD + Composition (or a CRD with an operator) exposing a simple Database API. Composite resources/CRDs hide provider details behind a small, validated API.
How should a CRD reject a spec where minReplicas > maxReplicas at admission time?
Answer: A CEL rule in x-kubernetes-validations in the CRD schema. CEL validation rules run in the API server without a webhook.
Requests are slow and you need to know which service in the chain adds the latency. Which signal?
Answer: Distributed traces (e.g. OpenTelemetry → Jaeger). Traces show per-hop timing for a single request across services.
An Argo CD Application is Synced but Pods are Pending. Where do you look first?
Answer: kubectl describe pod — scheduling events (insufficient resources, taints, quotas). Sync only means manifests were applied; Pending is a scheduling problem.
Only images from the internal registry may run anywhere on the platform. How?
Answer: A Kyverno or OPA Gatekeeper policy enforced at admission. Admission policies reject non-compliant Pods before they are created.
Builds must fail when an image has critical CVEs, and an SBOM must be kept. Where?
Answer: In the CI pipeline: scan + SBOM generation (e.g. Trivy), stored as attestations. Shift-left: catch it in the pipeline, verify the attestation at admission.