The Container Cost Problem

By the end of this lesson, you can…

• Explain why the tagging-based allocation that works on a cloud bill breaks down inside a single shared Kubernetes cluster.
• Walk the container cost stack from cluster to container and say, at each layer, what changes and what a FinOps analyst needs from it.
• Tell requests, limits, and actual usage apart on a real pod spec, and explain why each one produces a different number.
• Read kubectl top output and a cluster resource-usage export, and know exactly what they do and don't tell you about cost.
• Apply Inform → Optimize → Operate to a container platform, and explain why consistent namespace labeling is the prerequisite the rest of this course depends on.

Why Tagging Stops Working Inside a Cluster

Cost allocation on a cloud bill is mostly a tagging problem: attach a team or cost-center tag to a VM, an S3 bucket, or an RDS instance, and the billing export already carries that tag on every line item. It's imperfect — not everything gets tagged, and shared services still need a rule — but the resource and the bill line are the same object. One EC2 instance, one line item, one tag.

A Kubernetes cluster breaks that one-to-one mapping on purpose. The cloud bill has exactly one line for the nodes underneath your cluster — a handful of large VM instances — and every namespace, deployment, and pod scheduled onto those nodes is invisible to the cloud provider's billing system entirely. AWS, Azure, and GCP bill you for the node, not for the sixty pods from a dozen teams that happen to be running on it at any given moment. Pods are also transient in a way VMs rarely are: a pod can live for ninety seconds during a rolling deploy, get rescheduled to a different node an hour later, and never appear on a monthly cloud invoice as a distinct thing at all.

So the question this whole course answers is: once the cloud bill stops at the node, how do you get from "one shared node bill" back down to "what did the payments team's namespace actually cost this month"? That reconstruction has to happen entirely inside Kubernetes, using data the cloud bill never sees.

The Container Cost Stack

Six layers sit between "the cloud bill" and "a single container," and a different question gets asked at each one:

Namespace is the layer that matters most for this course. It's rarely a perfect proxy for "team" — some organizations run one namespace per team, others run one per environment per team, and shared infrastructure (ingress controllers, the observability stack, kube-system itself) lives in namespaces nobody owns as a product. But it's the layer every allocation method in Weeks 2–5 is built on, because it's the finest-grained boundary Kubernetes itself enforces with RBAC, ResourceQuotas, and NetworkPolicies — the same boundary that already carries organizational meaning is also the one the platform can technically enforce.

Three Numbers, Three Bills: Requests, Limits, and Usage

Every container in a Kubernetes pod can declare two numbers per resource (CPU and memory), and the cluster separately measures a third:

Requests

What a container asks the scheduler to reserve on a node before it will place the pod there. This is the number the scheduler treats as real — it will not place a pod on a node that can't satisfy every container's requests, even if the pod would never actually use that much.

Limits

The ceiling the kubelet enforces once the pod is running. A CPU limit throttles the container's CPU time once it's exceeded; a memory limit gets enforced by the kernel OOM killer terminating the container if it's exceeded. The two resources fail differently, which matters a great deal in Week 3.

Actual usage

What the container really consumes, moment to moment, as reported by the kubelet's cAdvisor via the Metrics API — the number requests and limits were only ever a prediction of.

containers:
  - name: payments-api
    image: registry.example.com/payments-api:1.14.2
    resources:
      requests:
        cpu: 1000m
        memory: 4Gi
      limits:
        cpu: 2000m
        memory: 4Gi

Here's why this matters for cost specifically: the scheduler only ever sees requests. A node with 8 vCPU of allocatable capacity and 6 vCPU already requested looks "full" to the scheduler once another 2 vCPU of requests would push it over — even if every pod on that node is, at that exact moment, using a combined 1.5 vCPU of actual CPU. That gap between what's requested and what's used is capacity you are paying for through the node's price, that no other workload can be scheduled into, and that shows zero activity in a dashboard that only plots usage. It is the single largest source of container waste, and it is the subject of Weeks 3 and 4 in full.

Limits sit in between the two, and they don't change what the scheduler reserves at all — a container can request 500m CPU and set a limit of 4000m, and the scheduler only ever reserves 500m against the node's capacity. Limits change what happens after the pod is already running, not where it gets placed.

Reading Real Resource-Usage Data

Two commands surface the raw material this entire course works from. kubectl top nodes reports each node's current CPU and memory usage against its allocatable capacity, sourced from the metrics-server add-on (not the kubelet directly — metrics-server aggregates cAdvisor data cluster-wide and serves it through the Kubernetes Metrics API):

$ kubectl top nodes
NAME                CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-pool-a-1       5960m        74%    22156Mi         67%
node-pool-a-2       5560m        69%    21308Mi         65%
node-pool-a-3       5160m        65%    19812Mi         60%
node-pool-a-4       4800m        60%    18944Mi         58%
node-pool-a-5       4400m        55%    17408Mi         53%
node-pool-a-6       4000m        50%    16000Mi         49%
node-pool-a-7       3200m        40%    13312Mi         41%
node-pool-a-8       2800m        35%    11776Mi         36%

kubectl top pods -n <namespace> does the same at pod grain:

$ kubectl top pods -n payments --sort-by=cpu
NAME                            CPU(cores)   MEMORY(bytes)
payments-api-7d8f9c-2xk4m       412m         2611Mi
payments-api-7d8f9c-9jn2p       405m         2588Mi
payments-api-7d8f9c-4tz8q       398m         2549Mi
payments-api-7d8f9c-1wq6r       390m         2502Mi
payments-api-7d8f9c-8mv3s       385m         2477Mi
payments-api-7d8f9c-5bx7t       378m         2440Mi
payments-api-7d8f9c-3ny1u       371m         2401Mi
payments-api-7d8f9c-6kp9v       365m         2378Mi
payments-api-7d8f9c-0hs5w       358m         2340Mi
payments-api-7d8f9c-2rt4x       338m         2314Mi
payments-webhook-99c6d-a1b2c    200m         900Mi
payments-webhook-99c6d-d3e4f    200m         800Mi

Both commands are point-in-time snapshots — they show usage right now, not a trend, a percentile, or a peak. A production resource-usage export (the kind you'll work with for the rest of this course) is this same data captured continuously by metrics-server or a Prometheus-based pipeline and aggregated over a window — hourly or daily rollups of average and peak usage per pod, joined against each pod's declared requests and limits. That join is what makes the requests-vs-usage gap visible at all: kubectl top alone never shows you requests, only usage.

Inform, Optimize, Operate — Applied to Containers

The FinOps Foundation's Calculating Container Costs guidance maps the same Inform → Optimize → Operate loop from general FinOps practice onto this container-specific data:

Every week that follows sits inside this loop: Weeks 1–2 are Inform, Weeks 3–4 are Optimize, Week 5 is Operate.

Labeling and Namespace Strategy: The Prerequisite for Everything Else

None of Weeks 2 through 5 work without one thing being true first: every namespace carries consistent, machine-readable ownership metadata. A showback report (Week 2) can't attribute a namespace's cost to a team if the namespace itself doesn't say which team owns it. A rightsizing recommendation (Week 3) can't get routed to the right engineer without knowing who to route it to.

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  labels:
    team: payments
    cost-center: cc-4471
    env: production
  annotations:
    finops.example.com/owner: "payments-eng@example.com"
    finops.example.com/business-unit: "commerce"

The specific label keys above (team, cost-center, env) are illustrative — your organization's schema will differ, and the FinOps Foundation's Container Cost Allocation Labels & Dictionary working group maintains a community-sourced reference for common label patterns if you're designing one from scratch. What matters isn't the exact key names; it's that the schema is applied consistently, enforced somewhere (an admission policy is the Week 5 topic), and treated as a prerequisite rather than a cleanup task for later. A cluster with inconsistent or missing namespace labels can still produce a resource-usage export, but every allocation method in this course stalls on "which team does this belong to" before it gets to "how much did it cost."

Quick Check: Can You Answer "What Does Payments Cost" Yet?

Exercise: Build the Namespace-Level Cost-Driver Map

Below is a resource-usage export summary for a real 8-node shared cluster (prod-shared-01, 8 nodes × 8 vCPU / 32 GiB allocatable each — 64 vCPU / 256 GiB total capacity), rolled up to namespace grain. This is the same cluster you'll allocate cost for in Week 2, rightsize a workload from in Week 3, and bin-pack in Week 4 — so it's worth understanding well now.

Reflection

Putting it together

A cloud bill's tagging model breaks the moment a cluster starts sharing one node's bill across dozens of pods from a dozen teams, which is why this whole course exists as a separate discipline from cloud FinOps generally. Requests, limits, and usage are three different numbers because they answer three different questions — what's reserved, what's the ceiling, and what's actually happening — and the gap between the first and third is where most container waste hides. You now have real resource-usage data and a namespace-level map of what it can and can't answer. Week 2 closes the biggest gap: attaching an actual dollar figure to every namespace on this same cluster.