Glenn's Digital Garden

Confidential Containers

Architecture

CNCF’s Confidential Containers architecture is composed of a few components:

  • Kata Containers provides the runtime that creates confidential virtual machines (CVMs) which serve as the secure enclaves.
  • CNCF Trustee, which provides attestation and verification services
  • CNCF guest components, which provides the clients that know how to talk to the guest’s hardware root of trust and Trustee

The flow of provisioning a confidential container is as follows:

  1. Proving what is running before anything is released. When the CVM boots, it contains no keys or credentials and must retrieve them from a trustworthy source: KBS. Running a container requires pulling an image security policy for the CVM from the KBS itself, pulling the image manifest and manifest signature from the container registry, then retrieving the signer’s public key from the KBS. Every KBS access must be preceded by attestation, where the CVM presents a hardware-signed fingerprint of its contents that the KBS/AS check against known-good values in RVPS. This attestation can be cached for a short duration, so this step is where the CVM proves what is running.
  2. Delivering the container without exposing it. Once the CVM has proof of attestation, it retrieves everything from the KBS that it needs to pull and decrypt the image: image policy for the CVM, image signer’s pubkey, and layer encryption keys. The image is pulled, decrypted, and verified entirely within the CVM, ensuring that it is never exposed to the untrusted host.
  3. Decrypting model weights and other proprietary data inside the container. Containers may contain encrypted data, keyed to different keys than those used to encrypt the container itself. Decrypting these in-container objects requires another decryption key, retrieved from the KMS by the KBS upon request by the CVM. This decryption key is injected directly into the container, and it is used to decrypt model weights into the CVM’s encrypted memory so that the weights are never exposed.

Hardware support

Unlike CPU technologies, GPUs only encrypt host-to-device (Hopper) and device-to-device communications (Blackwell and newer).1 Although NVIDIA’s own literature says “the private keys used to encrypt and decrypt the virtual machine and GPU memories,”2 all other circumstantial evidence suggests that GPU memory is protected at its boundaries, not through encryption.

CPUs

CPUs have supported trusted execution environments for a long time. They either use a dedicated microcontroller (the AMD ASP) or an isolated set of CPU resources inaccessible to the host OS or hypervisor (Intel TDX and Arm CCA) to serve as a trusted third party on the server. These technologies protect the vendor-signed, CPU-specific private key that each CPU is given at manufacturing and provide services for encrypting memory for virtual machines and providing signed attestations of the CPU and VM configurations.

GPUs

Adding NVIDIA GPUs to the mix adds complexity. Confidential containers (and trusted execution environments, TEEs) are designed to protect the memory of confidential containers from being read by host devices (like GPUs). However, confidential AI requires that GPUs have access to guest memory, and it also requires that the GPU’s own memory and communications with confidential host memory be encrypted. As a result, GPUs must be brought inside the TEE so they can access host memory that the host and its devices cannot.

The problem is that CPU attestation only describes the guest VM’s (the TEE’s) configuration and the silicon root of trust’s own firmware. It confirms that the VM is using encrypted memory, but that’s it. It says nothing about PCIe devices, host firmware versions, or anything else on the host, because all of that is assumed to be hostile.

Because the CPU attestation says nothing about the GPU’s use of encrypted memory, the GPU needs its own attestation process. And a verifier must also verify that the GPU’s firmware and VBIOS are not compromised, and that the GPU is running in Confidential Computing mode with encrypted host-device and device-device links.

CNCF’s Trustee, the reference implementation of Confidential Containers, contains support for adding NVIDIA GPUs into TEEs.

Just as the confidential VM collects signed evidence from its CPU root of trust to attest that the confidential VM is running in confidential mode and meets the minimum trusted computing base, the confidential VM also collects signed evidence from its GPU’s root of trust. Both CPU and GPU evidence are sent to the Attestation Service which evaluates the evidence against an attestation policy. If the evidence corresponds to a properly configured, trustworthy TEE, the KBS issues a token carrying the attestation claims that the confidential VM can use to retrieve secrets.

Container runtime support

Confidential containers rely on the following:

Kata

When a Kata VM runs, there is a kata-agent process running inside it that exposes a gRPC-based control API for the host to run commands inside the VM. Kata allows policies to be defined for this control API to limit what the host can do inside of the Kata container.

These policies are specified in a Kata initdata file, a TOML document that injects inline files into the confidential VM when it is provisioned. That initdata may look like this:

version = "0.1.0"
# this file is hashed with sha384 and its digest is included in the guest attestation report
algorithm = "sha384"
 
[data]
"policy.rego" = '''
package agent_policy
 
default CreateContainerRequest := false
default ExecProcessRequest := false
default ReadStreamRequest := false
# ...
'''
 
"aa.toml" = '''
[token_configs]
[token_configs.kbs]
url = "https://kbs.example.com:8080"
cert = "..."
'''
 
"cdh.toml" = '''
[kbc]
name = "cc_kbc"
url = "https://kbs.example.com:8080"
kbs_cert = "..."
 
[image]
image_security_policy_uri = "kbs:///default/security-policy/app"
'''

In the above example, there are three configuration files being injected into the confidential VM:

  1. policy.rego which defines what actions the host is allowed to do inside the guest VM/container. The Kata Agent is a process inside the VM evaluates every incoming request from the host against this policy and refuses anything that isn’t permitted.
  2. cdh.toml which configures the Confidential Data Hub, a process running inside the confidential VM that takes in requests for secrets from the Kata Agent and returns plaintext secrets. Under the hood, the CDH asks the Attestation Agent to generate the attestation evidence report that it presents to an external Key Broker Service (KBS).
  3. aa.toml which configures the Attestation Agent inside the confidential VM. This agent is a process that generates the hardware attestation evidence on-demand.

Together, these three files (and the agents that consume them) within the VM connect policies to identity. The whole initdata file is hashed, and the resulting fingerprint is recorded by the hardware root of trust as part of the VM’s immutable launch context. When the Attestation Agent inside the confidential VM is asked to provide attestation evidence, it retrieves a signed report from the hardware root of trust which includes this fingerprint so that the verifier (the KBS) can confirm that the VM was launched with the correct policies. Editing the initdata file (and the files embedded within it) to alter its policies would change its sha384 fingerprint, which would cause the KBS to reject the attestation and refuse to release any secrets. This means the host that runs the confidential container cannot weaken enforcement and still receive secrets.

Rego

Kata container policies are written in Rego, the policy language used by Open Policy Agent. For example,

You can only allow specific commands to be run inside the VM/container from the host:

# do not allow the host to execute commands inside containers by default
default ExecProcessRequest := false
 
# allow exec only if the whole command is simply whoami (whoami --help will not work)
ExecProcessRequest if {
    concat(" ", input.process.Args) == "whoami"
}

You can also allow only specific container images to be run inside the VM:

default CreateContainerRequest := false
 
CreateContainerRequest if {
    some pulled_image in input.storages
    pulled_image.driver == "image_guest_pull"
    pulled_image.source == "docker.io/me/mycontainer@sha256:47e2..."
}

Trustee

Trustee is the reference implementation of CNCF CoCo’s attestation and secret-release service. It is split into different microservices:

  • Attestation Service (AS) - Receives the evidence (sent by a CDH to the KBS) and produces an Entity Attestation Result (EAR). It parses platform-specific evidence, verifies the signature, appraises measurements against reference values from the RVPS, and applies the attestation policy.
  • Reference Value Provider Service (RVPS) - Supplies known-good values that appraisal policies compare against.
  • Key Broker Service (KBS) - The endpoint that CVMs talk to. Receives resource requests from CDH, forwards submitted evidence to the AS for appraisal, and evaluates its resource policy against the returned EAR to decide whether to release the requested resource.

The KBS also has a pluggable resource backend to store secrets; this can be a Key Management Service (KMS) like HashiCorp Vault. As of September 2026, Trustee’s KBS only supports Vault and the KMSes provided by Alibaba Cloud, AWS, and GCP.

Attestation flow

  1. The AA sends KBS an auth request with protocol version, TEE type, and extra params.
  2. The KBS responds to the AA by creating a session (tracked by cookie), generates a nonce, then returns that nonce as the challenge.
  3. The AA generates an ephemeral key pair for the CVM. It asks the hardware for an attestation report and includes a hash of the runtime data (nonce and ephemeral public key). The hardware returns the evidence and the runtime data hash, signed with a hardware attestation key.
  4. The AA sends the evidence to the KBS along with the full runtime data and initdata (which includes the agent policy)
  5. The KBS delegates appraisal of evidence to the AS.
  6. The AS appraises in two stages:
    • A TEE-specific verifier checks the hardware signature chain and parses the evidence into claims. It also recomputes the hash of the nonce and public key to validate it.
    • It checks the initdata (agent policy) against the hardware-signed digest that was included in the evidence.
    • Claims are compared to RVPS reference values. If policy allows, the AS then creates an EAR, which is a bearer token that contains all the annotated and signed evidence, encoded as a JWT. That EAR is handed back to KBS.
  7. The KBS returns the EAR token to the AA.
  8. The CDH gets the token from the AA.
  9. The CDH asks the KBS for a resource and uses the EAR as a bearer token to prove that it has attested.
  10. The KBS checks the token’s signature, then evaluates the claims against its resource policy. If it passes, the KBS fetches the resource from its backend KMS. It encrypts this resource using the ephemeral pubkey in the EAR, then sends it back to CDH.
  11. CDH decrypts the resource with its private key and returns the plaintext to its caller.

Launching pods

To run a confidential container, you create a pod manifest much in the same way you would any other pod. It would look something like this:

apiVersion: v1
kind: Pod
metadata:
  name: mycontainer
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: ...
spec:
  runtimeClassName: kata-qemu-snp
  containers:
    - name: mycontainer
      image: docker.io/me/mycontainer@sha256:47e2...
      imagePullPolicy: Always

Of note:

  • runtimeClassName is kata-qemu-snp to ensure the pod runs inside a QEMU-based VM with the Kata Agent, CDH, and Attestation Agent running. The VM also runs in a trusted execution environment backed by AMD’s SEV-SNP hardware root of trust.
  • io.katacontainers.config.hypervisor.cc_init_data annotation contains a base64-encoded, gzipped representation of the initdata from above.
  • The image is the same docker.io/me/mycontainer@sha256:47e2... that appears in the Rego policy in the initdata.
  • imagePullPolicy: Always pulls the container image from within the confidential VM (I don’t really understand this…it may be extraneous)

The flow:

  1. kubectl apply updates the pod object
  2. The scheduler detects a pod without an assigned node. It looks for nodes that support the desired runtimeClassName (and the required GPU resource and appropriate node labels), then updates the pod object’s node assignment with the name of one of those nodes.
  3. That worker node’s kubelet detects that there’s now a pod that is assigned to it. It tells containerd to create a sandbox for that pod, then create containers within that sandbox.
  4. containerd sees runtimeClassName: kata-qemu-nvidia-gpu-snp, looks up the handler for it, and hands the action off to the Kata shim.
  5. The Kata shim reads the cc_init_data annotation, decodes, and hashes it. It then boots a QEMU VM with SEV-SNP enabled, passes the initdata hash in the launch context, and attaches the initdata contents inside the VM. It also attaches the GPUs allocated for the pod as VFIO passthrough devices before any attestation happens.
  6. Inside the VM, NVIDIA’s NVRC loads kernel modules, detects that the VM has GPUs, and hands off to kata-agent which runs as PID 1.
    1. Before accepting host requests, kata-agent reads the initdata delivered on the attached device, writes aa.toml and cdh.toml where the Attestation Agent (AA) and Confidential Data Hub (CDH) will find them
    2. kata-agent applies policy.rego so every subsequent host request is evaluated against it.
    3. kata-agent then starts the GPU-aware AA and CDH.
  7. Once the VM is up and kata-agent is serving host requests through the policy, the Kata shim asks kata-agent to create a sandbox and a pause container. The policy must allow this though.
  8. The kubelet, reconciling the pod object against what’s actually running on the node, realizes it needs to create the workload container.
    1. kubelet asks containerd to create a workload container.
    2. containerd delegates to the kata shim which asks kata-agent (in the VM) to create a container for the main app (e.g., an model inference service).
    3. Because the node is configured for guest image pulling, kata-agent checks that the requested container image specified in the pod manifest (docker.io/me/mycontainer@sha256:47e2...) is allowed by policy before pulling it.
  9. kata-agent begins the image pull by handing the job to CDH. To pull an image, CDH first needs the image security policy, which lives in KBS.
    • CDH has no EAR token, so it needs to get one from the AA before it can request the image security policy from the KBS. It follows the steps described in Attestation flow.
    • With the EAR in hand, CDH asks KBS for the image security policy and gets it.
    • CDH applies the image policy. It then decides whether it can pull the image, then pulls the image. If the image is encrypted, it needs another secret from the KBS. If the image is signed, the same is true - image signatures are stored in the KBS.
  10. With the container images pulled, signature verified, and decrypted, the container starts up in the CVM.

Commercial support

NVIDIA supports Confidential Containers: NVIDIA Confidential Containers — NVIDIA Confidential Containers Architecture

OpenShift supports confidential containers: Use cases and ecosystem for OpenShift confidential containers

Secrets

There are a couple of places that a confidential container (or its VM) can have confidential secrets. In all cases, sealed secrets are used to ensure that secrets never appear in plaintext outside of the VM. Sealed secrets look like this in the pod manifest:

apiVersion: v1
kind: Secret
metadata: { name: rag-sealed }
type: Opaque
stringData:
  db-key: "sealed.<header>.<payload>.<signature>"

The payload offers two ways to deliver the secret from an external key management system (KMS).

Path 1: Proxied through KBS

The KBS can store a resource in an external KMS; this is a server-side configuration that is completely transparent to the application side. The secret’s payload simply looks like this:

{
  "version": "0.1.0",
  "type": "vault",
  "name": "kbs:///default/rag/db-key",
  "provider": "kbs",
  "provider_settings": {},
  "annotations": {}
}

It’s up to the KBS to know that kbs:///default/rag/db-key is backed by an external KMS. It then

  1. calls the external KMS with its own credentials
  2. gets the plaintext secret
  3. encrypts it with the VM’s throwaway key and returns it to the CDH

Path 2: CDH directly fetches from KMS

Alternatively, a CDH may be preconfigured with the location of a KMS from the pod manifest. In this case, it can retrieve a secret directly from the KMS. However, Trustee only supports AWS and Alibaba’s KWSes in this mode.3 Those secret payloads would look like this:

{
  "version": "0.1.0",
  "type": "vault",
  "name": "rag-db-key",
  "provider": "aws",
  "provider_settings": { "region": "us-east-1" },
  "annotations": {}
}

The credential to access the AWS KMS must be embedded in the CDH’s config, which itself is embedded as toml within the VM’s initdata:

[[credentials]]
resource_uri = "kbs:///default/aws-credential/rag"
path = "/run/confidential-containers/cdh/kbs/aws-credential/rag"

The KMS credential is stored in the KBS which is how attestation is required before retrieving this secret.

Useful terms

Confidential Containers terms:

  • The Key Broker Service (KBS) is the front door for the entire attestation service running within the trusted domain. CDHes talk to the KBS

Policies and secrets:

  • Attestation policy is a policy, enforced by the Attestation Service (AS) in Trustee, that determines whether the TEE is considered trustworthy..
  • Resource policy is a policy, enforced by the Key Broker Service (KBS) in Trustee, that determines whether a requestor is allowed to access a specific resource (typically a secret).
  • Agent policy is a policy, enforced by kata-agent within the CVM, that governs what actions the host hypervisor is allowed to take inside the CVM.
  • Image security policy4 protects a CVM from bad images and is stored in the KBS. The CDH must retrieve it from the KBS as part of the first image pull.
  • Sealed secret is a type of secret, specified within a pod manifest, that is only unsealed after attestation.5

General terms:

  • Claims are unverifiable. If a hardware root of trust says it is running firmware version 12, you just have to trust that is true.
  • Measurements are verifiable because they can be independently reproduced, typically because they are a hash/digest of content. If you know the expected content, you can hash it yourself and directly compare it to the reported measurement.

Footnotes

  1. [2606.23969v2] The Serialized Bridge: Understanding and Recovering LLM Serving Performance under Blackwell GPU Confidential Computing ↩

  2. nvidia-secure-ai-with-blackwell-and-hopper-gpus-whitepaper.pdf ↩

  3. https://github.com/confidential-containers/guest-components/blob/bdbc6ea59cc5bb31c1103a87fe170a2807ceb501/confidential-data-hub/kms/src/plugins/mod.rs#L54 ↩

  4. https://github.com/kata-containers/kata-containers/blob/3a9fd86ccc710ce21d477c888c400187889faaae/src/agent/README.md?plain=1#L136 ↩

  5. Sealed Secrets | Confidential Containers ↩

Graph View

Backlinks