Building container images
Build a Dockerfile in a task on a server with BuildKit, in a sandboxed runtime, and push the image with a short-lived token of a service account.
This guide builds a Dockerfile in a task that runs on a hammerkit server
and pushes the image to a registry. It sets up the server once, as its operator, and then
writes the build file: a task with build, which is the same on a
dev machine. The example pushes to Google's Artifact Registry from a server on
Kubernetes; the other registries follow the same steps, the differences are named where they
come up. A server on a docker host doesn't build images yet.
Why a server needs a sandbox for it
On a dev machine a task could mount the docker socket and run docker build. On a server that
would hand the task the daemon of the machine, and with it every container of every caller
on it, so the server refuses a mount of the socket. Docker in Docker needs a privileged
container, which a server never starts. BuildKit without root needs what the server takes
from every container: its seccomp profile, allowPrivilegeEscalation: false and the masked
/proc.
What works is BuildKit as root with one more capability, SYS_ADMIN, so it can mount the
layers it builds. SYS_ADMIN is root's power over the kernel, and the kernel of a node runs
the tasks of other callers too. So the server adds it only in a sandboxed runtime, which
gives the container a kernel of its own: gVisor handles the container's
system calls in a kernel it implements itself, Kata Containers
start a small virtual machine per pod. SYS_ADMIN in there reaches the sandbox's kernel, not
the node's. A task with build adds the capability, and the server's operator allows it in
backend.capabilities.
This guide was tested on GKE Autopilot with gVisor (BuildKit v0.25.1): the build, a RUN
step that downloads packages, and the push.
What you need
- A hammerkit server on Kubernetes, set up as in the Kubernetes guide, and the rights of its operator on the cluster.
- A registry to push to. Here, a Docker repository in Artifact Registry, and the right to grant roles on it.
- The variables of the Kubernetes guide in your shell:
NAME(the server's name and namespace),PROJECT,POOLandAUDIENCE.
1. A sandboxed runtime on the cluster
The cluster needs a RuntimeClass that starts pods in a sandbox. Look for one first:
kubectl get runtimeclass| Cluster | The RuntimeClass |
|---|---|
| GKE Autopilot | gvisor, there on every cluster (GKE Sandbox). Nothing to do. |
| GKE Standard | gvisor, once a node pool has it: gcloud container node-pools create sandbox --cluster <cluster> --sandbox type=gvisor. |
| AKS | kata-vm-isolation, with Pod Sandboxing on a node pool (--workload-runtime KataVmIsolation). |
| EKS and your own nodes | Install gVisor's runsc and its containerd shim on the nodes (gVisor's containerd guide), then add a RuntimeClass with handler: runsc. |
A RuntimeClass of your own looks like this:
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata: { name: gvisor }
handler: runscThe rest of this guide calls it gvisor. Use the name your cluster has.
2. Let the server add SYS_ADMIN there
Three changes, in this order, so no pod can add a capability outside the sandbox at any point.
The admission policies. A namespace of a caller enforces the Pod Security level baseline,
which refuses SYS_ADMIN even in a sandbox: it does not know about RuntimeClasses. With
capabilities, the server makes the namespaces of callers enforce privileged, and an admission
policy of yours refuses what baseline refuses, and any capability beyond it outside the
RuntimeClass. A second one keeps the namespaces the server makes to its own. Copy both from
Capabilities into admission.yaml, with your
RuntimeClass in place of gvisor, $NAME in place of hammerkit in the server's
ServiceAccount (system:serviceaccount:$NAME:hammerkit) and ^$NAME- in place of ^hammerkit-,
and apply it. They need Kubernetes 1.30 or later:
kubectl apply -f admission.yamlThe namespaces the server made. The server never changes the level of a namespace it made before (it has no right to), so set it on those of its prefix:
kubectl get namespace -l hammerkit.dev/managed=true -o name | grep "^namespace/$NAME-" | \
xargs -r -I{} kubectl label {} pod-security.kubernetes.io/enforce=privileged \
pod-security.kubernetes.io/audit=baseline --overwriteThe config. In server.yaml (the ConfigMap hammerkit-config), under backend:
backend:
type: kubernetes
capabilities:
allow: [SYS_ADMIN]
runtimeClass: gvisorRestart the server so it reads the config:
kubectl -n $NAME rollout restart deploy/hammerkitOnly the pods of tasks that add a capability run in gVisor; every other task runs as before.
Runs on sandboxes add none, unless you list them under
sandboxes.capabilities. On GKE Standard and AKS only the sandbox's node pool has the runtime:
the pods of these tasks go to the nodes of the first pool,
so its labels have to match that node pool, or a pod stays Pending.
3. An account that may push
The task pushes with a token of a service account, never with a key or a person's login. The server mints the token: a Kubernetes ServiceAccount of its namespace, which Google trusts through the identity pool of the Kubernetes guide, and which has the right to push to one repository and nothing else.
REGION=europe-west4 REPOSITORY=apps
gcloud artifacts repositories create $REPOSITORY --repository-format docker \
--location $REGION --project $PROJECT # skip this if the repository exists
# the identity that pushes: a ServiceAccount in the server's namespace, writer on the repository
kubectl -n $NAME create serviceaccount registry-push
gcloud artifacts repositories add-iam-policy-binding $REPOSITORY --location $REGION \
--project $PROJECT --role roles/artifactregistry.writer \
--member principal://iam.googleapis.com/$POOL/subject/system:serviceaccount:$NAME:registry-push
# how gcloud signs in as it: the token the server puts in the Job, exchanged at Google's STS
kubectl -n $NAME create secret generic registry-push-federation --from-literal=gcp.json="$(cat <<EOF
{ "type": "external_account",
"audience": "//iam.googleapis.com/$POOL/providers/cluster",
"subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
"token_url": "https://sts.googleapis.com/v1/token",
"credential_source": { "file": "/var/run/hammerkit/account/token" } }
EOF
)"
# the server may read that Secret and ask for tokens of that ServiceAccount, nothing else
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hammerkit-accounts, namespace: $NAME }
rules:
- { apiGroups: [''], resources: [secrets], verbs: [get], resourceNames: [registry-push-federation] }
- { apiGroups: [''], resources: [serviceaccounts/token], verbs: [create], resourceNames: [registry-push] }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: hammerkit-accounts, namespace: $NAME }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: hammerkit-accounts }
subjects: [{ kind: ServiceAccount, name: hammerkit, namespace: $NAME }]
EOFA new IAM binding can take a few minutes before it works. Then declare the account in
server.yaml, next to what is there, with your values in place of the examples:
secretProviders:
gcp-token:
command: [gcloud, auth, print-access-token] # prints a token of an hour
accounts:
registry-push:
description: Pushes images to Artifact Registry
providers:
gcp-token: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /var/run/hammerkit/account/gcp.json } }
kubernetes:
serviceAccount: registry-push
audience: hk # AUDIENCE of the Kubernetes guide
secret: registry-push-federation
source: [ref] # only from code in the repository
refs: [main] # and only from main
backend:
type: kubernetes
providers:
gcp-token:
image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
allow: [sts.googleapis.com] # where the command signs in
entitlements:
- subject: { iss: https://accounts.google.com, sub: '112233445566778899' } # PUBLISHER_ID
accounts: { registry-push: use } # add this to the publisher's entry
repos: [github.com/corp/app]
builds: publishsource and refs keep the account to reviewed code, see
Restricting the code: code nobody
reviewed could push any image under that repository. Accounts
and Secrets in the Job explain each key.
The network. A task that reads a secret reaches no more than
runs.secretNetwork allows, and the
Kubernetes guide sets it to none. The build pulls its base image, its RUN steps download
packages and the push goes to the registry, so name those hosts:
runs:
secretNetwork:
allow:
- europe-west4-docker.pkg.dev # the registry you push to
- registry-1.docker.io # Docker Hub, for FROM alpine
- auth.docker.io
- production.cloudflare.docker.com # Docker Hub's layers, from its CDNs or S3
- production.cloudfront.docker.com
- docker-images-prod.s3.dualstack.us-east-1.amazonaws.com
- dl-cdn.alpinelinux.org # what RUN apk add downloadsRestart the server again (kubectl -n $NAME rollout restart deploy/hammerkit).
4. The build file
Next to the Dockerfile, in the repository the server runs from:
FROM alpine:3.22
RUN apk add --no-cache curl
CMD ["curl", "--version"]tasks:
image:
description: build app/Dockerfile and push it
build:
context: app
tags: europe-west4-docker.pkg.dev/my-project/apps/hello:latest
push: true
login:
- registry: europe-west4-docker.pkg.dev
user: oauth2accesstoken
password: { from: 'gcp-token:registry', account: registry-push }
resources: { memory: 4Gi }
network:
allow:
- europe-west4-docker.pkg.dev
- registry-1.docker.io
- auth.docker.io
- production.cloudflare.docker.com
- production.cloudfront.docker.com
- docker-images-prod.s3.dualstack.us-east-1.amazonaws.com
- dl-cdn.alpinelinux.orgField by field:
buildmakes the task one that hammerkit runs with BuildKit, addingSYS_ADMIN: that is what makes the server run it in gVisor. See Image builds for every field.loginsigns in to Artifact Registry, which takes an access token as the password of the useroauth2accesstoken. The token comes from the server's providergcp-token, read asregistry-push: it lasts an hour and reaches only what that account may do.resources.memory: on a server BuildKit keeps its layers in memory, which counts against the task's.networkis an allow list, so hammerkit starts an egress proxy for the task. BuildKit pulls and pushes through it, and hammerkit hands it on to theRUNsteps of the Dockerfile.
Pushing to another registry changes only login and the hosts: Amazon ECR takes the user AWS
and the password of aws ecr get-login-password, Azure Container Registry the user
00000000-0000-0000-0000-000000000000 and the token of az acr login --expose-token, each
printed by a secret provider as the account of that cloud.
5. Publish and run
Commit both files to main, then publish the build file and run the task, as the publisher of
the Kubernetes guide:
export HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email --impersonate-service-account=$PUBLISHER"
hammerkit remote publish $NAME
hammerkit remote run image --server $NAME --repo github.com/$REPO --ref mainThe log shows BuildKit's steps, the RUN apk add and at the end the push:
[image] #9 pushing layers ... done
[image] #9 pushing manifest for europe-west4-docker.pkg.dev/my-project/apps/hello:latest ... doneA run whose app folder didn't change skips the task: the image of those sources is in the
registry already. Set cache: none on the task to push on every run. To deploy the image a
task built, by its digest, see Deploying the image by its digest.
On a docker host
A server with backend: { type: host } doesn't build images yet: it refuses a task with
build at publish. There a task that adds capabilities runs in gVisor (runsc), whose files
the server can't compare with the run's own before the task starts, so such a task may bind
none of the run's files, and a build reads its context from them. Build images on a server on
Kubernetes, or in CI; the build file is the same.
When it fails
| What you see | What it means |
|---|---|
image: builds an image, which a server builds only with a kubernetes backend for now at publish | The server runs tasks on a docker host, see On a docker host. |
image: adds the capability SYS_ADMIN, and this server allows none (backend.capabilities.allow) at publish | The server has no backend.capabilities, or wasn't restarted. |
... which this server does not allow (backend.capabilities.allow: ...) | The task adds a capability the operator didn't list. Ask whether the build needs it: BuildKit needs SYS_ADMIN only. |
image: adds the capabilities SYS_ADMIN, but the namespace ... was made before backend.capabilities | The caller's namespace still enforces baseline: run the kubectl label of the error, or the one of step 2 for all of them. |
The task's pod stays Pending | No node of the first pool has the RuntimeClass's runtime, see step 2. |
The pod is refused by hammerkit-capabilities | The admission policy names another RuntimeClass than backend.capabilities.runtimeClass. |
The task is killed (OOMKilled) during the build | Its layers fill its memory: raise resources.memory. |
401 or 403 when pushing | The IAM binding is new (give it a few minutes), names another repository, or another ServiceAccount than registry-push. |
RUN steps can't download (temporary error from apk) | A host isn't in the task's network and runs.secretNetwork. |