hammerkit

Building container images

Build a Dockerfile in a task on a server with BuildKit, in a sandboxed runtime, and push the image with a short-lived token of a service account.

This guide builds a Dockerfile in a task that runs on a hammerkit server and pushes the image to a registry. It sets up the server once, as its operator, and then writes the build file: a task with build, which is the same on a dev machine. The example pushes to Google's Artifact Registry from a server on Kubernetes; the other registries follow the same steps, the differences are named where they come up. A server on a docker host doesn't build images yet.

Why a server needs a sandbox for it

On a dev machine a task could mount the docker socket and run docker build. On a server that would hand the task the daemon of the machine, and with it every container of every caller on it, so the server refuses a mount of the socket. Docker in Docker needs a privileged container, which a server never starts. BuildKit without root needs what the server takes from every container: its seccomp profile, allowPrivilegeEscalation: false and the masked /proc.

What works is BuildKit as root with one more capability, SYS_ADMIN, so it can mount the layers it builds. SYS_ADMIN is root's power over the kernel, and the kernel of a node runs the tasks of other callers too. So the server adds it only in a sandboxed runtime, which gives the container a kernel of its own: gVisor handles the container's system calls in a kernel it implements itself, Kata Containers start a small virtual machine per pod. SYS_ADMIN in there reaches the sandbox's kernel, not the node's. A task with build adds the capability, and the server's operator allows it in backend.capabilities.

This guide was tested on GKE Autopilot with gVisor (BuildKit v0.25.1): the build, a RUN step that downloads packages, and the push.

What you need

  • A hammerkit server on Kubernetes, set up as in the Kubernetes guide, and the rights of its operator on the cluster.
  • A registry to push to. Here, a Docker repository in Artifact Registry, and the right to grant roles on it.
  • The variables of the Kubernetes guide in your shell: NAME (the server's name and namespace), PROJECT, POOL and AUDIENCE.

1. A sandboxed runtime on the cluster

The cluster needs a RuntimeClass that starts pods in a sandbox. Look for one first:

kubectl get runtimeclass
ClusterThe RuntimeClass
GKE Autopilotgvisor, there on every cluster (GKE Sandbox). Nothing to do.
GKE Standardgvisor, once a node pool has it: gcloud container node-pools create sandbox --cluster <cluster> --sandbox type=gvisor.
AKSkata-vm-isolation, with Pod Sandboxing on a node pool (--workload-runtime KataVmIsolation).
EKS and your own nodesInstall gVisor's runsc and its containerd shim on the nodes (gVisor's containerd guide), then add a RuntimeClass with handler: runsc.

A RuntimeClass of your own looks like this:

runtimeclass.yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata: { name: gvisor }
handler: runsc

The rest of this guide calls it gvisor. Use the name your cluster has.

2. Let the server add SYS_ADMIN there

Three changes, in this order, so no pod can add a capability outside the sandbox at any point.

The admission policies. A namespace of a caller enforces the Pod Security level baseline, which refuses SYS_ADMIN even in a sandbox: it does not know about RuntimeClasses. With capabilities, the server makes the namespaces of callers enforce privileged, and an admission policy of yours refuses what baseline refuses, and any capability beyond it outside the RuntimeClass. A second one keeps the namespaces the server makes to its own. Copy both from Capabilities into admission.yaml, with your RuntimeClass in place of gvisor, $NAME in place of hammerkit in the server's ServiceAccount (system:serviceaccount:$NAME:hammerkit) and ^$NAME- in place of ^hammerkit-, and apply it. They need Kubernetes 1.30 or later:

kubectl apply -f admission.yaml

The namespaces the server made. The server never changes the level of a namespace it made before (it has no right to), so set it on those of its prefix:

kubectl get namespace -l hammerkit.dev/managed=true -o name | grep "^namespace/$NAME-" | \
  xargs -r -I{} kubectl label {} pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/audit=baseline --overwrite

The config. In server.yaml (the ConfigMap hammerkit-config), under backend:

server.yaml
backend:
  type: kubernetes
  capabilities:
    allow: [SYS_ADMIN]
    runtimeClass: gvisor

Restart the server so it reads the config:

kubectl -n $NAME rollout restart deploy/hammerkit

Only the pods of tasks that add a capability run in gVisor; every other task runs as before. Runs on sandboxes add none, unless you list them under sandboxes.capabilities. On GKE Standard and AKS only the sandbox's node pool has the runtime: the pods of these tasks go to the nodes of the first pool, so its labels have to match that node pool, or a pod stays Pending.

3. An account that may push

The task pushes with a token of a service account, never with a key or a person's login. The server mints the token: a Kubernetes ServiceAccount of its namespace, which Google trusts through the identity pool of the Kubernetes guide, and which has the right to push to one repository and nothing else.

REGION=europe-west4 REPOSITORY=apps
gcloud artifacts repositories create $REPOSITORY --repository-format docker \
  --location $REGION --project $PROJECT    # skip this if the repository exists

# the identity that pushes: a ServiceAccount in the server's namespace, writer on the repository
kubectl -n $NAME create serviceaccount registry-push
gcloud artifacts repositories add-iam-policy-binding $REPOSITORY --location $REGION \
  --project $PROJECT --role roles/artifactregistry.writer \
  --member principal://iam.googleapis.com/$POOL/subject/system:serviceaccount:$NAME:registry-push

# how gcloud signs in as it: the token the server puts in the Job, exchanged at Google's STS
kubectl -n $NAME create secret generic registry-push-federation --from-literal=gcp.json="$(cat <<EOF
{ "type": "external_account",
  "audience": "//iam.googleapis.com/$POOL/providers/cluster",
  "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
  "token_url": "https://sts.googleapis.com/v1/token",
  "credential_source": { "file": "/var/run/hammerkit/account/token" } }
EOF
)"

# the server may read that Secret and ask for tokens of that ServiceAccount, nothing else
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hammerkit-accounts, namespace: $NAME }
rules:
  - { apiGroups: [''], resources: [secrets], verbs: [get], resourceNames: [registry-push-federation] }
  - { apiGroups: [''], resources: [serviceaccounts/token], verbs: [create], resourceNames: [registry-push] }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: hammerkit-accounts, namespace: $NAME }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: hammerkit-accounts }
subjects: [{ kind: ServiceAccount, name: hammerkit, namespace: $NAME }]
EOF

A new IAM binding can take a few minutes before it works. Then declare the account in server.yaml, next to what is there, with your values in place of the examples:

server.yaml
secretProviders:
  gcp-token:
    command: [gcloud, auth, print-access-token]   # prints a token of an hour
accounts:
  registry-push:
    description: Pushes images to Artifact Registry
    providers:
      gcp-token: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /var/run/hammerkit/account/gcp.json } }
    kubernetes:
      serviceAccount: registry-push
      audience: hk                                # AUDIENCE of the Kubernetes guide
      secret: registry-push-federation
    source: [ref]                                 # only from code in the repository
    refs: [main]                                  # and only from main
backend:
  type: kubernetes
  providers:
    gcp-token:
      image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
      allow: [sts.googleapis.com]                 # where the command signs in
entitlements:
  - subject: { iss: https://accounts.google.com, sub: '112233445566778899' }   # PUBLISHER_ID
    accounts: { registry-push: use }              # add this to the publisher's entry
    repos: [github.com/corp/app]
    builds: publish

source and refs keep the account to reviewed code, see Restricting the code: code nobody reviewed could push any image under that repository. Accounts and Secrets in the Job explain each key.

The network. A task that reads a secret reaches no more than runs.secretNetwork allows, and the Kubernetes guide sets it to none. The build pulls its base image, its RUN steps download packages and the push goes to the registry, so name those hosts:

server.yaml
runs:
  secretNetwork:
    allow:
      - europe-west4-docker.pkg.dev        # the registry you push to
      - registry-1.docker.io               # Docker Hub, for FROM alpine
      - auth.docker.io
      - production.cloudflare.docker.com   # Docker Hub's layers, from its CDNs or S3
      - production.cloudfront.docker.com
      - docker-images-prod.s3.dualstack.us-east-1.amazonaws.com
      - dl-cdn.alpinelinux.org             # what RUN apk add downloads

Restart the server again (kubectl -n $NAME rollout restart deploy/hammerkit).

4. The build file

Next to the Dockerfile, in the repository the server runs from:

app/Dockerfile
FROM alpine:3.22
RUN apk add --no-cache curl
CMD ["curl", "--version"]
.hammerkit.yaml
tasks:
  image:
    description: build app/Dockerfile and push it
    build:
      context: app
      tags: europe-west4-docker.pkg.dev/my-project/apps/hello:latest
      push: true
      login:
        - registry: europe-west4-docker.pkg.dev
          user: oauth2accesstoken
          password: { from: 'gcp-token:registry', account: registry-push }
    resources: { memory: 4Gi }
    network:
      allow:
        - europe-west4-docker.pkg.dev
        - registry-1.docker.io
        - auth.docker.io
        - production.cloudflare.docker.com
        - production.cloudfront.docker.com
        - docker-images-prod.s3.dualstack.us-east-1.amazonaws.com
        - dl-cdn.alpinelinux.org

Field by field:

  • build makes the task one that hammerkit runs with BuildKit, adding SYS_ADMIN: that is what makes the server run it in gVisor. See Image builds for every field.
  • login signs in to Artifact Registry, which takes an access token as the password of the user oauth2accesstoken. The token comes from the server's provider gcp-token, read as registry-push: it lasts an hour and reaches only what that account may do.
  • resources.memory: on a server BuildKit keeps its layers in memory, which counts against the task's.
  • network is an allow list, so hammerkit starts an egress proxy for the task. BuildKit pulls and pushes through it, and hammerkit hands it on to the RUN steps of the Dockerfile.

Pushing to another registry changes only login and the hosts: Amazon ECR takes the user AWS and the password of aws ecr get-login-password, Azure Container Registry the user 00000000-0000-0000-0000-000000000000 and the token of az acr login --expose-token, each printed by a secret provider as the account of that cloud.

5. Publish and run

Commit both files to main, then publish the build file and run the task, as the publisher of the Kubernetes guide:

export HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email --impersonate-service-account=$PUBLISHER"
hammerkit remote publish $NAME
hammerkit remote run image --server $NAME --repo github.com/$REPO --ref main

The log shows BuildKit's steps, the RUN apk add and at the end the push:

[image] #9 pushing layers ... done
[image] #9 pushing manifest for europe-west4-docker.pkg.dev/my-project/apps/hello:latest ... done

A run whose app folder didn't change skips the task: the image of those sources is in the registry already. Set cache: none on the task to push on every run. To deploy the image a task built, by its digest, see Deploying the image by its digest.

On a docker host

A server with backend: { type: host } doesn't build images yet: it refuses a task with build at publish. There a task that adds capabilities runs in gVisor (runsc), whose files the server can't compare with the run's own before the task starts, so such a task may bind none of the run's files, and a build reads its context from them. Build images on a server on Kubernetes, or in CI; the build file is the same.

When it fails

What you seeWhat it means
image: builds an image, which a server builds only with a kubernetes backend for now at publishThe server runs tasks on a docker host, see On a docker host.
image: adds the capability SYS_ADMIN, and this server allows none (backend.capabilities.allow) at publishThe server has no backend.capabilities, or wasn't restarted.
... which this server does not allow (backend.capabilities.allow: ...)The task adds a capability the operator didn't list. Ask whether the build needs it: BuildKit needs SYS_ADMIN only.
image: adds the capabilities SYS_ADMIN, but the namespace ... was made before backend.capabilitiesThe caller's namespace still enforces baseline: run the kubectl label of the error, or the one of step 2 for all of them.
The task's pod stays PendingNo node of the first pool has the RuntimeClass's runtime, see step 2.
The pod is refused by hammerkit-capabilitiesThe admission policy names another RuntimeClass than backend.capabilities.runtimeClass.
The task is killed (OOMKilled) during the buildIts layers fill its memory: raise resources.memory.
401 or 403 when pushingThe IAM binding is new (give it a few minutes), names another repository, or another ServiceAccount than registry-push.
RUN steps can't download (temporary error from apk)A host isn't in the task's network and runs.secretNetwork.

On this page