hammerkit
Agents on a server

Server on Kubernetes (GKE)

Run a hammerkit server for coding agents on a GKE cluster with the released image: four steps from an empty cluster to an agent pushing a branch.

Experimental, like the server. This guide was run as written on a GKE Autopilot cluster (Kubernetes 1.36) in europe-west4.

You end up with a hammerkit server in its own namespace of your cluster. Agents work on sandboxes there: every command they run is a Job in a namespace of its own, and their work leaves as a branch pushed by your GitHub App. No key is stored in the cluster, and no image has to be built: the server runs the released no0dles/hammerkit image.

What you need

  • A GKE cluster that enforces NetworkPolicy. Autopilot does; on a Standard cluster turn on Dataplane V2.
  • gcloud, kubectl and jq, signed in with the right to create ClusterRoles.
  • The hammerkit CLI of the server's version, on Node.js 24 or later: npm install -g hammerkit@$VERSION with VERSION from below. On an older Node.js npm warns EBADENGINE, and with a range instead of an exact version it quietly installs an old hammerkit without the commands this guide uses.
  • A GitHub App installed on your repository, with Contents: read and write, Metadata: read and nothing else. Download a private key for it (app.pem).
  • A build file with a shell task in the repository, see The build file.

Set these once; every command below uses them:

PROJECT=my-project
REGION=europe-west4
REPO=acme/shop          # your GitHub repository
GITHUB_APP_ID=123456    # on the App's settings page
VERSION=2.0.0           # one hammerkit version for the server, the CLI and the agent
NAME=hammerkit          # namespace and account names; another server on the cluster needs another
PROJECT_NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)')

1. Google: the App key and the accounts

The server reads the App's key from Secret Manager as its Kubernetes ServiceAccount. For that, an identity pool trusts the cluster's tokens, and only the server's ServiceAccount may read the key. Two Google service accounts give the agent and the publisher an identity to sign in with.

gcloud services enable sts.googleapis.com secretmanager.googleapis.com iam.googleapis.com \
  iamcredentials.googleapis.com --project $PROJECT

# let the cluster's ServiceAccounts sign in to Google
gcloud iam workload-identity-pools create $NAME --location global --project $PROJECT
gcloud iam workload-identity-pools providers create-oidc cluster --workload-identity-pool $NAME \
  --location global --project $PROJECT --allowed-audiences $NAME \
  --issuer-uri $(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer) \
  --attribute-mapping google.subject=assertion.sub
POOL=projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$NAME

# the App's key, readable by the server only
gcloud secrets create $NAME-github-app-key --project $PROJECT --data-file app.pem
gcloud secrets add-iam-policy-binding $NAME-github-app-key --project $PROJECT \
  --role roles/secretmanager.secretAccessor \
  --member principal://iam.googleapis.com/$POOL/subject/system:serviceaccount:$NAME:hammerkit

# identities for the agent and the publisher, which you may sign in as
ME=user:$(gcloud config get account)    # as a service account: ME=serviceAccount:<its email>; empty: set it yourself
for who in agent publisher; do
  gcloud iam service-accounts create $NAME-$who --project $PROJECT
  gcloud iam service-accounts add-iam-policy-binding $NAME-$who@$PROJECT.iam.gserviceaccount.com \
    --project $PROJECT --role roles/iam.serviceAccountTokenCreator --member $ME
done

# what the next steps use
AUDIENCE=$NAME
KEY_SECRET=projects/$PROJECT/secrets/$NAME-github-app-key/versions/latest
AGENT=$NAME-agent@$PROJECT.iam.gserviceaccount.com
PUBLISHER=$NAME-publisher@$PROJECT.iam.gserviceaccount.com
AGENT_ID=$(gcloud iam service-accounts describe $AGENT --project $PROJECT --format='value(uniqueId)')
PUBLISHER_ID=$(gcloud iam service-accounts describe $PUBLISHER --project $PROJECT --format='value(uniqueId)')

Delete app.pem now. The two service accounts hold no role in the project: only the server reads their tokens. A new IAM binding can take a few minutes before it works. AUDIENCE is what the pool accepts from the server's pod; it is unrelated to the audience: hammerkit of the sign-in tokens in step 2.

Did someone else do this step? Ask them for the values the last block sets, and set them in your shell: POOL (the pool's full name, as above), AUDIENCE (the audience its provider allows), KEY_SECRET (the Secret Manager version of the App's key), AGENT and PUBLISHER (the emails of the two service accounts) and AGENT_ID and PUBLISHER_ID (their unique ids). The pool's provider has to be named cluster, or change providers/cluster in gcp.json of step 2. They also have to let the server's ServiceAccount, system:serviceaccount:$NAME:hammerkit, read the key, and let you act as the two accounts. Service accounts on Google Cloud explains these two accounts, and how to let your team act as the agent.

2. Deploy the server

One manifest holds everything the server needs in the cluster: its namespace and ServiceAccount, its rights, its config, a disk and the Deployment. It uses the variables of step 1; the first line stops when one of them is not set, for example in a new terminal.

: "${NAME:?}" "${REPO:?}" "${GITHUB_APP_ID:?}" "${VERSION:?}" "${POOL:?}" "${AUDIENCE:?}" \
  "${KEY_SECRET:?}" "${AGENT_ID:?}" "${PUBLISHER_ID:?}"
kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata: { name: $NAME }
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: hammerkit, namespace: $NAME }
---
# may create the namespaces of callers and bind $NAME-runs in them, and delete those of
# service runs
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: $NAME-server }
rules:
  - { apiGroups: [''], resources: [namespaces], verbs: [get, list, create, delete] }
  - { apiGroups: [rbac.authorization.k8s.io], resources: [rolebindings], verbs: [get, create, update] }
  - { apiGroups: [rbac.authorization.k8s.io], resources: [clusterroles], verbs: [bind], resourceNames: [$NAME-runs] }
  # what the nodes have and use, for the capacity the dashboard shows
  - { apiGroups: [''], resources: [nodes], verbs: [list] }
  - { apiGroups: [metrics.k8s.io], resources: [nodes], verbs: [list] }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: $NAME-server }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: $NAME-server }
subjects: [{ kind: ServiceAccount, name: hammerkit, namespace: $NAME }]
---
# what the server may do in a caller's namespace, bound there by the server itself
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: $NAME-runs }
rules:
  - { apiGroups: [''], resources: [resourcequotas, limitranges], verbs: [get, create, update, delete] }
  - apiGroups: ['']
    resources: [pods, pods/exec, pods/log, persistentvolumeclaims, secrets, services]
    verbs: [get, list, create, patch, delete, deletecollection]
  - { apiGroups: [''], resources: [events], verbs: [list] }
  - { apiGroups: [metrics.k8s.io], resources: [pods], verbs: [list] }
  - { apiGroups: [batch], resources: [jobs, jobs/status], verbs: [get, list, create, patch, delete, deletecollection] }
  - { apiGroups: [apps], resources: [deployments], verbs: [get, list, create, patch, delete, deletecollection] }
  - { apiGroups: [networking.k8s.io], resources: [networkpolicies], verbs: [get, list, create, update, patch, delete] }
---
# the server binds $NAME-runs only in namespaces it made, not in another server's
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: $NAME-managed-namespaces }
spec:
  matchConstraints:
    resourceRules:
      - { apiGroups: [rbac.authorization.k8s.io], apiVersions: ['*'], operations: [CREATE, UPDATE], resources: [rolebindings] }
  matchConditions:
    - { name: server, expression: "request.userInfo.username == 'system:serviceaccount:$NAME:hammerkit'" }
  validations:
    - expression: "namespaceObject.metadata.?labels['hammerkit.dev/managed'].orValue('') == 'true'"
    - expression: "namespaceObject.metadata.name.matches('^$NAME-[0-9a-f]{16}$')"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata: { name: $NAME-managed-namespaces }
spec: { policyName: $NAME-managed-namespaces, validationActions: [Deny] }
---
# the server deletes only the namespaces of its service runs
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: $NAME-service-run-namespaces }
spec:
  matchConstraints:
    resourceRules:
      - { apiGroups: [''], apiVersions: ['*'], operations: [DELETE], resources: [namespaces] }
  matchConditions:
    - { name: server, expression: "request.userInfo.username == 'system:serviceaccount:$NAME:hammerkit'" }
  validations:
    - expression: "oldObject.metadata.?labels['hammerkit.dev/service-run'].hasValue()"
    - expression: "oldObject.metadata.name.matches('^$NAME-[0-9a-f]{16}$')"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata: { name: $NAME-service-run-namespaces }
spec: { policyName: $NAME-service-run-namespaces, validationActions: [Deny] }
---
apiVersion: v1
kind: ConfigMap
metadata: { name: hammerkit-config, namespace: $NAME }
data:
  # how the server signs in to Google: its ServiceAccount token, exchanged at Google's STS
  gcp.json: |
    { "type": "external_account",
      "audience": "//iam.googleapis.com/$POOL/providers/cluster",
      "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
      "token_url": "https://sts.googleapis.com/v1/token",
      "credential_source": { "file": "/var/run/hammerkit-google/token" } }
  server.yaml: |
    listen: { host: 0.0.0.0, port: 8480 }
    behindTlsProxy: true    # TLS ends at kubectl port-forward or your load balancer, see below
    storage: { directory: /data }
    issuers:
      - { issuer: https://accounts.google.com, audience: hammerkit }
    backend:
      type: kubernetes
      namespacePrefix: $NAME-
      runRole: $NAME-runs
      networkPolicy: enforced
      maxVolumeSize: 10Gi    # the default: the most a task's claim may request
      # the default (metadata and private ranges) and GKE's service range, which is outside of them
      deniedEgress:
        [169.254.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, fe80::/10, fc00::/7, 34.118.224.0/20]
      sandboxes: { image: 'no0dles/hammerkit:$VERSION' }
    secretProviders:
      github-app: { command: [hammerkit, github-app-token, '{{ref}}'] }
    accounts:
      github:
        providers:
          github-app:
            env:
              GITHUB_APP_ID: '$GITHUB_APP_ID'
              GITHUB_APP_KEY_SECRET: $KEY_SECRET
              GOOGLE_APPLICATION_CREDENTIALS: /etc/hammerkit/gcp.json
    sources:
      - match: github.com/$REPO
        secret: { from: 'github-app:$REPO', account: github }
    runs:
      env: { allow: [PATH] }    # the variables a command gets from the server
      args: declared            # only tasks with args: true take arguments
      secretNetwork: none       # what a task that reads a secret reaches
    sandboxes:
      network: { allow: [registry.npmjs.org] }    # what the agent's commands may reach
    entitlements:
      - subject: { iss: https://accounts.google.com, sub: '$PUBLISHER_ID' }   # $PUBLISHER
        repos: [github.com/$REPO]
        builds: publish
      - subject: { iss: https://accounts.google.com, sub: '$AGENT_ID' }       # $AGENT
        repos: [github.com/$REPO]
        builds: run
        sandboxes:
          - repo: github.com/$REPO
            branches: { main: pull, 'agent/**': push }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: hammerkit-data, namespace: $NAME }
spec:
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 30Gi } }
---
apiVersion: v1
kind: Service
metadata: { name: hammerkit, namespace: $NAME }
spec:
  selector: { app: hammerkit }
  ports: [{ port: 8480 }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: hammerkit, namespace: $NAME }
spec:
  replicas: 1
  strategy: { type: Recreate }
  selector: { matchLabels: { app: hammerkit } }
  template:
    metadata: { labels: { app: hammerkit } }
    spec:
      serviceAccountName: hammerkit
      containers:
        - name: server
          image: no0dles/hammerkit:$VERSION
          args: [server, --config, /etc/hammerkit/server.yaml]
          ports: [{ containerPort: 8480 }]
          readinessProbe: { httpGet: { path: /.well-known/hammerkit, port: 8480 } }
          resources: { requests: { cpu: '1', memory: 2Gi } }
          volumeMounts:
            - { name: config, mountPath: /etc/hammerkit }
            - { name: data, mountPath: /data }
            - { name: google, mountPath: /var/run/hammerkit-google }
      volumes:
        - { name: config, configMap: { name: hammerkit-config } }
        - { name: data, persistentVolumeClaim: { claimName: hammerkit-data } }
        - name: google
          projected:
            sources: [{ serviceAccountToken: { audience: $AUDIENCE, expirationSeconds: 3600, path: token } }]
EOF
kubectl -n $NAME rollout status deploy/hammerkit --timeout 10m

On Autopilot the first rollout waits a few minutes for a node, and kubectl apply warns that GKE changed the admission policy to fit Autopilot; both are expected.

Check that the server can read the App's key and reach your repository. This runs the command the server mints its GitHub tokens with, as the server's ServiceAccount, and throws the token away:

kubectl -n $NAME exec deploy/hammerkit -- env GITHUB_APP_ID=$GITHUB_APP_ID \
  GITHUB_APP_KEY_SECRET=$KEY_SECRET GOOGLE_APPLICATION_CREDENTIALS=/etc/hammerkit/gcp.json \
  hammerkit github-app-token $REPO > /dev/null && echo "the server can read $REPO"

When it fails, it says which step did:

  • Secret Manager did not give …: 403 means the server's ServiceAccount may not read the key: the binding of step 1 is missing, names another namespace, or is not in effect yet.
  • Google STS did not exchange the token … means the pool or its provider does not match: compare POOL and AUDIENCE with step 1.
  • the GitHub App … is not installed on … means the App is not installed on the repository.

An agent whose sandbox can not be created gets the same reason from the server.

3. Connect and publish

Forward the server's port to your machine and register it:

while true; do kubectl -n $NAME port-forward svc/hammerkit 8480:8480; sleep 1; done &
hammerkit remote add $NAME http://127.0.0.1:8480 --issuer https://accounts.google.com

kubectl port-forward ends when the server's pod is replaced, for example after a kubectl apply, and now and then when its connection drops; the loop starts it again. When it prints the same error every second, for example because port 8480 is taken or the cluster can not be reached, stop it with kill %1 and fix that first.

Before you publish, check that the build file's shell task has mounts: [.], args: true and cmds: [sh -c], as in The build file. Without the mount, the agent's commands run in an empty directory; without args: true this server refuses them. Then publish the build file from a checkout of the repository, signed in as the publisher:

HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \
  --impersonate-service-account=$PUBLISHER" \
  hammerkit remote publish $NAME

It prints the hash of the published version, sha256: and 64 hex digits. The agent names the build file by that hash, so a later publish never changes what a running agent uses.

4. Start an agent

Follow pi or opencode, with the server registered as above, the build file's hash and this token command:

export HAMMERKIT_BUILD=sha256:9f2c…    # the hash hammerkit remote publish printed
export HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \
  --impersonate-service-account=$AGENT"

The tutorials write the accounts as hammerkit-agent@my-project… and hammerkit-publisher@my-project…: use $AGENT and $PUBLISHER from step 1 instead.

The agent's branches show up in the repository as agent/…, pushed by your GitHub App.

Reaching the server over HTTPS

kubectl port-forward needs access to the cluster on every machine. A team, or a CI job such as GitHub Actions, reaches the server through an HTTPS load balancer instead: a GKE Ingress with a Google-managed certificate for a domain of yours. The config already expects TLS to end in front of the server (behindTlsProxy), and the server takes nothing without a token of one of its issuers, so the address can be public.

Reserve an address and point your domain at it, with an A record at your DNS provider:

DOMAIN=hammerkit.example.com    # a name in a zone you control
gcloud compute addresses create $NAME --global --project $PROJECT
gcloud compute addresses describe $NAME --global --project $PROJECT --format='value(address)'

Then the certificate and the load balancer:

: "${NAME:?}" "${DOMAIN:?}"
kubectl -n $NAME annotate --overwrite svc hammerkit \
  cloud.google.com/neg='{"ingress": true}' cloud.google.com/backend-config='{"default": "hammerkit"}'
kubectl apply -f - <<EOF
# a run's log and events are followed over long requests: the load balancer's default is 30s
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata: { name: hammerkit, namespace: $NAME }
spec:
  timeoutSec: 3600
  healthCheck: { type: HTTP, requestPath: /.well-known/hammerkit, port: 8480 }
---
# plain http only answers with a redirect: a token is never sent over it
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata: { name: hammerkit, namespace: $NAME }
spec: { redirectToHttps: { enabled: true } }
---
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata: { name: hammerkit, namespace: $NAME }
spec: { domains: [$DOMAIN] }
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hammerkit
  namespace: $NAME
  annotations:
    kubernetes.io/ingress.global-static-ip-name: $NAME
    networking.gke.io/managed-certificates: hammerkit
    networking.gke.io/v1beta1.FrontendConfig: hammerkit
spec:
  defaultBackend: { service: { name: hammerkit, port: { number: 8480 } } }
EOF

The load balancer takes about five minutes, the certificate up to an hour once the domain points at the address. kubectl -n $NAME get managedcertificate hammerkit says Active when it is done; until then curl https://$DOMAIN/.well-known/hammerkit fails with a TLS error. Then register the server under its domain, on every machine that uses it:

curl -sSf https://$DOMAIN/.well-known/hammerkit && echo
hammerkit remote add $NAME https://$DOMAIN --issuer https://accounts.google.com --force

--force replaces the http://127.0.0.1:8480 of step 3; the port-forward loop can stop then (kill %1).

What the pieces are for

  • The pool and gcp.json. The server's pod gets a token of its ServiceAccount for the pool's audience. Google's STS exchanges it for a short-lived access token, which reads the App's key. Nothing else is granted to it. See A GitHub App for the token command.
  • The two ClusterRoles and the admission policies. The server creates a namespace per caller and binds $NAME-runs there, and nowhere else. It deletes only the namespaces of service runs, which it makes per run. See What the server may do.
  • deniedEgress. A task with an open network can not reach the metadata server or your cluster's other workloads: the private ranges are denied by default, GKE's service range has to be added. gcloud container clusters describe <cluster> --region $REGION --format='value(clusterIpv4Cidr,servicesIpv4Cidr)' shows your ranges. Without the right to describe the cluster, kubectl get svc -A -o jsonpath='{..clusterIP}' shows service addresses that have to be in the denied ranges. An internal address tasks need goes in allowedEgress.
  • networkPolicy: enforced. The agent's commands only reach what sandboxes.network allows, here npm. See Network.
  • runs.args: declared. Arguments reach only the tasks that say args: true, so an agent can't append its own flags to the last command of any other task, for example to an install that has to keep --ignore-scripts. See Arguments.
  • runs.secretNetwork: none. A task that reads a secret reaches only the services it needs, so a tool it runs can't send the value anywhere. Give such a task network: none, or set an allow list here and in the task. See Tasks that read secrets.
  • One replica. The disk holds the server's records of sandboxes and its cache.

Secrets for builds

The agent's shell task reads no secrets. Other tasks of the build file can, for example a test against a staging API: the Job of such a task reads the secret itself, as a Kubernetes ServiceAccount of its account. See Secrets in the Job for the setup with Secret Manager. With secretNetwork: none such a task has a limited network, so its provider needs the hosts it reaches, which its egress proxy lets only the reading of the secret through:

server.yaml
backend:
  type: kubernetes
  providers:
    gsm:
      image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
      allow: [sts.googleapis.com, secretmanager.googleapis.com]

Cleaning up

Stop the port-forward loop first (kill %1, or close its terminal), then:

# the namespaces of this server's callers and service runs: namespacePrefix and 16 hex digits; look before you delete
kubectl get ns -l hammerkit.dev/managed -o name | grep -E "^namespace/$NAME-[0-9a-f]{16}$"
kubectl get ns -l hammerkit.dev/managed -o name | grep -E "^namespace/$NAME-[0-9a-f]{16}$" | xargs -r kubectl delete
kubectl delete ns $NAME
kubectl delete clusterrole $NAME-server $NAME-runs
kubectl delete clusterrolebinding $NAME-server
kubectl delete validatingadmissionpolicybinding,validatingadmissionpolicy $NAME-managed-namespaces $NAME-service-run-namespaces
gcloud iam workload-identity-pools delete $NAME --location global --project $PROJECT --quiet
gcloud secrets delete $NAME-github-app-key --project $PROJECT --quiet
gcloud compute addresses delete $NAME --global --project $PROJECT --quiet   # if you reserved one for HTTPS
for who in agent publisher; do
  gcloud iam service-accounts delete $NAME-$who@$PROJECT.iam.gserviceaccount.com --project $PROJECT --quiet
done

Every server labels the namespaces it makes hammerkit.dev/managed, so on a cluster with several servers delete by the whole name, as above, not by the label alone. A deleted pool keeps its name for 30 days.

On this page