Server on Kubernetes (GKE)
Run a hammerkit server for coding agents on a GKE cluster with the released image: four steps from an empty cluster to an agent pushing a branch.
Experimental, like the server. This guide was run as written on a GKE Autopilot cluster
(Kubernetes 1.36) in europe-west4.
You end up with a hammerkit server in its own namespace of your cluster. Agents work on
sandboxes there: every command they run is a Job in a
namespace of its own, and their work leaves as a branch pushed by your GitHub App. No key is
stored in the cluster, and no image has to be built: the server runs the released
no0dles/hammerkit image.
What you need
- A GKE cluster that enforces
NetworkPolicy. Autopilot does; on a Standard cluster turn on Dataplane V2. gcloud,kubectlandjq, signed in with the right to create ClusterRoles.- The hammerkit CLI of the server's version, on Node.js 24 or later:
npm install -g hammerkit@$VERSIONwithVERSIONfrom below. On an older Node.js npm warnsEBADENGINE, and with a range instead of an exact version it quietly installs an old hammerkit without the commands this guide uses. - A GitHub App installed on your repository, with Contents: read and write, Metadata:
read and nothing else. Download a private key for it (
app.pem). - A build file with a shell task in the repository, see The build file.
Set these once; every command below uses them:
PROJECT=my-project
REGION=europe-west4
REPO=acme/shop # your GitHub repository
GITHUB_APP_ID=123456 # on the App's settings page
VERSION=2.0.0 # one hammerkit version for the server, the CLI and the agent
NAME=hammerkit # namespace and account names; another server on the cluster needs another
PROJECT_NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)')1. Google: the App key and the accounts
The server reads the App's key from Secret Manager as its Kubernetes ServiceAccount. For that, an identity pool trusts the cluster's tokens, and only the server's ServiceAccount may read the key. Two Google service accounts give the agent and the publisher an identity to sign in with.
gcloud services enable sts.googleapis.com secretmanager.googleapis.com iam.googleapis.com \
iamcredentials.googleapis.com --project $PROJECT
# let the cluster's ServiceAccounts sign in to Google
gcloud iam workload-identity-pools create $NAME --location global --project $PROJECT
gcloud iam workload-identity-pools providers create-oidc cluster --workload-identity-pool $NAME \
--location global --project $PROJECT --allowed-audiences $NAME \
--issuer-uri $(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer) \
--attribute-mapping google.subject=assertion.sub
POOL=projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$NAME
# the App's key, readable by the server only
gcloud secrets create $NAME-github-app-key --project $PROJECT --data-file app.pem
gcloud secrets add-iam-policy-binding $NAME-github-app-key --project $PROJECT \
--role roles/secretmanager.secretAccessor \
--member principal://iam.googleapis.com/$POOL/subject/system:serviceaccount:$NAME:hammerkit
# identities for the agent and the publisher, which you may sign in as
ME=user:$(gcloud config get account) # as a service account: ME=serviceAccount:<its email>; empty: set it yourself
for who in agent publisher; do
gcloud iam service-accounts create $NAME-$who --project $PROJECT
gcloud iam service-accounts add-iam-policy-binding $NAME-$who@$PROJECT.iam.gserviceaccount.com \
--project $PROJECT --role roles/iam.serviceAccountTokenCreator --member $ME
done
# what the next steps use
AUDIENCE=$NAME
KEY_SECRET=projects/$PROJECT/secrets/$NAME-github-app-key/versions/latest
AGENT=$NAME-agent@$PROJECT.iam.gserviceaccount.com
PUBLISHER=$NAME-publisher@$PROJECT.iam.gserviceaccount.com
AGENT_ID=$(gcloud iam service-accounts describe $AGENT --project $PROJECT --format='value(uniqueId)')
PUBLISHER_ID=$(gcloud iam service-accounts describe $PUBLISHER --project $PROJECT --format='value(uniqueId)')Delete app.pem now. The two service accounts hold no role in the project: only the server
reads their tokens. A new IAM binding can take a few minutes before it works. AUDIENCE is
what the pool accepts from the server's pod; it is unrelated to the audience: hammerkit of
the sign-in tokens in step 2.
Did someone else do this step? Ask them for the values the last block sets, and set them
in your shell: POOL (the pool's full name, as above), AUDIENCE (the audience its provider
allows), KEY_SECRET (the Secret Manager version of the App's key), AGENT and PUBLISHER
(the emails of the two service accounts) and AGENT_ID and PUBLISHER_ID (their unique ids).
The pool's provider has to be named cluster, or change providers/cluster in gcp.json
of step 2. They also have to let the server's ServiceAccount,
system:serviceaccount:$NAME:hammerkit, read the key, and let you act as the two accounts.
Service accounts on Google Cloud explains these two
accounts, and how to let your team act as the agent.
2. Deploy the server
One manifest holds everything the server needs in the cluster: its namespace and ServiceAccount, its rights, its config, a disk and the Deployment. It uses the variables of step 1; the first line stops when one of them is not set, for example in a new terminal.
: "${NAME:?}" "${REPO:?}" "${GITHUB_APP_ID:?}" "${VERSION:?}" "${POOL:?}" "${AUDIENCE:?}" \
"${KEY_SECRET:?}" "${AGENT_ID:?}" "${PUBLISHER_ID:?}"
kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata: { name: $NAME }
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: hammerkit, namespace: $NAME }
---
# may create the namespaces of callers and bind $NAME-runs in them, and delete those of
# service runs
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: $NAME-server }
rules:
- { apiGroups: [''], resources: [namespaces], verbs: [get, list, create, delete] }
- { apiGroups: [rbac.authorization.k8s.io], resources: [rolebindings], verbs: [get, create, update] }
- { apiGroups: [rbac.authorization.k8s.io], resources: [clusterroles], verbs: [bind], resourceNames: [$NAME-runs] }
# what the nodes have and use, for the capacity the dashboard shows
- { apiGroups: [''], resources: [nodes], verbs: [list] }
- { apiGroups: [metrics.k8s.io], resources: [nodes], verbs: [list] }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: $NAME-server }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: $NAME-server }
subjects: [{ kind: ServiceAccount, name: hammerkit, namespace: $NAME }]
---
# what the server may do in a caller's namespace, bound there by the server itself
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: $NAME-runs }
rules:
- { apiGroups: [''], resources: [resourcequotas, limitranges], verbs: [get, create, update, delete] }
- apiGroups: ['']
resources: [pods, pods/exec, pods/log, persistentvolumeclaims, secrets, services]
verbs: [get, list, create, patch, delete, deletecollection]
- { apiGroups: [''], resources: [events], verbs: [list] }
- { apiGroups: [metrics.k8s.io], resources: [pods], verbs: [list] }
- { apiGroups: [batch], resources: [jobs, jobs/status], verbs: [get, list, create, patch, delete, deletecollection] }
- { apiGroups: [apps], resources: [deployments], verbs: [get, list, create, patch, delete, deletecollection] }
- { apiGroups: [networking.k8s.io], resources: [networkpolicies], verbs: [get, list, create, update, patch, delete] }
---
# the server binds $NAME-runs only in namespaces it made, not in another server's
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: $NAME-managed-namespaces }
spec:
matchConstraints:
resourceRules:
- { apiGroups: [rbac.authorization.k8s.io], apiVersions: ['*'], operations: [CREATE, UPDATE], resources: [rolebindings] }
matchConditions:
- { name: server, expression: "request.userInfo.username == 'system:serviceaccount:$NAME:hammerkit'" }
validations:
- expression: "namespaceObject.metadata.?labels['hammerkit.dev/managed'].orValue('') == 'true'"
- expression: "namespaceObject.metadata.name.matches('^$NAME-[0-9a-f]{16}$')"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata: { name: $NAME-managed-namespaces }
spec: { policyName: $NAME-managed-namespaces, validationActions: [Deny] }
---
# the server deletes only the namespaces of its service runs
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: $NAME-service-run-namespaces }
spec:
matchConstraints:
resourceRules:
- { apiGroups: [''], apiVersions: ['*'], operations: [DELETE], resources: [namespaces] }
matchConditions:
- { name: server, expression: "request.userInfo.username == 'system:serviceaccount:$NAME:hammerkit'" }
validations:
- expression: "oldObject.metadata.?labels['hammerkit.dev/service-run'].hasValue()"
- expression: "oldObject.metadata.name.matches('^$NAME-[0-9a-f]{16}$')"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata: { name: $NAME-service-run-namespaces }
spec: { policyName: $NAME-service-run-namespaces, validationActions: [Deny] }
---
apiVersion: v1
kind: ConfigMap
metadata: { name: hammerkit-config, namespace: $NAME }
data:
# how the server signs in to Google: its ServiceAccount token, exchanged at Google's STS
gcp.json: |
{ "type": "external_account",
"audience": "//iam.googleapis.com/$POOL/providers/cluster",
"subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
"token_url": "https://sts.googleapis.com/v1/token",
"credential_source": { "file": "/var/run/hammerkit-google/token" } }
server.yaml: |
listen: { host: 0.0.0.0, port: 8480 }
behindTlsProxy: true # TLS ends at kubectl port-forward or your load balancer, see below
storage: { directory: /data }
issuers:
- { issuer: https://accounts.google.com, audience: hammerkit }
backend:
type: kubernetes
namespacePrefix: $NAME-
runRole: $NAME-runs
networkPolicy: enforced
maxVolumeSize: 10Gi # the default: the most a task's claim may request
# the default (metadata and private ranges) and GKE's service range, which is outside of them
deniedEgress:
[169.254.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, fe80::/10, fc00::/7, 34.118.224.0/20]
sandboxes: { image: 'no0dles/hammerkit:$VERSION' }
secretProviders:
github-app: { command: [hammerkit, github-app-token, '{{ref}}'] }
accounts:
github:
providers:
github-app:
env:
GITHUB_APP_ID: '$GITHUB_APP_ID'
GITHUB_APP_KEY_SECRET: $KEY_SECRET
GOOGLE_APPLICATION_CREDENTIALS: /etc/hammerkit/gcp.json
sources:
- match: github.com/$REPO
secret: { from: 'github-app:$REPO', account: github }
runs:
env: { allow: [PATH] } # the variables a command gets from the server
args: declared # only tasks with args: true take arguments
secretNetwork: none # what a task that reads a secret reaches
sandboxes:
network: { allow: [registry.npmjs.org] } # what the agent's commands may reach
entitlements:
- subject: { iss: https://accounts.google.com, sub: '$PUBLISHER_ID' } # $PUBLISHER
repos: [github.com/$REPO]
builds: publish
- subject: { iss: https://accounts.google.com, sub: '$AGENT_ID' } # $AGENT
repos: [github.com/$REPO]
builds: run
sandboxes:
- repo: github.com/$REPO
branches: { main: pull, 'agent/**': push }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: hammerkit-data, namespace: $NAME }
spec:
accessModes: [ReadWriteOnce]
resources: { requests: { storage: 30Gi } }
---
apiVersion: v1
kind: Service
metadata: { name: hammerkit, namespace: $NAME }
spec:
selector: { app: hammerkit }
ports: [{ port: 8480 }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: hammerkit, namespace: $NAME }
spec:
replicas: 1
strategy: { type: Recreate }
selector: { matchLabels: { app: hammerkit } }
template:
metadata: { labels: { app: hammerkit } }
spec:
serviceAccountName: hammerkit
containers:
- name: server
image: no0dles/hammerkit:$VERSION
args: [server, --config, /etc/hammerkit/server.yaml]
ports: [{ containerPort: 8480 }]
readinessProbe: { httpGet: { path: /.well-known/hammerkit, port: 8480 } }
resources: { requests: { cpu: '1', memory: 2Gi } }
volumeMounts:
- { name: config, mountPath: /etc/hammerkit }
- { name: data, mountPath: /data }
- { name: google, mountPath: /var/run/hammerkit-google }
volumes:
- { name: config, configMap: { name: hammerkit-config } }
- { name: data, persistentVolumeClaim: { claimName: hammerkit-data } }
- name: google
projected:
sources: [{ serviceAccountToken: { audience: $AUDIENCE, expirationSeconds: 3600, path: token } }]
EOF
kubectl -n $NAME rollout status deploy/hammerkit --timeout 10mOn Autopilot the first rollout waits a few minutes for a node, and kubectl apply warns that
GKE changed the admission policy to fit Autopilot; both are expected.
Check that the server can read the App's key and reach your repository. This runs the command the server mints its GitHub tokens with, as the server's ServiceAccount, and throws the token away:
kubectl -n $NAME exec deploy/hammerkit -- env GITHUB_APP_ID=$GITHUB_APP_ID \
GITHUB_APP_KEY_SECRET=$KEY_SECRET GOOGLE_APPLICATION_CREDENTIALS=/etc/hammerkit/gcp.json \
hammerkit github-app-token $REPO > /dev/null && echo "the server can read $REPO"When it fails, it says which step did:
Secret Manager did not give …: 403means the server's ServiceAccount may not read the key: the binding of step 1 is missing, names another namespace, or is not in effect yet.Google STS did not exchange the token …means the pool or its provider does not match: comparePOOLandAUDIENCEwith step 1.the GitHub App … is not installed on …means the App is not installed on the repository.
An agent whose sandbox can not be created gets the same reason from the server.
3. Connect and publish
Forward the server's port to your machine and register it:
while true; do kubectl -n $NAME port-forward svc/hammerkit 8480:8480; sleep 1; done &
hammerkit remote add $NAME http://127.0.0.1:8480 --issuer https://accounts.google.comkubectl port-forward ends when the server's pod is replaced, for example after a
kubectl apply, and now and then when its connection drops; the loop starts it again. When
it prints the same error every second, for example because port 8480 is taken or the cluster
can not be reached, stop it with kill %1 and fix that first.
Before you publish, check that the build file's shell task has mounts: [.], args: true
and cmds: [sh -c], as in The build file. Without the
mount, the agent's commands run in an empty directory; without args: true this server refuses
them. Then publish the build file from a
checkout of the repository, signed in as the publisher:
HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \
--impersonate-service-account=$PUBLISHER" \
hammerkit remote publish $NAMEIt prints the hash of the published version, sha256: and 64 hex digits. The agent names the
build file by that hash, so a later publish never changes what a running agent uses.
4. Start an agent
Follow pi or opencode, with the server registered as above, the build file's hash and this token command:
export HAMMERKIT_BUILD=sha256:9f2c… # the hash hammerkit remote publish printed
export HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \
--impersonate-service-account=$AGENT"The tutorials write the accounts as hammerkit-agent@my-project… and
hammerkit-publisher@my-project…: use $AGENT and $PUBLISHER from step 1 instead.
The agent's branches show up in the repository as agent/…, pushed by your GitHub App.
Reaching the server over HTTPS
kubectl port-forward needs access to the cluster on every machine. A team, or a CI job such
as GitHub Actions, reaches the server through an HTTPS
load balancer instead: a GKE Ingress with a Google-managed certificate for a domain of yours.
The config already expects TLS to end in front of the server (behindTlsProxy), and the
server takes nothing without a token of one of its issuers, so the address can be public.
Reserve an address and point your domain at it, with an A record at your DNS provider:
DOMAIN=hammerkit.example.com # a name in a zone you control
gcloud compute addresses create $NAME --global --project $PROJECT
gcloud compute addresses describe $NAME --global --project $PROJECT --format='value(address)'Then the certificate and the load balancer:
: "${NAME:?}" "${DOMAIN:?}"
kubectl -n $NAME annotate --overwrite svc hammerkit \
cloud.google.com/neg='{"ingress": true}' cloud.google.com/backend-config='{"default": "hammerkit"}'
kubectl apply -f - <<EOF
# a run's log and events are followed over long requests: the load balancer's default is 30s
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata: { name: hammerkit, namespace: $NAME }
spec:
timeoutSec: 3600
healthCheck: { type: HTTP, requestPath: /.well-known/hammerkit, port: 8480 }
---
# plain http only answers with a redirect: a token is never sent over it
apiVersion: networking.gke.io/v1beta1
kind: FrontendConfig
metadata: { name: hammerkit, namespace: $NAME }
spec: { redirectToHttps: { enabled: true } }
---
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata: { name: hammerkit, namespace: $NAME }
spec: { domains: [$DOMAIN] }
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hammerkit
namespace: $NAME
annotations:
kubernetes.io/ingress.global-static-ip-name: $NAME
networking.gke.io/managed-certificates: hammerkit
networking.gke.io/v1beta1.FrontendConfig: hammerkit
spec:
defaultBackend: { service: { name: hammerkit, port: { number: 8480 } } }
EOFThe load balancer takes about five minutes, the certificate up to an hour once the domain
points at the address. kubectl -n $NAME get managedcertificate hammerkit says Active when it
is done; until then curl https://$DOMAIN/.well-known/hammerkit fails with a TLS error.
Then register the server under its domain, on every machine that uses it:
curl -sSf https://$DOMAIN/.well-known/hammerkit && echo
hammerkit remote add $NAME https://$DOMAIN --issuer https://accounts.google.com --force--force replaces the http://127.0.0.1:8480 of step 3; the port-forward loop can stop then
(kill %1).
What the pieces are for
- The pool and
gcp.json. The server's pod gets a token of its ServiceAccount for the pool's audience. Google's STS exchanges it for a short-lived access token, which reads the App's key. Nothing else is granted to it. See A GitHub App for the token command. - The two ClusterRoles and the admission policies. The server creates a namespace per
caller and binds
$NAME-runsthere, and nowhere else. It deletes only the namespaces of service runs, which it makes per run. See What the server may do. deniedEgress. A task with an open network can not reach the metadata server or your cluster's other workloads: the private ranges are denied by default, GKE's service range has to be added.gcloud container clusters describe <cluster> --region $REGION --format='value(clusterIpv4Cidr,servicesIpv4Cidr)'shows your ranges. Without the right to describe the cluster,kubectl get svc -A -o jsonpath='{..clusterIP}'shows service addresses that have to be in the denied ranges. An internal address tasks need goes inallowedEgress.networkPolicy: enforced. The agent's commands only reach whatsandboxes.networkallows, here npm. See Network.runs.args: declared. Arguments reach only the tasks that sayargs: true, so an agent can't append its own flags to the last command of any other task, for example to an install that has to keep--ignore-scripts. See Arguments.runs.secretNetwork: none. A task that reads a secret reaches only the services it needs, so a tool it runs can't send the value anywhere. Give such a tasknetwork: none, or set an allow list here and in the task. See Tasks that read secrets.- One replica. The disk holds the server's records of sandboxes and its cache.
Secrets for builds
The agent's shell task reads no secrets. Other tasks of the build file can, for example a
test against a staging API: the Job of such a task reads the secret itself, as a Kubernetes
ServiceAccount of its account. See
Secrets in the Job for the setup with
Secret Manager. With secretNetwork: none such a task has a limited network, so its provider
needs the hosts it reaches, which its egress proxy lets only the reading of the secret through:
backend:
type: kubernetes
providers:
gsm:
image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
allow: [sts.googleapis.com, secretmanager.googleapis.com]Cleaning up
Stop the port-forward loop first (kill %1, or close its terminal), then:
# the namespaces of this server's callers and service runs: namespacePrefix and 16 hex digits; look before you delete
kubectl get ns -l hammerkit.dev/managed -o name | grep -E "^namespace/$NAME-[0-9a-f]{16}$"
kubectl get ns -l hammerkit.dev/managed -o name | grep -E "^namespace/$NAME-[0-9a-f]{16}$" | xargs -r kubectl delete
kubectl delete ns $NAME
kubectl delete clusterrole $NAME-server $NAME-runs
kubectl delete clusterrolebinding $NAME-server
kubectl delete validatingadmissionpolicybinding,validatingadmissionpolicy $NAME-managed-namespaces $NAME-service-run-namespaces
gcloud iam workload-identity-pools delete $NAME --location global --project $PROJECT --quiet
gcloud secrets delete $NAME-github-app-key --project $PROJECT --quiet
gcloud compute addresses delete $NAME --global --project $PROJECT --quiet # if you reserved one for HTTPS
for who in agent publisher; do
gcloud iam service-accounts delete $NAME-$who@$PROJECT.iam.gserviceaccount.com --project $PROJECT --quiet
doneEvery server labels the namespaces it makes hammerkit.dev/managed, so on a cluster with
several servers delete by the whole name, as above, not by the label alone. A deleted pool
keeps its name for 30 days.
Agents on a server
Set up a hammerkit server for coding agents and connect pi or opencode to it, so the agent's files and commands stay on the server.
Service accounts on Google Cloud
Create the Google service accounts an agent and a publisher sign in to a hammerkit server with, and let yourself and your team act as them without a key file.