# Caches (/docs/build-file/caches)
A cache combines two things: a **method** that decides if a task changed and a
**backend** that stores the task result so it can be reused later.
By default hammerkit caches task results on the local filesystem under
`~/.hammerkit/remote-cache`, so a result built in one checkout can be reused in
another on the same machine. Since `1.6.0` you can also declare named caches with
a remote **backend** (for example an S3 bucket or a container registry) so a task built on one machine can
be restored on another - ideal for sharing results between developers and CI runs.
## Which caching mechanism? [#which-caching-mechanism]
Hammerkit has three related features that all "save work". Pick by where the result
needs to go:
| Mechanism | What it does | Reach for it when |
| ---------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| **Cache method** (`checksum`/`modify-date`) | Decides whether a task can be **skipped** locally. | Always on — it's how a task knows it's up to date. |
| **Cache backend** (`local`/`s3`/`registry`, this page) | **Shares** the skip-or-restore result across machines automatically. | Developers and CI should reuse each other's results without scripting. |
| **[`export: true`](/docs/task#exporting-generated-files)** | Copies a container task's output **back into your workspace**. | Another tool outside hammerkit needs the produced files. |
| **[`store` / `restore`](/docs/cli/store-restore)** | Manually moves outputs + cache state to a directory. | Wiring hammerkit into a CI provider's own cache step (no remote backend). |
A remote **backend** and **store/restore** solve the same "share across CI runs"
problem two ways: with a backend, pull/push is automatic and you don't script the
cache step; with store/restore, you hand the directory to your CI's cache. If a
remote backend is configured, you usually don't need store/restore at all. See the
[CI caching guide](/docs/guides/ci-caching).
## Declaring caches [#declaring-caches]
Caches are declared in a top-level `caches:` block. Each entry has a `method` and
a `backend`.
```yaml title=".hammerkit.yaml"
caches:
remote:
method: checksum
backend:
type: s3
bucket: my-build-cache
region: eu-central-1
tasks:
build:
cache:
name: remote
src:
- src
generates:
- dist
cmds:
- tsc -b
```
See [task caching](/docs/task/caching) for how a task references a cache.
## Method [#method]
The method decides whether the source files of a task changed since the last run.
* `checksum` (default) - compares the content checksum of the source files.
* `modify-date` - compares the last modification date of the source files. Faster
on large folders, but less precise.
* `none` - never skips the task; the commands run every time.
## Backends [#backends]
The backend decides where task results are stored. Three backend types are built
in. Hammerkit pulls from the backend when the result is missing locally and pushes
to it after a task ran successfully.
### local [#local]
Stores cache entries on the local filesystem. Useful to share a cache between
projects on the same machine. This is the backend used by the built-in `default`
cache.
```yaml title=".hammerkit.yaml"
caches:
shared:
method: checksum
backend:
type: local
path: /shared/hammerkit-cache
```
| Field | Required | Description |
| ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `path` | no | Directory for the cache. Defaults to `~/.hammerkit/remote-cache`, or `$HAMMERKIT_DIR/remote-cache` when `HAMMERKIT_DIR` is set. |
### s3 [#s3]
Stores cache entries in an S3-compatible bucket. Works with AWS S3 and any
S3-compatible service such as MinIO, Cloudflare R2, Google Cloud Storage or Wasabi.
```yaml title=".hammerkit.yaml"
caches:
remote:
method: checksum
backend:
type: s3
bucket: my-build-cache
region: eu-central-1
endpoint: https://minio.example.com # for S3-compatible services
prefix: hammerkit
```
| Field | Required | Description |
| ---------------- | -------- | ------------------------------------------------------------------------------- |
| `bucket` | yes | Name of the bucket cache entries are written to. |
| `region` | no | Region of the bucket. |
| `endpoint` | no | Custom endpoint for S3-compatible services (MinIO, R2, ...). |
| `prefix` | no | Key prefix for all cache entries, useful to share a bucket with other data. |
| `forcePathStyle` | no | Use path-style bucket addressing. Defaults to `true` when an `endpoint` is set. |
Credentials are read from the standard AWS SDK credential chain, for example the
`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables or the
shared `~/.aws/config` file. They are never stored in the build file.
#### Cloudflare R2 [#cloudflare-r2]
R2 exposes an S3 API, so point `endpoint` at your account endpoint and use an `auto`
region:
```yaml
caches:
remote:
method: checksum
backend:
type: s3
bucket: my-build-cache
region: auto
endpoint: https://.r2.cloudflarestorage.com
```
#### Google Cloud Storage (GCS) [#google-cloud-storage-gcs]
GCS offers an S3-compatible XML API, so it works through the `s3` backend. Point
`endpoint` at `storage.googleapis.com`, use path-style addressing, and provide
[HMAC interoperability keys](https://cloud.google.com/storage/docs/authentication/hmackeys)
through the AWS credential chain (`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`):
```yaml
caches:
remote:
method: checksum
backend:
type: s3
bucket: my-build-cache
region: europe-west1
endpoint: https://storage.googleapis.com
forcePathStyle: true
```
Azure Blob Storage is **not** S3-compatible, so it is not supported through the `s3`
backend. The backend registry is pluggable, so an Azure backend could be added as a
custom backend in the future.
### registry [#registry]
Stores cache entries in any OCI container registry — GHCR, Docker Hub, ECR, GAR,
Artifactory, Harbor or a plain `registry:2`. No bucket to provision: use the
registry you already push images to. Each entry is stored as a small single-layer
image tagged `-`.
```yaml title=".hammerkit.yaml"
caches:
remote:
method: checksum
backend:
type: registry
repository: ghcr.io/my-org/hammerkit-cache
```
| Field | Required | Description |
| ------------ | -------- | ------------------------------------------------------------------------- |
| `repository` | yes | Repository to store entries in, without a tag (e.g. `ghcr.io/org/cache`). |
| `insecure` | no | Use plain http. Defaults to `true` only for `localhost` / `127.0.0.1`. |
Credentials come from the docker config that `docker login` (or a CI login step
such as `docker/login-action`) writes, including credential helpers
(`credHelpers` / `credsStore`). `DOCKER_CONFIG` is honored. They are never
stored in the build file. A read-only token is enough for runners that only pull.
They are only sent to the registry itself: an upload location or a next page on another
host, port or scheme gets none, as a redirect to one does.
Removing entries (`hammerkit clean --cache`) requires the registry to allow
manifest deletion; retention is otherwise best left to the registry's own
cleanup policies.
## Retention [#retention]
Caches grow with every new version of every task. A `retention` block bounds a
cache:
```yaml title=".hammerkit.yaml"
caches:
default:
method: checksum
backend:
type: local
retention:
maxAge: 30d # remove entries not used for 30 days
maxSize: 20Gi # then remove least recently used entries above 20Gi
keepPerTask: 3 # keep at most the 3 newest versions of each task
```
| Field | Description |
| ------------- | ---------------------------------------------------------------------------------------------- |
| `maxAge` | Remove entries not used within this long (`30s`, `5m`, `12h`, `30d`, …). |
| `maxSize` | Remove least recently used entries until the cache is at most this big (`500Mi`, `5Gi`, `2G`). |
| `keepPerTask` | Keep only the newest N versions (state keys) of each task. |
Policies apply in that order: `keepPerTask`, then `maxAge`, then `maxSize`.
Removing an entry never produces a wrong result — the task just rebuilds next time.
A **local** cache with `retention` is pruned automatically after every successful
run. Remote caches are only pruned when you ask for it:
```bash
hammerkit cache ls --remote shared # what's in it
hammerkit cache prune --remote shared --dry-run # what the policy would remove
hammerkit cache prune --remote shared --max-age 14d # prune, overriding the policy
```
See [cache ls / prune](/docs/cli/cache#cache-ls-and-cache-prune). The local backend
records when each entry was last used, so `maxAge` and `maxSize` evict what hasn't
been used recently. S3 and registries only know when an entry was created, so
there they evict by age of creation. Registries must allow deleting manifests
(`registry:2`, Harbor, ECR, GAR); for GHCR and Docker Hub use the registry's own
retention settings instead — for GHCR, for example, the
[`actions/delete-package-versions`](https://github.com/actions/delete-package-versions)
action. S3 lifecycle rules work alongside `cache prune` too.
## Built-in caches [#built-in-caches]
Two caches are always available without declaring them:
* `default` - uses the `local` backend with the `checksum` method. This is what a
task uses when it does not specify a cache, so results are cached under
`~/.hammerkit/remote-cache` by default (`$HAMMERKIT_DIR/remote-cache` when
`HAMMERKIT_DIR` is set).
* `none` - uses the `none` method, disabling skipping entirely.
### Changing the default for all tasks [#changing-the-default-for-all-tasks]
Declaring a `caches.default` entry overrides the built-in `default` cache, so the
new backend applies to **every task that does not reference a cache explicitly**.
This is the way to point all tasks at a shared local path or a remote bucket
without touching each task.
```yaml title=".hammerkit.yaml"
caches:
default:
method: checksum
backend:
type: s3
bucket: my-build-cache
region: eu-central-1
```
The `--cache` flag still overrides the *method* for these implicit tasks at
runtime (it defaults to `checksum`), so redeclaring `caches.default` is primarily
how you change the default *backend*. See
[task caching](/docs/task/caching#changing-the-default-for-all-tasks).
## Pull / push behavior [#pull--push-behavior]
When a cache has a remote backend (`local` or `s3`):
* **Pull** - before a task runs, if its result is not present locally, hammerkit
pulls it from the backend and skips the task if the sources are unchanged.
* **Push** - after a task ran successfully, hammerkit pushes the result to the
backend so other machines can reuse it.
Backend errors never fail the build. If the backend is unreachable, the pull or
push is skipped with a warning and the task runs or completes as usual.
# Environment Variables (/docs/build-file/environment-variables)
## Shell environment [#shell-environment]
Variables from the shell environment are passed to a command when the task declares them:
```yaml title=".hammerkit.yaml"
tasks:
example:
envs:
VERSION: $VERSION
cmds:
- echo $VERSION
```
Every environment variable a task uses should be declared in its `envs`. Hammerkit checks that each one is defined and otherwise fails with an error, to prevent undesired behavior.
## .env file [#env-file]
Environment variables can be defined in a `.env` file. This is recommended for secret values that should neither be committed nor made public.
```txt title=".env"
NPM_TOKEN=abc
```
```yaml title=".hammerkit.yaml"
tasks:
example:
envs:
NPM_TOKEN: $NPM_TOKEN
cmds:
- npm publish
```
## Defined values in the build file [#defined-values-in-the-build-file]
Environment variables can be defined in the build file itself. They can be defined for the entire build file or only for specific tasks.
```yaml title=".hammerkit.yaml"
envs:
NODE_VERSION: '22'
tasks:
example:
cmds:
- echo $NODE_VERSION
override_example:
envs:
NODE_VERSION: '20'
cmds:
- echo $NODE_VERSION
```
Variables defined in a build file are scoped to that file. Neither [referenced](/docs/build-file/references) nor [included](/docs/build-file/includes) tasks have access to them. Pass a value on to an include with [`with`](/docs/build-file/includes#inputs-of-a-local-include), as `NODE_VERSION: ${NODE_VERSION}`.
## Cache key [#cache-key]
Every variable a task receives, including each build-file `envs` value, is part of
its [cache key](/docs/task/caching): changing one reruns the task. A build-file
value only a few tasks need — an image tag, a test token — therefore reruns every
task in the file when it changes. Define such values on the tasks that use them.
[`validate`](/docs/cli/validate) warns about this: for each task with commands it
lists the build-file `envs` its definition never mentions as `$NAME` or `${NAME}`.
```
warn at lint build-file envs E2E_TOKEN are part of this task's cache key but not referenced by it: …
```
Hammerkit can't see what a tool reads at runtime, so a variable such as
`NODE_ENV` that every task's tooling picks up on its own belongs in the build-file
`envs` despite the warning.
## Precedence [#precedence]
A variable can be defined in more than one place. Hammerkit resolves it like this:
1. **Task `envs`** win over **build-file `envs`** on the same key. In the example
above `override_example` prints `20`, while `example` prints `22`.
2. A value of exactly `$NAME` is a *reference* that is filled in from, in order:
1. a **`--set NAME=value`** argument on the hammerkit command, then
2. the **shell / process environment**, then
3. a **`.env`** file in the build file's directory.
If a referenced `$NAME` is set nowhere, hammerkit aborts before running with
`missing environment variable NAME`. A variable set to an empty string counts as
set, as in a shell: `export NAME=` passes an empty value, which is how a CI job
provides a variable its provider leaves unset on some pipelines.
Every variable a task uses must be declared in its `envs` (directly or via
`$NAME`). Hammerkit only passes declared variables to the command, so an
undeclared shell variable is *not* leaked into the task.
## Interpolation [#interpolation]
There are two distinct places a variable can appear:
**Inside a command** (`cmds`), `$NAME` is expanded by the shell at runtime, using
the variables hammerkit passed in — exactly like a normal shell. This is the
`echo $NODE_VERSION` case above.
**Inside other task fields**, hammerkit substitutes `$NAME` itself when it plans
the task — before anything runs. This works for `image`, `src`, `generates`,
`mounts`, `ports`, `shell`, command working directories and a service's `image`,
so you can drive them from a single value:
```yaml title=".hammerkit.yaml"
envs:
NODE_VERSION: '24'
tasks:
build:
image: node:$NODE_VERSION-alpine # -> node:24-alpine
src:
- src
cmds:
- tsc -b
```
**Inside an `envs` value**, `${NAME}` is replaced anywhere in the string, and
`${NAME:-default}` falls back to `default` when `NAME` is set nowhere. A name comes
from the shell / process environment, then a `.env` file, then a literal value of
the same `envs` (build file and task merged), then the default:
```yaml title=".hammerkit.yaml"
envs:
DB_USER: app
tasks:
e2e:
envs:
DATABASE_URL: postgres://${DB_USER}@db:5432/app # -> postgres://app@db:5432/app
SPEC: ${E2E_SPEC:-e2e/**/*.cy.ts} # E2E_SPEC=e2e/login.cy.ts narrows it
MR_IID: ${CI_MERGE_REQUEST_IID:-} # empty outside merge request pipelines
cmds:
- cypress run --spec "$SPEC"
```
A `${NAME}` without a default that is set nowhere fails like a missing `$NAME`,
when a task that declares it runs.
The whole value `$NAME` (no braces) is a reference to the shell or `.env` only, as
before. Field substitution matches `$NAME` and `${NAME}` anywhere in the string and
is case-insensitive; a value defined in `envs` (like `NODE_VERSION: '24'`, or one
built with `${…}`) is what gets substituted into fields such as `image`. A name is
matched as the text it is (a `.` in it is a dot), and a value is put in as it is
written: a `$NAME` inside the value that was put in is not substituted again.
# Includes (/docs/build-file/includes)
Includes are ideal for defining repetitive tasks once and reusing them in multiple places.
Referenced files also split up tasks, but their working directory stays fixed to the directory they are in.
For includes, the working directory is the directory of the including file.
The following example is a TypeScript monorepo with two projects, `a` and `b`. Project `b` depends on `a`, and both need up-to-date dependencies before compiling the source code.
```yaml title="build.npm.yaml"
tasks:
install:
cmds:
- npm install
src:
- package.json
- package-lock.json
generates:
- node_modules
```
The npm install uses an include: each project has its own package.json, but the task definition is the same.
```yaml title="project/a/.hammerkit.yaml"
tasks:
build:
deps: [npm:install]
cmds:
- node_modules/.bin/tsc -b
includes:
npm: ../../build.npm.yaml
```
The build task [depends](/docs/task/dependencies) on the included install task, which ensures that node_modules is installed first.
```yaml title="project/b/.hammerkit.yaml"
tasks:
build:
deps: [npm:install, a:build]
cmds:
- node_modules/.bin/tsc -b
references:
a: ../a
includes:
npm: ../../build.npm.yaml
```
The build file of project `b` uses the same include for the npm install task.
An additional [reference](/docs/build-file/references) to `a` and a [task dependency](/docs/task/dependencies) on `a:build` ensure that project `a` is built before project `b` is compiled.
An include can also be an object with a `path`, which takes `publish` (and
[`with`](#inputs-of-a-local-include)):
```yaml
includes:
npm: { path: ../../build.npm.yaml }
dev: { path: ./dev.yaml, publish: false }
```
`publish: false` keeps what the included file declares on your machine: a
[server](/docs/remote-compute/build-files#only-part-of-a-build-file) leaves it out of the
version it publishes. A git include takes `publish` too.
## Include from a git repository [#include-from-a-git-repository]
Instead of a path, an include (or a [reference](/docs/build-file/references)) can name a
git repository. Use this to share build files between projects without copying them.
```yaml title=".hammerkit.yaml"
includes:
npm:
git: https://github.com/no0dles/hammerkit.git
ref: v1.16.0
path: catalog/npm
tasks:
build:
deps: [npm:install]
cmds:
- npm run build
```
| Field | Meaning |
| ------ | ------------------------------------------------------------------------------------------------- |
| `git` | Repository URL or path. Required. |
| `ref` | Branch, tag or commit SHA. The repository's `HEAD` when left out. |
| `path` | Build file, or a directory holding one, inside the repository. The repository root when left out. |
| `with` | Values for the inputs of the included file, see [inputs](#inputs). |
A plain string stays a local path, exactly as before. Prefer `includes` for shared task
definitions: they run in your directory, while a git `reference` runs inside the cached
checkout.
### Inputs [#inputs]
An included file declares its inputs as ordinary top-level `envs`, each with a default,
and uses them in its tasks. `with` replaces the defaults for one include:
```yaml title="build.npm.yaml (in the repository)"
envs:
NODE_VERSION: 22
tasks:
ci:
image: node:$NODE_VERSION-alpine
cmds:
- npm ci
src:
- package.json
- package-lock.json
generates:
- node_modules
```
```yaml title=".hammerkit.yaml"
includes:
npm:
git: https://github.com/no0dles/hammerkit.git
ref: v1.16.0
path: catalog/npm
with:
NODE_VERSION: 22
```
A name the included file does not declare is an error that lists the declared inputs, so a
typo cannot silently do nothing. Values are strings or numbers. A value is used as it is
written; to let CI set an input from outside, write `NODE_VERSION: ${NODE_VERSION:-22}`,
which is the environment's (or a `.env` file's) `NODE_VERSION`, else `22`. The same file
included under two names with different inputs gives two independent sets of tasks.
### Inputs of a local include [#inputs-of-a-local-include]
A local include (or reference) takes inputs too: write it as an object with a `path`.
```yaml title=".hammerkit.yaml"
envs:
NODE_VERSION: 24
includes:
npm:
path: build.npm.yaml
with:
NODE_VERSION: ${NODE_VERSION}
web:
path: web/build.yaml
with:
NODE_VERSION: ${NODE_VERSION}
```
`${NAME}` (or `${NAME:-default}`) in a `with` value reads as it does in an `envs` value of
the including file: the environment's (or a `.env` file's) `NAME` if it is set, else the
including file's own `envs` value. So the Node version lives in one place, every include
builds with it, and CI can still set it. A file that is included passes it on to its own
includes the same way. A name the including file does not declare stays as it is, and the
included file reads it from the environment as above. A value the including file builds
from other variables (`${BASE}-alpine`) is passed on as it is written. A plain string path stays a local include
without inputs, exactly as before.
### Limits [#limits]
Includes and references nest at most 32 levels deep, and one build file reads at most
1,000 files through them, 16 MiB of them, with at most 10,000 tasks and services; a file
included again with other inputs counts again. The
inputs one include passes on take at most 64 KiB. A file that includes itself with
inputs that change each time stops at these with an error that names the include.
### Caching and offline use [#caching-and-offline-use]
The first run fetches the repository at `ref` into `~/.hammerkit/includes` and every later
run uses that copy without touching the network, so a build keeps working offline and does
not change underneath you. The cache is the pin: there is no lockfile.
* A **commit SHA** (or a release tag you trust) gives the same content on every machine.
* A **branch** is frozen at the commit of the first fetch on *each machine*, so a fresh CI
runner can see newer content than a developer laptop. Pin a SHA when that matters.
* A repository that was never fetched and is unreachable fails with an error naming the
repository and ref. There is no silent fallback.
To update a branch or tag, run [`hammerkit includes pull`](/docs/cli/includes); a commit SHA
stays as it is. [`hammerkit clean --cache`](/docs/cli/clean) removes the cached repositories,
so the next run fetches them again.
### What a remote file may do [#what-a-remote-file-may-do]
A remote build file contributes tasks, services, caches and environments. Fetching it runs
nothing; commands run only when you run one of its tasks. Its own relative `includes` and
`references` resolve inside the same repository at the same commit, and a path that leaves
the repository is rejected. Private repositories use your usual git credentials (credential
helpers, SSH keys); hammerkit never prompts and has no secrets of its own for this.
## references vs. includes vs. extend — when to use which [#references-vs-includes-vs-extend--when-to-use-which]
All three reuse definitions, but they differ in *what* they reuse and *where it
runs*:
| | What it pulls in | Working directory | Use it when |
| --------------------------------------------- | ----------------------------------------------- | --------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| **[references](/docs/build-file/references)** | A whole other build file (its tasks + services) | The **referenced** file's own directory | The other file is its own project (a sub-package you also want to build in place). |
| **includes** | A whole other build file | The **including** file's directory | You want to reuse a task *definition* and run it here — the same `install`/`build` template across every package. |
| **[extend](/docs/task/extending)** | A single task, as a base template | The current task's file | One task is almost another, with a tweak or two. |
A common monorepo setup combines them: an `includes` brings in a shared
`build.npm.yaml` so each package runs `install` in its own directory, while
`references` wire up cross-package build order. See the
[monorepo guide](/docs/guides/monorepo).
# Build file (/docs/build-file)
The build file is a YAML file in the root of your project. It contains a list of [tasks](/docs/task) that can be executed from the [CLI](/docs/cli).
Hammerkit looks for the build file by name, in this order:
1. `.hammerkit.yaml`
2. `.hammerkit.yml`
3. `build.yaml`
All three are valid; the first one found is used. If more than one exists hammerkit warns about the ambiguity. These docs use `.hammerkit.yaml` as the canonical name in every example. To point hammerkit at a different file, pass [`--file `](/docs/cli).
```yaml title=".hammerkit.yaml"
envs:
NODE_VERSION: '22'
tasks:
build:
image: node:$NODE_VERSION
deps: [npm:install]
cmds:
- tsc -b
references:
subProject: ./packages/subproject
includes:
npm: ./build.npm.yaml
```
For more detail about the structure of the build file, check out the following sections.
# Build file reference (/docs/build-file/reference)
This is the complete reference for the hammerkit build file, derived from the
schema hammerkit validates against (`build.schema.json`). Unknown keys are
rejected, so this list is exhaustive. For prose and examples, follow the links to
the dedicated pages.
## Top-level keys [#top-level-keys]
| Key | Type | Description |
| -------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `envs` | map of string→string/number | Build-file-wide [environment variables](/docs/build-file/environment-variables). |
| `tasks` | map of name→task | The [tasks](/docs/task) in this file. |
| `services` | map of name→service | The [services](/docs/service) in this file. |
| `references` | map of prefix→path, `{path, with, publish}` or git source | [Reference](/docs/build-file/references) other build files (keep their own directory). |
| `includes` | map of prefix→path, `{path, with, publish}` or git source | [Include](/docs/build-file/includes) other build files (use this file's directory), with [inputs](/docs/build-file/includes#inputs-of-a-local-include). `publish: false` leaves the file out of a [published version](/docs/remote-compute/build-files#only-part-of-a-build-file). |
| `environments` | map of name→environment | [Run targets](/docs/task/kubernetes) (Kubernetes or remote Docker). |
| `caches` | map of name→cache | Named [cache](/docs/build-file/caches) declarations (method + backend). |
| `labels` | map of string→string/number | [Labels](/docs/labels) applied to every task/service in the file. |
| `runner` | string | Default [`runner`](/docs/task#runner) of the tasks in this file. |
## Task [#task]
A task is either a **container task** (it has an `image`, its own or the one of the
task it [extends](/docs/task/extending)), an **image build** (it has
[`build`](/docs/task/image-builds)) or a **local task** (neither). They share these fields
(an image build has no `cmds`, `shell`, `extend` or `args`):
| Field | Type | Description |
| ------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cmds` | array of string \| `{cmd, path}` | Commands to run, in order. `path` sets the working directory for that command. |
| `description` | string | Shown by [`ls`](/docs/cli/ls); a missing one is a [validate](/docs/cli/validate) warning. |
| `src` | array of string | Input files/folders/globs used for [caching](/docs/task/caching). |
| `generates` | array of string \| `{path, export, resetOnChange, name}` | [Output](/docs/task#generated-files-generates) paths. `export: true` copies back to the workspace when the task succeeds, `export: always` also when it fails; outputs start empty when the task runs, `resetOnChange: false` keeps their content (see [outputs start empty](/docs/task#outputs-start-empty)). |
| `deps` | array of string | [Tasks](/docs/task/dependencies) (`name` or `prefix:name`) that run first. |
| `needs` | array of string \| `{service, name}` | [Services](/docs/task/needs) that must be ready first. |
| `envs` | map of string→string/number | Task-level [environment variables](/docs/build-file/environment-variables) (override build-file `envs`). |
| `labels` | map of string→string/number | [Labels](/docs/labels) for filtering. |
| `cache` | string \| `{name, method}` | [Cache](/docs/task/caching) method shorthand or a named cache reference. |
| `extend` | string | [Base task](/docs/task/extending) to inherit from (`prefix:name`). |
| `shell` | string | Shell used to run `cmds` (default `/bin/sh`). |
| `timeout` | duration (`30s`, `10m`, `1h30m`) | [Maximum execution time](/docs/task#timeouts); the task fails when exceeded. |
| `secrets` | array of `{from, env \| path, cache}` | [Credentials](/docs/task/secrets) from `env:NAME` or `file:path`, injected as an env variable or a read-only file; kept out of the cache key and logs. |
| `resources` | `{cpus, memory}` \| `{requests, limits}` | [CPU and memory](/docs/task#resources) limits and requests (`2`, `500m`; `512Mi`, `2Gi`); the run is planned by the requests. Not enforced for a local task, not part of the cache key. |
| `network` | `open` \| `none` \| `{allow: [hosts]}` | What the container [reaches](/docs/task/network) besides the services in `needs` (default `open`); `allow` goes through an egress proxy, and a `host:port` entry forwards raw TCP. An error on a local task, except on a [server's cluster](/docs/remote-compute/backend#network); not part of the cache key. |
| `args` | boolean | Whether the last command takes [arguments](/docs/task/arguments#which-tasks-take-arguments) after `--`: `false` refuses them, `true` says it is meant to, which a server with `runs.args: declared` requires. |
| `runner` | string | Name of the [server](/docs/cli/remote) the task has to run on (experimental, see [runner](/docs/task#runner)). |
| `publish` | boolean | `false`: a server leaves the task out of a [published version](/docs/remote-compute/build-files#only-part-of-a-build-file). Locally it changes nothing. |
**Container task** adds:
| Field | Type | Description |
| ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `image` | string (required, unless the task [extends](/docs/task/extending#container-properties-without-an-image) a container task, whose image it takes) | The container image to run in. Its presence is what makes the task containerized. |
| `mounts` | array of string | Extra `host[:container][:ro]` [mounts](/docs/task/container#adding-mounts). |
| `volumeSize` | size (`5Gi`) | On Kubernetes, the size of the [claim](/docs/task/kubernetes#claims) that holds the sources, mounts and outputs of the task (the environment's `volumeSize` when left out, else `1Gi`). Ignored elsewhere, not part of the cache key. |
| `capAdd` | array of string | Linux [capabilities](/docs/task/container#capabilities) added to the container, such as `SYS_ADMIN`. |
**Image build** adds:
| Field | Type | Description |
| ------------ | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `build` | `{context, dockerfile, tags, push, cache, digest, login, insecure}` (required) | The Dockerfile to [build](/docs/task/image-builds) with BuildKit, as `docker build` does: its context, tags, whether to push, a registry `cache` for its layers, a file for the `digest` of the pushed image, and `login`s of `{registry, user, password: {from, account}}`. Without `src`, the task reads the context. |
| `volumeSize` | size (`5Gi`) | As for a container task. |
**Local task** adds:
| Field | Type | Description |
| ---------- | ------------ | ----------------------------------------------------------------------- |
| `platform` | `{os, arch}` | Restrict to `os` (`win`/`macos`/`linux`) and/or `arch` (`arm64`/`arm`). |
## Service [#service]
A service is either a **container service** (it has an `image`) or a **Kubernetes
service** (it has a `selector`).
**Container service:**
| Field | Type | Description |
| ------------- | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `image` | string (required) | The service image. |
| `cmd` | string | Override the container command. |
| `workdir` | string | Working directory inside the container; see [working directory](/docs/service/container#working-directory). Defaults to the build file's directory. |
| `shell` | string | Run `cmd` and the healthcheck as ` -c ""`; see [shell](/docs/service/container#shell). Without it both run in exec form. |
| `ports` | array of string/number | `containerPort` or `hostPort:containerPort` to publish to the host. |
| `healthcheck` | `{cmd, timeout}` | Readiness command, and how long the service may take to pass it once running (duration, default `20s`, `HAMMERKIT_HEALTHCHECK_TIMEOUT` changes the default); see [healthcheck](/docs/service/container#healthcheck). |
| `restarts` | integer 0 to 10 | On Kubernetes, how often the container may crash and be restarted while the service starts before it fails (default `2`); see [restarts](/docs/service/container#restarts). On Docker a crash fails it at once. |
| `init` | string | A task (`name` or `prefix:name`) run once the healthcheck passed; the service is ready for dependents only after it succeeded; see [init](/docs/service/container#init). |
| `envs` | map of string→string/number | Service environment variables. |
| `secrets` | array of `{from, env \| path}` | [Credentials](/docs/task/secrets) injected as an env variable or a read-only file. |
| `resources` | `{cpus, memory}` \| `{requests, limits}` | [CPU and memory](/docs/service/container#resources) limits and requests of the container, as for a task. |
| `network` | `open` \| `none` \| `{allow: [hosts]}` | What the service [reaches](/docs/task/network) besides the services it needs, as for a task; what needs the service always reaches it. |
| `args` | boolean | `false` refuses [arguments](/docs/task/arguments#services) after `--` of `up` for its `cmd`. |
| `volumeSize` | size (`20Gi`) | On Kubernetes, the size of the [claim](/docs/task/kubernetes#claims) that holds the service's sources, mounts and volumes, as for a task. |
| `volumes` | array of string | `name:containerPath[:ro]` volumes that [persist](/docs/service/container#volumes) across restarts. |
| `mounts` | array of string | `host:container[:ro]` config mounts. |
| `deps` | array of string | Tasks that must run before the service starts. |
| `needs` | array of string \| `{service, name}` | Other services this one depends on. |
| `src` | array of string | Sources, used for caching the service image build. |
| `labels` | map of string→string/number | Labels for filtering. |
| `description` | string | Description shown by `ls`. |
| `continuous` | boolean | The service [watches its own files](/docs/service/container#continuous), such as a dev server: `up --watch` leaves it running, and a move of a service run on Kubernetes writes the changed files into its pod. |
| `publish` | boolean | `false`: a server leaves the service out of a [published version](/docs/remote-compute/build-files#only-part-of-a-build-file). |
**Kubernetes service** (forwards an existing cluster resource — see
[Kubernetes service](/docs/service/kubernetes)):
| Field | Type | Description |
| ------------------------ | --------------------------------- | --------------------------------------------------------------------- |
| `selector` | `{type, name}` (required) | Resource `type` (`deployment`/`service`/`pod`) and `name` to forward. |
| `ports` | array of string/number (required) | `hostPort:containerPort` mappings to forward to localhost. |
| `context` | string | Override the environment's kube context. |
| `namespace` | string | Override the environment's namespace. |
| `kubeconfig` | string | Override the environment's kubeconfig path. |
| `deps` | array of string | Tasks that run before forwarding. |
| `labels` / `description` | – | As above. |
## Cache (`caches`) [#cache-caches]
Each named cache has a `method` and a `backend` (see [caches](/docs/build-file/caches)):
| Field | Type | Description |
| ----------------------------------------------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------------- |
| `method` | `checksum` \| `modify-date` \| `none` | How changes are detected. |
| `backend.type` | `local` \| `s3` \| `registry` | Where results are stored. |
| `backend.path` | string (`local`) | Directory; defaults to `~/.hammerkit/remote-cache`. |
| `backend.bucket` | string (`s3`, required) | Target bucket. |
| `backend.region` / `endpoint` / `prefix` / `forcePathStyle` | – (`s3`) | S3 connection options. |
| `backend.repository` | string (`registry`, required) | OCI repository without a tag, e.g. `ghcr.io/org/cache`. |
| `backend.insecure` | boolean (`registry`) | Use plain HTTP; defaults to `true` only for localhost. |
| `retention.maxAge` / `maxSize` / `keepPerTask` | duration / size / integer | [Retention policy](/docs/build-file/caches#retention) for `cache prune` and automatic pruning. |
A task's `cache` field is either the method shorthand (`cache: checksum`) or a
reference `{name, method}` to a declared cache.
## Environment (`environments`) [#environment-environments]
Each environment is a [run target](/docs/task/kubernetes) — exactly one of:
| Key | Fields | Description |
| ------------ | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kubernetes` | `context` (required), `namespace`, `kubeconfig`, `httpRoutes[]`, `networkPolicy` | Run tasks as jobs and services as deployments on a cluster. `networkPolicy: enforced` states that the cluster enforces NetworkPolicy, which [network limits](/docs/task/network#kubernetes) need. |
| `docker` | `host` | Run against a remote Docker daemon. |
An `httpRoutes` entry has `host`, `service` and `gateway` (all required),
`servicePort`, `path` and `gatewayNamespace` — see
[HTTP routes](/docs/task/kubernetes#http-routes).
# References (/docs/build-file/references)
Each build file can have a `references` map pointing at other build files. The
referenced file brings in more tasks and services, which you can then use directly
from the [cli](/docs/cli) or as [task dependencies](/docs/task/dependencies).
```yaml title=".hammerkit.yaml"
references:
foo: project/foo
```
```yaml title="project/foo/.hammerkit.yaml"
tasks:
example:
cmds:
- echo "foo bar"
```
## Naming and prefixes [#naming-and-prefixes]
A referenced task is addressed as `:`, where the prefix is the
key you gave it in the `references` map. The `example` task above becomes
`foo:example`:
```bash
hammerkit foo:example
```
The same `prefix:name` syntax is what you use in `deps` and `needs`:
```yaml title=".hammerkit.yaml"
references:
foo: project/foo
tasks:
build:
deps: [foo:example]
cmds:
- echo "after foo"
```
References can be nested — a referenced file can have its own references — and the
prefixes stack (`foo:bar:task`). This same prefix rule is shared with
[includes](/docs/build-file/includes) and [extend](/docs/task/extending).
## References keep their own working directory [#references-keep-their-own-working-directory]
The key difference from [includes](/docs/build-file/includes): a referenced task runs with **its
own** directory as the working directory. Commands and relative `src`/`generates`
paths resolve against the referenced file's location, not yours. Use a reference
when the other file is a real project in its own right; use an include when you
want to *reuse a task definition* in the current project's directory.
Build-file `envs` do **not** cross a reference. Variables declared at the top level
of one file are not visible to a referenced (or [included](/docs/build-file/includes)) file —
each file's `envs` are scoped to that file. Declare the variables a referenced task
needs in that file (or on the task itself).
## References from git [#references-from-git]
A reference can also point at a git repository, with `git`, an optional `ref` and an
optional `path`. See [include from a git repository](/docs/build-file/includes#include-from-a-git-repository)
for the fields and the caching rules. A git reference runs in its cached checkout, so
use an [include](/docs/build-file/includes) when the task should run in your directory.
## Only on your machine [#only-on-your-machine]
Like an include, a reference can be an object with a `path`, and `publish: false` keeps
what the referenced file declares on your machine: a
[server](/docs/remote-compute/build-files#only-part-of-a-build-file) leaves it out of the
version it publishes.
```yaml
references:
tools: { path: ./tools, publish: false }
```
# cargo (/docs/catalog/cargo)
Build and test a Rust crate or workspace, with the cargo registry kept on the host.
`build` compiles into `target`, `test` runs the tests after it. Both are skipped while
the `.rs` files and `Cargo.toml`/`Cargo.lock` are unchanged, and downloaded crates are
kept in `.cargo` of your home directory, so they are not fetched again.
```yaml title=".hammerkit.yaml"
includes:
cargo:
git: https://github.com/no0dles/hammerkit.git
ref: v1.19.0
path: catalog/cargo
with:
CARGO_ARGS: --locked --workspace
tasks:
check:
deps: [cargo:test]
```
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| ------- | ------------------------------ | -------------------------------------------------- | -------- |
| `build` | Build the crate into `target`. | every `Cargo.toml`, `Cargo.lock`, every `.rs` file | `target` |
| `test` | Run the tests of the crate. | every `Cargo.toml`, `Cargo.lock`, every `.rs` file | |
## Inputs [#inputs]
| Input | Default | Meaning |
| -------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `RUST_VERSION` | `1.90` | Tag of the `rust` image the tasks run in. The image is `rust:-slim`. Quote it in `with` (`'1.90'`), or YAML reads it as a number. |
| `CARGO_ARGS` | `--locked` | Arguments for `cargo build` and `cargo test`, for example `--locked --workspace`. |
## The entry [#the-entry]
```yaml title="catalog/cargo/build.yaml"
# Builds and tests a Rust crate or workspace with the cargo registry kept on the host.
envs:
# tag of the rust image the tasks run in, quoted so 1.90 stays 1.90
RUST_VERSION: '1.90'
# arguments for cargo, for example --locked --workspace
CARGO_ARGS: --locked
tasks:
build:
description: build the crate into target
image: rust:${RUST_VERSION}-slim
src:
- Cargo.toml
- Cargo.lock
- '**/Cargo.toml'
- '**/*.rs'
generates: [target]
mounts:
- $PWD/.cargo/registry:/usr/local/cargo/registry
- $PWD/.cargo/git:/usr/local/cargo/git
cmds:
- cargo build ${CARGO_ARGS}
test:
description: run the tests of the crate
image: rust:${RUST_VERSION}-slim
deps: [build]
src:
- Cargo.toml
- Cargo.lock
- '**/Cargo.toml'
- '**/*.rs'
mounts:
- $PWD/.cargo/registry:/usr/local/cargo/registry
- $PWD/.cargo/git:/usr/local/cargo/git
cmds:
- cargo test ${CARGO_ARGS}
```
# docker (/docs/catalog/docker)
Build a Docker image from a Dockerfile and push it, using the Docker of the machine
hammerkit runs on. `build` builds the image into the local Docker, `publish` builds it
and pushes it to its registry. Both are local tasks that call `docker`, so Docker has to
be installed and running where you run hammerkit.
```yaml title=".hammerkit.yaml"
includes:
docker:
git: https://github.com/no0dles/hammerkit.git
ref: v1.21.0
path: catalog/docker
with:
IMAGE: ghcr.io/acme/hello
IMAGE_TAG: ${VERSION:-dev}
tasks:
release:
deps: [docker:publish]
```
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| --------- | ----------------------------------------------------------------- | ----- | ------------------------- |
| `build` | Build the image and tag it `IMAGE:IMAGE_TAG` in the local Docker. | | the image |
| `publish` | Build the image for `PLATFORMS` and push it. | | the image in its registry |
Neither task declares `src` or `generates`, so both run every time. The image lives in
Docker, not in a folder hammerkit can check, and Docker's own layer cache makes a rebuild
of an unchanged context fast.
`publish` does not log in. Log in before you run it, with `docker login` or your CI's
login step, preferably with a short-lived token of a service account rather than a
personal login. The credentials stay in Docker's config and never reach the build file.
## Inputs [#inputs]
| Input | Default | Meaning |
| ------------ | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `IMAGE` | `app` | The image name, including the registry (`ghcr.io/acme/app`). |
| `IMAGE_TAG` | `dev` | The tag of the image. Take it from the environment with `${VERSION:-dev}` in `with`, as above, and run `VERSION=1.0.0 hammerkit release`. |
| `DOCKERFILE` | `Dockerfile` | The Dockerfile to build, relative to the working directory. |
| `CONTEXT` | `.` | The build context, relative to the working directory. |
| `PLATFORMS` | `linux/amd64` | The platforms `publish` builds, comma separated, for example `linux/amd64,linux/arm64`. More than one needs a builder that supports multi-platform images. |
## The entry [#the-entry]
```yaml title="catalog/docker/build.yaml"
# Builds a Docker image with the Docker of the machine running hammerkit, and pushes it.
envs:
# the image name, including the registry (ghcr.io/acme/app)
IMAGE: app
# the tag of the image
IMAGE_TAG: dev
# the Dockerfile to build, relative to the working directory
DOCKERFILE: Dockerfile
# the build context, relative to the working directory
CONTEXT: .
# the platforms to push, comma separated
PLATFORMS: linux/amd64
tasks:
build:
description: build the image into the local Docker
cmds:
- docker build --file ${DOCKERFILE} --tag ${IMAGE}:${IMAGE_TAG} ${CONTEXT}
publish:
description: build the image and push it to its registry
cmds:
- docker buildx build --file ${DOCKERFILE} --platform ${PLATFORMS} --tag ${IMAGE}:${IMAGE_TAG} --push ${CONTEXT}
```
# dotnet (/docs/catalog/dotnet)
Build and test a .NET project or solution, with the NuGet package cache kept on the host.
`build` compiles into `artifacts`, `test` runs the tests after it. Both are skipped while
the files they read are unchanged, and downloaded packages are kept in `.nuget/packages`
of your home directory, so they are not fetched again.
```yaml title=".hammerkit.yaml"
includes:
dotnet:
git: https://github.com/no0dles/hammerkit.git
ref: v1.20.0
path: catalog/dotnet
with:
DOTNET_VERSION: '10.0'
PROJECT: tests/Hello.Tests
tasks:
check:
deps: [dotnet:test]
```
`PROJECT` names what `dotnet build` and `dotnet test` take: a project, a solution, or
`.` for the one in the directory. Pointing it at the test project builds the app it
references as well.
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| ------- | ---------------------------------------------------- | ------------------------------ | ----------- |
| `build` | Build the project, restoring its packages first. | the files matched by `SOURCES` | `artifacts` |
| `test` | Run the tests of the project without building again. | the files matched by `SOURCES` | |
The tasks pass `-p:UseArtifactsOutput=true`, so `bin` and `obj` go to `artifacts/` at
the root instead of into the project folders. The project folders are the tasks'
inputs: build output written into them would change the inputs, and the next run would
miss the cache. A built app is in `artifacts/bin///`, in lower
case, for example `dotnet artifacts/bin/Hello/release/Hello.dll`.
## Inputs [#inputs]
| Input | Default | Meaning |
| ---------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DOTNET_VERSION` | `10.0` | Tag of the `mcr.microsoft.com/dotnet/sdk` image the tasks run in. Quote it in `with` (`'10.0'`), or YAML reads it as the number 10. |
| `PROJECT` | `.` | The project or solution to build and test. |
| `SOURCES` | `{src/**,tests/**,Directory.*,global.json,nuget.config}` | A glob of the files the tasks read: the code under `src` and `tests`, and the shared build files at the root. Replace it when your projects live elsewhere, for example `{app/**,Directory.*}`. |
| `CONFIGURATION` | `Release` | The build configuration. |
Packages are restored from the sources in your `nuget.config` and are not locked. The
[.NET tutorial](/docs/tutorials/dotnet) shows a build file that restores in its own task
with `--locked-mode`, for projects that want the restore to fail when a lock file no
longer matches.
## The entry [#the-entry]
```yaml title="catalog/dotnet/build.yaml"
# Builds and tests a .NET solution with the NuGet package cache kept on the host.
envs:
# tag of the SDK image the tasks run in, quoted so 10.0 stays 10.0
DOTNET_VERSION: '10.0'
# the project or solution to build and test, as `dotnet build` takes it
PROJECT: .
# a glob of the files the tasks read: the projects and their code, plus the shared
# build files at the root (Directory.Build.props, global.json, nuget.config)
SOURCES: '{src/**,tests/**,Directory.*,global.json,nuget.config}'
# the build configuration
CONFIGURATION: Release
tasks:
build:
description: build the project into artifacts
image: mcr.microsoft.com/dotnet/sdk:${DOTNET_VERSION}
src:
- '${SOURCES}'
generates: [artifacts]
mounts:
- $PWD/.nuget/packages:/nuget/packages
envs:
NUGET_PACKAGES: /nuget/packages
DOTNET_CLI_TELEMETRY_OPTOUT: '1'
cmds:
- dotnet build ${PROJECT} --configuration ${CONFIGURATION} -p:UseArtifactsOutput=true -p:ArtifactsPath=$PWD/artifacts --disable-build-servers
test:
description: run the tests of the project
image: mcr.microsoft.com/dotnet/sdk:${DOTNET_VERSION}
deps: [build]
src:
- '${SOURCES}'
mounts:
- $PWD/.nuget/packages:/nuget/packages
envs:
NUGET_PACKAGES: /nuget/packages
DOTNET_CLI_TELEMETRY_OPTOUT: '1'
cmds:
- dotnet test ${PROJECT} --configuration ${CONFIGURATION} --no-build -p:UseArtifactsOutput=true -p:ArtifactsPath=$PWD/artifacts --disable-build-servers
```
# eslint (/docs/catalog/eslint)
Lint a project with ESLint 9 or later and its flat config. The task runs in a `node` container and is skipped when the sources and the configuration are unchanged.
## Install first [#install-first]
The task runs the tool from `node_modules`, and a container task only sees the files it
declares and the outputs of its dependencies, so the install must be a dependency. Include
the entry of your package manager ([npm](/docs/catalog/npm), [yarn](/docs/catalog/yarn) or
[pnpm](/docs/catalog/pnpm)) and `extend` the task to add it, which works with any of them
and keeps the entry free of a package manager:
```yaml title=".hammerkit.yaml"
includes:
npm:
git: https://github.com/no0dles/hammerkit.git
ref: v1.22.0
path: catalog/npm
eslint:
git: https://github.com/no0dles/hammerkit.git
ref: v1.22.0
path: catalog/eslint
tasks:
lint:
extend: eslint:check
deps: [npm:install]
```
`extend` keeps the entry's image, sources and commands and adds the dependency
(see [extending](/docs/task/extending)).
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| ------- | ----------------------------- | ----------------------------------------------- | -------- |
| `check` | Lint the sources with eslint. | `package.json`, `eslint.config.*`, `SOURCE_DIR` | nothing |
## Inputs [#inputs]
| Input | Default | Meaning |
| -------------- | ------- | ------------------------------------------------------------------------------------------------------- |
| `NODE_VERSION` | `24` | Tag of the `node` image the task runs in, e.g. `22` or `24`. The image is `node:-alpine`. |
| `SOURCE_DIR` | `src` | Folder with the sources to lint, relative to the working directory. |
## Fixing [#fixing]
The entry only checks. A task that rewrites its own sources cannot be cached, so there is no
`--fix` task; run `npx eslint --fix` on your machine when you want one.
## The entry [#the-entry]
```yaml title="catalog/eslint/build.yaml"
# Lints a project with ESLint. The project's own dependencies must be installed first, see the
# page of this entry.
envs:
# tag of the node image eslint runs in
NODE_VERSION: 24
# folder with the sources to lint, relative to the working directory
SOURCE_DIR: src
tasks:
check:
description: lint the sources with eslint
image: node:${NODE_VERSION}-alpine
src: [package.json, 'eslint.config.*', '${SOURCE_DIR}']
cmds:
- node_modules/.bin/eslint ${SOURCE_DIR}
```
# go (/docs/catalog/go)
Build and test a Go module, with the module cache kept on the host.
`build` compiles the packages into `bin`, `test` runs their tests. Both are skipped
while the `.go` files and the `go.*` files are unchanged, and downloaded modules are
kept in `go/pkg/mod` of your home directory, so they are not fetched again.
```yaml title=".hammerkit.yaml"
includes:
go:
git: https://github.com/no0dles/hammerkit.git
ref: v1.19.0
path: catalog/go
with:
GO_VERSION: '1.25'
tasks:
check:
deps: [go:test]
```
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| ------- | ------------------------------ | ---------------------------------------------------------------- | -------- |
| `build` | Build the packages into `bin`. | `go.mod`, `go.sum` and `go.work` where present, every `.go` file | `bin` |
| `test` | Run the tests of the packages. | `go.mod`, `go.sum` and `go.work` where present, every `.go` file | |
## Inputs [#inputs]
| Input | Default | Meaning |
| ------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GO_VERSION` | `1.25` | Tag of the `golang` image the tasks run in. The image is `golang:-alpine`. Quote it in `with` (`'1.30'`), or YAML reads it as the number 1.3. |
| `PACKAGES` | `./...` | The packages to build and test. |
## The entry [#the-entry]
```yaml title="catalog/go/build.yaml"
# Builds and tests a Go module with the module cache kept on the host.
envs:
# tag of the golang image the tasks run in, quoted so 1.30 stays 1.30
GO_VERSION: '1.25'
# the packages to build and test
PACKAGES: ./...
tasks:
build:
description: build the packages into bin
image: golang:${GO_VERSION}-alpine
src:
# go.mod, and go.sum, go.work and go.work.sum where the module has them
- 'go.*'
- '**/*.go'
generates: [bin]
mounts:
- $PWD/go/pkg/mod:/go/pkg/mod
cmds:
- mkdir -p bin
- go build -o bin/ ${PACKAGES}
test:
description: run the tests of the packages
image: golang:${GO_VERSION}-alpine
deps: [build]
src:
# go.mod, and go.sum, go.work and go.work.sum where the module has them
- 'go.*'
- '**/*.go'
mounts:
- $PWD/go/pkg/mod:/go/pkg/mod
cmds:
- go test ${PACKAGES}
```
# helm (/docs/catalog/helm)
Lint and package a Helm chart. The tasks run in an `alpine/helm` container, need no cluster, and are skipped when the chart is unchanged.
```yaml title=".hammerkit.yaml"
includes:
helm:
git: https://github.com/no0dles/hammerkit.git
ref: v1.22.0
path: catalog/helm
tasks:
chart:
deps: [helm:lint, helm:package]
```
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| --------- | ------------------------------ | ----------- | -------- |
| `lint` | Lint the chart with helm lint. | `CHART_DIR` | nothing |
| `package` | Package the chart into dist. | `CHART_DIR` | `dist` |
## Inputs [#inputs]
| Input | Default | Meaning |
| -------------- | ------- | ------------------------------------------------------------------------------- |
| `HELM_VERSION` | `4.3.0` | Tag of the `alpine/helm` image the tasks run in, e.g. `3.19.5` or `4.3.0`. |
| `CHART_DIR` | `chart` | Folder of the chart (with its `Chart.yaml`), relative to the working directory. |
## Talking to a cluster [#talking-to-a-cluster]
The entry does not install or inspect releases, because that depends on a cluster and its
credentials and cannot be cached. Run `helm upgrade` from a task of your own, or in your
deploy pipeline.
## The entry [#the-entry]
```yaml title="catalog/helm/build.yaml"
# Lints and packages a Helm chart. Nothing here talks to a cluster.
envs:
# tag of the alpine/helm image helm runs in
HELM_VERSION: 4.3.0
# folder of the chart, relative to the working directory
CHART_DIR: chart
tasks:
lint:
description: lint the chart with helm lint
image: alpine/helm:${HELM_VERSION}
src: ['${CHART_DIR}']
cmds:
- helm lint ${CHART_DIR}
package:
description: package the chart into dist
image: alpine/helm:${HELM_VERSION}
src: ['${CHART_DIR}']
generates: [dist]
cmds:
- helm package ${CHART_DIR} --destination dist
```
# Catalog (/docs/catalog)
The catalog is a set of build files for common tools such as `npm ci` or
`yarn install`, kept in the hammerkit repository. Each entry is built the way this
documentation recommends, runs in a pinned container image, declares what it reads
and writes so it is cached, and is run by hammerkit's own test suite against a real
project on every change.
You use an entry by [including it from git](/docs/build-file/includes#include-from-a-git-repository)
and calling its tasks:
```yaml title=".hammerkit.yaml"
includes:
npm:
git: https://github.com/no0dles/hammerkit.git
ref: v1.16.0
path: catalog/npm
with:
NODE_VERSION: 22
tasks:
build:
image: node:22-alpine
deps: [npm:install]
src: [src, tsconfig.json]
generates: [dist]
cmds:
- npm run build
```
`hammerkit build` installs the dependencies (or reuses them when `package.json` and
`package-lock.json` are unchanged) and then builds. The include runs in your directory,
so every package of a monorepo can include the same entry.
## Entries [#entries]
| Entry | What it does |
| ---------------------------------- | ------------------------------------------------ |
| [npm](/docs/catalog/npm) | Install locked dependencies with `npm ci`. |
| [yarn](/docs/catalog/yarn) | Install locked dependencies with `yarn install`. |
| [pnpm](/docs/catalog/pnpm) | Install locked dependencies with `pnpm install`. |
| [pip](/docs/catalog/pip) | Install pinned requirements with `pip install`. |
| [go](/docs/catalog/go) | Build and test a Go module. |
| [cargo](/docs/catalog/cargo) | Build and test a Rust crate or workspace. |
| [dotnet](/docs/catalog/dotnet) | Build and test a .NET project or solution. |
| [docker](/docs/catalog/docker) | Build a Docker image and push it. |
| [tsc](/docs/catalog/tsc) | Compile a TypeScript project with `tsc`. |
| [eslint](/docs/catalog/eslint) | Lint a project with ESLint. |
| [prettier](/docs/catalog/prettier) | Check the formatting of a project with Prettier. |
| [jest](/docs/catalog/jest) | Run the tests of a project with Jest. |
| [helm](/docs/catalog/helm) | Lint and package a Helm chart. |
The Node tools (tsc, eslint, prettier, jest) run from `node_modules`, so they do not install
anything themselves: your task `extend`s the entry's task and adds the install of your
package manager as a dependency. Each of their pages shows it.
## Versions [#versions]
`ref` is a release tag of hammerkit: the catalog at `v1.16.0` is the one that release's
tests ran. A commit SHA pins exactly, and a branch such as `master` follows the latest
entries but can differ between machines (see
[caching and offline use](/docs/build-file/includes#caching-and-offline-use)). A task or
input of an entry is not renamed or removed within a major version of hammerkit.
## Inputs [#inputs]
An entry declares its inputs as top-level `envs` with defaults, and lists them on its
page. `with` replaces them for one include; a name the entry does not declare is an
error that lists the ones it does. See
[inputs](/docs/build-file/includes#inputs).
## Adapting an entry [#adapting-an-entry]
Tasks are building blocks, not templates you edit. To go beyond the inputs:
* **Depend on it.** Your own task lists the entry's task in `deps`, as above.
* **Extend it.** `extend: npm:install` adds to an entry task: more `src` entries, more
`cmds`, more `envs`. Add `reset: [src]` to drop what it brings for a property. See
[extending](/docs/task/extending).
* **Another layout.** An entry reads files relative to the directory that includes it, so a
project in `packages/web` includes it from `packages/web/.hammerkit.yaml`. For a lockfile
that lives elsewhere, extend the task and point `src` at it.
## What an include brings [#what-an-include-brings]
Before you adopt an entry, read its page: it shows the whole build file. After including
it, `hammerkit ls` lists its tasks and
[`hammerkit explain --definition npm:install`](/docs/cli/explain#showing-a-tasks-definition)
shows what one runs, with the repository, ref and commit it came from and where each env
value was set. An entry contributes tasks and never runs anything until you run one of them.
# jest (/docs/catalog/jest)
Run the tests of a project with Jest. The task runs in a `node` container, keeps the coverage report as its result, and is skipped when the sources and the Jest configuration are unchanged.
## Install first [#install-first]
The task runs the tool from `node_modules`, and a container task only sees the files it
declares and the outputs of its dependencies, so the install must be a dependency. Include
the entry of your package manager ([npm](/docs/catalog/npm), [yarn](/docs/catalog/yarn) or
[pnpm](/docs/catalog/pnpm)) and `extend` the task to add it, which works with any of them
and keeps the entry free of a package manager:
```yaml title=".hammerkit.yaml"
includes:
npm:
git: https://github.com/no0dles/hammerkit.git
ref: v1.22.0
path: catalog/npm
jest:
git: https://github.com/no0dles/hammerkit.git
ref: v1.22.0
path: catalog/jest
tasks:
test:
extend: jest:test
deps: [npm:install]
```
`extend` keeps the entry's image, sources and commands and adds the dependency
(see [extending](/docs/task/extending)).
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| ------ | --------------------------------------------- | --------------------------------------------- | ---------- |
| `test` | Run the tests with jest and collect coverage. | `package.json`, `jest.config.*`, `SOURCE_DIR` | `coverage` |
## Inputs [#inputs]
| Input | Default | Meaning |
| -------------- | ------- | ------------------------------------------------------------------------------------------------------- |
| `NODE_VERSION` | `24` | Tag of the `node` image the task runs in, e.g. `22` or `24`. The image is `node:-alpine`. |
| `SOURCE_DIR` | `src` | Folder with the sources and their tests, relative to the working directory. |
## Another layout [#another-layout]
Tests that live outside `SOURCE_DIR` are added to the task's `src` when it extends, for
example `src: [tests]`.
## The entry [#the-entry]
```yaml title="catalog/jest/build.yaml"
# Runs the tests of a project with Jest. The project's own dependencies must be installed
# first, see the page of this entry.
envs:
# tag of the node image jest runs in
NODE_VERSION: 24
# folder with the sources and their tests, relative to the working directory
SOURCE_DIR: src
tasks:
test:
description: run the tests with jest and collect coverage
image: node:${NODE_VERSION}-alpine
src: [package.json, 'jest.config.*', '${SOURCE_DIR}']
generates: [coverage]
cmds:
- node_modules/.bin/jest --ci --coverage
```
# npm (/docs/catalog/npm)
Install locked Node.js dependencies with npm ci.
The task runs in a `node` container and keeps `node_modules` as its output, so it
is skipped when `package.json` and `package-lock.json` are unchanged.
```yaml title=".hammerkit.yaml"
includes:
npm:
git: https://github.com/no0dles/hammerkit.git
ref: v1.16.0
path: catalog/npm
tasks:
build:
image: node:24-alpine
deps: [npm:install]
cmds:
- npm run build
```
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| --------- | ------------------------------------------------ | ----------------------------------- | -------------- |
| `install` | Install locked Node.js dependencies with npm ci. | `package.json`, `package-lock.json` | `node_modules` |
## Inputs [#inputs]
| Input | Default | Meaning |
| -------------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `NODE_VERSION` | `24` | Tag of the `node` image the install runs in, e.g. `22` or `24`. The image is `node:-alpine`. |
## The entry [#the-entry]
```yaml title="catalog/npm/build.yaml"
# Installs the locked dependencies of a Node.js project with npm.
envs:
# tag of the node image the install runs in
NODE_VERSION: 24
tasks:
install:
description: install the locked dependencies with npm ci
image: node:${NODE_VERSION}-alpine
src: [package.json, package-lock.json]
generates: [node_modules]
cmds:
- npm ci
```
# pip (/docs/catalog/pip)
Install the pinned requirements of a Python project with pip.
The task runs in a `python` container and installs into `__pypackages__`, so it is
skipped when the requirements file is unchanged. Tasks that need the packages set
`PYTHONPATH=__pypackages__`.
```yaml title=".hammerkit.yaml"
includes:
pip:
git: https://github.com/no0dles/hammerkit.git
ref: v1.19.0
path: catalog/pip
tasks:
test:
image: python:3.13-slim
deps: [pip:install]
src: [app, tests]
envs:
PYTHONPATH: __pypackages__
cmds:
- python -m unittest
```
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| --------- | ----------------------------------------- | ------------------------------------------ | ---------------- |
| `install` | Install the pinned requirements with pip. | the requirements file (`requirements.txt`) | `__pypackages__` |
## Inputs [#inputs]
| Input | Default | Meaning |
| ---------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `PYTHON_VERSION` | `3.13` | Tag of the `python` image the install runs in. The image is `python:-slim`. Quote it in `with` (`'3.10'`), or YAML reads it as the number 3.1. |
| `REQUIREMENTS` | `requirements.txt` | The file to install from, with exact versions (`pip freeze`, `pip-compile`). Use it for a different name or folder, for example `requirements/prod.txt`. |
## The entry [#the-entry]
```yaml title="catalog/pip/build.yaml"
# Installs the pinned requirements of a Python project with pip.
envs:
# tag of the python image the install runs in, quoted so 3.10 stays 3.10
PYTHON_VERSION: '3.13'
# the file to install from, with exact versions (pip freeze, pip-compile)
REQUIREMENTS: requirements.txt
tasks:
install:
description: install the pinned requirements with pip
image: python:${PYTHON_VERSION}-slim
src:
- '${REQUIREMENTS}'
generates: [__pypackages__]
cmds:
- pip install --no-cache-dir --disable-pip-version-check --target __pypackages__ -r ${REQUIREMENTS}
```
# pnpm (/docs/catalog/pnpm)
Install locked Node.js dependencies with pnpm.
The task runs in a `node` container and keeps `node_modules` as its output, so it
is skipped when `package.json` and `pnpm-lock.yaml` are unchanged.
```yaml title=".hammerkit.yaml"
includes:
pnpm:
git: https://github.com/no0dles/hammerkit.git
ref: v1.16.0
path: catalog/pnpm
tasks:
build:
image: node:24-alpine
deps: [pnpm:install]
cmds:
- npm run build
```
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| --------- | ---------------------------------------------- | -------------------------------- | -------------- |
| `install` | Install locked Node.js dependencies with pnpm. | `package.json`, `pnpm-lock.yaml` | `node_modules` |
## Inputs [#inputs]
| Input | Default | Meaning |
| -------------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `NODE_VERSION` | `24` | Tag of the `node` image the install runs in, e.g. `22` or `24`. The image is `node:-alpine`. |
| `PNPM_VERSION` | `10` | pnpm release to run, fetched with corepack the first time, e.g. `9` or `10.28.0`. |
## The entry [#the-entry]
```yaml title="catalog/pnpm/build.yaml"
# Installs the locked dependencies of a Node.js project with pnpm.
envs:
# tag of the node image the install runs in
NODE_VERSION: 24
# pnpm release to run, fetched with corepack
PNPM_VERSION: 10
tasks:
install:
description: install the locked dependencies with pnpm
image: node:${NODE_VERSION}-alpine
src: [package.json, pnpm-lock.yaml]
generates: [node_modules]
envs:
COREPACK_HOME: /tmp/corepack
cmds:
- corepack pnpm@${PNPM_VERSION} install --frozen-lockfile
```
# prettier (/docs/catalog/prettier)
Check the formatting of a project with Prettier. The task runs in a `node` container and is skipped when the sources and the Prettier configuration are unchanged.
## Install first [#install-first]
The task runs the tool from `node_modules`, and a container task only sees the files it
declares and the outputs of its dependencies, so the install must be a dependency. Include
the entry of your package manager ([npm](/docs/catalog/npm), [yarn](/docs/catalog/yarn) or
[pnpm](/docs/catalog/pnpm)) and `extend` the task to add it, which works with any of them
and keeps the entry free of a package manager:
```yaml title=".hammerkit.yaml"
includes:
npm:
git: https://github.com/no0dles/hammerkit.git
ref: v1.22.0
path: catalog/npm
prettier:
git: https://github.com/no0dles/hammerkit.git
ref: v1.22.0
path: catalog/prettier
tasks:
format:
extend: prettier:check
deps: [npm:install]
```
`extend` keeps the entry's image, sources and commands and adds the dependency
(see [extending](/docs/task/extending)).
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| ------- | -------------------------------------------------- | -------------------------------------------- | -------- |
| `check` | Check the formatting of the sources with prettier. | `package.json`, `.prettierrc*`, `SOURCE_DIR` | nothing |
## Inputs [#inputs]
| Input | Default | Meaning |
| -------------- | ------- | ------------------------------------------------------------------------------------------------------- |
| `NODE_VERSION` | `24` | Tag of the `node` image the task runs in, e.g. `22` or `24`. The image is `node:-alpine`. |
| `SOURCE_DIR` | `src` | Folder with the sources to check, relative to the working directory. |
## Writing the formatting [#writing-the-formatting]
The entry only checks. A task that rewrites its own sources cannot be cached, so there is no
`--write` task; run `npx prettier --write` on your machine when you want one. The
configuration is read from a `.prettierrc*` file; a project that keeps it in `package.json`
needs no extra step, because that file is read too.
## The entry [#the-entry]
```yaml title="catalog/prettier/build.yaml"
# Checks the formatting of a project with Prettier. The project's own dependencies must be
# installed first, see the page of this entry.
envs:
# tag of the node image prettier runs in
NODE_VERSION: 24
# folder with the sources to check, relative to the working directory
SOURCE_DIR: src
tasks:
check:
description: check the formatting of the sources with prettier
image: node:${NODE_VERSION}-alpine
src: [package.json, '.prettierrc*', '${SOURCE_DIR}']
cmds:
- node_modules/.bin/prettier --check ${SOURCE_DIR}
```
# tsc (/docs/catalog/tsc)
Compile a TypeScript project with tsc. The task runs in a `node` container, keeps the compiler output as its result, and is skipped when the sources and `tsconfig.json` are unchanged.
## Install first [#install-first]
The task runs the tool from `node_modules`, and a container task only sees the files it
declares and the outputs of its dependencies, so the install must be a dependency. Include
the entry of your package manager ([npm](/docs/catalog/npm), [yarn](/docs/catalog/yarn) or
[pnpm](/docs/catalog/pnpm)) and `extend` the task to add it, which works with any of them
and keeps the entry free of a package manager:
```yaml title=".hammerkit.yaml"
includes:
npm:
git: https://github.com/no0dles/hammerkit.git
ref: v1.22.0
path: catalog/npm
tsc:
git: https://github.com/no0dles/hammerkit.git
ref: v1.22.0
path: catalog/tsc
tasks:
build:
extend: tsc:build
deps: [npm:install]
```
`extend` keeps the entry's image, sources and commands and adds the dependency
(see [extending](/docs/task/extending)).
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| ------- | ----------------------------- | --------------------------------------------- | ------------ |
| `build` | Compile the project with tsc. | `package.json`, `tsconfig.json`, `SOURCE_DIR` | `OUTPUT_DIR` |
## Inputs [#inputs]
| Input | Default | Meaning |
| -------------- | ------- | ------------------------------------------------------------------------------------------------------- |
| `NODE_VERSION` | `24` | Tag of the `node` image the task runs in, e.g. `22` or `24`. The image is `node:-alpine`. |
| `SOURCE_DIR` | `src` | Folder with the TypeScript sources, relative to the working directory. |
| `OUTPUT_DIR` | `dist` | Folder tsc writes to. It must be the `outDir` of your `tsconfig.json`. |
## Another layout [#another-layout]
The entry reads `tsconfig.json` and the `SOURCE_DIR` folder. A monorepo that shares a base
configuration adds it to the task's `src` when it extends, for example
`src: [../../tsconfig.base.json]`.
## The entry [#the-entry]
```yaml title="catalog/tsc/build.yaml"
# Compiles a TypeScript project with tsc. The project's own dependencies must be installed
# first, see the page of this entry.
envs:
# tag of the node image tsc runs in
NODE_VERSION: 24
# folder with the TypeScript sources, relative to the working directory
SOURCE_DIR: src
# folder tsc writes to, the outDir of the tsconfig.json
OUTPUT_DIR: dist
tasks:
build:
description: compile the project with tsc
image: node:${NODE_VERSION}-alpine
src: [package.json, tsconfig.json, '${SOURCE_DIR}']
generates: ['${OUTPUT_DIR}']
cmds:
- node_modules/.bin/tsc
```
# yarn (/docs/catalog/yarn)
Install locked Node.js dependencies with yarn.
The task runs in a `node` container and keeps `node_modules` as its output, so it
is skipped when `package.json` and `yarn.lock` are unchanged.
```yaml title=".hammerkit.yaml"
includes:
yarn:
git: https://github.com/no0dles/hammerkit.git
ref: v1.16.0
path: catalog/yarn
tasks:
build:
image: node:24-alpine
deps: [yarn:install]
cmds:
- yarn build
```
## Tasks [#tasks]
| Task | What it does | Reads | Produces |
| --------- | ---------------------------------------------- | --------------------------- | -------------- |
| `install` | Install locked Node.js dependencies with yarn. | `package.json`, `yarn.lock` | `node_modules` |
## Inputs [#inputs]
| Input | Default | Meaning |
| -------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `NODE_VERSION` | `24` | Tag of the `node` image the install runs in, e.g. `22` or `24`. The image is `node:-alpine`. |
| `YARN_ARGS` | `--frozen-lockfile` | Arguments of `yarn install`. The image ships yarn 1; for yarn berry (2 and later) use a Node image that provides it and pass `--immutable`. |
## The entry [#the-entry]
```yaml title="catalog/yarn/build.yaml"
# Installs the locked dependencies of a Node.js project with yarn.
envs:
# tag of the node image the install runs in
NODE_VERSION: 24
# arguments of `yarn install`; yarn berry (2+) uses --immutable instead
YARN_ARGS: --frozen-lockfile
tasks:
install:
description: install the locked dependencies with yarn
image: node:${NODE_VERSION}-alpine
src: [package.json, yarn.lock]
generates: [node_modules]
cmds:
- yarn install $YARN_ARGS
```
# Cache (/docs/cli/cache)
`hammerkit cache pull` and `hammerkit cache push` move cache entries between each
task's own cache (by default the machine-local `default` cache) and a **remote**
cache declared in the [`caches:`](/docs/build-file/caches) block. They execute no
task and start no container.
Only the entries for the tasks' **current** state are moved — exactly what the next
build of this commit looks up — including all transitive dependencies of the
selected tasks.
```yaml title=".hammerkit.yaml"
caches:
shared:
method: checksum
backend:
type: s3
bucket: my-build-cache
```
```bash
# fast workspace setup: fetch everything the build needs, then build offline
hammerkit cache pull --remote shared
hammerkit run build
# after a trusted build: upload the results for everyone else
hammerkit cache push build --remote shared
```
## Options [#options]
```
Usage: hammerkit cache pull|push [options] [task]
Options:
--remote remote cache declared in the caches block
-f, --filter filter task and services with labels
-e, --exclude exclude task and services with labels
--env environment
--cache caching method to compare (choices: "checksum", "modify-date", "none")
```
## Behavior [#behavior]
* An entry already present at the destination is skipped, so both commands are
safe to re-run.
* An entry missing at the source is reported and skipped — the command still
succeeds.
* An unreachable or unauthorized remote **fails** the command (unlike a build,
where a cache error only degrades to a cache miss).
* `cache push` refuses to run when `HAMMERKIT_CACHE_READ_ONLY` is set.
* Tasks with `cache: none`, or tasks whose own cache already *is* the remote, are
skipped.
## cache ls and cache prune [#cache-ls-and-cache-prune]
`hammerkit cache ls` lists the entries of a cache — by default the machine-local
`default` cache, or any cache from the `caches:` block with `--remote ` — with
the task, size and when it was last used. `--json` prints them for tooling.
```
• build 3f2a91c07d4e 12.4Mi last used 2h ago
• e2e 91bc02f3aa10 1.2Ki last used 2h ago
2 entries, 12.4Mi total
```
`hammerkit cache prune` removes entries according to the cache's
[retention policy](/docs/build-file/caches#retention), or the policy given on the
command line, which takes precedence:
```bash
hammerkit cache prune --keep 1 # newest version of each task only
hammerkit cache prune --max-age 30d --dry-run # show what would be removed
hammerkit cache prune --remote shared --max-size 50Gi
```
```
Usage: hammerkit cache prune [options]
Options:
--remote cache declared in the caches block to prune
--max-age remove entries not used within this long (e.g. 30d)
--max-size remove least recently used entries above this size (e.g. 5Gi)
--keep keep only the newest versions per task
--dry-run show what would be removed without removing it
```
A local cache with a declared `retention` is also pruned automatically after every
successful `run`. A remote cache is only ever pruned when you name it with
`--remote`. Unlike [`clean --cache`](/docs/cli/clean), which drops everything stored for
the tasks of this build file, `prune` keeps what's recent and removes what's old.
# Clean (/docs/cli/clean)
This command cleans up all generated directories and files including the local
cache state of hammerkit.
This is useful for detecting errors in the build file and for verifying that all tasks work without any previous state.
```
hammerkit clean
```
## Clearing stored cache results [#clearing-stored-cache-results]
By default `clean` removes the generated outputs and the per-task cache state of
the current checkout. Add `--cache` to also drop the **stored results** kept by
the cache backend - the local result store under `~/.hammerkit/remote-cache` or a
configured remote backend (for example an S3 bucket or a registry — which must
allow deleting manifests).
```
hammerkit clean --cache
```
This forces the next run to rebuild from scratch instead of restoring a result
from the backend. See [caches](/docs/build-file/caches) for how backends work.
It also removes the cached checkouts of
[remote includes](/docs/build-file/includes#include-from-a-git-repository) in
`~/.hammerkit/includes`, which are shared by all your projects, so the next run fetches them
again. To update them without removing anything, use [`includes pull`](/docs/cli/includes).
`--filter`/`--exclude` label options apply to `clean` as well, so you can clean a
subset of tasks and services.
## Leftovers of killed runs [#leftovers-of-killed-runs]
A run that was killed outright (`kill -9`, an out-of-memory kill) can't remove its
containers. `clean` also removes every hammerkit container on the docker host that
was started on this machine by a hammerkit process that no longer exists, whatever
project it belongs to. Services started with [`up --daemon`](/docs/cli/up) are
kept: [`down`](/docs/cli/down) stops those.
# Down (/docs/cli/down)
Stops the services of your build file, for example the ones started with
[`hammerkit up`](/docs/cli/up) or left running by `up --daemon`.
```bash
hammerkit down
```
Name services to stop only those, or tasks to stop the services they need:
```bash
hammerkit down postgres
hammerkit down dev:serve
```
Names are the ones [`hammerkit ls`](/docs/cli/ls) lists, as for
[`up`](/docs/cli/up#start-by-name). A named service stops alone: the services it
needs keep running, since other services may use them too. A named task stops the
services it and its dependencies need. Only the services matching the
`--filter`/`--exclude` labels are stopped when those options are given. `down`
never runs a task, also not one it names.
Pass `--env ` to target the services of a configured environment, for
example a Kubernetes cluster declared under `environments:`.
## Options [#options]
```
Usage: hammerkit down [options] [names...]
Arguments:
names services to stop, or tasks whose services to stop, all when none is named
Options:
-f, --filter filter task and services with labels
-e, --exclude exclude task and services with labels
--env environment
-h, --help display help for command
```
# Execute (/docs/cli/execute)
Running `hammerkit` without a command (or with the explicit `run` command)
executes a task. It is the default command, so `hammerkit build` and
`hammerkit run build` are equivalent.
## Execute a task [#execute-a-task]
Tasks can be executed by name.
```bash
hammerkit example
```
## Pass arguments [#pass-arguments]
Arguments after `--` are appended to the task's last command, each quoted as one
argument. The task then skips the cache, unless `--cache ` is given; its
dependencies run without them, cached as usual.
```bash
hammerkit run test -- src/login.spec.ts -t "login flow"
```
A run that selects several tasks takes no arguments. See
[arguments](/docs/task/arguments).
## Execute tasks with labels [#execute-tasks-with-labels]
Tasks can be executed by matching labels or filtered by label values.
### By matching labels [#by-matching-labels]
`-f type=build` only executes tasks that have the given label value.
Their dependencies don't need to match the label.
```bash
hammerkit -f type=build
```
### By excluding labels [#by-excluding-labels]
`-e build=ios` excludes tasks that have the given label value.
A task without a matching label is excluded as well if any of its dependencies matches.
```bash
hammerkit -e build=ios
```
## Options [#options]
```
Usage: hammerkit run [options] [task] [-- args...]
Options:
-f, --filter filter task and services with labels
-e, --exclude exclude task and services with labels
-c, --concurrency parallel worker count (default: 4, unbounded when every task and
service requests cpus and memory)
--skip-resource-check start even when a task with its services requests more than
the host has
-w, --watch watch tasks (default: false)
--env environment
-l, --log log mode (choices: "interactive", "live", "grouped")
--cache caching method to compare (choices: "checksum", "modify-date", "none")
--no-summary do not print the end-of-run summary
--summary-json emit the end-of-run summary as JSON (default: false)
--explain print the cache-miss cause when a task rebuilds (default: false)
--dry-run print the execution plan with predicted cache hits/misses without running (default: false)
--cache-read-only restore from cache backends but never push to them (or set HAMMERKIT_CACHE_READ_ONLY=1)
--timeout fail tasks without their own timeout after this long (e.g. 10m)
--max-timeout fail any task after this long, also one with a longer timeout
of its own (e.g. 2m)
--network the most any task or service reaches besides its needs: none,
open or allow:,
--no-skip-deps run dependencies even when nothing needing them has to run
--no-restore only look cache hits up, download their outputs only when a
task that runs needs them
-h, --help display help for command
```
`--cache` defaults to `checksum` everywhere — the same comparison runs locally and
in CI, so a result cached on one is reused on the other. `--log` defaults to
`interactive` outside CI and `live` in CI (hammerkit auto-detects CI from the `CI`,
`CONTINUOUS_INTEGRATION`, `BUILD_NUMBER` or `RUN_ID` environment variables).
## Build summary [#build-summary]
After a run, hammerkit prints one line per task — executed or cached, and how long
it took — followed by the totals and the cache hit ratio:
```
Summary:
build cached 0ms
e2e executed 4.5s
install cached 0ms
1 executed, 2 cached (67% cache hit), 4.7s total
```
A dependency that wasn't needed because every task depending on it was a cache hit
is listed as `skipped` — see [dependencies of cached tasks](/docs/task/dependencies#dependencies-of-cached-tasks).
`--no-summary` hides it; `--summary-json` prints it as JSON on stdout (and nothing
else), for CI dashboards or agents. With `--explain`, every rebuilt task also
prints why it missed the cache, and the summary gets a cause column.
## Dry run [#dry-run]
`--dry-run` prints the execution plan in order, with the predicted cache hit or
miss (and its cause) for every task, without running anything:
```bash
hammerkit run e2e --dry-run
```
```
Dry run (no commands executed):
1. install: cache miss (never cached)
2. build: cache miss (source changed: src/app.js, dependency changed: install)
3. e2e: cache miss (source changed: src/app.js, dependency changed: build)
```
See also [`explain`](/docs/cli/explain) and [`graph`](/docs/cli/graph).
## Checking without restoring [#checking-without-restoring]
On a cache hit, hammerkit normally downloads the task's entry and restores its
`generates` into the checkout, so the outputs are there afterwards. A CI check
often only needs to know the task passed. With `--no-restore`, a hit is only
looked up in the cache backend, and nothing is downloaded:
```bash
hammerkit run test --no-restore
```
Outputs are still restored where a run needs them: when a task that runs (a
cache miss) or a service that starts depends on a hit, that hit's outputs are
downloaded before it starts. A task that only groups others (no `cmds`) runs
nothing, so it restores nothing either. `--no-restore` has no effect in watch
mode.
Leave it off when a later step uses the outputs, for example a deploy after
`build`, or a coverage upload after `test`. A hit leaves the checkout as it is, so
outputs already on disk may be from an older state.
If an entry is removed from the cache before a task that needs it starts, the run
fails. Run again without `--no-restore`.
## Read-only cache [#read-only-cache]
`--cache-read-only` (or `HAMMERKIT_CACHE_READ_ONLY=1`) restores results from cache
backends but never writes to them — for untrusted runners such as agent sandboxes
or pull requests from forks. See
[agents, workspaces and CI](/docs/guides/agents-and-ci#who-may-write-to-the-cache).
## Stopping a run [#stopping-a-run]
Ctrl-C (`SIGINT`), a cancelled CI job or a supervisor stopping hammerkit
(`SIGTERM`) and a closed terminal (`SIGHUP`) all stop a run the same way: running
tasks are cancelled, their containers and the run's services are removed, and each
local task's command gets `SIGTERM` for its whole process group. Whatever is still
alive in that group after 10 seconds is killed; set `HAMMERKIT_STOP_TIMEOUT_MS` to
change the grace period.
A container task's container is removed when the task ends, so `docker ps` shows
nothing from hammerkit once a run is over. If hammerkit itself is killed
(`SIGKILL`), [`clean`](/docs/cli/clean) removes what that run left behind.
# Explain (/docs/cli/explain)
`hammerkit explain [task]` predicts, for every task in scope, whether the next run
would be a **cache hit** or a **cache miss**, and names the input that caused each
miss. Nothing is executed and no container is started. It uses the same state-key
engine as `run`, so the prediction matches what `run` will do.
```bash
hammerkit explain e2e
```
```
• build: cache miss
cause: source changed: src/app.js
cause: dependency changed: install
• e2e: cache miss
cause: dependency changed: build
• install: cache hit
```
## Causes [#causes]
| Cause | Meaning |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `never cached` | No result for this task has been recorded yet. |
| `source changed: ` | A file in `src` was added, removed or changed. |
| `dependency changed: ` | A task in `deps` (transitively) has a different state. |
| `command`, `image`, `env`, `mounts`, … changed | The task definition itself changed (see [cache identity](/docs/task/caching#what-gets-cached-and-what-invalidates-it)). |
A task with `cache: none` is reported as `uncacheable`.
## JSON output [#json-output]
`--json` prints a machine-readable list, for CI annotations or agents:
```json
[
{
"taskId": "ea717e7627d2e12e24acd731af1d1b364c510f32",
"taskName": "build",
"status": "miss",
"causes": [
{ "kind": "source-changed", "identifier": "src/app.js" },
{ "kind": "dependency-changed", "identifier": "install" }
]
}
]
```
## Gating a CI step [#gating-a-ci-step]
`--check` turns the prediction into an exit code: `0` when no task in scope would
be a cache miss (`uncacheable` tasks do not count), `1` otherwise. After a
[`cache pull`](/docs/cli/cache) it tells a pipeline whether a run has anything to
do, so a job can skip expensive setup:
```bash
hammerkit cache pull --remote ci
if hammerkit explain test --check; then echo "up to date"; else hammerkit run test; fi
```
## Showing a task's definition [#showing-a-tasks-definition]
`--definition` prints, instead of the cache prediction, the task as hammerkit resolved it:
the image, commands, `src`, `generates`, `deps` and `envs` after includes, inputs and
`extend` were applied, and where each part came from. It is how you review what an
[included](/docs/build-file/includes) or [catalog](/docs/catalog) task does before you
run it. Nothing is executed.
```bash
hammerkit explain --definition npm:install
```
```
• npm:install
defined in: catalog/npm/build.yaml of https://github.com/no0dles/hammerkit.git at v1.16.0 (commit 7b183f08ebed)
included as: npm
description: install the locked dependencies with npm ci
image: node:22-alpine
src: package.json, package-lock.json
generates: node_modules
cmd: npm ci
env NODE_VERSION: 22 (input of include npm)
```
* **defined in** names the file, and for a git include the repository, ref and the commit
the cache holds, so you see exactly which version runs.
* Each env value shows where it was declared: `task`, `build file`, `input of include `
(a value your `with` passed) or `extended task`.
* A variable taken from your shell or a `.env` file is listed with `from $NAME` and whether
it is set, never its value.
`--json` prints the same information as a list for tools. Services are not part of it yet.
## Related [#related]
* `hammerkit run --explain` prints the miss cause inline for every task that
rebuilds, and adds a cause column to the build summary.
* `hammerkit run --dry-run` prints the execution plan in order — see
[execute](/docs/cli/execute#dry-run).
## Options [#options]
```
Usage: hammerkit explain [options] [task]
Options:
-f, --filter filter task and services with labels
-e, --exclude exclude task and services with labels
--env environment
--cache caching method to compare (choices: "checksum", "modify-date", "none", default: "checksum")
--json emit the explanation as JSON (default: false)
--check exit with code 1 when any task would be a cache miss (default: false)
--definition show the resolved definition of the tasks and where it came from instead (default: false)
```
# Graph (/docs/cli/graph)
`hammerkit graph [task]` serializes the build graph without running anything.
Tasks are boxes, services are rounded nodes; a solid edge is a `deps` edge, a
dotted edge is a `needs` edge. Paste the mermaid output into a GitHub comment or
markdown file to render it.
```bash
hammerkit graph e2e
```
```
graph TD
t0["build"]
t1["e2e"]
t2["install"]
s0(["db"])
t0 --> t2
t1 -.-> s0
t1 --> t0
```
```bash
hammerkit graph --format dot | dot -Tsvg > graph.svg
```
A cycle is reported as a warning after the graph is printed.
## Options [#options]
```
Usage: hammerkit graph [options] [task]
Options:
-f, --filter filter task and services with labels
-e, --exclude exclude task and services with labels
--env environment
--format output format (choices: "mermaid", "dot", default: "mermaid")
```
# Includes (/docs/cli/includes)
`hammerkit includes pull` fetches every [git repository](/docs/build-file/includes#include-from-a-git-repository)
named by `includes` and `references` again and replaces the cached copy with the current
commit of its `ref`. It executes no task.
```
hammerkit includes pull
```
A remote include is fetched once and then read from `~/.hammerkit/includes` on every run,
so a branch or tag does not change underneath you. Use `pull` when you want the update.
The command follows the includes of remote files too, fetches a repository named several
times once, and prints what happened to each:
```
https://github.com/no0dles/hammerkit.git at v1.16.0: up to date at 7b183f08ebed
https://github.com/acme/build-files.git at main: 4e1c0a77d2b9 -> c9f03b1e5a68
```
* A **branch** or **tag** moves to the commit it points to now.
* A **commit SHA** never changes, so it is not fetched again and shows as `pinned`.
* A repository that cannot be reached fails the command and keeps the cached copy.
To forget every cached remote include instead, so the next run fetches it again, use
[`hammerkit clean --cache`](/docs/cli/clean#clearing-stored-cache-results).
# CLI (/docs/cli)
```
Usage: hammerkit [options] [command]
Options:
-V, --version output the version number
--verbose log debugging information (default: false)
--file set build file (default: .hammerkit.yaml)
--set value for $NAME / ${NAME} references, ahead of
the shell and .env (repeatable)
-h, --help display help for command
Commands:
ls [options] list all tasks
clean [options] clear cache and generated
store [options] save task outputs into
restore [options] restore task outputs from
package [options] package services into a docker image
validate [options] validate hammerkit configurations
graph [options] [task] serialize the build graph (tasks, services,
deps/needs edges) as mermaid or dot
explain [options] [task] explain whether tasks would be a cache hit or
miss, without running them
includes manage remote includes (includes pull)
cache inspect, prune and move cache entries
(cache ls / prune / pull / push)
up [options] start services(s)
down [options] stop services(s)
run [options] [task] execute task (default command)
help [command] display help for command
```
When no build file is present, only the [`init`](/docs/cli/init) command is available,
which creates a default `.hammerkit.yaml`. Point hammerkit at a non-default file
with `--file `.
`--set NAME=value` works with every command and in any position, as often as needed.
It supplies the value of `$NAME` and `${NAME}` references in the build file, ahead of
the shell and `.env` files, so a run can be narrowed without exporting variables:
`hammerkit run e2e --set E2E_SPEC=e2e/login.cy.ts`. Like any env value it is part of
the cache key of the tasks that declare it.
## Common options [#common-options]
These options are shared by the task-running commands ([run](/docs/cli/execute),
[up](/docs/cli/up)) and, where they make sense, by `clean`, `store`, `restore` and
`validate`:
| Option | Default | Description |
| ---------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-f, --filter ` | – | Keep only tasks/services matching `key=value` [labels](/docs/labels). |
| `-e, --exclude ` | – | Drop tasks/services matching `key=value` labels. |
| `-c, --concurrency ` | `4` | Number of tasks run in parallel. Unbounded by default when every task and service requests `cpus` and `memory`: the [requests](/docs/task#planning-the-run) then bound the run. |
| `--skip-resource-check` | | Start even when a task with the services it needs [requests more](/docs/task#planning-the-run) than the host has. |
| `-w, --watch` | `false` | Re-run tasks when their `src` changes. |
| `--env ` | – | Run against a declared [environment](/docs/task/kubernetes) (e.g. a cluster). |
| `-l, --log ` | `interactive` (local) / `live` (CI) | `interactive`, `live` or `grouped`. |
| `--cache ` | `checksum` | `checksum`, `modify-date` or `none`. See [caching](/docs/task/caching). |
| `--timeout ` | – | Default [timeout](/docs/task#timeouts) for tasks without their own. |
| `--max-timeout ` | – | [Timeout](/docs/task#capping-every-task) for every task, also one with a longer timeout of its own: each task runs at most the shorter of the two. |
| `--network ` | – | The most any task or service of the run [reaches](/docs/task/network#a-ceiling-for-the-run) besides its needs: `none`, `open` or `allow:,` (a host may carry a port for a TCP forward), whatever it declares. Fails with a local task in the run. |
Label filters are always `key=value`; a key-only `-f dev` is rejected.
## Parallelism [#parallelism]
Independent tasks and dependencies run concurrently up to `--concurrency` workers
(default `4`). A task only starts once all of its `deps` have completed and all of
its `needs` are healthy; tasks on separate branches of the graph run at the same
time. If any task fails, or a service a task needs fails to start, hammerkit stops
scheduling new work and the run fails.
When more tasks are ready than there are free workers, the one on the **longest
remaining chain** starts first: the one whose own duration plus that of the longest
line of tasks waiting for it is the largest. See
[which ready task runs first](/docs/task/dependencies#which-ready-task-runs-first).
## Exit codes [#exit-codes]
| Code | Meaning |
| ----- | ---------------------------------------------------------------------------------------------------------------- |
| `0` | Everything requested succeeded (including a fully cached run). |
| `1` | A task, service, or command failed (build error, validation error, store/restore/package/clean/up/down failure). |
| `127` | The requested task was not found. |
These make hammerkit safe to chain in CI with `&&` or as a pipeline step.
## CI detection [#ci-detection]
Hammerkit auto-detects CI from the `CI`, `CONTINUOUS_INTEGRATION`, `BUILD_NUMBER`
or `RUN_ID` environment variables. The only thing this changes is the **default
log mode** (`live` in CI, `interactive` otherwise). The cache method default is
`checksum` everywhere, so a result cached locally is reused in CI and vice versa.
# Init (/docs/cli/init)
`init` is available only when no build file exists yet. It writes a minimal
`.hammerkit.yaml` with a single `example` task so you have something to run
immediately.
### Command [#command]
```bash
hammerkit init
```
```bash
npx hammerkit init
```
### Console output [#console-output]
```
created .hammerkit.yaml
```
### Result [#result]
```yaml title=".hammerkit.yaml"
envs: {}
tasks:
example:
image: alpine
cmds:
- echo "it's Hammer Time!"
```
Run it with `hammerkit example` (the task is called `example`, not `build`).
Hammerkit keeps its local cache state in a `.hammerkit` directory. Add it to your
`.gitignore` so it is not committed:
```
.hammerkit
```
# ls (/docs/cli/ls)
`ls` prints every task and service hammerkit can see, including the ones pulled in
through [references](/docs/build-file/references) and
[includes](/docs/build-file/includes). It's the fastest way to understand a repo
you just cloned — before guessing task names.
```bash
hammerkit ls
```
## Example output [#example-output]
```
Services:
• postgres
ports: 127.0.0.1:5432 -> 5432
image: postgres:16-alpine
Tasks:
• install
image: node:24-alpine
labels: stage=build
src: package.json package-lock.json
generates: node_modules
• api
needs: postgres
deps: install
image: node:24-alpine
labels: stage=run app=example
src: index.js config.json package.json package-lock.json
generates: node_modules
```
Each entry shows the image (or nothing, for a local task), labels, sources,
generated outputs, and the `deps`/`needs` wiring — the same fields you set in the
build file.
## JSON [#json]
`--json` prints the same tasks and services as JSON, for scripts and agents:
```bash
hammerkit ls --json
```
```json
{
"tasks": [
{
"name": "test",
"description": "run the unit tests",
"type": "container-task",
"image": "node:24-alpine",
"deps": ["install"],
"needs": [],
"labels": {},
"command": "npx vitest run"
}
],
"services": [
{
"name": "api",
"description": "",
"type": "container-service",
"image": "node:24-alpine",
"ports": [{ "host": 3000, "container": 3000 }],
"labels": { "app": ["api"] },
"command": "node server.js"
}
]
}
```
`command` is the task's last command, or the service's `cmd`: where
[arguments](/docs/task/arguments) after `--` are appended. It is `null` when there
is none.
## Options [#options]
```
Options:
-f, --filter filter task and services with labels
-e, --exclude exclude task and services with labels
--json emit the tasks and services as JSON (default: false)
-h, --help display help for command
```
The label options take `key=value` pairs, so you can scope the listing to one part
of a monorepo, for example `hammerkit ls -f project=api`. See
[labels](/docs/labels).
# Package (/docs/cli/package)
The `package` command turns the **container services** of your build file into
Docker images. For each service hammerkit builds a multi-stage image that bakes
in the service itself plus the outputs of every task the service depends on, so
the resulting image is self-contained and ready to run without hammerkit.
```bash
hammerkit package
```
The image for a service `api` is named `/api:`, where the tag
defaults to `latest` (see [Versioning](#versioning)). The `` argument is
required - it is the prefix of the produced image name and the target of `--push`.
Packaging only runs against the Docker runtime. Services without a container
image, and dependency tasks that run locally (without an `image`), are not
supported.
## What gets packaged [#what-gets-packaged]
For every container service hammerkit generates a `Dockerfile` that:
* starts `FROM` the service image,
* copies the generated outputs of the service's dependency tasks (`deps`) into the
image,
* applies the service `envs`, `ports` (as `EXPOSE`) and `cmd`,
* labels the image with `hammerkit.dev/name` and `hammerkit.dev/id`.
A service with no command and no dependencies still produces a valid image from
its base image and sources.
## Pushing to a registry [#pushing-to-a-registry]
Add `--push` to upload the built images to the registry. Credentials can be
passed with `-u/--username` and `-p/--password`.
```bash
hammerkit package registry.example.com --push -u ci -p $REGISTRY_TOKEN
```
### Registry examples [#registry-examples]
The `` argument is just the image-name prefix, so any registry works:
```bash
# Docker Hub - produces docker.io//api
hammerkit package docker.io/ --push -u -p $DOCKER_TOKEN
# GitHub Container Registry
hammerkit package ghcr.io/ --push -u -p $GITHUB_TOKEN
# GitLab Container Registry (inside CI the variables are provided for you)
hammerkit package registry.gitlab.com// \
--push -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
```
## Versioning [#versioning]
By default the produced image is tagged `latest`. Use `-t/--tag` to pin an
explicit version - for example a release number or a CI commit SHA - so the same
build can be promoted across environments:
```bash
hammerkit package registry.example.com -t 1.2.3 --push
```
produces and pushes `registry.example.com/api:1.2.3`.
## Build platform [#build-platform]
By default hammerkit builds for the **host architecture** (`linux/amd64` on x64,
`linux/arm64` on arm64 / Apple Silicon). Override it with `--platform` to build for
a different target, for example when building on arm for an amd64 cluster:
```bash
hammerkit package registry.example.com --platform linux/amd64
```
`--platform` is forwarded to the Docker build, so the Docker daemon must be able to
build that platform (BuildKit / emulation enabled for cross-architecture builds).
## Example [#example]
Given a service that depends on an `install` task:
```yaml title=".hammerkit.yaml"
services:
api:
image: node:alpine
deps: [install]
cmd: node server.js
ports: [3000]
mounts: [server.js]
tasks:
install:
image: node:alpine
src:
- package.json
- package-lock.json
generates:
- node_modules
cmds:
- npm ci
```
```bash
hammerkit package localhost:5000 --push
```
produces and pushes `localhost:5000/api`, with the `node_modules` produced by
`install` already baked in.
## Options [#options]
```
Options:
-f, --filter filter task and services with labels
-e, --exclude exclude task and services with labels
--push push image to registry (default: false)
--build-override-user create and run as a dedicated uid/gid 1000 user
-t, --tag image tag (default: "latest")
--platform target build platform (default: host architecture)
-u, --username registry username
-p, --password registry password
-h, --help display help for command
```
# remote mcp (/docs/cli/remote-mcp)
Experimental. The tools and options can still change between releases.
`remote mcp` is a [Model Context Protocol](https://modelcontextprotocol.io) server on stdio for one
[sandbox](/docs/remote-compute/sandboxes) on a server. An MCP client starts it as a command
and gets tools to read, search and edit the sandbox's files, run commands and published
tasks on it, check them before a push, show what changed and push it to a branch. Like `remote`, it needs no build file: the tasks are the
published ones. The tools, the sign-in and the shell task are described in
[Agents](/docs/remote-compute/agents). It is a [`remote`](/docs/cli/remote) command, so a task
named `mcp` still runs with `hammerkit mcp`.
```bash
hammerkit remote mcp --server corp --sandbox sbx_4vJ0mT2cX9qLw1aB --build sha256:9f2c...
```
| Option | What it does |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--server ` | The registered server, `HAMMERKIT_SERVER`. Default: the one `remote use` set, else the only one. |
| `--sandbox ` | A sandbox of yours on it, `HAMMERKIT_SANDBOX`. |
| `--repo `, `--ref [` | Instead: a sandbox of this repository at this branch, `HAMMERKIT_REPO` and `HAMMERKIT_REF`. One of yours without changes is checked out at the branch again, otherwise a new one is created. |
| `--build ` | The hash of the published version of the build file whose tasks run on the sandbox, as `remote publish` printed it, `HAMMERKIT_BUILD`. |
| `--shell ` | Its task whose last command is a shell, `sh` by default, `HAMMERKIT_SHELL`. `--shell ''` leaves out `bash`. |
Standard output carries the protocol only; what `remote mcp` has to say goes to standard
error, warnings about the build file in the working directory too. It stops when the client
closes its input.
# Remote (/docs/cli/remote)
Experimental. `remote run` runs tasks of a published build file from a git ref of the
repository. Sending a run of the files on your machine (`--on`, running a task with a
`runner:`) arrives in follow-up releases, see
the [remote compute spec](https://github.com/no0dles/hammerkit/blob/master/specs/remote-compute/spec.md).
`remote` manages the list of hammerkit servers your machine knows about. Each one
is registered under a short name, which later features (`--on `, a task's
`runner:`) refer to instead of a URL.
How a server is set up, and how it decides who may run what, is described in
[Remote compute](/docs/remote-compute).
```bash
hammerkit remote add build https://build.example.com --issuer https://login.example.com
hammerkit remote login build
hammerkit remote accounts build
hammerkit remote publish build
hammerkit remote run test --server build
hammerkit remote verify test --server build
hammerkit remote approvals build
hammerkit remote approve run_Qx3k9ZpL2mW7vN4a build
hammerkit remote list
hammerkit remote use build
hammerkit remote remove build
```
Except for `publish`, `run` and `verify`, the commands only touch your host config and talk to the
server, so they work with or without a build file in the working directory.
### Leaving out the server name [#leaving-out-the-server-name]
Every command that takes a server (`[name]`, `[server]` or `--server `) works without
one. It then uses the default set with `remote use`, and when there is none (or the default
is `local` or `auto`), the only registered server. So with one server you can write:
```bash
hammerkit remote add build https://build.example.com --issuer https://login.example.com
hammerkit remote login
hammerkit remote publish
hammerkit remote run test
```
With several servers and no default, hammerkit asks you to name one and lists them:
```
2 servers are registered (build and staging), name one as the last argument, or make one the default with: hammerkit remote use
```
A default that names a server which is no longer registered is an error, never a reason
to pick another one. `remote add`, `remote remove` and `remote use` always need the name. Where a run of
`hammerkit run` goes is a separate choice, see [Runner](/docs/task#runner): a single
registered server does not make those runs remote.
## Commands [#commands]
| Command | What it does |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `remote add --issuer ` | Register a server. `--issuer` is the OAuth issuer the server accepts tokens from, written exactly as the `iss` of its tokens (some providers end it with a slash). `--client-id`, `--audience` and `--scope` pin what a login asks for, see [Signing in](#signing-in). `--token-command ` takes the token for this server from a command instead of a login, see [A token command per server](#a-token-command-per-server); `--no-token-command` removes it. |
| `remote list` | Show the registered servers and which one is the default. |
| `remote use ` | Set the default location: `local`, `auto` or a registered name. |
| `remote remove ` | Forget a server. If it was the default, runs are local again. |
| `remote login [name]` | Sign in to the server with your identity provider. |
| `remote logout [name]` | Forget the sign-in. |
| `remote accounts [name]` | Show who you are on the server and which service accounts you may use. Without a name, the default server or the only one, see [Leaving out the server name](#leaving-out-the-server-name). |
| `remote publish [server]` | Publish the build file of the working directory (or the one `--file` names) and the files it includes, and print its hash, see [Publishing](#publishing-a-build-file). `--dry-run` only checks it against the server's policy. |
| `remote run ` | Run tasks of a published build file on a server from a git ref, print their output and exit as they did, see [Running tasks](#running-tasks). |
| `remote up ` | Start services of a published build file on a server from a git ref and keep them up, with `--endpoint service:port` at addresses of their own, see [Service runs](#service-runs). |
| `remote status [server]` | Show how the server stands for you: the places that run and for how long, the queue with how long each run waited and why, your limits and the runs that count against them, the service runs, and the capacity. `--json` for scripts, see [Status](/docs/remote-compute/status). |
| `remote tasks [server]` | Show what the past runs of each task took: how often it ran, failed and came from the cache, the median and 90th percentile of its time, and how long its runs waited. `--mine` for your runs only, see [Past tasks](/docs/remote-compute/status#past-tasks). |
| `remote capacity [server]` | Show what the machines of the server have and what its runs take: CPU and memory requested and in use, per namespace on Kubernetes, see [Capacity](/docs/remote-compute/status#capacity). |
| `remote dashboard [server]` | Print the address of the [dashboard](/docs/remote-compute/status#dashboard) of a server that has one. |
| `remote services [server]` | List the service runs that are starting, up or asleep, yours and those of others you may see, with their addresses where you may open them, and when they end or sleep. `--ended` adds the ended ones. |
| `remote runs [server]` | List your runs on a server, newest first: their state, what they ran, how long they waited and ran, where a queued one stands, and why one failed. `--limit ` shows more than 20; `--state`, `--services`, `--tasks` and `--all` filter, see [Status](/docs/remote-compute/status#service-runs-and-past-runs). |
| `remote logs [server]` | Print the log of a run, also of one that ended: what its tasks and services printed, and what the server and the cluster said. `--task ` keeps one task or service, `--tail ` the last lines, `--follow` keeps printing until the run ends, see [Reading a run again](#reading-a-run-again). |
| `remote extend [server]` | Keep a service run up for this long from now. |
| `remote stop [server]` | Stop a run; a service run takes its services and endpoints down. |
| `remote retry [server]` | Start a run that is over again as a new run of yours, at the commit it ran, see [Retrying a run](/docs/remote-compute/runs#retrying-a-run). |
| `remote verify ` | Check that the server already passed tasks for the files checked out here, without running them, see [Verifying checks](#verifying-checks). |
| `remote versions [server]` | List the published versions, newest first, with who published them, where they come from and whether they were yanked. `--filter key=value` (repeatable) keeps the ones with that [annotation](/docs/remote-compute/build-files#annotations). |
| `remote approvals [server]` | List the runs that wait for your approval. |
| `remote show [server]` | Show a run that waits for your approval, or a proposed or published version, in full: per task the commands, image, secrets and accounts, and every file of a version, see [Approving runs](#approving-runs) and [Proposing](#proposing-a-version). |
| `remote proposals [server]` | List the proposed versions of build files you made or may approve, see [Proposing](#proposing-a-version). |
| `remote approve [server]` | Let a run go ahead, or publish a proposed version. `--reason ` is optional. |
| `remote yank [server]` | Pull a published version back: no run starts from it. `--reason ` is shown to whoever tries to run it. Needs `builds: publish` or `use` of an account the version works with. |
| `remote unyank [server]` | Let a yanked version go on. Needs `builds: publish`. |
| `remote reject [server]` | End a run, or drop a proposed version. `--reason ` is shown to who asked for it. |
| `remote mcp` | Serve a sandbox on a server as MCP tools over stdio for a coding agent, see [remote mcp](/docs/cli/remote-mcp). |
Re-running `remote add` with the same values changes nothing. What you leave out keeps
what was registered: the pinned client, audience and scope, and the token command. To
point an existing name at a different server pass `--force`, so a typo can't silently
redirect your runs. A name pointed at another server starts without the old token
command. To drop a pin, remove the server and add it again.
`local` and `auto` are reserved and can't be used as server names.
## Host config [#host-config]
Registrations live in `~/.hammerkit/config.yaml`. Set `HAMMERKIT_CONFIG` to use a
different file.
```yaml
servers:
build:
url: https://build.example.com
issuer: https://login.example.com
ci:
url: https://ci.example.com
issuer: https://accounts.google.com
tokenCommands:
ci:
url: https://ci.example.com
issuer: https://accounts.google.com
command: gcloud auth print-identity-token --audiences=hammerkit --include-email --impersonate-service-account=hammerkit-ci@my-project.iam.gserviceaccount.com
run:
on: build
```
`tokenCommands` are written by [`remote add --token-command`](#a-token-command-per-server).
Each is bound to the URL and issuer it was registered for, like a sign-in, and is not used
once the name points elsewhere. They sit beside `servers` so that hammerkit before 2.0,
which refuses unknown keys of a server, still reads the file.
The config is deliberately not part of the project's build file. A checked-in file
naming a server URL could redirect your source code and OAuth tokens, so the
decision of where to send work stays on the machine.
The same file can hold the `secretProviders` and `secretAccounts` this machine reads
secrets with, see [On this machine only](/docs/task/secret-providers#on-this-machine-only).
## URLs [#urls]
Server URLs must use `https`. Plain `http` is accepted only for `localhost`,
`127.0.0.1` and `[::1]`, for local testing. URLs with credentials, a query string
or a fragment are rejected, and a trailing slash is dropped.
## Publishing a build file [#publishing-a-build-file]
A server runs published build files, it does not take task definitions from the one
who runs them. `remote publish` sends yours:
```
$ hammerkit remote publish build
published to build: sha256:9f2c61d0... (3 files)
tasks and services: build, deploy, more:lint
from: path=.hammerkit.yaml repo=github.com/corp/payments commit=4be1a07c... machine=ci-runner-7 user=runner
```
It reads the build file with the parser of a run and sends it together with every
local file it includes or references, as text. That is the bundle, up to 200 files
and 1 MiB. Includes have to be below the directory of the build file, because a
server could not reproduce anything else. That goes for where a file really is too: an
include that is a link (or sits in a linked directory) pointing outside of that
directory is refused, not followed, so publishing a checkout never sends a file of
your machine that the checkout only links to. Git includes are not sent, the server
fetches them and wants them pinned to a commit. The version printed is the hash of
the bundle, computed here the same way the server computes it, so the same files
always have the same version. That hash is the only name the version has: a run
names it, there is no `latest`. Publishing what a server already has says `already
published` and changes nothing.
With the bundle go [annotations](/docs/remote-compute/build-files#annotations) that say
where it comes from, the `from:` line: the path of the build file in its git repository,
the repository and commit, this machine and your user name. When a file of the bundle is
not as the commit has it (changed, or not committed at all), `dirty=true` says so, so the
version is not taken for that commit's. They are not part of the hash. `--annotation key=value` adds one or replaces one, `--annotation user=` leaves one
out, and it can be given more than once. `remote versions` lists what a server has and
finds it by them, and `remote show ` prints one whole. Anybody else sees them as
`from (unverified)`, since the server takes them as you sent them:
```
$ hammerkit remote versions build --filter repo=github.com/corp/payments
build: 1 published
sha256:9f2c61d0... published by ci at 2026-10-12T20:31:00.000Z
from (unverified): path=.hammerkit.yaml repo=github.com/corp/payments commit=4be1a07c... machine=ci-runner-7 user=runner
tasks and services: build, deploy, more:lint
accounts: op-payments-prod
```
You need `builds: publish` on the server, see
[entitlements](/docs/remote-compute/entitlements). A build file that reads secrets as an
account you do not `use` is proposed instead, see below. Without a server name the default
server (or the only one) is used, and `--file ` publishes another build file.
`--dry-run` publishes nothing: the server checks the bundle against its policy and says
what publishing it would do. Any `builds` level may check, `run` too:
```
$ hammerkit remote publish build --dry-run
checked with the policy of build: sha256:9f2c61d0... (3 files), nothing was published
tasks and services: build, deploy, more:lint
publishing it would propose it, it names accounts you do not use: op-payments-prod
```
Tasks and services marked `publish: false` are left out, see
[Only part of a build file](/docs/remote-compute/build-files#only-part-of-a-build-file).
### Proposing a version [#proposing-a-version]
With only `builds: propose`, or when the build file names an account you do not use,
the same command proposes the version. It does not become a version and can not run
until somebody who may publish, and uses those accounts, approves it:
```
$ hammerkit remote publish build
proposed to build: sha256:4be1a07c... (3 files)
tasks and services: build, deploy, more:lint
it can not run before somebody who may publish approves it, until 2026-10-11T11:41:00.000Z
```
The one who may publish reads the whole version before deciding:
```
$ hammerkit remote proposals build
build: 1 proposed
sha256:4be1a07c... proposed by coding-agent, waits for your approval
from (unverified): path=.hammerkit.yaml repo=github.com/corp/payments machine=agent-sandbox-3 user=agent
tasks and services: build, deploy, more:lint
accounts: op-payments-prod, the proposer does not use op-payments-prod
decide before 2026-10-11T11:41:00.000Z
$ hammerkit remote show sha256:4be1a07c... build
sha256:4be1a07c... on build, proposed by coding-agent (https://login.example.com), waiting
deploy task, docker node:22
$ ./deploy.sh
after: build
secret gsm:kube-token as op-payments-prod -> KUBE_TOKEN
...
--- .hammerkit.yaml (entry)
tasks:
build:
cmds: [npm run build, ./upload.sh]
...
$ hammerkit remote approve sha256:4be1a07c... build
approved sha256:4be1a07c... on build, it is published
```
`remote show` prints what each task does and then every file of the bundle, whole and
with control characters replaced by `?`, so nothing hides in it. It does not compare
with an earlier version: a version has no name and follows none, so a diff would only
show what the proposer chose to compare with. `remote reject` drops the proposal, with a
`--reason` the one who proposed it reads in `remote proposals`. Nobody approves their
own proposal, and what you may not decide does not exist for you. After a rejection you
can not propose again for ten minutes, and nobody that version (`approvals.cooldown`).
## Running tasks [#running-tasks]
`remote run` starts a run of a published version on a server, from a branch or tag of
your repository that the server checks out itself, and follows it to its end:
```
$ hammerkit remote run test --server build
run_Qx3k9ZpL2mW7vN4a on build: test from main at 4be1a07c...
[hammerkit] checked out github.com/corp/payments at 4be1a07c...
✓ 42 tests passed
run_Qx3k9ZpL2mW7vN4a succeeded
```
It prints every line of the run as the server logs it, with the task in front when the line
is not of the one task you named (`[hammerkit]` is the server's own, so a task of that name
is best called something else). It exits with `0` when
the run succeeded and with the exit code of the task that failed otherwise, `1` when the
run was rejected, cancelled or failed without one. Control characters of the output show as
`?`, so the output of a run can not rewrite your terminal; tabs stay.
Several tasks named (`remote run ci:check test:e2e`) run as one plan, side by side as far as
their dependencies allow and in no particular order, the
[longest remaining chain](/docs/task/dependencies#which-ready-task-runs-first) first. To run
one after the other, make the second depend on the first.
What it runs, when you leave the options out, is what is checked out where you are:
| Option | What it is | Left out |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--build ` | The published version to run, also `HAMMERKIT_BUILD`. | The hash of the build file here (or the one `--file` names), computed as `remote publish` does. When the server does not have it, see [A build file the server does not have](#a-build-file-the-server-does-not-have). |
| `--repo ` | The repository, such as `github.com/corp/payments`. The server reads it from its own [sources](/docs/remote-compute/sources). | The `origin` of the checkout. |
| `--ref ][` | The branch or tag. | The branch checked out here. In a detached checkout, the one branch or tag of `origin` at its commit, see below. |
| `--commit ` | The commit the ref has to be at. A ref that moved since is refused, so the run executes the code you meant: `main is at 9e1f... now, not at 4be1a07c...`. | The commit checked out here when `--ref` is left out too, else no pin. |
| `--server ` | The server. | The default of `remote use`, else the only registered server. |
| `--env`, `--cache`, `--timeout`, `--max-timeout` | As for [`run`](/docs/remote-compute/runs#submitting-a-run). | The server's and the build file's defaults. |
| `--detach` | Submit the run and exit, without following it: it prints the run and `follow it with: hammerkit remote logs --follow`. For a [check](#verifying-checks) that CI verifies later. | Follow the run to its end. |
| `--publish` | Publish the build file here first when the server does not have it, without asking. Not with `--build` or `HAMMERKIT_BUILD`. | Ask at a terminal, fail without one. |
A detached checkout, such as a worktree made with `git worktree add --detach`, is on no
branch. Without `--ref` it takes the one branch of `origin` (as your last fetch or push left it)
or tag whose head is its commit, pinned to that commit as `--commit` does, so a branch that
moved since is refused. With none, or several, it fails and names them:
```
the checkout here is on no branch, and its commit 4be1a07c... is the head of branch agent/fix and tag v1.2.0 of origin: name one with --ref
```
A name that looks like a commit is left out, since the server would run it as that commit, and
a branch and a tag of the same name can only be run by the commit (`--ref 4be1a07c...`).
What follows `--` goes to the one task named, as the
[arguments](/docs/remote-compute/runs#arguments) of a run. The server refuses them for a
task that reads a secret or depends on one that does, for a task that says `args: false`, and,
on a server with `runs.args: declared`, for every task that doesn't say `args: true`.
A run on an account you may only `request` waits for an approval, and says so:
```
waiting until somebody who uses deploy-prod approves it, before 2026-10-24T10:41:00.000Z: hammerkit remote approve run_Qx3k9ZpL2mW7vN4a
```
The command waits as long as the server lets the run wait (`approvals.timeout`, one hour by
default). Ctrl-C, or the cancel of a CI job, cancels the run on the server: the command waits
up to ten seconds for the server to take the cancel before it ends. The token is the one of
[Signing in](#signing-in), or `HAMMERKIT_TOKEN_COMMAND` / `HAMMERKIT_TOKEN`, which is how a CI
job signs in with its own identity, see
[CI through a server](/docs/guides/agents/github-actions).
### A build file the server does not have [#a-build-file-the-server-does-not-have]
When you leave `--build` out and the server does not have the build file here, a run can not
start, since a server only runs [published](#publishing-a-build-file) versions. At a terminal
the command offers to publish it and goes on once you answer `y`:
```
$ hammerkit remote up web --endpoint web:8080
the build file here is not published on build (sha256:9f2c61d0...), 3 files
tasks and services: build, web, more:lint
publish it to build now and go on? [y/N] y
published to build: sha256:9f2c61d0... (3 files)
tasks and services: build, web, more:lint
from: path=.hammerkit.yaml repo=github.com/corp/shop commit=4be1a07c... machine=laptop user=pascal
run_Vd1rX9pQ2mT7wN4a on build: web from feature/login at 4be1a07c...
```
It publishes exactly as `remote publish` does, with the same [entitlements](/docs/remote-compute/entitlements):
* Any other answer, or Ctrl-C, publishes nothing and the command fails.
* With only `builds: run` it does not ask: it says that somebody who may publish has to run
`remote publish`.
* When publishing would only [propose](#proposing-a-version) the version (`builds: propose`, or
an account you do not use), it asks `propose it to build now?` and says why. A proposal can
not run yet, so the command proposes it and fails with `is only proposed on build, nothing was
started`; run it again once it is approved.
Without a terminal, in CI or with stdin or stdout redirected, or with `CI` set, it asks
nothing and fails with `the build file here is not published on build (sha256:...), publish it
first with: hammerkit remote publish build, or pass --publish`. `--publish` publishes it
without asking, for a job that may publish. A job that should only run what was published
leaves it out, and the failure tells that the build file changed.
With no build file here at all, the command was most likely started in another folder than the
checkout meant. It fails and says so: `there is no build file at . Run this in the
checkout whose build file the server has, or name a published version with --build ,
listed by: hammerkit remote versions`.
## Service runs [#service-runs]
`remote up` starts a [service run](/docs/remote-compute/service-runs): services of a published
version from a branch or tag, kept up until their time ends, and follows it until they are
up. Then it prints the address of each endpoint and exits; the services stay.
```
$ hammerkit remote up web --endpoint web:8080 --ttl 3d --sleep-ttl 30m
run_Vd1rX9pQ2mT7wN4a on build: web from feature/login at 4be1a07c...
[hammerkit] checked out github.com/corp/shop at 4be1a07c...
run_Vd1rX9pQ2mT7wN4a is up until 2026-10-28T10:12:00.000Z, asleep after 30m without a request
web:8080 https://feature-login-8080-3fa2c1e09b7d.run.example.com/
```
| Option | What it is | Left out |
| -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ |
| `--endpoint ` | A port of a service to publish, the container port it declares. More than once for more. | No endpoints. |
| `--ttl ` | How long the run lasts once its services are ready, asleep or not; then they are removed. | The server's `runs.up.ttl`. |
| `--sleep-ttl ` | How long the services stay up without a request, then they [sleep](/docs/remote-compute/service-runs#sleeping) until the next request wakes them. | The server's `runs.up.sleepTtl`, or never. |
| `--name ` | Lower case letters, digits and `-`. A later `remote up` of yours with this name moves this run to its commit, or replaces it when it starts other services or another build file; the addresses stay. | The branch. |
| `--build`, `--repo`, `--ref`, `--commit`, `--server`, `--env`, `--publish` | As for [`remote run`](#running-tasks), see also [A build file the server does not have](#a-build-file-the-server-does-not-have). | As there. |
When a run of yours with the name is up or asleep and starts the same services of the same
build file version, `remote up` [moves it](/docs/remote-compute/service-runs#moving-to-another-commit)
to the commit instead: it keeps its id, addresses and the data of its services, and only
services that changed restart (on Kubernetes). It follows the move until the run is up at the
new commit, or for a run that is asleep, says which commit its next wake starts:
```
$ hammerkit remote up web --endpoint web:8080
run_Vd1rX9pQ2mT7wN4a on build (up) moves to feature/login at 9e01b77d...
[hammerkit] moving from 4be1a07c... to 9e01b77d...
run_Vd1rX9pQ2mT7wN4a is up until 2026-10-28T10:12:00.000Z, asleep after 30m without a request
web:8080 https://feature-login-8080-3fa2c1e09b7d.run.example.com/
```
A move that does not get there leaves the run where it was and exits with `1`:
`did not move to 9e01b77d...: ; it stays at 4be1a07c...`. Ctrl-C while it moves stops
following; the run moves on without.
It prints what the services print while they start, on Kubernetes too, and the server's
`[hammerkit]` lines, such as what a service waits for (`web waits for a node: 0/3 nodes are
available: 3 Insufficient cpu`). When the run fails before it is up, it says which service
failed and why, and exits with `1`:
```
[web] service web can not become ready, it keeps crashing: it exited with code 1 (Error), restarted 2 times
[hammerkit] what the cluster said in hk-run-3f2a...:
[hammerkit] pod web-6c9f...-x2k8q: Running, restarted: it exited with code 1 (Error), restarted 2 times
[hammerkit] event BackOff pod/web-6c9f...-x2k8q: Back-off restarting failed container web (4 times)
read its log again with: hammerkit remote logs run_Vd1rX9pQ2mT7wN4a
run_Vd1rX9pQ2mT7wN4a failed: web: service web can not become ready, it keeps crashing: it exited with code 1 (Error), restarted 2 times
```
Ctrl-C while it starts cancels it.
`remote extend 8h` keeps it for that long from now (never shorter than it was),
`remote stop ` takes it down, asleep or not.
## Reading a run again [#reading-a-run-again]
A run that failed says why when it ends: the error of the run, then each task that failed
with its own error or exit code, then where its log stays:
```
migrate: exit code 2
read its log again with: hammerkit remote logs run_Qx3k9ZpL2mW7vN4a
run_Qx3k9ZpL2mW7vN4a failed: migrate failed
```
The server keeps the log of your last runs (`runs.keep`, 200 by default) after they ended,
also over a restart, so the pods of a run can be gone and its log is still there.
`remote runs` lists them, newest first:
```
$ hammerkit remote runs
run_Vd1rX9pQ2mT7wN4a 2026-11-07T07:12:03.000Z failed up web: web: service web can not become ready, its image ghcr.io/corp/web:4be1a07 could not be pulled (ImagePullBackOff)
run_Qx3k9ZpL2mW7vN4a 2026-11-07T06:58:41.000Z succeeded test
```
`remote logs ` prints the log of one, each line with its task or service in front, then
how it ended:
| Option | What it does |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--task ` | Only the lines of this task or service (`hammerkit` for the server's own), without the name in front. |
| `--tail ` | Only the last lines, at most 1000. |
| `-f`, `--follow` | Keep printing what comes until the run ends, as `remote run` does. Ctrl-C stops following and leaves the run alone. For a service run that is up, that is the output of its services. |
`remote retry ` starts a run that is over again, as a new run of yours: the same tasks,
settings and arguments, at the commit it ran. It prints the new run, which you follow with
`remote logs --follow`:
```
$ hammerkit remote retry run_Qx3k9ZpL2mW7vN4a
run_Hk2v8RqW5nM3xT1b on build retries run_Qx3k9ZpL2mW7vN4a: queued, follow it with hammerkit remote logs run_Hk2v8RqW5nM3xT1b --follow
```
The server decides about it as about any run you start, see
[Retrying a run](/docs/remote-compute/runs#retrying-a-run).
A log longer than the server keeps of a run (`runs.maxLogBytes`) ends with the last lines of
each task, which the server always keeps. On Kubernetes the log of a run that failed also
holds what the cluster said before the server removed its pods: each pod that was not as it
should be, and the warning events of the run's objects, such as `FailedScheduling`, a failed
image pull or a container that was killed for its memory. A container that ended says its exit
code, the reason Kubernetes gave and, above 128, the signal that killed it
(`container web: it exited with code 137 (Error), killed by SIGKILL, restarted 0 times`); only
one with a healthcheck adds that a failing healthcheck kills it so. A pod that copies a task's
files into or out of its volume says it is a helper of that task, not the task. The output of a
failed probe is the service's own and is left out. The server needs `list` on events for that, see
[What the server may do](/docs/remote-compute/backend#what-the-server-may-do).
## Verifying checks [#verifying-checks]
`remote verify` tells whether the server already passed tasks for the files checked out here,
so a CI job does not have to run them:
```
$ hammerkit remote verify test e2e --server build
test: passed 2026-10-25T09:14:03.120Z in 1m12s, run_Qx3k9ZpL2mW7vN4a by agent-x, from a check of sandbox sbx_4vJ0mT2cX9qLw1aB, fetched at 4be1a07c1a2b
e2e: no run on build passed it for these files
not verified: e2e; run on the server: hammerkit remote run e2e
```
It computes the [check key](/docs/remote-compute/checks#check-keys) of each task over the files
here, runs nothing, and asks the server for a run that passed it: a run from a ref without
arguments, or a check of a sandbox. It exits `0` when there is one for every task, and `1` at
once otherwise, also for a task that has no key (no `src` that matches a file). It never waits
for a run and never starts one. The guide is
[Checks that ran before CI](/docs/guides/agents/verified-checks).
| Option | What it is | Left out |
| -------------------- | ------------------------------------------------ | ------------------------------------------------------------- |
| `--repo ` | The repository, as the server knows it. | The `origin` of the checkout. |
| `--env ` | The environment of the build file, as for `run`. | None. |
| `--server ` | The server. | The default of `remote use`, else the only registered server. |
The caller needs `checks: read` or a `builds` level, with `repos` that cover the repository,
see [Entitlements](/docs/remote-compute/entitlements).
## Approving runs [#approving-runs]
A run that reads secrets as a service account its caller may only
[request](/docs/remote-compute/entitlements#approvals) waits for somebody who uses that account to
approve it. If you use the account:
```
$ hammerkit remote approvals build
build: 1 waiting for your approval
run_Qx3k9ZpL2mW7vN4a requested by 4f1c2b (https://login.example.com)
build file sha256:9f2c61d0...
tasks: deploy
accounts: op-payments-prod, source ref refs/heads/main, env=staging
op-payments-prod: waits for somebody who uses it to approve
decide before 2026-10-11T10:41:00.000Z
$ hammerkit remote approve run_Qx3k9ZpL2mW7vN4a build
approved run_Qx3k9ZpL2mW7vN4a on build, it is queued
```
`remote show` prints what each task of the run does, so the decision does not need the
build file:
```
$ hammerkit remote show run_Qx3k9ZpL2mW7vN4a build
...
deploy task, docker node:22 [remote-include]
$ ./deploy.sh
after: build
secret gsm:kube-token as op-payments-prod -> KUBE_TOKEN
```
The line `look at:` of the list names the tasks with a flag: `local-task` runs on the host
and not in a container, `host-port` binds a port of the host and `remote-include`
includes a file from git. A secret's value is never shown, only where it is read from.
A run that passes [arguments](/docs/remote-compute/runs#arguments) uses no account, so it
waits for nobody; one stored before that rule lists them on a line of their own, each as a
JSON string (`arguments: "--region" "eu, us"`). `remote show` prints each command as it
runs. Arguments and commands are never cut; other text longer than 300 characters ends in `… (+N chars)`. Control
characters show as `?`, and bidi and zero-width characters are written out
(`\u{202e}`), so what you read is the order it runs in. A task that isn't `open` shows
its `network`.
What you see is what the decision rests on: who asked, which version of the build
file, the tasks, the service accounts they read secrets as and where the code comes
from. The version is the one the run is pinned to, so what you approve is what runs.
You approve for every account you use; a run on several waits until they are all
covered. You see the runs on the accounts you use, and no others.
## Signing in [#signing-in]
`remote login` uses the OAuth device authorization grant, so it works on a machine
without a browser:
```
$ hammerkit remote login build
signing in to https://build.example.com through https://login.example.com
client hammerkit-cli, audience hammerkit, scope openid api://hammerkit/.default
To sign in, open https://login.example.com/activate and enter the code ABCD-EFGH
signed in to build as 4f1c2b
service accounts: op-payments-dev (use), op-payments-prod (request), op-web-dev (use)
```
Open the address on any device, sign in as you normally do, and enter the code. The
server tells hammerkit which identity provider client, audience and scope to use, so
there is nothing to configure beyond `remote add`. The `--issuer` you registered has to be one the
server accepts, otherwise the login stops before the browser step.
The server is only trusted with that once. The first successful login pins the client,
audience and scope in the host config, and a later login refuses a server that asks
for something else (register it again with `remote add --force` if the change is
expected). Look at the line printed before the code: a client or audience you do not
recognise, such as one of a cloud provider's own APIs, means stop. To not rely on the
first use, give them to `remote add` (`--client-id`, `--audience`, `--scope`), the
values come from whoever runs the server. Before a token is sent to the server,
hammerkit checks that it was issued by the registered issuer for the pinned audience,
so a token for another service never leaves your machine. `offline_access` is dropped
from the scope: no refresh token is requested.
What is stored is one short-lived access token in `~/.hammerkit/credentials.json`
(`HAMMERKIT_CREDENTIALS` picks another file), readable by you only (a file others can read is not used; hammerkit says how to fix it). A refresh token
is not kept: when the token expires, `remote login` again. The token is bound to the
URL and issuer it was issued for, so registering the name again for another server
(`remote add --force`) does not send the old token there.
Agents and CI do not use `login`. They bring their own token, an agent from its own
client credentials and a CI job from the OIDC token of the job, and the server maps
it to service accounts through its [entitlements](/docs/remote-compute/entitlements).
The `remote` commands take such a token instead of a sign-in, in this order:
| Variable | What it is |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `HAMMERKIT_TOKEN_COMMAND` | A command that prints the token, run for each command, such as a script that asks the identity provider for a service account. |
| `HAMMERKIT_TOKEN` | The token itself, as a CI job gets it. |
```yaml title=".gitlab-ci.yml"
publish:
id_tokens:
HAMMERKIT_TOKEN:
aud: hammerkit
script:
- hammerkit remote add build https://build.example.com --issuer https://gitlab.com
- hammerkit remote publish build
```
The token goes only to the server the name is registered for.
### A token command per server [#a-token-command-per-server]
The variables hold for every server. When servers want tokens of different service
accounts or audiences, or you don't want to export a variable in every shell, register the
command with the server:
```bash
hammerkit remote add ci https://ci.example.com --issuer https://accounts.google.com \
--token-command 'gcloud auth print-identity-token --audiences=hammerkit --include-email --impersonate-service-account=hammerkit-ci@my-project.iam.gserviceaccount.com'
hammerkit remote publish ci
```
Every `remote` command and [`remote mcp`](/docs/cli/remote-mcp) that talks to `ci` then runs
it for a token, and runs it again when the server stops accepting the token. It runs with
your environment and identity, through the shell, and has a minute to print the token alone
on stdout. When it fails, hammerkit says how it ended and shows what it printed on stderr;
what it printed on stdout is never shown, as it can hold more than the token. The order is:
| Source | Used for |
| ---------------------------- | ----------------------------------------------------------------- |
| `HAMMERKIT_TOKEN_COMMAND` | Every server, so a CI job or an agent can bring its own identity. |
| `HAMMERKIT_TOKEN` | Every server. |
| `remote add --token-command` | The one server it is registered with. |
| `remote login` | The one server it signed in to. |
A server with a token command needs no `remote login`, and `remote login` says so. A server
that offers no sign-in for its issuer (an issuer without a `clientId` in the server's config)
can only be reached with a token from elsewhere, and `remote login` points you to
`--token-command`.
The command lives in your host config only, like the server's URL: a build file can't name
one, since whoever writes it decides what runs on your machine. For the same reason keep the
host config writable by you only: whoever may edit it, an agent too, can make hammerkit run
any command, also in the [agent plugins](/docs/remote-compute/agents). `remote add` and
`remote list` print the command, so don't put a secret in it; point it at a script that
holds one instead. Make it print a token of a service account, see
[Service accounts on Google Cloud](/docs/guides/agents/google-accounts), not of yourself.
# Server (/docs/cli/server)
Experimental. The config and the API can still change between releases.
`server` turns a machine into the target of remote runs. It reads its own config file, so
it needs no build file. What the config holds and how a server decides who may run what is
described in [Remote compute](/docs/remote-compute).
```bash
hammerkit server --config /etc/hammerkit/server.yaml
```
| Option | What it does |
| --------------------- | ------------------------------------------------------------------------------------------------ |
| `-c, --config ` | The [server config](/docs/remote-compute#config). `HAMMERKIT_SERVER_CONFIG` can stand in for it. |
The server runs until it is interrupted. `SIGTERM` stops it gracefully, and `SIGHUP` reads
the entitlements of the config again without a restart, see
[Changing entitlements](/docs/remote-compute/entitlements#changing-entitlements).
# Store / Restore (/docs/cli/store-restore)
The store/restore commands are intended for CI systems. The store command takes a destination folder, into which all generated files and folders are moved for later recovery. Restoring the previous state speeds up your CI workflows, because task caching then works across pipeline runs.
### Example workflow [#example-workflow]
To demonstrate the behavior, we use the following example: a simple Node.js TypeScript project that installs npm dependencies and compiles the TypeScript source code.
```yaml title=".hammerkit.yaml"
tasks:
install:
src:
- package.json
- package-lock.json
cmds:
- npm ci
generates:
- node_modules
build:
deps: [install]
src:
- src
- tsconfig.json
generates:
- dist
cmds:
- node_modules/.bin/tsc -b
```
The CI system runs the `build` task on every commit. The goal of store/restore is to prevent unnecessary work in CI. Together with hammerkit's caching, the build task is skipped if the source code hasn't changed. The npm install is skipped as well, and the node_modules folder is restored, if `package.json` and `package-lock.json` are unchanged.
### Saving state [#saving-state]
```
hammerkit store cache
```
After the store command has completed, the hammerkit cache, including all files from tasks that declare `generates`, is saved in the given destination folder. The folder can then be cached with your CI's caching mechanism.
### Restoring state [#restoring-state]
Restore the state before running the required tasks in your CI workflow: let your CI's caching mechanism restore the previously saved folder, then restore it with hammerkit.
```
hammerkit restore
```
### Example with GitLab CI [#example-with-gitlab-ci]
This example uses GitLab CI caching to speed up the build time of the `build` command. For further details, take a look at the [demo repository](https://gitlab.com/pascalbe/hammerkit-typescript-restore-demo).
```yaml title=".gitlab-ci.yml"
before_script:
- npm i -g hammerkit
build:
image: node:24
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- cache
policy: pull-push
script:
- hammerkit restore cache
- hammerkit build
- hammerkit store cache
```
# Up (/docs/cli/up)
Services normally start and stop automatically based on the `needs` of the tasks
you run. The `up` command starts them directly, which is useful to keep shared
services such as a database running across several task runs during development.
```bash
hammerkit up
```
This starts every service in the build file (or the [named ones](#start-by-name),
or the ones matching the `--filter`/`--exclude` labels) and waits until you stop it. Use `--daemon` to
start the services in the background and return immediately.
```bash
hammerkit up --daemon
```
With `--daemon`, `up` returns once every service passed its healthcheck
(`--wait ready`, the default). `--wait start` returns as soon as the containers
are started instead; services that other services need are still awaited, since
those can only start once their needs are ready.
A service that is already running is reused, unless its definition changed since
it was started (image, command, env values, ports, mounts, volumes, healthcheck, its
init task, or a service it needs): then `up` and `run` recreate it. A reused service
does not run its [init](/docs/service/container#init) again.
Stop services started this way with [`hammerkit down`](/docs/cli/down).
## Start by name [#start-by-name]
Name the services to start only those, with the services they need:
```bash
hammerkit up postgres redis
hammerkit up --daemon infra:postgres
```
A service of an [include](/docs/build-file/includes) or a
[reference](/docs/build-file/references) goes by its full name, as
[`hammerkit ls`](/docs/cli/ls) lists it. A task name starts the task, its
dependencies and the services it needs: `hammerkit up dev:serve` keeps a dev server
and its database running. A name that is no task or service is an error, and names
can't be combined with `--filter`/`--exclude`.
## Pass arguments [#pass-arguments]
Arguments after `--` are appended to the `cmd` of the one service (or task) `up`
starts:
```bash
hammerkit up api -- --inspect
hammerkit up --filter app=api -- --inspect
```
Names or a filter that select several services or tasks take no arguments. See
[arguments](/docs/task/arguments#services).
Pass `--env ` to start the services in a configured environment, for
example a Kubernetes cluster declared under `environments:`. See
[running on Kubernetes](/docs/task/kubernetes).
## Options [#options]
```
Usage: hammerkit up [options] [names...] [-- args...]
Arguments:
names tasks and services to start, all when none is named
Options:
-f, --filter filter task and services with labels
-e, --exclude exclude task and services with labels
-c, --concurrency parallel worker count (default: 4, unbounded when every task and
service requests cpus and memory)
--skip-resource-check start even when a task with its services requests more than
the host has
-w, --watch watch tasks (default: false)
--wait with --daemon, return once services are ready or
once they started (choices: "ready", "start",
default: "ready")
-d, --daemon run services in background (default: false)
--env environment
-l, --log log mode (choices: "interactive", "live", "grouped")
--cache caching method to compare (choices: "checksum", "modify-date", "none")
--cache-read-only restore from cache backends but never push to them (or set HAMMERKIT_CACHE_READ_ONLY=1)
--timeout fail tasks without their own timeout after this long (e.g. 10m)
--max-timeout fail any task after this long, also one with a longer timeout
of its own (e.g. 2m)
--network the most any task or service reaches besides its needs: none,
open or allow:,
-h, --help display help for command
```
# Validate (/docs/cli/validate)
`validate` checks the build file (and everything it references/includes) without
running anything, so you can catch mistakes early — ideal as a CI lint step.
```bash
hammerkit validate
```
It reports two levels:
* **error** — something that would break a run, such as a dependency cycle. Any
error makes `validate` exit with code `1`.
* **warn** — something suspicious that still runs, such as a task with no
description, a container service without a healthcheck, a `src`/`mount` path
that doesn't exist, or build-file `envs` a task never references (see
[environment variables](/docs/build-file/environment-variables#cache-key)).
Warnings do **not** change the exit code.
## A clean run [#a-clean-run]
With no problems, nothing is reported and the command exits `0`:
```
hammerkit validate
```
## A run with findings [#a-run-with-findings]
Each finding is grouped by the file it came from and shows its level, the item it
applies to, and a message:
```
.hammerkit.yaml
warn at postgres missing healthcheck
warn at build missing description
error at build task cycle detected build -> test -> build
```
Here the two warnings are advisory, but the cycle is an error, so the command exits
`1` and a CI job using it fails.
Run this command in CI alongside your other linting jobs.
A malformed build file (an unknown key, a missing required `image`, the wrong
type) is rejected by the schema as soon as hammerkit reads it — on *any* command,
not just `validate`. `validate` adds the semantic checks above on top of that.
## Options [#options]
```
Options:
-f, --filter filter task and services with labels
-e, --exclude exclude task and services with labels
-h, --help display help for command
```
# Concepts (/docs/concepts)
Hammerkit introduces a handful of terms that are easy to mix up at first. This page
defines each one in a line or two and links to the page that covers it in depth.
The example below uses most of them together.
```yaml title=".hammerkit.yaml"
envs:
NODE_VERSION: '22'
tasks:
install:
image: node:$NODE_VERSION-alpine # has an image -> runs in a container
src:
- package.json
- package-lock.json
generates:
- node_modules
cmds:
- npm ci
test:
image: node:$NODE_VERSION-alpine
deps: [install] # another task must run first
needs: [postgres] # a service must be running first
src:
- src
cmds:
- npm test
services:
postgres:
image: postgres:16-alpine
healthcheck:
cmd: "pg_isready -U postgres"
ports:
- 5432
```
## Task vs. service [#task-vs-service]
* A **[task](/docs/task)** is a piece of work that runs and finishes — compile,
test, lint, deploy.
* A **[service](/docs/service)** is a long-running process a task depends on —
a database, a message broker, an API. Hammerkit starts services when a task
`needs` them and stops them when nothing needs them anymore.
## Local vs. container [#local-vs-container]
The single rule: **an `image` means the task runs in a container; no `image` means
it runs on your host.** Container tasks get their tools from the image (recommended
— nothing to install locally, identical everywhere); local tasks use the tools
installed on the machine. See [run a task in a container](/docs/task/container).
## `src`, `generates`, `mounts` [#src-generates-mounts]
* **`src`** — the input files/folders a task reads. They drive
[caching](/docs/task/caching): unchanged `src` lets a task be skipped.
* **`generates`** — the output files/folders a task produces. This is what gets
cached, [stored and restored](/docs/cli/store-restore).
* **`mounts`** — extra paths a *container* task needs that aren't sources or
outputs (config files, caches). See [container](/docs/task/container#adding-mounts).
## `deps` vs. `needs` [#deps-vs-needs]
* **`deps`** — task → task. The dependency tasks run (or are restored from cache)
first. See [dependencies](/docs/task/dependencies).
* **`needs`** — task → service. The needed services are started and become healthy
before the task runs. See [needs](/docs/task/needs).
## `references` vs. `includes` vs. `extend` [#references-vs-includes-vs-extend]
All three pull in definitions from other build files, but differently:
* **[`references`](/docs/build-file/references)** — point at another build file; its
tasks are usable as `prefix:taskName`, keeping *their own* directory as the
working directory.
* **[`includes`](/docs/build-file/includes)** — like references, but the included
tasks adopt the *including* file's working directory. Ideal for reusable task
templates across a monorepo.
* **[`extend`](/docs/task/extending)** — a per-task field that uses another task as a
base template and overrides only what differs.
## Cache vs. store/restore [#cache-vs-storerestore]
* **[Caching](/docs/task/caching)** is automatic: hammerkit skips a task whose `src`
is unchanged, and with a cache **backend** it can share results across machines.
* **[`store` / `restore`](/docs/cli/store-restore)** are explicit commands that move
generated outputs and cache state in and out of a directory — typically wired
into a CI provider's own cache step.
These overlap; the [CI caching guide](/docs/guides/ci-caching) explains which to reach
for.
# Release procedure (/docs/contribution/publish)
Releases are derived from commit messages. Nobody bumps a version or pushes a tag by hand.
## Commit messages [#commit-messages]
`master` is squash-merged, so the **pull request title** is the commit message. It follows
[Conventional Commits](https://www.conventionalcommits.org/): `: `.
| Type | Release |
| -------------------------------------------------------------- | --------------------------- |
| `feat` | minor (`1.11.3` → `1.12.0`) |
| `fix`, `perf` | patch (`1.11.3` → `1.11.4`) |
| `docs`, `refactor`, `test`, `build`, `ci`, `chore`, `style` | none |
| a `!` after the type (`feat!:`) or a `BREAKING CHANGE:` footer | **major** |
A title in another shape (`Add x`, `1.12.0: add x`) is not an error, it just does not release.
A breaking change releases a major version on its own, so it is only ever written on request.
AI agents working in this repository ask first and never add the marker themselves.
## What happens on a merge to master [#what-happens-on-a-merge-to-master]
[semantic-release](https://semantic-release.gitbook.io) (`.github/workflows/release.yaml`,
`.releaserc.json`) reads the commits since the last tag. When one of them releases, it:
1. bumps `package.json` and `package-lock.json` and adds the section to `CHANGELOG.md`, in a
`chore(release): x.y.z [skip ci]` commit on master,
2. tags `vX.Y.Z`,
3. starts the publish workflow (`.github/workflows/push.yaml`) for the tag: it creates the GitHub
release with the changelog section as notes and the binaries attached, publishes the Docker
Hub image and publishes to npm. For a stable release it then regenerates the
formula in the [Homebrew tap](https://github.com/no0dles/homebrew-hammerkit) from the
binaries' checksums and pushes it to the tap's `main`,
4. starts the website workflow (`.github/workflows/website.yaml`) on master, which rebuilds
[hammerkit.dev](https://hammerkit.dev). Its [changelog](/changelog) page is generated from
`CHANGELOG.md` (`website/scripts/changelog.mjs`), so it lists the new release without further
work. Releases before 1.12 have hand-written entries in `website/content/changelog`, which win
over a generated one for the same version.
## Setup and recovery [#setup-and-recovery]
* `master` must accept pushes from the workflow (the release commit and the tag).
* The Homebrew step needs the secret `HOMEBREW_TAP_TOKEN`, a fine-grained token with *Contents:
write* on `no0dles/homebrew-hammerkit` only. Without it the step is skipped and the formula is
updated by hand (`hammerkit.rb.tmpl` in the tap lists the placeholders).
* The repository's releases are immutable, so the release is created together with its binaries
and cannot be edited afterwards. If publishing fails before the release exists, fix the cause and
re-run it from the tag: *Actions → publish → Run workflow*, choosing the `vX.Y.Z` tag as the ref.
* A prerelease (`1.12.0-alpha.0`) is still cut by hand with `npm version prerelease --preid alpha`
and `git push origin --follow-tags`; the npm dist-tag follows the preid.
# Roadmap (/docs/contribution/roadmap)
The topics below are planned for upcoming releases, roughly in priority order. Each
one has a specification in the repository's
[`specs/`](https://github.com/no0dles/hammerkit/tree/master/specs) folder; the
order may change as real-world feedback comes in.
## Next [#next]
### Source exclusions [#source-exclusions]
Exclude paths from a task's `src` with `!` entries (`src: [src, '!src/**/*.md']`),
so a source folder can be narrowed without long globs. An excluded path is hidden
from container tasks too, so a task can't read what isn't part of its cache key.
### Container runtime options [#container-runtime-options]
Declare Docker runtime and security options such as `--shm-size` or a seccomp
profile on container tasks and services, for workloads like browsers or sandboxed
workers that need them.
## Later [#later]
* **Task resources** — CPU and memory requests/limits per task, enforced on Docker
and Kubernetes and used for resource-aware local scheduling instead of a flat
worker count.
* **Matrix tasks** — define a task once with a `matrix` of variants (Node versions,
architectures) and run one independently cached task per combination.
* **Service state snapshots** — cache the state a task leaves *inside* a service,
such as a seeded database, so it can be restored instead of recreated.
* **Multi-arch packaging** — let `hammerkit package` build and push multi-platform
images.
* **Distributed computing** — offload compute-intensive tasks to remote hardware
with very little configuration. Server registration (`hammerkit remote`) has
landed; see the [spec](https://github.com/no0dles/hammerkit/blob/master/specs/remote-compute/spec.md).
# Shipped [#shipped]
## Task catalog (1.22.0) [#task-catalog-1220]
Tested, versioned build files for common tools that a project includes straight from
git instead of copying them.
Shipped: [includes from a git repository](/docs/build-file/includes#include-from-a-git-repository)
with `with` inputs, and the [catalog](/docs/catalog) entries for npm, Yarn, pnpm, pip,
Go, Cargo, .NET, Docker, tsc, ESLint, Prettier, Jest and Helm.
## Agent-ready caching (1.7.0) [#agent-ready-caching-170]
Make one shared cache work across coding-agent sandboxes, developer workspaces and
CI, so work done in one place is never repeated in another.
Shipped in 1.7.0: cache keys independent of the checkout location, the `registry`
cache backend, [`cache pull` / `cache push`](/docs/cli/cache), read-only mode,
[cache retention](/docs/build-file/caches#retention), skipping
[dependencies of cached tasks](/docs/task/dependencies#dependencies-of-cached-tasks),
[task timeouts](/docs/task#timeouts), and [`explain`](/docs/cli/explain),
[`graph`](/docs/cli/graph), `run --dry-run` and the build summary. See the
[release blog](/docs/release-blog/release-1.7.0) and
[agents, workspaces and CI](/docs/guides/agents-and-ci).
## Distributed caching (1.6.0) [#distributed-caching-160]
Distribute the local cache, so other developers and CI can reuse already built
results.
Shipped in 1.6.0 as [pluggable cache backends](/docs/build-file/caches): declare a
`local` or `s3` cache backend and hammerkit pulls and pushes task results
automatically, sharing them between developers and CI runs.
## Run on Kubernetes (1.6.0) [#run-on-kubernetes-160]
Run the same build file on the local Docker daemon or on a Kubernetes cluster.
Shipped in 1.6.0 through [environments](/docs/task/kubernetes).
## Services (1.5.0) [#services-150]
Tasks sometimes require a service to run, for example a database for an API or an
integration test. The goal was to define those services and have hammerkit start
them for a task and stop them when they are no longer needed.
Implemented in 1.5.0 with [container services](/docs/service/container) and
[Kubernetes services](/docs/service/kubernetes).
## Platform requirements (1.5.0) [#platform-requirements-150]
Define local tasks that require a specific platform. The goal was to either skip
local tasks where the platform requirements are not met or run them on a suitable
machine.
Labels, available since 1.5.0, can be used to achieve this.
# FAQ & Troubleshooting (/docs/faq)
## Where did my `dist` go? My container task ran but the files aren't in my project. [#where-did-my-dist-go-my-container-task-ran-but-the-files-arent-in-my-project]
By default a container task's `generates` stay **inside the container volume** —
fast, and reused on the next run, but not copied back to your working directory.
Mark a generate with `export: true` to copy it back out:
```yaml title=".hammerkit.yaml"
tasks:
build:
image: node:24-alpine
generates:
- path: dist
export: true
cmds:
- npm run build
```
See [exporting generated files](/docs/task#exporting-generated-files).
## Why does my task run every time, even when nothing changed? [#why-does-my-task-run-every-time-even-when-nothing-changed]
A task can only be skipped if it declares `src` of its own **and** all of its
dependencies can also be skipped. A task with no `src` (or whose `src` matches no
file — check the warnings), or one that depends on such a task, runs every time —
hammerkit can't prove it's up to date. Give each task accurate `src`/`generates`.
See [caching](/docs/task/caching).
## Why did my task rebuild? / Why was it skipped? [#why-did-my-task-rebuild--why-was-it-skipped]
Run [`hammerkit explain `](/docs/cli/explain): it names the input that caused a
miss — a changed `src` file, a changed dependency, or a changed task definition
(`cmds`, `image`, `envs`, `mounts`, …). Editing a command or bumping an image
invalidates the cache just like a source change. To force a run regardless, use
`--cache none` or [`hammerkit clean`](/docs/cli/clean). See
[what invalidates a key](/docs/task/caching#what-gets-cached-and-what-invalidates-it).
## No cache hits in CI, even though the code didn't change. [#no-cache-hits-in-ci-even-though-the-code-didnt-change]
Almost always the `modify-date` method on a fresh checkout: `git clone` resets file
modification times on every run, so a `modify-date` task always looks changed. Use
the default `checksum` method in CI (it compares content, which is stable across
checkouts). See [checksum vs. modify-date](/docs/task/caching#checksum-vs-modify-date)
and the [CI caching guide](/docs/guides/ci-caching).
## `Cannot connect to the Docker daemon` / `docker: command not found`. [#cannot-connect-to-the-docker-daemon--docker-command-not-found]
A task with an `image` needs a running container engine. Verify with `docker info`.
If you don't want Docker for a step, remove its `image` so it runs on the host. See
[installation](/docs/installation).
## A task fails with `missing environment variable NAME`. [#a-task-fails-with-missing-environment-variable-name]
A `$NAME` reference in `envs` couldn't be resolved from the shell or a `.env` file.
Either export the variable, add it to `.env`, or give the `env` a literal value.
Hammerkit fails fast here on purpose so a build never runs with a silently-empty
variable. See [environment variables](/docs/build-file/environment-variables#precedence).
## My task hangs "waiting" for a service that's running. [#my-task-hangs-waiting-for-a-service-thats-running]
The service's [healthcheck](/docs/service/container#timing-and-failure) never exits
`0`, so hammerkit never considers it ready. After the readiness deadline
(`healthcheck.timeout`, default `20s`, or `HAMMERKIT_HEALTHCHECK_TIMEOUT`) the
service fails and the run stops with the command that never passed. Make the `cmd`
a real readiness probe (like `pg_isready`) and confirm it succeeds inside the
container; raise the `timeout` if the service is just slow to start.
## My container task can't reach the service. [#my-container-task-cant-reach-the-service]
A container task reaches a service by its **name** as the hostname on the service's
**container port** (e.g. `postgres:5432`) — not via a published `ports` entry, and
not via the `HAMMERKIT_*` variables (those are injected only for **local** tasks).
See [services & networking](/docs/guides/services-networking).
## Generated files are owned by `root` (Linux). [#generated-files-are-owned-by-root-linux]
On Linux a container task runs as your `uid:gid` and hammerkit `chown`s the working
directory and mounts to you, so new files come back owned by you. The `chown` is
not recursive, so root-owned files baked into the image are left as-is. On
macOS/Windows the Docker Desktop VM handles this. See
[file permission](/docs/task/container#file-permission).
## `No tasks found` (exit code 127). [#no-tasks-found-exit-code-127]
The task name you asked for doesn't exist in the resolved build file. Run
[`hammerkit ls`](/docs/cli/ls) to see the available task names (including
`prefix:name` ones from references/includes). For the full exit-code contract see
the [CLI reference](/docs/cli#exit-codes).
## Which file does hammerkit use as the build file? [#which-file-does-hammerkit-use-as-the-build-file]
It looks for `.hammerkit.yaml`, then `.hammerkit.yml`, then `build.yaml`, and uses
the first it finds (warning if several exist). Override with `--file `. See
the [build file](/docs/build-file).
# Getting started (/docs/getting-started)
## Prerequisites [#prerequisites]
Hammerkit itself only needs [Node.js](https://nodejs.org) (see [installation](/docs/installation)
for binary and Homebrew options that don't).
Tasks that declare an `image` run **inside a container**, so you also need a
running container engine — [Docker](https://docs.docker.com/get-docker/) or any
Docker-compatible daemon. Containers are the recommended way to run tasks: the
build then uses the exact same tools on every machine and in CI, instead of
whatever happens to be installed locally. Check the daemon is up with:
```bash
docker info
```
If that prints server information you're ready. If it errors, start Docker (or
remove the `image:` line from the example below to run on your host instead).
## Create your first build file [#create-your-first-build-file]
To get started with hammerkit, you will need to create your first build file. Build files are usually called `.hammerkit.yaml` and can be created from scratch manually or by using the init command.
```bash
hammerkit init
```
```bash
npx hammerkit init
```
If you created your build file with the init command, the build file will look like the example below.
```yaml title=".hammerkit.yaml"
envs: {}
tasks:
example:
image: alpine
cmds:
- echo it's Hammer Time!
```
This build file contains an example command which prints a statement to the console inside an alpine container.
The `image: alpine` line is what makes this task run in a container. Remove that
line and the task runs directly on your host instead — no Docker required. See
[run a task in a container](/docs/task/container) for the full rule.
## Run your first task [#run-your-first-task]
To run your first command, make sure you're in the same directory as the build file and run:
```bash
hammerkit example
```
```bash
npx hammerkit example
```
The first run pulls the `alpine` image if it isn't on your machine yet (later runs
reuse it), starts a container, executes the echo command `it's Hammer Time!` and
exits with code `0`. Run it a second time and it runs again: the task declares no
`src`, so hammerkit can't tell whether anything it reads has changed. Declare the
files a task reads under `src`, and an unchanged task is reported as cached and
skipped — that's [caching](/docs/task/caching) at work.
## Discover the tasks in a project [#discover-the-tasks-in-a-project]
When you clone a repository that already uses hammerkit, list everything it
defines with:
```bash
hammerkit ls
```
This prints every task and service with its image, sources, generated outputs and
dependencies. For the full command list and global flags use:
```bash
hammerkit --help
```
## Where to go next [#where-to-go-next]
* [Why hammerkit](/docs/why-hammerkit) — what it does that Make / npm scripts / Taskfile don't.
* [Concepts](/docs/concepts) — local vs. container, `src`/`generates`/`mounts`, `deps` vs. `needs`.
* [Tutorials](/docs/tutorials) — build, test and cache a project in Node.js, Python, Go, Rust or Java.
* [Task](/docs/task) — sources, outputs, dependencies and caching in depth.
* The runnable [`examples/`](https://github.com/no0dles/hammerkit/tree/master/examples) folder in the repository.
## Conclusion [#conclusion]
In this short introduction you created your first build file and executed your first task. The build file and the task in this example were very basic; to see the full functionality of hammerkit, take a look at the rest of the documentation.
# Agents, workspaces and CI (/docs/guides/agents-and-ci)
Hammerkit's cache key is a hash of a task's inputs — command, image, sources and
dependencies — computed relative to the project root. The same commit therefore
produces the same keys in a coding agent's sandbox, on a CI runner and on a
laptop. Put those results in a shared remote cache and:
* **CI does not repeat what an agent already ran** — an agent that ran the e2e
suite on a commit leaves the result behind; CI restores it and skips the job,
including starting its database.
* **New workspaces start warm** — a fresh sandbox pulls the dependency install
and build outputs instead of rebuilding them.
* **Less compute overall** — each unique input is built once, wherever it ran
first.
## 1. Declare a remote cache [#1-declare-a-remote-cache]
Any OCI registry works; use the one you already push images to. (An
[S3-compatible bucket](/docs/build-file/caches#s3) works the same way.)
```yaml title=".hammerkit.yaml"
caches:
shared:
method: checksum
backend:
type: registry
repository: ghcr.io/my-org/hammerkit-cache
services:
db:
image: postgres:16-alpine
envs:
POSTGRES_PASSWORD: postgres
ports:
- :5432 # random host port: parallel workspaces never collide
healthcheck:
cmd: pg_isready -U postgres
tasks:
install:
image: node:24-alpine
src: [package.json, package-lock.json]
generates: [node_modules]
cmds: [npm ci]
build:
image: node:24-alpine
deps: [install]
src: [src]
generates: [dist]
cmds: [npm run build]
e2e:
image: cypress/included:13.15.0
deps: [build]
needs: [db]
src: [cypress]
cmds: [cypress run]
```
Tasks keep caching to the machine-local `default` cache; the `shared` cache is
only touched by `cache pull` / `cache push`, so builds never wait on the network.
Use **container tasks** for anything you want to share: a local task includes the
host OS in its cache key, so a macOS laptop and a Linux runner would not share it.
## 2. In the agent sandbox [#2-in-the-agent-sandbox]
Set up the workspace from the cache, work, then publish the results:
```bash
docker login ghcr.io # or mount a docker config with a token
hammerkit cache pull --remote shared # warm workspace: deps + build restored
hammerkit run e2e # only what changed runs
hammerkit cache push e2e --remote shared # leave the results for CI and others
```
## 3. In CI [#3-in-ci]
```bash
hammerkit cache pull --remote shared
hammerkit run e2e # cache hits execute nothing
hammerkit cache push --remote shared # publish anything CI had to build
```
A task whose inputs were already run by an agent completes as a cache hit in
milliseconds, and services it needs are not started.
## Who may write to the cache? [#who-may-write-to-the-cache]
Pushing is trusting: a cache hit means "this exact input already produced this
result", and CI will not re-check it. Decide per runner:
| Policy | Agents | CI | Trade-off |
| ------------------------ | ----------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------- |
| **Agents write** (above) | `cache push` | reads, pushes misses | Maximum reuse; CI trusts results produced in sandboxes. |
| **CI writes only** | read-only token + `HAMMERKIT_CACHE_READ_ONLY=1` | reads + pushes | Agents still start warm; CI re-runs what agents ran. |
| **Split** | push to `agents` cache | pulls `agents` for PR checks, only pushes to `shared` on `main` | Agent results speed up PR feedback, never the release path. |
Enforce the policy with registry credentials (a pull-only token for read-only
runners); `HAMMERKIT_CACHE_READ_ONLY=1` makes hammerkit refuse pushes as well,
so a misconfigured runner fails loudly instead of writing.
## Checking why something rebuilt [#checking-why-something-rebuilt]
```bash
hammerkit explain e2e # hit or miss, and which input changed
hammerkit run --explain # the miss cause next to every rebuilt task
hammerkit run --dry-run # predicted hits/misses without running anything
```
# CI through the server (/docs/guides/agents/github-actions)
Experimental. The commands and settings can still change between releases. The server side
of this guide was tested on a GKE server with a stand-in for the CI job's identity; the
workflow of step 4 has not run on GitHub's runners yet.
A CI job usually builds, tests and deploys on its runner, with the credentials of the cloud
and the registry in the job. Whoever can change a workflow or a build script can read them.
With a hammerkit server the job only says what to run:
1. The job signs in to the server with the token GitHub gives every job, which names the
repository and the branch it runs for. Nothing else is stored in GitHub.
2. [`hammerkit remote run`](/docs/cli/remote#running-tasks) asks the server to run a task of the
published build file at the job's commit. The server checks the commit out itself, runs
the task in its cluster and streams the output back. The job fails when the task does.
3. A task that reads a secret reads it on the server, as a service
[account](/docs/remote-compute/accounts) of the server. A production deploy account is one the
job may only request: the run waits until a person who uses the account approves it.
This guide adds that to a server set up with the [Kubernetes guide](/docs/guides/agents/kubernetes).
Each `server.yaml` block below goes into the config of that guide's ConfigMap, next to the keys
it already has, and the shell variables are those of its step 1.
## What you need [#what-you-need]
* The server of the [Kubernetes guide](/docs/guides/agents/kubernetes), reachable from the
internet over [HTTPS](/docs/guides/agents/kubernetes#reaching-the-server-over-https): GitHub's
runners can not use a port-forward. The examples write it as `https://hammerkit.example.com`.
* The repository's build file, with the tasks the jobs run. Here `test`, and `deploy`, which
reads a secret and reaches the deploy target, written as `deploy.example.com`:
```yaml title=".hammerkit.yaml"
tasks:
test:
image: node:22-alpine
src: [src/**, test/**]
cmds: [node --test]
deploy:
image: node:22-alpine
deps: [test]
secrets:
- { from: 'gsm:deploy-token', account: deploy, env: DEPLOY_TOKEN }
# a task that reads a secret gets no open network on this server, see step 3
network: { allow: [deploy.example.com] }
# it reaches the target through the proxy; Node's fetch reads HTTPS_PROXY only with this,
# from Node 22.21 and 24.5 (22 warns that it is experimental)
envs: { NODE_USE_ENV_PROXY: '1' }
cmds: [node deploy.js]
```
* The repository's id, which stays the same when the repository is renamed or moved:
`curl -sS https://api.github.com/repos/acme/shop | jq .id` (with a token for a private one,
or `gh api repos/acme/shop --jq .id`).
## 1. Let the server take GitHub's tokens [#1-let-the-server-take-githubs-tokens]
Add GitHub's issuer to `issuers` in the server's config (the ConfigMap of step 2 of the
Kubernetes guide), next to Google's:
```yaml title="server.yaml"
issuers:
- { issuer: https://accounts.google.com, audience: hammerkit }
- { issuer: https://token.actions.githubusercontent.com, audience: https://hammerkit.example.com }
```
The `audience` is what a job asks its token for. The address of the server makes it a token
for this server alone: a token a job got for another service, or another hammerkit server, is
refused here. The server checks the token's signature with GitHub's keys and takes the claims
GitHub puts in it, such as `repository_id` and `ref`, as they are. See
[GitHub Actions](/docs/remote-compute/authentication#github-actions).
## 2. Say what the jobs may do [#2-say-what-the-jobs-may-do]
```yaml title="server.yaml"
entitlements:
# the ci workflow, on a push to main: publish the build file, run it, and ask for deploys
- claims:
iss: https://token.actions.githubusercontent.com
match:
repository_id: '123456789'
ref: refs/heads/main
event_name: push
job_workflow_ref: acme/shop/.github/workflows/ci.yaml@refs/heads/main
repos: [github.com/acme/shop]
builds: publish
accounts: { deploy: request }
# every job of the repository, pull requests too: run what is published
- claims:
iss: https://token.actions.githubusercontent.com
match: { repository_id: '123456789' }
repos: [github.com/acme/shop]
builds: run
# who approves deploys, and new versions of what they run: a person, or a group
- subject: { iss: https://accounts.google.com, sub: '' }
repos: [github.com/acme/shop]
builds: publish
accounts: { deploy: use }
```
A claim matches when it is exactly the value written, and every claim of `match` has to.
`ref: refs/heads/main` alone would be any job on `main`: a scheduled one, one started by hand,
or one of another workflow file with `id-token: write`. `event_name` and `job_workflow_ref` keep
it to the `ci` workflow of step 4 on a push. Entitlements add up: that job matches both, and
gets the higher level.
That job publishes. What the build file's tasks run is then what the `ci` workflow on `main`
publishes, and not whatever somebody with a publisher account uploads. Once this works, take
`builds: publish` away from the people and service accounts that only published, such as the
publisher of the Kubernetes guide.
The server still takes the files the job uploads as they are: any step of that job, an action
or an install script of a package, could upload another build file. Tasks that read secrets as
`deploy` are protected by the approval of
[When the build file changes](#when-the-build-file-changes); tasks without secrets are
published at once.
The approver keeps `builds: publish`: a new version that changes what `deploy` runs needs their
approval (see [When the build file changes](#when-the-build-file-changes)), and approving a
version takes `publish`. It also lets them upload a version themselves, so keep it to the
few people who approve deploys.
## 3. Keep the deploy account to main [#3-keep-the-deploy-account-to-main]
The `deploy` account is set up like any account that reads secrets on the cluster, see
[Secrets in the Job](/docs/remote-compute/backend#secrets-in-the-job). Two lines keep it to
reviewed code:
```yaml title="server.yaml"
accounts:
# next to github
deploy:
description: production deploys
source: [ref] # only code the server checked out itself, never an upload or a sandbox
refs: [main] # and only from main
providers:
gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /var/run/hammerkit/account/gcp.json } }
kubernetes:
serviceAccount: deploy # a ServiceAccount of the server's namespace, trusted by your cloud
audience: $AUDIENCE # what the pool of the Kubernetes guide's step 1 accepts
secret: deploy-federation
```
The Kubernetes guide's server gives a task that reads a secret no network at all
(`runs.secretNetwork: none`), and refuses `deploy`, which has to reach its target. Allow the
target there, and give Secret Manager's client the hosts it reads the secret from:
```yaml title="server.yaml"
runs:
env: { allow: [PATH] }
args: declared
# what a task that reads a secret may reach: nothing, or the deploy target
secretNetwork: { allow: [deploy.example.com] }
secretProviders:
github-app: { command: [hammerkit, github-app-token, '{{ref}}'] }
# the project of the secrets: a federation config names none
gsm: { command: [gcloud, secrets, versions, access, latest, '--secret={{ref}}', '--project=$PROJECT'] }
backend:
type: kubernetes # its other keys as in the Kubernetes guide, and providers
providers:
gsm:
image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
allow: [sts.googleapis.com, secretmanager.googleapis.com]
```
`secretNetwork` is a ceiling for every task that reads a secret: each may name
`deploy.example.com` now, or say `network: none`. One that leaves `network` out asks for an open
network, and a version with it, or with another host, is refused at publish:
`deploy: asks for network open, more than this server allows to what reads the secret
DEPLOY_TOKEN (runs.secretNetwork: allow deploy.example.com)`. The command of `deploy`
does not reach Secret Manager: its egress proxy lets only the container that reads the secret
through to `providers.gsm.allow`, see
[Secrets of a task with a limited network](/docs/remote-compute/backend#secrets-of-a-task-with-a-limited-network).
The server makes a token of the account's ServiceAccount for each Job of `deploy`, and reads
its Secret, so it needs both, by name, in its namespace:
```bash
kubectl apply -f - < deploy-gcp.json <-
curl -sSf -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN"
"$ACTIONS_ID_TOKEN_REQUEST_URL&audience=https://hammerkit.example.com" | jq -r .value
# the branch and commit the server checks out: a pull request's head, or the pushed commit
REF: ${{ github.head_ref || github.ref_name }}
SHA: ${{ github.event.pull_request.head.sha || github.sha }}
# a deploy that waits for its approval holds the job, an hour at most
timeout-minutes: 75
concurrency:
group: ci-${{ github.ref }}
# a newer push to a pull request stops the older job, and its run on the server
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
steps:
- uses: actions/checkout@v4
with: { ref: '${{ env.SHA }}' }
- uses: actions/setup-node@v4
with: { node-version: 24 }
- run: npm install -g hammerkit@2.0.0
- run: |
hammerkit remote add hammerkit https://hammerkit.example.com --issuer https://token.actions.githubusercontent.com
hammerkit remote use hammerkit
- if: github.event_name == 'push'
run: hammerkit remote publish
- run: hammerkit remote run test --ref "$REF" --commit "$SHA"
- if: github.event_name == 'push'
run: hammerkit remote run deploy --ref "$REF" --commit "$SHA"
```
What each step does:
* **The token.** `id-token: write` lets the job ask GitHub for a token. hammerkit runs
`HAMMERKIT_TOKEN_COMMAND` when it first calls the server, and again when the server no
longer accepts the token, so a long run outlives one token.
* **`remote add`** registers the server with GitHub as the issuer of the job's tokens. It
checks that the server takes them.
* **`remote publish`**, on `main` only, publishes the build file of the commit. A build file
that did not change since the last push is `already published`.
* **`REF` and `SHA`** come from the event: a pull request's job would check out a merge commit
the server does not know, so the job checks out the head of the branch and names it. They go
to the commands as variables, not into the script: a branch name is chosen by whoever opens
the pull request, and may hold shell.
* **`remote run test`** runs `test` at that commit. It names the build file by the hash of the
one checked out. A ref that moved since, after a newer push, is refused, and the newer push's
job runs it.
* **A pull request that changes the build file** fails at `remote run` with
`the build file here is not published on hammerkit (sha256:…), publish it first`: its job may only run
what is published, and its version is published once it is merged. Read that failure as
"the tasks changed", and look at what `main`'s job does after the merge.
* **`remote run deploy`** runs `deploy` from `main` at the pushed commit.
* **`concurrency`** stops an older job of a pull request when a newer push arrives, which
cancels its run on the server. Jobs on `main` are not stopped, so a deploy is never cut off
halfway: a newer push waits until the older deploy is approved or rejected.
The job exits as the task did, with its exit code, and prints its output as it comes. The
runner holds no cloud credentials, no registry login and no deploy key: nothing a changed
workflow could print.
## 5. Approve a deploy [#5-approve-a-deploy]
The job of a push to `main` waits at `deploy`:
```
run_Qx3k9ZpL2mW7vN4a on hammerkit: deploy from main at 4be1a07c...
waiting until somebody who uses deploy approves it, before 2026-10-24T10:41:00.000Z: hammerkit remote approve run_Qx3k9ZpL2mW7vN4a
```
The approver looks at it and lets it go, signed in as themselves:
```bash
hammerkit remote approvals hammerkit
hammerkit remote show run_Qx3k9ZpL2mW7vN4a hammerkit
hammerkit remote approve run_Qx3k9ZpL2mW7vN4a hammerkit
```
`remote show` lists every command the run executes, its image and the secrets it reads, so the
approver does not need to read the build file. The job then prints the deploy's output and
ends. A run nobody approves within an hour (`approvals.timeout`) is rejected, and the job
fails. See [Approving runs](/docs/cli/remote#approving-runs).
## When the build file changes [#when-the-build-file-changes]
The job of `main` may only request `deploy`. A version of the build file that reads secrets as
`deploy` is therefore not published by it but proposed: somebody who uses `deploy` and may
publish approves it first, so a change to what the deploy runs takes a second person. The job prints
`proposed to hammerkit`, and the runs after it fail with
`the build file here is not published on hammerkit (sha256:…), publish it first`. The approver:
```bash
hammerkit remote proposals hammerkit
hammerkit remote show sha256:9f2c… hammerkit # every file of the new version
hammerkit remote approve sha256:9f2c… hammerkit
```
Then re-run the job. A version that changes only tasks without secrets is published at once.
## Checks without waiting [#checks-without-waiting]
The `test` step above holds the runner while the server runs the tests. When the agent or the
person who pushed already ran them on the server, the job only has to verify that they passed
for the files of its commit, which takes seconds: see
[Checks that ran before CI](/docs/guides/agents/verified-checks).
## What is not covered yet [#what-is-not-covered-yet]
* The server does not watch the repository itself: a job still starts each run, so GitHub
Actions is still needed.
* Image builds need a Docker daemon, which the cluster does not offer to tasks.
* Pull requests from forks get no token from GitHub, so their jobs can not use the server.
# Service accounts on Google Cloud (/docs/guides/agents/google-accounts)
An agent never signs in to the server as you. It signs in as a **service account** of its own,
and the server's [entitlements](/docs/remote-compute/entitlements) decide what that account
may do. The guides use two accounts:
| Account | Signs in to | Entitlement on the server |
| --------------------- | --------------------------------------------------- | ------------------------------------------------ |
| `hammerkit-agent` | start sandboxes, run tasks, push `agent/…` branches | `builds: run` and `sandboxes` |
| `hammerkit-publisher` | publish build files (`hammerkit remote publish`) | `builds: publish` (includes run), no `sandboxes` |
Neither account gets a key file or a role in your project. Instead, you are allowed to
**impersonate** them: `gcloud` asks Google for a short-lived token of the account on your
behalf, and that is the token hammerkit sends to the server. This page sets that up.
If you followed [Server on Kubernetes](/docs/guides/agents/kubernetes), its step 1 already
created both accounts for you, named `$NAME-agent` and `$NAME-publisher`; skip to
[Let your team act as them](#3-let-your-team-act-as-them)
to add more people.
## What you need [#what-you-need]
* A Google Cloud project, and `gcloud` signed in as a user who may create service accounts in
it (for example the `Service Account Admin` role).
```bash
PROJECT=my-project
ME=$(gcloud config get account)
```
## 1. Create the accounts [#1-create-the-accounts]
```bash
gcloud services enable iam.googleapis.com iamcredentials.googleapis.com --project $PROJECT
gcloud iam service-accounts create hammerkit-agent --project $PROJECT --display-name "hammerkit agent"
gcloud iam service-accounts create hammerkit-publisher --project $PROJECT --display-name "hammerkit publisher"
```
`iamcredentials.googleapis.com` is the API that hands out tokens of a service account to
someone allowed to impersonate it.
## 2. Allow yourself to act as them [#2-allow-yourself-to-act-as-them]
Impersonating an account needs the role **Service Account Token Creator** on that account.
Grant it to yourself, on these two accounts only:
```bash
for who in agent publisher; do
gcloud iam service-accounts add-iam-policy-binding hammerkit-$who@$PROJECT.iam.gserviceaccount.com \
--project $PROJECT --role roles/iam.serviceAccountTokenCreator --member user:$ME
done
```
The narrower role Service Account OpenID Token Creator is not enough: `gcloud` with
`--impersonate-service-account` also asks for an access token of the account.
Signed in to `gcloud` as a service account yourself, for example in CI, write
`--member serviceAccount:…` instead of `user:…`. A new binding can take a minute before it
works.
## 3. Let your team act as them [#3-let-your-team-act-as-them]
Everybody who starts an agent needs the same role on `hammerkit-agent`. Give it to each person,
or to a Google group:
```bash
gcloud iam service-accounts add-iam-policy-binding hammerkit-agent@$PROJECT.iam.gserviceaccount.com \
--project $PROJECT --role roles/iam.serviceAccountTokenCreator --member group:dev-team@example.com
```
Keep `hammerkit-publisher` to the people or the CI job that publish build files: whoever can
act as it decides what the agent's commands run in.
## 4. Check that a token comes back [#4-check-that-a-token-comes-back]
```bash
gcloud auth print-identity-token --audiences=hammerkit --include-email \
--impersonate-service-account=hammerkit-agent@$PROJECT.iam.gserviceaccount.com
```
This is the command the tutorials put in `HAMMERKIT_TOKEN_COMMAND`, or that you register with a
server with [`remote add --token-command`](/docs/cli/remote#a-token-command-per-server). It prints a long string
starting with `eyJ`, a token that is valid for an hour, and a warning on stderr that it
impersonates the account, which is expected. `--audiences=hammerkit` has to match
the `audience` of `issuers` in the server's config.
A `Permission 'iam.serviceAccounts.getAccessToken' denied` (or `getOpenIdToken`) error means step 2 or 3 is missing for
your user, or has not taken effect yet.
## 5. Give the server the accounts' ids [#5-give-the-server-the-accounts-ids]
The server recognizes an account by its unique id, the `sub` of its tokens, not by its email:
```bash
gcloud iam service-accounts describe hammerkit-agent@$PROJECT.iam.gserviceaccount.com \
--project $PROJECT --format='value(uniqueId)'
gcloud iam service-accounts describe hammerkit-publisher@$PROJECT.iam.gserviceaccount.com \
--project $PROJECT --format='value(uniqueId)'
```
Each prints a number such as `104857204856125339187`. Whoever runs the server puts them in the
entitlements of its config, here for the repository `github.com/acme/shop`:
```yaml title="server.yaml"
issuers:
- { issuer: https://accounts.google.com, audience: hammerkit }
entitlements:
- subject: { iss: https://accounts.google.com, sub: '' }
repos: [github.com/acme/shop]
builds: publish
- subject: { iss: https://accounts.google.com, sub: '' }
repos: [github.com/acme/shop]
builds: run
sandboxes:
- repo: github.com/acme/shop
branches: { main: pull, 'agent/**': push } # start from main, push to agent/…
```
Then follow [pi on a sandbox](/docs/guides/agents/pi) or
[opencode on a sandbox](/docs/guides/agents/opencode).
# Agents on a server (/docs/guides/agents)
Experimental. The tools and settings can still change between releases.
A coding agent on a hammerkit server works on a [sandbox](/docs/remote-compute/sandboxes): a
checkout of a branch that stays on the server. The agent runs where you start it, on a laptop
or in CI, but every file it reads or edits and every command it runs is on the server, in the
images, network and accounts of a published build file. When it is done it pushes a branch
for review.
These guides set it up end to end, the way it was run to check them:
**A server.** [On Kubernetes (GKE)](/docs/guides/agents/kubernetes), where every command of
an agent is a Job in a namespace of its own, or [on one machine](/docs/remote-compute#setting-up-a-server).
**A build file with a shell task**, published to the server (below).
**An identity for the agent**: a service account of your identity provider with an
[entitlement](/docs/remote-compute/entitlements) that may start sandboxes from `main` and
push to `agent/**`.
**A harness**: [pi](/docs/guides/agents/pi), [opencode](/docs/guides/agents/opencode), or any
MCP client with [`hammerkit remote mcp`](/docs/remote-compute/agents#hammerkit-remote-mcp).
## The build file [#the-build-file]
The agent's commands run as the arguments of a [shell task](/docs/remote-compute/agents#the-shell-task),
so the build file decides the image, the network and the accounts of everything the agent
runs. Other tasks, such as `test`, are there for the agent to run as they are.
```yaml title=".hammerkit.yaml"
tasks:
sh:
image: node:22-alpine
mounts: [.]
args: true
cmds: [sh -c]
network: { allow: [registry.npmjs.org] }
test:
image: node:22-alpine
src: [src/**, test/**]
args: true
cmds: [node --test]
```
`mounts: [.]` puts the files of the sandbox into the container, writable, and `args: true`
marks the tasks that take arguments, the agent's commands and a test file: a server with
[`runs.args: declared`](/docs/remote-compute/runs#arguments) passes them to no other.
The task reads no secrets: a task that does takes no arguments on a server. Publish it with
[`hammerkit remote publish`](/docs/cli/remote#publishing-a-build-file) as an identity with
`builds: publish`; it prints the hash of the version. The agent then names the repository,
the branch and that hash:
| Variable | Example | What it names |
| ------------------------- | ------------------------------------ | ---------------------------------------------------------- |
| `HAMMERKIT_SERVER` | `hammerkit` | The server, as `hammerkit remote add` registered it. |
| `HAMMERKIT_REPO` | `github.com/acme/shop` | The repository of the sandbox. |
| `HAMMERKIT_REF` | `main` | The branch it starts from. |
| `HAMMERKIT_BUILD` | `sha256:9f2c…` | The published version of the build file. |
| `HAMMERKIT_TOKEN_COMMAND` | `gcloud auth print-identity-token …` | Prints a short-lived token of the agent's service account. |
A run names exactly the version it runs, so nothing follows a newer one: when the build file
changes, publish it again and set the new hash. See [Settings](/docs/remote-compute/agents#settings) for the rest, such as `HAMMERKIT_SANDBOX`
to come back to a sandbox with work in progress.
## On the agent's machine [#on-the-agents-machine]
The server is registered per user, in `~/.hammerkit/config.yaml`. Register it as the user
that starts the agent, with the issuer of the agent's tokens:
```bash
npx hammerkit remote add hammerkit https://hammerkit.example.com --issuer https://accounts.google.com
```
The token command runs again whenever the server stops accepting a token, so a session
outlives one token. It is the agent's own identity, never a person's sign-in.
## What it is not yet [#what-it-is-not-yet]
* A dev server or another service on the sandbox for the agent to send requests to.
* Commands that read input, or keep running in the background.
## CI on the same server [#ci-on-the-same-server]
CI jobs can use the server too, with nothing but their own token: see
[CI through the server](/docs/guides/agents/github-actions). A job does not even have to wait
for the tests when the agent checked them before its push: see
[Checks that ran before CI](/docs/guides/agents/verified-checks).
# Server on Kubernetes (GKE) (/docs/guides/agents/kubernetes)
Experimental, like the server. This guide was run as written on a GKE Autopilot cluster
(Kubernetes 1.36) in `europe-west4`.
You end up with a hammerkit server in its own namespace of your cluster. Agents work on
[sandboxes](/docs/remote-compute/sandboxes) there: every command they run is a Job in a
namespace of its own, and their work leaves as a branch pushed by your GitHub App. No key is
stored in the cluster, and no image has to be built: the server runs the released
`no0dles/hammerkit` image.
## What you need [#what-you-need]
* A GKE cluster that enforces `NetworkPolicy`. Autopilot does; on a Standard cluster turn on
Dataplane V2.
* `gcloud`, `kubectl` and `jq`, signed in with the right to create ClusterRoles.
* The hammerkit CLI of the server's version, on Node.js 24 or later:
`npm install -g hammerkit@$VERSION` with `VERSION` from below. On an older Node.js npm
warns `EBADENGINE`, and with a range instead of an exact version it quietly installs an
old hammerkit without the commands this guide uses.
* A GitHub App installed on your repository, with **Contents: read and write**, **Metadata:
read** and nothing else. Download a private key for it (`app.pem`).
* A build file with a shell task in the repository, see
[The build file](/docs/guides/agents#the-build-file).
Set these once; every command below uses them:
```bash
PROJECT=my-project
REGION=europe-west4
REPO=acme/shop # your GitHub repository
GITHUB_APP_ID=123456 # on the App's settings page
VERSION=2.0.0 # one hammerkit version for the server, the CLI and the agent
NAME=hammerkit # namespace and account names; another server on the cluster needs another
PROJECT_NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)')
```
## 1. Google: the App key and the accounts [#1-google-the-app-key-and-the-accounts]
The server reads the App's key from Secret Manager as its Kubernetes ServiceAccount. For that,
an identity pool trusts the cluster's tokens, and only the server's ServiceAccount may read the
key. Two Google service accounts give the agent and the publisher an identity to sign in with.
```bash
gcloud services enable sts.googleapis.com secretmanager.googleapis.com iam.googleapis.com \
iamcredentials.googleapis.com --project $PROJECT
# let the cluster's ServiceAccounts sign in to Google
gcloud iam workload-identity-pools create $NAME --location global --project $PROJECT
gcloud iam workload-identity-pools providers create-oidc cluster --workload-identity-pool $NAME \
--location global --project $PROJECT --allowed-audiences $NAME \
--issuer-uri $(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer) \
--attribute-mapping google.subject=assertion.sub
POOL=projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$NAME
# the App's key, readable by the server only
gcloud secrets create $NAME-github-app-key --project $PROJECT --data-file app.pem
gcloud secrets add-iam-policy-binding $NAME-github-app-key --project $PROJECT \
--role roles/secretmanager.secretAccessor \
--member principal://iam.googleapis.com/$POOL/subject/system:serviceaccount:$NAME:hammerkit
# identities for the agent and the publisher, which you may sign in as
ME=user:$(gcloud config get account) # as a service account: ME=serviceAccount:; empty: set it yourself
for who in agent publisher; do
gcloud iam service-accounts create $NAME-$who --project $PROJECT
gcloud iam service-accounts add-iam-policy-binding $NAME-$who@$PROJECT.iam.gserviceaccount.com \
--project $PROJECT --role roles/iam.serviceAccountTokenCreator --member $ME
done
# what the next steps use
AUDIENCE=$NAME
KEY_SECRET=projects/$PROJECT/secrets/$NAME-github-app-key/versions/latest
AGENT=$NAME-agent@$PROJECT.iam.gserviceaccount.com
PUBLISHER=$NAME-publisher@$PROJECT.iam.gserviceaccount.com
AGENT_ID=$(gcloud iam service-accounts describe $AGENT --project $PROJECT --format='value(uniqueId)')
PUBLISHER_ID=$(gcloud iam service-accounts describe $PUBLISHER --project $PROJECT --format='value(uniqueId)')
```
Delete `app.pem` now. The two service accounts hold no role in the project: only the server
reads their tokens. A new IAM binding can take a few minutes before it works. `AUDIENCE` is
what the pool accepts from the server's pod; it is unrelated to the `audience: hammerkit` of
the sign-in tokens in step 2.
**Did someone else do this step?** Ask them for the values the last block sets, and set them
in your shell: `POOL` (the pool's full name, as above), `AUDIENCE` (the audience its provider
allows), `KEY_SECRET` (the Secret Manager version of the App's key), `AGENT` and `PUBLISHER`
(the emails of the two service accounts) and `AGENT_ID` and `PUBLISHER_ID` (their unique ids).
The pool's provider has to be named `cluster`, or change `providers/cluster` in `gcp.json`
of step 2. They also have to let the server's ServiceAccount,
`system:serviceaccount:$NAME:hammerkit`, read the key, and let you act as the two accounts.
[Service accounts on Google Cloud](/docs/guides/agents/google-accounts) explains these two
accounts, and how to let your team act as the agent.
## 2. Deploy the server [#2-deploy-the-server]
One manifest holds everything the server needs in the cluster: its namespace and
ServiceAccount, its rights, its config, a disk and the Deployment. It uses the variables of
step 1; the first line stops when one of them is not set, for example in a new terminal.
```bash
: "${NAME:?}" "${REPO:?}" "${GITHUB_APP_ID:?}" "${VERSION:?}" "${POOL:?}" "${AUDIENCE:?}" \
"${KEY_SECRET:?}" "${AGENT_ID:?}" "${PUBLISHER_ID:?}"
kubectl apply -f - < /dev/null && echo "the server can read $REPO"
```
When it fails, it says which step did:
* `Secret Manager did not give …: 403` means the server's ServiceAccount may not read the key:
the binding of step 1 is missing, names another namespace, or is not in effect yet.
* `Google STS did not exchange the token …` means the pool or its provider does not match:
compare `POOL` and `AUDIENCE` with step 1.
* `the GitHub App … is not installed on …` means the App is not installed on the repository.
An agent whose sandbox can not be created gets the same reason from the server.
## 3. Connect and publish [#3-connect-and-publish]
Forward the server's port to your machine and register it:
```bash
while true; do kubectl -n $NAME port-forward svc/hammerkit 8480:8480; sleep 1; done &
hammerkit remote add $NAME http://127.0.0.1:8480 --issuer https://accounts.google.com
```
`kubectl port-forward` ends when the server's pod is replaced, for example after a
`kubectl apply`, and now and then when its connection drops; the loop starts it again. When
it prints the same error every second, for example because port 8480 is taken or the cluster
can not be reached, stop it with `kill %1` and fix that first.
Before you publish, check that the build file's shell task has `mounts: [.]`, `args: true`
and `cmds: [sh -c]`, as in [The build file](/docs/guides/agents#the-build-file). Without the
mount, the agent's commands run in an empty directory; without `args: true` this server refuses
them. Then publish the build file from a
checkout of the repository, signed in as the publisher:
```bash
HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \
--impersonate-service-account=$PUBLISHER" \
hammerkit remote publish $NAME
```
It prints the hash of the published version, `sha256:` and 64 hex digits. The agent names the
build file by that hash, so a later publish never changes what a running agent uses.
## 4. Start an agent [#4-start-an-agent]
Follow [pi](/docs/guides/agents/pi) or [opencode](/docs/guides/agents/opencode), with the
server registered as above, the build file's hash and this token command:
```bash
export HAMMERKIT_BUILD=sha256:9f2c… # the hash hammerkit remote publish printed
export HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \
--impersonate-service-account=$AGENT"
```
The tutorials write the accounts as `hammerkit-agent@my-project…` and
`hammerkit-publisher@my-project…`: use `$AGENT` and `$PUBLISHER` from step 1 instead.
The agent's branches show up in the repository as `agent/…`, pushed by your GitHub App.
## Reaching the server over HTTPS [#reaching-the-server-over-https]
`kubectl port-forward` needs access to the cluster on every machine. A team, or a CI job such
as [GitHub Actions](/docs/guides/agents/github-actions), reaches the server through an HTTPS
load balancer instead: a GKE Ingress with a Google-managed certificate for a domain of yours.
The config already expects TLS to end in front of the server (`behindTlsProxy`), and the
server takes nothing without a token of one of its issuers, so the address can be public.
Reserve an address and point your domain at it, with an `A` record at your DNS provider:
```bash
DOMAIN=hammerkit.example.com # a name in a zone you control
gcloud compute addresses create $NAME --global --project $PROJECT
gcloud compute addresses describe $NAME --global --project $PROJECT --format='value(address)'
```
Then the certificate and the load balancer:
```bash
: "${NAME:?}" "${DOMAIN:?}"
kubectl -n $NAME annotate --overwrite svc hammerkit \
cloud.google.com/neg='{"ingress": true}' cloud.google.com/backend-config='{"default": "hammerkit"}'
kubectl apply -f - < --region $REGION --format='value(clusterIpv4Cidr,servicesIpv4Cidr)'` shows your ranges. Without the right to
describe the cluster, `kubectl get svc -A -o jsonpath='{..clusterIP}'` shows service addresses
that have to be in the denied ranges. An internal address tasks need goes in `allowedEgress`.
* **`networkPolicy: enforced`.** The agent's commands only reach what `sandboxes.network`
allows, here npm. See [Network](/docs/remote-compute/backend#network).
* **`runs.args: declared`.** Arguments reach only the tasks that say `args: true`, so an agent
can't append its own flags to the last command of any other task, for example to an install
that has to keep `--ignore-scripts`. See [Arguments](/docs/remote-compute/runs#arguments).
* **`runs.secretNetwork: none`.** A task that reads a secret reaches only the services it needs,
so a tool it runs can't send the value anywhere. Give such a task `network: none`, or set an
allow list here and in the task. See
[Tasks that read secrets](/docs/remote-compute/runs#tasks-that-read-secrets).
* **One replica.** The disk holds the server's records of sandboxes and its cache.
## Secrets for builds [#secrets-for-builds]
The agent's shell task reads no secrets. Other tasks of the build file can, for example a
test against a staging API: the Job of such a task reads the secret itself, as a Kubernetes
ServiceAccount of its [account](/docs/remote-compute/accounts). See
[Secrets in the Job](/docs/remote-compute/backend#secrets-in-the-job) for the setup with
Secret Manager. With `secretNetwork: none` such a task has a limited network, so its provider
needs the hosts it reaches, which its egress proxy lets only the reading of the secret through:
```yaml title="server.yaml"
backend:
type: kubernetes
providers:
gsm:
image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
allow: [sts.googleapis.com, secretmanager.googleapis.com]
```
## Cleaning up [#cleaning-up]
Stop the port-forward loop first (`kill %1`, or close its terminal), then:
```bash
# the namespaces of this server's callers and service runs: namespacePrefix and 16 hex digits; look before you delete
kubectl get ns -l hammerkit.dev/managed -o name | grep -E "^namespace/$NAME-[0-9a-f]{16}$"
kubectl get ns -l hammerkit.dev/managed -o name | grep -E "^namespace/$NAME-[0-9a-f]{16}$" | xargs -r kubectl delete
kubectl delete ns $NAME
kubectl delete clusterrole $NAME-server $NAME-runs
kubectl delete clusterrolebinding $NAME-server
kubectl delete validatingadmissionpolicybinding,validatingadmissionpolicy $NAME-managed-namespaces $NAME-service-run-namespaces
gcloud iam workload-identity-pools delete $NAME --location global --project $PROJECT --quiet
gcloud secrets delete $NAME-github-app-key --project $PROJECT --quiet
gcloud compute addresses delete $NAME --global --project $PROJECT --quiet # if you reserved one for HTTPS
for who in agent publisher; do
gcloud iam service-accounts delete $NAME-$who@$PROJECT.iam.gserviceaccount.com --project $PROJECT --quiet
done
```
Every server labels the namespaces it makes `hammerkit.dev/managed`, so on a cluster with
several servers delete by the whole name, as above, not by the label alone. A deleted pool
keeps its name for 30 days.
# opencode on a sandbox (/docs/guides/agents/opencode)
Experimental. The tools and settings can still change between releases.
At the end of this tutorial you start [opencode](https://opencode.ai) on your machine, give it
a task for your project, and it works on a copy of your repository on a hammerkit server.
Your own checkout stays untouched: opencode never reads or writes a file of it and never runs
a command on your machine. When it is done, its work shows up as a branch in your repository,
ready for a pull request.
This page was run as written with opencode 2.0.30 against the server of
[Server on Kubernetes](/docs/guides/agents/kubernetes).
## How it fits together [#how-it-fits-together]
Four pieces work together. You set up the last three in this tutorial.
* **The server** keeps a **sandbox** for the agent: a checkout of your repository at a branch,
for example `main`. Every file opencode reads or edits is a file of the sandbox.
* **Your build file** has a **shell task**. Every command opencode runs, such as `npm test`,
is handed to that task as its argument and runs on the server, in the task's image. The
build file is **published** to the server once; the server answers with a hash that names
exactly that version.
* **The agent's identity** is a service account, not you. opencode signs in to the server as
that account, and the server only lets it start sandboxes from `main` and push to
`agent/…` branches.
* **opencode** runs on your machine and talks to the language model as usual. hammerkit's
opencode plugin replaces opencode's tools that read, edit, search and run commands with
ones of the same name that work on the sandbox, and adds a `push` tool.
## Before you start [#before-you-start]
**A hammerkit server for your GitHub repository.** [Server on Kubernetes](/docs/guides/agents/kubernetes)
sets one up on GKE: the server checks out and pushes to your repository with a GitHub App,
and accepts Google sign-ins. If someone else runs the server, ask them for its address and
the issuer of the tokens it accepts, for example `https://accounts.google.com`.
**Service accounts for the agent and the publisher**, and the right to act as them.
[Service accounts on Google Cloud](/docs/guides/agents/google-accounts) creates
`hammerkit-agent` and `hammerkit-publisher`, gives you the role that lets `gcloud` print their
tokens, and shows the entitlements the server needs for them. The Kubernetes guide does this
in its step 1; with a server someone else runs, have them add you as in
[Let your team act as them](/docs/guides/agents/google-accounts#3-let-your-team-act-as-them).
**Node.js 24 or later** on your machine, for opencode and the hammerkit CLI.
**An API key of a model provider** that opencode supports. The examples use OpenRouter.
The examples below use the repository `github.com/acme/shop`, a server registered as
`hammerkit`, and Google service accounts. Replace them with yours.
## 1. Install the hammerkit CLI [#1-install-the-hammerkit-cli]
```bash
npm install -g hammerkit@2.0.0 # the server's version
hammerkit --version
```
`hammerkit --version` has to print the server's version. hammerkit needs Node.js 24: on an
older one npm warns `EBADENGINE`, and with a range instead of an exact version it quietly
installs an old hammerkit without the agent clients.
Use the same version as the server. The CLI registers the server in step 2 and publishes the
build file in step 3.
## 2. Register the server on your machine [#2-register-the-server-on-your-machine]
The CLI keeps a list of servers for your user, in its config file. Add the server
under a short name of your choice, with the issuer of the tokens it accepts:
```bash
hammerkit remote add hammerkit http://127.0.0.1:8480 --issuer https://accounts.google.com
```
`hammerkit` is the name you use from now on. The address here is the
`kubectl port-forward` of the Kubernetes guide, which has to keep running while opencode
works. Plain `http` is only accepted for `127.0.0.1` and `localhost`; a server on any other
address needs `https`.
Register it as the same user that starts opencode later: the plugin reads this list.
## 3. Add a shell task to your project and publish it [#3-add-a-shell-task-to-your-project-and-publish-it]
Open your project's `.hammerkit.yaml`, or create one in the root of the repository, and add a
task for the agent's commands. For a Node.js project:
```yaml title=".hammerkit.yaml"
tasks:
sh:
image: node:22-alpine
mounts: [.]
args: true
cmds: [sh -c]
network: { allow: [registry.npmjs.org] }
```
Line by line:
* `sh` is the task's name. The plugin looks for a task called `sh` unless you set
`HAMMERKIT_SHELL`.
* `image` is the container every command of the agent runs in. Pick one with the tools your
project needs: the agent can only run `npm`, `go` or `python` if the image has them.
* `mounts: [.]` puts the files of the sandbox into the container, writable. Without it, the
commands would run in an empty directory.
* `args: true` says the task takes [arguments](/docs/task/arguments). A server can be set to
pass arguments to such tasks only, so mark the shell task even where nothing requires it.
* `cmds: [sh -c]` is the trick: the agent's command is appended to the last command as one
argument, so `npm test` runs as `sh -c 'npm test'`.
* `network` lists what the commands may reach. The sandbox is a fresh checkout without
`node_modules`, so the agent installs dependencies itself and needs the registry. The
server can narrow this further, never widen it.
Leave secrets out of this task: a task that reads a secret takes no arguments on a server.
Your other tasks, such as `test` or `lint`, stay as they are.
If you have Docker, try the task on your machine before you publish it:
```bash
hammerkit run sh -- 'ls && node --version'
```
Commit the file and get it onto `main`: the agent's commands run in the published version,
and its sandbox is a checkout of `main`, so the two should not drift apart.
Then publish it from the root of your checkout, signed in as the publisher's service account:
```bash
HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \
--impersonate-service-account=hammerkit-publisher@my-project.iam.gserviceaccount.com" \
hammerkit remote publish hammerkit
```
```text
published to hammerkit: sha256:9f2c61d0… (1 file)
```
If this exact version was published before, it prints `already published to hammerkit:`
with the same hash instead; that is fine.
`HAMMERKIT_TOKEN_COMMAND` makes this one command sign in as `hammerkit-publisher`: `gcloud`
prints a token of that service account for you. That only works once you may act as it, see
[Service accounts on Google Cloud](/docs/guides/agents/google-accounts). Write your project
instead of `my-project`; with the Kubernetes guide, the account is `$PUBLISHER` of its step 1.
Copy the hash, `sha256:` and 64 hex digits: it names this version of the build file, and
opencode uses it in step 5. The server never takes task definitions from the agent itself, so
the agent can not change what its commands run in. When you change the build file later,
publish it again and use the new hash.
## 4. Install opencode and the plugin [#4-install-opencode-and-the-plugin]
opencode loads plugins from the `.opencode/plugins/` directory of the directory it starts in.
Make a new, empty directory for the agent with that plugin. It stands in for the root of the
sandbox; nothing else in it is read or changed. Do not use your project's checkout for this.
```bash
npm install -g @opencode/cli
mkdir -p shop-agent/.opencode/plugins && cd shop-agent
cat > .opencode/package.json <<'EOF'
{ "dependencies": { "hammerkit": "2.0.0" } }
EOF
cat > .opencode/plugins/hammerkit.ts <<'EOF'
export { default } from 'hammerkit/harness/opencode'
EOF
npm install --prefix .opencode
npm ls --prefix .opencode hammerkit
```
* `.opencode/package.json` makes hammerkit a dependency of opencode's directory. Write the
server's exact version, not a range: `^1` would install a hammerkit without the plugin.
`npm ls` has to show that version.
* `.opencode/plugins/hammerkit.ts` is the plugin file: it hands hammerkit's plugin to
opencode as is.
* `npm install --prefix .opencode` installs it. opencode 2.0 does not install the
dependencies of plugins itself.
## 5. Tell the plugin where to work [#5-tell-the-plugin-where-to-work]
The plugin reads its settings from environment variables. Put them in a file, so every
session starts the same way:
```bash title="agent.env"
export HAMMERKIT_SERVER=hammerkit # the name from step 2
export HAMMERKIT_REPO=github.com/acme/shop # the repository of the sandbox
export HAMMERKIT_REF=main # the branch the sandbox starts from
export HAMMERKIT_BUILD=sha256:9f2c61d0… # the hash from step 3
export HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \
--impersonate-service-account=hammerkit-agent@my-project.iam.gserviceaccount.com"
export OPENROUTER_API_KEY=… # for the model, not for hammerkit
```
`HAMMERKIT_TOKEN_COMMAND` is how opencode signs in as the agent: a command that prints a
short-lived token of the agent's service account. The plugin runs it again whenever the
server stops accepting the token, so a long session needs no new sign-in. The command above
works once you may act as `hammerkit-agent` (with the Kubernetes guide, `$AGENT` of its step 1), see
[Service accounts on Google Cloud](/docs/guides/agents/google-accounts); check it there with
[a token test](/docs/guides/agents/google-accounts#4-check-that-a-token-comes-back). With
another identity provider, use any command that prints a token of the agent's account.
## 6. Check that the tools are hammerkit's [#6-check-that-the-tools-are-hammerkits]
When a plugin does not load, opencode starts without it and says nothing. Its own tools then
work on your machine instead of the sandbox. Do this check before you give opencode any work.
```bash
source agent.env
opencode service stop
opencode run --standalone -m openrouter/anthropic/claude-haiku-5.5 \
'List the files of the project root with the read tool.'
```
opencode 2.0 runs a background service that keeps the environment it was first started with.
`opencode service stop` ends it, so the next start sees the variables of `agent.env`; do that
whenever you change them. `--standalone` runs this one check without the service, and `-m`
picks the model, so the check does not depend on your opencode's default.
The answer has to list your repository's files. If it lists only `.opencode`, or says a file
is not found, the plugin did not load: check that `npm install --prefix .opencode` ran and
that the plugin file has the `export { default }` line.
## 7. Give it work and get it back [#7-give-it-work-and-get-it-back]
Start opencode in the agent's directory and give it a task the way you would in your own
checkout, telling it where to push:
```bash
opencode
```
> The cart total skips the last item. Fix it, run the tests and push the fix to
> `agent/fix-cart`.
On its first tool call the plugin signs in and takes a sandbox of the agent at `main` that
has no changes, or creates one. opencode edits the files of the sandbox, runs `npm ci` and
`npm test` as the shell task on the server, and finally calls `push`. The server commits the
changes as the agent and pushes them with its own GitHub credentials. Only branches the
entitlement allows are accepted, here `agent/…`.
The branch now shows up in your repository. Open a pull request from it and review it like
any other.
A sandbox stays on the server with whatever opencode changed in it, but the next start takes
a clean one again. Have opencode push before you end a session, or set
`HAMMERKIT_SANDBOX=` in `agent.env` when you know the id of the sandbox to come back to.
To run one prompt without the terminal interface, for example in CI:
```bash
BRANCH=agent/fix-cart-$(date +%Y%m%d%H%M%S) # a new branch for every run
opencode run --auto -m openrouter/anthropic/claude-haiku-5.5 \
"The cart total skips the last item. Fix it, run the tests and push to $BRANCH." < /dev/null
git -C "$CHECKOUT" ls-remote --exit-code origin "refs/heads/$BRANCH"
```
`--auto` approves the tool calls opencode would otherwise ask you about. opencode waits for
input on stdin, so close it when nothing feeds it. It exits with 0 even when its tools failed
and nothing was pushed, so the last line fails the job when the branch is not there.
`$CHECKOUT` is a checkout of the repository that may read it, for example the CI job's own;
a branch name of its own for every run keeps an earlier run's branch from passing the check.
## When it does not work [#when-it-does-not-work]
| What you see | What it means |
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| opencode lists only `.opencode`, or files of your machine | The plugin is not loaded, see [step 6](#6-check-that-the-tools-are-hammerkits). |
| The old settings are still used after a change of `agent.env` | The background service kept them: run `opencode service stop`. |
| `server hammerkit is not registered` | Step 2 was run as another user, or `HAMMERKIT_SERVER` names another server. |
| `… is not the hash of a published version` | `HAMMERKIT_BUILD` holds a name or a shortened hash. Use the full hash that step 3 printed. |
| `unable to reach http://…: fetch failed: connect ECONNREFUSED …` | Nothing listens at the server's address, for example `kubectl port-forward` has stopped. |
| `unable to reach https://…: fetch failed: …, this machine does not trust the certificate of the server` | The server's certificate comes from a company or private CA: name that CA's file in `NODE_EXTRA_CA_CERTS`, which takes one file. |
| `error: lost connection to pod`, `Unable to connect`, or an empty reply from the server | `kubectl port-forward` ended, because the server's pod was replaced or the connection dropped: start it again, or run it in the loop of the Kubernetes guide. |
| `could not read the source credential …` | The server can not mint its token for the repository; the rest of the message says why. See [the check](/docs/guides/agents/kubernetes#2-deploy-the-server) after the deploy. |
| `Permission 'iam.serviceAccounts.getAccessToken' denied` | You may not act as the service account in `HAMMERKIT_TOKEN_COMMAND` yet, see [Service accounts on Google Cloud](/docs/guides/agents/google-accounts#2-allow-yourself-to-act-as-them). |
| `you are not entitled to keep sandboxes on this server` | The token is not the agent's, or its entitlement has no `sandboxes`: check the account in `HAMMERKIT_TOKEN_COMMAND`. |
| The shell task needs `mounts: [.]` | The shell task does not mount the sandbox: add `mounts: [.]` (step 3), publish again and set the new hash. |
| `npm: not found`, or a download that times out | The shell task's image lacks the tool, or its `network` does not allow the host. Change the task, publish again and set the new hash. |
See [Agents](/docs/remote-compute/agents#opencode) for what each tool does on the sandbox.
# pi on a sandbox (/docs/guides/agents/pi)
Experimental. The tools and settings can still change between releases.
At the end of this tutorial you start [pi](https://pi.dev) on your machine, give it a task
for your project, and it works on a copy of your repository on a hammerkit server. Your own
checkout stays untouched: pi never reads or writes a file on your machine and never runs a
command there. When it is done, its work shows up as a branch in your repository, ready for
a pull request.
This page was run as written with pi 1.2.0 against the server of
[Server on Kubernetes](/docs/guides/agents/kubernetes).
## How it fits together [#how-it-fits-together]
Four pieces work together. You set up the last three in this tutorial.
* **The server** keeps a **sandbox** for the agent: a checkout of your repository at a branch,
for example `main`. Every file pi reads or edits is a file of the sandbox.
* **Your build file** has a **shell task**. Every command pi runs, such as `npm test`, is
handed to that task as its argument and runs on the server, in the task's image. The build
file is **published** to the server once; the server answers with a hash that names exactly
that version.
* **The agent's identity** is a service account, not you. pi signs in to the server as that
account, and the server only lets it start sandboxes from `main` and push to `agent/…`
branches.
* **pi** runs on your machine and talks to the language model as usual. hammerkit's pi
extension swaps pi's tools (`read`, `write`, `edit`, `bash`, `ls`, `find`, `grep`) for ones
that work on the sandbox, and adds a `push` tool.
## Before you start [#before-you-start]
**A hammerkit server for your GitHub repository.** [Server on Kubernetes](/docs/guides/agents/kubernetes)
sets one up on GKE: the server checks out and pushes to your repository with a GitHub App,
and accepts Google sign-ins. If someone else runs the server, ask them for its address and
the issuer of the tokens it accepts, for example `https://accounts.google.com`.
**Service accounts for the agent and the publisher**, and the right to act as them.
[Service accounts on Google Cloud](/docs/guides/agents/google-accounts) creates
`hammerkit-agent` and `hammerkit-publisher`, gives you the role that lets `gcloud` print their
tokens, and shows the entitlements the server needs for them. The Kubernetes guide does this
in its step 1; with a server someone else runs, have them add you as in
[Let your team act as them](/docs/guides/agents/google-accounts#3-let-your-team-act-as-them).
**Node.js 24 or later** on your machine, for pi and the hammerkit CLI.
**An API key of a model provider** that pi supports. The examples use OpenRouter.
The examples below use the repository `github.com/acme/shop`, a server registered as
`hammerkit`, and Google service accounts. Replace them with yours.
## 1. Install the hammerkit CLI [#1-install-the-hammerkit-cli]
```bash
npm install -g hammerkit@2.0.0 # the server's version
hammerkit --version
```
`hammerkit --version` has to print the server's version. hammerkit needs Node.js 24: on an
older one npm warns `EBADENGINE`, and with a range instead of an exact version it quietly
installs an old hammerkit without the agent clients.
Use the same version as the server. The CLI registers the server in step 2 and publishes the
build file in step 3; pi's extension in step 4 comes from the same package.
## 2. Register the server on your machine [#2-register-the-server-on-your-machine]
The CLI keeps a list of servers for your user, in its config file. Add the server
under a short name of your choice, with the issuer of the tokens it accepts:
```bash
hammerkit remote add hammerkit http://127.0.0.1:8480 --issuer https://accounts.google.com
```
`hammerkit` is the name you use from now on. The address here is the
`kubectl port-forward` of the Kubernetes guide, which has to keep running while pi works.
Plain `http` is only accepted for `127.0.0.1` and `localhost`; a server on any other address
needs `https`.
Register it as the same user that starts pi later: pi's extension reads this list.
## 3. Add a shell task to your project and publish it [#3-add-a-shell-task-to-your-project-and-publish-it]
Open your project's `.hammerkit.yaml`, or create one in the root of the repository, and add a
task for the agent's commands. For a Node.js project:
```yaml title=".hammerkit.yaml"
tasks:
sh:
image: node:22-alpine
mounts: [.]
args: true
cmds: [sh -c]
network: { allow: [registry.npmjs.org] }
```
Line by line:
* `sh` is the task's name. The extension looks for a task called `sh` unless you set
`HAMMERKIT_SHELL`.
* `image` is the container every command of the agent runs in. Pick one with the tools your
project needs: the agent can only run `npm`, `go` or `python` if the image has them.
* `mounts: [.]` puts the files of the sandbox into the container, writable. Without it, the
commands would run in an empty directory.
* `args: true` says the task takes [arguments](/docs/task/arguments). A server can be set to
pass arguments to such tasks only, so mark the shell task even where nothing requires it.
* `cmds: [sh -c]` is the trick: the agent's command is appended to the last command as one
argument, so `npm test` runs as `sh -c 'npm test'`.
* `network` lists what the commands may reach. The sandbox is a fresh checkout without
`node_modules`, so the agent installs dependencies itself and needs the registry. The
server can narrow this further, never widen it.
Leave secrets out of this task: a task that reads a secret takes no arguments on a server.
Your other tasks, such as `test` or `lint`, stay as they are, and pi can run them too.
If you have Docker, try the task on your machine before you publish it:
```bash
hammerkit run sh -- 'ls && node --version'
```
Commit the file and get it onto `main`: the agent's commands run in the published version,
and its sandbox is a checkout of `main`, so the two should not drift apart.
Then publish it from the root of your checkout, signed in as the publisher's service account:
```bash
HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \
--impersonate-service-account=hammerkit-publisher@my-project.iam.gserviceaccount.com" \
hammerkit remote publish hammerkit
```
```text
published to hammerkit: sha256:9f2c61d0… (1 file)
```
If this exact version was published before, it prints `already published to hammerkit:`
with the same hash instead; that is fine.
`HAMMERKIT_TOKEN_COMMAND` makes this one command sign in as `hammerkit-publisher`: `gcloud`
prints a token of that service account for you. That only works once you may act as it, see
[Service accounts on Google Cloud](/docs/guides/agents/google-accounts). Write your project
instead of `my-project`; with the Kubernetes guide, the account is `$PUBLISHER` of its step 1.
Copy the hash, `sha256:` and 64 hex digits: it names this version of the build file, and pi
uses it in step 5. The server never takes task definitions from the agent itself, so the
agent can not change what its commands run in. When you change the build file later, publish
it again and use the new hash; an agent that is already running keeps the old one.
## 4. Install pi and the extension [#4-install-pi-and-the-extension]
```bash
npm install -g @earendil-works/pi-coding-agent
pi install npm:hammerkit@2.0.0 # the server's version
pi list
```
`pi install` adds hammerkit as a [pi package](https://pi.dev/packages) for every pi session of
your user. `pi list` shows it under the user packages.
## 5. Tell the extension where to work [#5-tell-the-extension-where-to-work]
The extension reads its settings from environment variables. Put them in a file, so every
session starts the same way:
```bash title="agent.env"
export HAMMERKIT_SERVER=hammerkit # the name from step 2
export HAMMERKIT_REPO=github.com/acme/shop # the repository of the sandbox
export HAMMERKIT_REF=main # the branch the sandbox starts from
export HAMMERKIT_BUILD=sha256:9f2c61d0… # the hash from step 3
export HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \
--impersonate-service-account=hammerkit-agent@my-project.iam.gserviceaccount.com"
export OPENROUTER_API_KEY=… # for the model, not for hammerkit
```
`HAMMERKIT_TOKEN_COMMAND` is how pi signs in as the agent: a command that prints a
short-lived token of the agent's service account. The extension runs it again whenever the
server stops accepting the token, so a long session needs no new sign-in. The command above
works once you may act as `hammerkit-agent` (with the Kubernetes guide, `$AGENT` of its step 1), see
[Service accounts on Google Cloud](/docs/guides/agents/google-accounts); check it there with
[a token test](/docs/guides/agents/google-accounts#4-check-that-a-token-comes-back). With
another identity provider, use any command that prints a token of the agent's account.
## 6. Start pi and check where it works [#6-start-pi-and-check-where-it-works]
Start pi in a new, empty directory. pi treats its working directory as the root of the
sandbox, so this directory only stands in for it: nothing in it is read or changed.
```bash
mkdir shop-agent && cd shop-agent
source ../agent.env
pi --provider openrouter --model anthropic/claude-haiku-5.5
```
Ask pi first: "List the files of the project." The answer has to show your repository's
files, not an empty directory. On this first tool call the extension signs in, takes a
sandbox of the agent at `main` that has no changes (or creates one) and shows its id, for
example `sbx_4vJ0mT2cX9qLw1aB`. Every tool works on that sandbox from then on.
A sandbox stays on the server until it is deleted, with whatever pi changed in it. To come
back to unfinished work in a later session, add `export HAMMERKIT_SANDBOX=` to
`agent.env`; without it, pi starts from a clean checkout of `main` again.
## 7. Give it work and get it back [#7-give-it-work-and-get-it-back]
Give pi a task the way you would in your own checkout, and tell it where to push, for example:
> The cart total skips the last item. Fix it, run the tests and push the fix to
> `agent/fix-cart`.
pi edits the files of the sandbox, runs `npm ci` and `npm test` as the shell task on the
server, and finally calls `push`. The server commits the changes as the agent and pushes
them with its own GitHub credentials. Only branches the entitlement allows are accepted, here
`agent/…`; for any other branch, pi is told which ones it may push to.
The branch now shows up in your repository. Open a pull request from it and review it like
any other.
## Without a terminal [#without-a-terminal]
`-p` runs one prompt and exits, for example in CI. pi waits for input on stdin, so close it
when nothing feeds it:
```bash
BRANCH=agent/fix-cart-$(date +%Y%m%d%H%M%S) # a new branch for every run
pi --provider openrouter --model anthropic/claude-haiku-5.5 --no-session \
-p "The cart total skips the last item. Fix it, run the tests and push to $BRANCH." < /dev/null
git -C "$CHECKOUT" ls-remote --exit-code origin "refs/heads/$BRANCH"
```
pi can end without pushing, for example when its tools failed, so the last line fails the job
when the branch is not there. `$CHECKOUT` is a checkout of the repository that may read it,
for example the CI job's own; a branch name of its own for every run keeps an earlier run's
branch from passing the check.
`--mode json` prints every tool call and result as a line of JSON.
## When it does not work [#when-it-does-not-work]
| What you see | What it means |
| ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| pi lists an empty directory, or files of your machine | The extension is not loaded: check `pi list`. |
| `server hammerkit is not registered` | Step 2 was run as another user, or `HAMMERKIT_SERVER` names another server. |
| `… is not the hash of a published version` | `HAMMERKIT_BUILD` holds a name or a shortened hash. Use the full hash that step 3 printed. |
| `unable to reach http://…: fetch failed: connect ECONNREFUSED …` | Nothing listens at the server's address, for example `kubectl port-forward` has stopped. |
| `unable to reach https://…: fetch failed: …, this machine does not trust the certificate of the server` | The server's certificate comes from a company or private CA: name that CA's file in `NODE_EXTRA_CA_CERTS`, which takes one file. |
| `error: lost connection to pod`, `Unable to connect`, or an empty reply from the server | `kubectl port-forward` ended, because the server's pod was replaced or the connection dropped: start it again, or run it in the loop of the Kubernetes guide. |
| `could not read the source credential …` | The server can not mint its token for the repository; the rest of the message says why. See [the check](/docs/guides/agents/kubernetes#2-deploy-the-server) after the deploy. |
| `Permission 'iam.serviceAccounts.getAccessToken' denied` | You may not act as the service account in `HAMMERKIT_TOKEN_COMMAND` yet, see [Service accounts on Google Cloud](/docs/guides/agents/google-accounts#2-allow-yourself-to-act-as-them). |
| `you are not entitled to keep sandboxes on this server` | The token is not the agent's, or its entitlement has no `sandboxes`: check the account in `HAMMERKIT_TOKEN_COMMAND`. |
| The shell task needs `mounts: [.]` | The shell task does not mount the sandbox: add `mounts: [.]` (step 3), publish again and set the new hash. |
| `npm: not found`, or a download that times out | The shell task's image lacks the tool, or its `network` does not allow the host. Change the task, publish again and set the new hash. |
See [Agents](/docs/remote-compute/agents#pi) for what the extension does with each of pi's tools.
# Checks that ran before CI (/docs/guides/agents/verified-checks)
Experimental. The commands and settings can still change between releases. GitLab CI has
not been tested against a real GitLab yet.
[CI through the server](/docs/guides/agents/github-actions) moves the work of a CI job to the
server, but the job still waits for it: a runner that does nothing for the minutes the tests
take. A check task, such as `test` or `e2e`, does not have to run in CI at all when it already
passed on the server for the same files:
1. **Before the push**, an agent runs the task as a *check* on its sandbox, or a person runs it
on the server for the branch they pushed. The server notes the task, the
[check key](/docs/remote-compute/checks#check-keys) of the files it read, and that it passed.
2. **The CI job** runs [`hammerkit remote verify test e2e`](/docs/cli/remote#verifying-checks)
in its checkout. It computes the same keys over the checked-out files, asks the server, and
passes when the server has a passed check for each. Anything else fails at once: the job
never waits for a run and never starts one.
A check counts for every commit with the same files, whoever ran it and where: a commit that
only changes a README, an amended message or the squash commit on `main` is still verified. A
change to a file a task reads is not.
Deploys stay as in [CI through the server](/docs/guides/agents/github-actions): a deploy is
something to do, not something to have done.
This guide adds checks to a server set up with the [Kubernetes guide](/docs/guides/agents/kubernetes)
and the CI guide. Each `server.yaml` block goes into the config of that guide's ConfigMap.
## 1. Tasks that can be checked [#1-tasks-that-can-be-checked]
A check is only as complete as the `src` of its task: the key covers the files `src` names,
of the task and of the tasks it depends on, and nothing else. A test that reads a fixture
`src` leaves out would still be verified after the fixture changed. Name every folder the
task reads:
```yaml title=".hammerkit.yaml"
tasks:
install:
image: node:24-alpine
src: [package.json, package-lock.json]
generates: [node_modules]
cmds: [npm ci]
test:
image: node:24-alpine
deps: [install]
src: [src, test, tsconfig.json]
cmds: [npm test]
```
What the key covers, besides the files:
* The task's definition: commands, image, variables, mounts and the definitions of its
dependencies. A check of another version of the build file does not count when it changed
the task, so `cmds: [true]` published on the side does not pass for the real `test`. It
also means a check runs with the version of the build file the commit holds: when a change
touches a check task, publish the new version first and check with it.
* The services the task `needs`: their definitions, the files of their `src`, and the keys of
the tasks they depend on. An `e2e` that needs an `api` service is not verified after the
source of `api` changed.
* What dependencies generate (`node_modules` above) only as far as the commit holds it: the key
is computed on the checkout before anything ran in it, where those files are not there
unless they are committed. They are made again from their own `src`, which is in the key.
* Not `.git`, and not the run's arguments: a run with arguments is never a check.
A task without `src`, or whose `src` matches no file, here, in a dependency or in a task a
needed service depends on, can not be verified. `remote verify` says so and fails: `e2e: can
not be verified, it, or a task it or a service it needs depends on, declares no src that
matches a file`. The cache is less strict there: a task that resets the data of a service needs
no `src` for the task needing the service to be cached.
The CI runner and the server have to compute the same key. They do for tasks with an image.
A task whose variables read the variables of the process (`$HOME`, `$CI`) can get another key
in the CI job than on the server: `verify` then finds nothing and fails, it never passes by
mistake. Keep check tasks to fixed values.
## 2. Let the CI job read checks [#2-let-the-ci-job-read-checks]
The CI job only needs to read what passed. `checks: read` allows that for the repositories of
`repos`, and nothing else: no runs, no build files.
```yaml title="server.yaml"
entitlements:
# every job of the repository, pull requests too: read its checks
- claims:
iss: https://token.actions.githubusercontent.com
match: { repository_id: '123456789' }
repos: [github.com/acme/shop]
checks: read
```
`builds: run` or more also lets a caller read the checks of its `repos`, so the entitlements
of the CI guide already do. Who may check a sandbox needs `repos` for the repository as well,
and a branch of it to push to: a check counts like a run from a branch they pushed.
```yaml title="server.yaml"
entitlements:
# the agent of the Kubernetes guide: sandboxes, runs, and checks of its repositories
- subject: { iss: https://accounts.google.com, sub: '' }
repos: [github.com/acme/shop]
builds: run
sandboxes: [{ repo: github.com/acme/shop, branches: { main: pull, 'agent/**': push } }]
```
How long a check is found after it passed is `runs.checksKeep`, 30 days by default.
## 3. Check before the push [#3-check-before-the-push]
**An agent** on a sandbox calls its `check` tool with the task, `test`, after it is done
with its changes and before it calls `push`. All three [clients](/docs/remote-compute/agents)
have it. The tool answers whether the check passed:
```
check test passed (run run_Qx3k9ZpL2mW7vN4a): CI can verify it for a push of these files
```
The check does not run in the sandbox's directory but on a copy of the files a push would
commit: what `.gitignore` leaves out is not there. In its sandbox an agent can write any
ignored file, such as a `node_modules/.bin/vitest` that exits 0, and no key would notice it.
In the copy the dependencies are made from their `src`, or taken from the cache of the
repository's [ref runs](/docs/remote-compute/runs#source-files).
The pushed commit then has exactly the files the check ran on. An edit after the check
changes the key of every task that reads the file, so check again before pushing it.
**A person** pushes their branch and starts the tasks on the server for it, without waiting:
```bash
git push
hammerkit remote run test e2e --detach
```
```
run_Qx3k9ZpL2mW7vN4a on build: test, e2e from feature-x at 4be1a07c...
follow it with: hammerkit remote logs run_Qx3k9ZpL2mW7vN4a --follow
```
A run from a branch, without arguments, is a check: what it passes counts. Open the pull
request once it passed, or re-run its CI job.
## 4. The CI job [#4-the-ci-job]
```yaml title=".github/workflows/checks.yaml"
name: checks
on:
pull_request:
push: { branches: [main] }
permissions:
contents: read
id-token: write
jobs:
checks:
runs-on: ubuntu-latest
timeout-minutes: 5
env:
HAMMERKIT_TOKEN_COMMAND: >-
curl -sSf -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN"
"$ACTIONS_ID_TOKEN_REQUEST_URL&audience=https://hammerkit.example.com" | jq -r .value
steps:
- uses: actions/checkout@v4
with: { ref: '${{ github.event.pull_request.head.sha || github.sha }}' }
- run: npm install -g hammerkit@2.0.0
- run: |
hammerkit remote add hammerkit https://hammerkit.example.com --issuer https://token.actions.githubusercontent.com
hammerkit remote use hammerkit
- run: hammerkit remote verify test e2e --repo github.com/acme/shop
```
The checkout is of the pull request's head, the commit that was pushed, not the merge commit
GitHub makes. Both usually have the same files for `test`, but the head is what was checked.
`--repo` names the repository as the server knows it; left out, it is the `origin` of the
checkout. Verify right after the checkout, before any other step writes files.
A job that passes prints what passed, and when:
```
test: passed 2026-10-25T09:14:03.120Z in 1m12s, run_Qx3k9ZpL2mW7vN4a by agent-x, from a check of sandbox sbx_4vJ0mT2cX9qLw1aB, fetched at 4be1a07c1a2b
e2e: passed 2026-10-25T09:15:40.002Z from the cache, run_8Lw2… by 1180…, from feature-x at 9e1f20aa31c4
```
A job that does not, fails in the seconds it took to hash the files:
```
test: no run on hammerkit passed it for these files
not verified: test; run on the server: hammerkit remote run test
```
Re-run the job after the check passed. The deploy job of the CI guide stays: it runs on
`main` after the checks.
### GitLab CI [#gitlab-ci]
```yaml title=".gitlab-ci.yml"
checks:
image: node:24-alpine
id_tokens:
HAMMERKIT_TOKEN: { aud: https://hammerkit.example.com }
script:
- npm install -g hammerkit@2.0.0
- hammerkit remote add hammerkit https://hammerkit.example.com --issuer https://gitlab.com
- hammerkit remote use hammerkit
- hammerkit remote verify test e2e --repo gitlab.com/acme/shop
```
With an entitlement for GitLab's tokens, see [GitLab CI](/docs/remote-compute/authentication#gitlab-ci):
```yaml title="server.yaml"
entitlements:
- claims: { iss: https://gitlab.com, match: { project_id: '4711' } }
repos: [gitlab.com/acme/shop]
checks: read
```
## What a check can not tell [#what-a-check-can-not-tell]
* **Files `src` leaves out.** See [step 1](#1-tasks-that-can-be-checked).
* **Who wrote the code.** A check says the tasks passed for these files, not that anybody
reviewed them. Branch protection and reviews still decide what is merged.
* **Flaky tests.** A test that passed once for the files counts, as it would in the cache.
* **A yanked version.** A check that passed with a version of the build file that was
[yanked](/docs/remote-compute/build-files) since does not count any more.
# Caching strategy in CI (/docs/guides/ci-caching)
Locally, hammerkit caches automatically: a task whose `src` is unchanged is
skipped. The challenge in CI is that each run usually starts from a **fresh
checkout** on a **fresh machine**, so there is no local cache to hit. This guide
covers the two ways to carry results across CI runs and which to choose.
Keep the cache method at the default **`checksum`** in CI. `modify-date` compares
file timestamps, which `git clone` resets on every run — so a `modify-date` task
never hits the cache on a fresh runner. See [caching](/docs/task/caching#checksum-vs-modify-date).
## Option A — a remote cache backend (recommended) [#option-a--a-remote-cache-backend-recommended]
Point the built-in `default` cache at an S3-compatible bucket (or a container
registry — see [caches](/docs/build-file/caches#registry)). Hammerkit then
**pulls** a task's result before running it and **pushes** the result after — no
cache scripting in your pipeline at all.
```yaml title=".hammerkit.yaml"
caches:
default:
method: checksum
backend:
type: s3
bucket: my-build-cache
region: eu-central-1
tasks:
build:
image: node:24-alpine
src:
- src
generates:
- dist
cmds:
- npm run build
```
Credentials come from the standard AWS SDK chain (`AWS_ACCESS_KEY_ID` /
`AWS_SECRET_ACCESS_KEY`), so set those as CI secrets. Backend errors never fail the
build — an unreachable bucket just falls back to running the task. The same bucket
works for developers locally, so a result built on a laptop is reused in CI and
vice versa. See [caches](/docs/build-file/caches) for the full backend reference
(MinIO, R2, GCS).
### Sharing across checkouts, runners and agent sandboxes [#sharing-across-checkouts-runners-and-agent-sandboxes]
Cache ids are computed from paths relative to the project root (the git root), so
the same commit produces the same ids wherever it is checked out — a CI runner at
`/home/runner/work/app/app`, an agent sandbox at `/workspace/app` and a laptop all
hit the same entries. Use **container tasks** for anything you want to share
between macOS and Linux machines: a local task includes the host OS in its identity.
### Checks that don't need the outputs [#checks-that-dont-need-the-outputs]
A job that only gates on a task passing doesn't need its outputs. Run it with
`--no-restore`: a hit is then only looked up in the bucket, and nothing is
downloaded unless a task that has to run depends on it. See
[checking without restoring](/docs/cli/execute#checking-without-restoring).
```bash
hammerkit run test --no-restore
```
This only saves the download when the task's own cache is the remote one. After a
`cache pull` (below), the entry is already on the machine.
### Read-only runners [#read-only-runners]
Only trusted runners should write to the shared cache. Give untrusted runners —
coding-agent sandboxes, pull requests from forks — read-only credentials and run
them in read-only mode, so they restore results but never push:
```bash
export HAMMERKIT_CACHE_READ_ONLY=1 # or pass --cache-read-only to run / up
hammerkit run test
```
Read-only mode skips every backend push, including the built-in local `default`
cache. Results a read-only run builds are still reused by later runs in the same
checkout.
### Splitting network from compute [#splitting-network-from-compute]
Instead of letting every task talk to the bucket during the build, keep tasks on
the machine-local `default` cache and move entries explicitly with
[`cache pull` / `cache push`](/docs/cli/cache). The build itself then does no
network I/O, and a workspace can be warmed before anyone runs anything:
```bash
hammerkit cache pull --remote shared # network only
hammerkit run # compute only
hammerkit cache push --remote shared # trusted runners only
```
## Option B — store / restore with the CI's own cache [#option-b--store--restore-with-the-cis-own-cache]
If you'd rather use your CI provider's built-in caching, wrap the run with
[`store` / `restore`](/docs/cli/store-restore): `restore` pulls the previous
results out of a directory before the build, `store` writes them back after, and
the CI caches that directory.
```bash
hammerkit restore cache # before the build
hammerkit build
hammerkit store cache # after the build
```
A GitHub Actions job wiring that to `actions/cache`:
```yaml
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '24'
- run: npm i -g hammerkit
- uses: actions/cache@v4
with:
path: cache
key: hammerkit-${{ hashFiles('**/package-lock.json') }}
restore-keys: hammerkit-
- run: hammerkit restore cache
- run: hammerkit build
- run: hammerkit store cache
```
## Example: hammerkit's own CI [#example-hammerkits-own-ci]
Hammerkit builds and tests itself this way. The repository's
[`.hammerkit.yaml`](https://github.com/no0dles/hammerkit/blob/master/.hammerkit.yaml)
declares `lint`, `build`, `test`, and the integration suite as `test:docker` and
`test:kubernetes` (`test:integration` runs both) plus a `ci` cache on a GHCR
repository, and each workflow job does the same three things: `cache pull`, `run`,
`cache push`. The two integration tasks run as parallel jobs, so the Kubernetes
cluster is only started for the one that needs it. A push whose tree was already
checked or tested skips the work — including that cluster, which the workflow only
starts when `hammerkit explain` reports a miss. Every cache step may fail
without failing the job, so an unreachable registry just means a cold run. A test
that is flaky is a cached pass once it passes, so a run can be forced to ignore the
cache: start the workflow by hand, or put `[no-cache]` in the head commit message.
That run neither pulls nor pushes. A failed task is never cached.
## Which one? [#which-one]
| | Remote backend | store / restore |
| -------------------------- | -------------------------- | ---------------------------------------------- |
| Pipeline wiring | None — automatic pull/push | You add `restore`/`store` steps + a cache step |
| Granularity | Per task | One directory for the whole run |
| Shared with local dev | Yes (same bucket) | No (CI cache only) |
| Needs a bucket or registry | Yes | No |
A remote backend is the lower-maintenance choice and is shared with local
development. If you already rely on your CI's cache and don't want a bucket,
store/restore is fine. You rarely need both — **if a remote backend is configured,
store/restore is redundant.**
# Building container images (/docs/guides/container-images)
This guide builds a Dockerfile in a task that runs on a [hammerkit server](/docs/remote-compute)
and pushes the image to a registry. It sets up the server once, as its operator, and then
writes the build file: a task with [`build`](/docs/task/image-builds), which is the same on a
dev machine. The example pushes to Google's Artifact Registry from a server on
Kubernetes; the other registries follow the same steps, the differences are named where they
come up. A server on a [docker host](#on-a-docker-host) doesn't build images yet.
## Why a server needs a sandbox for it [#why-a-server-needs-a-sandbox-for-it]
On a dev machine a task could mount the docker socket and run `docker build`. On a server that
would hand the task the daemon of the machine, and with it every container of every caller
on it, so the server refuses a mount of the socket. Docker in Docker needs a `privileged`
container, which a server never starts. BuildKit without root needs what the server takes
from every container: its seccomp profile, `allowPrivilegeEscalation: false` and the masked
`/proc`.
What works is BuildKit as root with one more capability, `SYS_ADMIN`, so it can mount the
layers it builds. `SYS_ADMIN` is root's power over the kernel, and the kernel of a node runs
the tasks of other callers too. So the server adds it only in a **sandboxed runtime**, which
gives the container a kernel of its own: [gVisor](https://gvisor.dev) handles the container's
system calls in a kernel it implements itself, [Kata Containers](https://katacontainers.io)
start a small virtual machine per pod. `SYS_ADMIN` in there reaches the sandbox's kernel, not
the node's. A task with `build` adds the capability, and the server's operator allows it in
[`backend.capabilities`](/docs/remote-compute/backend#capabilities).
This guide was tested on GKE Autopilot with gVisor (BuildKit v0.25.1): the build, a `RUN`
step that downloads packages, and the push.
## What you need [#what-you-need]
* A hammerkit server on Kubernetes, set up as in
[the Kubernetes guide](/docs/guides/agents/kubernetes), and the rights of its operator on
the cluster.
* A registry to push to. Here, a Docker repository in Artifact Registry, and the right to
grant roles on it.
* The variables of the Kubernetes guide in your shell: `NAME` (the server's name and
namespace), `PROJECT`, `POOL` and `AUDIENCE`.
## 1. A sandboxed runtime on the cluster [#1-a-sandboxed-runtime-on-the-cluster]
The cluster needs a RuntimeClass that starts pods in a sandbox. Look for one first:
```bash
kubectl get runtimeclass
```
| Cluster | The RuntimeClass |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GKE Autopilot | `gvisor`, there on every cluster ([GKE Sandbox](https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods)). Nothing to do. |
| GKE Standard | `gvisor`, once a node pool has it: `gcloud container node-pools create sandbox --cluster --sandbox type=gvisor`. |
| AKS | `kata-vm-isolation`, with [Pod Sandboxing](https://learn.microsoft.com/azure/aks/use-pod-sandboxing) on a node pool (`--workload-runtime KataVmIsolation`). |
| EKS and your own nodes | Install gVisor's `runsc` and its containerd shim on the nodes ([gVisor's containerd guide](https://gvisor.dev/docs/user_guide/containerd/quick_start/)), then add a RuntimeClass with `handler: runsc`. |
A RuntimeClass of your own looks like this:
```yaml title="runtimeclass.yaml"
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata: { name: gvisor }
handler: runsc
```
The rest of this guide calls it `gvisor`. Use the name your cluster has.
## 2. Let the server add `SYS_ADMIN` there [#2-let-the-server-add-sys_admin-there]
Three changes, in this order, so no pod can add a capability outside the sandbox at any point.
**The admission policies.** A namespace of a caller enforces the Pod Security level `baseline`,
which refuses `SYS_ADMIN` even in a sandbox: it does not know about RuntimeClasses. With
capabilities, the server makes the namespaces of callers enforce `privileged`, and an admission
policy of yours refuses what `baseline` refuses, and any capability beyond it outside the
RuntimeClass. A second one keeps the namespaces the server makes to its own. Copy both from
[Capabilities](/docs/remote-compute/backend#capabilities) into `admission.yaml`, with your
RuntimeClass in place of `gvisor`, `$NAME` in place of `hammerkit` in the server's
ServiceAccount (`system:serviceaccount:$NAME:hammerkit`) and `^$NAME-` in place of `^hammerkit-`,
and apply it. They need Kubernetes 1.30 or later:
```bash
kubectl apply -f admission.yaml
```
**The namespaces the server made.** The server never changes the level of a namespace it made
before (it has no right to), so set it on those of its prefix:
```bash
kubectl get namespace -l hammerkit.dev/managed=true -o name | grep "^namespace/$NAME-" | \
xargs -r -I{} kubectl label {} pod-security.kubernetes.io/enforce=privileged \
pod-security.kubernetes.io/audit=baseline --overwrite
```
**The config.** In `server.yaml` (the ConfigMap `hammerkit-config`), under `backend`:
```yaml title="server.yaml"
backend:
type: kubernetes
capabilities:
allow: [SYS_ADMIN]
runtimeClass: gvisor
```
Restart the server so it reads the config:
```bash
kubectl -n $NAME rollout restart deploy/hammerkit
```
Only the pods of tasks that add a capability run in gVisor; every other task runs as before.
Runs on [sandboxes](/docs/remote-compute/sandboxes) add none, unless you list them under
`sandboxes.capabilities`. On GKE Standard and AKS only the sandbox's node pool has the runtime:
the pods of these tasks go to the nodes of the first [pool](/docs/remote-compute/backend#pools),
so its labels have to match that node pool, or a pod stays `Pending`.
## 3. An account that may push [#3-an-account-that-may-push]
The task pushes with a token of a service account, never with a key or a person's login. The
server mints the token: a Kubernetes ServiceAccount of its namespace, which Google trusts through
the identity pool of the Kubernetes guide, and which has the right to push to one repository and
nothing else.
```bash
REGION=europe-west4 REPOSITORY=apps
gcloud artifacts repositories create $REPOSITORY --repository-format docker \
--location $REGION --project $PROJECT # skip this if the repository exists
# the identity that pushes: a ServiceAccount in the server's namespace, writer on the repository
kubectl -n $NAME create serviceaccount registry-push
gcloud artifacts repositories add-iam-policy-binding $REPOSITORY --location $REGION \
--project $PROJECT --role roles/artifactregistry.writer \
--member principal://iam.googleapis.com/$POOL/subject/system:serviceaccount:$NAME:registry-push
# how gcloud signs in as it: the token the server puts in the Job, exchanged at Google's STS
kubectl -n $NAME create secret generic registry-push-federation --from-literal=gcp.json="$(cat <
Local-vs-container still applies in dev: a task with an `image` runs in a container
(consistent tooling, recommended); drop the `image` to run on the host when you
need a tool that's only installed locally. See
[local vs. container tasks](/docs/guides/local-vs-container).
# Local vs. container tasks (/docs/guides/local-vs-container)
The single rule, restated: **a task with an `image` runs in a container; a task
without one runs on your host.** Everything else — `src`, `generates`, `deps`,
`needs`, caching — behaves the same either way. This guide is about *choosing*
between them.
## Prefer containers [#prefer-containers]
Containers are the recommended default, and the reason hammerkit exists:
* **No local tooling.** The compiler, SDK, CLI — all come from the image. Nobody on
the team installs Node, .NET, Helm or the AWS CLI; they just need a container
engine.
* **Repeatable everywhere.** The build behaves identically on every laptop and
every CI runner, because it's the same image. No "works on my machine".
* **Per-task images.** Each task can use a *different* image, so you don't maintain
one giant build image with every tool in it.
* **Isolation.** Only declared `src`/`generates`/`mounts` are visible, so side
effects are contained.
```yaml
tasks:
build:
image: node:24-alpine # tools come from here
src:
- src
cmds:
- npm run build
```
## When a local task makes sense [#when-a-local-task-makes-sense]
Drop the `image` to run on the host when:
* the tool **can't** run in a container here — it needs host hardware or a platform
SDK (Xcode on macOS, signing tools, a local GPU);
* you're calling something every developer already has and containerizing adds no
value;
* you're driving the host's own container engine or `kubectl` and want host
credentials/context.
```yaml
tasks:
package-ios:
platform:
os: macos
cmds:
- xcodebuild ...
```
A local task can declare a [`platform`](/docs/build-file/reference#task) so
hammerkit only runs it where it's supported — useful in a CI matrix. Use
[labels](/docs/labels) to route such tasks to the right runner (a macOS host
for iOS, Linux for the rest).
## Mixing them [#mixing-them]
You can freely mix: a local task can depend on a container task and vice versa.
Sources and outputs of a container task's dependencies are mounted in automatically,
so the chain stays consistent. A local task that `needs` a service gets the
connection details as [environment hints](/docs/service#environment-hints-for-local-tasks),
since it isn't on the container network.
# CI migration guide (for coding agents) (/docs/guides/migrate-ci/agent-guide)
You are migrating this repository's CI pipeline to
[hammerkit](https://github.com/no0dles/hammerkit). Hammerkit runs build steps as
**tasks** declared in a `.hammerkit.yaml` file, each in a container image, and
skips a task when its declared inputs are unchanged. The CI then shrinks to
installing hammerkit and running tasks; the same tasks run on a laptop.
Two words carry the bar for this work:
* **Parity** — the migrated pipeline runs the same checks and produces the same
artifacts as the old one. Nothing the old pipeline verified goes unverified.
* **Hermetic** — a task's result depends only on what it declares: its image, its
commands, its `envs`, its `src` files and its `deps`. The cache trusts this; an
undeclared input lets a stale result through without any error.
Work through the steps in order. Each ends on a **done when** check; finish it
before moving on. The reference sections after the steps hold the syntax and
mappings you need along the way.
## Step 1 — Take stock of the current pipeline [#step-1--take-stock-of-the-current-pipeline]
Read every CI definition in the repository and everything it calls:
* `.github/workflows/*.yml`, `.gitlab-ci.yml`, `Jenkinsfile`, `.circleci/config.yml`,
`azure-pipelines.yml`, `bitbucket-pipelines.yml`, `.buildkite/`, `.drone.yml`;
* the scripts those run: `Makefile`, `package.json` scripts, `scripts/*.sh`,
`docker-compose*.yml` used by tests.
Build an inventory table with one row per job, recording: trigger and conditions
(branches, paths, PR vs push, schedules), toolchain (runner image, `container:`,
setup actions and their versions), each step's command, services (databases,
queues), environment variables and secrets, caches, artifacts produced and
consumed, matrix axes, job dependencies, timeouts, and steps allowed to fail.
**Done when** every job and every step of every CI file appears in the inventory.
## Step 2 — Classify every step [#step-2--classify-every-step]
Give each step exactly one class:
| Class | Examples | Becomes |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| **Task** | install dependencies, lint, typecheck, unit/integration tests, build, e2e | a hammerkit task (step 3) |
| **Toolchain** | `actions/setup-node`, `setup-java`, `container: node:20`, `apt-get install` of build tools | the task's `image` |
| **Service** | `services: postgres`, `docker compose up db` for tests | a hammerkit `service` plus `needs` (a whole Compose stack: see [Compose test stacks](#compose-test-stacks)) |
| **Plumbing** | checkout, `actions/cache`, cache keys, upload/download of artifacts *between jobs* | dropped: hammerkit caches per task and passes outputs along `deps` |
| **CI-owned** | triggers, branch and path conditions, secret injection, permissions, publishing releases or packages, deployments, notifications, uploading artifacts for humans | stays in the CI config |
**Done when** every inventory row has exactly one class, and every Plumbing step
has a note on what replaces it.
## Step 3 — Write the build file [#step-3--write-the-build-file]
Create `.hammerkit.yaml` at the repository root (in a monorepo, see
[monorepo layout](/docs/guides/monorepo)). For every Task-class step group:
1. Make it a task named after what it does (`install`, `lint`, `test`, `build`,
`e2e`). Put its commands in `cmds`, in order.
2. Give it an `image` carrying the toolchain the CI used, at the same version
(`setup-node` with `node-version: 20` → `node:20-alpine`, or the Debian variant
when the commands need glibc or bash). Use a local task (no `image`) only for
steps that need the host itself: Xcode, signing tools, running `docker` or
`kubectl` with host credentials.
3. Add `deps` from the data flow: a task depends on every task whose outputs it
reads (`build` reads `node_modules`, so `deps: [install]`).
4. Add `needs` for every service its commands connect to, and declare the service.
5. Declare each environment variable its commands read in `envs`. Hammerkit passes
only declared variables to a task. Secrets come from the CI environment as
`NAME: $NAME`.
6. For every CI job, add an aggregate task with only `deps` listing what that job
ran (`ci: { deps: [lint, test, build] }`), so each CI job runs one hammerkit
command.
A CI matrix becomes one task per variant, sharing a base through
[`extend`](/docs/task/extending), aggregated by a task with `deps` on all variants.
Pick images with these in mind:
* **Tasks run as your user, not root.** On Linux a container task runs with the
host's uid and `HOME=/tmp`, so `apt-get install` or `apk add` in `cmds` fails.
Every tool the commands call (git, jq, python3, bash, a docker CLI) must already
be in the image. When the old job installed tools first, use an image that has
them (the `mcr.microsoft.com/devcontainers/*` images carry git, bash and python)
or a tools task that downloads a pinned binary into its `generates`.
* **One C library per dependency tree.** Native modules (`node_modules` with
compiled addons, Python wheels) are built for the image that installed them.
Every task that loads `install`'s output needs the same libc: all Alpine (musl)
or all Debian (glibc), not a mix.
**Done when** every Task-class step appears in exactly one task's `cmds`, and
`hammerkit validate` reports no errors.
## Step 4 — Declare inputs and outputs [#step-4--declare-inputs-and-outputs]
This step decides whether the cache is correct. For every task:
* **`src`** — every file and folder its commands read: sources, configs
(`tsconfig.json`, `eslint.config.js`, `.babelrc`), lockfiles, test fixtures, and
`package.json` when scripts are run through it. Prefer folders (`src`) over
narrowing globs when the command reads more than the glob would match. A task
without `src` of its own runs on every invocation — fine for aggregate tasks, not
for the work itself.
* **`generates`** — everything a later task or the CI reads: `node_modules`, `dist`,
coverage and test reports. Add `export: true` to outputs a CI step reads from the
workspace (an artifact upload) or that a human inspects.
* **Everything a command writes is an output**, even inside a source folder:
tool state (`.yarn/install-state.gz`, `.eslintcache`), generated files a later
command reads (generated API docs), reports. Declare the path in `generates`,
or point the tool somewhere declared (`YARN_INSTALL_STATE_PATH`). A path that's
in neither `src` nor `generates` doesn't exist in the container, and the task's
user can't create it there: writing to it fails with `EACCES` or `ENOENT`.
* **Monorepos have one `node_modules` per workspace.** `generates` takes paths, not
globs: list `node_modules` and each `packages//node_modules` the install
creates (`find . -name node_modules -prune -not -path './node_modules/*'` after
a local install shows them).
Then apply the rules in [hermetic tasks](#hermetic-tasks) below.
**Done when**, for every task, you can name each file its commands read and it is
covered by that task's `src` or by a dependency's `generates`.
## Step 5 — Rewrite the CI config [#step-5--rewrite-the-ci-config]
Keep everything CI-owned from step 2. Replace the rest of each job with:
1. install a pinned hammerkit version,
2. log in to the registry used as remote cache (if you add one),
3. `hammerkit cache pull --remote shared`,
4. `hammerkit run `,
5. `hammerkit cache push --remote shared`, only on trusted refs (the default
branch).
Use the [GitHub Actions template](#ci-template-github-actions) below; for other
providers keep the same sequence (see [installation](/docs/installation) for the
container image and the GitLab example). Add `.hammerkit` to `.gitignore`.
On GitLab:
* Container tasks bind-mount the checkout, so the Docker daemon must see it at the
same path. A shell runner with the host's Docker, or the Docker executor with a
`docker:dind` service (the [installation](/docs/installation) example), both work.
The Docker executor with the host's socket mounted in does not: the host has no
`/builds/...`.
* A shell runner reuses its working copy. Keep hammerkit's project state across
jobs with `GIT_CLEAN_FLAGS: -ffdx -e .hammerkit`, and install the pinned binary
under the runner's `$HOME`, which also keeps `~/.hammerkit` between jobs.
* For a registry remote cache in GitLab's container registry, log in with the job
token: `docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"`.
**Done when** every CI job consists of CI-owned steps plus one hammerkit run, and
no Plumbing step remains.
## Step 6 — Verify [#step-6--verify]
Run these and check each expected result:
| Command | Expected |
| ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| `hammerkit validate` | no errors (warnings about descriptions are fine) |
| `hammerkit ls` | every task from step 3, with the images, `src` and `generates` you meant |
| `hammerkit run ci --dry-run` | the tasks in a sensible order, nothing missing |
| `hammerkit run ci` | succeeds; the same tests run and pass as in the old pipeline |
| `hammerkit run ci` again | `0 executed` in the summary: every task `cached`, or `skipped` because what needs it was cached |
| change one source file, then `hammerkit run ci --dry-run` | only the tasks reading that file, and tasks depending on them, are misses |
| `hammerkit run ci --cache none` | succeeds, and the outputs match the cached run |
| `docker ps --filter label=hammerkit-type=service` after the runs | empty: no service outlived its run |
A task that rebuilds without a change reads something that differs between runs;
`hammerkit explain ` names the input. A step that fails only under hammerkit
usually reads a file or variable it doesn't declare: fix the declaration, not the
command.
While you iterate:
* Run with `--log live` to get plain line-by-line output that you can read back
from a file. The interactive view redraws the terminal.
* Every `$NAME` that a task in the run declares must be set, even if empty;
tasks outside the run don't need theirs. For a variable the CI provider sets
only on some pipelines (merge request ids, branch names), declare a default
instead: `MR_IID: ${CI_MERGE_REQUEST_IID:-}`.
* Stopping a run with Ctrl-C (or `SIGTERM`) removes its services and stops its
local tasks. A run killed outright (`kill -9`) can't: `hammerkit clean` removes
what it left behind.
* Output volumes of every state you tried stay on the Docker host until
`hammerkit clean`. On a small CI or workspace disk, watch `docker system df`.
**Done when** every row matches, and the test and artifact list matches the old
pipeline (parity).
## Step 7 — Report [#step-7--report]
Write the migration report (as `MIGRATION.md` or the pull request description):
* a table mapping each old CI job and step to its task, or to "kept in CI" or
"dropped" with the reason;
* the verification results from step 6;
* everything that needs a human: secrets to configure, registry permissions for
the remote cache, deployment steps left untouched, and any input you couldn't
pin down.
**Done when** the report covers every inventory row.
***
## Reference: CI concepts in hammerkit [#reference-ci-concepts-in-hammerkit]
| CI concept | GitHub Actions / GitLab CI | Hammerkit |
| ---------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| Job | `jobs.` / a job | a task, or an aggregate task with `deps` |
| Steps | `run:` / `script:` | `cmds` |
| Toolchain | `runs-on`, `container:`, `setup-*` actions / `image:` | `image` |
| Job order | `needs:` / `stages`, `needs:` | `deps` |
| Artifacts between jobs | `upload-artifact` + `download-artifact` / `artifacts` + `dependencies` | `generates` on one task, `deps` on the next |
| Caches | `actions/cache` / `cache:` | none — per-task caching from `src`, shared via a [remote cache](/docs/build-file/caches) |
| Service containers | `services:` | `services` + `needs`; reach them by service name |
| Environment | `env:` / `variables:` | `envs` (build-file or task level) |
| Secrets | `${{ secrets.X }}` / masked variables | CI exports `X`; the task declares [`secrets: [{ from: env:X, env: X }]`](/docs/task/secrets) — out of the cache key and masked in logs |
| Matrix | `strategy.matrix` / `parallel:matrix` | one task per variant via `extend` |
| Timeout | `timeout-minutes` / `timeout` | `timeout: 20m` |
| Working directory | `working-directory` / `cd` | `{cmd, path}` in `cmds`, or a [referenced](/docs/build-file/references) build file per package |
| Conditions, triggers | `on:`, `if:` / `rules:` | stay in CI; CI picks which task to run |
| Allowed failure | `continue-on-error` / `allow_failure` | stays in CI as a separate job running that task |
## Reference: the build file [#reference-the-build-file]
A complete example covering the common shapes — a dependency install, lint, tests
against a database, a build whose output CI uploads, and e2e tests:
```yaml title=".hammerkit.yaml"
envs:
NODE_IMAGE: node:24-alpine
POSTGRES_IMAGE: postgres:16-alpine # part of every task's key in this file (`validate` warns)
DATABASE_URL: postgres://app:app@db:5432/app
caches:
shared:
method: checksum
backend:
type: registry
repository: ghcr.io/my-org/hammerkit-cache
services:
db:
description: database for tests
image: $POSTGRES_IMAGE
envs:
POSTGRES_USER: app
POSTGRES_PASSWORD: app
POSTGRES_DB: app
healthcheck:
cmd: pg_isready -U app
tasks:
install:
description: install npm dependencies
image: $NODE_IMAGE
src: [package.json, package-lock.json]
generates: [node_modules]
cmds: [npm ci]
lint:
description: lint the sources
image: $NODE_IMAGE
deps: [install]
src: [src, eslint.config.js, package.json]
cmds: [npm run lint]
test:
description: unit and integration tests
image: $NODE_IMAGE
deps: [install]
needs: [db]
src: [src, test, tsconfig.json, package.json]
cmds: [npm test]
build:
description: compile the app
image: $NODE_IMAGE
deps: [install]
src: [src, tsconfig.json, package.json]
generates:
- path: dist
export: true # CI uploads dist as an artifact
cmds: [npm run build]
e2e:
description: end-to-end tests against the built app
image: cypress/included:13.15.0
deps: [build]
needs: [db]
src: [cypress, cypress.config.ts]
timeout: 20m
cmds: [cypress run]
ci:
description: everything a pull request must pass
deps: [lint, test, build]
```
How it behaves:
* Commands run with `/bin/sh` in the image, in the build file's directory; relative
paths work as on the host. Alpine images have no bash: use a Debian-based image
and `shell: bash` for bash scripts.
* A container task sees its `src`, its `mounts`, and the sources and outputs of its
dependencies. Its outputs are what it declares in `generates`; other files it
writes inside the container are discarded. `src` and `mounts` are mounted
read-write, so a command writing into a source folder changes the checkout and
the task's own inputs — keep outputs out of source folders.
* A task reaches a service by its name on the service's container port (`db:5432`).
Without a `healthcheck`, a task may start before the service accepts connections.
* `$NAME` in `image`, `src`, `generates`, `mounts` and service images is replaced
with the build-file or task `envs` value. In `cmds`, the shell expands it at run
time from the declared `envs`.
* A dependency whose dependants are all cache hits is skipped. Request a task by
name (or use `--no-skip-deps`) when CI needs its outputs in the workspace.
Every key, including references, includes, labels and caches, is listed in the
[build file reference](/docs/build-file/reference).
## Hermetic tasks [#hermetic-tasks]
The cache replays a task's first result for as long as its declared inputs stay
the same. Keep each task's result a function of those inputs:
* Every file a command reads is in `src` or in a dependency's `generates`.
* `mounts` hold caches and tooling only (`~/.npm`, the docker socket): their
contents are not part of the key.
* Images are pinned to the CI's toolchain version, ideally by digest
(`node:24-alpine@sha256:…`); a moving tag reuses results built with the old image.
* Dependencies are installed from a lockfile in `src`, at pinned versions.
* Build stamps — dates, `git describe`, `$GITHUB_SHA`, build numbers — live in a
small final task with `cache: none` that reads the cached outputs. Inside a
cached task they either change the key on every commit or go stale.
* Credentials are declared only on the tasks that use them: env values are part
of the key, so a rotated token rebuilds every task declaring it.
* Publishing, deploying and anything with external side effects stays in CI, or
runs as a task with `cache: none` on trusted refs only.
The full list of what the cache can't see is in
[caching limitations](/docs/task/caching#limitations-what-the-cache-cant-see).
## Compose test stacks [#compose-test-stacks]
Integration and e2e suites often start a whole `docker-compose.yml`: a database,
a queue, object storage, a stub of an external API, and the app built from the
repository. Each compose service becomes a hammerkit service; the test runner
becomes a task that `needs` them:
| Compose | Hammerkit |
| ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `image:` | `image`, pinned by the digest of the multi-arch index (`docker buildx imagetools inspect`), so every machine pulls its native platform |
| `environment:` | `envs` |
| `command:` | `cmd` (replaces the image's `CMD`; split into words, quotes group them; with `shell: sh` it runs through the shell) |
| `working_dir:`, or the image's `WORKDIR` when the command relies on it | `workdir` |
| `volumes:` with a config file | `mounts` (`./config.json:/etc/app/config.json`) |
| `volumes:` with data | `volumes`, or nothing: a test stack usually starts empty |
| `depends_on: { condition: service_healthy }` | `needs` plus a `healthcheck` on the needed service |
| `healthcheck: test:` | `healthcheck: cmd:` (`CMD-SHELL` checks need `shell: sh` on the service) |
| `healthcheck: start_period`, `retries` | `healthcheck: timeout:` (default `20s`) |
| `ports:` | only for reaching the service from your host; tasks use `name:containerPort` |
| a `build:` of the app under test | a service on the toolchain image with `deps: [install, build]`, running the built app |
| a one-shot setup container (`createbuckets`, `rs.initiate`, migrations) | a task named by the service's [`init`](/docs/service/container#init): runs after the healthcheck passed, and the service is ready for anything needing it only once it succeeded |
Things that behave differently from Compose:
* **A service's working directory is the project directory**, not the image's
`WORKDIR`, so its `src` and dependency outputs are found by relative paths. An
image whose command relies on its own directory needs `workdir: /app`.
* **`cmd` and `healthcheck.cmd` are not run by a shell** unless the service sets
`shell: sh`. Pipes, `&&` and variable expansion need it.
* **Healthchecks start one second after the container, as the user of its main
process,** and repeat about once a second. Use `127.0.0.1` in HTTP probes:
`localhost` may resolve to `::1` while the server listens on IPv4 only.
* **A healthcheck that hasn't passed after `timeout` (default `20s`) fails the
run**, naming the check. Give slow starters (Elasticsearch, a JVM app) a longer
`timeout`, and get each service healthy with `hammerkit up` first, watching its
log.
* **Setup goes into an `init` task, written to succeed when run again.** Services
are recreated on every run (unless kept up with `hammerkit up --daemon`), so the
init runs again too: `mc mb --ignore-existing`, "initiate the replica set unless
it exists".
* **A name resolves only for direct `needs`.** A container reaches the services
it lists itself, not the ones those need. When a service hands out another
service's address (an object store's presigned URLs, a callback URL), every
consumer of that address needs that service too: the test task, and a renderer
downloading from it.
* **A service reaches itself by its name** (it is the container's hostname), so
software that advertises its own address (a MongoDB replica-set member
`mongodb:27017`, Kafka) advertises the service name.
Run several targets against one stack instead of one stack per target. Where the
old pipeline started a Compose project per tenant or test target, keep one database,
queue and storage service and give each target its own database name, key
prefix, vhost or bucket. Only the parts that really differ become one service per
target (`api-a`, `api-b`), and the test tasks per target run in parallel because
nothing orders them.
For a fast loop while iterating, keep the stack up and narrow the run.
`hammerkit up --daemon -f stack=e2e` starts the labelled services once, and a task
that needs them reuses the running containers. A spec filter with a default
(`SPEC: ${E2E_SPEC:-e2e/**/*.cy.ts}`) runs one test:
`E2E_SPEC=e2e/login.cy.ts hammerkit run e2e-a`. Services that stay up keep their
data, so reset the target's database, prefix or bucket at the start of the test
task.
The same services can replace a Compose file for local development: publish the
ports the app on the host uses, give data a named `volume`, and start them with
`hammerkit up --daemon`. Values only some machines have (a secret file path, a
public origin) get a default (`${GEMINI_KEY_FILE:-/dev/null}`), so one build file
serves laptops, workspaces and CI.
## CI template: GitHub Actions [#ci-template-github-actions]
```yaml title=".github/workflows/ci.yml"
name: ci
on:
push:
branches: [main]
pull_request:
jobs:
ci:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write # the remote cache lives in GHCR
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '24'
- run: npm install -g hammerkit@1.9.0 # pin the version
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- run: hammerkit cache pull --remote shared
- run: hammerkit run ci
- run: hammerkit cache push --remote shared
if: github.ref == 'refs/heads/main' # only trusted refs write
- uses: actions/upload-artifact@v4
with:
name: dist
path: dist # exported by the build task
```
Notes:
* Container tasks need a Docker daemon that can bind-mount the checkout. GitHub's
Ubuntu runners have one. On other providers, mount the host's Docker socket; a
Docker-in-Docker daemon must see the checkout at the same path.
* Without a remote cache, drop the login, pull and push lines: the run still
works, it just starts cold on every runner. See
[caching strategy in CI](/docs/guides/ci-caching) for the alternatives.
* Pull requests from forks get read-only credentials; set
`HAMMERKIT_CACHE_READ_ONLY=1` there. Who should write to a shared cache is
covered in [agents, workspaces and CI](/docs/guides/agents-and-ci#who-may-write-to-the-cache).
## Commands [#commands]
| Command | Use |
| ----------------------------------------------- | ---------------------------------------------------------------- |
| `hammerkit run ` | run a task and its dependencies (`hammerkit ` is the same) |
| `hammerkit run -f key=value` | run every task with a label (`-e` excludes) |
| `hammerkit run --dry-run` | the plan with predicted cache hits and misses, running nothing |
| `hammerkit explain ` | why a task is a cache hit or miss |
| `hammerkit validate` | check the build file |
| `hammerkit ls` | list tasks and services as hammerkit resolved them |
| `hammerkit cache pull` / `push --remote ` | move cache entries for the current commit |
| `hammerkit clean` | remove outputs and local cache state |
`hammerkit run` exits non-zero when a task fails. In CI it logs line by line;
`--summary-json` prints a machine-readable summary.
# Migrate your CI with a coding agent (/docs/guides/migrate-ci)
Moving an existing pipeline to hammerkit is mostly mechanical: every CI job becomes
a task, every setup step becomes an image, every cache and artifact step goes away.
That makes it a good job for a coding agent (Claude Code, Cursor, Copilot, …).
The [CI migration guide](/docs/guides/migrate-ci/agent-guide) is written for the agent, not for you: it
walks it through taking stock of your pipeline, writing the build file, rewriting
the CI config and proving the result runs the same checks as before.
## Hand it to your agent [#hand-it-to-your-agent]
Give your agent the repository and this prompt:
```
Migrate this repository's CI to hammerkit. Follow the guide at
https://hammerkit.dev/llms.mdx/docs/guides/migrate-ci/agent-guide/content.md
step by step, including its verification and its final report.
```
If your agent can't read URLs, copy
[`website/content/docs/guides/migrate-ci/agent-guide.mdx`](https://github.com/no0dles/hammerkit/blob/master/website/content/docs/guides/migrate-ci/agent-guide.mdx)
into your repository and point it at the file instead.
You need [hammerkit](/docs/installation) and a running Docker daemon where the agent
works, so it can run the build it writes.
## What you get back [#what-you-get-back]
* a `.hammerkit.yaml` (plus included files in a monorepo) with one task per build,
lint and test step;
* your CI config rewritten to install hammerkit and run those tasks, keeping
triggers, conditions, secrets and deployments where they were;
* `.hammerkit` added to `.gitignore`;
* a report: which job became which task, what was dropped and why, and what still
needs you (secrets, registry permissions, deploy steps).
## Review it [#review-it]
Check the report first, then:
* **Same checks.** Every job of the old pipeline maps to a task or is listed as
deliberately kept in CI.
* **Every input declared.** For each task, the files its commands read are in its
`src` — see [what the cache can't see](/docs/task/caching#limitations-what-the-cache-cant-see).
This is where a migration can go wrong without anything failing.
* **Pinned toolchain.** Images carry the versions your CI used, ideally pinned by
digest.
* **Runs twice.** `hammerkit run ci` twice in a row: the second run executes nothing.
Then try it on a branch and compare the run with the old pipeline before switching
over.
# Monorepo layout (/docs/guides/monorepo)
In a monorepo every package needs the same `install`/`build` steps, and packages
depend on each other in a specific order. Hammerkit handles both with
[includes](/docs/build-file/includes) (reuse a task *definition* in each package's
directory) and [references](/docs/build-file/references) (wire up cross-package
order). This guide mirrors the runnable
[`examples/monorepo`](https://github.com/no0dles/hammerkit/tree/master/examples/monorepo).
## The layout [#the-layout]
```
.hammerkit.yaml # root: build everything
build.npm.yaml # shared: how to install
build.tsc.yaml # shared: how to compile
projects/
a/.hammerkit.yaml # package a
b/.hammerkit.yaml # package b (depends on a)
```
## Shared templates [#shared-templates]
Define each repeated step once. `build.npm.yaml` knows how to install; `build.tsc.yaml`
knows how to compile and itself includes the npm install:
```yaml title="build.npm.yaml"
tasks:
install:
image: node:24-alpine
src:
- package.json
- package-lock.json
generates:
- node_modules
cmds:
- npm ci
```
```yaml title="build.tsc.yaml"
tasks:
build:
image: node:24-alpine
deps: [npm:install]
src:
- tsconfig.json
- src
generates:
- dist
cmds:
- node_modules/.bin/tsc -b
includes:
npm: ./build.npm.yaml
```
Because these are **includes**, when a package uses them the `install`/`build`
commands run in *that package's* directory, against *its* `package.json` — exactly
what you want.
## A package [#a-package]
Each package's build file is tiny: it [extends](/docs/task/extending) the shared
`tsc:build` template and adds its cross-package dependencies. Package `b` depends
on package `a`, so it references `a` and adds `a:build` to its deps:
```yaml title="projects/a/.hammerkit.yaml"
tasks:
build:
extend: tsc:build
includes:
tsc: ../../build.tsc.yaml
```
```yaml title="projects/b/.hammerkit.yaml"
tasks:
build:
deps: [a:build]
extend: tsc:build
references:
a: ../a
includes:
tsc: ../../build.tsc.yaml
```
## Building everything [#building-everything]
The root build file references each package and depends on all of them:
```yaml title=".hammerkit.yaml"
tasks:
build:
deps:
- a:build
- b:build
references:
a: projects/a
b: projects/b
```
```bash
hammerkit build
```
Hammerkit builds the dependency graph across files, runs independent packages in
parallel, builds `a` before `b`, and [caches](/docs/task/caching) each package
separately — change one package and only it (and anything downstream) rebuilds.
## Scoping with labels [#scoping-with-labels]
Add [labels](/docs/labels) to build or test a subset, e.g. label each
package's tasks with `project: a` and run `hammerkit -f project=a`. Combine with
[store / restore](/docs/cli/store-restore) to move outputs between CI jobs that
each build part of the repo.
# Services & networking (/docs/guides/services-networking)
A [service](/docs/service) is a long-running dependency — a database, a
queue, an API — that a task `needs`. This guide covers how a task actually connects
to one, and how services start and stop.
## Lifecycle [#lifecycle]
You don't start services by hand. When you run a task that `needs` a service,
hammerkit:
1. starts the service container,
2. waits for its [healthcheck](/docs/service/container#healthcheck) to pass,
3. runs the task, and
4. stops the service once nothing else needs it.
For development you can keep services up across many task runs with
[`hammerkit up`](/docs/cli/up) (and `--daemon` to background them), stopping them
later with [`hammerkit down`](/docs/cli/down).
## Connecting: container task vs. local task [#connecting-container-task-vs-local-task]
How a task reaches a service depends on whether it runs in a container:
* **Container task** (`image` set): shares a network with the service and reaches
it by the **service name** as hostname, on the service's **container port**. You
do **not** need to publish a port for this.
* **Local task** (no `image`): isn't on that network, so hammerkit injects
`HAMMERKIT__HOST` and `HAMMERKIT__PORT` env vars instead.
The `HAMMERKIT_*` hints are **local-task only** — a container task won't get them,
because it doesn't need them (it uses the service name directly).
## A pattern that works for both [#a-pattern-that-works-for-both]
Put the connection string in a shared top-level `envs` block and reference it from
the task. For a container task, point it at the service name; the same value reads
naturally in your code:
```yaml title=".hammerkit.yaml"
envs:
DATABASE_URL: postgres://api@postgres:5432/api
services:
postgres:
image: postgres:16-alpine
envs:
POSTGRES_USER: api
POSTGRES_DB: api
POSTGRES_HOST_AUTH_METHOD: trust
healthcheck:
cmd: "pg_isready -U api"
ports:
- 5432
tasks:
api:
image: node:24-alpine
needs: [postgres]
envs:
DATABASE_URL: $DATABASE_URL
cmds:
- node index.js
```
For a **local** `api` task (no `image`), build the URL from the injected hints in
your start script instead, e.g. `postgres://api@$HAMMERKIT_POSTGRES_HOST:$HAMMERKIT_POSTGRES_PORT/api`.
## Ports are for your host, not for tasks [#ports-are-for-your-host-not-for-tasks]
`ports` publish a service to your **host** machine — handy for connecting a GUI
client or `psql` to a running database during development. Tasks share the
service's network and can reach every container port directly, so they don't need
`ports` at all. hammerkit publishes them on `hammerkit up`, and during `hammerkit run`
only when a local task needs the service.
## Readiness [#readiness]
Always give a service a `healthcheck` when a task depends on it being usable, not
just started. Hammerkit polls the `cmd` about once a second and only starts the
dependent task once it exits `0`; there are no `interval`/`retries` knobs, and a
check that hasn't passed within its `timeout` (default `20s`) fails the run. See
[healthcheck timing](/docs/service/container#timing-and-failure).
## Persisting data [#persisting-data]
A service container is recreated on each start, so its data is lost unless you
attach a [volume](/docs/service/container#volumes). Use volumes to keep a database
between runs, and [`store` / `restore`](/docs/cli/store-restore) to seed or back up
that data.
## Sharing and Kubernetes [#sharing-and-kubernetes]
Services can be [referenced/included](/docs/build-file/includes) like tasks, so one
database definition is reused across projects. To forward a service from a real
cluster instead of running it locally, see
[Kubernetes services](/docs/service/kubernetes).
# Introduction (/docs)
Hammerkit runs your build steps as **tasks** declared in a `.hammerkit.yaml` file.
Each task runs in the container image it names, declares what it reads and what it
produces, and is skipped when nothing it reads has changed.
```yaml title=".hammerkit.yaml"
tasks:
install:
image: node:24-alpine
src: [package.json, package-lock.json]
generates: [node_modules]
cmds: [npm ci]
build:
image: node:24-alpine
deps: [install]
src: [src, tsconfig.json]
generates: [dist]
cmds: [npm run build]
```
`hammerkit run build` on your laptop is the same build CI runs. Its results can be
shared through a cache, so work done once — anywhere — isn't repeated anywhere else.
## Why hammerkit [#why-hammerkit]
Tools come from the task's image, not from what's installed on the machine.
A task runs again only when its inputs change — locally and in CI.
Agents, laptops and CI runners reuse each other's results through a remote cache.
Point your coding agent at the guide and it rewrites your CI into tasks.
Run a server on Kubernetes and let pi or opencode work on sandboxes there.
# Installation (/docs/installation)
Tasks that declare an `image` run in a container, so you also need a running
container engine such as [Docker](https://docs.docker.com/get-docker/). Check it
with `docker info`.
### Install hammerkit [#install-hammerkit]
```bash
npm install --global hammerkit
```
```bash
brew tap no0dles/hammerkit
brew install hammerkit
```
Run it without installing — always the latest release:
```bash
npx hammerkit --version
```
Download the binary for your platform from the
[latest release](https://github.com/no0dles/hammerkit/releases/latest) — Linux,
macOS and Windows, x64 and arm64 — and put it on your `PATH`. No Node.js needed.
```bash
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
-v "$PWD:$PWD" -w "$PWD" no0dles/hammerkit --version
```
### Describe a task [#describe-a-task]
Create a `.hammerkit.yaml` next to your code. A task names the image it runs in,
the files it reads (`src`) and its commands:
```yaml title=".hammerkit.yaml"
tasks:
hello:
image: alpine:3.21
src: [hello.txt]
cmds:
- cat hello.txt
```
```bash
echo "It's hammer time" > hello.txt
```
### Run it [#run-it]
```bash
hammerkit run hello
```
```txt
task: hello - It's hammer time
Summary:
hello executed 2.3s
1 executed, 0 cached (0% cache hit), 2.3s total
```
### Run it again [#run-it-again]
Nothing in `src` changed, so hammerkit reuses the result instead of running the
task:
```txt
Summary:
hello cached 0ms
0 executed, 1 cached (100% cache hit), 11ms total
```
Change `hello.txt` and it runs again. That's the whole idea —
[caching](/docs/task/caching) explains what counts as an input.
## In CI [#in-ci]
The [container image](https://hub.docker.com/r/no0dles/hammerkit) contains
hammerkit and can be used with Docker-in-Docker (dind) — the usual approach for
container builds on CI systems.
The `no0dles/hammerkit-github-action` action installs hammerkit; it needs
`actions/setup-node` first.
```yaml title=".github/workflows/ci.yml"
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '24'
- uses: no0dles/hammerkit-github-action@v1.3
- run: hammerkit build
```
```yaml title=".gitlab-ci.yml"
build:
image:
name: no0dles/hammerkit
entrypoint: [''] # the image's entrypoint is hammerkit itself
services:
- docker:dind
variables:
DOCKER_HOST: tcp://docker:2375 # the docker:dind service
DOCKER_TLS_CERTDIR: ''
script:
- hammerkit build
```
GitLab runs the job's script through the image's entrypoint, so it has to be
cleared. Tasks run in the `docker:dind` service, which sees the checkout under
`/builds` at the same path, so sources mount as they do locally.
To reuse results between CI runs and machines, see
[caching strategy in CI](/docs/guides/ci-caching).
## Next steps [#next-steps]
# Labels (/docs/labels)
A label is a key-value pair.
Labels can be defined on tasks or on build files.
Labeling your tasks lets you run multiple task groups or exclude specific tasks.
## Example use cases [#example-use-cases]
The following examples show some of the ways labels can be used.
### Separating tasks by area [#separating-tasks-by-area]
Labels can be useful to separate tasks,
for example if you share one build file across different parts of your application, or in a bigger project or monorepo.
Add a `project` label to your tasks to build projects independently with a single command: `hammerkit -f project=a` or `hammerkit -f project=b`.
Label filters are always given as `key=value`. A bare `-f project` (key only) is
rejected. Use `-f` (`--filter`) to keep only matching tasks and `-e` (`--exclude`)
to drop them; both can be repeated or take multiple values.
```yaml
tasks:
build-a:
deps: [install-a]
labels:
project: a
cmds:
- tsc -b ./project/a
test-a:
deps: [install-a]
labels:
project: a
cmds:
- jest
build-b:
deps: [install-b]
labels:
project: b
cmds:
- tsc -b ./project/b
```
### Separate by platform [#separate-by-platform]
Labels can be used to split your tasks across CI runners.
For example, a CI environment might have two runners, one on macOS and one on Linux, and the macOS host should only run the iOS-related code.
Add a `platform=ios` label to the tasks that require macOS, and configure your CI to run `hammerkit -f platform=ios` on the macOS host and `hammerkit -e platform=ios` on the Linux host.
If tasks on the two platforms depend on each other, use [store / restore](/docs/cli/store-restore) to move generated outputs and cache state between the hosts.
Both commands accept label filters too, for example `hammerkit store cache -e platform=ios`.
```yaml
tasks:
build-ios:
labels:
platform: ios
cmds:
- xcodebuild ...
build-api:
image: node
cmds:
- tsc -b
publish-testflight:
deps: [build-ios]
labels:
platform: ios
cmds:
- altool upload-app ...
```
### Group by purpose [#group-by-purpose]
Simplify your workflow by grouping tasks: run one command for multiple tasks that don't otherwise depend on each other.
For example, use a `task` label for developing or releasing.
`hammerkit -f task=dev` starts your API server together with the frontend development build.
`hammerkit -f task=release` pushes your container image to Docker Hub and builds and uploads the frontend to the staging environment.
```yaml
tasks:
frontend:
labels:
task: dev
cmds:
- ng serve
api:
labels:
task: dev
cmds:
- dotnet run
publish-s3:
labels:
task: release
cmds:
- ng build
- aws s3 cp dist/frontend s3://staging-web
publish-api:
labels:
task: release
cmds:
- docker build ...
- docker push ...
```
# Release 1.4.0 (/docs/release-blog/release-1.4.0)
A short retrospective on the changes and findings
## Log Modes [#log-modes]
The new log implementation supports three new modes.
`interactive` will be the default for regular use,
`live` is used if a CI environment is detected.
The default can be changed with the `-l`, `--log` flag.
### Interactive [#interactive]
Print the most recent log for each running task and always see an overview of the overall progress for all dependencies.
Failed tasks get printed in full, after completion.
]
### Live [#live]
Print all logs from all tasks instantly.
Get feedback fast, but makes reading logs harder, if there are many things running in parallel.
### Grouped [#grouped]
Print all logs after a task has finished.
Feedback is delayed, but not interrupted by other tasks.
## Watch improvements [#watch-improvements]
Before 1.4.0 a task could be marked as watchable with `watch: true`.
When executing the task, the source files were watched for changes and the task got restarted on every file change event.
With the new release the `watch` flag can be removed from the build file and instead be passed to the cli.
Every task with source files can be watchable and therefore the flag is not needed anymore.
The new implementation also detects changes on dependent tasks and restarts them if needed.
This change makes it possible to watch multiple build steps and reduces the need for manual interaction.
As a simple showcase, let's take a quick look at a node server application.
It requires some npm packages to be installed.
The package list is defined in the `package.json` file and installed with `npm install`.
After installation the api can be started.
If a new package gets added to the `package.json` it requires another `npm install` before restarting the api to take advantage of the new package.
With the following build file all the mentioned steps are executed by running `hammerkit api --watch`.
When the content of the package.json gets modified, it will first install the npm packages and then restart the api.
Removing the need for any manual interaction.
```yaml title=".hammerkit.yaml"
tasks:
install:
image: node:16.6.0-alpine
mounts:
- $PWD/.npm:/.npm
- $PWD/.config:/.config
cmds:
- npm install
generates:
- node_modules
src:
- package.json
- package-lock.json
api:
image: node:16.6.0-alpine
deps: [install]
ports:
- 8080
src:
- src
cmds:
- node src/index.js
```
## Docker performance improvements [#docker-performance-improvements]
The overall performance on file heavy tasks improved a lot by switching generated files from mounts to volumes.
Tasks like `npm install` with several thousand files led to unresponsiveness in the Docker API and required restarts.
Debugging and reading the docker-for-mac/windows [user manual](https://docs.docker.com/desktop/mac/#file-sharing) led to the conclusion **source files get mounted, generates use volumes**.
The change improved the stability and execution time, but required some changes to the store/restore functionality.
## Moved cache location [#moved-cache-location]
Hammerkit keeps track of all used source files and task definitions.
This is used to identify, if a task can be skipped because the source has not changed and therefore the old output should still be the same.
These cache files were located in the same directory as the build file in a `.hammerkit` subfolder.
Those files needed to be excluded from git, build steps and IDEs, because they were usually located on the same level as the source code.
To reduce those kind of issues, they were moved into the user directory.
Removing the need for special treatments.
## Automated test runner for macOS and Linux [#automated-test-runner-for-macos-and-linux]
To fully test hammerkit, a running docker daemon is required.
For macOS and Linux there is now a GitHub Action that tests all `examples` in the repository for every merge request.
Windows is hopefully coming soon as well, but even though there are Windows runners available on GitHub, the docker setup requires still some work.
## Update node to v16 and make use of the AbortController [#update-node-to-v16-and-make-use-of-the-abortcontroller]
During testing hammerkit on Linux, some examples ended in an `unhandledPromise` error.
After some investigation, the reason could be pinned down to the usage of the `Defer` class, that was used a lot in hammerkit.
The Defer class was a wrapper, to reduce repeating code handling callback results.
The issue was that the `Defer` class created a promise with the default constructor without adding a catch handler.
Since the `Defer` pattern is not a recommended approach anymore (very well explained [here](https://stackoverflow.com/questions/34971078/how-to-replace-promise-defer-with-new-promise)) and in newer versions of node `unhandledPromise` result in process failures, the removal was a no-brainer.
The `Defer` class was mainly used for callback functions and the cancellation logic of tasks.
A Defer got resolved when the user pressed Ctrl-C and the task execution handler ended the task early when that promise got resolved.
The `AbortController` was the perfect replacement and has just made it into the [Web Spec](https://dom.spec.whatwg.org/#interface-abortcontroller) and is available in node environments since [v15](https://nodejs.org/api/all.html#globals_class_abortcontroller).
## Adding best practices [#adding-best-practices]
In the repository a new folder `best-practices` got added with tooling specific examples.
The build files are intended to be project-independent and showcase how popular build tools should be used together with hammerkit.
## Next release [#next-release]
The next release will focus on adding a service concept.
Services will be a solution to allow long-running tasks like databases to be connected with the existing tasks.
The goal will be to find an efficient way of spinning up and shutting them down, so they only run, when they are needed by a task actively.
That will require also a concept of how to deal with readiness detection of services.
# Release 1.5.0 (/docs/release-blog/release-1.5.0)
A short retrospective on the changes and findings
## Services [#services]
Services have been added to assist your tasks.
They can be used to start up databases and other service applications and get attached to the task if needed.
They are similar to tasks but usually are continuously running in the background.\
Executing tasks that require a service, will trigger a service to start.
As soon as they are ready the task can execute.
When there are no pending tasks left that need the service, they will be terminated.
There are two types of services hammerkit supports:
### Container Services [#container-services]
Similar to [docker-compose](https://docs.docker.com/compose/) but offer some further advantages:
* Ensure services are ready before your tasks use them with healthchecks.
* Automatic DNS resolution between container task and services
* Dynamic start/termination of services depending on the need
* Watch mode for automated restarts on file mounts changes
```
services:
postgres:
image: postgres
healthcheck:
cmd: "pg_isready -U postgres"
ports:
- 5432
tasks:
api:
image: node:alpine
needs: [postgres]
cmds:
- node index.js
```
### Kubernetes services [#kubernetes-services]
Use `kubectl port-forward` to connect pods, services and deployments from Kubernetes to your task.
Enabling easy integration with staging/development environments and offload the need to run everything on one machine.
```
services:
postgres:
context: staging
ports:
- 5432:5432
selector:
type: deployment
name: postgres
tasks:
api:
image: node
needs: [postgres]
cmds:
- node index.js
```
## Labels [#labels]
Labels got added to categorize tasks and services.
Allowing to group or divide tasks from each other.
The cli commands [exec](/docs/cli/execute), [store/restore](/docs/cli/store-restore), [clean](/docs/cli/clean), [validate](/docs/cli/validate) and [ls](/docs/cli/ls) support label arguments to limit the scope.
The new `--filter, -f` option reduces the scope to tasks/services only matching the specified label.
For example `hammerkit clean --filter project=a` will only run the clean command on tasks that have the label `project=a` or are a dependency of such a task.
The other new `--exclude, -e` option works the opposite way and reduces the scope to task/services that do not have a matching label.
For example `hammerkit -e project=a` will execute all tasks that do not have a label `project=a`.
Further examples and use cases can be found [here](/docs/labels).
## CLI start performance [#cli-start-performance]
The cli responsiveness has been improved by reading tasks and service definition as late as possible.
Before the build file was parsed on startup including all references/includes.
This was needed for the preparation of the cli and help page that displayed all available tasks.
With the new redesign it's not needed until the moment of execution of a specific task or the new [ls command](/docs/cli/ls).
# Release 1.6.0 (/docs/release-blog/release-1.6.0)
A retro about the changes in 1.6.0. This release focuses on sharing work across
machines: caches can now live in a remote backend, and the same build file can run
on the local docker daemon or on a Kubernetes cluster.
## Remote caching with pluggable backends [#remote-caching-with-pluggable-backends]
Until now the cache state lived only on the machine that ran a task. With 1.6.0 a
cache can be backed by a remote **backend**, so a task built on one machine can be
restored on another - perfect for sharing results between developers and CI.
Caches are declared in a top-level `caches:` block with a method and a backend.
Two backends are built in: `local` (a shared filesystem directory, used by the
built-in `default` cache) and `s3` (AWS S3 and any S3-compatible service like
MinIO, R2 or Wasabi).
```yaml
caches:
remote:
method: checksum
backend:
type: s3
bucket: my-build-cache
region: eu-central-1
tasks:
build:
cache:
name: remote
src: [src]
generates: [dist]
cmds:
- tsc -b
```
Hammerkit pulls a result from the backend before a task runs if it is missing
locally, and pushes it after a successful run. Backend errors never fail the build,
they only produce a warning. More details in [caches](/docs/build-file/caches).
## Checksum is now the default everywhere [#checksum-is-now-the-default-everywhere]
Until now the cache method defaulted to `modify-date` locally and only `checksum`
in CI. With remote backends that split no longer makes sense: a result is shared
between machines, and `modify-date` compares file modification times, which a
`git clone` resets on every fresh checkout — so a developer's `modify-date` result
would never line up with a CI runner's. The whole point of an S3 backend is to
reuse work *across* machines, and only `checksum` (which compares file content) is
stable across them.
So `checksum` is now the default in **all** environments. The same comparison runs
locally and in CI, so a result cached on one is correctly reused on the other.
This is a behavior change, almost a breaking one: if you relied on the local
`modify-date` default (for example to avoid hashing very large source folders), set
it explicitly with `cache: modify-date` on the task or `--cache modify-date` on the
command. See [caching](/docs/task/caching#checksum-vs-modify-date).
## Run on Kubernetes [#run-on-kubernetes]
A new runtime abstraction lets the same build file run on the local docker daemon or
on a Kubernetes cluster. Declare an `environments:` block and select it with
`--env `. Tasks run as jobs, container services run as deployments, and a
service healthcheck is turned into readiness and liveness probes.
```yaml
environments:
default:
kubernetes:
context: docker-desktop
```
```bash
hammerkit api --env default
```
The store and restore commands accept `--env` too, so results can be stored from or
restored to a cluster. See [running on Kubernetes](/docs/task/kubernetes).
## Service improvements [#service-improvements]
Services got more capable in this release:
* Services can declare `deps` and `needs`, just like tasks.
* Multiple services can run in parallel and be referenced across projects.
* The new `up` and `down` commands start and stop services directly.
* Local tasks receive the connection details of needed services as environment
variables (`HAMMERKIT__HOST`, `HAMMERKIT__PORT`), since they are not
part of the container network.
For [Kubernetes services](/docs/service/kubernetes) the port-forwarding no longer
requires the `kubectl` binary, and a `namespace` can be set per service.
## Exporting build outputs [#exporting-build-outputs]
Generated files can be marked with `export: true` to copy them back into your
workspace after a task ran, so other tools can pick them up.
```yaml
tasks:
build:
image: node:alpine
generates:
- path: dist
export: true
cmds:
- tsc -b
```
## Reliability [#reliability]
A few robustness improvements landed as well: build graphs that mix `deps` and
`needs` in a cycle are now detected instead of deadlocking, local tasks use a pid
file to prevent concurrent runs of the same task, and a crashing docker service is
reported as a crash instead of terminating silently.
# Release 1.7.0 (/docs/release-blog/release-1.7.0)
A retrospective on 1.7.0. Release 1.6.0 made caches shareable through remote
backends; this release makes that sharing actually work across machines, and
makes it cheap. The goal: work done anywhere — on a laptop, in a coding agent's
sandbox or on a CI runner — is never repeated anywhere else.
## Cache keys no longer depend on where you check out [#cache-keys-no-longer-depend-on-where-you-check-out]
A task's cache key is a hash of its definition (commands, image, env, paths) and
its inputs. Until now the paths in that hash were absolute. The same commit
checked out at `/home/runner/work/app/app` on a CI runner and at `/workspace/app`
in a sandbox therefore produced *different* keys, and a shared remote cache never
hit between them.
In 1.7.0 every path in the key is relative to the **project root** — the git root,
or the directory of the main build file. The same commit now has the same keys on
every machine. Machine-local state (Docker containers, staging directories) stays
scoped to each checkout, so two worktrees on one machine share cached results but
never step on each other's running containers.
Every cache key changes once with this release, so the first run after upgrading
is a full rebuild.
## Use the registry you already have [#use-the-registry-you-already-have]
The new `registry` backend stores cache entries in any OCI container registry —
GHCR, Docker Hub, ECR, GAR, Artifactory or a plain `registry:2`. There is no bucket
to provision: use the registry you already push images to, with the credentials
`docker login` already set up.
```yaml
caches:
shared:
method: checksum
backend:
type: registry
repository: ghcr.io/my-org/hammerkit-cache
```
Each entry is a small single-layer image tagged `-`. It talks to
the registry directly (no Docker daemon or extra tools involved) and writes the
manifest last, so a half-finished upload is never visible to anyone else. See
[caches](/docs/build-file/caches#registry).
## `cache pull` and `cache push` [#cache-pull-and-cache-push]
With 1.6.0, a task whose cache was remote talked to the bucket during the build.
1.7.0 adds a second way that keeps the network out of the build entirely: tasks
cache to the machine-local default cache, and two commands move entries between
that and a remote cache, without running anything.
```bash
hammerkit cache pull --remote shared # warm the workspace: network only
hammerkit run # build: compute only
hammerkit cache push --remote shared # publish what was built
```
`cache pull` fetches exactly the entries the current commit will look up,
including all dependencies. That makes it the fastest way to set up a new
workspace. `cache push` publishes results even when they were never written to
the local cache. Both are safe to re-run. See [cache pull / push](/docs/cli/cache).
## Don't rebuild what nothing needs [#dont-rebuild-what-nothing-needs]
A cache hit used to save only the task itself. When CI pulled the result of an e2e
job, the e2e task was skipped — but the app build it depends on still ran, because
that build's own result wasn't on the machine. Nothing in the run used it.
Dependencies now run only when a task that needs them has to run. A task pulled into
the run purely as a dependency waits until something depending on it misses the
cache; if everything depending on it is a hit, it is **skipped** and shows up as
such in the build summary. The decision follows the real cache outcome rather than a
prediction, so a restore that fails still gets its dependencies built. Tasks you
name or select with labels always run, and `--no-skip-deps` restores the old
"run the whole graph" behavior. See
[dependencies of cached tasks](/docs/task/dependencies#dependencies-of-cached-tasks).
## Keep shared caches small [#keep-shared-caches-small]
Once every CI run and every workspace pushes to a shared cache, it grows without
bound. Caches can now declare a **retention** policy:
```yaml
caches:
default:
method: checksum
backend:
type: local
retention:
maxAge: 30d
maxSize: 20Gi
keepPerTask: 3
```
`hammerkit cache ls` shows what a cache holds, and `hammerkit cache prune` applies
the policy (or one given on the command line, with `--dry-run` to preview). A local
cache with a policy prunes itself after every successful run; remote caches are only
pruned when you name them with `--remote`. The local cache tracks when each entry
was last used, so what you still use stays. See
[retention](/docs/build-file/caches#retention).
## Timeouts [#timeouts]
A hanging test used to block a CI job until the provider killed it. Tasks can now
declare a `timeout` (and `--timeout` sets a default for all of them):
```yaml
tasks:
e2e:
image: cypress/included:13.15.0
timeout: 15m
cmds:
- cypress run
```
A task that runs longer fails with `timed out after 15m`, its process, container or
Kubernetes job is cleaned up, and nothing is written to the cache. See
[timeouts](/docs/task#timeouts).
## Agents, workspaces and CI share one cache [#agents-workspaces-and-ci-share-one-cache]
Put together, these change how CI and coding agents can work together. An agent
that ran the e2e suite on a commit in its sandbox pushes the result; CI pulls it and
the e2e job completes as a cache hit in milliseconds. The database it needs isn't
even started. A new workspace pulls the dependency install and the build instead
of redoing them, so only what actually changed runs.
We tried this on a small stand-in project (`npm install` → build → an e2e job
needing Postgres) across three simulated machines and a local registry:
| Scenario | Result |
| -------------------------------------------------------- | -------------------------------------------------------- |
| Agent sandbox, cold run + `cache push` | 3 tasks executed, 3 entries pushed |
| CI runner (other machine, other path) after `cache pull` | 0 executed, 100% cache hits, database not started |
| New agent workspace, `cache pull build` | install and build restored; only the edited e2e task ran |
Who may write to a shared cache is a policy decision: a cache hit means CI trusts
the result. The new **read-only mode** (`--cache-read-only` or
`HAMMERKIT_CACHE_READ_ONLY=1`) lets a runner restore from the cache while never
writing to it. That fits agent sandboxes you don't fully trust, or pull requests
from forks. The [agents, workspaces and CI guide](/docs/guides/agents-and-ci)
walks through the setup and the trade-offs.
## See why something ran [#see-why-something-ran]
Caching is only useful if you can tell why it didn't hit. 1.7.0 adds:
* **[`hammerkit explain`](/docs/cli/explain)** — for each task, whether the next
run is a cache hit and which input caused a miss: a changed source file, a
changed dependency, a changed command or image. `--json` for tooling.
* **`run --explain`** — the miss cause printed inline for every task that rebuilds.
* **`run --dry-run`** — the execution plan in order, with predicted hits and misses,
without running anything.
* **Build summary** — after every run, a line per task (executed or cached, and how
long it took) plus the cache hit ratio; `--summary-json` for dashboards and
agents.
* **[`hammerkit graph`](/docs/cli/graph)** — the task and service graph as mermaid
or dot.
## Fixes that mattered [#fixes-that-mattered]
Running the new setup end to end surfaced several bugs, all fixed in this release:
* **Container task outputs restored from a cache were lost.** They were written into
a throwaway container instead of the task's volume, so the task reported a cache
hit while its outputs were empty. This affected remote caches and
`store`/`restore`.
* **A cached task that needs a service failed the run.** The service was left
waiting and then cancelled. Now it isn't started at all.
* **A service that fails to start hung the run forever** (a missing image, a port
already in use). The task waiting for it now fails right away.
* **Tasks without `src` were cached after one run.** As documented, a task without
`src` — and everything depending on it — now always runs, since hammerkit can't
prove it's up to date.
* **Cache hits after a change.** Before relying on a shared cache in production we
went looking for false hits — a task reported as cached although an input it
reads had changed — and found several: globs like `src/**/*.ts` ignored every
subdirectory, a dependency's command or env could change without rebuilding the
tasks depending on it, deleted outputs stayed "cached", binary sources were
compared as text, and a `src` typo kept a task cached forever. All are fixed, and
the cache contract is now guarded by an invariant suite (every observable change
is a miss, every unobservable one a hit), a property test over random file trees
and globs, and mutation testing. See the [changelog](/changelog#v1.7.0).
* **Tasks on Kubernetes didn't wait for their jobs.** A task running as a
Kubernetes job was reported as completed as soon as the job was created, and a
failing job was never detected. Tasks now wait for the job, fail when it fails,
and delete it when the run is cancelled or times out.
* **S3 errors looked like cache misses.** An unreachable bucket or bad credentials
are now reported as errors by `cache pull`/`push`; builds still degrade to a
miss with a warning.
## Next release [#next-release]
Next up are source exclusions (`!` entries in `src`, hidden from container tasks
too), first-class secrets for tasks and services, and container runtime options
such as `--shm-size` for browser-based tests. See the
[roadmap](/docs/contribution/roadmap).
# Release 2.0.0 (/docs/release-blog/release-2.0.0)
Since 1.24 a hammerkit server runs the tasks of a published build file for people and CI,
with the credentials of service accounts instead of anyone's own. 2.0.0 makes that server
the place where coding agents work: they get a sandbox with the repository, run the same
tasks you run, start a preview of their change behind your sign-in, and leave checks behind
that CI can trust instead of running everything again. The server runs on one Docker host or
on a Kubernetes cluster, and you can watch it from a dashboard.
The version goes to 2.0 for one breaking change in build files: `continuous` is for
services only. The [changelog](/changelog#v2.0.0) lists every change and what to do when
you upgrade.
Remote compute is experimental: its config and API can still change between releases.
## Sandboxes for coding agents [#sandboxes-for-coding-agents]
An agent that writes code needs somewhere to run it. Giving it your laptop gives it your
credentials; giving it a CI runner makes it wait for a pipeline. A
[sandbox](/docs/remote-compute/sandboxes) is a copy of a repository on the server that an
agent reads, searches and edits through the server's API, runs the build file's tasks on,
and pushes to a branch it is allowed to push to. The tasks run as they would anywhere else,
with the build file that was published, so an agent can't change a task's commands, image or
the secrets it declares; what the task runs from the sandbox's files is the agent's work.
The [agent clients](/docs/remote-compute/agents) bring this into the agent's own tools:
`hammerkit remote mcp` for any MCP client, an extension for [pi](/docs/guides/agents/pi)
and a plugin for [opencode](/docs/guides/agents/opencode). The guides start from nothing
and end with an agent that runs your tests on your server.
## Previews behind your sign-in [#previews-behind-your-sign-in]
A [service run](/docs/remote-compute/service-runs) starts the services of a build file on
the server, with endpoints at their own addresses behind the server's sign-in, so a
reviewer opens the app an agent just changed instead of reading its diff. A service run
sleeps when nobody uses it and keeps its data while it sleeps. When the branch moves on,
`hammerkit remote up` with the same name moves the run to the new commit in place: same
addresses, same data, and only the services that changed restart.
Runs hand their tasks [the repository, branch and commit](/docs/remote-compute/runs#executing-a-run)
they run from, so a preview can deploy per branch while it reads a secret.
## Checks CI can trust [#checks-ci-can-trust]
When an agent ran the tests of a commit on the server, CI shouldn't have to run them
again. A check task that passed on the server is recorded by its cache key, and
`hammerkit remote verify` in CI only asks the server whether the checks for the commit's
files passed. The [verified checks guide](/docs/guides/agents/verified-checks) sets it up
for GitHub Actions and GitLab CI.
## A Docker host or a Kubernetes cluster [#a-docker-host-or-a-kubernetes-cluster]
A server runs its tasks on the Docker host it runs on, or as Jobs on a
[Kubernetes backend](/docs/remote-compute/backend#kubernetes): a namespace per caller,
claims kept per branch so the next run starts warm, network policies, and quotas that say
why a pod waits. Several runs of one caller go side by side, and the tasks of a run start
[longest chain first](/docs/task/dependencies#which-ready-task-runs-first), weighted by how
long they took before.
What a run may reach is the operator's call: per-task arguments, network ceilings for tasks
that read secrets, hosts a provider may reach through the egress proxy, and Linux
capabilities only in a sandboxed runtime such as gVisor.
## Image builds in a task [#image-builds-in-a-task]
A task with [`build`](/docs/task/image-builds) builds a Dockerfile with BuildKit, the
builder behind `docker build`: tags, push, a registry layer cache, logins from secrets and
several platforms under one name. The same task builds on your laptop, in CI and on a
Kubernetes server, where no task gets a Docker daemon.
[Building container images](/docs/guides/container-images) sets a server up for it.
## Seeing how a server stands [#seeing-how-a-server-stands]
`hammerkit remote status` tells you where your run is in the queue and why it waits,
what counts against your limits, and how long your tasks usually take. A server with a
sign-in also serves a [dashboard](/docs/remote-compute/status#dashboard) with runs, their
graph, timeline and logs, build file versions and proposals to publish or reject,
sandboxes, and a button to retry a run that failed.
## The breaking change: `continuous` is for services [#the-breaking-change-continuous-is-for-services]
A process that keeps running belongs in a service, which hammerkit starts, health-checks
and stops. In 2.0 a task with `continuous` is refused with a message that names it. Make it a container service with
[`continuous: true`](/docs/service/container#continuous): its `cmds` become one `cmd`, and
it keeps its `src` and `ports`. `up --watch` starts it once, and on a service run that moves
to a new commit it gets the changed files and reloads.
## Upgrading [#upgrading]
Every cache misses once after the upgrade, and `hammerkit down` now only stops services.
Upgrade a server before its clients, since a 2.0 client sends a timeout a 1.26 server
refuses. The [changelog](/changelog#v2.0.0) has the full list.
# Accounts (/docs/remote-compute/accounts)
A service account is a name the operator defines for one set of credentials, such as the
test or the production key of a team. The credentials are files and variables of the
operator's machine, they never reach the config: an account gives each secret provider
command the environment that makes it act as that account.
```yaml title="server.yaml"
secretProviders:
gsm:
command: [gcloud, secrets, versions, access, latest, '--secret={{ref}}']
accounts:
op-payments-dev:
description: Payments, test secrets
providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/payments-dev.json } } }
op-payments-prod:
description: Payments, production
providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/payments-prod.json } } }
source: [ref] # only from reviewed code
refs: [main, 'release/*'] # and only from these branches
scm-read: # reads the repositories, no entitlement may name it
description: Read-only access to source repositories
providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/scm-read.json } } }
```
| Key | What it does |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `description` | Shown to callers in `GET /v1/accounts` and `remote accounts`, up to 200 characters. |
| `providers` | Per secret provider, the `env` its command runs with when it reads a secret as this account. The provider has to be declared under `secretProviders`. |
| `kubernetes` | On a [Kubernetes backend](/docs/remote-compute/backend#secrets-in-the-job), the credential the Job of a task gets to read the account's secrets: a token of a ServiceAccount, the files of a Secret, or both. |
| `source` | Where the code of a run using the account may come from, see [Restricting the code](#restricting-the-code). |
| `refs` | The branches or tags such a run may start from. |
An account name starts with a letter and holds letters, digits, `-` and `_`.
[Entitlements](/docs/remote-compute/entitlements) say who may `use` or `request` an
account. An account a [source](/docs/remote-compute/sources) reads repositories with is the
server's own: no entitlement may name it and no build file may read a secret as it.
## Secret providers [#secret-providers]
`secretProviders` are the commands that read secrets, the same as the `secretProviders` of
a build file, see [Secret providers](/docs/task/secret-providers): a `command` as a list of
arguments, run without a shell, where `{{ref}}` is replaced by the reference of the secret,
an optional static `env` and a `timeout` (`30s` by default). Only the operator declares them;
a published build file can not.
## How a run gets its accounts [#how-a-run-gets-its-accounts]
The secrets in a published build file say which account each is read as (`account:` on the secret), and the server
derives from that which accounts a run uses: a task without secrets uses none and holds
no credentials. Nothing in a run request, no default and no `account` on a task or
service chooses one, so the person who asks for a run cannot pick the account they
prefer.
```yaml title=".hammerkit.yaml"
tasks:
test:
cmds: [npm test] # no secrets, no account
deploy:
secrets:
- from: gsm:kube-token
account: op-payments-prod
env: KUBE_TOKEN
cmds: [./deploy.sh]
```
## Restricting the code [#restricting-the-code]
An account can limit where the code of a run that uses it comes from. `source` lists the
kinds a run may use, `upload` for files the caller sends from their working tree, `ref`
for a git ref of a repository the server reads itself (see [Sources](/docs/remote-compute/sources)),
and `sandbox` for a [sandbox](/docs/remote-compute/sandboxes) of the caller's.
`refs` narrows `ref` to some branches or tags. These are the combinations:
| `source` | `refs` | A run that uses the account may come from |
| ---------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| left out | | an upload or any ref, not a sandbox. The default, for accounts without credentials that matter. |
| `[upload, ref]` | | the same, written out. |
| `[ref, sandbox]` | | any ref, or a sandbox. A sandbox holds changes nobody reviewed, so an account takes its runs only when it lists `sandbox`. |
| `[ref]` | | any branch, tag or 40-digit commit of a repository the caller may name. |
| `[ref]` | `[main, 'release/*']` | only those branches or tags. `*` matches within one level, so `release/*` takes `release/1.2` but not `release/1/2`; a last `**` takes every level below, so `release/**` takes both. A commit hash is refused, because it names no branch. |
| `[upload]` | | an upload only. |
`refs` is only accepted together with `source: [ref]`: an upload names no ref, so the server
refuses a config that sets `refs` with `upload` in `source` or without `source`.
The server does not execute runs from an upload yet (`501`), so today every run is from a
ref, whatever `source` says. An account with `source: [upload]` can not be used until
uploads are supported. Set `source: [ref]` on the accounts that hold production
credentials now, so they stay limited to reviewed code once uploads arrive.
```yaml title="server.yaml"
accounts:
op-payments-prod:
providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/payments-prod.json } } }
source: [ref]
refs: [main, 'release/*']
```
A run that breaks either rule is refused with `403` when it is submitted. Approving a run
or a build file does not approve the code its commands execute, so an account that matters
should be limited to `source: [ref]` and the branches that are reviewed.
## Secrets on a server [#secrets-on-a-server]
Because the account is chosen by the published file, a server applies stricter rules
to secrets than a local run does: a provider secret names its account (`account:`
is required, a task's `account:` and `default: true` are not used), `file:` secrets
are refused, and an `env:` secret may only read a variable listed in `runs.env.allow`.
A build file that declares `secretProviders` or `secretAccounts` is refused, and so is
a project `.env` file read: the operator's providers and accounts are the only ones.
# Agents (/docs/remote-compute/agents)
Experimental. The tools and settings can still change between releases.
A coding agent needs to read, edit and search files and to run commands. On a
[sandbox](/docs/remote-compute/sandboxes) the files stay on the server and the commands run
there, as published tasks in their containers. hammerkit connects agents to it three ways:
| Client | What it gets |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`hammerkit remote mcp`](#hammerkit-remote-mcp) | MCP tools for any client that speaks MCP: Claude Code, Claude Desktop, Cursor, a custom agent. |
| [pi](#pi) | pi's own `read`, `write`, `edit`, `bash`, `ls`, `find` and `grep`, working on the sandbox, and `check` and `push`. |
| [opencode](#opencode) | opencode's own `read`, `write`, `edit`, `patch`, `shell`, `grep` and `glob` (`apply_patch`, `bash` and `list` on opencode 1), working on the sandbox, `run_task`, `check`, `run_logs` and `push`. |
pi and opencode keep the tools their models are tuned for, with the same names, parameters
and output: only where they read, write and run moves. The MCP tools take the shapes models
are trained on: numbered reads with an offset and limit, an exact string replace, an
`apply_patch` body or a git diff, ripgrep's parameters and a shell command.
## The shell task [#the-shell-task]
There is no command endpoint on the server: commands bypassing the published build file
would bypass its image, network and accounts. A shell is a published task whose last command
is a shell, and every command of the agent is its [argument](/docs/task/arguments):
```yaml title=".hammerkit.yaml"
tasks:
sh:
image: node:24
mounts: [.]
args: true
cmds: [sh -c]
network: { allow: [registry.npmjs.org] }
timeout: 5m
```
`mounts: [.]` puts the files of the sandbox into the container, writable, as the task's
working directory. `args: true` marks it as a task that takes arguments, which a server with
[`runs.args: declared`](/docs/remote-compute/runs#arguments) requires. Without it a command in a container runs in an empty directory: the
clients then refuse to run it and say the shell task needs `mounts: [.]`. The
[`timeout`](/docs/task#timeouts) stops a command that hangs, such as a test that never ends,
before it holds the sandbox for the server's own limit. The clients pass the agent's own
timeout of a command too, and the command stops after the shorter of the two.
`bash` with `npm test -- -t login` then runs `sh -c 'npm test -- -t login'` in that image,
on the files of the sandbox. Name the task `sh` or tell the client its name. Use `bash -c`
when the image has bash. A task that reads secrets takes no arguments on a server, so the
shell holds no account. Its network is the lower of its own `network`,
[`runs.network`](/docs/remote-compute/runs#config) and
[`sandboxes.network`](/docs/remote-compute/sandboxes#config).
### Named tasks instead of a shell [#named-tasks-instead-of-a-shell]
A shell runs whatever the model types, past the guardrails a repository puts into its named
tasks: a `test` task with `--test-timeout=60000` does not help when the model runs
`node --test` itself, and a test that hangs holds the sandbox. `run_task` runs a published
task by its name, with arguments for its last command:
```yaml title=".hammerkit.yaml"
tasks:
test:
description: run the unit tests, args are test files or --test-name-pattern
image: node:24
mounts: [.]
args: true
timeout: 10m
cmds: [npm ci, node --test --test-force-exit --test-timeout=60000]
lint:
description: lint the sources
image: node:24
mounts: [.]
args: false
cmds: [npm ci, npx eslint .]
```
The description of `run_task` lists the tasks of the published version, so the model knows
what it can run without guessing: each with its `description` (its last command when it has
none) and whether a run of it takes arguments, as the server's runs see it. A task takes none
when it says `args: false`, when it or anything in its run reads a secret or a variable of the
server, and under [`runs.args: declared`](/docs/remote-compute/runs#arguments) when it does not
say `args: true`. The server still checks every run. The clients read the list of the version
`HAMMERKIT_BUILD` names when they start, and wait up to 15 seconds for it; without an answer the
tools work, and the description says the list is missing.
Set [`HAMMERKIT_SHELL`](#settings) to an empty value and the build file needs no shell task:
`hammerkit remote mcp` has no `bash` then, and the opencode plugin no shell (on opencode 1
its `bash` refuses, see [opencode](#opencode)). pi has no `run_task` yet and its `bash` refuses
without a shell, so keep a shell task for pi.
Leaving the shell out is a setting of the client, which whoever starts the agent picks. To keep
agents off a shell, publish a version without a shell task, or with `args: false` on it:
`run_task` runs every task of the version, a shell task included, and the server runs only what
the version has.
What follows from running a command as a task, a shell command or a named one:
* Commands run one at a time, and only the files are kept between them. A background process
ends with its command, and `cd` or `export` in one command does not reach the next. On a
host each command gets a fresh container. On
[Kubernetes](/docs/remote-compute/sandboxes#on-kubernetes) the commands of a sandbox run in
one container that stays, so a command starts in about a second instead of waiting for a pod:
before and after each command every process in it is killed and `/tmp` and `/dev/shm` are
emptied, but what a command writes elsewhere outside the sandbox, such as the home directory,
can still be there for the next one. The clients run a command in a directory with `cd || exit 1` on a
line before it.
* The command reads no input and gets none of the agent's variables.
* While a command runs, the sandbox is held: an edit in the meantime is `409`. Agents send
tool calls in parallel, so each client takes the calls of its model one at a time, in the
order the model made them: an edit sent along with `npm test` lands before the tests start.
A command of another client still holds the sandbox, and an edit then is refused.
* Agents on sandboxes of their own run their commands side by side, also when they sign in
as one identity, up to [`runs.limits.running`](/docs/remote-compute/runs#limits-and-restarts)
of that identity (2 by default); a server for more agents raises it.
* The timeout an agent gives a command is sent as the run's
[`maxTimeout`](/docs/task#capping-every-task): the command stops after the shorter of it and
the shell task's own `timeout`, with `timed out after `. Each task of the run gets
at most this long, so a dependency of the shell task counts on its own. On Kubernetes the
first command of a sandbox also waits for its pod to start within that time. A `timeout` on
the shell task bounds the commands that come without one, such as a test that never ends.
* A command starts in the root of the sandbox. The paths of the agent's machine do not exist
there, which the clients tell the model.
* What it printed comes from the [log of the run](/docs/remote-compute/runs#logs). A long
log is cut at `runs.maxLogBytes`; the last lines of the task are kept past it.
## Sign-in [#sign-in]
The clients send the token of the agent, not of a person, to the server registered in the
host config ([`hammerkit remote add`](/docs/cli/remote)), and nowhere else. They look for it
in this order:
| Source | What it is |
| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `HAMMERKIT_TOKEN_COMMAND` | A command that prints a short-lived access token of the agent's own identity, for example a service account or workload identity of the identity provider the server accepts. It runs again when the server stops accepting the token, so a long session outlives one token. |
| `HAMMERKIT_TOKEN` | Such a token, as the agent's platform hands it over. |
| `remote add --token-command` | The same as `HAMMERKIT_TOKEN_COMMAND`, registered with one server in the host config, see [A token command per server](/docs/cli/remote#a-token-command-per-server). |
| `hammerkit remote login ` | The sign-in stored on the machine, for an agent a person starts on their own machine. |
Give the agent's identity an [entitlement](/docs/remote-compute/entitlements) with its
repositories and branches under `sandboxes`, `builds: run`, and no account that holds
production secrets. `[{ repo: github.com/corp/*, branches: { main: pull, "agent/**": push } }]`
lets it start from `main` and push to `agent/**` with the [`push`](#taking-the-work-out) tool;
with only `pull` it hands back patches. `repos: [github.com/corp/*]`, with a branch to push to,
lets it [check](#checking-before-the-push) its work.
## Settings [#settings]
All three clients take the same settings, `hammerkit remote mcp` as options too:
| Variable | Option | What it names |
| --------------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `HAMMERKIT_SERVER` | `--server` | The registered server. Default: the one `hammerkit remote use` set, else the only registered one. |
| `HAMMERKIT_SANDBOX` | `--sandbox` | A sandbox of the agent, for example `sbx_4vJ0mT2cX9qLw1aB`. |
| `HAMMERKIT_REPO`, `HAMMERKIT_REF` | `--repo`, `--ref` | Instead of a sandbox: one of this repository at this branch. When the client starts, it takes a sandbox of yours of the same repository and branch that has no changes and checks it out at the branch again, or creates one. A sandbox stays until it is deleted and the server allows only a few, so set `HAMMERKIT_SANDBOX` to the id the client prints to come back to work in progress. |
| `HAMMERKIT_BUILD` | `--build` | The hash of the published version of the build file whose tasks run on the sandbox, as `hammerkit remote publish` printed it. A run names exactly the version it runs: when the build file changes, publish it again and set the new hash. |
| `HAMMERKIT_SHELL` | `--shell` | Its shell task, `sh` by default. Empty for none: the clients then leave their shell tool out and run [named tasks](#named-tasks-instead-of-a-shell) only. |
## hammerkit remote mcp [#hammerkit-remote-mcp]
[`hammerkit remote mcp`](/docs/cli/remote-mcp) is an MCP server on stdio for one sandbox.
Clients start it as a command:
```json title=".mcp.json"
{
"mcpServers": {
"hammerkit": {
"command": "npx",
"args": ["hammerkit", "remote", "mcp", "--server", "corp", "--build", "sha256:9f2c..."],
"env": { "HAMMERKIT_SANDBOX": "sbx_4vJ0mT2cX9qLw1aB", "HAMMERKIT_TOKEN_COMMAND": "agent-token" }
}
}
}
```
Paths are relative to the root of the sandbox. The tools:
| Tool | Parameters | What it does |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `read` | `path`, `offset`, `limit` | The lines of a file, numbered like `cat -n`, in parts for long files. |
| `write` | `path`, `content` | Writes a whole file, making its directories. |
| `edit` | `path`, `old_string`, `new_string`, `replace_all` | Replaces a string that matches exactly once, or every match. |
| `apply_patch` | `patch` | A `*** Begin Patch` body or a git diff; every file changes or none. |
| `search` | `pattern`, `path`, `glob`, `type`, `-i`, `-A`, `-B`, `-C`, `fixed_strings`, `multiline`, `output_mode`, `head_limit` | ripgrep on the server. |
| `glob` | `pattern`, `path` | The files that match a glob. |
| `bash` | `command`, `workdir`, `timeout` (seconds) | Runs a command as the shell task and answers its output and exit code. A long output keeps its first 50 and last 200 lines, about 30 KB; `run_logs` reads the rest. Left out without a shell task. |
| `run_task` | `task`, `args` | Runs another published task, such as `test`, and answers its [result](/docs/remote-compute/runs#results-and-waiting): status, exit code, why it was not taken from the cache, its last 20 lines. Its description [lists the tasks](#named-tasks-instead-of-a-shell). |
| `run_logs` | `run`, `task`, `tail`, `from`, `limit`, `grep`, `ignore_case`, `context` | Reads more of the [log of a run](/docs/remote-compute/runs#logs). |
| `changes` | `format`: `summary`, `diff` or `patch` | What changed since the sandbox was fetched. |
| `check` | `task` | Runs a task as a check on the files a push would commit, see [checking before the push](#checking-before-the-push). |
| `push` | `branch`, `message` | Commits the changes and pushes them to a branch, see [taking the work out](#taking-the-work-out). |
| `services_up`, `endpoint_get`, `services_stop` | see [Service runs](#service-runs) | Start services for a person to look at, read a page of one, take them down. |
Reads and searches are marked `readOnlyHint`, the edits `destructiveHint`, `push`
`openWorldHint`; `bash` and `run_task` keep the defaults, since a command can do anything
its task can. The server
also tells the client which sandbox, repository and commit the tools work on.
## pi [#pi]
hammerkit is a [pi package](https://pi.dev/packages): installing it adds the extension.
```bash
pi install npm:hammerkit
HAMMERKIT_SERVER=corp HAMMERKIT_BUILD=sha256:9f2c... HAMMERKIT_SANDBOX=sbx_4vJ0mT2cX9qLw1aB pi
```
The extension registers pi's own `read`, `write`, `edit`, `bash`, `ls`, `find` and `grep`
with the sandbox as their backend, the way pi's SSH example does, and runs the user's `!`
commands there too. pi's working directory stands for the root of the sandbox: a path below
it is the same path in the sandbox, anything else is refused. The system prompt names the
sandbox, so the model knows nothing is installed on the machine. `grep` searches with
ripgrep on the server. Images are read from the sandbox like any file.
## opencode [#opencode]
opencode lets a plugin replace a built-in tool by registering one of the same name.
hammerkit's plugin works with opencode 2 and with opencode 1 from 1.18.29. Add hammerkit to
the dependencies of the project's opencode directory, install them with `npm install` in
that directory (opencode 2 does not), and re-export the plugin as the default export. Write
the server's exact version: a range such as `^1` can install a hammerkit without the plugin.
```json title=".opencode/package.json"
{
"dependencies": { "hammerkit": "2.0.0" }
}
```
```ts title=".opencode/plugins/hammerkit.ts"
export { default } from 'hammerkit/harness/opencode'
```
opencode 1 also needs `"@opencode-ai/plugin": "*"` in these dependencies, which it adds
itself when it starts. Its plugin files that re-export `HammerkitPlugin` still work there;
opencode 2 loads only the default export.
Then start opencode with the [settings](#settings) as variables, as the user whose host
config has the server (`hammerkit remote add`). opencode 2 runs a background service that
keeps the variables it was started with: run `opencode service stop` after changing them, or
`opencode run --standalone` to start with the current ones.
| opencode 2 | opencode 1 | Parameters |
| ---------- | ------------- | ----------------------------------------------------------------------------------------- |
| `read` | `read` | `path` (`filePath` on opencode 1), `offset`, `limit`. A directory lists its entries. |
| `write` | `write` | `path` (`filePath`), `content` |
| `edit` | `edit` | `path` (`filePath`), `oldString`, `newString`, `replaceAll` |
| `patch` | `apply_patch` | `patchText`, a `*** Begin Patch` body or a git diff; every file changes or none. |
| `shell` | `bash` | `command`, `workdir`, `timeout` in milliseconds. There are no background commands. |
| `glob` | `glob` | `pattern`, `path`, and on opencode 2 `limit` |
| `grep` | `grep` | `pattern`, `path`, `include`, and on opencode 2 `literal`, `caseSensitive` and `limit` |
| | `list` | `path`, `ignore` |
| `run_task` | `run_task` | `task`, `args`, as the [MCP tool](#hammerkit-remote-mcp), with the same answer |
| `run_logs` | `run_logs` | `run`, `task`, `tail`, `from`, `limit`, `grep`, `ignore_case`, `context`, as the MCP tool |
| `check` | `check` | `task`, see [Checking before the push](#checking-before-the-push) |
| `push` | `push` | `branch`, `message`, see [Taking the work out](#taking-the-work-out) |
The tools answer in opencode's words, and opencode still picks the patch tool for GPT models
and `edit` and `write` for the others. Paths are relative to opencode's directory or absolute
below it, which stands for the root of the sandbox. A long shell output keeps its first and
last lines, like the MCP tool, and names the `run_logs` call that reads the rest. `run_task`
and `run_logs` take their turn with the other tools, in the order the model made the calls.
On opencode 2 the plugin also removes the worktree tools, which would make worktrees on the
machine opencode runs on.
With an empty `HAMMERKIT_SHELL` the plugin removes opencode 2's `shell`. opencode 1 has no way
to remove a tool, so its `bash` stays and refuses: its own would run on this machine. Turn it
off with `"tools": { "bash": false }` in the `opencode.json` of opencode 1 so the model does not
try it.
opencode carries on without a plugin that fails to load, and without a word: its own tools
then work on the machine opencode runs on. On opencode 1 hammerkit's plugin keeps that from
happening when it can. It takes the `@opencode-ai/plugin` of the opencode directory when
hammerkit is linked there (`npm install ../hammerkit`, `file:`) and does not see it, and when
it finds none at all, its tools still replace opencode's and refuse with what is missing. A
plugin file that does not load at all cannot do that: hammerkit not installed, or on
opencode 2 a file without the default export. Check once that the tools are hammerkit's, in a
directory with nothing but the opencode setup. The answer names the files of the sandbox,
not the ones of your directory:
```shell
opencode run --standalone 'List the files of the project root with the read tool.'
```
On opencode 1, leave out `--standalone`, or list them without a model:
`opencode debug agent build --tool list`.
Tools the plugin does not replace still work on the machine opencode runs on, such as
`webfetch` and `skill`. Turn off the ones that touch local files and that your opencode
version still has, for example `"tools": { "multiedit": false }` in the `opencode.json` of
opencode 1.
## Checking before the push [#checking-before-the-push]
Every client has a `check` tool with a `task`, such as `test`. It runs the task as a
[check](/docs/remote-compute/checks#checks-of-a-sandbox): on a copy of the files a push of the
sandbox would commit, not on the sandbox, whose ignored files the agent can write. When it
passes, a CI job that runs [`remote verify`](/docs/cli/remote#verifying-checks) for the pushed
commit passes without running the task, see
[Checks that ran before CI](/docs/guides/agents/verified-checks). The tool answers whether it
passed, and the last lines of a failure:
```
check test passed (run run_Qx3k9ZpL2mW7vN4a): CI can verify it for a push of these files
```
The tool tells the model to check right before `push` and to push without changing files in
between. A check needs `repos` for the sandbox's repository in the agent's entitlement, as a
run from a ref does.
## Taking the work out [#taking-the-work-out]
Every client has a `push` tool with a `branch` and a commit `message`. The server commits the
changes of the sandbox as the agent's identity and [pushes](/docs/remote-compute/sandboxes#pushing)
them with the source's own credentials, only to a branch the entitlement gives `push`, such
as `agent/**`; any other branch is refused with what the agent may push to. The sandbox is at
the pushed commit afterwards, so the next push carries only new work. Without a branch to push
to, the MCP tool `changes` with `format: patch` gives a patch to apply with `git am`.
## Service runs [#service-runs]
Every client has three tools for [service runs](/docs/remote-compute/service-runs), to bring
the app up for a person to check the change, or to try it out itself:
| Tool | Parameters | What it does |
| --------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `services_up` | `services`, `endpoints` (`service:port`, such as `web:8080`), `ttl`, `sleepTtl`, `name`, `branch` | Starts the services of the published build file with the files of the sandbox, or of a pushed `branch` of its repository, and answers once they are up: the address of each endpoint and until when they stay. The same `name` again replaces the run and keeps its addresses. |
| `endpoint_get` | `run`, `path`, `service` | A `GET` of a path of an endpoint of the run, signed in as the agent: its status and the first 20000 characters of the body (at most 256 KiB of it is read, the rest is not downloaded). Only at the addresses the server gave the run, so the token goes nowhere else. |
| `services_stop` | `run` | Takes the services down. |
The agent's identity needs an `endpoints` level `publish` for the repository to name
endpoints, and the people who look need `open`, see
[Who opens an endpoint](/docs/remote-compute/service-runs#who-opens-an-endpoint). On a
Kubernetes backend a service run starts from a branch only: `push` first, then
`services_up` with that `branch`.
## What is not covered yet [#what-is-not-covered-yet]
* Commands that read input, and long-running background commands.
# Authentication (/docs/remote-compute/authentication)
OAuth is the only way to call a server: there are no API keys and no passwords. The
`issuers` of the config list the identity providers whose tokens are accepted, at least one.
```yaml title="server.yaml"
issuers:
- issuer: https://login.example.com
audience: hammerkit
groupsClaim: groups # default
clientId: hammerkit-cli
scope: openid api://hammerkit/.default
- issuer: https://token.actions.githubusercontent.com # CI jobs, matched by claims
audience: hammerkit
```
| Key | What it does |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `issuer` | The issuer, written exactly as the `iss` of its tokens (some providers end it with a slash). An https URL, http only for `localhost`. Its keys are fetched from its OpenID discovery document. |
| `audience` | What the `aud` of a token has to contain: an API audience registered for the server. |
| `groupsClaim` | The claim that lists the groups of a caller, `groups` by default. [Entitlements](/docs/remote-compute/entitlements) with `group` match on it. |
| `clientId` | The public OAuth client that [`remote login`](/docs/cli/remote#signing-in) signs in with, by the device authorization grant. An issuer without one only takes tokens obtained elsewhere, such as a CI job's or an agent's own, or a service account's through [`remote add --token-command`](/docs/cli/remote#a-token-command-per-server). |
| `scope` | What `remote login` asks for, `openid` by default. Add `profile email` for the dashboard to name people by their token's `name` (else `preferred_username`, else `email`) instead of its subject, which on Google is a number. |
The server publishes the issuers with their client, audience and scope in its
[discovery document](/docs/remote-compute#discovery), so `remote login` needs nothing
configured beyond `remote add`.
## What is checked [#what-is-checked]
Every call except discovery and the health check carries an OAuth access token as
`Authorization: Bearer `. The server fetches the issuer's keys from its
OpenID discovery document and accepts a token only when
* it is signed with `RS256`, `RS384`, `RS512`, `PS256`, `ES256`, `ES384` or `ES512`
(never `none` or a shared secret),
* its `iss` is one of the configured issuers and its `aud` contains the configured
audience,
* it has not expired and is already valid (a minute of clock skew is allowed), and
* it has a `sub`.
Keys are cached for ten minutes, and fetched again, at most once a minute, when a
token names a key the server does not know yet, so a rotation needs no restart. A
refused token only gets `401 invalid token`; the reason goes to the server's log,
with everything a caller sent quoted and stripped of control characters, so a caller
can't forge log lines. Concurrent requests share one key fetch, a failing issuer is
left alone for ten seconds, and the previous keys keep working for up to an hour
while the issuer is down.
Two things the server does not check, so set them up on the identity provider:
* `aud` has to be an API audience for the server, not the OAuth client's own id, or an
ID token for that client would be accepted. The server does not look at the token
type (`typ`) or the authorized party (`azp`).
* A `claims` entitlement matches any string claim as it finds it. It does not know
that `email` needs `email_verified`; match on `sub`, a group or a claim the
provider controls.
## Identity providers [#identity-providers]
Any OpenID Connect provider works when it issues **JWT access tokens** for an API audience
of the server and publishes `/.well-known/openid-configuration` below the issuer, with an
`issuer` that is exactly the configured one. `remote login` also needs the provider's
device authorization endpoint and a public client. Some providers only hand out JWTs to
workloads (CI jobs, service accounts) and opaque tokens to people; those are listed
without `clientId`, and people sign in through another issuer of the same server.
| Provider | People (`remote login`) | Agents and CI |
| ----------------------------------------- | ---------------------------- | -------------------------------------- |
| [Keycloak](#keycloak) | yes | clients with a service account |
| [Microsoft Entra ID](#microsoft-entra-id) | yes | app registrations (client credentials) |
| [Google](#google) | no, access tokens are opaque | identity tokens of service accounts |
| [GitHub](#github-actions) | no, access tokens are opaque | GitHub Actions OIDC tokens |
| [GitLab](#gitlab-ci) | no, access tokens are opaque | GitLab CI ID tokens |
To let people with a GitHub or GitLab account sign in, broker that login through Keycloak
or Entra ID; the server then sees the broker's tokens.
### Keycloak [#keycloak]
```yaml title="server.yaml"
issuers:
- issuer: https://keycloak.example.com/realms/corp
audience: hammerkit
clientId: hammerkit-cli
entitlements:
- group: { iss: https://keycloak.example.com/realms/corp, name: platform }
builds: publish
```
* Create a public client `hammerkit-cli` with **OAuth 2.0 Device Authorization Grant**
enabled.
* Keycloak does not put an API audience into access tokens by default. Add a client
scope with an **Audience** mapper that includes the custom audience `hammerkit` in the
access token, and assign it to `hammerkit-cli` and to the agents' clients.
* Add a **Group Membership** mapper with the token claim name `groups` and **Full group
path** off, so groups arrive as `platform` and not `/platform`.
* An agent is a confidential client with **Service accounts roles** on; it gets its token
with the client credentials grant. Its `sub` is the id of the client's service account
user, which a `subject` entitlement names, or put the service account user in a group.
### Microsoft Entra ID [#microsoft-entra-id]
```yaml title="server.yaml"
issuers:
- issuer: https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0 # your tenant id
audience: 11111111-1111-1111-1111-111111111111 # client id of the API app
clientId: 22222222-2222-2222-2222-222222222222 # client id of the CLI app
scope: api://hammerkit/access
groupsClaim: roles
entitlements:
- group: { iss: https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0, name: platform }
builds: publish
```
* Register an API app: under **Expose an API** set the Application ID URI
`api://hammerkit` and add a scope `access`. Set `requestedAccessTokenVersion` to `2` in
its manifest: version 2 tokens carry the issuer above and the API's client id as `aud`.
Version 1 tokens are issued by `https://sts.windows.net//` instead.
* Register a CLI app with **Allow public client flows** on and the API permission
`access`; its client id is `clientId`.
* Use the tenant id in the issuer, not `common` or `organizations`: those have no single
issuer.
* The `groups` claim of Entra holds object ids, not names. Define **app roles** on the API
app (such as `platform`), assign them to groups, users or agent apps, and read them with
`groupsClaim: roles`.
* An agent is an app registration that gets `api://hammerkit/.default` with the client
credentials grant. Its `sub` is the object id of its service principal, and the app
roles assigned to it are in `roles` too.
### Google [#google]
```yaml title="server.yaml"
issuers:
- issuer: https://accounts.google.com
audience: hammerkit
entitlements:
- subject: { iss: https://accounts.google.com, sub: '104857382910563728194' } # unique id of the service account
builds: run
```
Google's access tokens are opaque, so a Google issuer takes the identity tokens of
service accounts, whose `aud` is whatever the caller asks for:
```bash
# a workload on GCE, GKE or Cloud Run
curl -s -H 'Metadata-Flavor: Google' \
'http://metadata.google.internal/computeMetadata/v1/instance/service-account/identity?audience=hammerkit'
# elsewhere, impersonating the service account
gcloud auth print-identity-token --impersonate-service-account=agent@corp-ci.iam.gserviceaccount.com --audiences=hammerkit --include-email
```
The `sub` of such a token is the unique id of the service account, shown by
`gcloud iam service-accounts describe`. There is no groups claim, so entitlements name
each `subject`. With `--include-email` the token carries the service account's email, which
the dashboard names its runs by; without it, it shows the `sub`, unless
the entitlement gives the subject a `name` (see [Who](/docs/remote-compute/entitlements#who)).
### GitHub Actions [#github-actions]
```yaml title="server.yaml"
issuers:
- issuer: https://token.actions.githubusercontent.com
audience: hammerkit
entitlements:
- claims:
iss: https://token.actions.githubusercontent.com
match: { repository: corp/payments, ref: refs/heads/main }
builds: run
```
A job with `permissions: { id-token: write }` asks for a token with the audience of the
server:
```bash
curl -sS -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
"$ACTIONS_ID_TOKEN_REQUEST_URL&audience=hammerkit" | jq -r .value
```
Match on `repository`, `ref`, `environment` or `job_workflow_ref`. `repository_id` and
`repository_owner_id` do not change when a repository is renamed or recreated under the
same name, so they are the safer choice for a long-lived entitlement.
### GitLab CI [#gitlab-ci]
```yaml title="server.yaml"
issuers:
- issuer: https://gitlab.com # or the address of your GitLab
audience: hammerkit
entitlements:
- claims:
iss: https://gitlab.com
match: { project_path: corp/payments, ref: main, ref_type: branch, ref_protected: 'true' }
builds: run
```
A job declares the token it needs, and GitLab puts it in the variable:
```yaml title=".gitlab-ci.yml"
deploy:
id_tokens:
HAMMERKIT_TOKEN:
aud: hammerkit
script:
- curl -H "Authorization: Bearer $HAMMERKIT_TOKEN" https://build.example.com/v1/me
```
`ref_protected` is the string `'true'` on a protected branch or tag, so only pipelines
that maintainers control match. `project_id` stays the same when a project is renamed.
## Transport [#transport]
Plain http carries OAuth tokens, so `listen` on anything but the loopback interface needs
`tls` (a certificate and key) or `behindTlsProxy: true` when a proxy in front terminates
TLS. A config without either is refused at startup.
```yaml title="server.yaml"
listen: { host: 0.0.0.0, port: 8480 }
tls:
cert: /etc/hammerkit/tls.crt
key: /etc/hammerkit/tls.key
```
# Backend (/docs/remote-compute/backend)
`backend` says where the tasks of runs execute. `host` is this machine, with the same engine
as `hammerkit run`. `kubernetes` runs them as Jobs on a cluster, see [Kubernetes](#kubernetes).
`runtimes` says which runtimes of the engine tasks may use.
```yaml title="server.yaml"
backend:
type: host
platform: { os: macos, arch: arm64 } # detected when left out
runtimes: [docker] # what tasks may run on, docker by default
```
| Key | What it does |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `backend.type` | `host` (this machine, the default) or `kubernetes` (a cluster). |
| `backend.platform` | `host` only. What the machine offers: `os` is `linux`, `macos` or `windows`, `arch` is `amd64` or `arm64`. Detected when left out. Clients read it from the [discovery document](/docs/remote-compute#discovery). |
| `runtimes` | What tasks may run on, see [Runtimes](#runtimes). |
## Runtimes [#runtimes]
`runtimes` lists what tasks may run on: `docker` (containers, the default) and `local`
(commands on this machine as the user of the server, which have to be allowed by name).
```yaml title="server.yaml"
runtimes: [docker, local]
```
A build file with a task on a runtime that is not listed is refused when it is published
or proposed, see [What a build file may not do](/docs/remote-compute/build-files#what-a-build-file-may-not-do).
The server logs a warning at startup when `local` is allowed on a host. A container task uses the
docker daemon of the server's environment, so name `DOCKER_HOST` in
[`runs.env.allow`](/docs/remote-compute/runs#config) when it is not the default.
On a cluster `local` means a task without an image runs in the `toolImage` of a pool, see
[Pools](#pools). An [aggregate task](/docs/task/dependencies#aggregate-tasks), which only
groups its dependencies, runs on no runtime: it is never refused for `runtimes`.
## One server is one trust level [#one-server-is-one-trust-level]
On a host, the process of a run is not a sandbox. It runs as the user
of the server, a `local` task reads everything that user can read, and a container task can
use the docker daemon the server is configured for. Accounts keep credentials apart (a task
only reads the secrets of the accounts its run was admitted for), but they do not keep code
apart on one machine. Do not mix production and untrusted agents on one server: run
them on different servers, each with the accounts of its own level, and with `local` only on
a server whose users you trust with the machine. The storage directory and the other runs of
a server are inside what a `local` task can read.
The containers of a run with an open network are on a network of the run, not on Docker's
default bridge, so they don't reach the containers of other runs or other callers. The volumes
of a service are its caller's: the server puts a scope of the caller and the source in front of
each name (`pg-data` becomes `hammerkit-ref--pg-data`), so a build file can't open, or
with `resetOnChange` remove, the data of somebody else's services. A volume named like a path
is refused, since docker would mount that path of the host. The outputs a service gets from
the tasks it depends on keep their names, which come from the run's own files. Nothing removes
the scoped volumes, and volumes of a server from before the scope are no run's any more: both
are listed by `docker volume ls --filter label=app=hammerkit`.
Each running run takes one Docker network, and each limited task or service one more, out of
the daemon's address pools, which hold about 30 by default. A server with many runs at a time
needs larger pools (`default-address-pools` in `daemon.json`). A daemon that runs Windows
containers has no such networks: there the open containers of all runs stay on its default
network, so it doesn't keep runs apart.
What lies beyond the host is not hammerkit's to block on Docker: an open container reaches the
LAN, the host's own services through the gateway of its network and the cloud's metadata
address (`169.254.169.254`), which hands out the machine's credentials. So `open` means the
internet only once the host's firewall says so. On a host that runs nothing but the server,
rules like these do it (the run networks are `br-*` bridges, the egress proxies of `allow`
items are on `docker0`). They need Docker's `iptables` firewall backend, which has the
`DOCKER-USER` chain (not `firewall-backend: nftables`), and a daemon with IPv6 off, its
default; with IPv6 on, the same rules are needed in `ip6tables` for `fe80::/10`, `fc00::/7`
and `fd00:ec2::/32`. The script can run again, replacing what it set, and the rules stay when
Docker restarts. Test them with an open task that has to fail `wget -T 3 http://169.254.169.254/`
and still resolve names:
```sh title="firewall of the server's host"
# the resolvers the containers use, the only private addresses they reach on port 53: like
# Docker, read systemd-resolved's upstream list when /etc/resolv.conf names only its stub
RESOLV_CONF=/etc/resolv.conf
grep -q '^nameserver 127\.0\.0\.53' "$RESOLV_CONF" && RESOLV_CONF=/run/systemd/resolve/resolv.conf
RESOLVERS=$(awk '/^nameserver/ && $2 !~ /^127\./ && !seen[$2]++ { print $2 }' "$RESOLV_CONF")
iptables -N HAMMERKIT-RUNS 2>/dev/null || iptables -F HAMMERKIT-RUNS
iptables -A HAMMERKIT-RUNS -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
for resolver in $RESOLVERS; do
iptables -A HAMMERKIT-RUNS -d "$resolver" -p udp --dport 53 -j RETURN
iptables -A HAMMERKIT-RUNS -d "$resolver" -p tcp --dport 53 -j RETURN
done
iptables -A HAMMERKIT-RUNS -d 10.20.0.15 -j RETURN # an internal registry runs may reach anyway
for range in 169.254.0.0/16 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10; do
iptables -A HAMMERKIT-RUNS -d "$range" -j DROP
done
# leaving a network of containers for anywhere else; Docker keeps the networks apart itself
for bridge in 'br-+' docker0; do
iptables -C DOCKER-USER -i "$bridge" ! -o "$bridge" -j HAMMERKIT-RUNS 2>/dev/null ||
iptables -I DOCKER-USER -i "$bridge" ! -o "$bridge" -j HAMMERKIT-RUNS
# the host itself, through the gateway of the network
iptables -C INPUT -i "$bridge" -m conntrack ! --ctstate ESTABLISHED,RELATED -j DROP 2>/dev/null ||
iptables -I INPUT -i "$bridge" -m conntrack ! --ctstate ESTABLISHED,RELATED -j DROP
done
```
Saved as `/usr/local/sbin/hammerkit-firewall.sh`, a unit puts the rules back after a reboot. Start
the server after it (`After=hammerkit-firewall.service` in the server's unit), so no run starts
before the rules are in:
```txt title="/etc/systemd/system/hammerkit-firewall.service"
[Unit]
Description=Firewall rules for the containers of hammerkit runs
After=docker.service
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh /usr/local/sbin/hammerkit-firewall.sh
[Install]
WantedBy=multi-user.target
```
`systemctl enable --now hammerkit-firewall` applies them now and on every boot.
On a cluster every task is a pod of its own in the caller's namespace, without a token of the
cluster. The boundary is the container, the namespace and the network policies the server puts
in it (see [Namespaces](#namespaces)), which only hold on a network plugin that enforces
`NetworkPolicy`. They keep an open pod off the metadata and private ranges by default, so there
`open` means the internet without a firewall of your own (see `deniedEgress` in [Kubernetes](#kubernetes)). Keep the server off the nodes of the pools, so a task that escapes its
container does not share a node with the server's token. The server still holds the
right to mint the tokens of every account (see [Secrets in the Job](#secrets-in-the-job)), so
the advice stands: one server, and one set of accounts, per trust level.
## Capabilities [#capabilities]
A container task can add Linux capabilities (`capAdd`, see
[Capabilities](/docs/task/container#capabilities)), such as `SYS_ADMIN` for a daemonless image
build. A capability is root's power over the kernel the container runs on, which the runs of
other callers share, so a server adds one only where its operator allows it, and then only in a
sandbox with a kernel of its own:
```yaml title="server.yaml"
backend:
type: kubernetes
capabilities:
allow: [SYS_ADMIN] # what a build file may add
runtimeClass: gvisor # the sandbox the pods of those tasks run in
sandboxes:
capabilities: [SYS_ADMIN] # runs on sandboxes add none unless named here
```
```yaml title="server.yaml"
backend:
type: host
capabilities:
allow: [SYS_ADMIN]
runtime: runsc # a runtime of the docker daemon: gVisor's
```
| Key | What it does |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `backend.capabilities.allow` | The capabilities a task may add. A build file with a task that adds another is refused when it is published or proposed (`400`); without `capabilities` the server allows none. |
| `backend.capabilities.runtimeClass` | `kubernetes` only. The RuntimeClass the pods of tasks that add capabilities run in. |
| `backend.capabilities.runtime` | `host` only. The docker runtime the containers of those tasks run in, as named in the daemon's `daemon.json`. |
| `sandboxes.capabilities` | What a run on a [sandbox](/docs/remote-compute/sandboxes) may add, out of `allow`. None by default: a sandbox holds code nobody reviewed. |
Only the containers of tasks that add capabilities run in the sandbox, the others run as before,
and such a task never runs in the [runner of a sandbox](#sandboxes). The sandbox has to be one:
hammerkit can't tell whether a RuntimeClass or a docker runtime keeps the capability from the
host, so name only one that does, such as gVisor (runsc, which handles the container's system
calls in a kernel of its own) or Kata Containers (a virtual machine per pod). gVisor is
`gvisor` on GKE (GKE Sandbox, on by default on Autopilot) and installed on the nodes elsewhere;
on a docker host `runsc install` adds it to the daemon. Kata is `kata-vm-isolation` with AKS Pod
Sandboxing. A task can't ask for `privileged`, on any backend. On a docker host, such a task
can't bind files of the run (`src`, file outputs, mounts) yet, see
[build files](/docs/remote-compute/build-files): build images with it on a Kubernetes backend.
The pod of such a task goes to the nodes of the first [pool](#pools), and on a sandbox to the
node of the sandbox's files: those nodes need the runtime. Where only a node pool of its own has
it (GKE Standard, AKS), name that pool's labels in the first pool, or make sure the
RuntimeClass's `scheduling` sends the pod there. A pod no node can take stays `Pending`, and
hammerkit doesn't fail it early yet.
Pod Security `baseline` refuses these capabilities in a sandbox too, it does not know about
RuntimeClasses. With `capabilities` the server makes the namespaces of new callers enforce
`privileged` and audit `baseline`, and an admission policy of yours refuses in them what
`baseline` refuses, and capabilities beyond `baseline` to every pod outside the sandbox. The server
never changes the level of a namespace it made before: that would take `patch` on every namespace
of the cluster. Until you relabel one, the tasks that add capabilities fail in it and say so, the
others run. The second policy keeps the namespaces the server makes to its own, so it can't make
one that enforces `privileged` outside them. Both need Kubernetes 1.30 or later
(ValidatingAdmissionPolicy). Apply them before you set `capabilities`, with the RuntimeClass you
name there, the server's ServiceAccount and its `namespacePrefix`:
```yaml title="admission.yaml"
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: hammerkit-capabilities }
spec:
matchConstraints:
namespaceSelector: { matchLabels: { hammerkit.dev/managed: 'true' } }
resourceRules:
- { apiGroups: [''], apiVersions: ['*'], operations: [CREATE, UPDATE], resources: [pods, pods/ephemeralcontainers] }
variables:
- name: sandboxed
expression: "object.spec.?runtimeClassName.orValue('') == 'gvisor'"
# the security context of every container, init and ephemeral ones too
- name: contexts
expression: >-
object.spec.containers.filter(c, has(c.securityContext)).map(c, c.securityContext)
+ object.spec.?initContainers.orValue([]).filter(c, has(c.securityContext)).map(c, c.securityContext)
+ object.spec.?ephemeralContainers.orValue([]).filter(c, has(c.securityContext)).map(c, c.securityContext)
- name: baseline
expression: "['AUDIT_WRITE', 'CHOWN', 'DAC_OVERRIDE', 'FOWNER', 'FSETID', 'KILL', 'MKNOD', 'NET_BIND_SERVICE', 'SETFCAP', 'SETGID', 'SETPCAP', 'SETUID', 'SYS_CHROOT']"
- name: seLinuxTypes
expression: "['', 'container_t', 'container_init_t', 'container_kvm_t', 'container_engine_t']"
- name: sysctls
expression: "['kernel.shm_rmid_forced', 'net.ipv4.ip_local_port_range', 'net.ipv4.ip_unprivileged_port_start', 'net.ipv4.tcp_syncookies', 'net.ipv4.ping_group_range', 'net.ipv4.ip_local_reserved_ports', 'net.ipv4.tcp_keepalive_time', 'net.ipv4.tcp_fin_timeout', 'net.ipv4.tcp_keepalive_intvl', 'net.ipv4.tcp_keepalive_probes', 'net.ipv4.tcp_rmem', 'net.ipv4.tcp_wmem']"
validations:
- expression: "variables.sandboxed || variables.contexts.all(s, s.?capabilities.?add.orValue([]).all(a, a in variables.baseline))"
message: capabilities beyond baseline are added in the RuntimeClass gvisor only
- expression: "!variables.contexts.exists(s, s.?privileged.orValue(false))"
message: no privileged containers
- expression: "!object.spec.?hostNetwork.orValue(false) && !object.spec.?hostPID.orValue(false) && !object.spec.?hostIPC.orValue(false)"
message: no host namespaces
- expression: "!object.spec.?volumes.orValue([]).exists(v, has(v.hostPath))"
message: no hostPath volumes
- expression: >-
!object.spec.containers.exists(c, c.?ports.orValue([]).exists(p, p.?hostPort.orValue(0) != 0))
&& !object.spec.?initContainers.orValue([]).exists(c, c.?ports.orValue([]).exists(p, p.?hostPort.orValue(0) != 0))
message: no host ports
- expression: "variables.contexts.all(s, s.?procMount.orValue('Default') == 'Default')"
message: no unmasked /proc
- expression: >-
object.spec.?securityContext.?seccompProfile.?type.orValue('') != 'Unconfined'
&& variables.contexts.all(s, s.?seccompProfile.?type.orValue('') != 'Unconfined')
message: no unconfined seccomp
- expression: >-
object.spec.?securityContext.?appArmorProfile.?type.orValue('') != 'Unconfined'
&& variables.contexts.all(s, s.?appArmorProfile.?type.orValue('') != 'Unconfined')
&& object.metadata.?annotations.orValue({}).all(k, !k.startsWith('container.apparmor.security.beta.kubernetes.io/')
|| object.metadata.annotations[k] == 'runtime/default' || object.metadata.annotations[k].startsWith('localhost/'))
message: no unconfined AppArmor
- expression: >-
object.spec.?securityContext.?seLinuxOptions.?type.orValue('') in variables.seLinuxTypes
&& object.spec.?securityContext.?seLinuxOptions.?user.orValue('') == '' && object.spec.?securityContext.?seLinuxOptions.?role.orValue('') == ''
&& variables.contexts.all(s, s.?seLinuxOptions.?type.orValue('') in variables.seLinuxTypes
&& s.?seLinuxOptions.?user.orValue('') == '' && s.?seLinuxOptions.?role.orValue('') == '')
message: no SELinux options beyond the container types
- expression: "object.spec.?securityContext.?sysctls.orValue([]).all(s, s.name in variables.sysctls)"
message: only the safe sysctls
- expression: >-
!object.spec.?securityContext.?windowsOptions.?hostProcess.orValue(false)
&& !variables.contexts.exists(s, s.?windowsOptions.?hostProcess.orValue(false))
message: no Windows host processes
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata: { name: hammerkit-capabilities }
spec: { policyName: hammerkit-capabilities, validationActions: [Deny] }
---
# the server makes the namespaces of its callers only, at the levels it sets
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: hammerkit-namespaces }
spec:
matchConstraints:
resourceRules:
- { apiGroups: [''], apiVersions: ['*'], operations: [CREATE, UPDATE], resources: [namespaces] }
matchConditions:
- { name: server, expression: "request.userInfo.username == 'system:serviceaccount:hammerkit:hammerkit'" }
validations:
- expression: "object.metadata.name.matches('^hammerkit-[0-9a-f]{16}$')"
message: the server makes the namespaces of its callers only
- expression: "object.metadata.?labels['hammerkit.dev/managed'].orValue('') == 'true'"
message: the namespaces of callers are labelled hammerkit.dev/managed
- expression: "object.metadata.?labels['pod-security.kubernetes.io/enforce'].orValue('') in ['baseline', 'privileged']"
message: the namespaces of callers enforce Pod Security baseline, or privileged with capabilities
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata: { name: hammerkit-namespaces }
spec: { policyName: hammerkit-namespaces, validationActions: [Deny] }
```
Then relabel the namespaces the server made before, those of its prefix:
```bash
kubectl get namespace -l hammerkit.dev/managed=true -o name | grep '^namespace/hammerkit-' | \
xargs -r -I{} kubectl label {} pod-security.kubernetes.io/enforce=privileged \
pod-security.kubernetes.io/audit=baseline --overwrite
```
When you remove `capabilities` again, set them back to `enforce=baseline` (and remove `audit`)
before the server restarts: it runs nothing in a namespace that still enforces `privileged`, and
its error names the command.
[Building container images](/docs/guides/container-images) walks through a build of a
Dockerfile with BuildKit in gVisor, pushed with a token of a service account.
## Kubernetes [#kubernetes]
Experimental, like the server. Linux nodes only.
```yaml title="server.yaml"
backend:
type: kubernetes
namespace: hammerkit # the server's own; the namespace of its pod when left out
# kubeconfig: /etc/hammerkit/kubeconfig # with context, for a cluster the server is not in
# context: prod
storageClass: standard-rwo # of the claims of tasks; the cluster's default when left out
volumeSize: 1Gi # default, of a task's claim that states no volumeSize
maxVolumeSize: 50Gi # the most a task may state; 10Gi (or volumeSize, if more) when left out
keepVolumes: 15m # default; how long a task's claim stays for the next run, false for not at all
pools:
- name: builds
platform: { os: linux, arch: amd64 }
nodeSelector: { cloud.google.com/gke-nodepool: builds }
tolerations: [{ key: builds, operator: Exists, effect: NoSchedule }]
toolImage: ghcr.io/corp/build-tools:2026.10
providers:
gsm: { image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim }
quota: { requests.cpu: '8', requests.memory: 16Gi, persistentvolumeclaims: '20', requests.storage: 100Gi }
defaultResources: { requests: { cpu: 250m, memory: 256Mi } }
sandboxes: { size: 4Gi, storageClass: sandboxes }
# runRole: hammerkit-runs # the ClusterRole the server binds in each caller's namespace
# deniedEgress: [169.254.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, fe80::/10, fc00::/7]
allowedEgress: [10.20.0.15/32] # an internal registry open tasks may reach anyway
networkPolicy: enforced # the network plugin enforces NetworkPolicy; limited tasks need it
# openNetworkGate: false # what open pods wait to no longer reach, see Namespaces
```
The server runs as before (it checks the code out, plans the run with the engine in a process
of its own and keeps the cache), but every container task of a run becomes a Job and every
service a Deployment, in a namespace of the caller.
| Key | What it does |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `namespace` | The server's own namespace, where the [credentials of the accounts](#secrets-in-the-job) are. Found on its own when the server runs in a pod. |
| `kubeconfig`, `context` | The cluster, for a server outside of it. Without them the server uses its default kubeconfig, else the cluster it runs in. |
| `namespacePrefix` | A caller's namespace is this and the first 16 hex digits of the sha-256 of their issuer and subject, `hammerkit-` by default. Two servers on one cluster need a prefix each. |
| `storageClass` | The storage class of the claims tasks get. |
| `volumeSize`, `maxVolumeSize` | The size of a task's or service's claim when it states no [`volumeSize`](/docs/task/kubernetes#claims) (`1Gi`), and the most one may state: `10Gi` when left out, or `volumeSize` if that is more, since a build file states its own. A task over it fails before its pods start. Each claim counts as its size in a `requests.storage` quota, which bounds what a caller's claims take together. |
| `keepVolumes` | How long the claim of a task stays after the run that mounted it last (`15m`), see [Kept claims](#kept-claims). `false` removes it at the end of the run. |
| `pools` | Where pods are placed, see [Pools](#pools). One `linux`/`amd64` pool when left out. |
| `providers` | Per secret provider, the image its command runs in. Every provider an account reads needs one. |
| `quota` | The `ResourceQuota` of each caller's namespace, as Kubernetes writes it (`requests.cpu`, `limits.memory`, `pods`, `persistentvolumeclaims`, `requests.storage`). |
| `defaultResources` | `requests` and `limits` (`cpu`, `memory`) for containers without [`resources`](/docs/task#resources), as a `LimitRange`, so they are admitted under a quota that limits compute. |
| `sandboxes` | Where [sandboxes](/docs/remote-compute/sandboxes#on-kubernetes) are kept: two claims per sandbox in its caller's namespace, their size and storage class, and the image and resources of the pod that reads and edits its files. |
| `runRole` | The ClusterRole the server binds to itself in each caller's namespace, `hammerkit-runs` by default, see [What the server may do](#what-the-server-may-do). |
| `deniedEgress` | Address ranges a pod of a run with an open network, or an egress proxy, may not connect to. By default the link-local ranges, where the clouds' metadata endpoints hand out the node's identity (`169.254.0.0/16`, `fe80::/10`), and the private ones (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `100.64.0.0/10`, `fc00::/7`), where the cluster's nodes, pods and services and the networks behind it are. So `open` means the internet. A cluster with pod or service ranges outside of them (GKE's `34.118.224.0/20`) adds those. |
| `openNetworkGate` | With `networkPolicy: enforced`, what a pod with an open network waits to no longer reach before it starts: `{ host, port }`, an IPv4 address inside `deniedEgress` that answers without the policy, by default the metadata endpoint `169.254.169.254:80`, or `false` for no wait. See [Namespaces](#namespaces). |
| `allowedEgress` | Addresses inside `deniedEgress` they may reach anyway, such as an internal registry, a package mirror, the secret manager a provider reads in a Job, or the database of an `allow` [TCP forward](/docs/task/network). None by default. The pods of the caller's own namespace and DNS are reachable without it. |
### Namespaces [#namespaces]
The first run of a caller makes their namespace, labelled `hammerkit.dev/managed`. A namespace
of that name without the label is refused, the server runs no task in a namespace it did not
make. When a run starts the server puts in it, replacing what was there:
* a RoleBinding of `runRole` to itself, its only rights in the namespace;
* a `NetworkPolicy` that lets each pod with an open network accept connections from the pods of
its namespace only (the services a task needs), so callers can not reach each other's pods;
* a `NetworkPolicy` for the pods with an open network and the egress proxies: they reach their
namespace, DNS, any address outside `deniedEgress` and the addresses of `allowedEgress`;
* a `NetworkPolicy` that lets nothing into an egress proxy, so a proxy takes connections only
from the task its own policy names, and from nobody once that policy is gone;
* a `NetworkPolicy` that lets nothing into or out of the pods that copy files into and out of
claims (labelled `hammerkit.dev/transfer`), which the server reaches through the API only;
* `quota` and `defaultResources`; removing one from the config removes it from the namespace.
The server does that again for each run, but not within 5 minutes of the last time for the
same namespace: a namespace that was deleted and made anew, or a server that restarted, gets
it at the next run.
Policies add up, so none of them widens a pod with a limited [network](#network) (labelled
`hammerkit.dev/network: limited`): its own policy alone decides what it reaches, DNS left out, and
what reaches it. An egress proxy takes connections from its task only.
Some network plugins apply a policy a moment after a pod started. With `networkPolicy: enforced`,
a pod with an open network therefore starts with the init container `hammerkit-network`, which
waits until the pod no longer reaches an address the namespace denies it, as a limited pod waits
for its own policy (see [Network](/docs/task/network#kubernetes)). By default that is the metadata
endpoint, `169.254.169.254:80`. Each open pod starts about a second later for it, the time the
gate waits for an answer that doesn't come, and each node pulls the proxy's image once. When the
policy doesn't hold within a minute, the pod fails rather than run open, and says so.
`openNetworkGate` changes what it waits for:
* an address and port inside `deniedEgress` that answers without the policy, such as a node's
kubelet (`{ host: 10.0.0.10, port: 10250 }`). Use one where the metadata endpoint doesn't
answer at all (bare metal, kind, k3s, AWS without IMDS): there the default gate opens at once
and proves nothing.
* `false` turns it off, for a cluster whose metadata traffic is redirected before the network
plugin sees it, where the gate would never open.
When `deniedEgress` doesn't cover the gate's address, or `allowedEgress` lets it in, open pods
start without the gate. An allow of the address the cloud redirects metadata to (GKE's
`169.254.169.252:988`) isn't seen: turn the gate off there.
The policies of a task or service that a [service run](/docs/remote-compute/service-runs) no
longer has after a move go once its pods have ended.
The namespace enforces the `baseline` [Pod Security Standard](https://kubernetes.io/docs/concepts/security/pod-security-standards/):
no privileged pods, host paths or host network (with [capabilities](#capabilities), an admission
policy of the operator in its place). Every container of a run runs with
`allowPrivilegeEscalation: false` (so `sudo` and other setuid programs do not work), without
the `NET_RAW` capability (no `ping`) and with the runtime's seccomp profile, and without a token
of the cluster.
What a run makes there carries the label `hammerkit.dev/run` and is deleted when the run ends,
whether it succeeded or not, and the end of the run waits until its pods and claims are gone:
an idle caller costs nothing on the cluster but the [claims it keeps](#kept-claims) for a while.
What a run left because the server stopped before its end is deleted when the caller's next run
starts: the first run of the caller after the server started, or after a run whose objects could
not all be deleted.
The runs of one caller execute side by side in their namespace, up to
[`runs.limits.running`](/docs/remote-compute/runs#limits-and-restarts), as the runs of
different callers do. Several agents of one identity, each on a sandbox of its own, do not wait
for each other. On a server the Jobs, Deployments, Services, Secrets and claims of a task are
named after its cache id (its first 32 characters) and the run (the first 8 digits of its
`hammerkit.dev/run` label), and found by both labels, so two runs of the same task have their
own objects and the end of one never removes what the other uses. Kept claims are named after
the source they are kept for instead. Two runs that share [kept claims](#kept-claims) execute
one after the other: runs of the same branch or tag of a repository, checks of the same
repository and runs from a commit hash of it, runs of the same sandbox, or the uploads of one
caller (a source of type `upload`), whose kept claims are the caller's. The second waits in the
queue, without counting against `runs.maxDuration`. Runs of two branches have kept claims of
their own and go side by side.
With `keepVolumes: false` nothing is shared and even the runs of one branch run side by side.
The runs of a caller share the quota of the namespace, so size it for `runs.limits.running`
runs of one caller at once, and for a task with all its services up: a service and the task
that needs it hold their room at the same time. Kept claims count once for each branch that
ran in the last `keepVolumes`, so set `requests.storage` to bound the disk they take. A pod that does not fit the nodes waits for one,
until `runs.maxDuration`. The pod of a task or service over the quota is not created until the
quota has room: it says so in its log, with what the quota refused, and waits. A pod that asks
for more than the quota allows at all fails at once. After 10 minutes the wait goes on only while
a pod that will end holds room in the namespace, such as another task of the run or of another
run; when only services hold it, they wait for what waits for them, and the task or service
fails. The wait of a task's pod does not count against its [`timeout`](/docs/task#timeouts),
and lasts at most that timeout in all. While the quota refuses a task's pod, the run starts no other task that is less urgent
than it (see [which ready task runs first](/docs/task/dependencies#which-ready-task-runs-first)),
so the room that frees goes to it. A task that needs a service is not held back: its service is up
already and holds its room until the task ran. The pods that copy a task's files into its claim and its outputs out of it wait 10
minutes. They ask for 64Mi and are limited to 512Mi of memory, so they take little of a quota
sized for the tasks, whatever `defaultResources` says. Each service pod also has a small `debug`
container beside the service, with the service's files, for a shell with `kubectl exec`; it asks
for 10m CPU and 16Mi and is limited to 100m and 256Mi. A claim over the quota (`persistentvolumeclaims`, `requests.storage`) does
not wait. The server first removes released kept claims of the caller to make room, and when
that is not enough the task fails with `exceeded quota`.
What changes objects of the whole namespace happens for one run at a time: making the namespace
and its policies and quota at the start of a run, removing what a stopped server left (the
objects of a run that does not execute) and removing kept claims that expired. These take a
moment at the start of a run, not the whole run.
### Kept claims [#kept-claims]
The claim of a container task stays after the run for `keepVolumes` (`15m`), and each run
that mounts it again starts the time anew. The next run of the same source finds the task's
outputs in it: when the task's inputs did not change, the task is cached and the tasks that
depend on it mount its outputs from there, without restoring them from the cache. A restore
copies the outputs into a new claim through a pod, which for a `node_modules` of an `npm ci`
takes a while; with the claim kept, a change to code the install does not read goes straight
to the compile and test steps.
When the inputs did change, the task runs on its kept claim emptied first, as on a new one,
so a run's outputs never mix with those of an earlier one. Without a kept claim, a cache hit
restores into a new claim as before.
A kept claim is the source's, within its [cache](#cache-and-sources): the runs of one branch
or tag of a repository share theirs, the checks of a repository and its runs from a commit
hash theirs, and a run on a sandbox
mounts only those of the same sandbox, never what a run of the repository or of another sandbox
kept. The approved runs of a sandbox share claims of their own, per sandbox: they never mount
what the sandbox's runs that needed no approval kept, as their cache is apart from theirs. The first run of a new branch finds none and restores the cached outputs, as without
kept claims; each active branch takes claims of its own until they expire. Its name carries a
hash of the source, and its label `hammerkit.dev/volume-scope` the source.
The disk they take is bounded by the time and the quota of the namespace: the server removes
a claim once it expired, and when a claim of a run does not fit a `requests.storage` or
`persistentvolumeclaims` quota, the kept claims the run does not use make room: those of the
caller's other sources that their last run released, and those of its own a task no longer has
(one renamed or changed). The claims a run of another source mounts while it executes stay, and
the sweep of expired claims leaves all claims of a source while one of its runs executes. Room
for a claim is made differently: a run of one branch may remove the released claims of another
branch that runs at the same time but has not mounted them yet, and that run's tasks then
restore from the cache. Size `requests.storage` for the branches that run at once to keep them
warm.
After a restart, the server finds the claims it kept before when its first run starts, and
removes them as they expire. Services keep no claim; theirs go with the run.
### Pools [#pools]
A pool is a set of nodes: `platform` (`linux`, with `amd64` or `arm64`), `nodeSelector`
(the labels besides `kubernetes.io/os` and `kubernetes.io/arch`, which come from `platform`)
and `tolerations`. Container tasks and services go to the first pool. A task without an image
runs in the `toolImage` of the first pool that has one and offers the `platform` the task
declares (`platform: { arch: arm64 }` picks an `arm64` pool). A task for macOS or
Windows has no pool and fails when the run starts. `runtimes` has to list `local` for such tasks,
and at least one pool needs a `toolImage` then.
### Secrets in the Job [#secrets-in-the-job]
The server reads no secret of a provider for a task on a cluster: the Job of the task does,
with only the credentials of the accounts it uses. Each provider secret gets an init container
in the provider's image, which runs the provider's command with the provider's and the account's
`env` and writes the value into a volume in memory. The task's container mounts only that
volume: a `path:` secret is the file, an `env:` secret is set before the command runs.
The credential of an account is set under `accounts..kubernetes` and is put at
`/var/run/hammerkit/account/` in the init containers of that account only:
```yaml title="server.yaml"
accounts:
payments-dev:
providers:
gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /var/run/hammerkit/account/gcp.json } }
kubernetes:
serviceAccount: payments-dev # a ServiceAccount in backend.namespace
audience: //iam.googleapis.com/projects/123/locations/global/workloadIdentityPools/hk/providers/cluster
secret: payments-dev-federation # a Secret in backend.namespace, each key a file
```
| Key | What it does |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `serviceAccount` | A ServiceAccount of the server's namespace. Right before each Job of the task, the server asks for a token of it (an hour, for `audience`) and puts it at `token`; the Secret that carries it is deleted when the Job is done. A cloud that trusts the cluster's issuer for that ServiceAccount (GCP workload identity federation, AWS `AssumeRoleWithWebIdentity`, Azure federated credentials) exchanges it for the account's credentials. Short-lived, and one trust binding per account whoever the caller is. |
| `audience` | The audience of that token; the cluster's default when left out. |
| `secret` | A Secret of the server's namespace, each key a file of the directory: the federation config that points at `token`, or a key file where nothing else works (long-lived, so the second choice). |
#### Secrets of a task with a limited network [#secrets-of-a-task-with-a-limited-network]
A task with `network: none` or an allow list reaches nothing else, and its init containers share
its policy. `backend.providers..allow` names the hosts a provider's command reaches:
```yaml title="server.yaml"
backend:
type: kubernetes
networkPolicy: enforced
providers:
gsm:
image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim
# Google's STS exchanges the token of the account's ServiceAccount, Secret Manager reads the value
allow: [sts.googleapis.com, secretmanager.googleapis.com]
runs:
secretNetwork: none
```
* The task's egress proxy lets a tunnel to one of those hosts, on port 443, through only when it
carries the proxy's token. The token is new for each run of the task, in a Secret of the
caller's namespace that the init containers read their proxy variables from
(`HTTPS_PROXY=http://hammerkit:@hammerkit-proxy:3128`). The task's own container gets
neither the token nor those variables, so its command reaches the provider's hosts as little
as any other. The Secret goes with the proxy.
* A task with `network: none` gets a proxy for this alone, which allows its own command no host.
* The names are exact names or `*.` patterns, without a port; like a pattern of `allow`, they
never reach a private address. Name the hosts the command needs, not their domain:
`*.googleapis.com` would let it reach every Google API. The provider's command has to read `HTTPS_PROXY`, user and
password included; `gcloud` does. A command that ignores it can't read a secret under a
limited network. A federation config that impersonates a Google service account
(`service_account_impersonation_url`) also needs `iamcredentials.googleapis.com`.
* With [`runs.secretNetwork`](/docs/remote-compute/runs#tasks-that-read-secrets) every task that
reads a secret has to be limited, so each provider of such a task needs `allow`.
On a cluster every account with `providers` needs `kubernetes`, or the server does not start:
its commands would otherwise run as whatever identity the node or the provider's image has,
the same for every account.
The values never reach the server, so it can not mask them in the log. The task's output is
masked in the pod instead, with `awk`, line by line, before it leaves the pod: the image of a
task that reads such a secret needs `awk` (busybox, alpine, debian and their kin have it), and
stdout and stderr arrive as one stream. Encodings of a value (base64, hex) are not masked
there. `env:` secrets (variables of `runs.env.allow`) are the server's own and are handed over
as before.
Not yet on a cluster, and refused when a build file is published: a provider secret on a
service, and a provider secret with `cache: true` (its digest would be needed before the Job
reads the value).
### Cache and sources [#cache-and-sources]
The cache stays on the server, one per source trust as on a host (a repository's runs share it).
After a task ran, its outputs are copied from its claim into the cache, and a later run that
hits restores them into a new claim before its dependents start, unless the claim is still
[kept](#kept-claims). The files of a task's `src`
are copied from the server's checkout into its claim. So on a cluster the storage directory
of the server (`storage.directory`) belongs on a volume of its own.
### Network [#network]
A task's [`network`](/docs/task/network) works on a cluster as in a
[Kubernetes environment](/docs/task/network#kubernetes): a limited task's Jobs get a `NetworkPolicy`
that lets them reach only the services they need and, under `allow`, their egress proxy, a
Deployment on the first pool. A task without an image runs in a container of the
[tool image](#pools), so it is limited like a container task: it may name its own `network`
on such a server (elsewhere a local task with a limited network is refused when the build
file is read), and `runs.network` and `runs.secretNetwork` hold it as well. The policy, the proxy and its
Service carry `hammerkit.dev/run` and go with the rest of the run. Pods with an open network are
governed by the policy of the caller's namespace, see [Namespaces](#namespaces).
The cluster's network plugin has to enforce `NetworkPolicy` (Calico and Cilium do, and recent
kind), for `network:` and for keeping callers apart. Without it nothing is isolated. Hammerkit
can't tell from the API, so `backend.networkPolicy: enforced` states it; without it a task with a
limited network fails instead of running open. With
[sandboxes](/docs/remote-compute/sandboxes#on-kubernetes), whose runs are limited by
`sandboxes.network`, a config without it is refused unless that is `open`, and so is one with
[`endpoints`](/docs/remote-compute/service-runs#on-kubernetes), whose services only the
server is to reach. Some apply a
policy a moment after the pod started, so a limited pod waits in an init container until its
policy holds, and fails after 60 seconds if it never does (see
[Kubernetes](/docs/task/network#kubernetes)). A run's policies go only after its pods are gone.
A pod with an open network has no such wait, so on those plugins it can reach a `deniedEgress`
address in its first moment.
A task that reads a provider's secret can have a limited network when the server names the
endpoints of the provider: its init containers are in the task's pod, under the same policy,
and reach the secret manager through the task's egress proxy (under `none` too). See
[Secrets in the Job](#secrets-in-the-job). Without them such a task keeps an open network: a
build file that limits it is refused when it is published.
### Sandboxes [#sandboxes]
A [sandbox](/docs/remote-compute/sandboxes#on-kubernetes) is two claims in its caller's
namespace, next to the namespace's runs: its files, and its git data. A short pod of
`backend.sandboxes.image` fetches into them, reads and edits the files and commits what the
server pushes; the Jobs of a run on the sandbox mount the files only. The quota of sandboxes
is a `ResourceQuota` on their storage class, when they have one of their own. The pod runs as
root with only the capabilities to own and change files (the `baseline` Pod Security level
allows it, `restricted` does not).
A task of a run on a sandbox that mounts all of its files and nothing else runs in the sandbox's runner, a pod
`hammerkit-runner-` that stays and takes each command over `pods/exec` (see
[sandboxes](/docs/remote-compute/sandboxes#on-kubernetes)). The pod of the files owns it, so
it goes with that pod after `sandboxes.idle`, and so does its `NetworkPolicy`, which outlives
the runner while the pod of the files stays. The runner carries no `hammerkit.dev/run` label:
the end of a run leaves it, and removing what a stopped server left skips it. It counts
against the quota like a task's pod, so a namespace with sandboxes needs room for one runner
per active sandbox besides the runs. It has the security context, resources and placement of
the task's Job, but no token and no service links, and an init container of
`busybox:1.37.0-musl` (pinned by digest) that copies the busybox the runner cleans up with:
a cluster that pulls only from a registry of its own needs that image there too. When the
runner can not be used (the cluster refuses it, it does not start, or another task of the run is
in it) the task runs as a Job.
### What the server may do [#what-the-server-may-do]
Across the cluster the server makes namespaces and binds `runRole` in the ones it made. With
[sandboxes](#sandboxes) it also lists namespaces when it starts, to find the claims of sandboxes
in the ones it made. With [service runs](/docs/remote-compute/service-runs#on-kubernetes) it
deletes the namespace of each at its end, and lists them to remove the ones a stopped server
left. It needs
no rights on Secrets or pods outside of them: those come from `runRole`, which it binds in each
caller's namespace (Kubernetes lets it bind that role only, by `bind`). In its own namespace it
reads the Secrets of the accounts and makes tokens of their ServiceAccounts, by name. The
processes of runs use the same identity. `list` on events is how the log of a run that failed
gets what the cluster said of its pods (a failed image pull, a pod no node takes); a
`runRole` without it still works, and the log says the events could not be read. `list` on
`pods.metrics.k8s.io` and, across the cluster, on nodes and `nodes.metrics.k8s.io` are only
for what [`remote capacity`](/docs/remote-compute/status#capacity) and the dashboard show of
the CPU and memory in use; without them, they say what they could not read.
```yaml title="rbac.yaml"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: hammerkit-server }
rules:
- { apiGroups: [''], resources: [namespaces], verbs: [get, list, create, delete] } # list: with sandboxes or service runs, delete: service runs
- { apiGroups: [rbac.authorization.k8s.io], resources: [rolebindings], verbs: [get, create, update] }
- apiGroups: [rbac.authorization.k8s.io]
resources: [clusterroles]
verbs: [bind]
resourceNames: [hammerkit-runs]
# optional, for the capacity: the nodes, what they have and use
- { apiGroups: [''], resources: [nodes], verbs: [list] }
- { apiGroups: [metrics.k8s.io], resources: [nodes], verbs: [list] }
---
# bound by the server in each caller's namespace, never across the cluster
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: hammerkit-runs }
rules:
- { apiGroups: [''], resources: [resourcequotas, limitranges], verbs: [get, create, update, delete] }
- apiGroups: ['']
resources: [pods, pods/exec, pods/log, persistentvolumeclaims, secrets, services]
verbs: [get, list, create, patch, delete, deletecollection]
- { apiGroups: [''], resources: [events], verbs: [list] } # what the cluster said of a run that failed
- { apiGroups: [metrics.k8s.io], resources: [pods], verbs: [list] } # what the pods of runs use, for the capacity
- apiGroups: [batch]
resources: [jobs, jobs/status]
verbs: [get, list, create, patch, delete, deletecollection]
- { apiGroups: [apps], resources: [deployments], verbs: [get, list, create, patch, delete, deletecollection] }
- apiGroups: [networking.k8s.io]
resources: [networkpolicies]
verbs: [get, list, create, update, patch, delete]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hammerkit-accounts, namespace: hammerkit }
rules:
# the secret and serviceAccount of each account
- { apiGroups: [''], resources: [secrets], verbs: [get], resourceNames: [payments-dev-federation] }
- { apiGroups: [''], resources: [serviceaccounts/token], verbs: [create], resourceNames: [payments-dev] }
```
Bind `hammerkit-server` with a ClusterRoleBinding and `hammerkit-accounts` with a RoleBinding to
the server's ServiceAccount; do not bind `hammerkit-runs` yourself. Making RoleBindings across the
cluster still lets the server bind `hammerkit-runs` in a namespace it did not make. To rule that
out, admit its RoleBindings only in the namespaces of its own callers, its `namespacePrefix`
and 16 hex digits, and not in those of another server on the cluster. The second policy keeps
the deletes of namespaces to those of its service runs:
```yaml title="admission.yaml"
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: hammerkit-managed-namespaces }
spec:
matchConstraints:
resourceRules:
- { apiGroups: [rbac.authorization.k8s.io], apiVersions: ['*'], operations: [CREATE, UPDATE], resources: [rolebindings] }
matchConditions:
- { name: server, expression: "request.userInfo.username == 'system:serviceaccount:hammerkit:hammerkit'" }
validations:
- expression: "namespaceObject.metadata.?labels['hammerkit.dev/managed'].orValue('') == 'true'"
- expression: "namespaceObject.metadata.name.matches('^hammerkit-[0-9a-f]{16}$')"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata: { name: hammerkit-managed-namespaces }
spec: { policyName: hammerkit-managed-namespaces, validationActions: [Deny] }
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata: { name: hammerkit-service-run-namespaces }
spec:
matchConstraints:
resourceRules:
- { apiGroups: [''], apiVersions: ['*'], operations: [DELETE], resources: [namespaces] }
matchConditions:
- { name: server, expression: "request.userInfo.username == 'system:serviceaccount:hammerkit:hammerkit'" }
validations:
- expression: "oldObject.metadata.?labels['hammerkit.dev/service-run'].hasValue()"
- expression: "oldObject.metadata.name.matches('^hammerkit-[0-9a-f]{16}$')"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata: { name: hammerkit-service-run-namespaces }
spec: { policyName: hammerkit-service-run-namespaces, validationActions: [Deny] }
```
Upgrading a server for [service runs](/docs/remote-compute/service-runs): add `delete` on
namespaces to `hammerkit-server` and the second policy. Without them a service run still
ends, but its namespace stays: the server removes what runs in it, and says in its log and
in the run's what its ClusterRole needs.
Not yet: Windows nodes, holding a pod with an open network back until the policy of its namespace holds, a cache in a registry or a bucket, a limited network for a task
that reads a provider's secret, and failing a task early whose pod can not start (an image that
can not be pulled, no node for its pool).
# Build files (/docs/remote-compute/build-files)
A run does not bring its own build file. The server keeps the build files that
were published to it, and a run says which one to use and which of its tasks to
execute. So whoever may run work cannot change what a task does, and what a task
does is decided by whoever may publish.
```
POST /v1/definitions
```
```json
{
"entry": ".hammerkit.yaml",
"files": {
".hammerkit.yaml": "tasks:\n build:\n cmds: [npm run build]\n",
"web/.hammerkit.yaml": "..."
}
}
```
A published build file is a bundle: the file that starts it (`entry`, the usual
`.hammerkit.yaml` when left out) and every file it includes, as text, up to 200
files and 1 MiB. It has no name: the server answers with its version, and a run
names that. The server reads the bundle the way a run would, in a directory of its own where
only the bundle's files exist, so one that does not load is refused when it is
published and not when somebody wants to run it (`400`). A bundle may not include
anything outside of itself. Includes from git have to be pinned to a commit and
come from an `https://` or `ssh://` address, and so do the git includes of the files
they include, because the version below can only vouch for what is in the bundle.
## Versions [#versions]
Every bundle has a version, the hash of its files:
`sha256:` of the JSON `[entry, [[path, sha-256 of the content], ...]]` with
the paths sorted. The same files have the same version anywhere, so a client can
compute it and compare it with the server's. A version is never changed; publishing
other files adds a version, publishing the same files again, by anybody, answers `200`
and changes nothing (the first publish answers `201`).
The hash is the only name a version has. There is no `latest` and no line of versions
one follows the other in: a name would be the publisher's choice, so a new name could
always start a new line, and a comparison with the version before would prove nothing.
Whoever reviews a version reads all of it, its [plan](#plan-and-accounts) and its files.
| Call | What it does |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `POST /v1/definitions` | Publish a bundle. Needs `builds: publish`, else `403`. With `builds: propose`, or when the bundle names accounts the caller does not `use`, it is a [proposal](#proposing-a-version) and answers `202`. The answer carries the `hash`. |
| `GET /v1/definitions` | Every version, newest first, with who published (and approved) it and when, its [annotations](#annotations), the tasks and services it has, the `accounts` it uses and whether it was yanked. `?repo=github.com/corp/payments&machine=ci-7` keeps the versions with those annotations. |
| `GET /v1/definitions/` | One version, with its `plan` and what each task works on (`inputs`). |
| `GET /v1/definitions//files` | The files of one version, to see what would run. |
| `POST /v1/definitions/plan` | What publishing the bundle would do, without storing anything. See [Checking before publishing](#checking-before-publishing). |
| `POST /v1/definitions//yank` | Pull a version back, see [Yanking](#yanking-a-version). The body is optional: `{ "reason": "..." }`. |
| `POST /v1/definitions//unyank` | Let a yanked version go on. Needs `builds: publish`. |
Reading needs a token and at least one service account or `builds` level. Versions are kept in the `storage` directory and survive a restart.
## Annotations [#annotations]
Without names, a version says where it comes from with annotations the publisher sends
next to the bundle, so the versions of one repository or one machine can be found, and
cleaned up, later:
```json
{
"entry": ".hammerkit.yaml",
"files": { ".hammerkit.yaml": "..." },
"annotations": {
"path": "web/.hammerkit.yaml",
"repo": "github.com/corp/payments",
"commit": "4be1a07c...",
"machine": "ci-runner-7",
"user": "runner"
}
}
```
`remote publish` fills in `path` (the build file's path in its git repository, or on the
machine outside of one), `repo` (the `origin` remote as `host/path`, left out when it is
not an https or ssh address), `commit`, `machine` and `user`. When a file of the bundle
differs from that commit, or is not committed, it adds `dirty=true`: the version has the
commit's files plus changes, not the commit's files. Any other key works too: up
to 20, lower case letters, digits, `.`, `_` and `-`, each value up to 500 characters.
Annotations are not part of the hash, so the same files sent from two machines are one
version, and it keeps the annotations of its first publish (a proposal keeps the
proposer's when it is approved). The server stores them as they were sent: they identify
a version, they prove nothing. Who published it is `publishedBy`, from the token.
## Inputs [#inputs]
A version lists, for every task, what it works on, so a client knows
which of its files a run needs without planning the build file itself:
```json
"inputs": {
"build": { "src": ["package.json", "src/**/*.ts", "!src/**/*.spec.ts"], "mounts": [], "deps": [], "needs": [] },
"test": { "src": [], "mounts": [".npm"], "deps": ["build"], "needs": ["db"] }
}
```
`src` are the patterns of the task's own `src` as the build file writes them
(environment variables are not substituted), `mounts` are the host paths a
container task mounts, `deps` the tasks and `needs` the services it waits for. Paths
are relative to the root of the bundle, which is also the root of the project's
files, and a path outside of it is left out because it cannot be sent. What a task
inherits from its dependencies is listed with those. Versions published before this
was recorded have no `inputs`.
## Plan and accounts [#plan-and-accounts]
A version also carries a `plan`, what each task and service
does as the engine reads it, and the `accounts` its secrets are read as. A reviewer
or an agent sees `deploy` uses `op-payments-prod` before anything runs:
```json
"accounts": ["op-payments-prod"],
"plan": {
"deploy": {
"kind": "task", "file": ".hammerkit.yaml", "runtime": "docker", "image": "node:22",
"cmds": ["./deploy.sh"], "command": "./deploy.sh", "network": "open",
"deps": [], "needs": [],
"accounts": ["op-payments-prod"],
"secrets": [{ "from": "gsm:kube-token", "account": "op-payments-prod", "target": "KUBE_TOKEN" }],
"serverEnvs": [],
"mounts": [], "ports": [], "generates": [],
"flags": ["local-task"]
}
}
```
`command` is the last command of a task (the `cmd` of a service), where a run's
[arguments](/docs/remote-compute/runs#arguments) land; `null` when there is none.
`serverEnvs` are the variables of the server its `envs` read (`$NAME`, `${NAME}`).
`network` is what it reaches besides its needs, as the build file declares it (`open`
when it declares nothing). Versions published before these were recorded have neither.
`file` is the build file that declares it, as a path in the bundle, or the repository and
the path in it for one of a git include. `deps` and `needs` name the tasks and services
as the plan does (`services:postgres` for `postgres` of an include). Versions published
before these were recorded have no `file`, and name what a task needs as its build file
does.
`runtime` is `docker`, `local`, or `none` for an
[aggregate task](/docs/task/dependencies#aggregate-tasks), which runs nothing.
The plan never holds a value of a secret, only where it is read from. `flags` are what a
reviewer looks at first: `local-task` (it runs on the host, not in a container),
`host-port` (it binds a port of the host), `remote-include` (it includes a file from
git) and `capabilities` (it adds the Linux capabilities in `capAdd`, see
[Capabilities](/docs/remote-compute/backend#capabilities)). A bundle that names an account the server does not define is refused and the
message lists the accounts that exist.
## Publishing and accounts [#publishing-and-accounts]
A bundle that uses an account the publisher does not
`use` becomes a [proposal](#proposing-a-version) even for `builds: publish`, and is
approved by somebody who may publish and who uses the accounts the proposer lacks. So
one person can not publish a task that reads the production key for somebody else to
run without a second pair of eyes.
## What a build file may not do [#what-a-build-file-may-not-do]
A build file is written by whoever publishes it and runs
with what the operator gave the server, so a bundle is refused (`400`, naming the file and
task) when it
* names `environments`: the docker host or Kubernetes cluster is the server's to choose,
* has a service for Kubernetes (`selector`, `kubeconfig` or `context`),
* names a cache backend other than the server's own (`s3`, `registry`, or a `path`),
* mounts a path outside of the bundle into a container, such as the docker socket, `/etc`
or a home directory, or has a `src` or `generates` outside of it (the server would copy its
own files into the task),
* has a service volume whose name is a path (`/var/run/docker.sock:/var/run/docker.sock`),
which docker would mount from the host,
* has a task on a runtime that `runtimes` does not allow,
* has a task that adds a capability (`capAdd`) that [`backend.capabilities`](/docs/remote-compute/backend#capabilities)
does not allow, which is any when the server names no sandbox for them,
* has a task or service whose `network` reaches more than [`runs.network`](/docs/remote-compute/runs#network) allows,
* declares `secretProviders` or `secretAccounts` (keep them in your
[host config](/docs/task/secret-providers#on-this-machine-only)), reads a `file:` secret, reads an `env:`
secret the server does not list in `runs.env.allow`, or reads a secret as the account
that reads repositories (see [Accounts](/docs/remote-compute/accounts) and [Sources](/docs/remote-compute/sources)).
* on a [Kubernetes backend](/docs/remote-compute/backend#secrets-in-the-job), reads a provider
secret in a service, one with `cache: true`, or one in a task with a limited
[`network`](/docs/remote-compute/backend#network).
A run checks the same paths again in its checkout, before it starts and before each task or
service runs: one whose `src`, `generates` or mount goes through a symbolic link of the
checkout fails, since docker and the copy into a cluster would follow the link out of it.
On a docker backend, once a container has started, the server also compares each folder
and file it binds with the one of the checkout, and a container that sees anything else
fails before anything of its image runs: a task's before any of its commands, a service's
before its own command or entrypoint. The server checks that with tools of its own, which it
puts in a folder of its own in each container (`/.hammerkit-` and a random name), so the image
needs nothing for it. They are a file in the `tools` folder of the server's hammerkit directory
(`HAMMERKIT_DIR`), bound by its path, so a server that runs in a container needs that folder at
the same path on the docker host, as it needs the files of its runs. No `src`, output, mount,
volume or secret file can be at `/` or at or in a folder of `/` whose name starts with
`.hammerkit`. A task that adds
[capabilities](/docs/remote-compute/backend#capabilities) runs in a sandboxed runtime on a docker
host, where what the container binds can't be compared, so there it can't bind files of the
run: its `src`, file outputs and mounts fail the run (cache volumes are fine).
Without this a task with no secrets would still reach what the operator's machine can: a
published file would be enough to send a run, or the cache, to a host of its choosing.
## Only part of a build file [#only-part-of-a-build-file]
One build file usually holds tasks for the server and tasks only your machine runs: a
watcher, a task that mounts the docker socket, a service for local development. Mark
those `publish: false`, and the server leaves them out of the version: they are not in
its tasks, not in its plan, and not checked against the rules above.
```yaml title=".hammerkit.yaml"
includes:
infra: { path: ./infra.yaml, publish: false } # everything of the file
services:
db:
image: postgres:16
publish: false
tasks:
build:
image: node:22-alpine
cmds: [npm run build]
watch:
cmds: [npm run watch]
publish: false
```
A file left out is not checked for what the list above refuses of a published one: its
`environments` (a local docker host or kind cluster), `caches`, Kubernetes services and
`secretProviders` stay on your machine. Neither is a Kubernetes service with
`publish: false`. A git include with `publish: false` is not fetched by the server, so its
`ref` may be a branch.
Locally `publish` changes nothing. A published task or service that depends on, needs,
extends or runs as `init` one that is left out is refused, naming both:
`test needs db, which is not published (publish: false)`.
`publish: false` belongs to the include, not to the file: a file that another published
file includes too is published through that one, under that one's name.
## Checking before publishing [#checking-before-publishing]
```shell
hammerkit remote publish --dry-run
```
The server reads the bundle with its policy as a publish would, and says what publishing
it would do: publish it, [propose](#proposing-a-version) it (and which accounts make it a
proposal), or that it is published already. A bundle the server would refuse prints why:
every problem with its policy, or the first git include it may not fetch. Nothing is stored. Any `builds` level may check, `run` too, so a developer can
check a build file before the account that publishes it does. As for a proposal, a caller
who may not publish can't make the server fetch a git include: a build file that includes
from git (and publishes that include) is checked by somebody with `builds: publish`. A
version that is published already is not checked again, the answer says when it is yanked.
```
POST /v1/definitions/plan
```
takes the bundle as `POST /v1/definitions` does and answers its `hash`, `tasks`,
`services`, `plan`, `accounts` and `outcome`: `published`, `proposed`, `already
published`, or `null` for who may only run (`missing` lists the accounts the caller does
not use, `yanked` is set for a yanked version).
A build file that is checked, proposed or published reads the variables of runs
(`runs.env.allow` and `runs.env.set`) by their names: `${NAME}` stands in for the value of
each but `PATH`, `HOME` and `CI`. So a `description` or an `image` that reads one shows
`${DEPLOY_REGION}` in the plan, never what a run would see; a run reads the values. A
version published by a server of before this keeps the plan it was published with: publish
the build file again (with any change, so it gets a new hash) when its `description` or
`image` reads a variable of runs.
## Proposing a version [#proposing-a-version]
Who defines the work decides what runs, so a new version of a build file can wait for
a second pair of eyes. `builds: propose` is the lower level of `publish`, like
`request` is the lower level of `use`:
```yaml title="server.yaml"
entitlements:
- group: { iss: https://login.example.com, name: platform }
accounts: { op-payments-prod: use }
builds: publish
- claims:
iss: https://login.example.com
match: { client: coding-agent }
builds: propose
```
A caller who may only propose sends the same `POST`. The server reads the bundle as it
does for a publish and refuses one that does not load, then keeps it as a
**proposal** (`202`). A publish also ends up here when the bundle names accounts the
publisher does not `use` (`missing` lists them). It is not a version: it can not be
run and it is not listed with the versions. Someone who
may `publish`, uses the accounts in `missing`, and did not propose it, approves it,
and only then it is published, as the proposer's version with the approver recorded as
`approvedBy`. That is the version runs refer to by hash, so what was approved is what
runs.
| Call | What it does |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /v1/proposals` | The proposals the caller made or may see, newest first, with `state` (`waiting`, `approved`, `rejected`, `expired`), who proposed it, the tasks and services, the `accounts` and the `missing` ones, `plan` and whether the caller `canDecide`. |
| `GET /v1/proposals/` | One of them. |
| `GET /v1/proposals//files` | Its files, to read what would be published. |
| `POST /v1/proposals//approve` | Publish it. The body is optional: `{ "reason": "..." }`. |
| `POST /v1/proposals//reject` | Drop it; the reason (up to 500 characters) is kept with it. |
A proposal is visible to its proposer and to everyone who may publish; what the caller
neither proposed nor may decide does not exist for them (`404`), and one that is
decided or expired is `409`. A proposal expires after the `approvals.timeout` like a
requested run does, and `approvals.maxWaiting` limits the proposals that wait (`503`).
Proposals are kept in memory, unlike runs: after a restart the proposer proposes again. Each
proposal and decision is logged with who made it.
Whoever decides reads the proposal whole, its plan and its files (`remote show`
does both). There is no diff against an earlier version: a proposal follows none.
### No git includes [#no-git-includes]
A proposal is read before anybody approved it. A bundle that
includes from git would make the server fetch from the network for a caller who may
only propose, so a proposal with `includes` or `references` from git is refused
(`400`). Whoever may publish can use them.
### Cooldown [#cooldown]
After a rejection, the proposer can not propose anything,
and nobody that version, for `approvals.cooldown` (default `10m`): `429`, with the time
left. A proposal that only ran out of time does not start one. It keeps the people who
decide from being asked again and again; like the proposals it is forgotten by a restart.
A proposal approves a new version of the build file, not the code its commands run:
`npm test` still executes the files a run brings. Production accounts stay behind
[`request`](/docs/remote-compute/entitlements#approvals) and `source: [ref]`.
## Yanking a version [#yanking-a-version]
A published version is never changed, but one can be pulled back when it should not run
any more (it leaks a token, it deploys to the wrong place). `POST
/v1/definitions//yank` is open to whoever has `builds: publish`, and to
whoever `use`s an account the version works with: the people who answer for what runs
as that account do not have to wait for a publisher. The body may carry a `reason`, up
to 500 characters.
A run that names a yanked version is refused with `409`
and the reason, and a run that was accepted before, and still waits for an approval or in
the queue, fails when it would start. Runs that already execute are not stopped; cancel
them. The version and its files stay on the server for the record, and the version list
shows who yanked it and why. Without names, yanking is the only way to stop a version:
every published hash stays runnable until it is yanked. `unyank` lets it go on, and needs `builds: publish`. Both are
logged with who did it, and a yank survives a restart.
From the command line, [`remote publish`](/docs/cli/remote#publishing-a-build-file) publishes
or proposes the build file of the working directory and prints its hash, and `remote
proposals`, `show`, `approve`, `reject`, `yank` and `unyank` work with the versions.
# Checks (/docs/remote-compute/checks)
Experimental. The keys, the API and the settings can still change between releases.
A CI job that only has to know whether a task passed for the files of its commit does not have
to run it: it asks the server with [`hammerkit remote verify`](/docs/cli/remote#verifying-checks).
The server answers from what its own runs passed. How to set that up is the guide
[Checks that ran before CI](/docs/guides/agents/verified-checks).
## Check runs [#check-runs]
Only the server writes checks, from runs it executed, for each task that `succeeded` or was
`cached`. A request names tasks and a source, never a result. The runs whose tasks count:
| Run | Counts |
| ---------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| From a ref (`source.type: ref`) without [arguments](/docs/remote-compute/runs#arguments) | yes |
| On a sandbox with `check: true`, see [below](#checks-of-a-sandbox) | yes |
| On a sandbox without `check` | no: ignored files of the sandbox can make a task pass |
| From uploaded files | no |
| Any run with arguments | no: it ran something other than the task |
A task without a [check key](#check-keys), and a task that failed, leave nothing. A `cached`
task counts because the cache found its entry by the same inputs the check key covers.
Checks hold only where tasks are isolated from the server: in containers or on a cluster. A
[`local`](/docs/remote-compute/runs#config) task runs as the user of the server, and could write
the server's records itself.
## Checks of a sandbox [#checks-of-a-sandbox]
```json
{
"definition": "sha256:9f2c...",
"tasks": ["test"],
"source": { "type": "sandbox", "id": "sbx_Jp0dY8m2kQ1aZt5x", "check": true }
}
```
A check of a [sandbox](/docs/remote-compute/sandboxes) runs on a copy of the files a push of
the sandbox would commit: the files git tracks and those `.gitignore` does not leave out, at
the run's `tree`. Nothing else of the sandbox is there, so a file the owner wrote where no
commit holds it can not make the task pass. What the tasks write stays in the copy, which is
removed when the run is over: the sandbox does not change.
Otherwise it is a run on the sandbox: it holds the sandbox while it runs, its network and
accounts are those of a run on a sandbox. Its cache is the one of the repository's runs from a
ref, since it runs on files a commit holds. Two more rules:
* The caller needs `repos` for the sandbox's repository, as for a run from a ref, and a branch of
it they may push to (`push` in `sandboxes`): `403` without. A check leaves what it made in the
repository's cache, which runs from `main` restore from, and so can a caller who pushes a
branch and runs it. A caller who may only pull can not.
* It takes no arguments: `400`.
* It takes a cached result only by the files' checksums, never by their dates
(`cache: modify-date` is taken as `checksum`).
The agents' [`check` tool](/docs/remote-compute/agents) submits such a run.
## Check keys [#check-keys]
A check is found by its key, which both the server and `remote verify` compute from the build
file and the files of a checkout, before anything ran in it:
* the task's definition, as for its [cache](/docs/task/caching): commands, image, variables,
mounts, the definitions of the services it `needs`, and so on,
* the sha256 of every file its `src` matches, by path, without `.git`,
* the keys of the tasks it depends on,
* for each service it needs, the files of the service's `src` and the keys of the tasks the
service depends on: an `e2e` that needs `api` gets another key when the source of `api`
changes.
So the key of a commit's task is the same wherever the commit is checked out, and a check
counts for every commit with the same files. It hashes with sha256 throughout, as the cache
does, so a file made to share the hash of another does not pass for it. Run `remote verify` in
a fresh checkout: a file a task generated there, where the `src` of a task reads, gives
another key than the commit has.
The build file is read as a server reads it (no `.env`, no placement on a cluster, any secret
provider and account). A task has no key when it, a task it depends on, or a task a service
it needs depends on, declares no `src` that matches a file, or when it reads a secret with
`cache: true`, whose value is part of its cache key. An
[aggregate](/docs/task/dependencies#aggregate-tasks) (`deps` and nothing to run) needs no
`src` of its own: it has a key when all its dependencies have one. Like any task, it counts
as verified only for a run that ran it: a run of the aggregate itself, which costs nothing
more when its dependencies already passed.
## Lookup [#lookup]
```
POST /v1/checks/lookup
{ "repo": "github.com/corp/payments", "keys": [{ "task": "test", "key": "4f0c…" }] }
```
Up to 100 keys. The answer has an entry per key, the newest run that passed it, or `null`:
```json
{
"checks": [
{
"task": "test",
"key": "4f0c…",
"passed": {
"task": "test",
"status": "succeeded",
"run": "run_Qx3k9ZpL2mW7vN4a",
"by": { "iss": "https://accounts.google.com", "sub": "1180…" },
"source": { "type": "sandbox", "sandbox": "sbx_4vJ0mT2cX9qLw1aB", "commit": "4be1a07c…", "tree": "a1f3…" },
"definition": "sha256:9f2c…",
"finishedAt": "2026-10-25T09:14:03.120Z",
"durationMs": 72000
}
}
]
}
```
A check that passed with a version of the build file that was [yanked](/docs/remote-compute/build-files)
since is `null`, with a `reason`. The caller needs `checks: read`, or a `builds` level, and
`repos` that cover the repository, see [Entitlements](/docs/remote-compute/entitlements): `403`
otherwise. The answer leaves out the accounts the run used: a CI job's log is often readable by
more people than the server.
## Config [#config]
```yaml title="server.yaml"
runs:
checksKeep: 30d # default
```
Checks are kept in the server's storage directory (`checks/`), one file per repository and
key, the newest run's. One older than `runs.checksKeep` is not found any more and is removed
within six hours.
# Entitlements (/docs/remote-compute/entitlements)
An entitlement says which service accounts a caller may use, and what they may do
with build files. The server matches the validated claims of the token only, never
anything the caller sends along. Nothing is allowed by default.
```yaml title="server.yaml"
entitlements:
- group: { iss: https://login.example.com, name: platform }
accounts: { op-payments-dev: use, op-payments-prod: use }
repos: [github.com/corp/*]
builds: publish
- group: { iss: https://login.example.com, name: eng-payments }
accounts: { op-payments-dev: use, op-payments-prod: request }
repos: [github.com/corp/payments]
builds: run
- claims:
iss: https://token.actions.githubusercontent.com
match: { repository: corp/payments, ref: refs/heads/main }
accounts: { op-payments-dev: use }
repos: [github.com/corp/payments]
builds: publish
```
| Key | What it does |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `subject`, `group` or `claims` | Who the entry is for, exactly one of them, see [Who](#who). |
| `accounts` | The [service accounts](/docs/remote-compute/accounts) the caller may `use` or `request`, see [Account levels](#account-levels). |
| `repos` | The repositories the caller may name in a run from a git ref, see [Repositories](#repositories). |
| `builds` | What the caller may do with build files: `run`, `propose` or `publish`, see [Build file levels](#build-file-levels). |
| `checks` | `read`: the caller may look up the [checks](/docs/remote-compute/checks) of the repositories of `repos`, and nothing else, for a CI job that only verifies. A `builds` level allows it too. Like the other levels it combines with the `repos` of every entitlement that matches the caller. |
| `sandboxes` | The repositories the caller may keep [sandboxes](/docs/remote-compute/sandboxes) of, one `repo` (`host/path` pattern) per entry with a `pull` or `push` type per branch pattern (`branches`). A list of its own: `repos` is for runs from a ref. |
| `endpoints` | The repositories whose [service runs](/docs/remote-compute/service-runs) the caller may open the endpoints of (`open`), or also publish endpoints in their own (`publish`), one `repo` pattern and `level` per entry, see [Who opens an endpoint](/docs/remote-compute/service-runs#who-opens-an-endpoint). |
| `observe` | `true`: the caller sees every run of the server whole, who started it, what its tasks run and its log, in [`remote status`](/docs/remote-compute/status), `remote runs --all`, `remote logs`, the dashboard and `GET /v1/runs?scope=all`, and the published build files as who has a `builds` level does. For operators; it stops, extends and approves nothing. |
An entry names accounts, a `builds` level, `checks`, `sandboxes`, `endpoints`, `observe` or more of them. `checks`
needs `repos`.
## Who [#who]
| Who | Matches |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `subject: {iss, sub}` | exactly that person or client; an optional `name` is how the dashboard names its runs when its tokens name nobody, such as a service account's without `--include-email` |
| `group: {iss, name}` | every caller of that issuer whose groups claim lists the name |
| `claims: {iss, match}` | every token of that issuer whose string claims equal all of `match`, for CI jobs (repository, ref, environment) |
A caller gets what all entitlements that match give, and nothing when no entry
matches. Every account that an entitlement names has to be defined under `accounts`,
a typo stops the server at startup.
## Account levels [#account-levels]
`accounts` maps an account to a level:
* `use` runs on the account at once and approves the runs others request on it.
* `request` lets the caller ask for runs on the account. Such a run waits until
somebody who may `use` the account approves it, see [Approvals](#approvals).
When several entitlements give a level for the same account, `use` wins. The account that
reads the repositories of [`sources`](/docs/remote-compute/sources) is the server's own: no
entitlement may name it.
## Build file levels [#build-file-levels]
`builds` is what the caller may do with [build files](/docs/remote-compute/build-files), one
level that includes the lower ones:
| `builds` | May |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `run` | start runs of published build files. Without a level nothing runs, an account alone does not allow it. |
| `propose` | also send new versions, which somebody who may publish approves, see [Proposing a version](/docs/remote-compute/build-files#proposing-a-version). |
| `publish` | also publish build files. Who defines the work does not have to be who runs it. |
A build file has no name, a version is the hash of its files, so the config never lists build files.
## Repositories [#repositories]
`repos` lists the repositories (`github.com/corp/payments`, `github.com/corp/*` for
one level, `gitlab.corp.com/platform/**` for a group with its subgroups) the caller
may name in a run from a git ref; the union of the matching entitlements applies. A
caller without `repos` can not start runs from a ref. The server also has to be able to
read the repository, see [Sources](/docs/remote-compute/sources).
## What a caller sees [#what-a-caller-sees]
Every grant is written to the log with issuer, subject, accounts and the entitlements
that matched. A caller with no account and no `builds` level can do nothing on the
server, they are not even shown which build files exist.
`GET /v1/me` returns the identity and what the caller may use:
```json
{
"iss": "https://login.example.com",
"sub": "4f1c2b",
"accounts": [
{ "name": "op-payments-dev", "level": "use" },
{ "name": "op-payments-prod", "level": "request" }
],
"repos": ["github.com/corp/payments"],
"builds": "run",
"checks": "read",
"sandboxes": [],
"endpoints": [],
"observe": false
}
```
`GET /v1/accounts` lists the accounts the caller has a level on, with the operator's
`description` and the `sources` the account allows (`upload`, `ref`, `sandbox`, or `upload`
and `ref` when the config does not say):
```json
{
"accounts": [
{ "name": "op-payments-dev", "description": "dev cluster", "level": "use", "sources": ["upload", "ref"] },
{ "name": "op-payments-prod", "level": "request", "sources": ["ref"] }
]
}
```
## Approvals [#approvals]
Some work should not happen because one identity asked for it: a deploy with the
production key, a task with access to customer data. Access to a service account comes
at two levels for that, see [Account levels](#account-levels). Whoever may **use** an
account runs on it directly. Whoever may only **request** it can ask for a run, which
waits in state `awaiting-approval` until somebody who may use that account approves it.
This suits an agent or a developer who proposes a deployment and a person who confirms
it.
```yaml title="server.yaml"
entitlements:
- group: { iss: https://login.example.com, name: release-managers }
accounts: { op-payments-prod: use }
- group: { iss: https://login.example.com, name: eng-payments }
accounts: { op-payments-dev: use, op-payments-prod: request }
builds: run
approvals:
timeout: 1h # how long a run waits, default 1h
maxWaiting: 100 # runs waiting at the same time, default 100
```
| Key | What it does |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `approvals.timeout` | How long a requested run, or a [proposed version](/docs/remote-compute/build-files#proposing-a-version), waits for its decision before it is rejected or expires. `1h` by default. |
| `approvals.maxWaiting` | Runs, and separately proposals, that wait at the same time; more are refused with `503`. `100` by default. |
| `approvals.cooldown` | After a proposal was rejected, how long its proposer can not propose that build file again, and nobody that version. `10m` by default. |
* The accounts of a run are derived from the secrets of the published, hash-pinned
build file, see [Accounts](/docs/remote-compute/accounts). Nothing in the request can change them, so
every run that reads a production secret as a requester is held, whatever its tasks
are called or labelled, and whoever may [publish](/docs/remote-compute/build-files) cannot lift it.
* A run needs an approval for the accounts the caller can only request, the others
they use. `deploy` waits for `op-payments-prod` while `test` in the same run does not.
* An approver covers every account they may `use`. A run that waits for accounts A and
B is released as soon as the approvals cover both: one person who uses both is
enough, otherwise two approve, one per account. Each account has its entry in the
run's `approvals`, with who approved it. They share one expiry: a run that nobody
covers within `timeout` is rejected, and so is one that any covering approver
rejects. More than `maxWaiting` waiting runs are refused with `503`. A run that waits
keeps waiting over a restart, for the time that is left, see [Limits and restarts](/docs/remote-compute/runs#limits-and-restarts).
* A request for an account that nobody may use is refused at once (`409 nobody can
approve `) and not left to expire.
* A caller approves what they have access to, and nobody approves their own run. A
caller who uses none of the accounts that wait sees nothing of the run. Approvers see
the published build files, so they can read what they approve, and nothing more.
* The approver is shown what to decide on: the tasks and services the run executes
with their dependencies as `plan` (commands, image, the secrets and the account each
is read as, mounts, ports and the `flags` to look at first, never a value), the
accounts, the version, the source and the options. Who uses an account a run needed
approved may also read that run (`GET /v1/runs/`, its events and its logs) and cancel it;
`GET /v1/runs` stays the caller's own.
* What an approval cannot cover is what the code does: pinning commands does not pin the
code they run, so an account that matters should also be limited to `source: [ref]`
and `refs`.
| Call | What it does |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /v1/approvals` | The runs waiting for an approval the caller can give, with who asked, the build file version, tasks, accounts, source and options. |
| `GET /v1/approvals/` | One of them. |
| `POST /v1/approvals//approve` | Approve the accounts the caller uses; the run is queued when all are covered (`already-approved` is `409`). The body is optional: `{ "reason": "..." }`. |
| `POST /v1/approvals//reject` | End the run; the reason (up to 500 characters) becomes its `error`. |
What a caller may not approve does not exist for them (`404`), and a run that is no
longer waiting is `409`. Each call is logged with who decided and the result, and the
decision is an event of the run, so the one who asked sees who approved it.
`hammerkit remote accounts` shows the caller's levels. The command line has [`remote approvals`, `approve` and `reject`](/docs/cli/remote#approving-runs).
## Changing entitlements [#changing-entitlements]
`kill -HUP ` (in a container `docker kill -s HUP `) reads the config file again and takes over
its `entitlements`: who may use which account, which repositories and which build files changes for the next
request, the server keeps running and no run is touched.
Entitlements that name an account or issuer the running server does not have are refused and the old ones
stay. Everything else (accounts, issuers, providers, sources, limits) is read at start only, the server says
which keys of the file differ and need a restart. As SIGHUP is also what a closed terminal sends, closing the
terminal of a server started in it does not stop it: stop it with Ctrl-C or `SIGTERM`. SIGHUP is not available
on Windows, where closing the console stops the server.
# Overview (/docs/remote-compute)
Experimental. The config and the API can still change between releases, see the
[remote compute spec](https://github.com/no0dles/hammerkit/blob/master/specs/remote-compute/spec.md).
A hammerkit server runs tasks for other people and agents on a machine that has what the
work needs: a Mac mini for iOS builds, a Linux box with big CPUs, the credentials to deploy.
Callers never get those credentials, and they can not change what a task does: they name a
published build file by its hash, the tasks to run and where the code comes from, and the server
decides with its config whether they may.
## How it fits together [#how-it-fits-together]
1. **Who calls.** Every call carries an OAuth token of an identity provider the server
trusts. The server knows people, CI jobs and agents by the claims of that token only,
see [Authentication](/docs/remote-compute/authentication).
2. **What they may do.** [Entitlements](/docs/remote-compute/entitlements) map callers to
service accounts (`use` or only `request`), to the repositories they may run from and to
what they may do with build files (`run`, `propose` or `publish`).
3. **Which credentials.** [Accounts](/docs/remote-compute/accounts) are the service accounts
of the operator, with the secret provider commands that read secrets as them. The secrets
of a published build file say which account each is read as, so the account of a run
follows from its tasks, never from the request.
4. **What runs.** A run executes tasks of a [published build file](/docs/remote-compute/build-files),
pinned to a version, so whoever may run can not change what a task does.
5. **Which code.** The code comes from a git ref of a repository the server reads itself,
see [Sources](/docs/remote-compute/sources), from files the caller uploads, or from a
[sandbox](/docs/remote-compute/sandboxes): a checkout of the caller's that stays on the server.
6. **Where it runs.** [Runs](/docs/remote-compute/runs) are queued, limited per caller and
executed in a process of their own on the [backend](/docs/remote-compute/backend). A
[service run](/docs/remote-compute/service-runs) keeps services up for a while instead,
with their ports at addresses of their own behind the server's sign-in.
## Starting a server [#starting-a-server]
[`hammerkit server`](/docs/cli/server) reads its own config file, so it needs no build file.
```bash
hammerkit server --config /etc/hammerkit/server.yaml
```
The server runs until it is interrupted. `HAMMERKIT_SERVER_CONFIG` can stand in for
`--config`.
## Setting up a server [#setting-up-a-server]
For coding agents on a GKE cluster, [Server on Kubernetes](/docs/guides/agents/kubernetes)
does all of this step by step. The pieces, in the order an operator sets them up.
1. **Identity provider.** Register an API audience for the server and list the issuer
under `issuers` ([Authentication](/docs/remote-compute/authentication)). Nothing else
identifies a caller.
2. **Accounts.** Define one service account per set of credentials a build needs, and the
secret provider commands that read them ([Accounts](/docs/remote-compute/accounts)). A
production account is limited to `source: [ref]` and the branches it may run from.
3. **Repositories.** List the repositories the server may read under `sources`, with a
token that a provider command mints, for example from a GitHub App
([Sources](/docs/remote-compute/sources)).
4. **Entitlements.** Say who may `use` or `request` which account, which repositories they
may name and whether they may `run`, `propose` or `publish` build files
([Entitlements](/docs/remote-compute/entitlements)).
5. **Start it.** `hammerkit server --config server.yaml`, behind a TLS proxy or with `tls`.
Send `SIGHUP` to read the entitlements again without a restart
([Changing entitlements](/docs/remote-compute/entitlements#changing-entitlements)).
A small config for one team on a Linux box, with a GitHub App token and a production
account that needs a second person:
```yaml title="server.yaml"
listen: { host: 0.0.0.0, port: 8480 }
behindTlsProxy: true
issuers:
- { issuer: https://login.example.com, audience: hammerkit, clientId: hammerkit-cli }
secretProviders:
gsm: { command: [gcloud, secrets, versions, access, latest, '--secret={{ref}}'] }
github-app: { command: [/usr/local/bin/github-app-token, '{{ref}}'] }
accounts:
scm-read: { providers: { github-app: { env: { APP_KEY_FILE: /etc/hammerkit/app.pem } } } }
op-payments-dev: { providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/dev.json } } } }
op-payments-prod:
providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/prod.json } } }
source: [ref]
refs: [main]
sources:
- match: github.com/corp/*
secret: { from: 'github-app:payments', account: scm-read }
runs:
env: { allow: [PATH, DOCKER_HOST] }
entitlements:
- group: { iss: https://login.example.com, name: platform }
accounts: { op-payments-dev: use, op-payments-prod: use }
repos: [github.com/corp/*]
builds: publish
- group: { iss: https://login.example.com, name: eng-payments }
accounts: { op-payments-dev: use, op-payments-prod: request }
repos: [github.com/corp/payments]
builds: run
```
A developer then registers and signs in with [`remote add` and `remote login`](/docs/cli/remote),
someone in `platform` publishes the build file with `remote publish`, which prints its
hash, and a run is a `POST /v1/runs` naming that hash (`remote run` does not exist yet). A run that reads the production secret
waits in `remote approvals` until a member of `platform` approves it.
## Config [#config]
The file belongs to whoever operates the server and is reviewed like code. Unknown
keys are an error, so a typo is not silently ignored.
```yaml title="server.yaml"
listen:
host: 127.0.0.1 # default
port: 8480 # default
tls:
cert: /etc/hammerkit/tls.crt
key: /etc/hammerkit/tls.key
issuers:
- issuer: https://login.example.com
audience: hammerkit
groupsClaim: groups # default
clientId: hammerkit-cli
scope: openid api://hammerkit/.default
- issuer: https://token.actions.githubusercontent.com # CI jobs, matched by claims
audience: hammerkit
secretProviders: # the commands that read secrets, see Secret providers
gsm:
command: [gcloud, secrets, versions, access, latest, '--secret={{ref}}']
accounts: # the service accounts a secret can be read as
op-payments-dev:
description: Payments, test secrets
providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/payments-dev.json } } }
op-payments-prod:
description: Payments, production
providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/payments-prod.json } } }
source: [ref] # only from reviewed code
refs: [main, 'release/*'] # and only from these branches
scm-read: # reads the repositories, no entitlement may name it
description: Read-only access to source repositories
providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/scm-read.json } } }
entitlements:
- subject: { iss: https://login.example.com, sub: 4f1c2b }
accounts: { op-payments-dev: use, op-payments-prod: request }
builds: run
- group: { iss: https://login.example.com, name: release-managers }
accounts: { op-payments-prod: use }
builds: publish
- claims:
iss: https://token.actions.githubusercontent.com
match: { repository: corp/payments, ref: refs/heads/main }
accounts: { op-payments-dev: use }
repos: [github.com/corp/payments]
builds: publish
sources: # the repositories a run may get its code from
- match: github.com/corp/* # an organisation
secret: { from: 'gsm:github-readonly-token', account: scm-read }
- match: gitlab.corp.com/platform/** # a group with its subgroups
secret: { from: 'gsm:gitlab-group-token', account: scm-read, username: oauth2 }
runtimes: [docker] # what tasks may run on, docker by default
approvals:
timeout: 1h # default
maxWaiting: 100 # default
cooldown: 10m # default, a rejected version can not be proposed again before
storage:
directory: /var/lib/hammerkit # default ~/.hammerkit/server
runs:
maxConcurrent: 4 # default
keep: 200 # default
limits: # per caller (issuer and subject), 429 over it
waiting: 10 # default, runs that wait for an approval
queued: 20 # default, runs that wait for a free place
running: 2 # default, runs that execute at once, the others wait in the queue
maxDuration: 1h # default, a longer run is stopped and fails
maxLogBytes: 1048576 # default, log output kept per run
env:
allow: [PATH, DOCKER_HOST] # the only variables of the server a run sees, none by default
set: { CI: 'true' }
network: open # default, the most any task reaches besides its needs
backend:
type: host
platform: { os: macos, arch: arm64 } # detected when left out
```
| Key | What it does |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `listen` | Interface and port. Port `0` picks a free one. |
| `tls` | Certificate and key to serve https. |
| `behindTlsProxy` | Set to `true` when a proxy in front terminates TLS. |
| `issuers` | The OAuth issuers whose tokens are accepted, see [Authentication](/docs/remote-compute/authentication). |
| `secretProviders` | The commands that read secrets, see [Accounts](/docs/remote-compute/accounts#secret-providers). |
| `accounts` | The service accounts a secret can be read as, see [Accounts](/docs/remote-compute/accounts). |
| `entitlements` | Who may use or request which account, which repositories and what they may do with build files, see [Entitlements](/docs/remote-compute/entitlements). |
| `approvals` | How long requested runs and proposals wait, see [Approvals](/docs/remote-compute/entitlements#approvals). |
| `sources` | The repositories a run may get its code from, and the credential the server reads and pushes them with, see [Sources](/docs/remote-compute/sources). |
| `runs` | Concurrency, limits per caller, durations, the environment and the network ceiling of runs, see [Runs](/docs/remote-compute/runs#config). |
| `sandboxes` | The runtimes, network and limits of sandboxes, see [Sandboxes](/docs/remote-compute/sandboxes#config). |
| `endpoints` | The domain, certificate and sign-in of the endpoints of service runs, see [Service runs](/docs/remote-compute/service-runs#server-config). |
| `runtimes` | What tasks may run on, see [Backend](/docs/remote-compute/backend#runtimes). |
| `backend` | Where tasks run and what the machine offers, see [Backend](/docs/remote-compute/backend). |
| `storage` | Where the server keeps what must survive a restart: the published build files, the uploaded source files, the sandboxes (`sandboxes/`), the task cache and the records of the runs (`records/`). Runs work in `runs/` below it while they run. |
Plain http carries OAuth tokens, so it is accepted on the loopback interface only.
Anywhere else the config needs `tls`, or `behindTlsProxy: true`.
The server checks the references within the file when it starts: an entitlement has to
name a configured issuer and defined accounts, an account only names declared secret
providers, and a source reads its token with a provider the account has credentials for.
## Discovery [#discovery]
`GET /.well-known/hammerkit` needs no token and returns what the server is, so a
client can check it before signing in:
```json
{
"name": "hammerkit",
"version": "1.25.0",
"protocol": 1,
"issuers": [
{
"issuer": "https://login.example.com",
"audience": "hammerkit",
"clientId": "hammerkit-cli",
"scope": "openid api://hammerkit/.default"
}
],
"backend": { "type": "host", "platform": { "os": "macos", "arch": "arm64" } },
"dashboard": "https://hammerkit.run.example.com/"
}
```
`dashboard` is there when the server has a [dashboard](/docs/remote-compute/status#dashboard).
`GET /healthz` answers `{"status":"ok"}` for a load balancer or a process manager.
Every other path needs a token, except `/` for a browser, which goes to the dashboard.
# Runs (/docs/remote-compute/runs)
A run is a request to execute tasks of a published [build file](/docs/remote-compute/build-files). The server writes every run
down in `storage` (`records/`, readable by the user of the server only), so a restart keeps them with their
logs, see [Limits and restarts](#limits-and-restarts). Every call needs a token, and a run can be
seen and cancelled only by the caller that submitted it: for anyone else its id is
`404`, the same as an id that does not exist. Who decides about it, and who may open a service run's endpoints, see it too; with `observe` a caller sees every run in lists, see [Who sees what](/docs/remote-compute/status#who-sees-what).
From the command line, [`remote approvals`](/docs/cli/remote#approving-runs) lists the runs
that wait for you. Submitting a run from the command line (`remote run`, `--on`) is not
there yet; a run is a `POST /v1/runs`.
## Submitting a run [#submitting-a-run]
| Call | What it does |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `POST /v1/runs` | Submit a run. Answers `202` with the run in state `queued`, or `awaiting-approval` when the caller only has the account on [`request`](/docs/remote-compute/entitlements#approvals). |
| `GET /v1/runs` | The caller's runs, newest first, without the `results` of their tasks. |
| `GET /v1/runs/` | One run: `state` is `awaiting-approval`, `queued`, `running`, `succeeded`, `failed`, `cancelled` or `rejected`; a failed or rejected run has an `error`. `results` says how each task went, and `?wait=30s` answers once the run is over, see [Results and waiting](#results-and-waiting). |
| `POST /v1/runs//cancel` | Cancel a run: one that is queued or waits for an approval at once, a running one when its tasks have stopped. |
| `POST /v1/runs//retry` | Start a run that is over again, as a new run of yours, see [Retrying a run](#retrying-a-run). |
| `GET /v1/runs//events` | The run's events as server-sent events, see [Events](#events). |
| `GET /v1/runs//logs` | The run's output as numbered lines, with a tail, a range and a search, see [Logs](#logs). |
```json
{
"definition": "sha256:9f2c...",
"tasks": ["build", "test"],
"source": { "type": "ref", "repo": "github.com/corp/payments", "ref": "main" },
"options": { "env": "staging", "cache": "checksum", "timeout": "30m" }
}
```
* `definition` is the hash of the published [version](/docs/remote-compute/build-files#versions)
to run, as `remote publish` printed it. The run is pinned to it, so what it executes
does not change under it. A hash that was not published is `400`, a yanked one `409`.
* `tasks` are task names as on the command line, up to 100. A task the build file
does not have is `400`. Instead of names, labels in `options.filter` can select
the tasks. The tasks named run as one plan, like those a label selects: side by
side as far as their dependencies allow, in no particular order, with the
[longest remaining chain](/docs/task/dependencies#which-ready-task-runs-first) first.
What a task usually takes comes from its last successful runs in the repository.
To run one task after another, make it depend on the other.
* There is no account in the request. The run uses the accounts of the secrets of the
tasks it executes, with their dependencies and the services they need, derived from
the pinned version, and the answer lists them as `accounts`. A caller who has no
level on one of them gets `403` naming the account, one who may only `request` it
gets a run in `awaiting-approval`, and a caller without `builds` level may not run.
* `source` says where the code comes from: `{ "type": "ref", "repo": "github.com/corp/payments", "ref": "main" }`
for a branch or tag of a repository (or a full commit hash) that the server checks out itself. The
server pins the ref to the commit it is at when the run is submitted; with `"commit": "4be1a07c..."`
(the full hash, 40 or 64 hex digits) the ref has to be at that commit, else the run is refused
with `409` (`main is at 9e1f... now, not at 4be1a07c...`), so a CI job runs the commit it was started
for and not one pushed after it. Or `{ "type": "upload", "snapshot": "sha256:..." }`
for the caller's working tree, announced and sent beforehand, see [Source
files](#source-files), or `{ "type": "sandbox", "id": "sbx_..." }` for a
[sandbox](/docs/remote-compute/sandboxes) of the caller's. A snapshot that is unknown, or that still lacks files, and a sandbox
that is not the caller's, are `400`. The kinds the accounts of the run list in `source` (and, for `refs`, the
branches) are the only ones allowed, anything else is `403`.
* `options` are the settings of `hammerkit run` that make sense on a server, all
optional: `env`, `filter` and `exclude` (labels), `cache` (`checksum`, `modify-date` or `none`),
`cacheReadOnly`, `timeout` (for example `30m`, for tasks without a timeout of their own),
`maxTimeout` (for every task, also one with a longer timeout of its own, see
[Capping every task](/docs/task#capping-every-task)) and `skipDeps`. A run as a whole is still
stopped after the server's `runs.maxDuration`. How many tasks run in
parallel and how much of the machine they use stay with the server, and `watch` and
the log mode are for terminals only, so they are not accepted.
* `options.args` are [arguments](/docs/task/arguments), what `hammerkit run -- ...`
passes after `--`. See [Arguments](#arguments).
The request names the work, it does not contain it: what the tasks do comes from the
published build file, and the files they work on come from `source`. A `ref` is
fetched by the server. An `upload` refers to a snapshot of the working tree that is
sent separately, so the request stays small however large the project is.
The body is JSON up to 64 KiB, and unknown fields are rejected. Each submitted run is
logged with who submitted it, the accounts and the source.
What `POST /v1/runs` answers, in the order the server checks:
| Status | When |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `202` | The run is accepted, `queued` or `awaiting-approval`. |
| `400` | The body is malformed or has unknown fields, the build file or version does not exist, a task is unknown, arguments go to anything but one task that reads no secret or server credential and takes arguments, the repository is not readable by the server or the ref is not one branch or tag, the snapshot is unknown or incomplete, or the sandbox is not the caller's. |
| `401` | The token is missing or invalid. |
| `403` | The caller has no `builds` level, may not run from that repository, has no level on an account of the run, or the account refuses that source or ref. A run on a sandbox also needs its repository and branch in the `sandboxes` of an entitlement. |
| `409` | The version was yanked, was published before accounts were recorded, nobody may approve an account the caller can only request, or the ref is no longer at the `commit` the request names. |
| `413` | The caller's [sandboxes](/docs/remote-compute/sandboxes) take more than their limit (runs on a sandbox only). |
| `429` | The caller is over their [limits](#limits-and-restarts). |
| `501` | The run is from an `upload`, which the executor does not support yet. |
| `502` | The server could not reach the repository to resolve the ref. |
| `503` | The server is shutting down, or too many runs wait for an approval. |
## Arguments [#arguments]
A run can pass arguments to one task, as `hammerkit run test -- src/login.spec.ts -t "login flow"`
does locally:
```json
{
"definition": "sha256:9f2c...",
"tasks": ["test"],
"source": { "type": "ref", "repo": "github.com/corp/payments", "ref": "main" },
"options": { "args": ["src/login.spec.ts", "-t", "login flow"] }
}
```
They are appended to the last command of that task, each quoted for its shell, so
shell syntax in an argument stays text. Its dependencies and the services it needs get
none. The run's `plan` shows the command as it runs, so the caller sees
`npx vitest run 'src/login.spec.ts' '-t' 'login flow'`, and a published version shows
the last command of each task as `command`, so a caller knows where the arguments land.
* `tasks` names exactly one task; labels can not select it. Several tasks, a service, or
only labels with arguments are `400`.
* Up to 100 arguments, 16 KiB together, and none with a NUL character.
* A run in which any task or service reads a secret, or takes a variable of the server
into its `envs` (`$NAME` or `${NAME}` of [`runs.env`](#config)) other than `PATH`,
`HOME`, `CI` and the [`HAMMERKIT_RUN_` variables](#executing-a-run), passes no
arguments. That covers the task itself, its dependencies and the services it needs: an
argument could make the command print what the task holds, and what a dependency wrote
into its outputs or a service answers, in a form no mask of
secrets knows (`rev token.txt`). The run is `400` naming the item and what it reads,
and the executor checks again before it starts. A shell task (`cmds: [sh -c]`) on a
server therefore runs without any account or credential of the server, nor next to one.
* A task with [`args: false`](/docs/task/arguments#which-tasks-take-arguments) takes none:
the run is `400`. With `runs.args: declared` in the [config](#config) the server passes
arguments only to the tasks that say `args: true`, any other task is `400` as well. Use it
when a published build file holds tasks nobody wrote for arguments: one of them that takes
them runs whatever its last command does with what a caller appends, so each would need
`args: false`, and a new task that forgets it takes them again. With `declared` the build
file names the tasks that take them, typically the [shell task](/docs/remote-compute/agents#the-shell-task)
of agents and a test task, and a reviewer reads `args` in the [plan](/docs/remote-compute/build-files)
of each.
* Such a run never uses an account, so it never waits for an approval.
* [`remote approvals`](/docs/cli/remote#approving-runs) still lists the arguments of a
run as JSON strings, and `remote show` prints every command whole, never cut, with
control and bidi characters written out.
* On a server that allows the `local` runtime, arguments to a local task run as commands
on the server's machine, as the code of the ref already does.
* A server never takes `--set`; variables stay what the published build file says.
As [locally](/docs/task/arguments), the task with arguments skips the cache: it reads no
cache entry and stores none, and a later run without arguments doesn't take its outputs
for a cache hit. Naming a method in `options.cache` (`"cache": "checksum"`) keeps its
caching, as `--cache` before `--` does. Its dependencies cache as always.
## Results and waiting [#results-and-waiting]
A run says how each of its tasks went in `results`, by task name, from the moment the
task starts. A client reads this first, and only reads the [log](#logs) when the last
lines are not enough:
```json
{
"id": "run_4vJ0mT2cX9qLw1aB",
"state": "failed",
"error": "test failed",
"results": {
"build": {
"status": "succeeded",
"startedAt": "2026-10-12T20:47:14.569Z",
"finishedAt": "2026-10-12T20:47:14.588Z",
"durationMs": 42,
"cache": { "hit": false, "causes": ["source changed: src/index.ts"] },
"commands": [{ "startedAt": "2026-10-12T20:47:14.571Z", "finishedAt": "2026-10-12T20:47:14.585Z" }],
"lastLog": ["compiled 12 files"]
},
"test": {
"status": "failed",
"startedAt": "2026-10-12T20:47:14.589Z",
"finishedAt": "2026-10-12T20:47:14.616Z",
"durationMs": 70,
"exitCode": 1,
"cache": { "hit": false },
"lastLog": ["FAIL src/index.spec.ts", " expected 1 to be 2"]
}
}
}
```
| Field | What it says |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status` | `running`, `succeeded`, `failed`, `cached` (taken from the cache, nothing ran), `skipped` (a dependency no task of the run needed, because every task depending on it was cached) or `cancelled` (stopped with the run, or never started because another task failed). A task the run names that is checked again as a dependency of a later one keeps what it did the first time. |
| `startedAt`, `finishedAt` | When the server heard that the task started and ended. A cached task has no `startedAt`. |
| `durationMs` | How long the task took, as the engine measured it. |
| `exitCode` | The exit code of the command that failed. |
| `error` | Why a task failed when no command exited, for example `timed out after 30m`. Masked like the log, up to 2 KiB. |
| `cache` | `hit: true` for a cached task. For a task that ran, `causes` says why it was not taken from the cache, as [`--explain`](/docs/cli/explain) would: `never cached`, `source changed: `, `command changed`, `no src declared (uncacheable)` and so on, at most 20 and then `… and N more`. |
| `commands` | When each command of a task that ran started and exited, the first 50, once the task is over: the rest of the task's time went to its container, its caches and, on Kubernetes, the pod's scheduling, image and init containers. Kubernetes gives the times of a pod in seconds. Left out where the runtime does not say, such as for a cached task or a shell in a sandbox. |
| `lastLog` | The last 20 lines the task printed, stdout and stderr as they came, the newest last, each cut at 1000 characters. They are kept even when the run's log was [cut off](#limits-and-restarts), so the end of a long log, where the error usually is, is never lost. |
What the run did before its tasks is in `phases`, in their order, each with its
`startedAt` and `finishedAt`: `source` got its files (the checkout of the commit, or those
of the sandbox), `namespace` prepared its place on a Kubernetes cluster, and `plan` started
the process of the run and read its build files. The time after `plan` until the first task
went to the keys of its tasks, computed from their files. A run of an older server has no
`phases`.
```json
"phases": [
{ "phase": "source", "startedAt": "2026-11-14T07:44:16.766Z", "finishedAt": "2026-11-14T07:44:17.916Z" },
{ "phase": "namespace", "startedAt": "2026-11-14T07:44:17.916Z", "finishedAt": "2026-11-14T07:44:18.166Z" },
{ "phase": "plan", "startedAt": "2026-11-14T07:44:18.216Z", "finishedAt": "2026-11-14T07:44:19.416Z" }
]
```
`GET /v1/runs/?wait=30s` holds the answer until the run is over (`succeeded`,
`failed`, `cancelled` or `rejected`) or the time is up, and then answers the run as it
is. A client that can not keep a stream of events open, such as a tool call of an agent,
waits with it and asks again while the run is not over. The wait is a duration like
`500ms` or `30s` (leave `wait` out not to wait, `0s` is not a wait), at most 50 seconds (a longer one is cut to 50 seconds, so the answer
comes before a proxy in front of the server closes the quiet connection, which nginx does
after 60); anything else is `400`. A run that waits
for an approval answers when the time is up, or when the server stops.
## Retrying a run [#retrying-a-run]
`POST /v1/runs//retry` (no body), `hammerkit remote retry ` or the retry button on the
run's page in the [dashboard](/docs/remote-compute/status#dashboard) starts a run that is over
(succeeded, failed, cancelled or rejected) again. The new run is yours, whoever started the
first one, and names it in `retryOf`. It asks for what the first one asked for: the same
published build file, the same tasks or services, settings and arguments, from the same source.
* **Who may retry it:** who may read the run whole, its owner, who decided about it, and
`observe`, since a retry runs its arguments again. `may.retry` says so on the run.
* **How it is decided:** as a run you start yourself. Your entitlements say whether you may
run from its repository, which accounts you may use, and whether one needs an approval.
Nothing carries over from the first run: an approval it got is not yours, so a retry of a
run that needed one waits for one again, and the server resolves its secrets as for any run.
* **What it runs:** a run from a branch or tag runs the commit the first one ran. When the
branch moved on since, the retry is refused with `409` and says so (for example
`main moved to since ran , start a new run from main`): the
accounts of a branch never run a commit that is no longer on it. A run on a
[sandbox](/docs/remote-compute/sandboxes) runs on the files the sandbox holds now, and
only its owner may retry it. A run of uploaded files is not retried.
* **Which tasks:** all the tasks the first run named, not only the ones that failed. What
passed is taken from the cache, so it costs little, and the run is whole again in one place.
* **A service run** starts again with its services, name, endpoints and the `ttl` and
`sleepTtl` it had, within today's `runs.up.maxTtl`. While a service run of yours with that
name is not over, the retry is refused (`409`): a new run of the name would replace it and
its volumes, so stop it first.
A run that is not over yet is not retried (`409`): stop it first.
## Logs [#logs]
`GET /v1/runs//logs` reads the output of a run as numbered lines, as far as the server
kept it. Without parameters it answers the first 200 lines:
```json
{
"lines": [{ "line": 2, "task": "test", "stream": "stdout", "text": " expected 1 to be 2", "match": true }],
"total": 214,
"next": 3,
"truncated": false
}
```
| Parameter | What it does |
| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `task` | Only the lines of this task. The lines are numbered from 1 over what `task` selects, so a later call with the same `task` reads around a line it found. Without it, every task's lines, and the server's own (`hammerkit`, the checkout). |
| `tail` | The last lines, up to 1000. With `grep`, the last lines of what it finds: the matches and their context, counted as lines. |
| `from` | The first line to read, by its number. Not together with `tail`. |
| `limit` | How many lines one answer has, up to 1000: 200 by default, or as many as `tail` asks for. When more follow, `next` is the `from` that continues; a match just before `from` still shows its lines after. |
| `grep` | Only lines that contain this text. It is a fixed string, not a regular expression, so a pattern can not keep the server busy. It can not span lines. |
| `i` | `true` to ignore case in `grep`. |
| `context` | Lines before and after each match, up to 20, marked `"match": false`. They are lines of the `task` asked for; without `task`, the lines around the match in the whole log, of any task. |
`total` counts the lines `task` selects, `truncated` says the log of the run was cut off at
`runs.maxLogBytes` and its end is not kept (the [results](#results-and-waiting) still have the
last lines). An unknown parameter or a value out of range is `400` and says which, a run of
somebody else's is `404`. A read costs the lines it answers, not the length of the log: the
server notes where the lines are once, as the log grows.
An agent that ran the tests reads `results.test.lastLog` first, then for example
`?task=test&grep=FAIL&context=5` to see every failing test with what it printed around it.
## Events [#events]
The events carry the sequence number as their `id`, so a client that lost the
connection continues with the `Last-Event-ID` header (or `?after=`) and misses
nothing. The stream replays what happened and then follows the run until it is over;
a finished run's events are replayed and the stream ends. Event types are `state`
(`awaiting-approval`, `queued`, `running` and the final state, with the `error` of a failed run), `approval`
(who approved or rejected the run), `task` (a task started or ended, with the `state` a
[result](#results-and-waiting) has and, once it ended, `durationMs`, `exitCode`, `error`
and `causes`) and `log` (a line of a task's output). A run that
produces more than 5000 events is cut off with one `truncated` event; the state
changes are always kept, and so are the results of the tasks, while their own events are
cut off with the log.
## Source files [#source-files]
A run from an `upload` works on the caller's files. They are sent in two steps so
that only what the server does not have yet travels: the caller announces the tree
as a list of files with their digests, the server says which digests it lacks, and
the caller sends those. The server takes the files already, but does not execute runs from
an upload yet: such a run is refused with `501`.
```
POST /v1/sources
```
```json
{
"files": [
{ "path": "package.json", "sha256": "9f2c...", "size": 812 },
{ "path": "bin/run.sh", "sha256": "41ab...", "size": 233, "mode": "exec" }
]
}
```
The answer names the tree, a **snapshot**, and says what is missing:
```json
{
"snapshot": "sha256:5d1e...",
"files": 2,
"bytes": 1045,
"createdAt": "2026-10-11T11:00:00.000Z",
"missing": ["9f2c...", "41ab..."],
"complete": false
}
```
The snapshot is the `sha256:` of the JSON `[[path, sha-256, mode], ...]` with
the paths sorted, the same construction as the version of a build file, so the same
files have the same name anywhere. It answers `201` for a new tree and `200` for
one the server already knows. Then each missing file is sent on its own:
| Call | What it does |
| -------------------------------- | ------------------------------------------------------------------------------------------------- |
| `POST /v1/sources` | Announce a tree. Up to 20,000 files, 32 MiB per file, 1 GiB per tree. |
| `PUT /v1/sources/blobs/` | Send the content of one file as the request body. Answers `201`, or `200` when the server had it. |
| `GET /v1/sources` | The trees the caller keeps, what they take and the limit. |
| `GET /v1/sources/` | One tree and the digests still missing; `complete` when none is. |
| `DELETE /v1/sources/` | Forget a tree and the content no other tree of the caller needs. |
A second run after one edited file announces the tree again and sends that file
only. A client can also skip the announcement when the snapshot it computes is the
one it used last and `GET /v1/sources/` says it is complete.
* **Only the caller's.** Trees and content belong to the caller who sent them;
another caller's snapshot is `404`. Content is never shared between callers: a
server that answers "I already have this file" to anybody would tell them what
other people work on.
* **Only announced content.** A file is taken only if a tree the caller announced
lists its digest, with the size announced, and only if the content has that
digest. Anything else is `400`.
* **Credentials do not leave.** A tree with a path that looks like a credential is
refused with `400` that names it, even when a `src` pattern reaches it: `.env`
(not `.env.example`, `.env.sample`, `.env.template`), `.npmrc`, `.netrc`,
`.pypirc`, `.git-credentials`, private keys (`id_rsa` and its kin, `*.pem`,
`*.p12`, `*.pfx`) and anything below `.git/`, `.ssh/`, `.aws/`, `.gnupg/` or
`.kube/`, in any case and under the names Windows also opens (`.SSH`, `.git.`,
`GIT~1`). Secrets belong in a secret provider; there is no override.
* **Quota.** A caller may keep 4 GiB of files, files still being sent included; a
file that would go over is `413`. A caller may keep 1,000 trees, whose
announcements take at most 64 MiB together; a tree over either is `413`. Nothing
is removed on its own, a caller deletes what they no longer need.
* **Who may.** Callers with a `builds` level; others get `403`. An account can
further restrict its `source`.
Paths are relative to the root of the project, forward slashes, nothing that climbs
out; `inputs` of a [build file version](/docs/remote-compute/build-files) say which files a run needs.
## Executing a run [#executing-a-run]
The server runs the tasks of an accepted run on its own machine, with the same engine
as `hammerkit run`, so the `local` and `docker` runtimes and the task cache work as
they do there. Each run executes in a process of its own (`hammerkit server-run`, started
by the server):
1. It fetches the ref of the repository, as pinned to its commit when the run was
submitted (see [Pinning](/docs/remote-compute/sources#pinning)), into a directory made for the run. A ref
that is not a plain branch, tag or commit name is refused.
2. It writes the files of the published version into a directory of the run that no task
mounts, and over the checkout, so the tasks see the published build files whatever the
repository says. A file that would land in `.git` (in any spelling a case-insensitive file
system reads as that) or behind a symbolic link of the checkout is refused.
3. It starts the process of the run in the published files, which it reads every build file
from, includes too: a task that writes a build file into the checkout, or a build file of
the repository beside an included one, changes nothing of what the run executes. The tasks
work in the checkout. It runs the tasks named (or the ones the labels select) as one
plan, side by side as far as their dependencies allow, the
[longest remaining chain](/docs/task/dependencies#which-ready-task-runs-first) first;
once one fails it starts no new task and the run fails. It reports each task and its
output as events. Cancelling the run tells the process so on
its standard input, the same on every platform, so the engine stops its tasks and
removes its containers; the process is killed if it is still running after 15 seconds.
On Windows the engine stops a local task by ending its shell, so a program the shell
started can keep running; the directory it holds is then left and reported (step 4).
4. It removes the directory, whether the run succeeded, failed or was cancelled. A directory
that can't be removed (a file still in use, for example) is left in place and reported in
the run's log and the server's log; the run still ends the way it ended.
The process starts with an environment the server builds, not with its own: the variables
`runs.env.allow` lists, plus `runs.env.set`, and a `HOME` that is an empty directory of the
run's alone. So what the engine reads of its process (the docker host, registry logins in
the home directory, cloud credentials) is what the operator gave runs and nothing else; a
private registry needs `DOCKER_CONFIG` in the allowlist, pointing to a login the operator
chose for runs. Tasks do not get the environment of the process either, only their own
`envs` and the secrets they declare.
The server also sets what the run is from, which a task or service reads into its `envs`
like any variable of the process:
| Variable | Value |
| ----------------------- | --------------------------------------------------------------------------------- |
| `HAMMERKIT_RUN_REPO` | the repository, such as `github.com/corp/payments` |
| `HAMMERKIT_RUN_REF` | the branch or tag the run was submitted for, as given (not set for a commit hash) |
| `HAMMERKIT_RUN_COMMIT` | the commit the run was pinned to (not on a sandbox, whose files may hold changes) |
| `HAMMERKIT_RUN_SANDBOX` | the sandbox of a run on a sandbox |
```yaml
tasks:
preview:
envs:
BRANCH: ${HAMMERKIT_RUN_REF:-local}
cmds:
- ./deploy-preview.sh "$BRANCH"
```
So a task that reads a secret, which takes no arguments on a server, still knows which
branch it deploys. They hold no credential: a task that reads them takes arguments and
keeps its network as before. The server sets them alone: a config that names one in
`runs.env` doesn't load. Outside a server they are not set, hence the default above.
A ref may hold `/`, `.` and upper case and be up to 200 characters long, which is no valid
image tag, DNS label or Kubernetes label. Turn it into one in the command, for example the
start of its hash: `printf %s "$BRANCH" | sha256sum | cut -c1-12`.
The plan of a run, which its approver reads, is made with the run's values. A value is part
of the task's cache key, so a task that reads `HAMMERKIT_RUN_COMMIT` runs again for every
commit. A task that reads any of them, or depends on one that does, records no
[check](/docs/remote-compute/checks): its result says nothing a CI job's `remote verify`,
which has no run, could look up. Keep them out of the tasks CI verifies.
The task cache is kept below the server's `storage` directory, one cache for the runs of
a repository and one per caller for uploads and for sandboxes. A run from an upload or a
sandbox can therefore not leave an entry that a run from a reviewed repository finds. A task is keyed by its definition
and its sources, not by where the checkout is.
A run fails with the reason when the server does not list the repository, when the ref
can not be fetched or has moved, when it comes from an `upload`: uploads are not
supported by the executor yet, use `source: ref`. Cancelling a run stops the fetch or
the running task.
Before it starts, the executor derives the accounts of the run again from the pinned
files with the run's environment, and fails the run if it would use one it was not
admitted for. The secrets of the tasks are read as the accounts of the server config, by
the provider commands, and a secret of any other account is refused when it is fetched.
A run on a [sandbox](/docs/remote-compute/sandboxes#running-on-a-sandbox) skips the fetch: the
published files are written to a directory of the run, and the process works on the files
of the sandbox, which stay when the run is over.
The process of a run is not an isolation boundary, see [One server is one trust level](/docs/remote-compute/backend#one-server-is-one-trust-level).
## Limits and restarts [#limits-and-restarts]
One caller can not take the machine. Per caller (issuer and subject) the server counts the runs that wait
for an approval (`runs.limits.waiting`) and the runs in the queue (`runs.limits.queued`); a submit over
either is answered `429` and says which. At most `runs.limits.running` runs of one caller execute at the
same time, and the next run in the queue whose caller has a place left starts first, so one caller with
many runs does not keep the others waiting. On a Kubernetes backend, runs of one caller that share
kept claims also run one at a time, see [Namespaces](/docs/remote-compute/backend#namespaces). [`remote status`](/docs/remote-compute/status) shows the queue, what holds each run in it, and which of your runs count against each limit. A run that executes longer than `runs.maxDuration` is stopped
and fails with that reason. The log of a run is cut off at `runs.maxLogBytes` with a `truncated` event;
the state changes and the results of the tasks, with their last lines, are always kept.
Runs are written down when their state changes. On start the server takes them back:
* finished runs keep their state, the results of their tasks, events and logs;
* runs that wait for an approval wait on, for the time that was left, and are rejected when it has passed;
* runs that were queued or running when the server stopped have failed with it: `the server stopped while
the run was running, submit it again`.
A graceful stop (SIGTERM) leaves the waiting runs waiting, and cancels the queued and running ones.
A [service run](/docs/remote-compute/service-runs) that is up gives its place back: it counts
against `runs.up` instead, and it is not kept over a restart either, since its services stop
with the server.
Changes to `runs` need a restart; only the entitlements are read again on `SIGHUP`, see
[Changing entitlements](/docs/remote-compute/entitlements#changing-entitlements).
## Config [#config]
```yaml title="server.yaml"
runs:
maxConcurrent: 4 # default
keep: 200 # default
activity: everyone # default, who sees the runs of others
limits: # per caller (issuer and subject), 429 over it
waiting: 10 # default, runs that wait for an approval
queued: 20 # default, runs that wait for a free place
running: 2 # default, runs that execute at once, the others wait in the queue
maxDuration: 1h # default, a longer run is stopped and fails
maxLogBytes: 1048576 # default, log output kept per run
env:
allow: [PATH, DOCKER_HOST] # the only variables of the server a run sees, none by default
set: { CI: 'true' }
network: { allow: [registry.npmjs.org] } # open by default
secretNetwork: none # open by default, what a task that reads a secret reaches
args: declared # any by default, which tasks take arguments
up: { ttl: 8h, maxTtl: 72h } # the defaults, sleepTtl unset, see Service runs
checksKeep: 30d # default, how long a check is found
```
| Key | What it does |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `maxConcurrent` | Runs that execute at the same time, the others wait in queue. `4` by default. |
| `keep` | Finished runs that are remembered, the oldest are dropped first. `200` by default. |
| `activity` | Who sees the runs of others, see [Who sees what](/docs/remote-compute/status#who-sees-what): `everyone` (the default), a caller with a `builds` level sees who runs what in the repositories they have access to, where it waits and how long it took, never its arguments, why it failed or its log; `involved` only the runs they decide about or may open the endpoints of. |
| `limits` | What one caller may have at a time, see [Limits and restarts](#limits-and-restarts). |
| `maxDuration` | A run that executes longer is stopped and fails. `1h` by default. |
| `maxLogBytes` | The log of a run that says more is cut off. 1 MiB by default. |
| `env.allow` | The variables of the server process that a run, its provider commands and the git fetch can see. Nothing else of the server's environment is passed on, and the list is empty by default. A server that runs tasks needs `PATH` in it, Windows needs its system variables (`SystemRoot`, `ComSpec`, `PATHEXT`, `TEMP`) as well. |
| `env.set` | Fixed values that are set in addition. Neither list may name a `HAMMERKIT_RUN_` variable. |
| `network` | The most any task or service of a run reaches besides the services it needs: `open` (the default), `none` or `{ allow: [hosts] }`, as the [`network`](/docs/task/network) of a task. See [Network](#network). |
| `secretNetwork` | The most a task or service that reads a secret, or a variable of the server other than `PATH`, `HOME` and `CI`, reaches: `open` (the default), `none` or `{ allow: [hosts] }`, within `network`. See [Tasks that read secrets](#tasks-that-read-secrets). |
| `args` | Which tasks take [arguments](#arguments): `any` (the default) but those with `args: false`, or `declared`, only those with `args: true`. |
| `up` | The time and limits of [service runs](/docs/remote-compute/service-runs#server-config). |
| `checksKeep` | How long a [check](/docs/remote-compute/checks) that a run passed is found by `remote verify`. `30d` by default. |
## Network [#network]
`runs.network` is a ceiling for every run of the server, as `hammerkit run --network`
is for a run on a laptop:
```yaml title="server.yaml"
runtimes: [docker]
runs:
network: { allow: [registry.npmjs.org, '*.githubusercontent.com'] }
```
* A version whose tasks or services ask for more is refused when it is published (or
proposed): with the ceiling above, a task without `network` (which is `open`), or one
that allows `pypi.org`, can't be published. The error names the task and what it asks for.
A wildcard of the server covers the names under it; a name does not cover a wildcard.
A name of the server covers a [TCP forward](/docs/task/network#tcp-forwards) to any port of it
(`db.corp.internal:5432`); a forward covers only itself, so `db.corp.internal:5432` on the
server lets a task forward to that port but not reach the name through the proxy.
* So a build file published to such a server names its `network` on every task and
service, which is also what a reviewer reads: the [plan](/docs/remote-compute/build-files) of a version shows each
one's `network`.
* Every run executes under the ceiling as well, and a version published before the
ceiling was set, which asks for more, is refused when a run of it is submitted.
* The services a task needs are reachable at every level.
* Local tasks can't be limited on the server's machine, so a server with a ceiling below `open`
allows only the `docker` runtime; `runtimes: [docker, local]` with it is a config error. On a
[Kubernetes backend](/docs/remote-compute/backend#network) a local task runs in a pod of a
pool's tool image and is limited like a container task, so there it is allowed, and a local
task may name its own `network`.
* On a Kubernetes backend the ceiling is a NetworkPolicy on every pod, so it needs
`backend.networkPolicy: enforced`; without that it is a config error. A task that reads a
provider's secret there can be limited only when the server names the provider's endpoints,
see [Secrets in the Job](/docs/remote-compute/backend#secrets-in-the-job).
### Tasks that read secrets [#tasks-that-read-secrets]
A task that reads a secret runs its tools next to it: `npm publish` with a token, a deploy
with a cloud credential. A tool that was changed, a dependency's install script or a
compromised release, can send the value anywhere the task reaches. `runs.secretNetwork`
holds every task and service that reads a secret, or takes a variable of the server into its
`envs` (other than `PATH`, `HOME` and `CI`), to less than the others:
```yaml title="server.yaml"
runs:
secretNetwork: { allow: [registry.npmjs.org] }
```
```yaml title=".hammerkit.yaml"
tasks:
publish:
image: node:24-alpine
deps: [build]
secrets:
- { from: 'gsm:npm-token', account: release, env: NPM_TOKEN }
network: { allow: [registry.npmjs.org] }
cmds: [npm publish]
```
* It is checked as `runs.network` is: a version in which such an item asks for more is
refused when it is published, proposed and when a run of it is submitted, naming the item
and what it reads. The other tasks keep what `runs.network` allows.
* `open`, the default, holds nothing beyond `runs.network`, so existing build files keep
working. `none` suits a server whose secret tasks only talk to the services they need.
* It limits the task while it holds the value, not what it hands on: a task that writes the
secret into its outputs, or a service it needs that has an open network, takes it further.
Those are in the build file, which a reviewer reads.
* An allow list is no wall around a shared host: a task that may reach `*.googleapis.com`
reaches another project's buckets there too. Allow the names a task needs, not their domain.
* Local tasks on the server's machine can't be limited, so with `secretNetwork` below `open`
a local task that reads a secret is refused, except on a Kubernetes backend.
# Sandboxes (/docs/remote-compute/sandboxes)
A sandbox is a caller's own checkout of a repository on the server, which runs work on.
It is for an agent without a machine of its own, or a person whose laptop should not
build: the files stay on the server between runs, so what `npm install`, a formatter or a
code generator wrote is there for the next run. It is not for production, see
[Accounts](#accounts).
The published build file always runs. A sandbox only supplies the files: its own
`.hammerkit.yaml` is just a file, and changing it changes nothing until a new version is
[proposed and published](/docs/remote-compute/build-files).
This page covers creating, listing, moving, deleting, running on sandboxes, reading,
searching and editing their files, and pushing their changes, on a host and on a
[Kubernetes backend](#on-kubernetes). [Agents](/docs/remote-compute/agents) covers the
clients that let a coding agent work on one.
## Who may keep one [#who-may-keep-one]
The `sandboxes` of an [entitlement](/docs/remote-compute/entitlements) list the repositories
the caller may keep sandboxes of, one `repo` (`host/path` pattern) per entry, and give each
branch pattern of it a type:
* `pull`: a sandbox may start from a matching branch, and fetch one.
* `push`: `pull`, and [pushing](#pushing) to the branch.
Without `sandboxes` the caller keeps none. It is a list of its own: `repos` are for runs
from a ref only, and a sandbox needs no `repos` entry for its repository. The entries of all
of the caller's entitlements whose `repo` covers a repository count, and where several
patterns match a branch, the higher type wins. An entry's branches never count for another
repository. The server has to read the repository too, the same check as for a
[run from a ref](/docs/remote-compute/sources).
```yaml title="server.yaml"
entitlements:
- subject: { iss: https://login.example.com, sub: agent-7 }
accounts: { op-payments-dev: use }
repos: [github.com/corp/payments]
builds: run
sandboxes:
- repo: github.com/corp/payments
branches:
'agent/**': push # agent/ and everything below it
main: pull
- repo: github.com/corp/*
branches: { main: pull }
```
A branch pattern is a branch name, where a segment may hold `*` (any characters but `/`)
and the last one may be `**`, everything below.
A sandbox belongs to the caller who created it. For anybody else its id is `404`, the same
as an id that does not exist, and it does not show in their list.
## Creating and managing [#creating-and-managing]
| Call | What it does |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `POST /v1/sandboxes` | `{ "repo": "github.com/corp/payments", "ref": "main" }`: fetch a branch into a new sandbox. Answers `201` with the sandbox. A sandbox starts from a branch, never from a tag or a commit, and keeps the commit the branch pointed to. |
| `GET /v1/sandboxes` | The caller's sandboxes, oldest first, with their `limits` and what they `used`. |
| `GET /v1/sandboxes/` | One sandbox. |
| `POST /v1/sandboxes//fetch` | `{ "ref": "next" }`: move a sandbox without changes to the commit a branch points to now. |
| `POST /v1/sandboxes//push` | `{ \"branch\": \"agent/fix-login\", \"message\": \"Fix the login\" }`: commit the changes and push them, see [Pushing](#pushing). |
| `DELETE /v1/sandboxes/` | Cancel the runs on the sandbox, wait for them to stop and remove its files. |
```json
{
"id": "sbx_Jp0dY8m2kQ1aZt5x",
"repo": "github.com/corp/payments",
"ref": "main",
"commit": "4f1c2a...",
"bytes": 18352211,
"createdAt": "2026-10-12T21:00:00.000Z",
"lastUsedAt": "2026-10-12T21:10:00.000Z",
"busy": true,
"run": "run_Wm3o0yN2e4cR9pLs"
}
```
After a [push](#pushing), `ref` is the branch it pushed to, `commit` what it pushed, and
`pushed` lists each branch it pushed with the commit it pushed there last.
Creating a sandbox is a fetch, not a run: it runs no task, holds no account and takes no
place of the [run limits](/docs/remote-compute/runs#limits-and-restarts). The server reads
the repository with the credential of its [sources](/docs/remote-compute/sources), as for a
run from a ref, and the request waits until the fetch is done. The checkout is shallow, the
commit and nothing before it.
A sandbox stays until it is deleted. Nothing expires and nothing is removed on its own,
`lastUsedAt` says when a run or a fetch last used it.
`fetch` moves a sandbox only while nothing changed its files (files that `.gitignore` leaves
out, such as `node_modules`, are not changes and stay) and no run holds it, else it is
`409`. While the caller's sandboxes take their limit of bytes, a fetch is `413`, and while
all sandboxes take the server's total it is `507`. The creates and fetches of one caller
run one at a time, so each sees what the one before it added. A fetch that takes the caller over
their limit is kept and measured: their next create, edit that adds bytes, or run on a
sandbox is refused until they delete one.
| Status | When |
| ------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `400` | The body is malformed, the repository is not `host/path`, the server can not read it, or `ref` is not a branch of it (a tag or a commit). |
| `403` | No entry in the `sandboxes` of the caller's entitlements covers the repository, or none of its branch patterns covers the branch. |
| `404` | The sandbox does not exist or is somebody else's. |
| `409` | A fetch of a sandbox with changes, or one a run holds. |
| `413` | The caller is at their limit of sandboxes or bytes. |
| `502` | The server could not reach the repository. |
| `507` | All sandboxes together take what the server allows. |
## Running on a sandbox [#running-on-a-sandbox]
A run names the sandbox as its `source`, the rest of the request is a normal
[run](/docs/remote-compute/runs#submitting-a-run):
```json
{
"definition": "sha256:9f2c...",
"tasks": ["test"],
"source": { "type": "sandbox", "id": "sbx_Jp0dY8m2kQ1aZt5x" }
}
```
* **The published files run on the sandbox.** The server reads the published version from a
directory of the run, and its tasks work on the sandbox: a build file published as
`ci/build.yaml` works in `ci/` of the sandbox. Nothing of the published version is written
into the sandbox.
* **One run at a time.** A run holds its sandbox from when it is submitted until it is over,
waiting for an approval and in the queue included. A second run on the same sandbox waits
in the queue for the first, runs on other sandboxes start next to it within the caller's
[limits](/docs/remote-compute/runs#limits-and-restarts) (`runs.limits.running`, 2 by
default), on [Kubernetes](/docs/remote-compute/backend#namespaces) too.
* **What it started from** is recorded in the run: the `commit` the sandbox was fetched at,
the git `tree` of its files when the run was submitted, which together name the changes
it held, and whether it held any (`clean`). The run logs the tree again when it starts,
since a run before it on the same sandbox may have changed the files.
* **An approved run works on a copy.** A run that waited for an
[approval](/docs/remote-compute/entitlements#approvals) runs on a copy of the tree it was
approved with, as a [check](/docs/remote-compute/checks#checks-of-a-sandbox) does, not on the
sandbox: files `.gitignore` leaves out (`node_modules`, outputs) are no part of the tree, an
approver does not see them, and they are not in the copy. Its tasks install or build what
they need, and what they write stays in the copy. On Kubernetes the copy is uploaded read
only, as for a run from a ref: a task writes only into what it `generates`. The run fails when the sandbox's tree is
no longer the approved one; submit it again.
* **No links out.** The files are written by tasks and by whoever pushed the commit, so the
server does not trust them. A `src`, mount or output of a task, or one it inherits from a
task it depends on, that leaves the sandbox or goes through a symbolic link fails the run,
and so does one the server can not look into. It is checked when the run starts and again
before each container is made, once it has started (where what the container binds is
compared with the sandbox's files, see [build files](/docs/remote-compute/build-files)) and
before outputs are copied: a task that leaves `src -> /` behind does
not give the next run the server's disk.
* **Containers only.** The tasks of a run on a sandbox may only use the runtimes in
`sandboxes.runtimes`, `docker` by default. A local task would run as the user of the
server and could read every other sandbox on the machine, so `local` has to be listed
there by name, and in `runtimes`.
* **Network.** No task or service of a run on a sandbox reaches more than
`sandboxes.network`, `none` by default: only the services it needs, nor more than
[`runs.network`](/docs/remote-compute/runs#config), which holds for every run. A build
file can narrow it, never widen it, see [network limits](/docs/task/network).
* **Variables.** An `env:` secret of a run on a sandbox reads only what
`sandboxes.env.allow` lists, nothing by default: the code that would read it is nobody's
reviewed code. So does a build file: `$NAME` or `${NAME}` in its `envs` may read `PATH`,
`HOME`, `CI` and what the server sets about the run, else only a variable
`sandboxes.env.allow` lists. A run that reads another is refused with which task reads
what.
* **Cache.** Runs on sandboxes share a cache per owner, never with runs from a ref: a
sandbox holds changes nobody reviewed, and an entry they leave must not reach a reviewed
run. An approved run has a cache of its own, apart from the owner's runs that needed no
approval.
* **Checks.** With `"check": true` in the `source` the run works on a copy of the files a
push would commit instead of the sandbox, and what it passes counts for a CI job that
verifies the pushed commit, see [Checks of a sandbox](/docs/remote-compute/checks#checks-of-a-sandbox).
What a run wrote stays in the sandbox (not what an approved run or a check wrote, they work
on a copy) and counts against the owner's bytes, the docker
volumes of its outputs included, whatever permissions a task gave its files. While a
caller's sandboxes take more than their limit, their runs on sandboxes are refused with
`413`.
| Status | When |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `400` | The sandbox does not exist or is somebody else's, or a task reads an `env:` secret that `sandboxes.env.allow` does not list. |
| `403` | No entry in the `sandboxes` of the caller's entitlements covers the repository and the branch of the sandbox (any more), a check's caller has no `repos` for the repository, an account of the run does not list `sandbox` in its `source`, or the server runs no tasks on sandboxes (only `runtimes: [local]`, see below). |
| `413` | The caller's sandboxes take more than their limit. |
## Files [#files]
The files of a sandbox are read, searched and edited through the API, in two ways:
* **Typed endpoints** for clients without file tools of their own, such as an MCP server: a
tree, numbered reads, ripgrep, batches of edits and the changes. Their parameters follow
the tools models are trained on: `offset`/`limit` reads in `cat -n` style, exact string
replacement with `old_string`/`new_string`, git diffs and OpenAI's `apply_patch`, and
ripgrep's options.
* **A raw surface** for agent harnesses that bring their own tools and only swap where they
run (pi's tool operations, for example): read bytes, write bytes, stat.
| Call | What it does |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /v1/sandboxes//tree` | The files and directories git sees, see [Tree](#tree). |
| `GET /v1/sandboxes//files/