# Caches (/docs/build-file/caches) A cache combines two things: a **method** that decides if a task changed and a **backend** that stores the task result so it can be reused later. By default hammerkit caches task results on the local filesystem under `~/.hammerkit/remote-cache`, so a result built in one checkout can be reused in another on the same machine. Since `1.6.0` you can also declare named caches with a remote **backend** (for example an S3 bucket or a container registry) so a task built on one machine can be restored on another - ideal for sharing results between developers and CI runs. ## Which caching mechanism? [#which-caching-mechanism] Hammerkit has three related features that all "save work". Pick by where the result needs to go: | Mechanism | What it does | Reach for it when | | ---------------------------------------------------------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------- | | **Cache method** (`checksum`/`modify-date`) | Decides whether a task can be **skipped** locally. | Always on — it's how a task knows it's up to date. | | **Cache backend** (`local`/`s3`/`registry`, this page) | **Shares** the skip-or-restore result across machines automatically. | Developers and CI should reuse each other's results without scripting. | | **[`export: true`](/docs/task#exporting-generated-files)** | Copies a container task's output **back into your workspace**. | Another tool outside hammerkit needs the produced files. | | **[`store` / `restore`](/docs/cli/store-restore)** | Manually moves outputs + cache state to a directory. | Wiring hammerkit into a CI provider's own cache step (no remote backend). | A remote **backend** and **store/restore** solve the same "share across CI runs" problem two ways: with a backend, pull/push is automatic and you don't script the cache step; with store/restore, you hand the directory to your CI's cache. If a remote backend is configured, you usually don't need store/restore at all. See the [CI caching guide](/docs/guides/ci-caching). ## Declaring caches [#declaring-caches] Caches are declared in a top-level `caches:` block. Each entry has a `method` and a `backend`. ```yaml title=".hammerkit.yaml" caches: remote: method: checksum backend: type: s3 bucket: my-build-cache region: eu-central-1 tasks: build: cache: name: remote src: - src generates: - dist cmds: - tsc -b ``` See [task caching](/docs/task/caching) for how a task references a cache. ## Method [#method] The method decides whether the source files of a task changed since the last run. * `checksum` (default) - compares the content checksum of the source files. * `modify-date` - compares the last modification date of the source files. Faster on large folders, but less precise. * `none` - never skips the task; the commands run every time. ## Backends [#backends] The backend decides where task results are stored. Three backend types are built in. Hammerkit pulls from the backend when the result is missing locally and pushes to it after a task ran successfully. ### local [#local] Stores cache entries on the local filesystem. Useful to share a cache between projects on the same machine. This is the backend used by the built-in `default` cache. ```yaml title=".hammerkit.yaml" caches: shared: method: checksum backend: type: local path: /shared/hammerkit-cache ``` | Field | Required | Description | | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------- | | `path` | no | Directory for the cache. Defaults to `~/.hammerkit/remote-cache`, or `$HAMMERKIT_DIR/remote-cache` when `HAMMERKIT_DIR` is set. | ### s3 [#s3] Stores cache entries in an S3-compatible bucket. Works with AWS S3 and any S3-compatible service such as MinIO, Cloudflare R2, Google Cloud Storage or Wasabi. ```yaml title=".hammerkit.yaml" caches: remote: method: checksum backend: type: s3 bucket: my-build-cache region: eu-central-1 endpoint: https://minio.example.com # for S3-compatible services prefix: hammerkit ``` | Field | Required | Description | | ---------------- | -------- | ------------------------------------------------------------------------------- | | `bucket` | yes | Name of the bucket cache entries are written to. | | `region` | no | Region of the bucket. | | `endpoint` | no | Custom endpoint for S3-compatible services (MinIO, R2, ...). | | `prefix` | no | Key prefix for all cache entries, useful to share a bucket with other data. | | `forcePathStyle` | no | Use path-style bucket addressing. Defaults to `true` when an `endpoint` is set. | Credentials are read from the standard AWS SDK credential chain, for example the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables or the shared `~/.aws/config` file. They are never stored in the build file. #### Cloudflare R2 [#cloudflare-r2] R2 exposes an S3 API, so point `endpoint` at your account endpoint and use an `auto` region: ```yaml caches: remote: method: checksum backend: type: s3 bucket: my-build-cache region: auto endpoint: https://.r2.cloudflarestorage.com ``` #### Google Cloud Storage (GCS) [#google-cloud-storage-gcs] GCS offers an S3-compatible XML API, so it works through the `s3` backend. Point `endpoint` at `storage.googleapis.com`, use path-style addressing, and provide [HMAC interoperability keys](https://cloud.google.com/storage/docs/authentication/hmackeys) through the AWS credential chain (`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`): ```yaml caches: remote: method: checksum backend: type: s3 bucket: my-build-cache region: europe-west1 endpoint: https://storage.googleapis.com forcePathStyle: true ``` Azure Blob Storage is **not** S3-compatible, so it is not supported through the `s3` backend. The backend registry is pluggable, so an Azure backend could be added as a custom backend in the future. ### registry [#registry] Stores cache entries in any OCI container registry — GHCR, Docker Hub, ECR, GAR, Artifactory, Harbor or a plain `registry:2`. No bucket to provision: use the registry you already push images to. Each entry is stored as a small single-layer image tagged `-`. ```yaml title=".hammerkit.yaml" caches: remote: method: checksum backend: type: registry repository: ghcr.io/my-org/hammerkit-cache ``` | Field | Required | Description | | ------------ | -------- | ------------------------------------------------------------------------- | | `repository` | yes | Repository to store entries in, without a tag (e.g. `ghcr.io/org/cache`). | | `insecure` | no | Use plain http. Defaults to `true` only for `localhost` / `127.0.0.1`. | Credentials come from the docker config that `docker login` (or a CI login step such as `docker/login-action`) writes, including credential helpers (`credHelpers` / `credsStore`). `DOCKER_CONFIG` is honored. They are never stored in the build file. A read-only token is enough for runners that only pull. They are only sent to the registry itself: an upload location or a next page on another host, port or scheme gets none, as a redirect to one does. Removing entries (`hammerkit clean --cache`) requires the registry to allow manifest deletion; retention is otherwise best left to the registry's own cleanup policies. ## Retention [#retention] Caches grow with every new version of every task. A `retention` block bounds a cache: ```yaml title=".hammerkit.yaml" caches: default: method: checksum backend: type: local retention: maxAge: 30d # remove entries not used for 30 days maxSize: 20Gi # then remove least recently used entries above 20Gi keepPerTask: 3 # keep at most the 3 newest versions of each task ``` | Field | Description | | ------------- | ---------------------------------------------------------------------------------------------- | | `maxAge` | Remove entries not used within this long (`30s`, `5m`, `12h`, `30d`, …). | | `maxSize` | Remove least recently used entries until the cache is at most this big (`500Mi`, `5Gi`, `2G`). | | `keepPerTask` | Keep only the newest N versions (state keys) of each task. | Policies apply in that order: `keepPerTask`, then `maxAge`, then `maxSize`. Removing an entry never produces a wrong result — the task just rebuilds next time. A **local** cache with `retention` is pruned automatically after every successful run. Remote caches are only pruned when you ask for it: ```bash hammerkit cache ls --remote shared # what's in it hammerkit cache prune --remote shared --dry-run # what the policy would remove hammerkit cache prune --remote shared --max-age 14d # prune, overriding the policy ``` See [cache ls / prune](/docs/cli/cache#cache-ls-and-cache-prune). The local backend records when each entry was last used, so `maxAge` and `maxSize` evict what hasn't been used recently. S3 and registries only know when an entry was created, so there they evict by age of creation. Registries must allow deleting manifests (`registry:2`, Harbor, ECR, GAR); for GHCR and Docker Hub use the registry's own retention settings instead — for GHCR, for example, the [`actions/delete-package-versions`](https://github.com/actions/delete-package-versions) action. S3 lifecycle rules work alongside `cache prune` too. ## Built-in caches [#built-in-caches] Two caches are always available without declaring them: * `default` - uses the `local` backend with the `checksum` method. This is what a task uses when it does not specify a cache, so results are cached under `~/.hammerkit/remote-cache` by default (`$HAMMERKIT_DIR/remote-cache` when `HAMMERKIT_DIR` is set). * `none` - uses the `none` method, disabling skipping entirely. ### Changing the default for all tasks [#changing-the-default-for-all-tasks] Declaring a `caches.default` entry overrides the built-in `default` cache, so the new backend applies to **every task that does not reference a cache explicitly**. This is the way to point all tasks at a shared local path or a remote bucket without touching each task. ```yaml title=".hammerkit.yaml" caches: default: method: checksum backend: type: s3 bucket: my-build-cache region: eu-central-1 ``` The `--cache` flag still overrides the *method* for these implicit tasks at runtime (it defaults to `checksum`), so redeclaring `caches.default` is primarily how you change the default *backend*. See [task caching](/docs/task/caching#changing-the-default-for-all-tasks). ## Pull / push behavior [#pull--push-behavior] When a cache has a remote backend (`local` or `s3`): * **Pull** - before a task runs, if its result is not present locally, hammerkit pulls it from the backend and skips the task if the sources are unchanged. * **Push** - after a task ran successfully, hammerkit pushes the result to the backend so other machines can reuse it. Backend errors never fail the build. If the backend is unreachable, the pull or push is skipped with a warning and the task runs or completes as usual. # Environment Variables (/docs/build-file/environment-variables) ## Shell environment [#shell-environment] Variables from the shell environment are passed to a command when the task declares them: ```yaml title=".hammerkit.yaml" tasks: example: envs: VERSION: $VERSION cmds: - echo $VERSION ``` Every environment variable a task uses should be declared in its `envs`. Hammerkit checks that each one is defined and otherwise fails with an error, to prevent undesired behavior. ## .env file [#env-file] Environment variables can be defined in a `.env` file. This is recommended for secret values that should neither be committed nor made public. ```txt title=".env" NPM_TOKEN=abc ``` ```yaml title=".hammerkit.yaml" tasks: example: envs: NPM_TOKEN: $NPM_TOKEN cmds: - npm publish ``` ## Defined values in the build file [#defined-values-in-the-build-file] Environment variables can be defined in the build file itself. They can be defined for the entire build file or only for specific tasks. ```yaml title=".hammerkit.yaml" envs: NODE_VERSION: '22' tasks: example: cmds: - echo $NODE_VERSION override_example: envs: NODE_VERSION: '20' cmds: - echo $NODE_VERSION ``` Variables defined in a build file are scoped to that file. Neither [referenced](/docs/build-file/references) nor [included](/docs/build-file/includes) tasks have access to them. Pass a value on to an include with [`with`](/docs/build-file/includes#inputs-of-a-local-include), as `NODE_VERSION: ${NODE_VERSION}`. ## Cache key [#cache-key] Every variable a task receives, including each build-file `envs` value, is part of its [cache key](/docs/task/caching): changing one reruns the task. A build-file value only a few tasks need — an image tag, a test token — therefore reruns every task in the file when it changes. Define such values on the tasks that use them. [`validate`](/docs/cli/validate) warns about this: for each task with commands it lists the build-file `envs` its definition never mentions as `$NAME` or `${NAME}`. ``` warn at lint build-file envs E2E_TOKEN are part of this task's cache key but not referenced by it: … ``` Hammerkit can't see what a tool reads at runtime, so a variable such as `NODE_ENV` that every task's tooling picks up on its own belongs in the build-file `envs` despite the warning. ## Precedence [#precedence] A variable can be defined in more than one place. Hammerkit resolves it like this: 1. **Task `envs`** win over **build-file `envs`** on the same key. In the example above `override_example` prints `20`, while `example` prints `22`. 2. A value of exactly `$NAME` is a *reference* that is filled in from, in order: 1. a **`--set NAME=value`** argument on the hammerkit command, then 2. the **shell / process environment**, then 3. a **`.env`** file in the build file's directory. If a referenced `$NAME` is set nowhere, hammerkit aborts before running with `missing environment variable NAME`. A variable set to an empty string counts as set, as in a shell: `export NAME=` passes an empty value, which is how a CI job provides a variable its provider leaves unset on some pipelines. Every variable a task uses must be declared in its `envs` (directly or via `$NAME`). Hammerkit only passes declared variables to the command, so an undeclared shell variable is *not* leaked into the task. ## Interpolation [#interpolation] There are two distinct places a variable can appear: **Inside a command** (`cmds`), `$NAME` is expanded by the shell at runtime, using the variables hammerkit passed in — exactly like a normal shell. This is the `echo $NODE_VERSION` case above. **Inside other task fields**, hammerkit substitutes `$NAME` itself when it plans the task — before anything runs. This works for `image`, `src`, `generates`, `mounts`, `ports`, `shell`, command working directories and a service's `image`, so you can drive them from a single value: ```yaml title=".hammerkit.yaml" envs: NODE_VERSION: '24' tasks: build: image: node:$NODE_VERSION-alpine # -> node:24-alpine src: - src cmds: - tsc -b ``` **Inside an `envs` value**, `${NAME}` is replaced anywhere in the string, and `${NAME:-default}` falls back to `default` when `NAME` is set nowhere. A name comes from the shell / process environment, then a `.env` file, then a literal value of the same `envs` (build file and task merged), then the default: ```yaml title=".hammerkit.yaml" envs: DB_USER: app tasks: e2e: envs: DATABASE_URL: postgres://${DB_USER}@db:5432/app # -> postgres://app@db:5432/app SPEC: ${E2E_SPEC:-e2e/**/*.cy.ts} # E2E_SPEC=e2e/login.cy.ts narrows it MR_IID: ${CI_MERGE_REQUEST_IID:-} # empty outside merge request pipelines cmds: - cypress run --spec "$SPEC" ``` A `${NAME}` without a default that is set nowhere fails like a missing `$NAME`, when a task that declares it runs. The whole value `$NAME` (no braces) is a reference to the shell or `.env` only, as before. Field substitution matches `$NAME` and `${NAME}` anywhere in the string and is case-insensitive; a value defined in `envs` (like `NODE_VERSION: '24'`, or one built with `${…}`) is what gets substituted into fields such as `image`. A name is matched as the text it is (a `.` in it is a dot), and a value is put in as it is written: a `$NAME` inside the value that was put in is not substituted again. # Includes (/docs/build-file/includes) Includes are ideal for defining repetitive tasks once and reusing them in multiple places. Referenced files also split up tasks, but their working directory stays fixed to the directory they are in. For includes, the working directory is the directory of the including file. The following example is a TypeScript monorepo with two projects, `a` and `b`. Project `b` depends on `a`, and both need up-to-date dependencies before compiling the source code. ```yaml title="build.npm.yaml" tasks: install: cmds: - npm install src: - package.json - package-lock.json generates: - node_modules ``` The npm install uses an include: each project has its own package.json, but the task definition is the same. ```yaml title="project/a/.hammerkit.yaml" tasks: build: deps: [npm:install] cmds: - node_modules/.bin/tsc -b includes: npm: ../../build.npm.yaml ``` The build task [depends](/docs/task/dependencies) on the included install task, which ensures that node_modules is installed first. ```yaml title="project/b/.hammerkit.yaml" tasks: build: deps: [npm:install, a:build] cmds: - node_modules/.bin/tsc -b references: a: ../a includes: npm: ../../build.npm.yaml ``` The build file of project `b` uses the same include for the npm install task. An additional [reference](/docs/build-file/references) to `a` and a [task dependency](/docs/task/dependencies) on `a:build` ensure that project `a` is built before project `b` is compiled. An include can also be an object with a `path`, which takes `publish` (and [`with`](#inputs-of-a-local-include)): ```yaml includes: npm: { path: ../../build.npm.yaml } dev: { path: ./dev.yaml, publish: false } ``` `publish: false` keeps what the included file declares on your machine: a [server](/docs/remote-compute/build-files#only-part-of-a-build-file) leaves it out of the version it publishes. A git include takes `publish` too. ## Include from a git repository [#include-from-a-git-repository] Instead of a path, an include (or a [reference](/docs/build-file/references)) can name a git repository. Use this to share build files between projects without copying them. ```yaml title=".hammerkit.yaml" includes: npm: git: https://github.com/no0dles/hammerkit.git ref: v1.16.0 path: catalog/npm tasks: build: deps: [npm:install] cmds: - npm run build ``` | Field | Meaning | | ------ | ------------------------------------------------------------------------------------------------- | | `git` | Repository URL or path. Required. | | `ref` | Branch, tag or commit SHA. The repository's `HEAD` when left out. | | `path` | Build file, or a directory holding one, inside the repository. The repository root when left out. | | `with` | Values for the inputs of the included file, see [inputs](#inputs). | A plain string stays a local path, exactly as before. Prefer `includes` for shared task definitions: they run in your directory, while a git `reference` runs inside the cached checkout. ### Inputs [#inputs] An included file declares its inputs as ordinary top-level `envs`, each with a default, and uses them in its tasks. `with` replaces the defaults for one include: ```yaml title="build.npm.yaml (in the repository)" envs: NODE_VERSION: 22 tasks: ci: image: node:$NODE_VERSION-alpine cmds: - npm ci src: - package.json - package-lock.json generates: - node_modules ``` ```yaml title=".hammerkit.yaml" includes: npm: git: https://github.com/no0dles/hammerkit.git ref: v1.16.0 path: catalog/npm with: NODE_VERSION: 22 ``` A name the included file does not declare is an error that lists the declared inputs, so a typo cannot silently do nothing. Values are strings or numbers. A value is used as it is written; to let CI set an input from outside, write `NODE_VERSION: ${NODE_VERSION:-22}`, which is the environment's (or a `.env` file's) `NODE_VERSION`, else `22`. The same file included under two names with different inputs gives two independent sets of tasks. ### Inputs of a local include [#inputs-of-a-local-include] A local include (or reference) takes inputs too: write it as an object with a `path`. ```yaml title=".hammerkit.yaml" envs: NODE_VERSION: 24 includes: npm: path: build.npm.yaml with: NODE_VERSION: ${NODE_VERSION} web: path: web/build.yaml with: NODE_VERSION: ${NODE_VERSION} ``` `${NAME}` (or `${NAME:-default}`) in a `with` value reads as it does in an `envs` value of the including file: the environment's (or a `.env` file's) `NAME` if it is set, else the including file's own `envs` value. So the Node version lives in one place, every include builds with it, and CI can still set it. A file that is included passes it on to its own includes the same way. A name the including file does not declare stays as it is, and the included file reads it from the environment as above. A value the including file builds from other variables (`${BASE}-alpine`) is passed on as it is written. A plain string path stays a local include without inputs, exactly as before. ### Limits [#limits] Includes and references nest at most 32 levels deep, and one build file reads at most 1,000 files through them, 16 MiB of them, with at most 10,000 tasks and services; a file included again with other inputs counts again. The inputs one include passes on take at most 64 KiB. A file that includes itself with inputs that change each time stops at these with an error that names the include. ### Caching and offline use [#caching-and-offline-use] The first run fetches the repository at `ref` into `~/.hammerkit/includes` and every later run uses that copy without touching the network, so a build keeps working offline and does not change underneath you. The cache is the pin: there is no lockfile. * A **commit SHA** (or a release tag you trust) gives the same content on every machine. * A **branch** is frozen at the commit of the first fetch on *each machine*, so a fresh CI runner can see newer content than a developer laptop. Pin a SHA when that matters. * A repository that was never fetched and is unreachable fails with an error naming the repository and ref. There is no silent fallback. To update a branch or tag, run [`hammerkit includes pull`](/docs/cli/includes); a commit SHA stays as it is. [`hammerkit clean --cache`](/docs/cli/clean) removes the cached repositories, so the next run fetches them again. ### What a remote file may do [#what-a-remote-file-may-do] A remote build file contributes tasks, services, caches and environments. Fetching it runs nothing; commands run only when you run one of its tasks. Its own relative `includes` and `references` resolve inside the same repository at the same commit, and a path that leaves the repository is rejected. Private repositories use your usual git credentials (credential helpers, SSH keys); hammerkit never prompts and has no secrets of its own for this. ## references vs. includes vs. extend — when to use which [#references-vs-includes-vs-extend--when-to-use-which] All three reuse definitions, but they differ in *what* they reuse and *where it runs*: | | What it pulls in | Working directory | Use it when | | --------------------------------------------- | ----------------------------------------------- | --------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | **[references](/docs/build-file/references)** | A whole other build file (its tasks + services) | The **referenced** file's own directory | The other file is its own project (a sub-package you also want to build in place). | | **includes** | A whole other build file | The **including** file's directory | You want to reuse a task *definition* and run it here — the same `install`/`build` template across every package. | | **[extend](/docs/task/extending)** | A single task, as a base template | The current task's file | One task is almost another, with a tweak or two. | A common monorepo setup combines them: an `includes` brings in a shared `build.npm.yaml` so each package runs `install` in its own directory, while `references` wire up cross-package build order. See the [monorepo guide](/docs/guides/monorepo). # Build file (/docs/build-file) The build file is a YAML file in the root of your project. It contains a list of [tasks](/docs/task) that can be executed from the [CLI](/docs/cli). Hammerkit looks for the build file by name, in this order: 1. `.hammerkit.yaml` 2. `.hammerkit.yml` 3. `build.yaml` All three are valid; the first one found is used. If more than one exists hammerkit warns about the ambiguity. These docs use `.hammerkit.yaml` as the canonical name in every example. To point hammerkit at a different file, pass [`--file `](/docs/cli). ```yaml title=".hammerkit.yaml" envs: NODE_VERSION: '22' tasks: build: image: node:$NODE_VERSION deps: [npm:install] cmds: - tsc -b references: subProject: ./packages/subproject includes: npm: ./build.npm.yaml ``` For more detail about the structure of the build file, check out the following sections. # Build file reference (/docs/build-file/reference) This is the complete reference for the hammerkit build file, derived from the schema hammerkit validates against (`build.schema.json`). Unknown keys are rejected, so this list is exhaustive. For prose and examples, follow the links to the dedicated pages. ## Top-level keys [#top-level-keys] | Key | Type | Description | | -------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `envs` | map of string→string/number | Build-file-wide [environment variables](/docs/build-file/environment-variables). | | `tasks` | map of name→task | The [tasks](/docs/task) in this file. | | `services` | map of name→service | The [services](/docs/service) in this file. | | `references` | map of prefix→path, `{path, with, publish}` or git source | [Reference](/docs/build-file/references) other build files (keep their own directory). | | `includes` | map of prefix→path, `{path, with, publish}` or git source | [Include](/docs/build-file/includes) other build files (use this file's directory), with [inputs](/docs/build-file/includes#inputs-of-a-local-include). `publish: false` leaves the file out of a [published version](/docs/remote-compute/build-files#only-part-of-a-build-file). | | `environments` | map of name→environment | [Run targets](/docs/task/kubernetes) (Kubernetes or remote Docker). | | `caches` | map of name→cache | Named [cache](/docs/build-file/caches) declarations (method + backend). | | `labels` | map of string→string/number | [Labels](/docs/labels) applied to every task/service in the file. | | `runner` | string | Default [`runner`](/docs/task#runner) of the tasks in this file. | ## Task [#task] A task is either a **container task** (it has an `image`, its own or the one of the task it [extends](/docs/task/extending)), an **image build** (it has [`build`](/docs/task/image-builds)) or a **local task** (neither). They share these fields (an image build has no `cmds`, `shell`, `extend` or `args`): | Field | Type | Description | | ------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cmds` | array of string \| `{cmd, path}` | Commands to run, in order. `path` sets the working directory for that command. | | `description` | string | Shown by [`ls`](/docs/cli/ls); a missing one is a [validate](/docs/cli/validate) warning. | | `src` | array of string | Input files/folders/globs used for [caching](/docs/task/caching). | | `generates` | array of string \| `{path, export, resetOnChange, name}` | [Output](/docs/task#generated-files-generates) paths. `export: true` copies back to the workspace when the task succeeds, `export: always` also when it fails; outputs start empty when the task runs, `resetOnChange: false` keeps their content (see [outputs start empty](/docs/task#outputs-start-empty)). | | `deps` | array of string | [Tasks](/docs/task/dependencies) (`name` or `prefix:name`) that run first. | | `needs` | array of string \| `{service, name}` | [Services](/docs/task/needs) that must be ready first. | | `envs` | map of string→string/number | Task-level [environment variables](/docs/build-file/environment-variables) (override build-file `envs`). | | `labels` | map of string→string/number | [Labels](/docs/labels) for filtering. | | `cache` | string \| `{name, method}` | [Cache](/docs/task/caching) method shorthand or a named cache reference. | | `extend` | string | [Base task](/docs/task/extending) to inherit from (`prefix:name`). | | `shell` | string | Shell used to run `cmds` (default `/bin/sh`). | | `timeout` | duration (`30s`, `10m`, `1h30m`) | [Maximum execution time](/docs/task#timeouts); the task fails when exceeded. | | `secrets` | array of `{from, env \| path, cache}` | [Credentials](/docs/task/secrets) from `env:NAME` or `file:path`, injected as an env variable or a read-only file; kept out of the cache key and logs. | | `resources` | `{cpus, memory}` \| `{requests, limits}` | [CPU and memory](/docs/task#resources) limits and requests (`2`, `500m`; `512Mi`, `2Gi`); the run is planned by the requests. Not enforced for a local task, not part of the cache key. | | `network` | `open` \| `none` \| `{allow: [hosts]}` | What the container [reaches](/docs/task/network) besides the services in `needs` (default `open`); `allow` goes through an egress proxy, and a `host:port` entry forwards raw TCP. An error on a local task, except on a [server's cluster](/docs/remote-compute/backend#network); not part of the cache key. | | `args` | boolean | Whether the last command takes [arguments](/docs/task/arguments#which-tasks-take-arguments) after `--`: `false` refuses them, `true` says it is meant to, which a server with `runs.args: declared` requires. | | `runner` | string | Name of the [server](/docs/cli/remote) the task has to run on (experimental, see [runner](/docs/task#runner)). | | `publish` | boolean | `false`: a server leaves the task out of a [published version](/docs/remote-compute/build-files#only-part-of-a-build-file). Locally it changes nothing. | **Container task** adds: | Field | Type | Description | | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `image` | string (required, unless the task [extends](/docs/task/extending#container-properties-without-an-image) a container task, whose image it takes) | The container image to run in. Its presence is what makes the task containerized. | | `mounts` | array of string | Extra `host[:container][:ro]` [mounts](/docs/task/container#adding-mounts). | | `volumeSize` | size (`5Gi`) | On Kubernetes, the size of the [claim](/docs/task/kubernetes#claims) that holds the sources, mounts and outputs of the task (the environment's `volumeSize` when left out, else `1Gi`). Ignored elsewhere, not part of the cache key. | | `capAdd` | array of string | Linux [capabilities](/docs/task/container#capabilities) added to the container, such as `SYS_ADMIN`. | **Image build** adds: | Field | Type | Description | | ------------ | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `build` | `{context, dockerfile, tags, push, cache, digest, login, insecure}` (required) | The Dockerfile to [build](/docs/task/image-builds) with BuildKit, as `docker build` does: its context, tags, whether to push, a registry `cache` for its layers, a file for the `digest` of the pushed image, and `login`s of `{registry, user, password: {from, account}}`. Without `src`, the task reads the context. | | `volumeSize` | size (`5Gi`) | As for a container task. | **Local task** adds: | Field | Type | Description | | ---------- | ------------ | ----------------------------------------------------------------------- | | `platform` | `{os, arch}` | Restrict to `os` (`win`/`macos`/`linux`) and/or `arch` (`arm64`/`arm`). | ## Service [#service] A service is either a **container service** (it has an `image`) or a **Kubernetes service** (it has a `selector`). **Container service:** | Field | Type | Description | | ------------- | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `image` | string (required) | The service image. | | `cmd` | string | Override the container command. | | `workdir` | string | Working directory inside the container; see [working directory](/docs/service/container#working-directory). Defaults to the build file's directory. | | `shell` | string | Run `cmd` and the healthcheck as ` -c ""`; see [shell](/docs/service/container#shell). Without it both run in exec form. | | `ports` | array of string/number | `containerPort` or `hostPort:containerPort` to publish to the host. | | `healthcheck` | `{cmd, timeout}` | Readiness command, and how long the service may take to pass it once running (duration, default `20s`, `HAMMERKIT_HEALTHCHECK_TIMEOUT` changes the default); see [healthcheck](/docs/service/container#healthcheck). | | `restarts` | integer 0 to 10 | On Kubernetes, how often the container may crash and be restarted while the service starts before it fails (default `2`); see [restarts](/docs/service/container#restarts). On Docker a crash fails it at once. | | `init` | string | A task (`name` or `prefix:name`) run once the healthcheck passed; the service is ready for dependents only after it succeeded; see [init](/docs/service/container#init). | | `envs` | map of string→string/number | Service environment variables. | | `secrets` | array of `{from, env \| path}` | [Credentials](/docs/task/secrets) injected as an env variable or a read-only file. | | `resources` | `{cpus, memory}` \| `{requests, limits}` | [CPU and memory](/docs/service/container#resources) limits and requests of the container, as for a task. | | `network` | `open` \| `none` \| `{allow: [hosts]}` | What the service [reaches](/docs/task/network) besides the services it needs, as for a task; what needs the service always reaches it. | | `args` | boolean | `false` refuses [arguments](/docs/task/arguments#services) after `--` of `up` for its `cmd`. | | `volumeSize` | size (`20Gi`) | On Kubernetes, the size of the [claim](/docs/task/kubernetes#claims) that holds the service's sources, mounts and volumes, as for a task. | | `volumes` | array of string | `name:containerPath[:ro]` volumes that [persist](/docs/service/container#volumes) across restarts. | | `mounts` | array of string | `host:container[:ro]` config mounts. | | `deps` | array of string | Tasks that must run before the service starts. | | `needs` | array of string \| `{service, name}` | Other services this one depends on. | | `src` | array of string | Sources, used for caching the service image build. | | `labels` | map of string→string/number | Labels for filtering. | | `description` | string | Description shown by `ls`. | | `continuous` | boolean | The service [watches its own files](/docs/service/container#continuous), such as a dev server: `up --watch` leaves it running, and a move of a service run on Kubernetes writes the changed files into its pod. | | `publish` | boolean | `false`: a server leaves the service out of a [published version](/docs/remote-compute/build-files#only-part-of-a-build-file). | **Kubernetes service** (forwards an existing cluster resource — see [Kubernetes service](/docs/service/kubernetes)): | Field | Type | Description | | ------------------------ | --------------------------------- | --------------------------------------------------------------------- | | `selector` | `{type, name}` (required) | Resource `type` (`deployment`/`service`/`pod`) and `name` to forward. | | `ports` | array of string/number (required) | `hostPort:containerPort` mappings to forward to localhost. | | `context` | string | Override the environment's kube context. | | `namespace` | string | Override the environment's namespace. | | `kubeconfig` | string | Override the environment's kubeconfig path. | | `deps` | array of string | Tasks that run before forwarding. | | `labels` / `description` | – | As above. | ## Cache (`caches`) [#cache-caches] Each named cache has a `method` and a `backend` (see [caches](/docs/build-file/caches)): | Field | Type | Description | | ----------------------------------------------------------- | ------------------------------------- | ---------------------------------------------------------------------------------------------- | | `method` | `checksum` \| `modify-date` \| `none` | How changes are detected. | | `backend.type` | `local` \| `s3` \| `registry` | Where results are stored. | | `backend.path` | string (`local`) | Directory; defaults to `~/.hammerkit/remote-cache`. | | `backend.bucket` | string (`s3`, required) | Target bucket. | | `backend.region` / `endpoint` / `prefix` / `forcePathStyle` | – (`s3`) | S3 connection options. | | `backend.repository` | string (`registry`, required) | OCI repository without a tag, e.g. `ghcr.io/org/cache`. | | `backend.insecure` | boolean (`registry`) | Use plain HTTP; defaults to `true` only for localhost. | | `retention.maxAge` / `maxSize` / `keepPerTask` | duration / size / integer | [Retention policy](/docs/build-file/caches#retention) for `cache prune` and automatic pruning. | A task's `cache` field is either the method shorthand (`cache: checksum`) or a reference `{name, method}` to a declared cache. ## Environment (`environments`) [#environment-environments] Each environment is a [run target](/docs/task/kubernetes) — exactly one of: | Key | Fields | Description | | ------------ | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `kubernetes` | `context` (required), `namespace`, `kubeconfig`, `httpRoutes[]`, `networkPolicy` | Run tasks as jobs and services as deployments on a cluster. `networkPolicy: enforced` states that the cluster enforces NetworkPolicy, which [network limits](/docs/task/network#kubernetes) need. | | `docker` | `host` | Run against a remote Docker daemon. | An `httpRoutes` entry has `host`, `service` and `gateway` (all required), `servicePort`, `path` and `gatewayNamespace` — see [HTTP routes](/docs/task/kubernetes#http-routes). # References (/docs/build-file/references) Each build file can have a `references` map pointing at other build files. The referenced file brings in more tasks and services, which you can then use directly from the [cli](/docs/cli) or as [task dependencies](/docs/task/dependencies). ```yaml title=".hammerkit.yaml" references: foo: project/foo ``` ```yaml title="project/foo/.hammerkit.yaml" tasks: example: cmds: - echo "foo bar" ``` ## Naming and prefixes [#naming-and-prefixes] A referenced task is addressed as `:`, where the prefix is the key you gave it in the `references` map. The `example` task above becomes `foo:example`: ```bash hammerkit foo:example ``` The same `prefix:name` syntax is what you use in `deps` and `needs`: ```yaml title=".hammerkit.yaml" references: foo: project/foo tasks: build: deps: [foo:example] cmds: - echo "after foo" ``` References can be nested — a referenced file can have its own references — and the prefixes stack (`foo:bar:task`). This same prefix rule is shared with [includes](/docs/build-file/includes) and [extend](/docs/task/extending). ## References keep their own working directory [#references-keep-their-own-working-directory] The key difference from [includes](/docs/build-file/includes): a referenced task runs with **its own** directory as the working directory. Commands and relative `src`/`generates` paths resolve against the referenced file's location, not yours. Use a reference when the other file is a real project in its own right; use an include when you want to *reuse a task definition* in the current project's directory. Build-file `envs` do **not** cross a reference. Variables declared at the top level of one file are not visible to a referenced (or [included](/docs/build-file/includes)) file — each file's `envs` are scoped to that file. Declare the variables a referenced task needs in that file (or on the task itself). ## References from git [#references-from-git] A reference can also point at a git repository, with `git`, an optional `ref` and an optional `path`. See [include from a git repository](/docs/build-file/includes#include-from-a-git-repository) for the fields and the caching rules. A git reference runs in its cached checkout, so use an [include](/docs/build-file/includes) when the task should run in your directory. ## Only on your machine [#only-on-your-machine] Like an include, a reference can be an object with a `path`, and `publish: false` keeps what the referenced file declares on your machine: a [server](/docs/remote-compute/build-files#only-part-of-a-build-file) leaves it out of the version it publishes. ```yaml references: tools: { path: ./tools, publish: false } ``` # cargo (/docs/catalog/cargo) Build and test a Rust crate or workspace, with the cargo registry kept on the host. `build` compiles into `target`, `test` runs the tests after it. Both are skipped while the `.rs` files and `Cargo.toml`/`Cargo.lock` are unchanged, and downloaded crates are kept in `.cargo` of your home directory, so they are not fetched again. ```yaml title=".hammerkit.yaml" includes: cargo: git: https://github.com/no0dles/hammerkit.git ref: v1.19.0 path: catalog/cargo with: CARGO_ARGS: --locked --workspace tasks: check: deps: [cargo:test] ``` ## Tasks [#tasks] | Task | What it does | Reads | Produces | | ------- | ------------------------------ | -------------------------------------------------- | -------- | | `build` | Build the crate into `target`. | every `Cargo.toml`, `Cargo.lock`, every `.rs` file | `target` | | `test` | Run the tests of the crate. | every `Cargo.toml`, `Cargo.lock`, every `.rs` file | | ## Inputs [#inputs] | Input | Default | Meaning | | -------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `RUST_VERSION` | `1.90` | Tag of the `rust` image the tasks run in. The image is `rust:-slim`. Quote it in `with` (`'1.90'`), or YAML reads it as a number. | | `CARGO_ARGS` | `--locked` | Arguments for `cargo build` and `cargo test`, for example `--locked --workspace`. | ## The entry [#the-entry] ```yaml title="catalog/cargo/build.yaml" # Builds and tests a Rust crate or workspace with the cargo registry kept on the host. envs: # tag of the rust image the tasks run in, quoted so 1.90 stays 1.90 RUST_VERSION: '1.90' # arguments for cargo, for example --locked --workspace CARGO_ARGS: --locked tasks: build: description: build the crate into target image: rust:${RUST_VERSION}-slim src: - Cargo.toml - Cargo.lock - '**/Cargo.toml' - '**/*.rs' generates: [target] mounts: - $PWD/.cargo/registry:/usr/local/cargo/registry - $PWD/.cargo/git:/usr/local/cargo/git cmds: - cargo build ${CARGO_ARGS} test: description: run the tests of the crate image: rust:${RUST_VERSION}-slim deps: [build] src: - Cargo.toml - Cargo.lock - '**/Cargo.toml' - '**/*.rs' mounts: - $PWD/.cargo/registry:/usr/local/cargo/registry - $PWD/.cargo/git:/usr/local/cargo/git cmds: - cargo test ${CARGO_ARGS} ``` # docker (/docs/catalog/docker) Build a Docker image from a Dockerfile and push it, using the Docker of the machine hammerkit runs on. `build` builds the image into the local Docker, `publish` builds it and pushes it to its registry. Both are local tasks that call `docker`, so Docker has to be installed and running where you run hammerkit. ```yaml title=".hammerkit.yaml" includes: docker: git: https://github.com/no0dles/hammerkit.git ref: v1.21.0 path: catalog/docker with: IMAGE: ghcr.io/acme/hello IMAGE_TAG: ${VERSION:-dev} tasks: release: deps: [docker:publish] ``` ## Tasks [#tasks] | Task | What it does | Reads | Produces | | --------- | ----------------------------------------------------------------- | ----- | ------------------------- | | `build` | Build the image and tag it `IMAGE:IMAGE_TAG` in the local Docker. | | the image | | `publish` | Build the image for `PLATFORMS` and push it. | | the image in its registry | Neither task declares `src` or `generates`, so both run every time. The image lives in Docker, not in a folder hammerkit can check, and Docker's own layer cache makes a rebuild of an unchanged context fast. `publish` does not log in. Log in before you run it, with `docker login` or your CI's login step, preferably with a short-lived token of a service account rather than a personal login. The credentials stay in Docker's config and never reach the build file. ## Inputs [#inputs] | Input | Default | Meaning | | ------------ | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `IMAGE` | `app` | The image name, including the registry (`ghcr.io/acme/app`). | | `IMAGE_TAG` | `dev` | The tag of the image. Take it from the environment with `${VERSION:-dev}` in `with`, as above, and run `VERSION=1.0.0 hammerkit release`. | | `DOCKERFILE` | `Dockerfile` | The Dockerfile to build, relative to the working directory. | | `CONTEXT` | `.` | The build context, relative to the working directory. | | `PLATFORMS` | `linux/amd64` | The platforms `publish` builds, comma separated, for example `linux/amd64,linux/arm64`. More than one needs a builder that supports multi-platform images. | ## The entry [#the-entry] ```yaml title="catalog/docker/build.yaml" # Builds a Docker image with the Docker of the machine running hammerkit, and pushes it. envs: # the image name, including the registry (ghcr.io/acme/app) IMAGE: app # the tag of the image IMAGE_TAG: dev # the Dockerfile to build, relative to the working directory DOCKERFILE: Dockerfile # the build context, relative to the working directory CONTEXT: . # the platforms to push, comma separated PLATFORMS: linux/amd64 tasks: build: description: build the image into the local Docker cmds: - docker build --file ${DOCKERFILE} --tag ${IMAGE}:${IMAGE_TAG} ${CONTEXT} publish: description: build the image and push it to its registry cmds: - docker buildx build --file ${DOCKERFILE} --platform ${PLATFORMS} --tag ${IMAGE}:${IMAGE_TAG} --push ${CONTEXT} ``` # dotnet (/docs/catalog/dotnet) Build and test a .NET project or solution, with the NuGet package cache kept on the host. `build` compiles into `artifacts`, `test` runs the tests after it. Both are skipped while the files they read are unchanged, and downloaded packages are kept in `.nuget/packages` of your home directory, so they are not fetched again. ```yaml title=".hammerkit.yaml" includes: dotnet: git: https://github.com/no0dles/hammerkit.git ref: v1.20.0 path: catalog/dotnet with: DOTNET_VERSION: '10.0' PROJECT: tests/Hello.Tests tasks: check: deps: [dotnet:test] ``` `PROJECT` names what `dotnet build` and `dotnet test` take: a project, a solution, or `.` for the one in the directory. Pointing it at the test project builds the app it references as well. ## Tasks [#tasks] | Task | What it does | Reads | Produces | | ------- | ---------------------------------------------------- | ------------------------------ | ----------- | | `build` | Build the project, restoring its packages first. | the files matched by `SOURCES` | `artifacts` | | `test` | Run the tests of the project without building again. | the files matched by `SOURCES` | | The tasks pass `-p:UseArtifactsOutput=true`, so `bin` and `obj` go to `artifacts/` at the root instead of into the project folders. The project folders are the tasks' inputs: build output written into them would change the inputs, and the next run would miss the cache. A built app is in `artifacts/bin///`, in lower case, for example `dotnet artifacts/bin/Hello/release/Hello.dll`. ## Inputs [#inputs] | Input | Default | Meaning | | ---------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DOTNET_VERSION` | `10.0` | Tag of the `mcr.microsoft.com/dotnet/sdk` image the tasks run in. Quote it in `with` (`'10.0'`), or YAML reads it as the number 10. | | `PROJECT` | `.` | The project or solution to build and test. | | `SOURCES` | `{src/**,tests/**,Directory.*,global.json,nuget.config}` | A glob of the files the tasks read: the code under `src` and `tests`, and the shared build files at the root. Replace it when your projects live elsewhere, for example `{app/**,Directory.*}`. | | `CONFIGURATION` | `Release` | The build configuration. | Packages are restored from the sources in your `nuget.config` and are not locked. The [.NET tutorial](/docs/tutorials/dotnet) shows a build file that restores in its own task with `--locked-mode`, for projects that want the restore to fail when a lock file no longer matches. ## The entry [#the-entry] ```yaml title="catalog/dotnet/build.yaml" # Builds and tests a .NET solution with the NuGet package cache kept on the host. envs: # tag of the SDK image the tasks run in, quoted so 10.0 stays 10.0 DOTNET_VERSION: '10.0' # the project or solution to build and test, as `dotnet build` takes it PROJECT: . # a glob of the files the tasks read: the projects and their code, plus the shared # build files at the root (Directory.Build.props, global.json, nuget.config) SOURCES: '{src/**,tests/**,Directory.*,global.json,nuget.config}' # the build configuration CONFIGURATION: Release tasks: build: description: build the project into artifacts image: mcr.microsoft.com/dotnet/sdk:${DOTNET_VERSION} src: - '${SOURCES}' generates: [artifacts] mounts: - $PWD/.nuget/packages:/nuget/packages envs: NUGET_PACKAGES: /nuget/packages DOTNET_CLI_TELEMETRY_OPTOUT: '1' cmds: - dotnet build ${PROJECT} --configuration ${CONFIGURATION} -p:UseArtifactsOutput=true -p:ArtifactsPath=$PWD/artifacts --disable-build-servers test: description: run the tests of the project image: mcr.microsoft.com/dotnet/sdk:${DOTNET_VERSION} deps: [build] src: - '${SOURCES}' mounts: - $PWD/.nuget/packages:/nuget/packages envs: NUGET_PACKAGES: /nuget/packages DOTNET_CLI_TELEMETRY_OPTOUT: '1' cmds: - dotnet test ${PROJECT} --configuration ${CONFIGURATION} --no-build -p:UseArtifactsOutput=true -p:ArtifactsPath=$PWD/artifacts --disable-build-servers ``` # eslint (/docs/catalog/eslint) Lint a project with ESLint 9 or later and its flat config. The task runs in a `node` container and is skipped when the sources and the configuration are unchanged. ## Install first [#install-first] The task runs the tool from `node_modules`, and a container task only sees the files it declares and the outputs of its dependencies, so the install must be a dependency. Include the entry of your package manager ([npm](/docs/catalog/npm), [yarn](/docs/catalog/yarn) or [pnpm](/docs/catalog/pnpm)) and `extend` the task to add it, which works with any of them and keeps the entry free of a package manager: ```yaml title=".hammerkit.yaml" includes: npm: git: https://github.com/no0dles/hammerkit.git ref: v1.22.0 path: catalog/npm eslint: git: https://github.com/no0dles/hammerkit.git ref: v1.22.0 path: catalog/eslint tasks: lint: extend: eslint:check deps: [npm:install] ``` `extend` keeps the entry's image, sources and commands and adds the dependency (see [extending](/docs/task/extending)). ## Tasks [#tasks] | Task | What it does | Reads | Produces | | ------- | ----------------------------- | ----------------------------------------------- | -------- | | `check` | Lint the sources with eslint. | `package.json`, `eslint.config.*`, `SOURCE_DIR` | nothing | ## Inputs [#inputs] | Input | Default | Meaning | | -------------- | ------- | ------------------------------------------------------------------------------------------------------- | | `NODE_VERSION` | `24` | Tag of the `node` image the task runs in, e.g. `22` or `24`. The image is `node:-alpine`. | | `SOURCE_DIR` | `src` | Folder with the sources to lint, relative to the working directory. | ## Fixing [#fixing] The entry only checks. A task that rewrites its own sources cannot be cached, so there is no `--fix` task; run `npx eslint --fix` on your machine when you want one. ## The entry [#the-entry] ```yaml title="catalog/eslint/build.yaml" # Lints a project with ESLint. The project's own dependencies must be installed first, see the # page of this entry. envs: # tag of the node image eslint runs in NODE_VERSION: 24 # folder with the sources to lint, relative to the working directory SOURCE_DIR: src tasks: check: description: lint the sources with eslint image: node:${NODE_VERSION}-alpine src: [package.json, 'eslint.config.*', '${SOURCE_DIR}'] cmds: - node_modules/.bin/eslint ${SOURCE_DIR} ``` # go (/docs/catalog/go) Build and test a Go module, with the module cache kept on the host. `build` compiles the packages into `bin`, `test` runs their tests. Both are skipped while the `.go` files and the `go.*` files are unchanged, and downloaded modules are kept in `go/pkg/mod` of your home directory, so they are not fetched again. ```yaml title=".hammerkit.yaml" includes: go: git: https://github.com/no0dles/hammerkit.git ref: v1.19.0 path: catalog/go with: GO_VERSION: '1.25' tasks: check: deps: [go:test] ``` ## Tasks [#tasks] | Task | What it does | Reads | Produces | | ------- | ------------------------------ | ---------------------------------------------------------------- | -------- | | `build` | Build the packages into `bin`. | `go.mod`, `go.sum` and `go.work` where present, every `.go` file | `bin` | | `test` | Run the tests of the packages. | `go.mod`, `go.sum` and `go.work` where present, every `.go` file | | ## Inputs [#inputs] | Input | Default | Meaning | | ------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GO_VERSION` | `1.25` | Tag of the `golang` image the tasks run in. The image is `golang:-alpine`. Quote it in `with` (`'1.30'`), or YAML reads it as the number 1.3. | | `PACKAGES` | `./...` | The packages to build and test. | ## The entry [#the-entry] ```yaml title="catalog/go/build.yaml" # Builds and tests a Go module with the module cache kept on the host. envs: # tag of the golang image the tasks run in, quoted so 1.30 stays 1.30 GO_VERSION: '1.25' # the packages to build and test PACKAGES: ./... tasks: build: description: build the packages into bin image: golang:${GO_VERSION}-alpine src: # go.mod, and go.sum, go.work and go.work.sum where the module has them - 'go.*' - '**/*.go' generates: [bin] mounts: - $PWD/go/pkg/mod:/go/pkg/mod cmds: - mkdir -p bin - go build -o bin/ ${PACKAGES} test: description: run the tests of the packages image: golang:${GO_VERSION}-alpine deps: [build] src: # go.mod, and go.sum, go.work and go.work.sum where the module has them - 'go.*' - '**/*.go' mounts: - $PWD/go/pkg/mod:/go/pkg/mod cmds: - go test ${PACKAGES} ``` # helm (/docs/catalog/helm) Lint and package a Helm chart. The tasks run in an `alpine/helm` container, need no cluster, and are skipped when the chart is unchanged. ```yaml title=".hammerkit.yaml" includes: helm: git: https://github.com/no0dles/hammerkit.git ref: v1.22.0 path: catalog/helm tasks: chart: deps: [helm:lint, helm:package] ``` ## Tasks [#tasks] | Task | What it does | Reads | Produces | | --------- | ------------------------------ | ----------- | -------- | | `lint` | Lint the chart with helm lint. | `CHART_DIR` | nothing | | `package` | Package the chart into dist. | `CHART_DIR` | `dist` | ## Inputs [#inputs] | Input | Default | Meaning | | -------------- | ------- | ------------------------------------------------------------------------------- | | `HELM_VERSION` | `4.3.0` | Tag of the `alpine/helm` image the tasks run in, e.g. `3.19.5` or `4.3.0`. | | `CHART_DIR` | `chart` | Folder of the chart (with its `Chart.yaml`), relative to the working directory. | ## Talking to a cluster [#talking-to-a-cluster] The entry does not install or inspect releases, because that depends on a cluster and its credentials and cannot be cached. Run `helm upgrade` from a task of your own, or in your deploy pipeline. ## The entry [#the-entry] ```yaml title="catalog/helm/build.yaml" # Lints and packages a Helm chart. Nothing here talks to a cluster. envs: # tag of the alpine/helm image helm runs in HELM_VERSION: 4.3.0 # folder of the chart, relative to the working directory CHART_DIR: chart tasks: lint: description: lint the chart with helm lint image: alpine/helm:${HELM_VERSION} src: ['${CHART_DIR}'] cmds: - helm lint ${CHART_DIR} package: description: package the chart into dist image: alpine/helm:${HELM_VERSION} src: ['${CHART_DIR}'] generates: [dist] cmds: - helm package ${CHART_DIR} --destination dist ``` # Catalog (/docs/catalog) The catalog is a set of build files for common tools such as `npm ci` or `yarn install`, kept in the hammerkit repository. Each entry is built the way this documentation recommends, runs in a pinned container image, declares what it reads and writes so it is cached, and is run by hammerkit's own test suite against a real project on every change. You use an entry by [including it from git](/docs/build-file/includes#include-from-a-git-repository) and calling its tasks: ```yaml title=".hammerkit.yaml" includes: npm: git: https://github.com/no0dles/hammerkit.git ref: v1.16.0 path: catalog/npm with: NODE_VERSION: 22 tasks: build: image: node:22-alpine deps: [npm:install] src: [src, tsconfig.json] generates: [dist] cmds: - npm run build ``` `hammerkit build` installs the dependencies (or reuses them when `package.json` and `package-lock.json` are unchanged) and then builds. The include runs in your directory, so every package of a monorepo can include the same entry. ## Entries [#entries] | Entry | What it does | | ---------------------------------- | ------------------------------------------------ | | [npm](/docs/catalog/npm) | Install locked dependencies with `npm ci`. | | [yarn](/docs/catalog/yarn) | Install locked dependencies with `yarn install`. | | [pnpm](/docs/catalog/pnpm) | Install locked dependencies with `pnpm install`. | | [pip](/docs/catalog/pip) | Install pinned requirements with `pip install`. | | [go](/docs/catalog/go) | Build and test a Go module. | | [cargo](/docs/catalog/cargo) | Build and test a Rust crate or workspace. | | [dotnet](/docs/catalog/dotnet) | Build and test a .NET project or solution. | | [docker](/docs/catalog/docker) | Build a Docker image and push it. | | [tsc](/docs/catalog/tsc) | Compile a TypeScript project with `tsc`. | | [eslint](/docs/catalog/eslint) | Lint a project with ESLint. | | [prettier](/docs/catalog/prettier) | Check the formatting of a project with Prettier. | | [jest](/docs/catalog/jest) | Run the tests of a project with Jest. | | [helm](/docs/catalog/helm) | Lint and package a Helm chart. | The Node tools (tsc, eslint, prettier, jest) run from `node_modules`, so they do not install anything themselves: your task `extend`s the entry's task and adds the install of your package manager as a dependency. Each of their pages shows it. ## Versions [#versions] `ref` is a release tag of hammerkit: the catalog at `v1.16.0` is the one that release's tests ran. A commit SHA pins exactly, and a branch such as `master` follows the latest entries but can differ between machines (see [caching and offline use](/docs/build-file/includes#caching-and-offline-use)). A task or input of an entry is not renamed or removed within a major version of hammerkit. ## Inputs [#inputs] An entry declares its inputs as top-level `envs` with defaults, and lists them on its page. `with` replaces them for one include; a name the entry does not declare is an error that lists the ones it does. See [inputs](/docs/build-file/includes#inputs). ## Adapting an entry [#adapting-an-entry] Tasks are building blocks, not templates you edit. To go beyond the inputs: * **Depend on it.** Your own task lists the entry's task in `deps`, as above. * **Extend it.** `extend: npm:install` adds to an entry task: more `src` entries, more `cmds`, more `envs`. Add `reset: [src]` to drop what it brings for a property. See [extending](/docs/task/extending). * **Another layout.** An entry reads files relative to the directory that includes it, so a project in `packages/web` includes it from `packages/web/.hammerkit.yaml`. For a lockfile that lives elsewhere, extend the task and point `src` at it. ## What an include brings [#what-an-include-brings] Before you adopt an entry, read its page: it shows the whole build file. After including it, `hammerkit ls` lists its tasks and [`hammerkit explain --definition npm:install`](/docs/cli/explain#showing-a-tasks-definition) shows what one runs, with the repository, ref and commit it came from and where each env value was set. An entry contributes tasks and never runs anything until you run one of them. # jest (/docs/catalog/jest) Run the tests of a project with Jest. The task runs in a `node` container, keeps the coverage report as its result, and is skipped when the sources and the Jest configuration are unchanged. ## Install first [#install-first] The task runs the tool from `node_modules`, and a container task only sees the files it declares and the outputs of its dependencies, so the install must be a dependency. Include the entry of your package manager ([npm](/docs/catalog/npm), [yarn](/docs/catalog/yarn) or [pnpm](/docs/catalog/pnpm)) and `extend` the task to add it, which works with any of them and keeps the entry free of a package manager: ```yaml title=".hammerkit.yaml" includes: npm: git: https://github.com/no0dles/hammerkit.git ref: v1.22.0 path: catalog/npm jest: git: https://github.com/no0dles/hammerkit.git ref: v1.22.0 path: catalog/jest tasks: test: extend: jest:test deps: [npm:install] ``` `extend` keeps the entry's image, sources and commands and adds the dependency (see [extending](/docs/task/extending)). ## Tasks [#tasks] | Task | What it does | Reads | Produces | | ------ | --------------------------------------------- | --------------------------------------------- | ---------- | | `test` | Run the tests with jest and collect coverage. | `package.json`, `jest.config.*`, `SOURCE_DIR` | `coverage` | ## Inputs [#inputs] | Input | Default | Meaning | | -------------- | ------- | ------------------------------------------------------------------------------------------------------- | | `NODE_VERSION` | `24` | Tag of the `node` image the task runs in, e.g. `22` or `24`. The image is `node:-alpine`. | | `SOURCE_DIR` | `src` | Folder with the sources and their tests, relative to the working directory. | ## Another layout [#another-layout] Tests that live outside `SOURCE_DIR` are added to the task's `src` when it extends, for example `src: [tests]`. ## The entry [#the-entry] ```yaml title="catalog/jest/build.yaml" # Runs the tests of a project with Jest. The project's own dependencies must be installed # first, see the page of this entry. envs: # tag of the node image jest runs in NODE_VERSION: 24 # folder with the sources and their tests, relative to the working directory SOURCE_DIR: src tasks: test: description: run the tests with jest and collect coverage image: node:${NODE_VERSION}-alpine src: [package.json, 'jest.config.*', '${SOURCE_DIR}'] generates: [coverage] cmds: - node_modules/.bin/jest --ci --coverage ``` # npm (/docs/catalog/npm) Install locked Node.js dependencies with npm ci. The task runs in a `node` container and keeps `node_modules` as its output, so it is skipped when `package.json` and `package-lock.json` are unchanged. ```yaml title=".hammerkit.yaml" includes: npm: git: https://github.com/no0dles/hammerkit.git ref: v1.16.0 path: catalog/npm tasks: build: image: node:24-alpine deps: [npm:install] cmds: - npm run build ``` ## Tasks [#tasks] | Task | What it does | Reads | Produces | | --------- | ------------------------------------------------ | ----------------------------------- | -------------- | | `install` | Install locked Node.js dependencies with npm ci. | `package.json`, `package-lock.json` | `node_modules` | ## Inputs [#inputs] | Input | Default | Meaning | | -------------- | ------- | ---------------------------------------------------------------------------------------------------------- | | `NODE_VERSION` | `24` | Tag of the `node` image the install runs in, e.g. `22` or `24`. The image is `node:-alpine`. | ## The entry [#the-entry] ```yaml title="catalog/npm/build.yaml" # Installs the locked dependencies of a Node.js project with npm. envs: # tag of the node image the install runs in NODE_VERSION: 24 tasks: install: description: install the locked dependencies with npm ci image: node:${NODE_VERSION}-alpine src: [package.json, package-lock.json] generates: [node_modules] cmds: - npm ci ``` # pip (/docs/catalog/pip) Install the pinned requirements of a Python project with pip. The task runs in a `python` container and installs into `__pypackages__`, so it is skipped when the requirements file is unchanged. Tasks that need the packages set `PYTHONPATH=__pypackages__`. ```yaml title=".hammerkit.yaml" includes: pip: git: https://github.com/no0dles/hammerkit.git ref: v1.19.0 path: catalog/pip tasks: test: image: python:3.13-slim deps: [pip:install] src: [app, tests] envs: PYTHONPATH: __pypackages__ cmds: - python -m unittest ``` ## Tasks [#tasks] | Task | What it does | Reads | Produces | | --------- | ----------------------------------------- | ------------------------------------------ | ---------------- | | `install` | Install the pinned requirements with pip. | the requirements file (`requirements.txt`) | `__pypackages__` | ## Inputs [#inputs] | Input | Default | Meaning | | ---------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `PYTHON_VERSION` | `3.13` | Tag of the `python` image the install runs in. The image is `python:-slim`. Quote it in `with` (`'3.10'`), or YAML reads it as the number 3.1. | | `REQUIREMENTS` | `requirements.txt` | The file to install from, with exact versions (`pip freeze`, `pip-compile`). Use it for a different name or folder, for example `requirements/prod.txt`. | ## The entry [#the-entry] ```yaml title="catalog/pip/build.yaml" # Installs the pinned requirements of a Python project with pip. envs: # tag of the python image the install runs in, quoted so 3.10 stays 3.10 PYTHON_VERSION: '3.13' # the file to install from, with exact versions (pip freeze, pip-compile) REQUIREMENTS: requirements.txt tasks: install: description: install the pinned requirements with pip image: python:${PYTHON_VERSION}-slim src: - '${REQUIREMENTS}' generates: [__pypackages__] cmds: - pip install --no-cache-dir --disable-pip-version-check --target __pypackages__ -r ${REQUIREMENTS} ``` # pnpm (/docs/catalog/pnpm) Install locked Node.js dependencies with pnpm. The task runs in a `node` container and keeps `node_modules` as its output, so it is skipped when `package.json` and `pnpm-lock.yaml` are unchanged. ```yaml title=".hammerkit.yaml" includes: pnpm: git: https://github.com/no0dles/hammerkit.git ref: v1.16.0 path: catalog/pnpm tasks: build: image: node:24-alpine deps: [pnpm:install] cmds: - npm run build ``` ## Tasks [#tasks] | Task | What it does | Reads | Produces | | --------- | ---------------------------------------------- | -------------------------------- | -------------- | | `install` | Install locked Node.js dependencies with pnpm. | `package.json`, `pnpm-lock.yaml` | `node_modules` | ## Inputs [#inputs] | Input | Default | Meaning | | -------------- | ------- | ---------------------------------------------------------------------------------------------------------- | | `NODE_VERSION` | `24` | Tag of the `node` image the install runs in, e.g. `22` or `24`. The image is `node:-alpine`. | | `PNPM_VERSION` | `10` | pnpm release to run, fetched with corepack the first time, e.g. `9` or `10.28.0`. | ## The entry [#the-entry] ```yaml title="catalog/pnpm/build.yaml" # Installs the locked dependencies of a Node.js project with pnpm. envs: # tag of the node image the install runs in NODE_VERSION: 24 # pnpm release to run, fetched with corepack PNPM_VERSION: 10 tasks: install: description: install the locked dependencies with pnpm image: node:${NODE_VERSION}-alpine src: [package.json, pnpm-lock.yaml] generates: [node_modules] envs: COREPACK_HOME: /tmp/corepack cmds: - corepack pnpm@${PNPM_VERSION} install --frozen-lockfile ``` # prettier (/docs/catalog/prettier) Check the formatting of a project with Prettier. The task runs in a `node` container and is skipped when the sources and the Prettier configuration are unchanged. ## Install first [#install-first] The task runs the tool from `node_modules`, and a container task only sees the files it declares and the outputs of its dependencies, so the install must be a dependency. Include the entry of your package manager ([npm](/docs/catalog/npm), [yarn](/docs/catalog/yarn) or [pnpm](/docs/catalog/pnpm)) and `extend` the task to add it, which works with any of them and keeps the entry free of a package manager: ```yaml title=".hammerkit.yaml" includes: npm: git: https://github.com/no0dles/hammerkit.git ref: v1.22.0 path: catalog/npm prettier: git: https://github.com/no0dles/hammerkit.git ref: v1.22.0 path: catalog/prettier tasks: format: extend: prettier:check deps: [npm:install] ``` `extend` keeps the entry's image, sources and commands and adds the dependency (see [extending](/docs/task/extending)). ## Tasks [#tasks] | Task | What it does | Reads | Produces | | ------- | -------------------------------------------------- | -------------------------------------------- | -------- | | `check` | Check the formatting of the sources with prettier. | `package.json`, `.prettierrc*`, `SOURCE_DIR` | nothing | ## Inputs [#inputs] | Input | Default | Meaning | | -------------- | ------- | ------------------------------------------------------------------------------------------------------- | | `NODE_VERSION` | `24` | Tag of the `node` image the task runs in, e.g. `22` or `24`. The image is `node:-alpine`. | | `SOURCE_DIR` | `src` | Folder with the sources to check, relative to the working directory. | ## Writing the formatting [#writing-the-formatting] The entry only checks. A task that rewrites its own sources cannot be cached, so there is no `--write` task; run `npx prettier --write` on your machine when you want one. The configuration is read from a `.prettierrc*` file; a project that keeps it in `package.json` needs no extra step, because that file is read too. ## The entry [#the-entry] ```yaml title="catalog/prettier/build.yaml" # Checks the formatting of a project with Prettier. The project's own dependencies must be # installed first, see the page of this entry. envs: # tag of the node image prettier runs in NODE_VERSION: 24 # folder with the sources to check, relative to the working directory SOURCE_DIR: src tasks: check: description: check the formatting of the sources with prettier image: node:${NODE_VERSION}-alpine src: [package.json, '.prettierrc*', '${SOURCE_DIR}'] cmds: - node_modules/.bin/prettier --check ${SOURCE_DIR} ``` # tsc (/docs/catalog/tsc) Compile a TypeScript project with tsc. The task runs in a `node` container, keeps the compiler output as its result, and is skipped when the sources and `tsconfig.json` are unchanged. ## Install first [#install-first] The task runs the tool from `node_modules`, and a container task only sees the files it declares and the outputs of its dependencies, so the install must be a dependency. Include the entry of your package manager ([npm](/docs/catalog/npm), [yarn](/docs/catalog/yarn) or [pnpm](/docs/catalog/pnpm)) and `extend` the task to add it, which works with any of them and keeps the entry free of a package manager: ```yaml title=".hammerkit.yaml" includes: npm: git: https://github.com/no0dles/hammerkit.git ref: v1.22.0 path: catalog/npm tsc: git: https://github.com/no0dles/hammerkit.git ref: v1.22.0 path: catalog/tsc tasks: build: extend: tsc:build deps: [npm:install] ``` `extend` keeps the entry's image, sources and commands and adds the dependency (see [extending](/docs/task/extending)). ## Tasks [#tasks] | Task | What it does | Reads | Produces | | ------- | ----------------------------- | --------------------------------------------- | ------------ | | `build` | Compile the project with tsc. | `package.json`, `tsconfig.json`, `SOURCE_DIR` | `OUTPUT_DIR` | ## Inputs [#inputs] | Input | Default | Meaning | | -------------- | ------- | ------------------------------------------------------------------------------------------------------- | | `NODE_VERSION` | `24` | Tag of the `node` image the task runs in, e.g. `22` or `24`. The image is `node:-alpine`. | | `SOURCE_DIR` | `src` | Folder with the TypeScript sources, relative to the working directory. | | `OUTPUT_DIR` | `dist` | Folder tsc writes to. It must be the `outDir` of your `tsconfig.json`. | ## Another layout [#another-layout] The entry reads `tsconfig.json` and the `SOURCE_DIR` folder. A monorepo that shares a base configuration adds it to the task's `src` when it extends, for example `src: [../../tsconfig.base.json]`. ## The entry [#the-entry] ```yaml title="catalog/tsc/build.yaml" # Compiles a TypeScript project with tsc. The project's own dependencies must be installed # first, see the page of this entry. envs: # tag of the node image tsc runs in NODE_VERSION: 24 # folder with the TypeScript sources, relative to the working directory SOURCE_DIR: src # folder tsc writes to, the outDir of the tsconfig.json OUTPUT_DIR: dist tasks: build: description: compile the project with tsc image: node:${NODE_VERSION}-alpine src: [package.json, tsconfig.json, '${SOURCE_DIR}'] generates: ['${OUTPUT_DIR}'] cmds: - node_modules/.bin/tsc ``` # yarn (/docs/catalog/yarn) Install locked Node.js dependencies with yarn. The task runs in a `node` container and keeps `node_modules` as its output, so it is skipped when `package.json` and `yarn.lock` are unchanged. ```yaml title=".hammerkit.yaml" includes: yarn: git: https://github.com/no0dles/hammerkit.git ref: v1.16.0 path: catalog/yarn tasks: build: image: node:24-alpine deps: [yarn:install] cmds: - yarn build ``` ## Tasks [#tasks] | Task | What it does | Reads | Produces | | --------- | ---------------------------------------------- | --------------------------- | -------------- | | `install` | Install locked Node.js dependencies with yarn. | `package.json`, `yarn.lock` | `node_modules` | ## Inputs [#inputs] | Input | Default | Meaning | | -------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `NODE_VERSION` | `24` | Tag of the `node` image the install runs in, e.g. `22` or `24`. The image is `node:-alpine`. | | `YARN_ARGS` | `--frozen-lockfile` | Arguments of `yarn install`. The image ships yarn 1; for yarn berry (2 and later) use a Node image that provides it and pass `--immutable`. | ## The entry [#the-entry] ```yaml title="catalog/yarn/build.yaml" # Installs the locked dependencies of a Node.js project with yarn. envs: # tag of the node image the install runs in NODE_VERSION: 24 # arguments of `yarn install`; yarn berry (2+) uses --immutable instead YARN_ARGS: --frozen-lockfile tasks: install: description: install the locked dependencies with yarn image: node:${NODE_VERSION}-alpine src: [package.json, yarn.lock] generates: [node_modules] cmds: - yarn install $YARN_ARGS ``` # Cache (/docs/cli/cache) `hammerkit cache pull` and `hammerkit cache push` move cache entries between each task's own cache (by default the machine-local `default` cache) and a **remote** cache declared in the [`caches:`](/docs/build-file/caches) block. They execute no task and start no container. Only the entries for the tasks' **current** state are moved — exactly what the next build of this commit looks up — including all transitive dependencies of the selected tasks. ```yaml title=".hammerkit.yaml" caches: shared: method: checksum backend: type: s3 bucket: my-build-cache ``` ```bash # fast workspace setup: fetch everything the build needs, then build offline hammerkit cache pull --remote shared hammerkit run build # after a trusted build: upload the results for everyone else hammerkit cache push build --remote shared ``` ## Options [#options] ``` Usage: hammerkit cache pull|push [options] [task] Options: --remote remote cache declared in the caches block -f, --filter filter task and services with labels -e, --exclude exclude task and services with labels --env environment --cache caching method to compare (choices: "checksum", "modify-date", "none") ``` ## Behavior [#behavior] * An entry already present at the destination is skipped, so both commands are safe to re-run. * An entry missing at the source is reported and skipped — the command still succeeds. * An unreachable or unauthorized remote **fails** the command (unlike a build, where a cache error only degrades to a cache miss). * `cache push` refuses to run when `HAMMERKIT_CACHE_READ_ONLY` is set. * Tasks with `cache: none`, or tasks whose own cache already *is* the remote, are skipped. ## cache ls and cache prune [#cache-ls-and-cache-prune] `hammerkit cache ls` lists the entries of a cache — by default the machine-local `default` cache, or any cache from the `caches:` block with `--remote ` — with the task, size and when it was last used. `--json` prints them for tooling. ``` • build 3f2a91c07d4e 12.4Mi last used 2h ago • e2e 91bc02f3aa10 1.2Ki last used 2h ago 2 entries, 12.4Mi total ``` `hammerkit cache prune` removes entries according to the cache's [retention policy](/docs/build-file/caches#retention), or the policy given on the command line, which takes precedence: ```bash hammerkit cache prune --keep 1 # newest version of each task only hammerkit cache prune --max-age 30d --dry-run # show what would be removed hammerkit cache prune --remote shared --max-size 50Gi ``` ``` Usage: hammerkit cache prune [options] Options: --remote cache declared in the caches block to prune --max-age remove entries not used within this long (e.g. 30d) --max-size remove least recently used entries above this size (e.g. 5Gi) --keep keep only the newest versions per task --dry-run show what would be removed without removing it ``` A local cache with a declared `retention` is also pruned automatically after every successful `run`. A remote cache is only ever pruned when you name it with `--remote`. Unlike [`clean --cache`](/docs/cli/clean), which drops everything stored for the tasks of this build file, `prune` keeps what's recent and removes what's old. # Clean (/docs/cli/clean) This command cleans up all generated directories and files including the local cache state of hammerkit. This is useful for detecting errors in the build file and for verifying that all tasks work without any previous state. ``` hammerkit clean ``` ## Clearing stored cache results [#clearing-stored-cache-results] By default `clean` removes the generated outputs and the per-task cache state of the current checkout. Add `--cache` to also drop the **stored results** kept by the cache backend - the local result store under `~/.hammerkit/remote-cache` or a configured remote backend (for example an S3 bucket or a registry — which must allow deleting manifests). ``` hammerkit clean --cache ``` This forces the next run to rebuild from scratch instead of restoring a result from the backend. See [caches](/docs/build-file/caches) for how backends work. It also removes the cached checkouts of [remote includes](/docs/build-file/includes#include-from-a-git-repository) in `~/.hammerkit/includes`, which are shared by all your projects, so the next run fetches them again. To update them without removing anything, use [`includes pull`](/docs/cli/includes). `--filter`/`--exclude` label options apply to `clean` as well, so you can clean a subset of tasks and services. ## Leftovers of killed runs [#leftovers-of-killed-runs] A run that was killed outright (`kill -9`, an out-of-memory kill) can't remove its containers. `clean` also removes every hammerkit container on the docker host that was started on this machine by a hammerkit process that no longer exists, whatever project it belongs to. Services started with [`up --daemon`](/docs/cli/up) are kept: [`down`](/docs/cli/down) stops those. # Down (/docs/cli/down) Stops the services of your build file, for example the ones started with [`hammerkit up`](/docs/cli/up) or left running by `up --daemon`. ```bash hammerkit down ``` Name services to stop only those, or tasks to stop the services they need: ```bash hammerkit down postgres hammerkit down dev:serve ``` Names are the ones [`hammerkit ls`](/docs/cli/ls) lists, as for [`up`](/docs/cli/up#start-by-name). A named service stops alone: the services it needs keep running, since other services may use them too. A named task stops the services it and its dependencies need. Only the services matching the `--filter`/`--exclude` labels are stopped when those options are given. `down` never runs a task, also not one it names. Pass `--env ` to target the services of a configured environment, for example a Kubernetes cluster declared under `environments:`. ## Options [#options] ``` Usage: hammerkit down [options] [names...] Arguments: names services to stop, or tasks whose services to stop, all when none is named Options: -f, --filter filter task and services with labels -e, --exclude exclude task and services with labels --env environment -h, --help display help for command ``` # Execute (/docs/cli/execute) Running `hammerkit` without a command (or with the explicit `run` command) executes a task. It is the default command, so `hammerkit build` and `hammerkit run build` are equivalent. ## Execute a task [#execute-a-task] Tasks can be executed by name. ```bash hammerkit example ``` ## Pass arguments [#pass-arguments] Arguments after `--` are appended to the task's last command, each quoted as one argument. The task then skips the cache, unless `--cache ` is given; its dependencies run without them, cached as usual. ```bash hammerkit run test -- src/login.spec.ts -t "login flow" ``` A run that selects several tasks takes no arguments. See [arguments](/docs/task/arguments). ## Execute tasks with labels [#execute-tasks-with-labels] Tasks can be executed by matching labels or filtered by label values. ### By matching labels [#by-matching-labels] `-f type=build` only executes tasks that have the given label value. Their dependencies don't need to match the label. ```bash hammerkit -f type=build ``` ### By excluding labels [#by-excluding-labels] `-e build=ios` excludes tasks that have the given label value. A task without a matching label is excluded as well if any of its dependencies matches. ```bash hammerkit -e build=ios ``` ## Options [#options] ``` Usage: hammerkit run [options] [task] [-- args...] Options: -f, --filter filter task and services with labels -e, --exclude exclude task and services with labels -c, --concurrency parallel worker count (default: 4, unbounded when every task and service requests cpus and memory) --skip-resource-check start even when a task with its services requests more than the host has -w, --watch watch tasks (default: false) --env environment -l, --log log mode (choices: "interactive", "live", "grouped") --cache caching method to compare (choices: "checksum", "modify-date", "none") --no-summary do not print the end-of-run summary --summary-json emit the end-of-run summary as JSON (default: false) --explain print the cache-miss cause when a task rebuilds (default: false) --dry-run print the execution plan with predicted cache hits/misses without running (default: false) --cache-read-only restore from cache backends but never push to them (or set HAMMERKIT_CACHE_READ_ONLY=1) --timeout fail tasks without their own timeout after this long (e.g. 10m) --max-timeout fail any task after this long, also one with a longer timeout of its own (e.g. 2m) --network the most any task or service reaches besides its needs: none, open or allow:, --no-skip-deps run dependencies even when nothing needing them has to run --no-restore only look cache hits up, download their outputs only when a task that runs needs them -h, --help display help for command ``` `--cache` defaults to `checksum` everywhere — the same comparison runs locally and in CI, so a result cached on one is reused on the other. `--log` defaults to `interactive` outside CI and `live` in CI (hammerkit auto-detects CI from the `CI`, `CONTINUOUS_INTEGRATION`, `BUILD_NUMBER` or `RUN_ID` environment variables). ## Build summary [#build-summary] After a run, hammerkit prints one line per task — executed or cached, and how long it took — followed by the totals and the cache hit ratio: ``` Summary: build cached 0ms e2e executed 4.5s install cached 0ms 1 executed, 2 cached (67% cache hit), 4.7s total ``` A dependency that wasn't needed because every task depending on it was a cache hit is listed as `skipped` — see [dependencies of cached tasks](/docs/task/dependencies#dependencies-of-cached-tasks). `--no-summary` hides it; `--summary-json` prints it as JSON on stdout (and nothing else), for CI dashboards or agents. With `--explain`, every rebuilt task also prints why it missed the cache, and the summary gets a cause column. ## Dry run [#dry-run] `--dry-run` prints the execution plan in order, with the predicted cache hit or miss (and its cause) for every task, without running anything: ```bash hammerkit run e2e --dry-run ``` ``` Dry run (no commands executed): 1. install: cache miss (never cached) 2. build: cache miss (source changed: src/app.js, dependency changed: install) 3. e2e: cache miss (source changed: src/app.js, dependency changed: build) ``` See also [`explain`](/docs/cli/explain) and [`graph`](/docs/cli/graph). ## Checking without restoring [#checking-without-restoring] On a cache hit, hammerkit normally downloads the task's entry and restores its `generates` into the checkout, so the outputs are there afterwards. A CI check often only needs to know the task passed. With `--no-restore`, a hit is only looked up in the cache backend, and nothing is downloaded: ```bash hammerkit run test --no-restore ``` Outputs are still restored where a run needs them: when a task that runs (a cache miss) or a service that starts depends on a hit, that hit's outputs are downloaded before it starts. A task that only groups others (no `cmds`) runs nothing, so it restores nothing either. `--no-restore` has no effect in watch mode. Leave it off when a later step uses the outputs, for example a deploy after `build`, or a coverage upload after `test`. A hit leaves the checkout as it is, so outputs already on disk may be from an older state. If an entry is removed from the cache before a task that needs it starts, the run fails. Run again without `--no-restore`. ## Read-only cache [#read-only-cache] `--cache-read-only` (or `HAMMERKIT_CACHE_READ_ONLY=1`) restores results from cache backends but never writes to them — for untrusted runners such as agent sandboxes or pull requests from forks. See [agents, workspaces and CI](/docs/guides/agents-and-ci#who-may-write-to-the-cache). ## Stopping a run [#stopping-a-run] Ctrl-C (`SIGINT`), a cancelled CI job or a supervisor stopping hammerkit (`SIGTERM`) and a closed terminal (`SIGHUP`) all stop a run the same way: running tasks are cancelled, their containers and the run's services are removed, and each local task's command gets `SIGTERM` for its whole process group. Whatever is still alive in that group after 10 seconds is killed; set `HAMMERKIT_STOP_TIMEOUT_MS` to change the grace period. A container task's container is removed when the task ends, so `docker ps` shows nothing from hammerkit once a run is over. If hammerkit itself is killed (`SIGKILL`), [`clean`](/docs/cli/clean) removes what that run left behind. # Explain (/docs/cli/explain) `hammerkit explain [task]` predicts, for every task in scope, whether the next run would be a **cache hit** or a **cache miss**, and names the input that caused each miss. Nothing is executed and no container is started. It uses the same state-key engine as `run`, so the prediction matches what `run` will do. ```bash hammerkit explain e2e ``` ``` • build: cache miss cause: source changed: src/app.js cause: dependency changed: install • e2e: cache miss cause: dependency changed: build • install: cache hit ``` ## Causes [#causes] | Cause | Meaning | | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `never cached` | No result for this task has been recorded yet. | | `source changed: ` | A file in `src` was added, removed or changed. | | `dependency changed: ` | A task in `deps` (transitively) has a different state. | | `command`, `image`, `env`, `mounts`, … changed | The task definition itself changed (see [cache identity](/docs/task/caching#what-gets-cached-and-what-invalidates-it)). | A task with `cache: none` is reported as `uncacheable`. ## JSON output [#json-output] `--json` prints a machine-readable list, for CI annotations or agents: ```json [ { "taskId": "ea717e7627d2e12e24acd731af1d1b364c510f32", "taskName": "build", "status": "miss", "causes": [ { "kind": "source-changed", "identifier": "src/app.js" }, { "kind": "dependency-changed", "identifier": "install" } ] } ] ``` ## Gating a CI step [#gating-a-ci-step] `--check` turns the prediction into an exit code: `0` when no task in scope would be a cache miss (`uncacheable` tasks do not count), `1` otherwise. After a [`cache pull`](/docs/cli/cache) it tells a pipeline whether a run has anything to do, so a job can skip expensive setup: ```bash hammerkit cache pull --remote ci if hammerkit explain test --check; then echo "up to date"; else hammerkit run test; fi ``` ## Showing a task's definition [#showing-a-tasks-definition] `--definition` prints, instead of the cache prediction, the task as hammerkit resolved it: the image, commands, `src`, `generates`, `deps` and `envs` after includes, inputs and `extend` were applied, and where each part came from. It is how you review what an [included](/docs/build-file/includes) or [catalog](/docs/catalog) task does before you run it. Nothing is executed. ```bash hammerkit explain --definition npm:install ``` ``` • npm:install defined in: catalog/npm/build.yaml of https://github.com/no0dles/hammerkit.git at v1.16.0 (commit 7b183f08ebed) included as: npm description: install the locked dependencies with npm ci image: node:22-alpine src: package.json, package-lock.json generates: node_modules cmd: npm ci env NODE_VERSION: 22 (input of include npm) ``` * **defined in** names the file, and for a git include the repository, ref and the commit the cache holds, so you see exactly which version runs. * Each env value shows where it was declared: `task`, `build file`, `input of include ` (a value your `with` passed) or `extended task`. * A variable taken from your shell or a `.env` file is listed with `from $NAME` and whether it is set, never its value. `--json` prints the same information as a list for tools. Services are not part of it yet. ## Related [#related] * `hammerkit run --explain` prints the miss cause inline for every task that rebuilds, and adds a cause column to the build summary. * `hammerkit run --dry-run` prints the execution plan in order — see [execute](/docs/cli/execute#dry-run). ## Options [#options] ``` Usage: hammerkit explain [options] [task] Options: -f, --filter filter task and services with labels -e, --exclude exclude task and services with labels --env environment --cache caching method to compare (choices: "checksum", "modify-date", "none", default: "checksum") --json emit the explanation as JSON (default: false) --check exit with code 1 when any task would be a cache miss (default: false) --definition show the resolved definition of the tasks and where it came from instead (default: false) ``` # Graph (/docs/cli/graph) `hammerkit graph [task]` serializes the build graph without running anything. Tasks are boxes, services are rounded nodes; a solid edge is a `deps` edge, a dotted edge is a `needs` edge. Paste the mermaid output into a GitHub comment or markdown file to render it. ```bash hammerkit graph e2e ``` ``` graph TD t0["build"] t1["e2e"] t2["install"] s0(["db"]) t0 --> t2 t1 -.-> s0 t1 --> t0 ``` ```bash hammerkit graph --format dot | dot -Tsvg > graph.svg ``` A cycle is reported as a warning after the graph is printed. ## Options [#options] ``` Usage: hammerkit graph [options] [task] Options: -f, --filter filter task and services with labels -e, --exclude exclude task and services with labels --env environment --format output format (choices: "mermaid", "dot", default: "mermaid") ``` # Includes (/docs/cli/includes) `hammerkit includes pull` fetches every [git repository](/docs/build-file/includes#include-from-a-git-repository) named by `includes` and `references` again and replaces the cached copy with the current commit of its `ref`. It executes no task. ``` hammerkit includes pull ``` A remote include is fetched once and then read from `~/.hammerkit/includes` on every run, so a branch or tag does not change underneath you. Use `pull` when you want the update. The command follows the includes of remote files too, fetches a repository named several times once, and prints what happened to each: ``` https://github.com/no0dles/hammerkit.git at v1.16.0: up to date at 7b183f08ebed https://github.com/acme/build-files.git at main: 4e1c0a77d2b9 -> c9f03b1e5a68 ``` * A **branch** or **tag** moves to the commit it points to now. * A **commit SHA** never changes, so it is not fetched again and shows as `pinned`. * A repository that cannot be reached fails the command and keeps the cached copy. To forget every cached remote include instead, so the next run fetches it again, use [`hammerkit clean --cache`](/docs/cli/clean#clearing-stored-cache-results). # CLI (/docs/cli) ``` Usage: hammerkit [options] [command] Options: -V, --version output the version number --verbose log debugging information (default: false) --file set build file (default: .hammerkit.yaml) --set value for $NAME / ${NAME} references, ahead of the shell and .env (repeatable) -h, --help display help for command Commands: ls [options] list all tasks clean [options] clear cache and generated store [options] save task outputs into restore [options] restore task outputs from package [options] package services into a docker image validate [options] validate hammerkit configurations graph [options] [task] serialize the build graph (tasks, services, deps/needs edges) as mermaid or dot explain [options] [task] explain whether tasks would be a cache hit or miss, without running them includes manage remote includes (includes pull) cache inspect, prune and move cache entries (cache ls / prune / pull / push) up [options] start services(s) down [options] stop services(s) run [options] [task] execute task (default command) help [command] display help for command ``` When no build file is present, only the [`init`](/docs/cli/init) command is available, which creates a default `.hammerkit.yaml`. Point hammerkit at a non-default file with `--file `. `--set NAME=value` works with every command and in any position, as often as needed. It supplies the value of `$NAME` and `${NAME}` references in the build file, ahead of the shell and `.env` files, so a run can be narrowed without exporting variables: `hammerkit run e2e --set E2E_SPEC=e2e/login.cy.ts`. Like any env value it is part of the cache key of the tasks that declare it. ## Common options [#common-options] These options are shared by the task-running commands ([run](/docs/cli/execute), [up](/docs/cli/up)) and, where they make sense, by `clean`, `store`, `restore` and `validate`: | Option | Default | Description | | ---------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `-f, --filter ` | – | Keep only tasks/services matching `key=value` [labels](/docs/labels). | | `-e, --exclude ` | – | Drop tasks/services matching `key=value` labels. | | `-c, --concurrency ` | `4` | Number of tasks run in parallel. Unbounded by default when every task and service requests `cpus` and `memory`: the [requests](/docs/task#planning-the-run) then bound the run. | | `--skip-resource-check` | | Start even when a task with the services it needs [requests more](/docs/task#planning-the-run) than the host has. | | `-w, --watch` | `false` | Re-run tasks when their `src` changes. | | `--env ` | – | Run against a declared [environment](/docs/task/kubernetes) (e.g. a cluster). | | `-l, --log ` | `interactive` (local) / `live` (CI) | `interactive`, `live` or `grouped`. | | `--cache ` | `checksum` | `checksum`, `modify-date` or `none`. See [caching](/docs/task/caching). | | `--timeout ` | – | Default [timeout](/docs/task#timeouts) for tasks without their own. | | `--max-timeout ` | – | [Timeout](/docs/task#capping-every-task) for every task, also one with a longer timeout of its own: each task runs at most the shorter of the two. | | `--network ` | – | The most any task or service of the run [reaches](/docs/task/network#a-ceiling-for-the-run) besides its needs: `none`, `open` or `allow:,` (a host may carry a port for a TCP forward), whatever it declares. Fails with a local task in the run. | Label filters are always `key=value`; a key-only `-f dev` is rejected. ## Parallelism [#parallelism] Independent tasks and dependencies run concurrently up to `--concurrency` workers (default `4`). A task only starts once all of its `deps` have completed and all of its `needs` are healthy; tasks on separate branches of the graph run at the same time. If any task fails, or a service a task needs fails to start, hammerkit stops scheduling new work and the run fails. When more tasks are ready than there are free workers, the one on the **longest remaining chain** starts first: the one whose own duration plus that of the longest line of tasks waiting for it is the largest. See [which ready task runs first](/docs/task/dependencies#which-ready-task-runs-first). ## Exit codes [#exit-codes] | Code | Meaning | | ----- | ---------------------------------------------------------------------------------------------------------------- | | `0` | Everything requested succeeded (including a fully cached run). | | `1` | A task, service, or command failed (build error, validation error, store/restore/package/clean/up/down failure). | | `127` | The requested task was not found. | These make hammerkit safe to chain in CI with `&&` or as a pipeline step. ## CI detection [#ci-detection] Hammerkit auto-detects CI from the `CI`, `CONTINUOUS_INTEGRATION`, `BUILD_NUMBER` or `RUN_ID` environment variables. The only thing this changes is the **default log mode** (`live` in CI, `interactive` otherwise). The cache method default is `checksum` everywhere, so a result cached locally is reused in CI and vice versa. # Init (/docs/cli/init) `init` is available only when no build file exists yet. It writes a minimal `.hammerkit.yaml` with a single `example` task so you have something to run immediately. ### Command [#command] ```bash hammerkit init ``` ```bash npx hammerkit init ``` ### Console output [#console-output] ``` created .hammerkit.yaml ``` ### Result [#result] ```yaml title=".hammerkit.yaml" envs: {} tasks: example: image: alpine cmds: - echo "it's Hammer Time!" ``` Run it with `hammerkit example` (the task is called `example`, not `build`). Hammerkit keeps its local cache state in a `.hammerkit` directory. Add it to your `.gitignore` so it is not committed: ``` .hammerkit ``` # ls (/docs/cli/ls) `ls` prints every task and service hammerkit can see, including the ones pulled in through [references](/docs/build-file/references) and [includes](/docs/build-file/includes). It's the fastest way to understand a repo you just cloned — before guessing task names. ```bash hammerkit ls ``` ## Example output [#example-output] ``` Services: • postgres ports: 127.0.0.1:5432 -> 5432 image: postgres:16-alpine Tasks: • install image: node:24-alpine labels: stage=build src: package.json package-lock.json generates: node_modules • api needs: postgres deps: install image: node:24-alpine labels: stage=run app=example src: index.js config.json package.json package-lock.json generates: node_modules ``` Each entry shows the image (or nothing, for a local task), labels, sources, generated outputs, and the `deps`/`needs` wiring — the same fields you set in the build file. ## JSON [#json] `--json` prints the same tasks and services as JSON, for scripts and agents: ```bash hammerkit ls --json ``` ```json { "tasks": [ { "name": "test", "description": "run the unit tests", "type": "container-task", "image": "node:24-alpine", "deps": ["install"], "needs": [], "labels": {}, "command": "npx vitest run" } ], "services": [ { "name": "api", "description": "", "type": "container-service", "image": "node:24-alpine", "ports": [{ "host": 3000, "container": 3000 }], "labels": { "app": ["api"] }, "command": "node server.js" } ] } ``` `command` is the task's last command, or the service's `cmd`: where [arguments](/docs/task/arguments) after `--` are appended. It is `null` when there is none. ## Options [#options] ``` Options: -f, --filter filter task and services with labels -e, --exclude exclude task and services with labels --json emit the tasks and services as JSON (default: false) -h, --help display help for command ``` The label options take `key=value` pairs, so you can scope the listing to one part of a monorepo, for example `hammerkit ls -f project=api`. See [labels](/docs/labels). # Package (/docs/cli/package) The `package` command turns the **container services** of your build file into Docker images. For each service hammerkit builds a multi-stage image that bakes in the service itself plus the outputs of every task the service depends on, so the resulting image is self-contained and ready to run without hammerkit. ```bash hammerkit package ``` The image for a service `api` is named `/api:`, where the tag defaults to `latest` (see [Versioning](#versioning)). The `` argument is required - it is the prefix of the produced image name and the target of `--push`. Packaging only runs against the Docker runtime. Services without a container image, and dependency tasks that run locally (without an `image`), are not supported. ## What gets packaged [#what-gets-packaged] For every container service hammerkit generates a `Dockerfile` that: * starts `FROM` the service image, * copies the generated outputs of the service's dependency tasks (`deps`) into the image, * applies the service `envs`, `ports` (as `EXPOSE`) and `cmd`, * labels the image with `hammerkit.dev/name` and `hammerkit.dev/id`. A service with no command and no dependencies still produces a valid image from its base image and sources. ## Pushing to a registry [#pushing-to-a-registry] Add `--push` to upload the built images to the registry. Credentials can be passed with `-u/--username` and `-p/--password`. ```bash hammerkit package registry.example.com --push -u ci -p $REGISTRY_TOKEN ``` ### Registry examples [#registry-examples] The `` argument is just the image-name prefix, so any registry works: ```bash # Docker Hub - produces docker.io//api hammerkit package docker.io/ --push -u -p $DOCKER_TOKEN # GitHub Container Registry hammerkit package ghcr.io/ --push -u -p $GITHUB_TOKEN # GitLab Container Registry (inside CI the variables are provided for you) hammerkit package registry.gitlab.com// \ --push -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD ``` ## Versioning [#versioning] By default the produced image is tagged `latest`. Use `-t/--tag` to pin an explicit version - for example a release number or a CI commit SHA - so the same build can be promoted across environments: ```bash hammerkit package registry.example.com -t 1.2.3 --push ``` produces and pushes `registry.example.com/api:1.2.3`. ## Build platform [#build-platform] By default hammerkit builds for the **host architecture** (`linux/amd64` on x64, `linux/arm64` on arm64 / Apple Silicon). Override it with `--platform` to build for a different target, for example when building on arm for an amd64 cluster: ```bash hammerkit package registry.example.com --platform linux/amd64 ``` `--platform` is forwarded to the Docker build, so the Docker daemon must be able to build that platform (BuildKit / emulation enabled for cross-architecture builds). ## Example [#example] Given a service that depends on an `install` task: ```yaml title=".hammerkit.yaml" services: api: image: node:alpine deps: [install] cmd: node server.js ports: [3000] mounts: [server.js] tasks: install: image: node:alpine src: - package.json - package-lock.json generates: - node_modules cmds: - npm ci ``` ```bash hammerkit package localhost:5000 --push ``` produces and pushes `localhost:5000/api`, with the `node_modules` produced by `install` already baked in. ## Options [#options] ``` Options: -f, --filter filter task and services with labels -e, --exclude exclude task and services with labels --push push image to registry (default: false) --build-override-user create and run as a dedicated uid/gid 1000 user -t, --tag image tag (default: "latest") --platform target build platform (default: host architecture) -u, --username registry username -p, --password registry password -h, --help display help for command ``` # remote mcp (/docs/cli/remote-mcp) Experimental. The tools and options can still change between releases. `remote mcp` is a [Model Context Protocol](https://modelcontextprotocol.io) server on stdio for one [sandbox](/docs/remote-compute/sandboxes) on a server. An MCP client starts it as a command and gets tools to read, search and edit the sandbox's files, run commands and published tasks on it, check them before a push, show what changed and push it to a branch. Like `remote`, it needs no build file: the tasks are the published ones. The tools, the sign-in and the shell task are described in [Agents](/docs/remote-compute/agents). It is a [`remote`](/docs/cli/remote) command, so a task named `mcp` still runs with `hammerkit mcp`. ```bash hammerkit remote mcp --server corp --sandbox sbx_4vJ0mT2cX9qLw1aB --build sha256:9f2c... ``` | Option | What it does | | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--server ` | The registered server, `HAMMERKIT_SERVER`. Default: the one `remote use` set, else the only one. | | `--sandbox ` | A sandbox of yours on it, `HAMMERKIT_SANDBOX`. | | `--repo `, `--ref ` | Instead: a sandbox of this repository at this branch, `HAMMERKIT_REPO` and `HAMMERKIT_REF`. One of yours without changes is checked out at the branch again, otherwise a new one is created. | | `--build ` | The hash of the published version of the build file whose tasks run on the sandbox, as `remote publish` printed it, `HAMMERKIT_BUILD`. | | `--shell ` | Its task whose last command is a shell, `sh` by default, `HAMMERKIT_SHELL`. `--shell ''` leaves out `bash`. | Standard output carries the protocol only; what `remote mcp` has to say goes to standard error, warnings about the build file in the working directory too. It stops when the client closes its input. # Remote (/docs/cli/remote) Experimental. `remote run` runs tasks of a published build file from a git ref of the repository. Sending a run of the files on your machine (`--on`, running a task with a `runner:`) arrives in follow-up releases, see the [remote compute spec](https://github.com/no0dles/hammerkit/blob/master/specs/remote-compute/spec.md). `remote` manages the list of hammerkit servers your machine knows about. Each one is registered under a short name, which later features (`--on `, a task's `runner:`) refer to instead of a URL. How a server is set up, and how it decides who may run what, is described in [Remote compute](/docs/remote-compute). ```bash hammerkit remote add build https://build.example.com --issuer https://login.example.com hammerkit remote login build hammerkit remote accounts build hammerkit remote publish build hammerkit remote run test --server build hammerkit remote verify test --server build hammerkit remote approvals build hammerkit remote approve run_Qx3k9ZpL2mW7vN4a build hammerkit remote list hammerkit remote use build hammerkit remote remove build ``` Except for `publish`, `run` and `verify`, the commands only touch your host config and talk to the server, so they work with or without a build file in the working directory. ### Leaving out the server name [#leaving-out-the-server-name] Every command that takes a server (`[name]`, `[server]` or `--server `) works without one. It then uses the default set with `remote use`, and when there is none (or the default is `local` or `auto`), the only registered server. So with one server you can write: ```bash hammerkit remote add build https://build.example.com --issuer https://login.example.com hammerkit remote login hammerkit remote publish hammerkit remote run test ``` With several servers and no default, hammerkit asks you to name one and lists them: ``` 2 servers are registered (build and staging), name one as the last argument, or make one the default with: hammerkit remote use ``` A default that names a server which is no longer registered is an error, never a reason to pick another one. `remote add`, `remote remove` and `remote use` always need the name. Where a run of `hammerkit run` goes is a separate choice, see [Runner](/docs/task#runner): a single registered server does not make those runs remote. ## Commands [#commands] | Command | What it does | | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `remote add --issuer ` | Register a server. `--issuer` is the OAuth issuer the server accepts tokens from, written exactly as the `iss` of its tokens (some providers end it with a slash). `--client-id`, `--audience` and `--scope` pin what a login asks for, see [Signing in](#signing-in). `--token-command ` takes the token for this server from a command instead of a login, see [A token command per server](#a-token-command-per-server); `--no-token-command` removes it. | | `remote list` | Show the registered servers and which one is the default. | | `remote use ` | Set the default location: `local`, `auto` or a registered name. | | `remote remove ` | Forget a server. If it was the default, runs are local again. | | `remote login [name]` | Sign in to the server with your identity provider. | | `remote logout [name]` | Forget the sign-in. | | `remote accounts [name]` | Show who you are on the server and which service accounts you may use. Without a name, the default server or the only one, see [Leaving out the server name](#leaving-out-the-server-name). | | `remote publish [server]` | Publish the build file of the working directory (or the one `--file` names) and the files it includes, and print its hash, see [Publishing](#publishing-a-build-file). `--dry-run` only checks it against the server's policy. | | `remote run ` | Run tasks of a published build file on a server from a git ref, print their output and exit as they did, see [Running tasks](#running-tasks). | | `remote up ` | Start services of a published build file on a server from a git ref and keep them up, with `--endpoint service:port` at addresses of their own, see [Service runs](#service-runs). | | `remote status [server]` | Show how the server stands for you: the places that run and for how long, the queue with how long each run waited and why, your limits and the runs that count against them, the service runs, and the capacity. `--json` for scripts, see [Status](/docs/remote-compute/status). | | `remote tasks [server]` | Show what the past runs of each task took: how often it ran, failed and came from the cache, the median and 90th percentile of its time, and how long its runs waited. `--mine` for your runs only, see [Past tasks](/docs/remote-compute/status#past-tasks). | | `remote capacity [server]` | Show what the machines of the server have and what its runs take: CPU and memory requested and in use, per namespace on Kubernetes, see [Capacity](/docs/remote-compute/status#capacity). | | `remote dashboard [server]` | Print the address of the [dashboard](/docs/remote-compute/status#dashboard) of a server that has one. | | `remote services [server]` | List the service runs that are starting, up or asleep, yours and those of others you may see, with their addresses where you may open them, and when they end or sleep. `--ended` adds the ended ones. | | `remote runs [server]` | List your runs on a server, newest first: their state, what they ran, how long they waited and ran, where a queued one stands, and why one failed. `--limit ` shows more than 20; `--state`, `--services`, `--tasks` and `--all` filter, see [Status](/docs/remote-compute/status#service-runs-and-past-runs). | | `remote logs [server]` | Print the log of a run, also of one that ended: what its tasks and services printed, and what the server and the cluster said. `--task ` keeps one task or service, `--tail ` the last lines, `--follow` keeps printing until the run ends, see [Reading a run again](#reading-a-run-again). | | `remote extend [server]` | Keep a service run up for this long from now. | | `remote stop [server]` | Stop a run; a service run takes its services and endpoints down. | | `remote retry [server]` | Start a run that is over again as a new run of yours, at the commit it ran, see [Retrying a run](/docs/remote-compute/runs#retrying-a-run). | | `remote verify ` | Check that the server already passed tasks for the files checked out here, without running them, see [Verifying checks](#verifying-checks). | | `remote versions [server]` | List the published versions, newest first, with who published them, where they come from and whether they were yanked. `--filter key=value` (repeatable) keeps the ones with that [annotation](/docs/remote-compute/build-files#annotations). | | `remote approvals [server]` | List the runs that wait for your approval. | | `remote show [server]` | Show a run that waits for your approval, or a proposed or published version, in full: per task the commands, image, secrets and accounts, and every file of a version, see [Approving runs](#approving-runs) and [Proposing](#proposing-a-version). | | `remote proposals [server]` | List the proposed versions of build files you made or may approve, see [Proposing](#proposing-a-version). | | `remote approve [server]` | Let a run go ahead, or publish a proposed version. `--reason ` is optional. | | `remote yank [server]` | Pull a published version back: no run starts from it. `--reason ` is shown to whoever tries to run it. Needs `builds: publish` or `use` of an account the version works with. | | `remote unyank [server]` | Let a yanked version go on. Needs `builds: publish`. | | `remote reject [server]` | End a run, or drop a proposed version. `--reason ` is shown to who asked for it. | | `remote mcp` | Serve a sandbox on a server as MCP tools over stdio for a coding agent, see [remote mcp](/docs/cli/remote-mcp). | Re-running `remote add` with the same values changes nothing. What you leave out keeps what was registered: the pinned client, audience and scope, and the token command. To point an existing name at a different server pass `--force`, so a typo can't silently redirect your runs. A name pointed at another server starts without the old token command. To drop a pin, remove the server and add it again. `local` and `auto` are reserved and can't be used as server names. ## Host config [#host-config] Registrations live in `~/.hammerkit/config.yaml`. Set `HAMMERKIT_CONFIG` to use a different file. ```yaml servers: build: url: https://build.example.com issuer: https://login.example.com ci: url: https://ci.example.com issuer: https://accounts.google.com tokenCommands: ci: url: https://ci.example.com issuer: https://accounts.google.com command: gcloud auth print-identity-token --audiences=hammerkit --include-email --impersonate-service-account=hammerkit-ci@my-project.iam.gserviceaccount.com run: on: build ``` `tokenCommands` are written by [`remote add --token-command`](#a-token-command-per-server). Each is bound to the URL and issuer it was registered for, like a sign-in, and is not used once the name points elsewhere. They sit beside `servers` so that hammerkit before 2.0, which refuses unknown keys of a server, still reads the file. The config is deliberately not part of the project's build file. A checked-in file naming a server URL could redirect your source code and OAuth tokens, so the decision of where to send work stays on the machine. The same file can hold the `secretProviders` and `secretAccounts` this machine reads secrets with, see [On this machine only](/docs/task/secret-providers#on-this-machine-only). ## URLs [#urls] Server URLs must use `https`. Plain `http` is accepted only for `localhost`, `127.0.0.1` and `[::1]`, for local testing. URLs with credentials, a query string or a fragment are rejected, and a trailing slash is dropped. ## Publishing a build file [#publishing-a-build-file] A server runs published build files, it does not take task definitions from the one who runs them. `remote publish` sends yours: ``` $ hammerkit remote publish build published to build: sha256:9f2c61d0... (3 files) tasks and services: build, deploy, more:lint from: path=.hammerkit.yaml repo=github.com/corp/payments commit=4be1a07c... machine=ci-runner-7 user=runner ``` It reads the build file with the parser of a run and sends it together with every local file it includes or references, as text. That is the bundle, up to 200 files and 1 MiB. Includes have to be below the directory of the build file, because a server could not reproduce anything else. That goes for where a file really is too: an include that is a link (or sits in a linked directory) pointing outside of that directory is refused, not followed, so publishing a checkout never sends a file of your machine that the checkout only links to. Git includes are not sent, the server fetches them and wants them pinned to a commit. The version printed is the hash of the bundle, computed here the same way the server computes it, so the same files always have the same version. That hash is the only name the version has: a run names it, there is no `latest`. Publishing what a server already has says `already published` and changes nothing. With the bundle go [annotations](/docs/remote-compute/build-files#annotations) that say where it comes from, the `from:` line: the path of the build file in its git repository, the repository and commit, this machine and your user name. When a file of the bundle is not as the commit has it (changed, or not committed at all), `dirty=true` says so, so the version is not taken for that commit's. They are not part of the hash. `--annotation key=value` adds one or replaces one, `--annotation user=` leaves one out, and it can be given more than once. `remote versions` lists what a server has and finds it by them, and `remote show ` prints one whole. Anybody else sees them as `from (unverified)`, since the server takes them as you sent them: ``` $ hammerkit remote versions build --filter repo=github.com/corp/payments build: 1 published sha256:9f2c61d0... published by ci at 2026-10-12T20:31:00.000Z from (unverified): path=.hammerkit.yaml repo=github.com/corp/payments commit=4be1a07c... machine=ci-runner-7 user=runner tasks and services: build, deploy, more:lint accounts: op-payments-prod ``` You need `builds: publish` on the server, see [entitlements](/docs/remote-compute/entitlements). A build file that reads secrets as an account you do not `use` is proposed instead, see below. Without a server name the default server (or the only one) is used, and `--file ` publishes another build file. `--dry-run` publishes nothing: the server checks the bundle against its policy and says what publishing it would do. Any `builds` level may check, `run` too: ``` $ hammerkit remote publish build --dry-run checked with the policy of build: sha256:9f2c61d0... (3 files), nothing was published tasks and services: build, deploy, more:lint publishing it would propose it, it names accounts you do not use: op-payments-prod ``` Tasks and services marked `publish: false` are left out, see [Only part of a build file](/docs/remote-compute/build-files#only-part-of-a-build-file). ### Proposing a version [#proposing-a-version] With only `builds: propose`, or when the build file names an account you do not use, the same command proposes the version. It does not become a version and can not run until somebody who may publish, and uses those accounts, approves it: ``` $ hammerkit remote publish build proposed to build: sha256:4be1a07c... (3 files) tasks and services: build, deploy, more:lint it can not run before somebody who may publish approves it, until 2026-10-11T11:41:00.000Z ``` The one who may publish reads the whole version before deciding: ``` $ hammerkit remote proposals build build: 1 proposed sha256:4be1a07c... proposed by coding-agent, waits for your approval from (unverified): path=.hammerkit.yaml repo=github.com/corp/payments machine=agent-sandbox-3 user=agent tasks and services: build, deploy, more:lint accounts: op-payments-prod, the proposer does not use op-payments-prod decide before 2026-10-11T11:41:00.000Z $ hammerkit remote show sha256:4be1a07c... build sha256:4be1a07c... on build, proposed by coding-agent (https://login.example.com), waiting deploy task, docker node:22 $ ./deploy.sh after: build secret gsm:kube-token as op-payments-prod -> KUBE_TOKEN ... --- .hammerkit.yaml (entry) tasks: build: cmds: [npm run build, ./upload.sh] ... $ hammerkit remote approve sha256:4be1a07c... build approved sha256:4be1a07c... on build, it is published ``` `remote show` prints what each task does and then every file of the bundle, whole and with control characters replaced by `?`, so nothing hides in it. It does not compare with an earlier version: a version has no name and follows none, so a diff would only show what the proposer chose to compare with. `remote reject` drops the proposal, with a `--reason` the one who proposed it reads in `remote proposals`. Nobody approves their own proposal, and what you may not decide does not exist for you. After a rejection you can not propose again for ten minutes, and nobody that version (`approvals.cooldown`). ## Running tasks [#running-tasks] `remote run` starts a run of a published version on a server, from a branch or tag of your repository that the server checks out itself, and follows it to its end: ``` $ hammerkit remote run test --server build run_Qx3k9ZpL2mW7vN4a on build: test from main at 4be1a07c... [hammerkit] checked out github.com/corp/payments at 4be1a07c... ✓ 42 tests passed run_Qx3k9ZpL2mW7vN4a succeeded ``` It prints every line of the run as the server logs it, with the task in front when the line is not of the one task you named (`[hammerkit]` is the server's own, so a task of that name is best called something else). It exits with `0` when the run succeeded and with the exit code of the task that failed otherwise, `1` when the run was rejected, cancelled or failed without one. Control characters of the output show as `?`, so the output of a run can not rewrite your terminal; tabs stay. Several tasks named (`remote run ci:check test:e2e`) run as one plan, side by side as far as their dependencies allow and in no particular order, the [longest remaining chain](/docs/task/dependencies#which-ready-task-runs-first) first. To run one after the other, make the second depend on the first. What it runs, when you leave the options out, is what is checked out where you are: | Option | What it is | Left out | | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--build ` | The published version to run, also `HAMMERKIT_BUILD`. | The hash of the build file here (or the one `--file` names), computed as `remote publish` does. When the server does not have it, see [A build file the server does not have](#a-build-file-the-server-does-not-have). | | `--repo ` | The repository, such as `github.com/corp/payments`. The server reads it from its own [sources](/docs/remote-compute/sources). | The `origin` of the checkout. | | `--ref ` | The branch or tag. | The branch checked out here. In a detached checkout, the one branch or tag of `origin` at its commit, see below. | | `--commit ` | The commit the ref has to be at. A ref that moved since is refused, so the run executes the code you meant: `main is at 9e1f... now, not at 4be1a07c...`. | The commit checked out here when `--ref` is left out too, else no pin. | | `--server ` | The server. | The default of `remote use`, else the only registered server. | | `--env`, `--cache`, `--timeout`, `--max-timeout` | As for [`run`](/docs/remote-compute/runs#submitting-a-run). | The server's and the build file's defaults. | | `--detach` | Submit the run and exit, without following it: it prints the run and `follow it with: hammerkit remote logs --follow`. For a [check](#verifying-checks) that CI verifies later. | Follow the run to its end. | | `--publish` | Publish the build file here first when the server does not have it, without asking. Not with `--build` or `HAMMERKIT_BUILD`. | Ask at a terminal, fail without one. | A detached checkout, such as a worktree made with `git worktree add --detach`, is on no branch. Without `--ref` it takes the one branch of `origin` (as your last fetch or push left it) or tag whose head is its commit, pinned to that commit as `--commit` does, so a branch that moved since is refused. With none, or several, it fails and names them: ``` the checkout here is on no branch, and its commit 4be1a07c... is the head of branch agent/fix and tag v1.2.0 of origin: name one with --ref ``` A name that looks like a commit is left out, since the server would run it as that commit, and a branch and a tag of the same name can only be run by the commit (`--ref 4be1a07c...`). What follows `--` goes to the one task named, as the [arguments](/docs/remote-compute/runs#arguments) of a run. The server refuses them for a task that reads a secret or depends on one that does, for a task that says `args: false`, and, on a server with `runs.args: declared`, for every task that doesn't say `args: true`. A run on an account you may only `request` waits for an approval, and says so: ``` waiting until somebody who uses deploy-prod approves it, before 2026-10-24T10:41:00.000Z: hammerkit remote approve run_Qx3k9ZpL2mW7vN4a ``` The command waits as long as the server lets the run wait (`approvals.timeout`, one hour by default). Ctrl-C, or the cancel of a CI job, cancels the run on the server: the command waits up to ten seconds for the server to take the cancel before it ends. The token is the one of [Signing in](#signing-in), or `HAMMERKIT_TOKEN_COMMAND` / `HAMMERKIT_TOKEN`, which is how a CI job signs in with its own identity, see [CI through a server](/docs/guides/agents/github-actions). ### A build file the server does not have [#a-build-file-the-server-does-not-have] When you leave `--build` out and the server does not have the build file here, a run can not start, since a server only runs [published](#publishing-a-build-file) versions. At a terminal the command offers to publish it and goes on once you answer `y`: ``` $ hammerkit remote up web --endpoint web:8080 the build file here is not published on build (sha256:9f2c61d0...), 3 files tasks and services: build, web, more:lint publish it to build now and go on? [y/N] y published to build: sha256:9f2c61d0... (3 files) tasks and services: build, web, more:lint from: path=.hammerkit.yaml repo=github.com/corp/shop commit=4be1a07c... machine=laptop user=pascal run_Vd1rX9pQ2mT7wN4a on build: web from feature/login at 4be1a07c... ``` It publishes exactly as `remote publish` does, with the same [entitlements](/docs/remote-compute/entitlements): * Any other answer, or Ctrl-C, publishes nothing and the command fails. * With only `builds: run` it does not ask: it says that somebody who may publish has to run `remote publish`. * When publishing would only [propose](#proposing-a-version) the version (`builds: propose`, or an account you do not use), it asks `propose it to build now?` and says why. A proposal can not run yet, so the command proposes it and fails with `is only proposed on build, nothing was started`; run it again once it is approved. Without a terminal, in CI or with stdin or stdout redirected, or with `CI` set, it asks nothing and fails with `the build file here is not published on build (sha256:...), publish it first with: hammerkit remote publish build, or pass --publish`. `--publish` publishes it without asking, for a job that may publish. A job that should only run what was published leaves it out, and the failure tells that the build file changed. With no build file here at all, the command was most likely started in another folder than the checkout meant. It fails and says so: `there is no build file at . Run this in the checkout whose build file the server has, or name a published version with --build , listed by: hammerkit remote versions`. ## Service runs [#service-runs] `remote up` starts a [service run](/docs/remote-compute/service-runs): services of a published version from a branch or tag, kept up until their time ends, and follows it until they are up. Then it prints the address of each endpoint and exits; the services stay. ``` $ hammerkit remote up web --endpoint web:8080 --ttl 3d --sleep-ttl 30m run_Vd1rX9pQ2mT7wN4a on build: web from feature/login at 4be1a07c... [hammerkit] checked out github.com/corp/shop at 4be1a07c... run_Vd1rX9pQ2mT7wN4a is up until 2026-10-28T10:12:00.000Z, asleep after 30m without a request web:8080 https://feature-login-8080-3fa2c1e09b7d.run.example.com/ ``` | Option | What it is | Left out | | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | | `--endpoint ` | A port of a service to publish, the container port it declares. More than once for more. | No endpoints. | | `--ttl ` | How long the run lasts once its services are ready, asleep or not; then they are removed. | The server's `runs.up.ttl`. | | `--sleep-ttl ` | How long the services stay up without a request, then they [sleep](/docs/remote-compute/service-runs#sleeping) until the next request wakes them. | The server's `runs.up.sleepTtl`, or never. | | `--name ` | Lower case letters, digits and `-`. A later `remote up` of yours with this name moves this run to its commit, or replaces it when it starts other services or another build file; the addresses stay. | The branch. | | `--build`, `--repo`, `--ref`, `--commit`, `--server`, `--env`, `--publish` | As for [`remote run`](#running-tasks), see also [A build file the server does not have](#a-build-file-the-server-does-not-have). | As there. | When a run of yours with the name is up or asleep and starts the same services of the same build file version, `remote up` [moves it](/docs/remote-compute/service-runs#moving-to-another-commit) to the commit instead: it keeps its id, addresses and the data of its services, and only services that changed restart (on Kubernetes). It follows the move until the run is up at the new commit, or for a run that is asleep, says which commit its next wake starts: ``` $ hammerkit remote up web --endpoint web:8080 run_Vd1rX9pQ2mT7wN4a on build (up) moves to feature/login at 9e01b77d... [hammerkit] moving from 4be1a07c... to 9e01b77d... run_Vd1rX9pQ2mT7wN4a is up until 2026-10-28T10:12:00.000Z, asleep after 30m without a request web:8080 https://feature-login-8080-3fa2c1e09b7d.run.example.com/ ``` A move that does not get there leaves the run where it was and exits with `1`: `did not move to 9e01b77d...: ; it stays at 4be1a07c...`. Ctrl-C while it moves stops following; the run moves on without. It prints what the services print while they start, on Kubernetes too, and the server's `[hammerkit]` lines, such as what a service waits for (`web waits for a node: 0/3 nodes are available: 3 Insufficient cpu`). When the run fails before it is up, it says which service failed and why, and exits with `1`: ``` [web] service web can not become ready, it keeps crashing: it exited with code 1 (Error), restarted 2 times [hammerkit] what the cluster said in hk-run-3f2a...: [hammerkit] pod web-6c9f...-x2k8q: Running, restarted: it exited with code 1 (Error), restarted 2 times [hammerkit] event BackOff pod/web-6c9f...-x2k8q: Back-off restarting failed container web (4 times) read its log again with: hammerkit remote logs run_Vd1rX9pQ2mT7wN4a run_Vd1rX9pQ2mT7wN4a failed: web: service web can not become ready, it keeps crashing: it exited with code 1 (Error), restarted 2 times ``` Ctrl-C while it starts cancels it. `remote extend 8h` keeps it for that long from now (never shorter than it was), `remote stop ` takes it down, asleep or not. ## Reading a run again [#reading-a-run-again] A run that failed says why when it ends: the error of the run, then each task that failed with its own error or exit code, then where its log stays: ``` migrate: exit code 2 read its log again with: hammerkit remote logs run_Qx3k9ZpL2mW7vN4a run_Qx3k9ZpL2mW7vN4a failed: migrate failed ``` The server keeps the log of your last runs (`runs.keep`, 200 by default) after they ended, also over a restart, so the pods of a run can be gone and its log is still there. `remote runs` lists them, newest first: ``` $ hammerkit remote runs run_Vd1rX9pQ2mT7wN4a 2026-11-07T07:12:03.000Z failed up web: web: service web can not become ready, its image ghcr.io/corp/web:4be1a07 could not be pulled (ImagePullBackOff) run_Qx3k9ZpL2mW7vN4a 2026-11-07T06:58:41.000Z succeeded test ``` `remote logs ` prints the log of one, each line with its task or service in front, then how it ended: | Option | What it does | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--task ` | Only the lines of this task or service (`hammerkit` for the server's own), without the name in front. | | `--tail ` | Only the last lines, at most 1000. | | `-f`, `--follow` | Keep printing what comes until the run ends, as `remote run` does. Ctrl-C stops following and leaves the run alone. For a service run that is up, that is the output of its services. | `remote retry ` starts a run that is over again, as a new run of yours: the same tasks, settings and arguments, at the commit it ran. It prints the new run, which you follow with `remote logs --follow`: ``` $ hammerkit remote retry run_Qx3k9ZpL2mW7vN4a run_Hk2v8RqW5nM3xT1b on build retries run_Qx3k9ZpL2mW7vN4a: queued, follow it with hammerkit remote logs run_Hk2v8RqW5nM3xT1b --follow ``` The server decides about it as about any run you start, see [Retrying a run](/docs/remote-compute/runs#retrying-a-run). A log longer than the server keeps of a run (`runs.maxLogBytes`) ends with the last lines of each task, which the server always keeps. On Kubernetes the log of a run that failed also holds what the cluster said before the server removed its pods: each pod that was not as it should be, and the warning events of the run's objects, such as `FailedScheduling`, a failed image pull or a container that was killed for its memory. A container that ended says its exit code, the reason Kubernetes gave and, above 128, the signal that killed it (`container web: it exited with code 137 (Error), killed by SIGKILL, restarted 0 times`); only one with a healthcheck adds that a failing healthcheck kills it so. A pod that copies a task's files into or out of its volume says it is a helper of that task, not the task. The output of a failed probe is the service's own and is left out. The server needs `list` on events for that, see [What the server may do](/docs/remote-compute/backend#what-the-server-may-do). ## Verifying checks [#verifying-checks] `remote verify` tells whether the server already passed tasks for the files checked out here, so a CI job does not have to run them: ``` $ hammerkit remote verify test e2e --server build test: passed 2026-10-25T09:14:03.120Z in 1m12s, run_Qx3k9ZpL2mW7vN4a by agent-x, from a check of sandbox sbx_4vJ0mT2cX9qLw1aB, fetched at 4be1a07c1a2b e2e: no run on build passed it for these files not verified: e2e; run on the server: hammerkit remote run e2e ``` It computes the [check key](/docs/remote-compute/checks#check-keys) of each task over the files here, runs nothing, and asks the server for a run that passed it: a run from a ref without arguments, or a check of a sandbox. It exits `0` when there is one for every task, and `1` at once otherwise, also for a task that has no key (no `src` that matches a file). It never waits for a run and never starts one. The guide is [Checks that ran before CI](/docs/guides/agents/verified-checks). | Option | What it is | Left out | | -------------------- | ------------------------------------------------ | ------------------------------------------------------------- | | `--repo ` | The repository, as the server knows it. | The `origin` of the checkout. | | `--env ` | The environment of the build file, as for `run`. | None. | | `--server ` | The server. | The default of `remote use`, else the only registered server. | The caller needs `checks: read` or a `builds` level, with `repos` that cover the repository, see [Entitlements](/docs/remote-compute/entitlements). ## Approving runs [#approving-runs] A run that reads secrets as a service account its caller may only [request](/docs/remote-compute/entitlements#approvals) waits for somebody who uses that account to approve it. If you use the account: ``` $ hammerkit remote approvals build build: 1 waiting for your approval run_Qx3k9ZpL2mW7vN4a requested by 4f1c2b (https://login.example.com) build file sha256:9f2c61d0... tasks: deploy accounts: op-payments-prod, source ref refs/heads/main, env=staging op-payments-prod: waits for somebody who uses it to approve decide before 2026-10-11T10:41:00.000Z $ hammerkit remote approve run_Qx3k9ZpL2mW7vN4a build approved run_Qx3k9ZpL2mW7vN4a on build, it is queued ``` `remote show` prints what each task of the run does, so the decision does not need the build file: ``` $ hammerkit remote show run_Qx3k9ZpL2mW7vN4a build ... deploy task, docker node:22 [remote-include] $ ./deploy.sh after: build secret gsm:kube-token as op-payments-prod -> KUBE_TOKEN ``` The line `look at:` of the list names the tasks with a flag: `local-task` runs on the host and not in a container, `host-port` binds a port of the host and `remote-include` includes a file from git. A secret's value is never shown, only where it is read from. A run that passes [arguments](/docs/remote-compute/runs#arguments) uses no account, so it waits for nobody; one stored before that rule lists them on a line of their own, each as a JSON string (`arguments: "--region" "eu, us"`). `remote show` prints each command as it runs. Arguments and commands are never cut; other text longer than 300 characters ends in `… (+N chars)`. Control characters show as `?`, and bidi and zero-width characters are written out (`\u{202e}`), so what you read is the order it runs in. A task that isn't `open` shows its `network`. What you see is what the decision rests on: who asked, which version of the build file, the tasks, the service accounts they read secrets as and where the code comes from. The version is the one the run is pinned to, so what you approve is what runs. You approve for every account you use; a run on several waits until they are all covered. You see the runs on the accounts you use, and no others. ## Signing in [#signing-in] `remote login` uses the OAuth device authorization grant, so it works on a machine without a browser: ``` $ hammerkit remote login build signing in to https://build.example.com through https://login.example.com client hammerkit-cli, audience hammerkit, scope openid api://hammerkit/.default To sign in, open https://login.example.com/activate and enter the code ABCD-EFGH signed in to build as 4f1c2b service accounts: op-payments-dev (use), op-payments-prod (request), op-web-dev (use) ``` Open the address on any device, sign in as you normally do, and enter the code. The server tells hammerkit which identity provider client, audience and scope to use, so there is nothing to configure beyond `remote add`. The `--issuer` you registered has to be one the server accepts, otherwise the login stops before the browser step. The server is only trusted with that once. The first successful login pins the client, audience and scope in the host config, and a later login refuses a server that asks for something else (register it again with `remote add --force` if the change is expected). Look at the line printed before the code: a client or audience you do not recognise, such as one of a cloud provider's own APIs, means stop. To not rely on the first use, give them to `remote add` (`--client-id`, `--audience`, `--scope`), the values come from whoever runs the server. Before a token is sent to the server, hammerkit checks that it was issued by the registered issuer for the pinned audience, so a token for another service never leaves your machine. `offline_access` is dropped from the scope: no refresh token is requested. What is stored is one short-lived access token in `~/.hammerkit/credentials.json` (`HAMMERKIT_CREDENTIALS` picks another file), readable by you only (a file others can read is not used; hammerkit says how to fix it). A refresh token is not kept: when the token expires, `remote login` again. The token is bound to the URL and issuer it was issued for, so registering the name again for another server (`remote add --force`) does not send the old token there. Agents and CI do not use `login`. They bring their own token, an agent from its own client credentials and a CI job from the OIDC token of the job, and the server maps it to service accounts through its [entitlements](/docs/remote-compute/entitlements). The `remote` commands take such a token instead of a sign-in, in this order: | Variable | What it is | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `HAMMERKIT_TOKEN_COMMAND` | A command that prints the token, run for each command, such as a script that asks the identity provider for a service account. | | `HAMMERKIT_TOKEN` | The token itself, as a CI job gets it. | ```yaml title=".gitlab-ci.yml" publish: id_tokens: HAMMERKIT_TOKEN: aud: hammerkit script: - hammerkit remote add build https://build.example.com --issuer https://gitlab.com - hammerkit remote publish build ``` The token goes only to the server the name is registered for. ### A token command per server [#a-token-command-per-server] The variables hold for every server. When servers want tokens of different service accounts or audiences, or you don't want to export a variable in every shell, register the command with the server: ```bash hammerkit remote add ci https://ci.example.com --issuer https://accounts.google.com \ --token-command 'gcloud auth print-identity-token --audiences=hammerkit --include-email --impersonate-service-account=hammerkit-ci@my-project.iam.gserviceaccount.com' hammerkit remote publish ci ``` Every `remote` command and [`remote mcp`](/docs/cli/remote-mcp) that talks to `ci` then runs it for a token, and runs it again when the server stops accepting the token. It runs with your environment and identity, through the shell, and has a minute to print the token alone on stdout. When it fails, hammerkit says how it ended and shows what it printed on stderr; what it printed on stdout is never shown, as it can hold more than the token. The order is: | Source | Used for | | ---------------------------- | ----------------------------------------------------------------- | | `HAMMERKIT_TOKEN_COMMAND` | Every server, so a CI job or an agent can bring its own identity. | | `HAMMERKIT_TOKEN` | Every server. | | `remote add --token-command` | The one server it is registered with. | | `remote login` | The one server it signed in to. | A server with a token command needs no `remote login`, and `remote login` says so. A server that offers no sign-in for its issuer (an issuer without a `clientId` in the server's config) can only be reached with a token from elsewhere, and `remote login` points you to `--token-command`. The command lives in your host config only, like the server's URL: a build file can't name one, since whoever writes it decides what runs on your machine. For the same reason keep the host config writable by you only: whoever may edit it, an agent too, can make hammerkit run any command, also in the [agent plugins](/docs/remote-compute/agents). `remote add` and `remote list` print the command, so don't put a secret in it; point it at a script that holds one instead. Make it print a token of a service account, see [Service accounts on Google Cloud](/docs/guides/agents/google-accounts), not of yourself. # Server (/docs/cli/server) Experimental. The config and the API can still change between releases. `server` turns a machine into the target of remote runs. It reads its own config file, so it needs no build file. What the config holds and how a server decides who may run what is described in [Remote compute](/docs/remote-compute). ```bash hammerkit server --config /etc/hammerkit/server.yaml ``` | Option | What it does | | --------------------- | ------------------------------------------------------------------------------------------------ | | `-c, --config ` | The [server config](/docs/remote-compute#config). `HAMMERKIT_SERVER_CONFIG` can stand in for it. | The server runs until it is interrupted. `SIGTERM` stops it gracefully, and `SIGHUP` reads the entitlements of the config again without a restart, see [Changing entitlements](/docs/remote-compute/entitlements#changing-entitlements). # Store / Restore (/docs/cli/store-restore) The store/restore commands are intended for CI systems. The store command takes a destination folder, into which all generated files and folders are moved for later recovery. Restoring the previous state speeds up your CI workflows, because task caching then works across pipeline runs. ### Example workflow [#example-workflow] To demonstrate the behavior, we use the following example: a simple Node.js TypeScript project that installs npm dependencies and compiles the TypeScript source code. ```yaml title=".hammerkit.yaml" tasks: install: src: - package.json - package-lock.json cmds: - npm ci generates: - node_modules build: deps: [install] src: - src - tsconfig.json generates: - dist cmds: - node_modules/.bin/tsc -b ``` The CI system runs the `build` task on every commit. The goal of store/restore is to prevent unnecessary work in CI. Together with hammerkit's caching, the build task is skipped if the source code hasn't changed. The npm install is skipped as well, and the node_modules folder is restored, if `package.json` and `package-lock.json` are unchanged. ### Saving state [#saving-state] ``` hammerkit store cache ``` After the store command has completed, the hammerkit cache, including all files from tasks that declare `generates`, is saved in the given destination folder. The folder can then be cached with your CI's caching mechanism. ### Restoring state [#restoring-state] Restore the state before running the required tasks in your CI workflow: let your CI's caching mechanism restore the previously saved folder, then restore it with hammerkit. ``` hammerkit restore ``` ### Example with GitLab CI [#example-with-gitlab-ci] This example uses GitLab CI caching to speed up the build time of the `build` command. For further details, take a look at the [demo repository](https://gitlab.com/pascalbe/hammerkit-typescript-restore-demo). ```yaml title=".gitlab-ci.yml" before_script: - npm i -g hammerkit build: image: node:24 cache: key: ${CI_COMMIT_REF_SLUG} paths: - cache policy: pull-push script: - hammerkit restore cache - hammerkit build - hammerkit store cache ``` # Up (/docs/cli/up) Services normally start and stop automatically based on the `needs` of the tasks you run. The `up` command starts them directly, which is useful to keep shared services such as a database running across several task runs during development. ```bash hammerkit up ``` This starts every service in the build file (or the [named ones](#start-by-name), or the ones matching the `--filter`/`--exclude` labels) and waits until you stop it. Use `--daemon` to start the services in the background and return immediately. ```bash hammerkit up --daemon ``` With `--daemon`, `up` returns once every service passed its healthcheck (`--wait ready`, the default). `--wait start` returns as soon as the containers are started instead; services that other services need are still awaited, since those can only start once their needs are ready. A service that is already running is reused, unless its definition changed since it was started (image, command, env values, ports, mounts, volumes, healthcheck, its init task, or a service it needs): then `up` and `run` recreate it. A reused service does not run its [init](/docs/service/container#init) again. Stop services started this way with [`hammerkit down`](/docs/cli/down). ## Start by name [#start-by-name] Name the services to start only those, with the services they need: ```bash hammerkit up postgres redis hammerkit up --daemon infra:postgres ``` A service of an [include](/docs/build-file/includes) or a [reference](/docs/build-file/references) goes by its full name, as [`hammerkit ls`](/docs/cli/ls) lists it. A task name starts the task, its dependencies and the services it needs: `hammerkit up dev:serve` keeps a dev server and its database running. A name that is no task or service is an error, and names can't be combined with `--filter`/`--exclude`. ## Pass arguments [#pass-arguments] Arguments after `--` are appended to the `cmd` of the one service (or task) `up` starts: ```bash hammerkit up api -- --inspect hammerkit up --filter app=api -- --inspect ``` Names or a filter that select several services or tasks take no arguments. See [arguments](/docs/task/arguments#services). Pass `--env ` to start the services in a configured environment, for example a Kubernetes cluster declared under `environments:`. See [running on Kubernetes](/docs/task/kubernetes). ## Options [#options] ``` Usage: hammerkit up [options] [names...] [-- args...] Arguments: names tasks and services to start, all when none is named Options: -f, --filter filter task and services with labels -e, --exclude exclude task and services with labels -c, --concurrency parallel worker count (default: 4, unbounded when every task and service requests cpus and memory) --skip-resource-check start even when a task with its services requests more than the host has -w, --watch watch tasks (default: false) --wait with --daemon, return once services are ready or once they started (choices: "ready", "start", default: "ready") -d, --daemon run services in background (default: false) --env environment -l, --log log mode (choices: "interactive", "live", "grouped") --cache caching method to compare (choices: "checksum", "modify-date", "none") --cache-read-only restore from cache backends but never push to them (or set HAMMERKIT_CACHE_READ_ONLY=1) --timeout fail tasks without their own timeout after this long (e.g. 10m) --max-timeout fail any task after this long, also one with a longer timeout of its own (e.g. 2m) --network the most any task or service reaches besides its needs: none, open or allow:, -h, --help display help for command ``` # Validate (/docs/cli/validate) `validate` checks the build file (and everything it references/includes) without running anything, so you can catch mistakes early — ideal as a CI lint step. ```bash hammerkit validate ``` It reports two levels: * **error** — something that would break a run, such as a dependency cycle. Any error makes `validate` exit with code `1`. * **warn** — something suspicious that still runs, such as a task with no description, a container service without a healthcheck, a `src`/`mount` path that doesn't exist, or build-file `envs` a task never references (see [environment variables](/docs/build-file/environment-variables#cache-key)). Warnings do **not** change the exit code. ## A clean run [#a-clean-run] With no problems, nothing is reported and the command exits `0`: ``` hammerkit validate ``` ## A run with findings [#a-run-with-findings] Each finding is grouped by the file it came from and shows its level, the item it applies to, and a message: ``` .hammerkit.yaml warn at postgres missing healthcheck warn at build missing description error at build task cycle detected build -> test -> build ``` Here the two warnings are advisory, but the cycle is an error, so the command exits `1` and a CI job using it fails. Run this command in CI alongside your other linting jobs. A malformed build file (an unknown key, a missing required `image`, the wrong type) is rejected by the schema as soon as hammerkit reads it — on *any* command, not just `validate`. `validate` adds the semantic checks above on top of that. ## Options [#options] ``` Options: -f, --filter filter task and services with labels -e, --exclude exclude task and services with labels -h, --help display help for command ``` # Concepts (/docs/concepts) Hammerkit introduces a handful of terms that are easy to mix up at first. This page defines each one in a line or two and links to the page that covers it in depth. The example below uses most of them together. ```yaml title=".hammerkit.yaml" envs: NODE_VERSION: '22' tasks: install: image: node:$NODE_VERSION-alpine # has an image -> runs in a container src: - package.json - package-lock.json generates: - node_modules cmds: - npm ci test: image: node:$NODE_VERSION-alpine deps: [install] # another task must run first needs: [postgres] # a service must be running first src: - src cmds: - npm test services: postgres: image: postgres:16-alpine healthcheck: cmd: "pg_isready -U postgres" ports: - 5432 ``` ## Task vs. service [#task-vs-service] * A **[task](/docs/task)** is a piece of work that runs and finishes — compile, test, lint, deploy. * A **[service](/docs/service)** is a long-running process a task depends on — a database, a message broker, an API. Hammerkit starts services when a task `needs` them and stops them when nothing needs them anymore. ## Local vs. container [#local-vs-container] The single rule: **an `image` means the task runs in a container; no `image` means it runs on your host.** Container tasks get their tools from the image (recommended — nothing to install locally, identical everywhere); local tasks use the tools installed on the machine. See [run a task in a container](/docs/task/container). ## `src`, `generates`, `mounts` [#src-generates-mounts] * **`src`** — the input files/folders a task reads. They drive [caching](/docs/task/caching): unchanged `src` lets a task be skipped. * **`generates`** — the output files/folders a task produces. This is what gets cached, [stored and restored](/docs/cli/store-restore). * **`mounts`** — extra paths a *container* task needs that aren't sources or outputs (config files, caches). See [container](/docs/task/container#adding-mounts). ## `deps` vs. `needs` [#deps-vs-needs] * **`deps`** — task → task. The dependency tasks run (or are restored from cache) first. See [dependencies](/docs/task/dependencies). * **`needs`** — task → service. The needed services are started and become healthy before the task runs. See [needs](/docs/task/needs). ## `references` vs. `includes` vs. `extend` [#references-vs-includes-vs-extend] All three pull in definitions from other build files, but differently: * **[`references`](/docs/build-file/references)** — point at another build file; its tasks are usable as `prefix:taskName`, keeping *their own* directory as the working directory. * **[`includes`](/docs/build-file/includes)** — like references, but the included tasks adopt the *including* file's working directory. Ideal for reusable task templates across a monorepo. * **[`extend`](/docs/task/extending)** — a per-task field that uses another task as a base template and overrides only what differs. ## Cache vs. store/restore [#cache-vs-storerestore] * **[Caching](/docs/task/caching)** is automatic: hammerkit skips a task whose `src` is unchanged, and with a cache **backend** it can share results across machines. * **[`store` / `restore`](/docs/cli/store-restore)** are explicit commands that move generated outputs and cache state in and out of a directory — typically wired into a CI provider's own cache step. These overlap; the [CI caching guide](/docs/guides/ci-caching) explains which to reach for. # Release procedure (/docs/contribution/publish) Releases are derived from commit messages. Nobody bumps a version or pushes a tag by hand. ## Commit messages [#commit-messages] `master` is squash-merged, so the **pull request title** is the commit message. It follows [Conventional Commits](https://www.conventionalcommits.org/): `: `. | Type | Release | | -------------------------------------------------------------- | --------------------------- | | `feat` | minor (`1.11.3` → `1.12.0`) | | `fix`, `perf` | patch (`1.11.3` → `1.11.4`) | | `docs`, `refactor`, `test`, `build`, `ci`, `chore`, `style` | none | | a `!` after the type (`feat!:`) or a `BREAKING CHANGE:` footer | **major** | A title in another shape (`Add x`, `1.12.0: add x`) is not an error, it just does not release. A breaking change releases a major version on its own, so it is only ever written on request. AI agents working in this repository ask first and never add the marker themselves. ## What happens on a merge to master [#what-happens-on-a-merge-to-master] [semantic-release](https://semantic-release.gitbook.io) (`.github/workflows/release.yaml`, `.releaserc.json`) reads the commits since the last tag. When one of them releases, it: 1. bumps `package.json` and `package-lock.json` and adds the section to `CHANGELOG.md`, in a `chore(release): x.y.z [skip ci]` commit on master, 2. tags `vX.Y.Z`, 3. starts the publish workflow (`.github/workflows/push.yaml`) for the tag: it creates the GitHub release with the changelog section as notes and the binaries attached, publishes the Docker Hub image and publishes to npm. For a stable release it then regenerates the formula in the [Homebrew tap](https://github.com/no0dles/homebrew-hammerkit) from the binaries' checksums and pushes it to the tap's `main`, 4. starts the website workflow (`.github/workflows/website.yaml`) on master, which rebuilds [hammerkit.dev](https://hammerkit.dev). Its [changelog](/changelog) page is generated from `CHANGELOG.md` (`website/scripts/changelog.mjs`), so it lists the new release without further work. Releases before 1.12 have hand-written entries in `website/content/changelog`, which win over a generated one for the same version. ## Setup and recovery [#setup-and-recovery] * `master` must accept pushes from the workflow (the release commit and the tag). * The Homebrew step needs the secret `HOMEBREW_TAP_TOKEN`, a fine-grained token with *Contents: write* on `no0dles/homebrew-hammerkit` only. Without it the step is skipped and the formula is updated by hand (`hammerkit.rb.tmpl` in the tap lists the placeholders). * The repository's releases are immutable, so the release is created together with its binaries and cannot be edited afterwards. If publishing fails before the release exists, fix the cause and re-run it from the tag: *Actions → publish → Run workflow*, choosing the `vX.Y.Z` tag as the ref. * A prerelease (`1.12.0-alpha.0`) is still cut by hand with `npm version prerelease --preid alpha` and `git push origin --follow-tags`; the npm dist-tag follows the preid. # Roadmap (/docs/contribution/roadmap) The topics below are planned for upcoming releases, roughly in priority order. Each one has a specification in the repository's [`specs/`](https://github.com/no0dles/hammerkit/tree/master/specs) folder; the order may change as real-world feedback comes in. ## Next [#next] ### Source exclusions [#source-exclusions] Exclude paths from a task's `src` with `!` entries (`src: [src, '!src/**/*.md']`), so a source folder can be narrowed without long globs. An excluded path is hidden from container tasks too, so a task can't read what isn't part of its cache key. ### Container runtime options [#container-runtime-options] Declare Docker runtime and security options such as `--shm-size` or a seccomp profile on container tasks and services, for workloads like browsers or sandboxed workers that need them. ## Later [#later] * **Task resources** — CPU and memory requests/limits per task, enforced on Docker and Kubernetes and used for resource-aware local scheduling instead of a flat worker count. * **Matrix tasks** — define a task once with a `matrix` of variants (Node versions, architectures) and run one independently cached task per combination. * **Service state snapshots** — cache the state a task leaves *inside* a service, such as a seeded database, so it can be restored instead of recreated. * **Multi-arch packaging** — let `hammerkit package` build and push multi-platform images. * **Distributed computing** — offload compute-intensive tasks to remote hardware with very little configuration. Server registration (`hammerkit remote`) has landed; see the [spec](https://github.com/no0dles/hammerkit/blob/master/specs/remote-compute/spec.md). # Shipped [#shipped] ## Task catalog (1.22.0) [#task-catalog-1220] Tested, versioned build files for common tools that a project includes straight from git instead of copying them. Shipped: [includes from a git repository](/docs/build-file/includes#include-from-a-git-repository) with `with` inputs, and the [catalog](/docs/catalog) entries for npm, Yarn, pnpm, pip, Go, Cargo, .NET, Docker, tsc, ESLint, Prettier, Jest and Helm. ## Agent-ready caching (1.7.0) [#agent-ready-caching-170] Make one shared cache work across coding-agent sandboxes, developer workspaces and CI, so work done in one place is never repeated in another. Shipped in 1.7.0: cache keys independent of the checkout location, the `registry` cache backend, [`cache pull` / `cache push`](/docs/cli/cache), read-only mode, [cache retention](/docs/build-file/caches#retention), skipping [dependencies of cached tasks](/docs/task/dependencies#dependencies-of-cached-tasks), [task timeouts](/docs/task#timeouts), and [`explain`](/docs/cli/explain), [`graph`](/docs/cli/graph), `run --dry-run` and the build summary. See the [release blog](/docs/release-blog/release-1.7.0) and [agents, workspaces and CI](/docs/guides/agents-and-ci). ## Distributed caching (1.6.0) [#distributed-caching-160] Distribute the local cache, so other developers and CI can reuse already built results. Shipped in 1.6.0 as [pluggable cache backends](/docs/build-file/caches): declare a `local` or `s3` cache backend and hammerkit pulls and pushes task results automatically, sharing them between developers and CI runs. ## Run on Kubernetes (1.6.0) [#run-on-kubernetes-160] Run the same build file on the local Docker daemon or on a Kubernetes cluster. Shipped in 1.6.0 through [environments](/docs/task/kubernetes). ## Services (1.5.0) [#services-150] Tasks sometimes require a service to run, for example a database for an API or an integration test. The goal was to define those services and have hammerkit start them for a task and stop them when they are no longer needed. Implemented in 1.5.0 with [container services](/docs/service/container) and [Kubernetes services](/docs/service/kubernetes). ## Platform requirements (1.5.0) [#platform-requirements-150] Define local tasks that require a specific platform. The goal was to either skip local tasks where the platform requirements are not met or run them on a suitable machine. Labels, available since 1.5.0, can be used to achieve this. # FAQ & Troubleshooting (/docs/faq) ## Where did my `dist` go? My container task ran but the files aren't in my project. [#where-did-my-dist-go-my-container-task-ran-but-the-files-arent-in-my-project] By default a container task's `generates` stay **inside the container volume** — fast, and reused on the next run, but not copied back to your working directory. Mark a generate with `export: true` to copy it back out: ```yaml title=".hammerkit.yaml" tasks: build: image: node:24-alpine generates: - path: dist export: true cmds: - npm run build ``` See [exporting generated files](/docs/task#exporting-generated-files). ## Why does my task run every time, even when nothing changed? [#why-does-my-task-run-every-time-even-when-nothing-changed] A task can only be skipped if it declares `src` of its own **and** all of its dependencies can also be skipped. A task with no `src` (or whose `src` matches no file — check the warnings), or one that depends on such a task, runs every time — hammerkit can't prove it's up to date. Give each task accurate `src`/`generates`. See [caching](/docs/task/caching). ## Why did my task rebuild? / Why was it skipped? [#why-did-my-task-rebuild--why-was-it-skipped] Run [`hammerkit explain `](/docs/cli/explain): it names the input that caused a miss — a changed `src` file, a changed dependency, or a changed task definition (`cmds`, `image`, `envs`, `mounts`, …). Editing a command or bumping an image invalidates the cache just like a source change. To force a run regardless, use `--cache none` or [`hammerkit clean`](/docs/cli/clean). See [what invalidates a key](/docs/task/caching#what-gets-cached-and-what-invalidates-it). ## No cache hits in CI, even though the code didn't change. [#no-cache-hits-in-ci-even-though-the-code-didnt-change] Almost always the `modify-date` method on a fresh checkout: `git clone` resets file modification times on every run, so a `modify-date` task always looks changed. Use the default `checksum` method in CI (it compares content, which is stable across checkouts). See [checksum vs. modify-date](/docs/task/caching#checksum-vs-modify-date) and the [CI caching guide](/docs/guides/ci-caching). ## `Cannot connect to the Docker daemon` / `docker: command not found`. [#cannot-connect-to-the-docker-daemon--docker-command-not-found] A task with an `image` needs a running container engine. Verify with `docker info`. If you don't want Docker for a step, remove its `image` so it runs on the host. See [installation](/docs/installation). ## A task fails with `missing environment variable NAME`. [#a-task-fails-with-missing-environment-variable-name] A `$NAME` reference in `envs` couldn't be resolved from the shell or a `.env` file. Either export the variable, add it to `.env`, or give the `env` a literal value. Hammerkit fails fast here on purpose so a build never runs with a silently-empty variable. See [environment variables](/docs/build-file/environment-variables#precedence). ## My task hangs "waiting" for a service that's running. [#my-task-hangs-waiting-for-a-service-thats-running] The service's [healthcheck](/docs/service/container#timing-and-failure) never exits `0`, so hammerkit never considers it ready. After the readiness deadline (`healthcheck.timeout`, default `20s`, or `HAMMERKIT_HEALTHCHECK_TIMEOUT`) the service fails and the run stops with the command that never passed. Make the `cmd` a real readiness probe (like `pg_isready`) and confirm it succeeds inside the container; raise the `timeout` if the service is just slow to start. ## My container task can't reach the service. [#my-container-task-cant-reach-the-service] A container task reaches a service by its **name** as the hostname on the service's **container port** (e.g. `postgres:5432`) — not via a published `ports` entry, and not via the `HAMMERKIT_*` variables (those are injected only for **local** tasks). See [services & networking](/docs/guides/services-networking). ## Generated files are owned by `root` (Linux). [#generated-files-are-owned-by-root-linux] On Linux a container task runs as your `uid:gid` and hammerkit `chown`s the working directory and mounts to you, so new files come back owned by you. The `chown` is not recursive, so root-owned files baked into the image are left as-is. On macOS/Windows the Docker Desktop VM handles this. See [file permission](/docs/task/container#file-permission). ## `No tasks found` (exit code 127). [#no-tasks-found-exit-code-127] The task name you asked for doesn't exist in the resolved build file. Run [`hammerkit ls`](/docs/cli/ls) to see the available task names (including `prefix:name` ones from references/includes). For the full exit-code contract see the [CLI reference](/docs/cli#exit-codes). ## Which file does hammerkit use as the build file? [#which-file-does-hammerkit-use-as-the-build-file] It looks for `.hammerkit.yaml`, then `.hammerkit.yml`, then `build.yaml`, and uses the first it finds (warning if several exist). Override with `--file `. See the [build file](/docs/build-file). # Getting started (/docs/getting-started) ## Prerequisites [#prerequisites] Hammerkit itself only needs [Node.js](https://nodejs.org) (see [installation](/docs/installation) for binary and Homebrew options that don't). Tasks that declare an `image` run **inside a container**, so you also need a running container engine — [Docker](https://docs.docker.com/get-docker/) or any Docker-compatible daemon. Containers are the recommended way to run tasks: the build then uses the exact same tools on every machine and in CI, instead of whatever happens to be installed locally. Check the daemon is up with: ```bash docker info ``` If that prints server information you're ready. If it errors, start Docker (or remove the `image:` line from the example below to run on your host instead). ## Create your first build file [#create-your-first-build-file] To get started with hammerkit, you will need to create your first build file. Build files are usually called `.hammerkit.yaml` and can be created from scratch manually or by using the init command. ```bash hammerkit init ``` ```bash npx hammerkit init ``` If you created your build file with the init command, the build file will look like the example below. ```yaml title=".hammerkit.yaml" envs: {} tasks: example: image: alpine cmds: - echo it's Hammer Time! ``` This build file contains an example command which prints a statement to the console inside an alpine container. The `image: alpine` line is what makes this task run in a container. Remove that line and the task runs directly on your host instead — no Docker required. See [run a task in a container](/docs/task/container) for the full rule. ## Run your first task [#run-your-first-task] To run your first command, make sure you're in the same directory as the build file and run: ```bash hammerkit example ``` ```bash npx hammerkit example ``` The first run pulls the `alpine` image if it isn't on your machine yet (later runs reuse it), starts a container, executes the echo command `it's Hammer Time!` and exits with code `0`. Run it a second time and it runs again: the task declares no `src`, so hammerkit can't tell whether anything it reads has changed. Declare the files a task reads under `src`, and an unchanged task is reported as cached and skipped — that's [caching](/docs/task/caching) at work. ## Discover the tasks in a project [#discover-the-tasks-in-a-project] When you clone a repository that already uses hammerkit, list everything it defines with: ```bash hammerkit ls ``` This prints every task and service with its image, sources, generated outputs and dependencies. For the full command list and global flags use: ```bash hammerkit --help ``` ## Where to go next [#where-to-go-next] * [Why hammerkit](/docs/why-hammerkit) — what it does that Make / npm scripts / Taskfile don't. * [Concepts](/docs/concepts) — local vs. container, `src`/`generates`/`mounts`, `deps` vs. `needs`. * [Tutorials](/docs/tutorials) — build, test and cache a project in Node.js, Python, Go, Rust or Java. * [Task](/docs/task) — sources, outputs, dependencies and caching in depth. * The runnable [`examples/`](https://github.com/no0dles/hammerkit/tree/master/examples) folder in the repository. ## Conclusion [#conclusion] In this short introduction you created your first build file and executed your first task. The build file and the task in this example were very basic; to see the full functionality of hammerkit, take a look at the rest of the documentation. # Agents, workspaces and CI (/docs/guides/agents-and-ci) Hammerkit's cache key is a hash of a task's inputs — command, image, sources and dependencies — computed relative to the project root. The same commit therefore produces the same keys in a coding agent's sandbox, on a CI runner and on a laptop. Put those results in a shared remote cache and: * **CI does not repeat what an agent already ran** — an agent that ran the e2e suite on a commit leaves the result behind; CI restores it and skips the job, including starting its database. * **New workspaces start warm** — a fresh sandbox pulls the dependency install and build outputs instead of rebuilding them. * **Less compute overall** — each unique input is built once, wherever it ran first. ## 1. Declare a remote cache [#1-declare-a-remote-cache] Any OCI registry works; use the one you already push images to. (An [S3-compatible bucket](/docs/build-file/caches#s3) works the same way.) ```yaml title=".hammerkit.yaml" caches: shared: method: checksum backend: type: registry repository: ghcr.io/my-org/hammerkit-cache services: db: image: postgres:16-alpine envs: POSTGRES_PASSWORD: postgres ports: - :5432 # random host port: parallel workspaces never collide healthcheck: cmd: pg_isready -U postgres tasks: install: image: node:24-alpine src: [package.json, package-lock.json] generates: [node_modules] cmds: [npm ci] build: image: node:24-alpine deps: [install] src: [src] generates: [dist] cmds: [npm run build] e2e: image: cypress/included:13.15.0 deps: [build] needs: [db] src: [cypress] cmds: [cypress run] ``` Tasks keep caching to the machine-local `default` cache; the `shared` cache is only touched by `cache pull` / `cache push`, so builds never wait on the network. Use **container tasks** for anything you want to share: a local task includes the host OS in its cache key, so a macOS laptop and a Linux runner would not share it. ## 2. In the agent sandbox [#2-in-the-agent-sandbox] Set up the workspace from the cache, work, then publish the results: ```bash docker login ghcr.io # or mount a docker config with a token hammerkit cache pull --remote shared # warm workspace: deps + build restored hammerkit run e2e # only what changed runs hammerkit cache push e2e --remote shared # leave the results for CI and others ``` ## 3. In CI [#3-in-ci] ```bash hammerkit cache pull --remote shared hammerkit run e2e # cache hits execute nothing hammerkit cache push --remote shared # publish anything CI had to build ``` A task whose inputs were already run by an agent completes as a cache hit in milliseconds, and services it needs are not started. ## Who may write to the cache? [#who-may-write-to-the-cache] Pushing is trusting: a cache hit means "this exact input already produced this result", and CI will not re-check it. Decide per runner: | Policy | Agents | CI | Trade-off | | ------------------------ | ----------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------- | | **Agents write** (above) | `cache push` | reads, pushes misses | Maximum reuse; CI trusts results produced in sandboxes. | | **CI writes only** | read-only token + `HAMMERKIT_CACHE_READ_ONLY=1` | reads + pushes | Agents still start warm; CI re-runs what agents ran. | | **Split** | push to `agents` cache | pulls `agents` for PR checks, only pushes to `shared` on `main` | Agent results speed up PR feedback, never the release path. | Enforce the policy with registry credentials (a pull-only token for read-only runners); `HAMMERKIT_CACHE_READ_ONLY=1` makes hammerkit refuse pushes as well, so a misconfigured runner fails loudly instead of writing. ## Checking why something rebuilt [#checking-why-something-rebuilt] ```bash hammerkit explain e2e # hit or miss, and which input changed hammerkit run --explain # the miss cause next to every rebuilt task hammerkit run --dry-run # predicted hits/misses without running anything ``` # CI through the server (/docs/guides/agents/github-actions) Experimental. The commands and settings can still change between releases. The server side of this guide was tested on a GKE server with a stand-in for the CI job's identity; the workflow of step 4 has not run on GitHub's runners yet. A CI job usually builds, tests and deploys on its runner, with the credentials of the cloud and the registry in the job. Whoever can change a workflow or a build script can read them. With a hammerkit server the job only says what to run: 1. The job signs in to the server with the token GitHub gives every job, which names the repository and the branch it runs for. Nothing else is stored in GitHub. 2. [`hammerkit remote run`](/docs/cli/remote#running-tasks) asks the server to run a task of the published build file at the job's commit. The server checks the commit out itself, runs the task in its cluster and streams the output back. The job fails when the task does. 3. A task that reads a secret reads it on the server, as a service [account](/docs/remote-compute/accounts) of the server. A production deploy account is one the job may only request: the run waits until a person who uses the account approves it. This guide adds that to a server set up with the [Kubernetes guide](/docs/guides/agents/kubernetes). Each `server.yaml` block below goes into the config of that guide's ConfigMap, next to the keys it already has, and the shell variables are those of its step 1. ## What you need [#what-you-need] * The server of the [Kubernetes guide](/docs/guides/agents/kubernetes), reachable from the internet over [HTTPS](/docs/guides/agents/kubernetes#reaching-the-server-over-https): GitHub's runners can not use a port-forward. The examples write it as `https://hammerkit.example.com`. * The repository's build file, with the tasks the jobs run. Here `test`, and `deploy`, which reads a secret and reaches the deploy target, written as `deploy.example.com`: ```yaml title=".hammerkit.yaml" tasks: test: image: node:22-alpine src: [src/**, test/**] cmds: [node --test] deploy: image: node:22-alpine deps: [test] secrets: - { from: 'gsm:deploy-token', account: deploy, env: DEPLOY_TOKEN } # a task that reads a secret gets no open network on this server, see step 3 network: { allow: [deploy.example.com] } # it reaches the target through the proxy; Node's fetch reads HTTPS_PROXY only with this, # from Node 22.21 and 24.5 (22 warns that it is experimental) envs: { NODE_USE_ENV_PROXY: '1' } cmds: [node deploy.js] ``` * The repository's id, which stays the same when the repository is renamed or moved: `curl -sS https://api.github.com/repos/acme/shop | jq .id` (with a token for a private one, or `gh api repos/acme/shop --jq .id`). ## 1. Let the server take GitHub's tokens [#1-let-the-server-take-githubs-tokens] Add GitHub's issuer to `issuers` in the server's config (the ConfigMap of step 2 of the Kubernetes guide), next to Google's: ```yaml title="server.yaml" issuers: - { issuer: https://accounts.google.com, audience: hammerkit } - { issuer: https://token.actions.githubusercontent.com, audience: https://hammerkit.example.com } ``` The `audience` is what a job asks its token for. The address of the server makes it a token for this server alone: a token a job got for another service, or another hammerkit server, is refused here. The server checks the token's signature with GitHub's keys and takes the claims GitHub puts in it, such as `repository_id` and `ref`, as they are. See [GitHub Actions](/docs/remote-compute/authentication#github-actions). ## 2. Say what the jobs may do [#2-say-what-the-jobs-may-do] ```yaml title="server.yaml" entitlements: # the ci workflow, on a push to main: publish the build file, run it, and ask for deploys - claims: iss: https://token.actions.githubusercontent.com match: repository_id: '123456789' ref: refs/heads/main event_name: push job_workflow_ref: acme/shop/.github/workflows/ci.yaml@refs/heads/main repos: [github.com/acme/shop] builds: publish accounts: { deploy: request } # every job of the repository, pull requests too: run what is published - claims: iss: https://token.actions.githubusercontent.com match: { repository_id: '123456789' } repos: [github.com/acme/shop] builds: run # who approves deploys, and new versions of what they run: a person, or a group - subject: { iss: https://accounts.google.com, sub: '' } repos: [github.com/acme/shop] builds: publish accounts: { deploy: use } ``` A claim matches when it is exactly the value written, and every claim of `match` has to. `ref: refs/heads/main` alone would be any job on `main`: a scheduled one, one started by hand, or one of another workflow file with `id-token: write`. `event_name` and `job_workflow_ref` keep it to the `ci` workflow of step 4 on a push. Entitlements add up: that job matches both, and gets the higher level. That job publishes. What the build file's tasks run is then what the `ci` workflow on `main` publishes, and not whatever somebody with a publisher account uploads. Once this works, take `builds: publish` away from the people and service accounts that only published, such as the publisher of the Kubernetes guide. The server still takes the files the job uploads as they are: any step of that job, an action or an install script of a package, could upload another build file. Tasks that read secrets as `deploy` are protected by the approval of [When the build file changes](#when-the-build-file-changes); tasks without secrets are published at once. The approver keeps `builds: publish`: a new version that changes what `deploy` runs needs their approval (see [When the build file changes](#when-the-build-file-changes)), and approving a version takes `publish`. It also lets them upload a version themselves, so keep it to the few people who approve deploys. ## 3. Keep the deploy account to main [#3-keep-the-deploy-account-to-main] The `deploy` account is set up like any account that reads secrets on the cluster, see [Secrets in the Job](/docs/remote-compute/backend#secrets-in-the-job). Two lines keep it to reviewed code: ```yaml title="server.yaml" accounts: # next to github deploy: description: production deploys source: [ref] # only code the server checked out itself, never an upload or a sandbox refs: [main] # and only from main providers: gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /var/run/hammerkit/account/gcp.json } } kubernetes: serviceAccount: deploy # a ServiceAccount of the server's namespace, trusted by your cloud audience: $AUDIENCE # what the pool of the Kubernetes guide's step 1 accepts secret: deploy-federation ``` The Kubernetes guide's server gives a task that reads a secret no network at all (`runs.secretNetwork: none`), and refuses `deploy`, which has to reach its target. Allow the target there, and give Secret Manager's client the hosts it reads the secret from: ```yaml title="server.yaml" runs: env: { allow: [PATH] } args: declared # what a task that reads a secret may reach: nothing, or the deploy target secretNetwork: { allow: [deploy.example.com] } secretProviders: github-app: { command: [hammerkit, github-app-token, '{{ref}}'] } # the project of the secrets: a federation config names none gsm: { command: [gcloud, secrets, versions, access, latest, '--secret={{ref}}', '--project=$PROJECT'] } backend: type: kubernetes # its other keys as in the Kubernetes guide, and providers providers: gsm: image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim allow: [sts.googleapis.com, secretmanager.googleapis.com] ``` `secretNetwork` is a ceiling for every task that reads a secret: each may name `deploy.example.com` now, or say `network: none`. One that leaves `network` out asks for an open network, and a version with it, or with another host, is refused at publish: `deploy: asks for network open, more than this server allows to what reads the secret DEPLOY_TOKEN (runs.secretNetwork: allow deploy.example.com)`. The command of `deploy` does not reach Secret Manager: its egress proxy lets only the container that reads the secret through to `providers.gsm.allow`, see [Secrets of a task with a limited network](/docs/remote-compute/backend#secrets-of-a-task-with-a-limited-network). The server makes a token of the account's ServiceAccount for each Job of `deploy`, and reads its Secret, so it needs both, by name, in its namespace: ```bash kubectl apply -f - < deploy-gcp.json <- curl -sSf -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=https://hammerkit.example.com" | jq -r .value # the branch and commit the server checks out: a pull request's head, or the pushed commit REF: ${{ github.head_ref || github.ref_name }} SHA: ${{ github.event.pull_request.head.sha || github.sha }} # a deploy that waits for its approval holds the job, an hour at most timeout-minutes: 75 concurrency: group: ci-${{ github.ref }} # a newer push to a pull request stops the older job, and its run on the server cancel-in-progress: ${{ github.event_name == 'pull_request' }} steps: - uses: actions/checkout@v4 with: { ref: '${{ env.SHA }}' } - uses: actions/setup-node@v4 with: { node-version: 24 } - run: npm install -g hammerkit@2.0.0 - run: | hammerkit remote add hammerkit https://hammerkit.example.com --issuer https://token.actions.githubusercontent.com hammerkit remote use hammerkit - if: github.event_name == 'push' run: hammerkit remote publish - run: hammerkit remote run test --ref "$REF" --commit "$SHA" - if: github.event_name == 'push' run: hammerkit remote run deploy --ref "$REF" --commit "$SHA" ``` What each step does: * **The token.** `id-token: write` lets the job ask GitHub for a token. hammerkit runs `HAMMERKIT_TOKEN_COMMAND` when it first calls the server, and again when the server no longer accepts the token, so a long run outlives one token. * **`remote add`** registers the server with GitHub as the issuer of the job's tokens. It checks that the server takes them. * **`remote publish`**, on `main` only, publishes the build file of the commit. A build file that did not change since the last push is `already published`. * **`REF` and `SHA`** come from the event: a pull request's job would check out a merge commit the server does not know, so the job checks out the head of the branch and names it. They go to the commands as variables, not into the script: a branch name is chosen by whoever opens the pull request, and may hold shell. * **`remote run test`** runs `test` at that commit. It names the build file by the hash of the one checked out. A ref that moved since, after a newer push, is refused, and the newer push's job runs it. * **A pull request that changes the build file** fails at `remote run` with `the build file here is not published on hammerkit (sha256:…), publish it first`: its job may only run what is published, and its version is published once it is merged. Read that failure as "the tasks changed", and look at what `main`'s job does after the merge. * **`remote run deploy`** runs `deploy` from `main` at the pushed commit. * **`concurrency`** stops an older job of a pull request when a newer push arrives, which cancels its run on the server. Jobs on `main` are not stopped, so a deploy is never cut off halfway: a newer push waits until the older deploy is approved or rejected. The job exits as the task did, with its exit code, and prints its output as it comes. The runner holds no cloud credentials, no registry login and no deploy key: nothing a changed workflow could print. ## 5. Approve a deploy [#5-approve-a-deploy] The job of a push to `main` waits at `deploy`: ``` run_Qx3k9ZpL2mW7vN4a on hammerkit: deploy from main at 4be1a07c... waiting until somebody who uses deploy approves it, before 2026-10-24T10:41:00.000Z: hammerkit remote approve run_Qx3k9ZpL2mW7vN4a ``` The approver looks at it and lets it go, signed in as themselves: ```bash hammerkit remote approvals hammerkit hammerkit remote show run_Qx3k9ZpL2mW7vN4a hammerkit hammerkit remote approve run_Qx3k9ZpL2mW7vN4a hammerkit ``` `remote show` lists every command the run executes, its image and the secrets it reads, so the approver does not need to read the build file. The job then prints the deploy's output and ends. A run nobody approves within an hour (`approvals.timeout`) is rejected, and the job fails. See [Approving runs](/docs/cli/remote#approving-runs). ## When the build file changes [#when-the-build-file-changes] The job of `main` may only request `deploy`. A version of the build file that reads secrets as `deploy` is therefore not published by it but proposed: somebody who uses `deploy` and may publish approves it first, so a change to what the deploy runs takes a second person. The job prints `proposed to hammerkit`, and the runs after it fail with `the build file here is not published on hammerkit (sha256:…), publish it first`. The approver: ```bash hammerkit remote proposals hammerkit hammerkit remote show sha256:9f2c… hammerkit # every file of the new version hammerkit remote approve sha256:9f2c… hammerkit ``` Then re-run the job. A version that changes only tasks without secrets is published at once. ## Checks without waiting [#checks-without-waiting] The `test` step above holds the runner while the server runs the tests. When the agent or the person who pushed already ran them on the server, the job only has to verify that they passed for the files of its commit, which takes seconds: see [Checks that ran before CI](/docs/guides/agents/verified-checks). ## What is not covered yet [#what-is-not-covered-yet] * The server does not watch the repository itself: a job still starts each run, so GitHub Actions is still needed. * Image builds need a Docker daemon, which the cluster does not offer to tasks. * Pull requests from forks get no token from GitHub, so their jobs can not use the server. # Service accounts on Google Cloud (/docs/guides/agents/google-accounts) An agent never signs in to the server as you. It signs in as a **service account** of its own, and the server's [entitlements](/docs/remote-compute/entitlements) decide what that account may do. The guides use two accounts: | Account | Signs in to | Entitlement on the server | | --------------------- | --------------------------------------------------- | ------------------------------------------------ | | `hammerkit-agent` | start sandboxes, run tasks, push `agent/…` branches | `builds: run` and `sandboxes` | | `hammerkit-publisher` | publish build files (`hammerkit remote publish`) | `builds: publish` (includes run), no `sandboxes` | Neither account gets a key file or a role in your project. Instead, you are allowed to **impersonate** them: `gcloud` asks Google for a short-lived token of the account on your behalf, and that is the token hammerkit sends to the server. This page sets that up. If you followed [Server on Kubernetes](/docs/guides/agents/kubernetes), its step 1 already created both accounts for you, named `$NAME-agent` and `$NAME-publisher`; skip to [Let your team act as them](#3-let-your-team-act-as-them) to add more people. ## What you need [#what-you-need] * A Google Cloud project, and `gcloud` signed in as a user who may create service accounts in it (for example the `Service Account Admin` role). ```bash PROJECT=my-project ME=$(gcloud config get account) ``` ## 1. Create the accounts [#1-create-the-accounts] ```bash gcloud services enable iam.googleapis.com iamcredentials.googleapis.com --project $PROJECT gcloud iam service-accounts create hammerkit-agent --project $PROJECT --display-name "hammerkit agent" gcloud iam service-accounts create hammerkit-publisher --project $PROJECT --display-name "hammerkit publisher" ``` `iamcredentials.googleapis.com` is the API that hands out tokens of a service account to someone allowed to impersonate it. ## 2. Allow yourself to act as them [#2-allow-yourself-to-act-as-them] Impersonating an account needs the role **Service Account Token Creator** on that account. Grant it to yourself, on these two accounts only: ```bash for who in agent publisher; do gcloud iam service-accounts add-iam-policy-binding hammerkit-$who@$PROJECT.iam.gserviceaccount.com \ --project $PROJECT --role roles/iam.serviceAccountTokenCreator --member user:$ME done ``` The narrower role Service Account OpenID Token Creator is not enough: `gcloud` with `--impersonate-service-account` also asks for an access token of the account. Signed in to `gcloud` as a service account yourself, for example in CI, write `--member serviceAccount:…` instead of `user:…`. A new binding can take a minute before it works. ## 3. Let your team act as them [#3-let-your-team-act-as-them] Everybody who starts an agent needs the same role on `hammerkit-agent`. Give it to each person, or to a Google group: ```bash gcloud iam service-accounts add-iam-policy-binding hammerkit-agent@$PROJECT.iam.gserviceaccount.com \ --project $PROJECT --role roles/iam.serviceAccountTokenCreator --member group:dev-team@example.com ``` Keep `hammerkit-publisher` to the people or the CI job that publish build files: whoever can act as it decides what the agent's commands run in. ## 4. Check that a token comes back [#4-check-that-a-token-comes-back] ```bash gcloud auth print-identity-token --audiences=hammerkit --include-email \ --impersonate-service-account=hammerkit-agent@$PROJECT.iam.gserviceaccount.com ``` This is the command the tutorials put in `HAMMERKIT_TOKEN_COMMAND`, or that you register with a server with [`remote add --token-command`](/docs/cli/remote#a-token-command-per-server). It prints a long string starting with `eyJ`, a token that is valid for an hour, and a warning on stderr that it impersonates the account, which is expected. `--audiences=hammerkit` has to match the `audience` of `issuers` in the server's config. A `Permission 'iam.serviceAccounts.getAccessToken' denied` (or `getOpenIdToken`) error means step 2 or 3 is missing for your user, or has not taken effect yet. ## 5. Give the server the accounts' ids [#5-give-the-server-the-accounts-ids] The server recognizes an account by its unique id, the `sub` of its tokens, not by its email: ```bash gcloud iam service-accounts describe hammerkit-agent@$PROJECT.iam.gserviceaccount.com \ --project $PROJECT --format='value(uniqueId)' gcloud iam service-accounts describe hammerkit-publisher@$PROJECT.iam.gserviceaccount.com \ --project $PROJECT --format='value(uniqueId)' ``` Each prints a number such as `104857204856125339187`. Whoever runs the server puts them in the entitlements of its config, here for the repository `github.com/acme/shop`: ```yaml title="server.yaml" issuers: - { issuer: https://accounts.google.com, audience: hammerkit } entitlements: - subject: { iss: https://accounts.google.com, sub: '' } repos: [github.com/acme/shop] builds: publish - subject: { iss: https://accounts.google.com, sub: '' } repos: [github.com/acme/shop] builds: run sandboxes: - repo: github.com/acme/shop branches: { main: pull, 'agent/**': push } # start from main, push to agent/… ``` Then follow [pi on a sandbox](/docs/guides/agents/pi) or [opencode on a sandbox](/docs/guides/agents/opencode). # Agents on a server (/docs/guides/agents) Experimental. The tools and settings can still change between releases. A coding agent on a hammerkit server works on a [sandbox](/docs/remote-compute/sandboxes): a checkout of a branch that stays on the server. The agent runs where you start it, on a laptop or in CI, but every file it reads or edits and every command it runs is on the server, in the images, network and accounts of a published build file. When it is done it pushes a branch for review. These guides set it up end to end, the way it was run to check them: **A server.** [On Kubernetes (GKE)](/docs/guides/agents/kubernetes), where every command of an agent is a Job in a namespace of its own, or [on one machine](/docs/remote-compute#setting-up-a-server). **A build file with a shell task**, published to the server (below). **An identity for the agent**: a service account of your identity provider with an [entitlement](/docs/remote-compute/entitlements) that may start sandboxes from `main` and push to `agent/**`. **A harness**: [pi](/docs/guides/agents/pi), [opencode](/docs/guides/agents/opencode), or any MCP client with [`hammerkit remote mcp`](/docs/remote-compute/agents#hammerkit-remote-mcp). ## The build file [#the-build-file] The agent's commands run as the arguments of a [shell task](/docs/remote-compute/agents#the-shell-task), so the build file decides the image, the network and the accounts of everything the agent runs. Other tasks, such as `test`, are there for the agent to run as they are. ```yaml title=".hammerkit.yaml" tasks: sh: image: node:22-alpine mounts: [.] args: true cmds: [sh -c] network: { allow: [registry.npmjs.org] } test: image: node:22-alpine src: [src/**, test/**] args: true cmds: [node --test] ``` `mounts: [.]` puts the files of the sandbox into the container, writable, and `args: true` marks the tasks that take arguments, the agent's commands and a test file: a server with [`runs.args: declared`](/docs/remote-compute/runs#arguments) passes them to no other. The task reads no secrets: a task that does takes no arguments on a server. Publish it with [`hammerkit remote publish`](/docs/cli/remote#publishing-a-build-file) as an identity with `builds: publish`; it prints the hash of the version. The agent then names the repository, the branch and that hash: | Variable | Example | What it names | | ------------------------- | ------------------------------------ | ---------------------------------------------------------- | | `HAMMERKIT_SERVER` | `hammerkit` | The server, as `hammerkit remote add` registered it. | | `HAMMERKIT_REPO` | `github.com/acme/shop` | The repository of the sandbox. | | `HAMMERKIT_REF` | `main` | The branch it starts from. | | `HAMMERKIT_BUILD` | `sha256:9f2c…` | The published version of the build file. | | `HAMMERKIT_TOKEN_COMMAND` | `gcloud auth print-identity-token …` | Prints a short-lived token of the agent's service account. | A run names exactly the version it runs, so nothing follows a newer one: when the build file changes, publish it again and set the new hash. See [Settings](/docs/remote-compute/agents#settings) for the rest, such as `HAMMERKIT_SANDBOX` to come back to a sandbox with work in progress. ## On the agent's machine [#on-the-agents-machine] The server is registered per user, in `~/.hammerkit/config.yaml`. Register it as the user that starts the agent, with the issuer of the agent's tokens: ```bash npx hammerkit remote add hammerkit https://hammerkit.example.com --issuer https://accounts.google.com ``` The token command runs again whenever the server stops accepting a token, so a session outlives one token. It is the agent's own identity, never a person's sign-in. ## What it is not yet [#what-it-is-not-yet] * A dev server or another service on the sandbox for the agent to send requests to. * Commands that read input, or keep running in the background. ## CI on the same server [#ci-on-the-same-server] CI jobs can use the server too, with nothing but their own token: see [CI through the server](/docs/guides/agents/github-actions). A job does not even have to wait for the tests when the agent checked them before its push: see [Checks that ran before CI](/docs/guides/agents/verified-checks). # Server on Kubernetes (GKE) (/docs/guides/agents/kubernetes) Experimental, like the server. This guide was run as written on a GKE Autopilot cluster (Kubernetes 1.36) in `europe-west4`. You end up with a hammerkit server in its own namespace of your cluster. Agents work on [sandboxes](/docs/remote-compute/sandboxes) there: every command they run is a Job in a namespace of its own, and their work leaves as a branch pushed by your GitHub App. No key is stored in the cluster, and no image has to be built: the server runs the released `no0dles/hammerkit` image. ## What you need [#what-you-need] * A GKE cluster that enforces `NetworkPolicy`. Autopilot does; on a Standard cluster turn on Dataplane V2. * `gcloud`, `kubectl` and `jq`, signed in with the right to create ClusterRoles. * The hammerkit CLI of the server's version, on Node.js 24 or later: `npm install -g hammerkit@$VERSION` with `VERSION` from below. On an older Node.js npm warns `EBADENGINE`, and with a range instead of an exact version it quietly installs an old hammerkit without the commands this guide uses. * A GitHub App installed on your repository, with **Contents: read and write**, **Metadata: read** and nothing else. Download a private key for it (`app.pem`). * A build file with a shell task in the repository, see [The build file](/docs/guides/agents#the-build-file). Set these once; every command below uses them: ```bash PROJECT=my-project REGION=europe-west4 REPO=acme/shop # your GitHub repository GITHUB_APP_ID=123456 # on the App's settings page VERSION=2.0.0 # one hammerkit version for the server, the CLI and the agent NAME=hammerkit # namespace and account names; another server on the cluster needs another PROJECT_NUMBER=$(gcloud projects describe $PROJECT --format='value(projectNumber)') ``` ## 1. Google: the App key and the accounts [#1-google-the-app-key-and-the-accounts] The server reads the App's key from Secret Manager as its Kubernetes ServiceAccount. For that, an identity pool trusts the cluster's tokens, and only the server's ServiceAccount may read the key. Two Google service accounts give the agent and the publisher an identity to sign in with. ```bash gcloud services enable sts.googleapis.com secretmanager.googleapis.com iam.googleapis.com \ iamcredentials.googleapis.com --project $PROJECT # let the cluster's ServiceAccounts sign in to Google gcloud iam workload-identity-pools create $NAME --location global --project $PROJECT gcloud iam workload-identity-pools providers create-oidc cluster --workload-identity-pool $NAME \ --location global --project $PROJECT --allowed-audiences $NAME \ --issuer-uri $(kubectl get --raw /.well-known/openid-configuration | jq -r .issuer) \ --attribute-mapping google.subject=assertion.sub POOL=projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$NAME # the App's key, readable by the server only gcloud secrets create $NAME-github-app-key --project $PROJECT --data-file app.pem gcloud secrets add-iam-policy-binding $NAME-github-app-key --project $PROJECT \ --role roles/secretmanager.secretAccessor \ --member principal://iam.googleapis.com/$POOL/subject/system:serviceaccount:$NAME:hammerkit # identities for the agent and the publisher, which you may sign in as ME=user:$(gcloud config get account) # as a service account: ME=serviceAccount:; empty: set it yourself for who in agent publisher; do gcloud iam service-accounts create $NAME-$who --project $PROJECT gcloud iam service-accounts add-iam-policy-binding $NAME-$who@$PROJECT.iam.gserviceaccount.com \ --project $PROJECT --role roles/iam.serviceAccountTokenCreator --member $ME done # what the next steps use AUDIENCE=$NAME KEY_SECRET=projects/$PROJECT/secrets/$NAME-github-app-key/versions/latest AGENT=$NAME-agent@$PROJECT.iam.gserviceaccount.com PUBLISHER=$NAME-publisher@$PROJECT.iam.gserviceaccount.com AGENT_ID=$(gcloud iam service-accounts describe $AGENT --project $PROJECT --format='value(uniqueId)') PUBLISHER_ID=$(gcloud iam service-accounts describe $PUBLISHER --project $PROJECT --format='value(uniqueId)') ``` Delete `app.pem` now. The two service accounts hold no role in the project: only the server reads their tokens. A new IAM binding can take a few minutes before it works. `AUDIENCE` is what the pool accepts from the server's pod; it is unrelated to the `audience: hammerkit` of the sign-in tokens in step 2. **Did someone else do this step?** Ask them for the values the last block sets, and set them in your shell: `POOL` (the pool's full name, as above), `AUDIENCE` (the audience its provider allows), `KEY_SECRET` (the Secret Manager version of the App's key), `AGENT` and `PUBLISHER` (the emails of the two service accounts) and `AGENT_ID` and `PUBLISHER_ID` (their unique ids). The pool's provider has to be named `cluster`, or change `providers/cluster` in `gcp.json` of step 2. They also have to let the server's ServiceAccount, `system:serviceaccount:$NAME:hammerkit`, read the key, and let you act as the two accounts. [Service accounts on Google Cloud](/docs/guides/agents/google-accounts) explains these two accounts, and how to let your team act as the agent. ## 2. Deploy the server [#2-deploy-the-server] One manifest holds everything the server needs in the cluster: its namespace and ServiceAccount, its rights, its config, a disk and the Deployment. It uses the variables of step 1; the first line stops when one of them is not set, for example in a new terminal. ```bash : "${NAME:?}" "${REPO:?}" "${GITHUB_APP_ID:?}" "${VERSION:?}" "${POOL:?}" "${AUDIENCE:?}" \ "${KEY_SECRET:?}" "${AGENT_ID:?}" "${PUBLISHER_ID:?}" kubectl apply -f - < /dev/null && echo "the server can read $REPO" ``` When it fails, it says which step did: * `Secret Manager did not give …: 403` means the server's ServiceAccount may not read the key: the binding of step 1 is missing, names another namespace, or is not in effect yet. * `Google STS did not exchange the token …` means the pool or its provider does not match: compare `POOL` and `AUDIENCE` with step 1. * `the GitHub App … is not installed on …` means the App is not installed on the repository. An agent whose sandbox can not be created gets the same reason from the server. ## 3. Connect and publish [#3-connect-and-publish] Forward the server's port to your machine and register it: ```bash while true; do kubectl -n $NAME port-forward svc/hammerkit 8480:8480; sleep 1; done & hammerkit remote add $NAME http://127.0.0.1:8480 --issuer https://accounts.google.com ``` `kubectl port-forward` ends when the server's pod is replaced, for example after a `kubectl apply`, and now and then when its connection drops; the loop starts it again. When it prints the same error every second, for example because port 8480 is taken or the cluster can not be reached, stop it with `kill %1` and fix that first. Before you publish, check that the build file's shell task has `mounts: [.]`, `args: true` and `cmds: [sh -c]`, as in [The build file](/docs/guides/agents#the-build-file). Without the mount, the agent's commands run in an empty directory; without `args: true` this server refuses them. Then publish the build file from a checkout of the repository, signed in as the publisher: ```bash HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \ --impersonate-service-account=$PUBLISHER" \ hammerkit remote publish $NAME ``` It prints the hash of the published version, `sha256:` and 64 hex digits. The agent names the build file by that hash, so a later publish never changes what a running agent uses. ## 4. Start an agent [#4-start-an-agent] Follow [pi](/docs/guides/agents/pi) or [opencode](/docs/guides/agents/opencode), with the server registered as above, the build file's hash and this token command: ```bash export HAMMERKIT_BUILD=sha256:9f2c… # the hash hammerkit remote publish printed export HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \ --impersonate-service-account=$AGENT" ``` The tutorials write the accounts as `hammerkit-agent@my-project…` and `hammerkit-publisher@my-project…`: use `$AGENT` and `$PUBLISHER` from step 1 instead. The agent's branches show up in the repository as `agent/…`, pushed by your GitHub App. ## Reaching the server over HTTPS [#reaching-the-server-over-https] `kubectl port-forward` needs access to the cluster on every machine. A team, or a CI job such as [GitHub Actions](/docs/guides/agents/github-actions), reaches the server through an HTTPS load balancer instead: a GKE Ingress with a Google-managed certificate for a domain of yours. The config already expects TLS to end in front of the server (`behindTlsProxy`), and the server takes nothing without a token of one of its issuers, so the address can be public. Reserve an address and point your domain at it, with an `A` record at your DNS provider: ```bash DOMAIN=hammerkit.example.com # a name in a zone you control gcloud compute addresses create $NAME --global --project $PROJECT gcloud compute addresses describe $NAME --global --project $PROJECT --format='value(address)' ``` Then the certificate and the load balancer: ```bash : "${NAME:?}" "${DOMAIN:?}" kubectl -n $NAME annotate --overwrite svc hammerkit \ cloud.google.com/neg='{"ingress": true}' cloud.google.com/backend-config='{"default": "hammerkit"}' kubectl apply -f - < --region $REGION --format='value(clusterIpv4Cidr,servicesIpv4Cidr)'` shows your ranges. Without the right to describe the cluster, `kubectl get svc -A -o jsonpath='{..clusterIP}'` shows service addresses that have to be in the denied ranges. An internal address tasks need goes in `allowedEgress`. * **`networkPolicy: enforced`.** The agent's commands only reach what `sandboxes.network` allows, here npm. See [Network](/docs/remote-compute/backend#network). * **`runs.args: declared`.** Arguments reach only the tasks that say `args: true`, so an agent can't append its own flags to the last command of any other task, for example to an install that has to keep `--ignore-scripts`. See [Arguments](/docs/remote-compute/runs#arguments). * **`runs.secretNetwork: none`.** A task that reads a secret reaches only the services it needs, so a tool it runs can't send the value anywhere. Give such a task `network: none`, or set an allow list here and in the task. See [Tasks that read secrets](/docs/remote-compute/runs#tasks-that-read-secrets). * **One replica.** The disk holds the server's records of sandboxes and its cache. ## Secrets for builds [#secrets-for-builds] The agent's shell task reads no secrets. Other tasks of the build file can, for example a test against a staging API: the Job of such a task reads the secret itself, as a Kubernetes ServiceAccount of its [account](/docs/remote-compute/accounts). See [Secrets in the Job](/docs/remote-compute/backend#secrets-in-the-job) for the setup with Secret Manager. With `secretNetwork: none` such a task has a limited network, so its provider needs the hosts it reaches, which its egress proxy lets only the reading of the secret through: ```yaml title="server.yaml" backend: type: kubernetes providers: gsm: image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim allow: [sts.googleapis.com, secretmanager.googleapis.com] ``` ## Cleaning up [#cleaning-up] Stop the port-forward loop first (`kill %1`, or close its terminal), then: ```bash # the namespaces of this server's callers and service runs: namespacePrefix and 16 hex digits; look before you delete kubectl get ns -l hammerkit.dev/managed -o name | grep -E "^namespace/$NAME-[0-9a-f]{16}$" kubectl get ns -l hammerkit.dev/managed -o name | grep -E "^namespace/$NAME-[0-9a-f]{16}$" | xargs -r kubectl delete kubectl delete ns $NAME kubectl delete clusterrole $NAME-server $NAME-runs kubectl delete clusterrolebinding $NAME-server kubectl delete validatingadmissionpolicybinding,validatingadmissionpolicy $NAME-managed-namespaces $NAME-service-run-namespaces gcloud iam workload-identity-pools delete $NAME --location global --project $PROJECT --quiet gcloud secrets delete $NAME-github-app-key --project $PROJECT --quiet gcloud compute addresses delete $NAME --global --project $PROJECT --quiet # if you reserved one for HTTPS for who in agent publisher; do gcloud iam service-accounts delete $NAME-$who@$PROJECT.iam.gserviceaccount.com --project $PROJECT --quiet done ``` Every server labels the namespaces it makes `hammerkit.dev/managed`, so on a cluster with several servers delete by the whole name, as above, not by the label alone. A deleted pool keeps its name for 30 days. # opencode on a sandbox (/docs/guides/agents/opencode) Experimental. The tools and settings can still change between releases. At the end of this tutorial you start [opencode](https://opencode.ai) on your machine, give it a task for your project, and it works on a copy of your repository on a hammerkit server. Your own checkout stays untouched: opencode never reads or writes a file of it and never runs a command on your machine. When it is done, its work shows up as a branch in your repository, ready for a pull request. This page was run as written with opencode 2.0.30 against the server of [Server on Kubernetes](/docs/guides/agents/kubernetes). ## How it fits together [#how-it-fits-together] Four pieces work together. You set up the last three in this tutorial. * **The server** keeps a **sandbox** for the agent: a checkout of your repository at a branch, for example `main`. Every file opencode reads or edits is a file of the sandbox. * **Your build file** has a **shell task**. Every command opencode runs, such as `npm test`, is handed to that task as its argument and runs on the server, in the task's image. The build file is **published** to the server once; the server answers with a hash that names exactly that version. * **The agent's identity** is a service account, not you. opencode signs in to the server as that account, and the server only lets it start sandboxes from `main` and push to `agent/…` branches. * **opencode** runs on your machine and talks to the language model as usual. hammerkit's opencode plugin replaces opencode's tools that read, edit, search and run commands with ones of the same name that work on the sandbox, and adds a `push` tool. ## Before you start [#before-you-start] **A hammerkit server for your GitHub repository.** [Server on Kubernetes](/docs/guides/agents/kubernetes) sets one up on GKE: the server checks out and pushes to your repository with a GitHub App, and accepts Google sign-ins. If someone else runs the server, ask them for its address and the issuer of the tokens it accepts, for example `https://accounts.google.com`. **Service accounts for the agent and the publisher**, and the right to act as them. [Service accounts on Google Cloud](/docs/guides/agents/google-accounts) creates `hammerkit-agent` and `hammerkit-publisher`, gives you the role that lets `gcloud` print their tokens, and shows the entitlements the server needs for them. The Kubernetes guide does this in its step 1; with a server someone else runs, have them add you as in [Let your team act as them](/docs/guides/agents/google-accounts#3-let-your-team-act-as-them). **Node.js 24 or later** on your machine, for opencode and the hammerkit CLI. **An API key of a model provider** that opencode supports. The examples use OpenRouter. The examples below use the repository `github.com/acme/shop`, a server registered as `hammerkit`, and Google service accounts. Replace them with yours. ## 1. Install the hammerkit CLI [#1-install-the-hammerkit-cli] ```bash npm install -g hammerkit@2.0.0 # the server's version hammerkit --version ``` `hammerkit --version` has to print the server's version. hammerkit needs Node.js 24: on an older one npm warns `EBADENGINE`, and with a range instead of an exact version it quietly installs an old hammerkit without the agent clients. Use the same version as the server. The CLI registers the server in step 2 and publishes the build file in step 3. ## 2. Register the server on your machine [#2-register-the-server-on-your-machine] The CLI keeps a list of servers for your user, in its config file. Add the server under a short name of your choice, with the issuer of the tokens it accepts: ```bash hammerkit remote add hammerkit http://127.0.0.1:8480 --issuer https://accounts.google.com ``` `hammerkit` is the name you use from now on. The address here is the `kubectl port-forward` of the Kubernetes guide, which has to keep running while opencode works. Plain `http` is only accepted for `127.0.0.1` and `localhost`; a server on any other address needs `https`. Register it as the same user that starts opencode later: the plugin reads this list. ## 3. Add a shell task to your project and publish it [#3-add-a-shell-task-to-your-project-and-publish-it] Open your project's `.hammerkit.yaml`, or create one in the root of the repository, and add a task for the agent's commands. For a Node.js project: ```yaml title=".hammerkit.yaml" tasks: sh: image: node:22-alpine mounts: [.] args: true cmds: [sh -c] network: { allow: [registry.npmjs.org] } ``` Line by line: * `sh` is the task's name. The plugin looks for a task called `sh` unless you set `HAMMERKIT_SHELL`. * `image` is the container every command of the agent runs in. Pick one with the tools your project needs: the agent can only run `npm`, `go` or `python` if the image has them. * `mounts: [.]` puts the files of the sandbox into the container, writable. Without it, the commands would run in an empty directory. * `args: true` says the task takes [arguments](/docs/task/arguments). A server can be set to pass arguments to such tasks only, so mark the shell task even where nothing requires it. * `cmds: [sh -c]` is the trick: the agent's command is appended to the last command as one argument, so `npm test` runs as `sh -c 'npm test'`. * `network` lists what the commands may reach. The sandbox is a fresh checkout without `node_modules`, so the agent installs dependencies itself and needs the registry. The server can narrow this further, never widen it. Leave secrets out of this task: a task that reads a secret takes no arguments on a server. Your other tasks, such as `test` or `lint`, stay as they are. If you have Docker, try the task on your machine before you publish it: ```bash hammerkit run sh -- 'ls && node --version' ``` Commit the file and get it onto `main`: the agent's commands run in the published version, and its sandbox is a checkout of `main`, so the two should not drift apart. Then publish it from the root of your checkout, signed in as the publisher's service account: ```bash HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \ --impersonate-service-account=hammerkit-publisher@my-project.iam.gserviceaccount.com" \ hammerkit remote publish hammerkit ``` ```text published to hammerkit: sha256:9f2c61d0… (1 file) ``` If this exact version was published before, it prints `already published to hammerkit:` with the same hash instead; that is fine. `HAMMERKIT_TOKEN_COMMAND` makes this one command sign in as `hammerkit-publisher`: `gcloud` prints a token of that service account for you. That only works once you may act as it, see [Service accounts on Google Cloud](/docs/guides/agents/google-accounts). Write your project instead of `my-project`; with the Kubernetes guide, the account is `$PUBLISHER` of its step 1. Copy the hash, `sha256:` and 64 hex digits: it names this version of the build file, and opencode uses it in step 5. The server never takes task definitions from the agent itself, so the agent can not change what its commands run in. When you change the build file later, publish it again and use the new hash. ## 4. Install opencode and the plugin [#4-install-opencode-and-the-plugin] opencode loads plugins from the `.opencode/plugins/` directory of the directory it starts in. Make a new, empty directory for the agent with that plugin. It stands in for the root of the sandbox; nothing else in it is read or changed. Do not use your project's checkout for this. ```bash npm install -g @opencode/cli mkdir -p shop-agent/.opencode/plugins && cd shop-agent cat > .opencode/package.json <<'EOF' { "dependencies": { "hammerkit": "2.0.0" } } EOF cat > .opencode/plugins/hammerkit.ts <<'EOF' export { default } from 'hammerkit/harness/opencode' EOF npm install --prefix .opencode npm ls --prefix .opencode hammerkit ``` * `.opencode/package.json` makes hammerkit a dependency of opencode's directory. Write the server's exact version, not a range: `^1` would install a hammerkit without the plugin. `npm ls` has to show that version. * `.opencode/plugins/hammerkit.ts` is the plugin file: it hands hammerkit's plugin to opencode as is. * `npm install --prefix .opencode` installs it. opencode 2.0 does not install the dependencies of plugins itself. ## 5. Tell the plugin where to work [#5-tell-the-plugin-where-to-work] The plugin reads its settings from environment variables. Put them in a file, so every session starts the same way: ```bash title="agent.env" export HAMMERKIT_SERVER=hammerkit # the name from step 2 export HAMMERKIT_REPO=github.com/acme/shop # the repository of the sandbox export HAMMERKIT_REF=main # the branch the sandbox starts from export HAMMERKIT_BUILD=sha256:9f2c61d0… # the hash from step 3 export HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \ --impersonate-service-account=hammerkit-agent@my-project.iam.gserviceaccount.com" export OPENROUTER_API_KEY=… # for the model, not for hammerkit ``` `HAMMERKIT_TOKEN_COMMAND` is how opencode signs in as the agent: a command that prints a short-lived token of the agent's service account. The plugin runs it again whenever the server stops accepting the token, so a long session needs no new sign-in. The command above works once you may act as `hammerkit-agent` (with the Kubernetes guide, `$AGENT` of its step 1), see [Service accounts on Google Cloud](/docs/guides/agents/google-accounts); check it there with [a token test](/docs/guides/agents/google-accounts#4-check-that-a-token-comes-back). With another identity provider, use any command that prints a token of the agent's account. ## 6. Check that the tools are hammerkit's [#6-check-that-the-tools-are-hammerkits] When a plugin does not load, opencode starts without it and says nothing. Its own tools then work on your machine instead of the sandbox. Do this check before you give opencode any work. ```bash source agent.env opencode service stop opencode run --standalone -m openrouter/anthropic/claude-haiku-5.5 \ 'List the files of the project root with the read tool.' ``` opencode 2.0 runs a background service that keeps the environment it was first started with. `opencode service stop` ends it, so the next start sees the variables of `agent.env`; do that whenever you change them. `--standalone` runs this one check without the service, and `-m` picks the model, so the check does not depend on your opencode's default. The answer has to list your repository's files. If it lists only `.opencode`, or says a file is not found, the plugin did not load: check that `npm install --prefix .opencode` ran and that the plugin file has the `export { default }` line. ## 7. Give it work and get it back [#7-give-it-work-and-get-it-back] Start opencode in the agent's directory and give it a task the way you would in your own checkout, telling it where to push: ```bash opencode ``` > The cart total skips the last item. Fix it, run the tests and push the fix to > `agent/fix-cart`. On its first tool call the plugin signs in and takes a sandbox of the agent at `main` that has no changes, or creates one. opencode edits the files of the sandbox, runs `npm ci` and `npm test` as the shell task on the server, and finally calls `push`. The server commits the changes as the agent and pushes them with its own GitHub credentials. Only branches the entitlement allows are accepted, here `agent/…`. The branch now shows up in your repository. Open a pull request from it and review it like any other. A sandbox stays on the server with whatever opencode changed in it, but the next start takes a clean one again. Have opencode push before you end a session, or set `HAMMERKIT_SANDBOX=` in `agent.env` when you know the id of the sandbox to come back to. To run one prompt without the terminal interface, for example in CI: ```bash BRANCH=agent/fix-cart-$(date +%Y%m%d%H%M%S) # a new branch for every run opencode run --auto -m openrouter/anthropic/claude-haiku-5.5 \ "The cart total skips the last item. Fix it, run the tests and push to $BRANCH." < /dev/null git -C "$CHECKOUT" ls-remote --exit-code origin "refs/heads/$BRANCH" ``` `--auto` approves the tool calls opencode would otherwise ask you about. opencode waits for input on stdin, so close it when nothing feeds it. It exits with 0 even when its tools failed and nothing was pushed, so the last line fails the job when the branch is not there. `$CHECKOUT` is a checkout of the repository that may read it, for example the CI job's own; a branch name of its own for every run keeps an earlier run's branch from passing the check. ## When it does not work [#when-it-does-not-work] | What you see | What it means | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | opencode lists only `.opencode`, or files of your machine | The plugin is not loaded, see [step 6](#6-check-that-the-tools-are-hammerkits). | | The old settings are still used after a change of `agent.env` | The background service kept them: run `opencode service stop`. | | `server hammerkit is not registered` | Step 2 was run as another user, or `HAMMERKIT_SERVER` names another server. | | `… is not the hash of a published version` | `HAMMERKIT_BUILD` holds a name or a shortened hash. Use the full hash that step 3 printed. | | `unable to reach http://…: fetch failed: connect ECONNREFUSED …` | Nothing listens at the server's address, for example `kubectl port-forward` has stopped. | | `unable to reach https://…: fetch failed: …, this machine does not trust the certificate of the server` | The server's certificate comes from a company or private CA: name that CA's file in `NODE_EXTRA_CA_CERTS`, which takes one file. | | `error: lost connection to pod`, `Unable to connect`, or an empty reply from the server | `kubectl port-forward` ended, because the server's pod was replaced or the connection dropped: start it again, or run it in the loop of the Kubernetes guide. | | `could not read the source credential …` | The server can not mint its token for the repository; the rest of the message says why. See [the check](/docs/guides/agents/kubernetes#2-deploy-the-server) after the deploy. | | `Permission 'iam.serviceAccounts.getAccessToken' denied` | You may not act as the service account in `HAMMERKIT_TOKEN_COMMAND` yet, see [Service accounts on Google Cloud](/docs/guides/agents/google-accounts#2-allow-yourself-to-act-as-them). | | `you are not entitled to keep sandboxes on this server` | The token is not the agent's, or its entitlement has no `sandboxes`: check the account in `HAMMERKIT_TOKEN_COMMAND`. | | The shell task needs `mounts: [.]` | The shell task does not mount the sandbox: add `mounts: [.]` (step 3), publish again and set the new hash. | | `npm: not found`, or a download that times out | The shell task's image lacks the tool, or its `network` does not allow the host. Change the task, publish again and set the new hash. | See [Agents](/docs/remote-compute/agents#opencode) for what each tool does on the sandbox. # pi on a sandbox (/docs/guides/agents/pi) Experimental. The tools and settings can still change between releases. At the end of this tutorial you start [pi](https://pi.dev) on your machine, give it a task for your project, and it works on a copy of your repository on a hammerkit server. Your own checkout stays untouched: pi never reads or writes a file on your machine and never runs a command there. When it is done, its work shows up as a branch in your repository, ready for a pull request. This page was run as written with pi 1.2.0 against the server of [Server on Kubernetes](/docs/guides/agents/kubernetes). ## How it fits together [#how-it-fits-together] Four pieces work together. You set up the last three in this tutorial. * **The server** keeps a **sandbox** for the agent: a checkout of your repository at a branch, for example `main`. Every file pi reads or edits is a file of the sandbox. * **Your build file** has a **shell task**. Every command pi runs, such as `npm test`, is handed to that task as its argument and runs on the server, in the task's image. The build file is **published** to the server once; the server answers with a hash that names exactly that version. * **The agent's identity** is a service account, not you. pi signs in to the server as that account, and the server only lets it start sandboxes from `main` and push to `agent/…` branches. * **pi** runs on your machine and talks to the language model as usual. hammerkit's pi extension swaps pi's tools (`read`, `write`, `edit`, `bash`, `ls`, `find`, `grep`) for ones that work on the sandbox, and adds a `push` tool. ## Before you start [#before-you-start] **A hammerkit server for your GitHub repository.** [Server on Kubernetes](/docs/guides/agents/kubernetes) sets one up on GKE: the server checks out and pushes to your repository with a GitHub App, and accepts Google sign-ins. If someone else runs the server, ask them for its address and the issuer of the tokens it accepts, for example `https://accounts.google.com`. **Service accounts for the agent and the publisher**, and the right to act as them. [Service accounts on Google Cloud](/docs/guides/agents/google-accounts) creates `hammerkit-agent` and `hammerkit-publisher`, gives you the role that lets `gcloud` print their tokens, and shows the entitlements the server needs for them. The Kubernetes guide does this in its step 1; with a server someone else runs, have them add you as in [Let your team act as them](/docs/guides/agents/google-accounts#3-let-your-team-act-as-them). **Node.js 24 or later** on your machine, for pi and the hammerkit CLI. **An API key of a model provider** that pi supports. The examples use OpenRouter. The examples below use the repository `github.com/acme/shop`, a server registered as `hammerkit`, and Google service accounts. Replace them with yours. ## 1. Install the hammerkit CLI [#1-install-the-hammerkit-cli] ```bash npm install -g hammerkit@2.0.0 # the server's version hammerkit --version ``` `hammerkit --version` has to print the server's version. hammerkit needs Node.js 24: on an older one npm warns `EBADENGINE`, and with a range instead of an exact version it quietly installs an old hammerkit without the agent clients. Use the same version as the server. The CLI registers the server in step 2 and publishes the build file in step 3; pi's extension in step 4 comes from the same package. ## 2. Register the server on your machine [#2-register-the-server-on-your-machine] The CLI keeps a list of servers for your user, in its config file. Add the server under a short name of your choice, with the issuer of the tokens it accepts: ```bash hammerkit remote add hammerkit http://127.0.0.1:8480 --issuer https://accounts.google.com ``` `hammerkit` is the name you use from now on. The address here is the `kubectl port-forward` of the Kubernetes guide, which has to keep running while pi works. Plain `http` is only accepted for `127.0.0.1` and `localhost`; a server on any other address needs `https`. Register it as the same user that starts pi later: pi's extension reads this list. ## 3. Add a shell task to your project and publish it [#3-add-a-shell-task-to-your-project-and-publish-it] Open your project's `.hammerkit.yaml`, or create one in the root of the repository, and add a task for the agent's commands. For a Node.js project: ```yaml title=".hammerkit.yaml" tasks: sh: image: node:22-alpine mounts: [.] args: true cmds: [sh -c] network: { allow: [registry.npmjs.org] } ``` Line by line: * `sh` is the task's name. The extension looks for a task called `sh` unless you set `HAMMERKIT_SHELL`. * `image` is the container every command of the agent runs in. Pick one with the tools your project needs: the agent can only run `npm`, `go` or `python` if the image has them. * `mounts: [.]` puts the files of the sandbox into the container, writable. Without it, the commands would run in an empty directory. * `args: true` says the task takes [arguments](/docs/task/arguments). A server can be set to pass arguments to such tasks only, so mark the shell task even where nothing requires it. * `cmds: [sh -c]` is the trick: the agent's command is appended to the last command as one argument, so `npm test` runs as `sh -c 'npm test'`. * `network` lists what the commands may reach. The sandbox is a fresh checkout without `node_modules`, so the agent installs dependencies itself and needs the registry. The server can narrow this further, never widen it. Leave secrets out of this task: a task that reads a secret takes no arguments on a server. Your other tasks, such as `test` or `lint`, stay as they are, and pi can run them too. If you have Docker, try the task on your machine before you publish it: ```bash hammerkit run sh -- 'ls && node --version' ``` Commit the file and get it onto `main`: the agent's commands run in the published version, and its sandbox is a checkout of `main`, so the two should not drift apart. Then publish it from the root of your checkout, signed in as the publisher's service account: ```bash HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \ --impersonate-service-account=hammerkit-publisher@my-project.iam.gserviceaccount.com" \ hammerkit remote publish hammerkit ``` ```text published to hammerkit: sha256:9f2c61d0… (1 file) ``` If this exact version was published before, it prints `already published to hammerkit:` with the same hash instead; that is fine. `HAMMERKIT_TOKEN_COMMAND` makes this one command sign in as `hammerkit-publisher`: `gcloud` prints a token of that service account for you. That only works once you may act as it, see [Service accounts on Google Cloud](/docs/guides/agents/google-accounts). Write your project instead of `my-project`; with the Kubernetes guide, the account is `$PUBLISHER` of its step 1. Copy the hash, `sha256:` and 64 hex digits: it names this version of the build file, and pi uses it in step 5. The server never takes task definitions from the agent itself, so the agent can not change what its commands run in. When you change the build file later, publish it again and use the new hash; an agent that is already running keeps the old one. ## 4. Install pi and the extension [#4-install-pi-and-the-extension] ```bash npm install -g @earendil-works/pi-coding-agent pi install npm:hammerkit@2.0.0 # the server's version pi list ``` `pi install` adds hammerkit as a [pi package](https://pi.dev/packages) for every pi session of your user. `pi list` shows it under the user packages. ## 5. Tell the extension where to work [#5-tell-the-extension-where-to-work] The extension reads its settings from environment variables. Put them in a file, so every session starts the same way: ```bash title="agent.env" export HAMMERKIT_SERVER=hammerkit # the name from step 2 export HAMMERKIT_REPO=github.com/acme/shop # the repository of the sandbox export HAMMERKIT_REF=main # the branch the sandbox starts from export HAMMERKIT_BUILD=sha256:9f2c61d0… # the hash from step 3 export HAMMERKIT_TOKEN_COMMAND="gcloud auth print-identity-token --audiences=hammerkit --include-email \ --impersonate-service-account=hammerkit-agent@my-project.iam.gserviceaccount.com" export OPENROUTER_API_KEY=… # for the model, not for hammerkit ``` `HAMMERKIT_TOKEN_COMMAND` is how pi signs in as the agent: a command that prints a short-lived token of the agent's service account. The extension runs it again whenever the server stops accepting the token, so a long session needs no new sign-in. The command above works once you may act as `hammerkit-agent` (with the Kubernetes guide, `$AGENT` of its step 1), see [Service accounts on Google Cloud](/docs/guides/agents/google-accounts); check it there with [a token test](/docs/guides/agents/google-accounts#4-check-that-a-token-comes-back). With another identity provider, use any command that prints a token of the agent's account. ## 6. Start pi and check where it works [#6-start-pi-and-check-where-it-works] Start pi in a new, empty directory. pi treats its working directory as the root of the sandbox, so this directory only stands in for it: nothing in it is read or changed. ```bash mkdir shop-agent && cd shop-agent source ../agent.env pi --provider openrouter --model anthropic/claude-haiku-5.5 ``` Ask pi first: "List the files of the project." The answer has to show your repository's files, not an empty directory. On this first tool call the extension signs in, takes a sandbox of the agent at `main` that has no changes (or creates one) and shows its id, for example `sbx_4vJ0mT2cX9qLw1aB`. Every tool works on that sandbox from then on. A sandbox stays on the server until it is deleted, with whatever pi changed in it. To come back to unfinished work in a later session, add `export HAMMERKIT_SANDBOX=` to `agent.env`; without it, pi starts from a clean checkout of `main` again. ## 7. Give it work and get it back [#7-give-it-work-and-get-it-back] Give pi a task the way you would in your own checkout, and tell it where to push, for example: > The cart total skips the last item. Fix it, run the tests and push the fix to > `agent/fix-cart`. pi edits the files of the sandbox, runs `npm ci` and `npm test` as the shell task on the server, and finally calls `push`. The server commits the changes as the agent and pushes them with its own GitHub credentials. Only branches the entitlement allows are accepted, here `agent/…`; for any other branch, pi is told which ones it may push to. The branch now shows up in your repository. Open a pull request from it and review it like any other. ## Without a terminal [#without-a-terminal] `-p` runs one prompt and exits, for example in CI. pi waits for input on stdin, so close it when nothing feeds it: ```bash BRANCH=agent/fix-cart-$(date +%Y%m%d%H%M%S) # a new branch for every run pi --provider openrouter --model anthropic/claude-haiku-5.5 --no-session \ -p "The cart total skips the last item. Fix it, run the tests and push to $BRANCH." < /dev/null git -C "$CHECKOUT" ls-remote --exit-code origin "refs/heads/$BRANCH" ``` pi can end without pushing, for example when its tools failed, so the last line fails the job when the branch is not there. `$CHECKOUT` is a checkout of the repository that may read it, for example the CI job's own; a branch name of its own for every run keeps an earlier run's branch from passing the check. `--mode json` prints every tool call and result as a line of JSON. ## When it does not work [#when-it-does-not-work] | What you see | What it means | | ------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | pi lists an empty directory, or files of your machine | The extension is not loaded: check `pi list`. | | `server hammerkit is not registered` | Step 2 was run as another user, or `HAMMERKIT_SERVER` names another server. | | `… is not the hash of a published version` | `HAMMERKIT_BUILD` holds a name or a shortened hash. Use the full hash that step 3 printed. | | `unable to reach http://…: fetch failed: connect ECONNREFUSED …` | Nothing listens at the server's address, for example `kubectl port-forward` has stopped. | | `unable to reach https://…: fetch failed: …, this machine does not trust the certificate of the server` | The server's certificate comes from a company or private CA: name that CA's file in `NODE_EXTRA_CA_CERTS`, which takes one file. | | `error: lost connection to pod`, `Unable to connect`, or an empty reply from the server | `kubectl port-forward` ended, because the server's pod was replaced or the connection dropped: start it again, or run it in the loop of the Kubernetes guide. | | `could not read the source credential …` | The server can not mint its token for the repository; the rest of the message says why. See [the check](/docs/guides/agents/kubernetes#2-deploy-the-server) after the deploy. | | `Permission 'iam.serviceAccounts.getAccessToken' denied` | You may not act as the service account in `HAMMERKIT_TOKEN_COMMAND` yet, see [Service accounts on Google Cloud](/docs/guides/agents/google-accounts#2-allow-yourself-to-act-as-them). | | `you are not entitled to keep sandboxes on this server` | The token is not the agent's, or its entitlement has no `sandboxes`: check the account in `HAMMERKIT_TOKEN_COMMAND`. | | The shell task needs `mounts: [.]` | The shell task does not mount the sandbox: add `mounts: [.]` (step 3), publish again and set the new hash. | | `npm: not found`, or a download that times out | The shell task's image lacks the tool, or its `network` does not allow the host. Change the task, publish again and set the new hash. | See [Agents](/docs/remote-compute/agents#pi) for what the extension does with each of pi's tools. # Checks that ran before CI (/docs/guides/agents/verified-checks) Experimental. The commands and settings can still change between releases. GitLab CI has not been tested against a real GitLab yet. [CI through the server](/docs/guides/agents/github-actions) moves the work of a CI job to the server, but the job still waits for it: a runner that does nothing for the minutes the tests take. A check task, such as `test` or `e2e`, does not have to run in CI at all when it already passed on the server for the same files: 1. **Before the push**, an agent runs the task as a *check* on its sandbox, or a person runs it on the server for the branch they pushed. The server notes the task, the [check key](/docs/remote-compute/checks#check-keys) of the files it read, and that it passed. 2. **The CI job** runs [`hammerkit remote verify test e2e`](/docs/cli/remote#verifying-checks) in its checkout. It computes the same keys over the checked-out files, asks the server, and passes when the server has a passed check for each. Anything else fails at once: the job never waits for a run and never starts one. A check counts for every commit with the same files, whoever ran it and where: a commit that only changes a README, an amended message or the squash commit on `main` is still verified. A change to a file a task reads is not. Deploys stay as in [CI through the server](/docs/guides/agents/github-actions): a deploy is something to do, not something to have done. This guide adds checks to a server set up with the [Kubernetes guide](/docs/guides/agents/kubernetes) and the CI guide. Each `server.yaml` block goes into the config of that guide's ConfigMap. ## 1. Tasks that can be checked [#1-tasks-that-can-be-checked] A check is only as complete as the `src` of its task: the key covers the files `src` names, of the task and of the tasks it depends on, and nothing else. A test that reads a fixture `src` leaves out would still be verified after the fixture changed. Name every folder the task reads: ```yaml title=".hammerkit.yaml" tasks: install: image: node:24-alpine src: [package.json, package-lock.json] generates: [node_modules] cmds: [npm ci] test: image: node:24-alpine deps: [install] src: [src, test, tsconfig.json] cmds: [npm test] ``` What the key covers, besides the files: * The task's definition: commands, image, variables, mounts and the definitions of its dependencies. A check of another version of the build file does not count when it changed the task, so `cmds: [true]` published on the side does not pass for the real `test`. It also means a check runs with the version of the build file the commit holds: when a change touches a check task, publish the new version first and check with it. * The services the task `needs`: their definitions, the files of their `src`, and the keys of the tasks they depend on. An `e2e` that needs an `api` service is not verified after the source of `api` changed. * What dependencies generate (`node_modules` above) only as far as the commit holds it: the key is computed on the checkout before anything ran in it, where those files are not there unless they are committed. They are made again from their own `src`, which is in the key. * Not `.git`, and not the run's arguments: a run with arguments is never a check. A task without `src`, or whose `src` matches no file, here, in a dependency or in a task a needed service depends on, can not be verified. `remote verify` says so and fails: `e2e: can not be verified, it, or a task it or a service it needs depends on, declares no src that matches a file`. The cache is less strict there: a task that resets the data of a service needs no `src` for the task needing the service to be cached. The CI runner and the server have to compute the same key. They do for tasks with an image. A task whose variables read the variables of the process (`$HOME`, `$CI`) can get another key in the CI job than on the server: `verify` then finds nothing and fails, it never passes by mistake. Keep check tasks to fixed values. ## 2. Let the CI job read checks [#2-let-the-ci-job-read-checks] The CI job only needs to read what passed. `checks: read` allows that for the repositories of `repos`, and nothing else: no runs, no build files. ```yaml title="server.yaml" entitlements: # every job of the repository, pull requests too: read its checks - claims: iss: https://token.actions.githubusercontent.com match: { repository_id: '123456789' } repos: [github.com/acme/shop] checks: read ``` `builds: run` or more also lets a caller read the checks of its `repos`, so the entitlements of the CI guide already do. Who may check a sandbox needs `repos` for the repository as well, and a branch of it to push to: a check counts like a run from a branch they pushed. ```yaml title="server.yaml" entitlements: # the agent of the Kubernetes guide: sandboxes, runs, and checks of its repositories - subject: { iss: https://accounts.google.com, sub: '' } repos: [github.com/acme/shop] builds: run sandboxes: [{ repo: github.com/acme/shop, branches: { main: pull, 'agent/**': push } }] ``` How long a check is found after it passed is `runs.checksKeep`, 30 days by default. ## 3. Check before the push [#3-check-before-the-push] **An agent** on a sandbox calls its `check` tool with the task, `test`, after it is done with its changes and before it calls `push`. All three [clients](/docs/remote-compute/agents) have it. The tool answers whether the check passed: ``` check test passed (run run_Qx3k9ZpL2mW7vN4a): CI can verify it for a push of these files ``` The check does not run in the sandbox's directory but on a copy of the files a push would commit: what `.gitignore` leaves out is not there. In its sandbox an agent can write any ignored file, such as a `node_modules/.bin/vitest` that exits 0, and no key would notice it. In the copy the dependencies are made from their `src`, or taken from the cache of the repository's [ref runs](/docs/remote-compute/runs#source-files). The pushed commit then has exactly the files the check ran on. An edit after the check changes the key of every task that reads the file, so check again before pushing it. **A person** pushes their branch and starts the tasks on the server for it, without waiting: ```bash git push hammerkit remote run test e2e --detach ``` ``` run_Qx3k9ZpL2mW7vN4a on build: test, e2e from feature-x at 4be1a07c... follow it with: hammerkit remote logs run_Qx3k9ZpL2mW7vN4a --follow ``` A run from a branch, without arguments, is a check: what it passes counts. Open the pull request once it passed, or re-run its CI job. ## 4. The CI job [#4-the-ci-job] ```yaml title=".github/workflows/checks.yaml" name: checks on: pull_request: push: { branches: [main] } permissions: contents: read id-token: write jobs: checks: runs-on: ubuntu-latest timeout-minutes: 5 env: HAMMERKIT_TOKEN_COMMAND: >- curl -sSf -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=https://hammerkit.example.com" | jq -r .value steps: - uses: actions/checkout@v4 with: { ref: '${{ github.event.pull_request.head.sha || github.sha }}' } - run: npm install -g hammerkit@2.0.0 - run: | hammerkit remote add hammerkit https://hammerkit.example.com --issuer https://token.actions.githubusercontent.com hammerkit remote use hammerkit - run: hammerkit remote verify test e2e --repo github.com/acme/shop ``` The checkout is of the pull request's head, the commit that was pushed, not the merge commit GitHub makes. Both usually have the same files for `test`, but the head is what was checked. `--repo` names the repository as the server knows it; left out, it is the `origin` of the checkout. Verify right after the checkout, before any other step writes files. A job that passes prints what passed, and when: ``` test: passed 2026-10-25T09:14:03.120Z in 1m12s, run_Qx3k9ZpL2mW7vN4a by agent-x, from a check of sandbox sbx_4vJ0mT2cX9qLw1aB, fetched at 4be1a07c1a2b e2e: passed 2026-10-25T09:15:40.002Z from the cache, run_8Lw2… by 1180…, from feature-x at 9e1f20aa31c4 ``` A job that does not, fails in the seconds it took to hash the files: ``` test: no run on hammerkit passed it for these files not verified: test; run on the server: hammerkit remote run test ``` Re-run the job after the check passed. The deploy job of the CI guide stays: it runs on `main` after the checks. ### GitLab CI [#gitlab-ci] ```yaml title=".gitlab-ci.yml" checks: image: node:24-alpine id_tokens: HAMMERKIT_TOKEN: { aud: https://hammerkit.example.com } script: - npm install -g hammerkit@2.0.0 - hammerkit remote add hammerkit https://hammerkit.example.com --issuer https://gitlab.com - hammerkit remote use hammerkit - hammerkit remote verify test e2e --repo gitlab.com/acme/shop ``` With an entitlement for GitLab's tokens, see [GitLab CI](/docs/remote-compute/authentication#gitlab-ci): ```yaml title="server.yaml" entitlements: - claims: { iss: https://gitlab.com, match: { project_id: '4711' } } repos: [gitlab.com/acme/shop] checks: read ``` ## What a check can not tell [#what-a-check-can-not-tell] * **Files `src` leaves out.** See [step 1](#1-tasks-that-can-be-checked). * **Who wrote the code.** A check says the tasks passed for these files, not that anybody reviewed them. Branch protection and reviews still decide what is merged. * **Flaky tests.** A test that passed once for the files counts, as it would in the cache. * **A yanked version.** A check that passed with a version of the build file that was [yanked](/docs/remote-compute/build-files) since does not count any more. # Caching strategy in CI (/docs/guides/ci-caching) Locally, hammerkit caches automatically: a task whose `src` is unchanged is skipped. The challenge in CI is that each run usually starts from a **fresh checkout** on a **fresh machine**, so there is no local cache to hit. This guide covers the two ways to carry results across CI runs and which to choose. Keep the cache method at the default **`checksum`** in CI. `modify-date` compares file timestamps, which `git clone` resets on every run — so a `modify-date` task never hits the cache on a fresh runner. See [caching](/docs/task/caching#checksum-vs-modify-date). ## Option A — a remote cache backend (recommended) [#option-a--a-remote-cache-backend-recommended] Point the built-in `default` cache at an S3-compatible bucket (or a container registry — see [caches](/docs/build-file/caches#registry)). Hammerkit then **pulls** a task's result before running it and **pushes** the result after — no cache scripting in your pipeline at all. ```yaml title=".hammerkit.yaml" caches: default: method: checksum backend: type: s3 bucket: my-build-cache region: eu-central-1 tasks: build: image: node:24-alpine src: - src generates: - dist cmds: - npm run build ``` Credentials come from the standard AWS SDK chain (`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`), so set those as CI secrets. Backend errors never fail the build — an unreachable bucket just falls back to running the task. The same bucket works for developers locally, so a result built on a laptop is reused in CI and vice versa. See [caches](/docs/build-file/caches) for the full backend reference (MinIO, R2, GCS). ### Sharing across checkouts, runners and agent sandboxes [#sharing-across-checkouts-runners-and-agent-sandboxes] Cache ids are computed from paths relative to the project root (the git root), so the same commit produces the same ids wherever it is checked out — a CI runner at `/home/runner/work/app/app`, an agent sandbox at `/workspace/app` and a laptop all hit the same entries. Use **container tasks** for anything you want to share between macOS and Linux machines: a local task includes the host OS in its identity. ### Checks that don't need the outputs [#checks-that-dont-need-the-outputs] A job that only gates on a task passing doesn't need its outputs. Run it with `--no-restore`: a hit is then only looked up in the bucket, and nothing is downloaded unless a task that has to run depends on it. See [checking without restoring](/docs/cli/execute#checking-without-restoring). ```bash hammerkit run test --no-restore ``` This only saves the download when the task's own cache is the remote one. After a `cache pull` (below), the entry is already on the machine. ### Read-only runners [#read-only-runners] Only trusted runners should write to the shared cache. Give untrusted runners — coding-agent sandboxes, pull requests from forks — read-only credentials and run them in read-only mode, so they restore results but never push: ```bash export HAMMERKIT_CACHE_READ_ONLY=1 # or pass --cache-read-only to run / up hammerkit run test ``` Read-only mode skips every backend push, including the built-in local `default` cache. Results a read-only run builds are still reused by later runs in the same checkout. ### Splitting network from compute [#splitting-network-from-compute] Instead of letting every task talk to the bucket during the build, keep tasks on the machine-local `default` cache and move entries explicitly with [`cache pull` / `cache push`](/docs/cli/cache). The build itself then does no network I/O, and a workspace can be warmed before anyone runs anything: ```bash hammerkit cache pull --remote shared # network only hammerkit run # compute only hammerkit cache push --remote shared # trusted runners only ``` ## Option B — store / restore with the CI's own cache [#option-b--store--restore-with-the-cis-own-cache] If you'd rather use your CI provider's built-in caching, wrap the run with [`store` / `restore`](/docs/cli/store-restore): `restore` pulls the previous results out of a directory before the build, `store` writes them back after, and the CI caches that directory. ```bash hammerkit restore cache # before the build hammerkit build hammerkit store cache # after the build ``` A GitHub Actions job wiring that to `actions/cache`: ```yaml jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: '24' - run: npm i -g hammerkit - uses: actions/cache@v4 with: path: cache key: hammerkit-${{ hashFiles('**/package-lock.json') }} restore-keys: hammerkit- - run: hammerkit restore cache - run: hammerkit build - run: hammerkit store cache ``` ## Example: hammerkit's own CI [#example-hammerkits-own-ci] Hammerkit builds and tests itself this way. The repository's [`.hammerkit.yaml`](https://github.com/no0dles/hammerkit/blob/master/.hammerkit.yaml) declares `lint`, `build`, `test`, and the integration suite as `test:docker` and `test:kubernetes` (`test:integration` runs both) plus a `ci` cache on a GHCR repository, and each workflow job does the same three things: `cache pull`, `run`, `cache push`. The two integration tasks run as parallel jobs, so the Kubernetes cluster is only started for the one that needs it. A push whose tree was already checked or tested skips the work — including that cluster, which the workflow only starts when `hammerkit explain` reports a miss. Every cache step may fail without failing the job, so an unreachable registry just means a cold run. A test that is flaky is a cached pass once it passes, so a run can be forced to ignore the cache: start the workflow by hand, or put `[no-cache]` in the head commit message. That run neither pulls nor pushes. A failed task is never cached. ## Which one? [#which-one] | | Remote backend | store / restore | | -------------------------- | -------------------------- | ---------------------------------------------- | | Pipeline wiring | None — automatic pull/push | You add `restore`/`store` steps + a cache step | | Granularity | Per task | One directory for the whole run | | Shared with local dev | Yes (same bucket) | No (CI cache only) | | Needs a bucket or registry | Yes | No | A remote backend is the lower-maintenance choice and is shared with local development. If you already rely on your CI's cache and don't want a bucket, store/restore is fine. You rarely need both — **if a remote backend is configured, store/restore is redundant.** # Building container images (/docs/guides/container-images) This guide builds a Dockerfile in a task that runs on a [hammerkit server](/docs/remote-compute) and pushes the image to a registry. It sets up the server once, as its operator, and then writes the build file: a task with [`build`](/docs/task/image-builds), which is the same on a dev machine. The example pushes to Google's Artifact Registry from a server on Kubernetes; the other registries follow the same steps, the differences are named where they come up. A server on a [docker host](#on-a-docker-host) doesn't build images yet. ## Why a server needs a sandbox for it [#why-a-server-needs-a-sandbox-for-it] On a dev machine a task could mount the docker socket and run `docker build`. On a server that would hand the task the daemon of the machine, and with it every container of every caller on it, so the server refuses a mount of the socket. Docker in Docker needs a `privileged` container, which a server never starts. BuildKit without root needs what the server takes from every container: its seccomp profile, `allowPrivilegeEscalation: false` and the masked `/proc`. What works is BuildKit as root with one more capability, `SYS_ADMIN`, so it can mount the layers it builds. `SYS_ADMIN` is root's power over the kernel, and the kernel of a node runs the tasks of other callers too. So the server adds it only in a **sandboxed runtime**, which gives the container a kernel of its own: [gVisor](https://gvisor.dev) handles the container's system calls in a kernel it implements itself, [Kata Containers](https://katacontainers.io) start a small virtual machine per pod. `SYS_ADMIN` in there reaches the sandbox's kernel, not the node's. A task with `build` adds the capability, and the server's operator allows it in [`backend.capabilities`](/docs/remote-compute/backend#capabilities). This guide was tested on GKE Autopilot with gVisor (BuildKit v0.25.1): the build, a `RUN` step that downloads packages, and the push. ## What you need [#what-you-need] * A hammerkit server on Kubernetes, set up as in [the Kubernetes guide](/docs/guides/agents/kubernetes), and the rights of its operator on the cluster. * A registry to push to. Here, a Docker repository in Artifact Registry, and the right to grant roles on it. * The variables of the Kubernetes guide in your shell: `NAME` (the server's name and namespace), `PROJECT`, `POOL` and `AUDIENCE`. ## 1. A sandboxed runtime on the cluster [#1-a-sandboxed-runtime-on-the-cluster] The cluster needs a RuntimeClass that starts pods in a sandbox. Look for one first: ```bash kubectl get runtimeclass ``` | Cluster | The RuntimeClass | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GKE Autopilot | `gvisor`, there on every cluster ([GKE Sandbox](https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods)). Nothing to do. | | GKE Standard | `gvisor`, once a node pool has it: `gcloud container node-pools create sandbox --cluster --sandbox type=gvisor`. | | AKS | `kata-vm-isolation`, with [Pod Sandboxing](https://learn.microsoft.com/azure/aks/use-pod-sandboxing) on a node pool (`--workload-runtime KataVmIsolation`). | | EKS and your own nodes | Install gVisor's `runsc` and its containerd shim on the nodes ([gVisor's containerd guide](https://gvisor.dev/docs/user_guide/containerd/quick_start/)), then add a RuntimeClass with `handler: runsc`. | A RuntimeClass of your own looks like this: ```yaml title="runtimeclass.yaml" apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: { name: gvisor } handler: runsc ``` The rest of this guide calls it `gvisor`. Use the name your cluster has. ## 2. Let the server add `SYS_ADMIN` there [#2-let-the-server-add-sys_admin-there] Three changes, in this order, so no pod can add a capability outside the sandbox at any point. **The admission policies.** A namespace of a caller enforces the Pod Security level `baseline`, which refuses `SYS_ADMIN` even in a sandbox: it does not know about RuntimeClasses. With capabilities, the server makes the namespaces of callers enforce `privileged`, and an admission policy of yours refuses what `baseline` refuses, and any capability beyond it outside the RuntimeClass. A second one keeps the namespaces the server makes to its own. Copy both from [Capabilities](/docs/remote-compute/backend#capabilities) into `admission.yaml`, with your RuntimeClass in place of `gvisor`, `$NAME` in place of `hammerkit` in the server's ServiceAccount (`system:serviceaccount:$NAME:hammerkit`) and `^$NAME-` in place of `^hammerkit-`, and apply it. They need Kubernetes 1.30 or later: ```bash kubectl apply -f admission.yaml ``` **The namespaces the server made.** The server never changes the level of a namespace it made before (it has no right to), so set it on those of its prefix: ```bash kubectl get namespace -l hammerkit.dev/managed=true -o name | grep "^namespace/$NAME-" | \ xargs -r -I{} kubectl label {} pod-security.kubernetes.io/enforce=privileged \ pod-security.kubernetes.io/audit=baseline --overwrite ``` **The config.** In `server.yaml` (the ConfigMap `hammerkit-config`), under `backend`: ```yaml title="server.yaml" backend: type: kubernetes capabilities: allow: [SYS_ADMIN] runtimeClass: gvisor ``` Restart the server so it reads the config: ```bash kubectl -n $NAME rollout restart deploy/hammerkit ``` Only the pods of tasks that add a capability run in gVisor; every other task runs as before. Runs on [sandboxes](/docs/remote-compute/sandboxes) add none, unless you list them under `sandboxes.capabilities`. On GKE Standard and AKS only the sandbox's node pool has the runtime: the pods of these tasks go to the nodes of the first [pool](/docs/remote-compute/backend#pools), so its labels have to match that node pool, or a pod stays `Pending`. ## 3. An account that may push [#3-an-account-that-may-push] The task pushes with a token of a service account, never with a key or a person's login. The server mints the token: a Kubernetes ServiceAccount of its namespace, which Google trusts through the identity pool of the Kubernetes guide, and which has the right to push to one repository and nothing else. ```bash REGION=europe-west4 REPOSITORY=apps gcloud artifacts repositories create $REPOSITORY --repository-format docker \ --location $REGION --project $PROJECT # skip this if the repository exists # the identity that pushes: a ServiceAccount in the server's namespace, writer on the repository kubectl -n $NAME create serviceaccount registry-push gcloud artifacts repositories add-iam-policy-binding $REPOSITORY --location $REGION \ --project $PROJECT --role roles/artifactregistry.writer \ --member principal://iam.googleapis.com/$POOL/subject/system:serviceaccount:$NAME:registry-push # how gcloud signs in as it: the token the server puts in the Job, exchanged at Google's STS kubectl -n $NAME create secret generic registry-push-federation --from-literal=gcp.json="$(cat < Local-vs-container still applies in dev: a task with an `image` runs in a container (consistent tooling, recommended); drop the `image` to run on the host when you need a tool that's only installed locally. See [local vs. container tasks](/docs/guides/local-vs-container). # Local vs. container tasks (/docs/guides/local-vs-container) The single rule, restated: **a task with an `image` runs in a container; a task without one runs on your host.** Everything else — `src`, `generates`, `deps`, `needs`, caching — behaves the same either way. This guide is about *choosing* between them. ## Prefer containers [#prefer-containers] Containers are the recommended default, and the reason hammerkit exists: * **No local tooling.** The compiler, SDK, CLI — all come from the image. Nobody on the team installs Node, .NET, Helm or the AWS CLI; they just need a container engine. * **Repeatable everywhere.** The build behaves identically on every laptop and every CI runner, because it's the same image. No "works on my machine". * **Per-task images.** Each task can use a *different* image, so you don't maintain one giant build image with every tool in it. * **Isolation.** Only declared `src`/`generates`/`mounts` are visible, so side effects are contained. ```yaml tasks: build: image: node:24-alpine # tools come from here src: - src cmds: - npm run build ``` ## When a local task makes sense [#when-a-local-task-makes-sense] Drop the `image` to run on the host when: * the tool **can't** run in a container here — it needs host hardware or a platform SDK (Xcode on macOS, signing tools, a local GPU); * you're calling something every developer already has and containerizing adds no value; * you're driving the host's own container engine or `kubectl` and want host credentials/context. ```yaml tasks: package-ios: platform: os: macos cmds: - xcodebuild ... ``` A local task can declare a [`platform`](/docs/build-file/reference#task) so hammerkit only runs it where it's supported — useful in a CI matrix. Use [labels](/docs/labels) to route such tasks to the right runner (a macOS host for iOS, Linux for the rest). ## Mixing them [#mixing-them] You can freely mix: a local task can depend on a container task and vice versa. Sources and outputs of a container task's dependencies are mounted in automatically, so the chain stays consistent. A local task that `needs` a service gets the connection details as [environment hints](/docs/service#environment-hints-for-local-tasks), since it isn't on the container network. # CI migration guide (for coding agents) (/docs/guides/migrate-ci/agent-guide) You are migrating this repository's CI pipeline to [hammerkit](https://github.com/no0dles/hammerkit). Hammerkit runs build steps as **tasks** declared in a `.hammerkit.yaml` file, each in a container image, and skips a task when its declared inputs are unchanged. The CI then shrinks to installing hammerkit and running tasks; the same tasks run on a laptop. Two words carry the bar for this work: * **Parity** — the migrated pipeline runs the same checks and produces the same artifacts as the old one. Nothing the old pipeline verified goes unverified. * **Hermetic** — a task's result depends only on what it declares: its image, its commands, its `envs`, its `src` files and its `deps`. The cache trusts this; an undeclared input lets a stale result through without any error. Work through the steps in order. Each ends on a **done when** check; finish it before moving on. The reference sections after the steps hold the syntax and mappings you need along the way. ## Step 1 — Take stock of the current pipeline [#step-1--take-stock-of-the-current-pipeline] Read every CI definition in the repository and everything it calls: * `.github/workflows/*.yml`, `.gitlab-ci.yml`, `Jenkinsfile`, `.circleci/config.yml`, `azure-pipelines.yml`, `bitbucket-pipelines.yml`, `.buildkite/`, `.drone.yml`; * the scripts those run: `Makefile`, `package.json` scripts, `scripts/*.sh`, `docker-compose*.yml` used by tests. Build an inventory table with one row per job, recording: trigger and conditions (branches, paths, PR vs push, schedules), toolchain (runner image, `container:`, setup actions and their versions), each step's command, services (databases, queues), environment variables and secrets, caches, artifacts produced and consumed, matrix axes, job dependencies, timeouts, and steps allowed to fail. **Done when** every job and every step of every CI file appears in the inventory. ## Step 2 — Classify every step [#step-2--classify-every-step] Give each step exactly one class: | Class | Examples | Becomes | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | **Task** | install dependencies, lint, typecheck, unit/integration tests, build, e2e | a hammerkit task (step 3) | | **Toolchain** | `actions/setup-node`, `setup-java`, `container: node:20`, `apt-get install` of build tools | the task's `image` | | **Service** | `services: postgres`, `docker compose up db` for tests | a hammerkit `service` plus `needs` (a whole Compose stack: see [Compose test stacks](#compose-test-stacks)) | | **Plumbing** | checkout, `actions/cache`, cache keys, upload/download of artifacts *between jobs* | dropped: hammerkit caches per task and passes outputs along `deps` | | **CI-owned** | triggers, branch and path conditions, secret injection, permissions, publishing releases or packages, deployments, notifications, uploading artifacts for humans | stays in the CI config | **Done when** every inventory row has exactly one class, and every Plumbing step has a note on what replaces it. ## Step 3 — Write the build file [#step-3--write-the-build-file] Create `.hammerkit.yaml` at the repository root (in a monorepo, see [monorepo layout](/docs/guides/monorepo)). For every Task-class step group: 1. Make it a task named after what it does (`install`, `lint`, `test`, `build`, `e2e`). Put its commands in `cmds`, in order. 2. Give it an `image` carrying the toolchain the CI used, at the same version (`setup-node` with `node-version: 20` → `node:20-alpine`, or the Debian variant when the commands need glibc or bash). Use a local task (no `image`) only for steps that need the host itself: Xcode, signing tools, running `docker` or `kubectl` with host credentials. 3. Add `deps` from the data flow: a task depends on every task whose outputs it reads (`build` reads `node_modules`, so `deps: [install]`). 4. Add `needs` for every service its commands connect to, and declare the service. 5. Declare each environment variable its commands read in `envs`. Hammerkit passes only declared variables to a task. Secrets come from the CI environment as `NAME: $NAME`. 6. For every CI job, add an aggregate task with only `deps` listing what that job ran (`ci: { deps: [lint, test, build] }`), so each CI job runs one hammerkit command. A CI matrix becomes one task per variant, sharing a base through [`extend`](/docs/task/extending), aggregated by a task with `deps` on all variants. Pick images with these in mind: * **Tasks run as your user, not root.** On Linux a container task runs with the host's uid and `HOME=/tmp`, so `apt-get install` or `apk add` in `cmds` fails. Every tool the commands call (git, jq, python3, bash, a docker CLI) must already be in the image. When the old job installed tools first, use an image that has them (the `mcr.microsoft.com/devcontainers/*` images carry git, bash and python) or a tools task that downloads a pinned binary into its `generates`. * **One C library per dependency tree.** Native modules (`node_modules` with compiled addons, Python wheels) are built for the image that installed them. Every task that loads `install`'s output needs the same libc: all Alpine (musl) or all Debian (glibc), not a mix. **Done when** every Task-class step appears in exactly one task's `cmds`, and `hammerkit validate` reports no errors. ## Step 4 — Declare inputs and outputs [#step-4--declare-inputs-and-outputs] This step decides whether the cache is correct. For every task: * **`src`** — every file and folder its commands read: sources, configs (`tsconfig.json`, `eslint.config.js`, `.babelrc`), lockfiles, test fixtures, and `package.json` when scripts are run through it. Prefer folders (`src`) over narrowing globs when the command reads more than the glob would match. A task without `src` of its own runs on every invocation — fine for aggregate tasks, not for the work itself. * **`generates`** — everything a later task or the CI reads: `node_modules`, `dist`, coverage and test reports. Add `export: true` to outputs a CI step reads from the workspace (an artifact upload) or that a human inspects. * **Everything a command writes is an output**, even inside a source folder: tool state (`.yarn/install-state.gz`, `.eslintcache`), generated files a later command reads (generated API docs), reports. Declare the path in `generates`, or point the tool somewhere declared (`YARN_INSTALL_STATE_PATH`). A path that's in neither `src` nor `generates` doesn't exist in the container, and the task's user can't create it there: writing to it fails with `EACCES` or `ENOENT`. * **Monorepos have one `node_modules` per workspace.** `generates` takes paths, not globs: list `node_modules` and each `packages//node_modules` the install creates (`find . -name node_modules -prune -not -path './node_modules/*'` after a local install shows them). Then apply the rules in [hermetic tasks](#hermetic-tasks) below. **Done when**, for every task, you can name each file its commands read and it is covered by that task's `src` or by a dependency's `generates`. ## Step 5 — Rewrite the CI config [#step-5--rewrite-the-ci-config] Keep everything CI-owned from step 2. Replace the rest of each job with: 1. install a pinned hammerkit version, 2. log in to the registry used as remote cache (if you add one), 3. `hammerkit cache pull --remote shared`, 4. `hammerkit run `, 5. `hammerkit cache push --remote shared`, only on trusted refs (the default branch). Use the [GitHub Actions template](#ci-template-github-actions) below; for other providers keep the same sequence (see [installation](/docs/installation) for the container image and the GitLab example). Add `.hammerkit` to `.gitignore`. On GitLab: * Container tasks bind-mount the checkout, so the Docker daemon must see it at the same path. A shell runner with the host's Docker, or the Docker executor with a `docker:dind` service (the [installation](/docs/installation) example), both work. The Docker executor with the host's socket mounted in does not: the host has no `/builds/...`. * A shell runner reuses its working copy. Keep hammerkit's project state across jobs with `GIT_CLEAN_FLAGS: -ffdx -e .hammerkit`, and install the pinned binary under the runner's `$HOME`, which also keeps `~/.hammerkit` between jobs. * For a registry remote cache in GitLab's container registry, log in with the job token: `docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"`. **Done when** every CI job consists of CI-owned steps plus one hammerkit run, and no Plumbing step remains. ## Step 6 — Verify [#step-6--verify] Run these and check each expected result: | Command | Expected | | ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | `hammerkit validate` | no errors (warnings about descriptions are fine) | | `hammerkit ls` | every task from step 3, with the images, `src` and `generates` you meant | | `hammerkit run ci --dry-run` | the tasks in a sensible order, nothing missing | | `hammerkit run ci` | succeeds; the same tests run and pass as in the old pipeline | | `hammerkit run ci` again | `0 executed` in the summary: every task `cached`, or `skipped` because what needs it was cached | | change one source file, then `hammerkit run ci --dry-run` | only the tasks reading that file, and tasks depending on them, are misses | | `hammerkit run ci --cache none` | succeeds, and the outputs match the cached run | | `docker ps --filter label=hammerkit-type=service` after the runs | empty: no service outlived its run | A task that rebuilds without a change reads something that differs between runs; `hammerkit explain ` names the input. A step that fails only under hammerkit usually reads a file or variable it doesn't declare: fix the declaration, not the command. While you iterate: * Run with `--log live` to get plain line-by-line output that you can read back from a file. The interactive view redraws the terminal. * Every `$NAME` that a task in the run declares must be set, even if empty; tasks outside the run don't need theirs. For a variable the CI provider sets only on some pipelines (merge request ids, branch names), declare a default instead: `MR_IID: ${CI_MERGE_REQUEST_IID:-}`. * Stopping a run with Ctrl-C (or `SIGTERM`) removes its services and stops its local tasks. A run killed outright (`kill -9`) can't: `hammerkit clean` removes what it left behind. * Output volumes of every state you tried stay on the Docker host until `hammerkit clean`. On a small CI or workspace disk, watch `docker system df`. **Done when** every row matches, and the test and artifact list matches the old pipeline (parity). ## Step 7 — Report [#step-7--report] Write the migration report (as `MIGRATION.md` or the pull request description): * a table mapping each old CI job and step to its task, or to "kept in CI" or "dropped" with the reason; * the verification results from step 6; * everything that needs a human: secrets to configure, registry permissions for the remote cache, deployment steps left untouched, and any input you couldn't pin down. **Done when** the report covers every inventory row. *** ## Reference: CI concepts in hammerkit [#reference-ci-concepts-in-hammerkit] | CI concept | GitHub Actions / GitLab CI | Hammerkit | | ---------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | Job | `jobs.` / a job | a task, or an aggregate task with `deps` | | Steps | `run:` / `script:` | `cmds` | | Toolchain | `runs-on`, `container:`, `setup-*` actions / `image:` | `image` | | Job order | `needs:` / `stages`, `needs:` | `deps` | | Artifacts between jobs | `upload-artifact` + `download-artifact` / `artifacts` + `dependencies` | `generates` on one task, `deps` on the next | | Caches | `actions/cache` / `cache:` | none — per-task caching from `src`, shared via a [remote cache](/docs/build-file/caches) | | Service containers | `services:` | `services` + `needs`; reach them by service name | | Environment | `env:` / `variables:` | `envs` (build-file or task level) | | Secrets | `${{ secrets.X }}` / masked variables | CI exports `X`; the task declares [`secrets: [{ from: env:X, env: X }]`](/docs/task/secrets) — out of the cache key and masked in logs | | Matrix | `strategy.matrix` / `parallel:matrix` | one task per variant via `extend` | | Timeout | `timeout-minutes` / `timeout` | `timeout: 20m` | | Working directory | `working-directory` / `cd` | `{cmd, path}` in `cmds`, or a [referenced](/docs/build-file/references) build file per package | | Conditions, triggers | `on:`, `if:` / `rules:` | stay in CI; CI picks which task to run | | Allowed failure | `continue-on-error` / `allow_failure` | stays in CI as a separate job running that task | ## Reference: the build file [#reference-the-build-file] A complete example covering the common shapes — a dependency install, lint, tests against a database, a build whose output CI uploads, and e2e tests: ```yaml title=".hammerkit.yaml" envs: NODE_IMAGE: node:24-alpine POSTGRES_IMAGE: postgres:16-alpine # part of every task's key in this file (`validate` warns) DATABASE_URL: postgres://app:app@db:5432/app caches: shared: method: checksum backend: type: registry repository: ghcr.io/my-org/hammerkit-cache services: db: description: database for tests image: $POSTGRES_IMAGE envs: POSTGRES_USER: app POSTGRES_PASSWORD: app POSTGRES_DB: app healthcheck: cmd: pg_isready -U app tasks: install: description: install npm dependencies image: $NODE_IMAGE src: [package.json, package-lock.json] generates: [node_modules] cmds: [npm ci] lint: description: lint the sources image: $NODE_IMAGE deps: [install] src: [src, eslint.config.js, package.json] cmds: [npm run lint] test: description: unit and integration tests image: $NODE_IMAGE deps: [install] needs: [db] src: [src, test, tsconfig.json, package.json] cmds: [npm test] build: description: compile the app image: $NODE_IMAGE deps: [install] src: [src, tsconfig.json, package.json] generates: - path: dist export: true # CI uploads dist as an artifact cmds: [npm run build] e2e: description: end-to-end tests against the built app image: cypress/included:13.15.0 deps: [build] needs: [db] src: [cypress, cypress.config.ts] timeout: 20m cmds: [cypress run] ci: description: everything a pull request must pass deps: [lint, test, build] ``` How it behaves: * Commands run with `/bin/sh` in the image, in the build file's directory; relative paths work as on the host. Alpine images have no bash: use a Debian-based image and `shell: bash` for bash scripts. * A container task sees its `src`, its `mounts`, and the sources and outputs of its dependencies. Its outputs are what it declares in `generates`; other files it writes inside the container are discarded. `src` and `mounts` are mounted read-write, so a command writing into a source folder changes the checkout and the task's own inputs — keep outputs out of source folders. * A task reaches a service by its name on the service's container port (`db:5432`). Without a `healthcheck`, a task may start before the service accepts connections. * `$NAME` in `image`, `src`, `generates`, `mounts` and service images is replaced with the build-file or task `envs` value. In `cmds`, the shell expands it at run time from the declared `envs`. * A dependency whose dependants are all cache hits is skipped. Request a task by name (or use `--no-skip-deps`) when CI needs its outputs in the workspace. Every key, including references, includes, labels and caches, is listed in the [build file reference](/docs/build-file/reference). ## Hermetic tasks [#hermetic-tasks] The cache replays a task's first result for as long as its declared inputs stay the same. Keep each task's result a function of those inputs: * Every file a command reads is in `src` or in a dependency's `generates`. * `mounts` hold caches and tooling only (`~/.npm`, the docker socket): their contents are not part of the key. * Images are pinned to the CI's toolchain version, ideally by digest (`node:24-alpine@sha256:…`); a moving tag reuses results built with the old image. * Dependencies are installed from a lockfile in `src`, at pinned versions. * Build stamps — dates, `git describe`, `$GITHUB_SHA`, build numbers — live in a small final task with `cache: none` that reads the cached outputs. Inside a cached task they either change the key on every commit or go stale. * Credentials are declared only on the tasks that use them: env values are part of the key, so a rotated token rebuilds every task declaring it. * Publishing, deploying and anything with external side effects stays in CI, or runs as a task with `cache: none` on trusted refs only. The full list of what the cache can't see is in [caching limitations](/docs/task/caching#limitations-what-the-cache-cant-see). ## Compose test stacks [#compose-test-stacks] Integration and e2e suites often start a whole `docker-compose.yml`: a database, a queue, object storage, a stub of an external API, and the app built from the repository. Each compose service becomes a hammerkit service; the test runner becomes a task that `needs` them: | Compose | Hammerkit | | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `image:` | `image`, pinned by the digest of the multi-arch index (`docker buildx imagetools inspect`), so every machine pulls its native platform | | `environment:` | `envs` | | `command:` | `cmd` (replaces the image's `CMD`; split into words, quotes group them; with `shell: sh` it runs through the shell) | | `working_dir:`, or the image's `WORKDIR` when the command relies on it | `workdir` | | `volumes:` with a config file | `mounts` (`./config.json:/etc/app/config.json`) | | `volumes:` with data | `volumes`, or nothing: a test stack usually starts empty | | `depends_on: { condition: service_healthy }` | `needs` plus a `healthcheck` on the needed service | | `healthcheck: test:` | `healthcheck: cmd:` (`CMD-SHELL` checks need `shell: sh` on the service) | | `healthcheck: start_period`, `retries` | `healthcheck: timeout:` (default `20s`) | | `ports:` | only for reaching the service from your host; tasks use `name:containerPort` | | a `build:` of the app under test | a service on the toolchain image with `deps: [install, build]`, running the built app | | a one-shot setup container (`createbuckets`, `rs.initiate`, migrations) | a task named by the service's [`init`](/docs/service/container#init): runs after the healthcheck passed, and the service is ready for anything needing it only once it succeeded | Things that behave differently from Compose: * **A service's working directory is the project directory**, not the image's `WORKDIR`, so its `src` and dependency outputs are found by relative paths. An image whose command relies on its own directory needs `workdir: /app`. * **`cmd` and `healthcheck.cmd` are not run by a shell** unless the service sets `shell: sh`. Pipes, `&&` and variable expansion need it. * **Healthchecks start one second after the container, as the user of its main process,** and repeat about once a second. Use `127.0.0.1` in HTTP probes: `localhost` may resolve to `::1` while the server listens on IPv4 only. * **A healthcheck that hasn't passed after `timeout` (default `20s`) fails the run**, naming the check. Give slow starters (Elasticsearch, a JVM app) a longer `timeout`, and get each service healthy with `hammerkit up` first, watching its log. * **Setup goes into an `init` task, written to succeed when run again.** Services are recreated on every run (unless kept up with `hammerkit up --daemon`), so the init runs again too: `mc mb --ignore-existing`, "initiate the replica set unless it exists". * **A name resolves only for direct `needs`.** A container reaches the services it lists itself, not the ones those need. When a service hands out another service's address (an object store's presigned URLs, a callback URL), every consumer of that address needs that service too: the test task, and a renderer downloading from it. * **A service reaches itself by its name** (it is the container's hostname), so software that advertises its own address (a MongoDB replica-set member `mongodb:27017`, Kafka) advertises the service name. Run several targets against one stack instead of one stack per target. Where the old pipeline started a Compose project per tenant or test target, keep one database, queue and storage service and give each target its own database name, key prefix, vhost or bucket. Only the parts that really differ become one service per target (`api-a`, `api-b`), and the test tasks per target run in parallel because nothing orders them. For a fast loop while iterating, keep the stack up and narrow the run. `hammerkit up --daemon -f stack=e2e` starts the labelled services once, and a task that needs them reuses the running containers. A spec filter with a default (`SPEC: ${E2E_SPEC:-e2e/**/*.cy.ts}`) runs one test: `E2E_SPEC=e2e/login.cy.ts hammerkit run e2e-a`. Services that stay up keep their data, so reset the target's database, prefix or bucket at the start of the test task. The same services can replace a Compose file for local development: publish the ports the app on the host uses, give data a named `volume`, and start them with `hammerkit up --daemon`. Values only some machines have (a secret file path, a public origin) get a default (`${GEMINI_KEY_FILE:-/dev/null}`), so one build file serves laptops, workspaces and CI. ## CI template: GitHub Actions [#ci-template-github-actions] ```yaml title=".github/workflows/ci.yml" name: ci on: push: branches: [main] pull_request: jobs: ci: runs-on: ubuntu-latest permissions: contents: read packages: write # the remote cache lives in GHCR steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: '24' - run: npm install -g hammerkit@1.9.0 # pin the version - uses: docker/login-action@v3 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - run: hammerkit cache pull --remote shared - run: hammerkit run ci - run: hammerkit cache push --remote shared if: github.ref == 'refs/heads/main' # only trusted refs write - uses: actions/upload-artifact@v4 with: name: dist path: dist # exported by the build task ``` Notes: * Container tasks need a Docker daemon that can bind-mount the checkout. GitHub's Ubuntu runners have one. On other providers, mount the host's Docker socket; a Docker-in-Docker daemon must see the checkout at the same path. * Without a remote cache, drop the login, pull and push lines: the run still works, it just starts cold on every runner. See [caching strategy in CI](/docs/guides/ci-caching) for the alternatives. * Pull requests from forks get read-only credentials; set `HAMMERKIT_CACHE_READ_ONLY=1` there. Who should write to a shared cache is covered in [agents, workspaces and CI](/docs/guides/agents-and-ci#who-may-write-to-the-cache). ## Commands [#commands] | Command | Use | | ----------------------------------------------- | ---------------------------------------------------------------- | | `hammerkit run ` | run a task and its dependencies (`hammerkit ` is the same) | | `hammerkit run -f key=value` | run every task with a label (`-e` excludes) | | `hammerkit run --dry-run` | the plan with predicted cache hits and misses, running nothing | | `hammerkit explain ` | why a task is a cache hit or miss | | `hammerkit validate` | check the build file | | `hammerkit ls` | list tasks and services as hammerkit resolved them | | `hammerkit cache pull` / `push --remote ` | move cache entries for the current commit | | `hammerkit clean` | remove outputs and local cache state | `hammerkit run` exits non-zero when a task fails. In CI it logs line by line; `--summary-json` prints a machine-readable summary. # Migrate your CI with a coding agent (/docs/guides/migrate-ci) Moving an existing pipeline to hammerkit is mostly mechanical: every CI job becomes a task, every setup step becomes an image, every cache and artifact step goes away. That makes it a good job for a coding agent (Claude Code, Cursor, Copilot, …). The [CI migration guide](/docs/guides/migrate-ci/agent-guide) is written for the agent, not for you: it walks it through taking stock of your pipeline, writing the build file, rewriting the CI config and proving the result runs the same checks as before. ## Hand it to your agent [#hand-it-to-your-agent] Give your agent the repository and this prompt: ``` Migrate this repository's CI to hammerkit. Follow the guide at https://hammerkit.dev/llms.mdx/docs/guides/migrate-ci/agent-guide/content.md step by step, including its verification and its final report. ``` If your agent can't read URLs, copy [`website/content/docs/guides/migrate-ci/agent-guide.mdx`](https://github.com/no0dles/hammerkit/blob/master/website/content/docs/guides/migrate-ci/agent-guide.mdx) into your repository and point it at the file instead. You need [hammerkit](/docs/installation) and a running Docker daemon where the agent works, so it can run the build it writes. ## What you get back [#what-you-get-back] * a `.hammerkit.yaml` (plus included files in a monorepo) with one task per build, lint and test step; * your CI config rewritten to install hammerkit and run those tasks, keeping triggers, conditions, secrets and deployments where they were; * `.hammerkit` added to `.gitignore`; * a report: which job became which task, what was dropped and why, and what still needs you (secrets, registry permissions, deploy steps). ## Review it [#review-it] Check the report first, then: * **Same checks.** Every job of the old pipeline maps to a task or is listed as deliberately kept in CI. * **Every input declared.** For each task, the files its commands read are in its `src` — see [what the cache can't see](/docs/task/caching#limitations-what-the-cache-cant-see). This is where a migration can go wrong without anything failing. * **Pinned toolchain.** Images carry the versions your CI used, ideally pinned by digest. * **Runs twice.** `hammerkit run ci` twice in a row: the second run executes nothing. Then try it on a branch and compare the run with the old pipeline before switching over. # Monorepo layout (/docs/guides/monorepo) In a monorepo every package needs the same `install`/`build` steps, and packages depend on each other in a specific order. Hammerkit handles both with [includes](/docs/build-file/includes) (reuse a task *definition* in each package's directory) and [references](/docs/build-file/references) (wire up cross-package order). This guide mirrors the runnable [`examples/monorepo`](https://github.com/no0dles/hammerkit/tree/master/examples/monorepo). ## The layout [#the-layout] ``` .hammerkit.yaml # root: build everything build.npm.yaml # shared: how to install build.tsc.yaml # shared: how to compile projects/ a/.hammerkit.yaml # package a b/.hammerkit.yaml # package b (depends on a) ``` ## Shared templates [#shared-templates] Define each repeated step once. `build.npm.yaml` knows how to install; `build.tsc.yaml` knows how to compile and itself includes the npm install: ```yaml title="build.npm.yaml" tasks: install: image: node:24-alpine src: - package.json - package-lock.json generates: - node_modules cmds: - npm ci ``` ```yaml title="build.tsc.yaml" tasks: build: image: node:24-alpine deps: [npm:install] src: - tsconfig.json - src generates: - dist cmds: - node_modules/.bin/tsc -b includes: npm: ./build.npm.yaml ``` Because these are **includes**, when a package uses them the `install`/`build` commands run in *that package's* directory, against *its* `package.json` — exactly what you want. ## A package [#a-package] Each package's build file is tiny: it [extends](/docs/task/extending) the shared `tsc:build` template and adds its cross-package dependencies. Package `b` depends on package `a`, so it references `a` and adds `a:build` to its deps: ```yaml title="projects/a/.hammerkit.yaml" tasks: build: extend: tsc:build includes: tsc: ../../build.tsc.yaml ``` ```yaml title="projects/b/.hammerkit.yaml" tasks: build: deps: [a:build] extend: tsc:build references: a: ../a includes: tsc: ../../build.tsc.yaml ``` ## Building everything [#building-everything] The root build file references each package and depends on all of them: ```yaml title=".hammerkit.yaml" tasks: build: deps: - a:build - b:build references: a: projects/a b: projects/b ``` ```bash hammerkit build ``` Hammerkit builds the dependency graph across files, runs independent packages in parallel, builds `a` before `b`, and [caches](/docs/task/caching) each package separately — change one package and only it (and anything downstream) rebuilds. ## Scoping with labels [#scoping-with-labels] Add [labels](/docs/labels) to build or test a subset, e.g. label each package's tasks with `project: a` and run `hammerkit -f project=a`. Combine with [store / restore](/docs/cli/store-restore) to move outputs between CI jobs that each build part of the repo. # Services & networking (/docs/guides/services-networking) A [service](/docs/service) is a long-running dependency — a database, a queue, an API — that a task `needs`. This guide covers how a task actually connects to one, and how services start and stop. ## Lifecycle [#lifecycle] You don't start services by hand. When you run a task that `needs` a service, hammerkit: 1. starts the service container, 2. waits for its [healthcheck](/docs/service/container#healthcheck) to pass, 3. runs the task, and 4. stops the service once nothing else needs it. For development you can keep services up across many task runs with [`hammerkit up`](/docs/cli/up) (and `--daemon` to background them), stopping them later with [`hammerkit down`](/docs/cli/down). ## Connecting: container task vs. local task [#connecting-container-task-vs-local-task] How a task reaches a service depends on whether it runs in a container: * **Container task** (`image` set): shares a network with the service and reaches it by the **service name** as hostname, on the service's **container port**. You do **not** need to publish a port for this. * **Local task** (no `image`): isn't on that network, so hammerkit injects `HAMMERKIT__HOST` and `HAMMERKIT__PORT` env vars instead. The `HAMMERKIT_*` hints are **local-task only** — a container task won't get them, because it doesn't need them (it uses the service name directly). ## A pattern that works for both [#a-pattern-that-works-for-both] Put the connection string in a shared top-level `envs` block and reference it from the task. For a container task, point it at the service name; the same value reads naturally in your code: ```yaml title=".hammerkit.yaml" envs: DATABASE_URL: postgres://api@postgres:5432/api services: postgres: image: postgres:16-alpine envs: POSTGRES_USER: api POSTGRES_DB: api POSTGRES_HOST_AUTH_METHOD: trust healthcheck: cmd: "pg_isready -U api" ports: - 5432 tasks: api: image: node:24-alpine needs: [postgres] envs: DATABASE_URL: $DATABASE_URL cmds: - node index.js ``` For a **local** `api` task (no `image`), build the URL from the injected hints in your start script instead, e.g. `postgres://api@$HAMMERKIT_POSTGRES_HOST:$HAMMERKIT_POSTGRES_PORT/api`. ## Ports are for your host, not for tasks [#ports-are-for-your-host-not-for-tasks] `ports` publish a service to your **host** machine — handy for connecting a GUI client or `psql` to a running database during development. Tasks share the service's network and can reach every container port directly, so they don't need `ports` at all. hammerkit publishes them on `hammerkit up`, and during `hammerkit run` only when a local task needs the service. ## Readiness [#readiness] Always give a service a `healthcheck` when a task depends on it being usable, not just started. Hammerkit polls the `cmd` about once a second and only starts the dependent task once it exits `0`; there are no `interval`/`retries` knobs, and a check that hasn't passed within its `timeout` (default `20s`) fails the run. See [healthcheck timing](/docs/service/container#timing-and-failure). ## Persisting data [#persisting-data] A service container is recreated on each start, so its data is lost unless you attach a [volume](/docs/service/container#volumes). Use volumes to keep a database between runs, and [`store` / `restore`](/docs/cli/store-restore) to seed or back up that data. ## Sharing and Kubernetes [#sharing-and-kubernetes] Services can be [referenced/included](/docs/build-file/includes) like tasks, so one database definition is reused across projects. To forward a service from a real cluster instead of running it locally, see [Kubernetes services](/docs/service/kubernetes). # Introduction (/docs) Hammerkit runs your build steps as **tasks** declared in a `.hammerkit.yaml` file. Each task runs in the container image it names, declares what it reads and what it produces, and is skipped when nothing it reads has changed. ```yaml title=".hammerkit.yaml" tasks: install: image: node:24-alpine src: [package.json, package-lock.json] generates: [node_modules] cmds: [npm ci] build: image: node:24-alpine deps: [install] src: [src, tsconfig.json] generates: [dist] cmds: [npm run build] ``` `hammerkit run build` on your laptop is the same build CI runs. Its results can be shared through a cache, so work done once — anywhere — isn't repeated anywhere else. ## Why hammerkit [#why-hammerkit] Tools come from the task's image, not from what's installed on the machine. A task runs again only when its inputs change — locally and in CI. Agents, laptops and CI runners reuse each other's results through a remote cache. Point your coding agent at the guide and it rewrites your CI into tasks. Run a server on Kubernetes and let pi or opencode work on sandboxes there. # Installation (/docs/installation) Tasks that declare an `image` run in a container, so you also need a running container engine such as [Docker](https://docs.docker.com/get-docker/). Check it with `docker info`. ### Install hammerkit [#install-hammerkit] ```bash npm install --global hammerkit ``` ```bash brew tap no0dles/hammerkit brew install hammerkit ``` Run it without installing — always the latest release: ```bash npx hammerkit --version ``` Download the binary for your platform from the [latest release](https://github.com/no0dles/hammerkit/releases/latest) — Linux, macOS and Windows, x64 and arm64 — and put it on your `PATH`. No Node.js needed. ```bash docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ -v "$PWD:$PWD" -w "$PWD" no0dles/hammerkit --version ``` ### Describe a task [#describe-a-task] Create a `.hammerkit.yaml` next to your code. A task names the image it runs in, the files it reads (`src`) and its commands: ```yaml title=".hammerkit.yaml" tasks: hello: image: alpine:3.21 src: [hello.txt] cmds: - cat hello.txt ``` ```bash echo "It's hammer time" > hello.txt ``` ### Run it [#run-it] ```bash hammerkit run hello ``` ```txt task: hello - It's hammer time Summary: hello executed 2.3s 1 executed, 0 cached (0% cache hit), 2.3s total ``` ### Run it again [#run-it-again] Nothing in `src` changed, so hammerkit reuses the result instead of running the task: ```txt Summary: hello cached 0ms 0 executed, 1 cached (100% cache hit), 11ms total ``` Change `hello.txt` and it runs again. That's the whole idea — [caching](/docs/task/caching) explains what counts as an input. ## In CI [#in-ci] The [container image](https://hub.docker.com/r/no0dles/hammerkit) contains hammerkit and can be used with Docker-in-Docker (dind) — the usual approach for container builds on CI systems. The `no0dles/hammerkit-github-action` action installs hammerkit; it needs `actions/setup-node` first. ```yaml title=".github/workflows/ci.yml" jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: '24' - uses: no0dles/hammerkit-github-action@v1.3 - run: hammerkit build ``` ```yaml title=".gitlab-ci.yml" build: image: name: no0dles/hammerkit entrypoint: [''] # the image's entrypoint is hammerkit itself services: - docker:dind variables: DOCKER_HOST: tcp://docker:2375 # the docker:dind service DOCKER_TLS_CERTDIR: '' script: - hammerkit build ``` GitLab runs the job's script through the image's entrypoint, so it has to be cleared. Tasks run in the `docker:dind` service, which sees the checkout under `/builds` at the same path, so sources mount as they do locally. To reuse results between CI runs and machines, see [caching strategy in CI](/docs/guides/ci-caching). ## Next steps [#next-steps] # Labels (/docs/labels) A label is a key-value pair. Labels can be defined on tasks or on build files. Labeling your tasks lets you run multiple task groups or exclude specific tasks. ## Example use cases [#example-use-cases] The following examples show some of the ways labels can be used. ### Separating tasks by area [#separating-tasks-by-area] Labels can be useful to separate tasks, for example if you share one build file across different parts of your application, or in a bigger project or monorepo. Add a `project` label to your tasks to build projects independently with a single command: `hammerkit -f project=a` or `hammerkit -f project=b`. Label filters are always given as `key=value`. A bare `-f project` (key only) is rejected. Use `-f` (`--filter`) to keep only matching tasks and `-e` (`--exclude`) to drop them; both can be repeated or take multiple values. ```yaml tasks: build-a: deps: [install-a] labels: project: a cmds: - tsc -b ./project/a test-a: deps: [install-a] labels: project: a cmds: - jest build-b: deps: [install-b] labels: project: b cmds: - tsc -b ./project/b ``` ### Separate by platform [#separate-by-platform] Labels can be used to split your tasks across CI runners. For example, a CI environment might have two runners, one on macOS and one on Linux, and the macOS host should only run the iOS-related code. Add a `platform=ios` label to the tasks that require macOS, and configure your CI to run `hammerkit -f platform=ios` on the macOS host and `hammerkit -e platform=ios` on the Linux host. If tasks on the two platforms depend on each other, use [store / restore](/docs/cli/store-restore) to move generated outputs and cache state between the hosts. Both commands accept label filters too, for example `hammerkit store cache -e platform=ios`. ```yaml tasks: build-ios: labels: platform: ios cmds: - xcodebuild ... build-api: image: node cmds: - tsc -b publish-testflight: deps: [build-ios] labels: platform: ios cmds: - altool upload-app ... ``` ### Group by purpose [#group-by-purpose] Simplify your workflow by grouping tasks: run one command for multiple tasks that don't otherwise depend on each other. For example, use a `task` label for developing or releasing. `hammerkit -f task=dev` starts your API server together with the frontend development build. `hammerkit -f task=release` pushes your container image to Docker Hub and builds and uploads the frontend to the staging environment. ```yaml tasks: frontend: labels: task: dev cmds: - ng serve api: labels: task: dev cmds: - dotnet run publish-s3: labels: task: release cmds: - ng build - aws s3 cp dist/frontend s3://staging-web publish-api: labels: task: release cmds: - docker build ... - docker push ... ``` # Release 1.4.0 (/docs/release-blog/release-1.4.0) A short retrospective on the changes and findings ## Log Modes [#log-modes] The new log implementation supports three new modes. `interactive` will be the default for regular use, `live` is used if a CI environment is detected. The default can be changed with the `-l`, `--log` flag. ### Interactive [#interactive] Print the most recent log for each running task and always see an overview of the overall progress for all dependencies. Failed tasks get printed in full, after completion. ### Live [#live] Print all logs from all tasks instantly. Get feedback fast, but makes reading logs harder, if there are many things running in parallel. ### Grouped [#grouped] Print all logs after a task has finished. Feedback is delayed, but not interrupted by other tasks. ## Watch improvements [#watch-improvements] Before 1.4.0 a task could be marked as watchable with `watch: true`. When executing the task, the source files were watched for changes and the task got restarted on every file change event. With the new release the `watch` flag can be removed from the build file and instead be passed to the cli. Every task with source files can be watchable and therefore the flag is not needed anymore. The new implementation also detects changes on dependent tasks and restarts them if needed. This change makes it possible to watch multiple build steps and reduces the need for manual interaction. As a simple showcase, let's take a quick look at a node server application. It requires some npm packages to be installed. The package list is defined in the `package.json` file and installed with `npm install`. After installation the api can be started. If a new package gets added to the `package.json` it requires another `npm install` before restarting the api to take advantage of the new package. With the following build file all the mentioned steps are executed by running `hammerkit api --watch`. When the content of the package.json gets modified, it will first install the npm packages and then restart the api. Removing the need for any manual interaction. ```yaml title=".hammerkit.yaml" tasks: install: image: node:16.6.0-alpine mounts: - $PWD/.npm:/.npm - $PWD/.config:/.config cmds: - npm install generates: - node_modules src: - package.json - package-lock.json api: image: node:16.6.0-alpine deps: [install] ports: - 8080 src: - src cmds: - node src/index.js ``` ## Docker performance improvements [#docker-performance-improvements] The overall performance on file heavy tasks improved a lot by switching generated files from mounts to volumes. Tasks like `npm install` with several thousand files led to unresponsiveness in the Docker API and required restarts. Debugging and reading the docker-for-mac/windows [user manual](https://docs.docker.com/desktop/mac/#file-sharing) led to the conclusion **source files get mounted, generates use volumes**. The change improved the stability and execution time, but required some changes to the store/restore functionality. ## Moved cache location [#moved-cache-location] Hammerkit keeps track of all used source files and task definitions. This is used to identify, if a task can be skipped because the source has not changed and therefore the old output should still be the same. These cache files were located in the same directory as the build file in a `.hammerkit` subfolder. Those files needed to be excluded from git, build steps and IDEs, because they were usually located on the same level as the source code. To reduce those kind of issues, they were moved into the user directory. Removing the need for special treatments. ## Automated test runner for macOS and Linux [#automated-test-runner-for-macos-and-linux] To fully test hammerkit, a running docker daemon is required. For macOS and Linux there is now a GitHub Action that tests all `examples` in the repository for every merge request. Windows is hopefully coming soon as well, but even though there are Windows runners available on GitHub, the docker setup requires still some work. ## Update node to v16 and make use of the AbortController [#update-node-to-v16-and-make-use-of-the-abortcontroller] During testing hammerkit on Linux, some examples ended in an `unhandledPromise` error. After some investigation, the reason could be pinned down to the usage of the `Defer` class, that was used a lot in hammerkit. The Defer class was a wrapper, to reduce repeating code handling callback results. The issue was that the `Defer` class created a promise with the default constructor without adding a catch handler. Since the `Defer` pattern is not a recommended approach anymore (very well explained [here](https://stackoverflow.com/questions/34971078/how-to-replace-promise-defer-with-new-promise)) and in newer versions of node `unhandledPromise` result in process failures, the removal was a no-brainer. The `Defer` class was mainly used for callback functions and the cancellation logic of tasks. A Defer got resolved when the user pressed Ctrl-C and the task execution handler ended the task early when that promise got resolved. The `AbortController` was the perfect replacement and has just made it into the [Web Spec](https://dom.spec.whatwg.org/#interface-abortcontroller) and is available in node environments since [v15](https://nodejs.org/api/all.html#globals_class_abortcontroller). ## Adding best practices [#adding-best-practices] In the repository a new folder `best-practices` got added with tooling specific examples. The build files are intended to be project-independent and showcase how popular build tools should be used together with hammerkit. ## Next release [#next-release] The next release will focus on adding a service concept. Services will be a solution to allow long-running tasks like databases to be connected with the existing tasks. The goal will be to find an efficient way of spinning up and shutting them down, so they only run, when they are needed by a task actively. That will require also a concept of how to deal with readiness detection of services. # Release 1.5.0 (/docs/release-blog/release-1.5.0) A short retrospective on the changes and findings ## Services [#services] Services have been added to assist your tasks. They can be used to start up databases and other service applications and get attached to the task if needed. They are similar to tasks but usually are continuously running in the background.\ Executing tasks that require a service, will trigger a service to start. As soon as they are ready the task can execute. When there are no pending tasks left that need the service, they will be terminated. There are two types of services hammerkit supports: ### Container Services [#container-services] Similar to [docker-compose](https://docs.docker.com/compose/) but offer some further advantages: * Ensure services are ready before your tasks use them with healthchecks. * Automatic DNS resolution between container task and services * Dynamic start/termination of services depending on the need * Watch mode for automated restarts on file mounts changes ``` services: postgres: image: postgres healthcheck: cmd: "pg_isready -U postgres" ports: - 5432 tasks: api: image: node:alpine needs: [postgres] cmds: - node index.js ``` ### Kubernetes services [#kubernetes-services] Use `kubectl port-forward` to connect pods, services and deployments from Kubernetes to your task. Enabling easy integration with staging/development environments and offload the need to run everything on one machine. ``` services: postgres: context: staging ports: - 5432:5432 selector: type: deployment name: postgres tasks: api: image: node needs: [postgres] cmds: - node index.js ``` ## Labels [#labels] Labels got added to categorize tasks and services. Allowing to group or divide tasks from each other. The cli commands [exec](/docs/cli/execute), [store/restore](/docs/cli/store-restore), [clean](/docs/cli/clean), [validate](/docs/cli/validate) and [ls](/docs/cli/ls) support label arguments to limit the scope. The new `--filter, -f` option reduces the scope to tasks/services only matching the specified label. For example `hammerkit clean --filter project=a` will only run the clean command on tasks that have the label `project=a` or are a dependency of such a task. The other new `--exclude, -e` option works the opposite way and reduces the scope to task/services that do not have a matching label. For example `hammerkit -e project=a` will execute all tasks that do not have a label `project=a`. Further examples and use cases can be found [here](/docs/labels). ## CLI start performance [#cli-start-performance] The cli responsiveness has been improved by reading tasks and service definition as late as possible. Before the build file was parsed on startup including all references/includes. This was needed for the preparation of the cli and help page that displayed all available tasks. With the new redesign it's not needed until the moment of execution of a specific task or the new [ls command](/docs/cli/ls). # Release 1.6.0 (/docs/release-blog/release-1.6.0) A retro about the changes in 1.6.0. This release focuses on sharing work across machines: caches can now live in a remote backend, and the same build file can run on the local docker daemon or on a Kubernetes cluster. ## Remote caching with pluggable backends [#remote-caching-with-pluggable-backends] Until now the cache state lived only on the machine that ran a task. With 1.6.0 a cache can be backed by a remote **backend**, so a task built on one machine can be restored on another - perfect for sharing results between developers and CI. Caches are declared in a top-level `caches:` block with a method and a backend. Two backends are built in: `local` (a shared filesystem directory, used by the built-in `default` cache) and `s3` (AWS S3 and any S3-compatible service like MinIO, R2 or Wasabi). ```yaml caches: remote: method: checksum backend: type: s3 bucket: my-build-cache region: eu-central-1 tasks: build: cache: name: remote src: [src] generates: [dist] cmds: - tsc -b ``` Hammerkit pulls a result from the backend before a task runs if it is missing locally, and pushes it after a successful run. Backend errors never fail the build, they only produce a warning. More details in [caches](/docs/build-file/caches). ## Checksum is now the default everywhere [#checksum-is-now-the-default-everywhere] Until now the cache method defaulted to `modify-date` locally and only `checksum` in CI. With remote backends that split no longer makes sense: a result is shared between machines, and `modify-date` compares file modification times, which a `git clone` resets on every fresh checkout — so a developer's `modify-date` result would never line up with a CI runner's. The whole point of an S3 backend is to reuse work *across* machines, and only `checksum` (which compares file content) is stable across them. So `checksum` is now the default in **all** environments. The same comparison runs locally and in CI, so a result cached on one is correctly reused on the other. This is a behavior change, almost a breaking one: if you relied on the local `modify-date` default (for example to avoid hashing very large source folders), set it explicitly with `cache: modify-date` on the task or `--cache modify-date` on the command. See [caching](/docs/task/caching#checksum-vs-modify-date). ## Run on Kubernetes [#run-on-kubernetes] A new runtime abstraction lets the same build file run on the local docker daemon or on a Kubernetes cluster. Declare an `environments:` block and select it with `--env `. Tasks run as jobs, container services run as deployments, and a service healthcheck is turned into readiness and liveness probes. ```yaml environments: default: kubernetes: context: docker-desktop ``` ```bash hammerkit api --env default ``` The store and restore commands accept `--env` too, so results can be stored from or restored to a cluster. See [running on Kubernetes](/docs/task/kubernetes). ## Service improvements [#service-improvements] Services got more capable in this release: * Services can declare `deps` and `needs`, just like tasks. * Multiple services can run in parallel and be referenced across projects. * The new `up` and `down` commands start and stop services directly. * Local tasks receive the connection details of needed services as environment variables (`HAMMERKIT__HOST`, `HAMMERKIT__PORT`), since they are not part of the container network. For [Kubernetes services](/docs/service/kubernetes) the port-forwarding no longer requires the `kubectl` binary, and a `namespace` can be set per service. ## Exporting build outputs [#exporting-build-outputs] Generated files can be marked with `export: true` to copy them back into your workspace after a task ran, so other tools can pick them up. ```yaml tasks: build: image: node:alpine generates: - path: dist export: true cmds: - tsc -b ``` ## Reliability [#reliability] A few robustness improvements landed as well: build graphs that mix `deps` and `needs` in a cycle are now detected instead of deadlocking, local tasks use a pid file to prevent concurrent runs of the same task, and a crashing docker service is reported as a crash instead of terminating silently. # Release 1.7.0 (/docs/release-blog/release-1.7.0) A retrospective on 1.7.0. Release 1.6.0 made caches shareable through remote backends; this release makes that sharing actually work across machines, and makes it cheap. The goal: work done anywhere — on a laptop, in a coding agent's sandbox or on a CI runner — is never repeated anywhere else. ## Cache keys no longer depend on where you check out [#cache-keys-no-longer-depend-on-where-you-check-out] A task's cache key is a hash of its definition (commands, image, env, paths) and its inputs. Until now the paths in that hash were absolute. The same commit checked out at `/home/runner/work/app/app` on a CI runner and at `/workspace/app` in a sandbox therefore produced *different* keys, and a shared remote cache never hit between them. In 1.7.0 every path in the key is relative to the **project root** — the git root, or the directory of the main build file. The same commit now has the same keys on every machine. Machine-local state (Docker containers, staging directories) stays scoped to each checkout, so two worktrees on one machine share cached results but never step on each other's running containers. Every cache key changes once with this release, so the first run after upgrading is a full rebuild. ## Use the registry you already have [#use-the-registry-you-already-have] The new `registry` backend stores cache entries in any OCI container registry — GHCR, Docker Hub, ECR, GAR, Artifactory or a plain `registry:2`. There is no bucket to provision: use the registry you already push images to, with the credentials `docker login` already set up. ```yaml caches: shared: method: checksum backend: type: registry repository: ghcr.io/my-org/hammerkit-cache ``` Each entry is a small single-layer image tagged `-`. It talks to the registry directly (no Docker daemon or extra tools involved) and writes the manifest last, so a half-finished upload is never visible to anyone else. See [caches](/docs/build-file/caches#registry). ## `cache pull` and `cache push` [#cache-pull-and-cache-push] With 1.6.0, a task whose cache was remote talked to the bucket during the build. 1.7.0 adds a second way that keeps the network out of the build entirely: tasks cache to the machine-local default cache, and two commands move entries between that and a remote cache, without running anything. ```bash hammerkit cache pull --remote shared # warm the workspace: network only hammerkit run # build: compute only hammerkit cache push --remote shared # publish what was built ``` `cache pull` fetches exactly the entries the current commit will look up, including all dependencies. That makes it the fastest way to set up a new workspace. `cache push` publishes results even when they were never written to the local cache. Both are safe to re-run. See [cache pull / push](/docs/cli/cache). ## Don't rebuild what nothing needs [#dont-rebuild-what-nothing-needs] A cache hit used to save only the task itself. When CI pulled the result of an e2e job, the e2e task was skipped — but the app build it depends on still ran, because that build's own result wasn't on the machine. Nothing in the run used it. Dependencies now run only when a task that needs them has to run. A task pulled into the run purely as a dependency waits until something depending on it misses the cache; if everything depending on it is a hit, it is **skipped** and shows up as such in the build summary. The decision follows the real cache outcome rather than a prediction, so a restore that fails still gets its dependencies built. Tasks you name or select with labels always run, and `--no-skip-deps` restores the old "run the whole graph" behavior. See [dependencies of cached tasks](/docs/task/dependencies#dependencies-of-cached-tasks). ## Keep shared caches small [#keep-shared-caches-small] Once every CI run and every workspace pushes to a shared cache, it grows without bound. Caches can now declare a **retention** policy: ```yaml caches: default: method: checksum backend: type: local retention: maxAge: 30d maxSize: 20Gi keepPerTask: 3 ``` `hammerkit cache ls` shows what a cache holds, and `hammerkit cache prune` applies the policy (or one given on the command line, with `--dry-run` to preview). A local cache with a policy prunes itself after every successful run; remote caches are only pruned when you name them with `--remote`. The local cache tracks when each entry was last used, so what you still use stays. See [retention](/docs/build-file/caches#retention). ## Timeouts [#timeouts] A hanging test used to block a CI job until the provider killed it. Tasks can now declare a `timeout` (and `--timeout` sets a default for all of them): ```yaml tasks: e2e: image: cypress/included:13.15.0 timeout: 15m cmds: - cypress run ``` A task that runs longer fails with `timed out after 15m`, its process, container or Kubernetes job is cleaned up, and nothing is written to the cache. See [timeouts](/docs/task#timeouts). ## Agents, workspaces and CI share one cache [#agents-workspaces-and-ci-share-one-cache] Put together, these change how CI and coding agents can work together. An agent that ran the e2e suite on a commit in its sandbox pushes the result; CI pulls it and the e2e job completes as a cache hit in milliseconds. The database it needs isn't even started. A new workspace pulls the dependency install and the build instead of redoing them, so only what actually changed runs. We tried this on a small stand-in project (`npm install` → build → an e2e job needing Postgres) across three simulated machines and a local registry: | Scenario | Result | | -------------------------------------------------------- | -------------------------------------------------------- | | Agent sandbox, cold run + `cache push` | 3 tasks executed, 3 entries pushed | | CI runner (other machine, other path) after `cache pull` | 0 executed, 100% cache hits, database not started | | New agent workspace, `cache pull build` | install and build restored; only the edited e2e task ran | Who may write to a shared cache is a policy decision: a cache hit means CI trusts the result. The new **read-only mode** (`--cache-read-only` or `HAMMERKIT_CACHE_READ_ONLY=1`) lets a runner restore from the cache while never writing to it. That fits agent sandboxes you don't fully trust, or pull requests from forks. The [agents, workspaces and CI guide](/docs/guides/agents-and-ci) walks through the setup and the trade-offs. ## See why something ran [#see-why-something-ran] Caching is only useful if you can tell why it didn't hit. 1.7.0 adds: * **[`hammerkit explain`](/docs/cli/explain)** — for each task, whether the next run is a cache hit and which input caused a miss: a changed source file, a changed dependency, a changed command or image. `--json` for tooling. * **`run --explain`** — the miss cause printed inline for every task that rebuilds. * **`run --dry-run`** — the execution plan in order, with predicted hits and misses, without running anything. * **Build summary** — after every run, a line per task (executed or cached, and how long it took) plus the cache hit ratio; `--summary-json` for dashboards and agents. * **[`hammerkit graph`](/docs/cli/graph)** — the task and service graph as mermaid or dot. ## Fixes that mattered [#fixes-that-mattered] Running the new setup end to end surfaced several bugs, all fixed in this release: * **Container task outputs restored from a cache were lost.** They were written into a throwaway container instead of the task's volume, so the task reported a cache hit while its outputs were empty. This affected remote caches and `store`/`restore`. * **A cached task that needs a service failed the run.** The service was left waiting and then cancelled. Now it isn't started at all. * **A service that fails to start hung the run forever** (a missing image, a port already in use). The task waiting for it now fails right away. * **Tasks without `src` were cached after one run.** As documented, a task without `src` — and everything depending on it — now always runs, since hammerkit can't prove it's up to date. * **Cache hits after a change.** Before relying on a shared cache in production we went looking for false hits — a task reported as cached although an input it reads had changed — and found several: globs like `src/**/*.ts` ignored every subdirectory, a dependency's command or env could change without rebuilding the tasks depending on it, deleted outputs stayed "cached", binary sources were compared as text, and a `src` typo kept a task cached forever. All are fixed, and the cache contract is now guarded by an invariant suite (every observable change is a miss, every unobservable one a hit), a property test over random file trees and globs, and mutation testing. See the [changelog](/changelog#v1.7.0). * **Tasks on Kubernetes didn't wait for their jobs.** A task running as a Kubernetes job was reported as completed as soon as the job was created, and a failing job was never detected. Tasks now wait for the job, fail when it fails, and delete it when the run is cancelled or times out. * **S3 errors looked like cache misses.** An unreachable bucket or bad credentials are now reported as errors by `cache pull`/`push`; builds still degrade to a miss with a warning. ## Next release [#next-release] Next up are source exclusions (`!` entries in `src`, hidden from container tasks too), first-class secrets for tasks and services, and container runtime options such as `--shm-size` for browser-based tests. See the [roadmap](/docs/contribution/roadmap). # Release 2.0.0 (/docs/release-blog/release-2.0.0) Since 1.24 a hammerkit server runs the tasks of a published build file for people and CI, with the credentials of service accounts instead of anyone's own. 2.0.0 makes that server the place where coding agents work: they get a sandbox with the repository, run the same tasks you run, start a preview of their change behind your sign-in, and leave checks behind that CI can trust instead of running everything again. The server runs on one Docker host or on a Kubernetes cluster, and you can watch it from a dashboard. The version goes to 2.0 for one breaking change in build files: `continuous` is for services only. The [changelog](/changelog#v2.0.0) lists every change and what to do when you upgrade. Remote compute is experimental: its config and API can still change between releases. ## Sandboxes for coding agents [#sandboxes-for-coding-agents] An agent that writes code needs somewhere to run it. Giving it your laptop gives it your credentials; giving it a CI runner makes it wait for a pipeline. A [sandbox](/docs/remote-compute/sandboxes) is a copy of a repository on the server that an agent reads, searches and edits through the server's API, runs the build file's tasks on, and pushes to a branch it is allowed to push to. The tasks run as they would anywhere else, with the build file that was published, so an agent can't change a task's commands, image or the secrets it declares; what the task runs from the sandbox's files is the agent's work. The [agent clients](/docs/remote-compute/agents) bring this into the agent's own tools: `hammerkit remote mcp` for any MCP client, an extension for [pi](/docs/guides/agents/pi) and a plugin for [opencode](/docs/guides/agents/opencode). The guides start from nothing and end with an agent that runs your tests on your server. ## Previews behind your sign-in [#previews-behind-your-sign-in] A [service run](/docs/remote-compute/service-runs) starts the services of a build file on the server, with endpoints at their own addresses behind the server's sign-in, so a reviewer opens the app an agent just changed instead of reading its diff. A service run sleeps when nobody uses it and keeps its data while it sleeps. When the branch moves on, `hammerkit remote up` with the same name moves the run to the new commit in place: same addresses, same data, and only the services that changed restart. Runs hand their tasks [the repository, branch and commit](/docs/remote-compute/runs#executing-a-run) they run from, so a preview can deploy per branch while it reads a secret. ## Checks CI can trust [#checks-ci-can-trust] When an agent ran the tests of a commit on the server, CI shouldn't have to run them again. A check task that passed on the server is recorded by its cache key, and `hammerkit remote verify` in CI only asks the server whether the checks for the commit's files passed. The [verified checks guide](/docs/guides/agents/verified-checks) sets it up for GitHub Actions and GitLab CI. ## A Docker host or a Kubernetes cluster [#a-docker-host-or-a-kubernetes-cluster] A server runs its tasks on the Docker host it runs on, or as Jobs on a [Kubernetes backend](/docs/remote-compute/backend#kubernetes): a namespace per caller, claims kept per branch so the next run starts warm, network policies, and quotas that say why a pod waits. Several runs of one caller go side by side, and the tasks of a run start [longest chain first](/docs/task/dependencies#which-ready-task-runs-first), weighted by how long they took before. What a run may reach is the operator's call: per-task arguments, network ceilings for tasks that read secrets, hosts a provider may reach through the egress proxy, and Linux capabilities only in a sandboxed runtime such as gVisor. ## Image builds in a task [#image-builds-in-a-task] A task with [`build`](/docs/task/image-builds) builds a Dockerfile with BuildKit, the builder behind `docker build`: tags, push, a registry layer cache, logins from secrets and several platforms under one name. The same task builds on your laptop, in CI and on a Kubernetes server, where no task gets a Docker daemon. [Building container images](/docs/guides/container-images) sets a server up for it. ## Seeing how a server stands [#seeing-how-a-server-stands] `hammerkit remote status` tells you where your run is in the queue and why it waits, what counts against your limits, and how long your tasks usually take. A server with a sign-in also serves a [dashboard](/docs/remote-compute/status#dashboard) with runs, their graph, timeline and logs, build file versions and proposals to publish or reject, sandboxes, and a button to retry a run that failed. ## The breaking change: `continuous` is for services [#the-breaking-change-continuous-is-for-services] A process that keeps running belongs in a service, which hammerkit starts, health-checks and stops. In 2.0 a task with `continuous` is refused with a message that names it. Make it a container service with [`continuous: true`](/docs/service/container#continuous): its `cmds` become one `cmd`, and it keeps its `src` and `ports`. `up --watch` starts it once, and on a service run that moves to a new commit it gets the changed files and reloads. ## Upgrading [#upgrading] Every cache misses once after the upgrade, and `hammerkit down` now only stops services. Upgrade a server before its clients, since a 2.0 client sends a timeout a 1.26 server refuses. The [changelog](/changelog#v2.0.0) has the full list. # Accounts (/docs/remote-compute/accounts) A service account is a name the operator defines for one set of credentials, such as the test or the production key of a team. The credentials are files and variables of the operator's machine, they never reach the config: an account gives each secret provider command the environment that makes it act as that account. ```yaml title="server.yaml" secretProviders: gsm: command: [gcloud, secrets, versions, access, latest, '--secret={{ref}}'] accounts: op-payments-dev: description: Payments, test secrets providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/payments-dev.json } } } op-payments-prod: description: Payments, production providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/payments-prod.json } } } source: [ref] # only from reviewed code refs: [main, 'release/*'] # and only from these branches scm-read: # reads the repositories, no entitlement may name it description: Read-only access to source repositories providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/scm-read.json } } } ``` | Key | What it does | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `description` | Shown to callers in `GET /v1/accounts` and `remote accounts`, up to 200 characters. | | `providers` | Per secret provider, the `env` its command runs with when it reads a secret as this account. The provider has to be declared under `secretProviders`. | | `kubernetes` | On a [Kubernetes backend](/docs/remote-compute/backend#secrets-in-the-job), the credential the Job of a task gets to read the account's secrets: a token of a ServiceAccount, the files of a Secret, or both. | | `source` | Where the code of a run using the account may come from, see [Restricting the code](#restricting-the-code). | | `refs` | The branches or tags such a run may start from. | An account name starts with a letter and holds letters, digits, `-` and `_`. [Entitlements](/docs/remote-compute/entitlements) say who may `use` or `request` an account. An account a [source](/docs/remote-compute/sources) reads repositories with is the server's own: no entitlement may name it and no build file may read a secret as it. ## Secret providers [#secret-providers] `secretProviders` are the commands that read secrets, the same as the `secretProviders` of a build file, see [Secret providers](/docs/task/secret-providers): a `command` as a list of arguments, run without a shell, where `{{ref}}` is replaced by the reference of the secret, an optional static `env` and a `timeout` (`30s` by default). Only the operator declares them; a published build file can not. ## How a run gets its accounts [#how-a-run-gets-its-accounts] The secrets in a published build file say which account each is read as (`account:` on the secret), and the server derives from that which accounts a run uses: a task without secrets uses none and holds no credentials. Nothing in a run request, no default and no `account` on a task or service chooses one, so the person who asks for a run cannot pick the account they prefer. ```yaml title=".hammerkit.yaml" tasks: test: cmds: [npm test] # no secrets, no account deploy: secrets: - from: gsm:kube-token account: op-payments-prod env: KUBE_TOKEN cmds: [./deploy.sh] ``` ## Restricting the code [#restricting-the-code] An account can limit where the code of a run that uses it comes from. `source` lists the kinds a run may use, `upload` for files the caller sends from their working tree, `ref` for a git ref of a repository the server reads itself (see [Sources](/docs/remote-compute/sources)), and `sandbox` for a [sandbox](/docs/remote-compute/sandboxes) of the caller's. `refs` narrows `ref` to some branches or tags. These are the combinations: | `source` | `refs` | A run that uses the account may come from | | ---------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | left out | | an upload or any ref, not a sandbox. The default, for accounts without credentials that matter. | | `[upload, ref]` | | the same, written out. | | `[ref, sandbox]` | | any ref, or a sandbox. A sandbox holds changes nobody reviewed, so an account takes its runs only when it lists `sandbox`. | | `[ref]` | | any branch, tag or 40-digit commit of a repository the caller may name. | | `[ref]` | `[main, 'release/*']` | only those branches or tags. `*` matches within one level, so `release/*` takes `release/1.2` but not `release/1/2`; a last `**` takes every level below, so `release/**` takes both. A commit hash is refused, because it names no branch. | | `[upload]` | | an upload only. | `refs` is only accepted together with `source: [ref]`: an upload names no ref, so the server refuses a config that sets `refs` with `upload` in `source` or without `source`. The server does not execute runs from an upload yet (`501`), so today every run is from a ref, whatever `source` says. An account with `source: [upload]` can not be used until uploads are supported. Set `source: [ref]` on the accounts that hold production credentials now, so they stay limited to reviewed code once uploads arrive. ```yaml title="server.yaml" accounts: op-payments-prod: providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/payments-prod.json } } } source: [ref] refs: [main, 'release/*'] ``` A run that breaks either rule is refused with `403` when it is submitted. Approving a run or a build file does not approve the code its commands execute, so an account that matters should be limited to `source: [ref]` and the branches that are reviewed. ## Secrets on a server [#secrets-on-a-server] Because the account is chosen by the published file, a server applies stricter rules to secrets than a local run does: a provider secret names its account (`account:` is required, a task's `account:` and `default: true` are not used), `file:` secrets are refused, and an `env:` secret may only read a variable listed in `runs.env.allow`. A build file that declares `secretProviders` or `secretAccounts` is refused, and so is a project `.env` file read: the operator's providers and accounts are the only ones. # Agents (/docs/remote-compute/agents) Experimental. The tools and settings can still change between releases. A coding agent needs to read, edit and search files and to run commands. On a [sandbox](/docs/remote-compute/sandboxes) the files stay on the server and the commands run there, as published tasks in their containers. hammerkit connects agents to it three ways: | Client | What it gets | | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`hammerkit remote mcp`](#hammerkit-remote-mcp) | MCP tools for any client that speaks MCP: Claude Code, Claude Desktop, Cursor, a custom agent. | | [pi](#pi) | pi's own `read`, `write`, `edit`, `bash`, `ls`, `find` and `grep`, working on the sandbox, and `check` and `push`. | | [opencode](#opencode) | opencode's own `read`, `write`, `edit`, `patch`, `shell`, `grep` and `glob` (`apply_patch`, `bash` and `list` on opencode 1), working on the sandbox, `run_task`, `check`, `run_logs` and `push`. | pi and opencode keep the tools their models are tuned for, with the same names, parameters and output: only where they read, write and run moves. The MCP tools take the shapes models are trained on: numbered reads with an offset and limit, an exact string replace, an `apply_patch` body or a git diff, ripgrep's parameters and a shell command. ## The shell task [#the-shell-task] There is no command endpoint on the server: commands bypassing the published build file would bypass its image, network and accounts. A shell is a published task whose last command is a shell, and every command of the agent is its [argument](/docs/task/arguments): ```yaml title=".hammerkit.yaml" tasks: sh: image: node:24 mounts: [.] args: true cmds: [sh -c] network: { allow: [registry.npmjs.org] } timeout: 5m ``` `mounts: [.]` puts the files of the sandbox into the container, writable, as the task's working directory. `args: true` marks it as a task that takes arguments, which a server with [`runs.args: declared`](/docs/remote-compute/runs#arguments) requires. Without it a command in a container runs in an empty directory: the clients then refuse to run it and say the shell task needs `mounts: [.]`. The [`timeout`](/docs/task#timeouts) stops a command that hangs, such as a test that never ends, before it holds the sandbox for the server's own limit. The clients pass the agent's own timeout of a command too, and the command stops after the shorter of the two. `bash` with `npm test -- -t login` then runs `sh -c 'npm test -- -t login'` in that image, on the files of the sandbox. Name the task `sh` or tell the client its name. Use `bash -c` when the image has bash. A task that reads secrets takes no arguments on a server, so the shell holds no account. Its network is the lower of its own `network`, [`runs.network`](/docs/remote-compute/runs#config) and [`sandboxes.network`](/docs/remote-compute/sandboxes#config). ### Named tasks instead of a shell [#named-tasks-instead-of-a-shell] A shell runs whatever the model types, past the guardrails a repository puts into its named tasks: a `test` task with `--test-timeout=60000` does not help when the model runs `node --test` itself, and a test that hangs holds the sandbox. `run_task` runs a published task by its name, with arguments for its last command: ```yaml title=".hammerkit.yaml" tasks: test: description: run the unit tests, args are test files or --test-name-pattern image: node:24 mounts: [.] args: true timeout: 10m cmds: [npm ci, node --test --test-force-exit --test-timeout=60000] lint: description: lint the sources image: node:24 mounts: [.] args: false cmds: [npm ci, npx eslint .] ``` The description of `run_task` lists the tasks of the published version, so the model knows what it can run without guessing: each with its `description` (its last command when it has none) and whether a run of it takes arguments, as the server's runs see it. A task takes none when it says `args: false`, when it or anything in its run reads a secret or a variable of the server, and under [`runs.args: declared`](/docs/remote-compute/runs#arguments) when it does not say `args: true`. The server still checks every run. The clients read the list of the version `HAMMERKIT_BUILD` names when they start, and wait up to 15 seconds for it; without an answer the tools work, and the description says the list is missing. Set [`HAMMERKIT_SHELL`](#settings) to an empty value and the build file needs no shell task: `hammerkit remote mcp` has no `bash` then, and the opencode plugin no shell (on opencode 1 its `bash` refuses, see [opencode](#opencode)). pi has no `run_task` yet and its `bash` refuses without a shell, so keep a shell task for pi. Leaving the shell out is a setting of the client, which whoever starts the agent picks. To keep agents off a shell, publish a version without a shell task, or with `args: false` on it: `run_task` runs every task of the version, a shell task included, and the server runs only what the version has. What follows from running a command as a task, a shell command or a named one: * Commands run one at a time, and only the files are kept between them. A background process ends with its command, and `cd` or `export` in one command does not reach the next. On a host each command gets a fresh container. On [Kubernetes](/docs/remote-compute/sandboxes#on-kubernetes) the commands of a sandbox run in one container that stays, so a command starts in about a second instead of waiting for a pod: before and after each command every process in it is killed and `/tmp` and `/dev/shm` are emptied, but what a command writes elsewhere outside the sandbox, such as the home directory, can still be there for the next one. The clients run a command in a directory with `cd || exit 1` on a line before it. * The command reads no input and gets none of the agent's variables. * While a command runs, the sandbox is held: an edit in the meantime is `409`. Agents send tool calls in parallel, so each client takes the calls of its model one at a time, in the order the model made them: an edit sent along with `npm test` lands before the tests start. A command of another client still holds the sandbox, and an edit then is refused. * Agents on sandboxes of their own run their commands side by side, also when they sign in as one identity, up to [`runs.limits.running`](/docs/remote-compute/runs#limits-and-restarts) of that identity (2 by default); a server for more agents raises it. * The timeout an agent gives a command is sent as the run's [`maxTimeout`](/docs/task#capping-every-task): the command stops after the shorter of it and the shell task's own `timeout`, with `timed out after `. Each task of the run gets at most this long, so a dependency of the shell task counts on its own. On Kubernetes the first command of a sandbox also waits for its pod to start within that time. A `timeout` on the shell task bounds the commands that come without one, such as a test that never ends. * A command starts in the root of the sandbox. The paths of the agent's machine do not exist there, which the clients tell the model. * What it printed comes from the [log of the run](/docs/remote-compute/runs#logs). A long log is cut at `runs.maxLogBytes`; the last lines of the task are kept past it. ## Sign-in [#sign-in] The clients send the token of the agent, not of a person, to the server registered in the host config ([`hammerkit remote add`](/docs/cli/remote)), and nowhere else. They look for it in this order: | Source | What it is | | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `HAMMERKIT_TOKEN_COMMAND` | A command that prints a short-lived access token of the agent's own identity, for example a service account or workload identity of the identity provider the server accepts. It runs again when the server stops accepting the token, so a long session outlives one token. | | `HAMMERKIT_TOKEN` | Such a token, as the agent's platform hands it over. | | `remote add --token-command` | The same as `HAMMERKIT_TOKEN_COMMAND`, registered with one server in the host config, see [A token command per server](/docs/cli/remote#a-token-command-per-server). | | `hammerkit remote login ` | The sign-in stored on the machine, for an agent a person starts on their own machine. | Give the agent's identity an [entitlement](/docs/remote-compute/entitlements) with its repositories and branches under `sandboxes`, `builds: run`, and no account that holds production secrets. `[{ repo: github.com/corp/*, branches: { main: pull, "agent/**": push } }]` lets it start from `main` and push to `agent/**` with the [`push`](#taking-the-work-out) tool; with only `pull` it hands back patches. `repos: [github.com/corp/*]`, with a branch to push to, lets it [check](#checking-before-the-push) its work. ## Settings [#settings] All three clients take the same settings, `hammerkit remote mcp` as options too: | Variable | Option | What it names | | --------------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `HAMMERKIT_SERVER` | `--server` | The registered server. Default: the one `hammerkit remote use` set, else the only registered one. | | `HAMMERKIT_SANDBOX` | `--sandbox` | A sandbox of the agent, for example `sbx_4vJ0mT2cX9qLw1aB`. | | `HAMMERKIT_REPO`, `HAMMERKIT_REF` | `--repo`, `--ref` | Instead of a sandbox: one of this repository at this branch. When the client starts, it takes a sandbox of yours of the same repository and branch that has no changes and checks it out at the branch again, or creates one. A sandbox stays until it is deleted and the server allows only a few, so set `HAMMERKIT_SANDBOX` to the id the client prints to come back to work in progress. | | `HAMMERKIT_BUILD` | `--build` | The hash of the published version of the build file whose tasks run on the sandbox, as `hammerkit remote publish` printed it. A run names exactly the version it runs: when the build file changes, publish it again and set the new hash. | | `HAMMERKIT_SHELL` | `--shell` | Its shell task, `sh` by default. Empty for none: the clients then leave their shell tool out and run [named tasks](#named-tasks-instead-of-a-shell) only. | ## hammerkit remote mcp [#hammerkit-remote-mcp] [`hammerkit remote mcp`](/docs/cli/remote-mcp) is an MCP server on stdio for one sandbox. Clients start it as a command: ```json title=".mcp.json" { "mcpServers": { "hammerkit": { "command": "npx", "args": ["hammerkit", "remote", "mcp", "--server", "corp", "--build", "sha256:9f2c..."], "env": { "HAMMERKIT_SANDBOX": "sbx_4vJ0mT2cX9qLw1aB", "HAMMERKIT_TOKEN_COMMAND": "agent-token" } } } } ``` Paths are relative to the root of the sandbox. The tools: | Tool | Parameters | What it does | | ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `read` | `path`, `offset`, `limit` | The lines of a file, numbered like `cat -n`, in parts for long files. | | `write` | `path`, `content` | Writes a whole file, making its directories. | | `edit` | `path`, `old_string`, `new_string`, `replace_all` | Replaces a string that matches exactly once, or every match. | | `apply_patch` | `patch` | A `*** Begin Patch` body or a git diff; every file changes or none. | | `search` | `pattern`, `path`, `glob`, `type`, `-i`, `-A`, `-B`, `-C`, `fixed_strings`, `multiline`, `output_mode`, `head_limit` | ripgrep on the server. | | `glob` | `pattern`, `path` | The files that match a glob. | | `bash` | `command`, `workdir`, `timeout` (seconds) | Runs a command as the shell task and answers its output and exit code. A long output keeps its first 50 and last 200 lines, about 30 KB; `run_logs` reads the rest. Left out without a shell task. | | `run_task` | `task`, `args` | Runs another published task, such as `test`, and answers its [result](/docs/remote-compute/runs#results-and-waiting): status, exit code, why it was not taken from the cache, its last 20 lines. Its description [lists the tasks](#named-tasks-instead-of-a-shell). | | `run_logs` | `run`, `task`, `tail`, `from`, `limit`, `grep`, `ignore_case`, `context` | Reads more of the [log of a run](/docs/remote-compute/runs#logs). | | `changes` | `format`: `summary`, `diff` or `patch` | What changed since the sandbox was fetched. | | `check` | `task` | Runs a task as a check on the files a push would commit, see [checking before the push](#checking-before-the-push). | | `push` | `branch`, `message` | Commits the changes and pushes them to a branch, see [taking the work out](#taking-the-work-out). | | `services_up`, `endpoint_get`, `services_stop` | see [Service runs](#service-runs) | Start services for a person to look at, read a page of one, take them down. | Reads and searches are marked `readOnlyHint`, the edits `destructiveHint`, `push` `openWorldHint`; `bash` and `run_task` keep the defaults, since a command can do anything its task can. The server also tells the client which sandbox, repository and commit the tools work on. ## pi [#pi] hammerkit is a [pi package](https://pi.dev/packages): installing it adds the extension. ```bash pi install npm:hammerkit HAMMERKIT_SERVER=corp HAMMERKIT_BUILD=sha256:9f2c... HAMMERKIT_SANDBOX=sbx_4vJ0mT2cX9qLw1aB pi ``` The extension registers pi's own `read`, `write`, `edit`, `bash`, `ls`, `find` and `grep` with the sandbox as their backend, the way pi's SSH example does, and runs the user's `!` commands there too. pi's working directory stands for the root of the sandbox: a path below it is the same path in the sandbox, anything else is refused. The system prompt names the sandbox, so the model knows nothing is installed on the machine. `grep` searches with ripgrep on the server. Images are read from the sandbox like any file. ## opencode [#opencode] opencode lets a plugin replace a built-in tool by registering one of the same name. hammerkit's plugin works with opencode 2 and with opencode 1 from 1.18.29. Add hammerkit to the dependencies of the project's opencode directory, install them with `npm install` in that directory (opencode 2 does not), and re-export the plugin as the default export. Write the server's exact version: a range such as `^1` can install a hammerkit without the plugin. ```json title=".opencode/package.json" { "dependencies": { "hammerkit": "2.0.0" } } ``` ```ts title=".opencode/plugins/hammerkit.ts" export { default } from 'hammerkit/harness/opencode' ``` opencode 1 also needs `"@opencode-ai/plugin": "*"` in these dependencies, which it adds itself when it starts. Its plugin files that re-export `HammerkitPlugin` still work there; opencode 2 loads only the default export. Then start opencode with the [settings](#settings) as variables, as the user whose host config has the server (`hammerkit remote add`). opencode 2 runs a background service that keeps the variables it was started with: run `opencode service stop` after changing them, or `opencode run --standalone` to start with the current ones. | opencode 2 | opencode 1 | Parameters | | ---------- | ------------- | ----------------------------------------------------------------------------------------- | | `read` | `read` | `path` (`filePath` on opencode 1), `offset`, `limit`. A directory lists its entries. | | `write` | `write` | `path` (`filePath`), `content` | | `edit` | `edit` | `path` (`filePath`), `oldString`, `newString`, `replaceAll` | | `patch` | `apply_patch` | `patchText`, a `*** Begin Patch` body or a git diff; every file changes or none. | | `shell` | `bash` | `command`, `workdir`, `timeout` in milliseconds. There are no background commands. | | `glob` | `glob` | `pattern`, `path`, and on opencode 2 `limit` | | `grep` | `grep` | `pattern`, `path`, `include`, and on opencode 2 `literal`, `caseSensitive` and `limit` | | | `list` | `path`, `ignore` | | `run_task` | `run_task` | `task`, `args`, as the [MCP tool](#hammerkit-remote-mcp), with the same answer | | `run_logs` | `run_logs` | `run`, `task`, `tail`, `from`, `limit`, `grep`, `ignore_case`, `context`, as the MCP tool | | `check` | `check` | `task`, see [Checking before the push](#checking-before-the-push) | | `push` | `push` | `branch`, `message`, see [Taking the work out](#taking-the-work-out) | The tools answer in opencode's words, and opencode still picks the patch tool for GPT models and `edit` and `write` for the others. Paths are relative to opencode's directory or absolute below it, which stands for the root of the sandbox. A long shell output keeps its first and last lines, like the MCP tool, and names the `run_logs` call that reads the rest. `run_task` and `run_logs` take their turn with the other tools, in the order the model made the calls. On opencode 2 the plugin also removes the worktree tools, which would make worktrees on the machine opencode runs on. With an empty `HAMMERKIT_SHELL` the plugin removes opencode 2's `shell`. opencode 1 has no way to remove a tool, so its `bash` stays and refuses: its own would run on this machine. Turn it off with `"tools": { "bash": false }` in the `opencode.json` of opencode 1 so the model does not try it. opencode carries on without a plugin that fails to load, and without a word: its own tools then work on the machine opencode runs on. On opencode 1 hammerkit's plugin keeps that from happening when it can. It takes the `@opencode-ai/plugin` of the opencode directory when hammerkit is linked there (`npm install ../hammerkit`, `file:`) and does not see it, and when it finds none at all, its tools still replace opencode's and refuse with what is missing. A plugin file that does not load at all cannot do that: hammerkit not installed, or on opencode 2 a file without the default export. Check once that the tools are hammerkit's, in a directory with nothing but the opencode setup. The answer names the files of the sandbox, not the ones of your directory: ```shell opencode run --standalone 'List the files of the project root with the read tool.' ``` On opencode 1, leave out `--standalone`, or list them without a model: `opencode debug agent build --tool list`. Tools the plugin does not replace still work on the machine opencode runs on, such as `webfetch` and `skill`. Turn off the ones that touch local files and that your opencode version still has, for example `"tools": { "multiedit": false }` in the `opencode.json` of opencode 1. ## Checking before the push [#checking-before-the-push] Every client has a `check` tool with a `task`, such as `test`. It runs the task as a [check](/docs/remote-compute/checks#checks-of-a-sandbox): on a copy of the files a push of the sandbox would commit, not on the sandbox, whose ignored files the agent can write. When it passes, a CI job that runs [`remote verify`](/docs/cli/remote#verifying-checks) for the pushed commit passes without running the task, see [Checks that ran before CI](/docs/guides/agents/verified-checks). The tool answers whether it passed, and the last lines of a failure: ``` check test passed (run run_Qx3k9ZpL2mW7vN4a): CI can verify it for a push of these files ``` The tool tells the model to check right before `push` and to push without changing files in between. A check needs `repos` for the sandbox's repository in the agent's entitlement, as a run from a ref does. ## Taking the work out [#taking-the-work-out] Every client has a `push` tool with a `branch` and a commit `message`. The server commits the changes of the sandbox as the agent's identity and [pushes](/docs/remote-compute/sandboxes#pushing) them with the source's own credentials, only to a branch the entitlement gives `push`, such as `agent/**`; any other branch is refused with what the agent may push to. The sandbox is at the pushed commit afterwards, so the next push carries only new work. Without a branch to push to, the MCP tool `changes` with `format: patch` gives a patch to apply with `git am`. ## Service runs [#service-runs] Every client has three tools for [service runs](/docs/remote-compute/service-runs), to bring the app up for a person to check the change, or to try it out itself: | Tool | Parameters | What it does | | --------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `services_up` | `services`, `endpoints` (`service:port`, such as `web:8080`), `ttl`, `sleepTtl`, `name`, `branch` | Starts the services of the published build file with the files of the sandbox, or of a pushed `branch` of its repository, and answers once they are up: the address of each endpoint and until when they stay. The same `name` again replaces the run and keeps its addresses. | | `endpoint_get` | `run`, `path`, `service` | A `GET` of a path of an endpoint of the run, signed in as the agent: its status and the first 20000 characters of the body (at most 256 KiB of it is read, the rest is not downloaded). Only at the addresses the server gave the run, so the token goes nowhere else. | | `services_stop` | `run` | Takes the services down. | The agent's identity needs an `endpoints` level `publish` for the repository to name endpoints, and the people who look need `open`, see [Who opens an endpoint](/docs/remote-compute/service-runs#who-opens-an-endpoint). On a Kubernetes backend a service run starts from a branch only: `push` first, then `services_up` with that `branch`. ## What is not covered yet [#what-is-not-covered-yet] * Commands that read input, and long-running background commands. # Authentication (/docs/remote-compute/authentication) OAuth is the only way to call a server: there are no API keys and no passwords. The `issuers` of the config list the identity providers whose tokens are accepted, at least one. ```yaml title="server.yaml" issuers: - issuer: https://login.example.com audience: hammerkit groupsClaim: groups # default clientId: hammerkit-cli scope: openid api://hammerkit/.default - issuer: https://token.actions.githubusercontent.com # CI jobs, matched by claims audience: hammerkit ``` | Key | What it does | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `issuer` | The issuer, written exactly as the `iss` of its tokens (some providers end it with a slash). An https URL, http only for `localhost`. Its keys are fetched from its OpenID discovery document. | | `audience` | What the `aud` of a token has to contain: an API audience registered for the server. | | `groupsClaim` | The claim that lists the groups of a caller, `groups` by default. [Entitlements](/docs/remote-compute/entitlements) with `group` match on it. | | `clientId` | The public OAuth client that [`remote login`](/docs/cli/remote#signing-in) signs in with, by the device authorization grant. An issuer without one only takes tokens obtained elsewhere, such as a CI job's or an agent's own, or a service account's through [`remote add --token-command`](/docs/cli/remote#a-token-command-per-server). | | `scope` | What `remote login` asks for, `openid` by default. Add `profile email` for the dashboard to name people by their token's `name` (else `preferred_username`, else `email`) instead of its subject, which on Google is a number. | The server publishes the issuers with their client, audience and scope in its [discovery document](/docs/remote-compute#discovery), so `remote login` needs nothing configured beyond `remote add`. ## What is checked [#what-is-checked] Every call except discovery and the health check carries an OAuth access token as `Authorization: Bearer `. The server fetches the issuer's keys from its OpenID discovery document and accepts a token only when * it is signed with `RS256`, `RS384`, `RS512`, `PS256`, `ES256`, `ES384` or `ES512` (never `none` or a shared secret), * its `iss` is one of the configured issuers and its `aud` contains the configured audience, * it has not expired and is already valid (a minute of clock skew is allowed), and * it has a `sub`. Keys are cached for ten minutes, and fetched again, at most once a minute, when a token names a key the server does not know yet, so a rotation needs no restart. A refused token only gets `401 invalid token`; the reason goes to the server's log, with everything a caller sent quoted and stripped of control characters, so a caller can't forge log lines. Concurrent requests share one key fetch, a failing issuer is left alone for ten seconds, and the previous keys keep working for up to an hour while the issuer is down. Two things the server does not check, so set them up on the identity provider: * `aud` has to be an API audience for the server, not the OAuth client's own id, or an ID token for that client would be accepted. The server does not look at the token type (`typ`) or the authorized party (`azp`). * A `claims` entitlement matches any string claim as it finds it. It does not know that `email` needs `email_verified`; match on `sub`, a group or a claim the provider controls. ## Identity providers [#identity-providers] Any OpenID Connect provider works when it issues **JWT access tokens** for an API audience of the server and publishes `/.well-known/openid-configuration` below the issuer, with an `issuer` that is exactly the configured one. `remote login` also needs the provider's device authorization endpoint and a public client. Some providers only hand out JWTs to workloads (CI jobs, service accounts) and opaque tokens to people; those are listed without `clientId`, and people sign in through another issuer of the same server. | Provider | People (`remote login`) | Agents and CI | | ----------------------------------------- | ---------------------------- | -------------------------------------- | | [Keycloak](#keycloak) | yes | clients with a service account | | [Microsoft Entra ID](#microsoft-entra-id) | yes | app registrations (client credentials) | | [Google](#google) | no, access tokens are opaque | identity tokens of service accounts | | [GitHub](#github-actions) | no, access tokens are opaque | GitHub Actions OIDC tokens | | [GitLab](#gitlab-ci) | no, access tokens are opaque | GitLab CI ID tokens | To let people with a GitHub or GitLab account sign in, broker that login through Keycloak or Entra ID; the server then sees the broker's tokens. ### Keycloak [#keycloak] ```yaml title="server.yaml" issuers: - issuer: https://keycloak.example.com/realms/corp audience: hammerkit clientId: hammerkit-cli entitlements: - group: { iss: https://keycloak.example.com/realms/corp, name: platform } builds: publish ``` * Create a public client `hammerkit-cli` with **OAuth 2.0 Device Authorization Grant** enabled. * Keycloak does not put an API audience into access tokens by default. Add a client scope with an **Audience** mapper that includes the custom audience `hammerkit` in the access token, and assign it to `hammerkit-cli` and to the agents' clients. * Add a **Group Membership** mapper with the token claim name `groups` and **Full group path** off, so groups arrive as `platform` and not `/platform`. * An agent is a confidential client with **Service accounts roles** on; it gets its token with the client credentials grant. Its `sub` is the id of the client's service account user, which a `subject` entitlement names, or put the service account user in a group. ### Microsoft Entra ID [#microsoft-entra-id] ```yaml title="server.yaml" issuers: - issuer: https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0 # your tenant id audience: 11111111-1111-1111-1111-111111111111 # client id of the API app clientId: 22222222-2222-2222-2222-222222222222 # client id of the CLI app scope: api://hammerkit/access groupsClaim: roles entitlements: - group: { iss: https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0, name: platform } builds: publish ``` * Register an API app: under **Expose an API** set the Application ID URI `api://hammerkit` and add a scope `access`. Set `requestedAccessTokenVersion` to `2` in its manifest: version 2 tokens carry the issuer above and the API's client id as `aud`. Version 1 tokens are issued by `https://sts.windows.net//` instead. * Register a CLI app with **Allow public client flows** on and the API permission `access`; its client id is `clientId`. * Use the tenant id in the issuer, not `common` or `organizations`: those have no single issuer. * The `groups` claim of Entra holds object ids, not names. Define **app roles** on the API app (such as `platform`), assign them to groups, users or agent apps, and read them with `groupsClaim: roles`. * An agent is an app registration that gets `api://hammerkit/.default` with the client credentials grant. Its `sub` is the object id of its service principal, and the app roles assigned to it are in `roles` too. ### Google [#google] ```yaml title="server.yaml" issuers: - issuer: https://accounts.google.com audience: hammerkit entitlements: - subject: { iss: https://accounts.google.com, sub: '104857382910563728194' } # unique id of the service account builds: run ``` Google's access tokens are opaque, so a Google issuer takes the identity tokens of service accounts, whose `aud` is whatever the caller asks for: ```bash # a workload on GCE, GKE or Cloud Run curl -s -H 'Metadata-Flavor: Google' \ 'http://metadata.google.internal/computeMetadata/v1/instance/service-account/identity?audience=hammerkit' # elsewhere, impersonating the service account gcloud auth print-identity-token --impersonate-service-account=agent@corp-ci.iam.gserviceaccount.com --audiences=hammerkit --include-email ``` The `sub` of such a token is the unique id of the service account, shown by `gcloud iam service-accounts describe`. There is no groups claim, so entitlements name each `subject`. With `--include-email` the token carries the service account's email, which the dashboard names its runs by; without it, it shows the `sub`, unless the entitlement gives the subject a `name` (see [Who](/docs/remote-compute/entitlements#who)). ### GitHub Actions [#github-actions] ```yaml title="server.yaml" issuers: - issuer: https://token.actions.githubusercontent.com audience: hammerkit entitlements: - claims: iss: https://token.actions.githubusercontent.com match: { repository: corp/payments, ref: refs/heads/main } builds: run ``` A job with `permissions: { id-token: write }` asks for a token with the audience of the server: ```bash curl -sS -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \ "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=hammerkit" | jq -r .value ``` Match on `repository`, `ref`, `environment` or `job_workflow_ref`. `repository_id` and `repository_owner_id` do not change when a repository is renamed or recreated under the same name, so they are the safer choice for a long-lived entitlement. ### GitLab CI [#gitlab-ci] ```yaml title="server.yaml" issuers: - issuer: https://gitlab.com # or the address of your GitLab audience: hammerkit entitlements: - claims: iss: https://gitlab.com match: { project_path: corp/payments, ref: main, ref_type: branch, ref_protected: 'true' } builds: run ``` A job declares the token it needs, and GitLab puts it in the variable: ```yaml title=".gitlab-ci.yml" deploy: id_tokens: HAMMERKIT_TOKEN: aud: hammerkit script: - curl -H "Authorization: Bearer $HAMMERKIT_TOKEN" https://build.example.com/v1/me ``` `ref_protected` is the string `'true'` on a protected branch or tag, so only pipelines that maintainers control match. `project_id` stays the same when a project is renamed. ## Transport [#transport] Plain http carries OAuth tokens, so `listen` on anything but the loopback interface needs `tls` (a certificate and key) or `behindTlsProxy: true` when a proxy in front terminates TLS. A config without either is refused at startup. ```yaml title="server.yaml" listen: { host: 0.0.0.0, port: 8480 } tls: cert: /etc/hammerkit/tls.crt key: /etc/hammerkit/tls.key ``` # Backend (/docs/remote-compute/backend) `backend` says where the tasks of runs execute. `host` is this machine, with the same engine as `hammerkit run`. `kubernetes` runs them as Jobs on a cluster, see [Kubernetes](#kubernetes). `runtimes` says which runtimes of the engine tasks may use. ```yaml title="server.yaml" backend: type: host platform: { os: macos, arch: arm64 } # detected when left out runtimes: [docker] # what tasks may run on, docker by default ``` | Key | What it does | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `backend.type` | `host` (this machine, the default) or `kubernetes` (a cluster). | | `backend.platform` | `host` only. What the machine offers: `os` is `linux`, `macos` or `windows`, `arch` is `amd64` or `arm64`. Detected when left out. Clients read it from the [discovery document](/docs/remote-compute#discovery). | | `runtimes` | What tasks may run on, see [Runtimes](#runtimes). | ## Runtimes [#runtimes] `runtimes` lists what tasks may run on: `docker` (containers, the default) and `local` (commands on this machine as the user of the server, which have to be allowed by name). ```yaml title="server.yaml" runtimes: [docker, local] ``` A build file with a task on a runtime that is not listed is refused when it is published or proposed, see [What a build file may not do](/docs/remote-compute/build-files#what-a-build-file-may-not-do). The server logs a warning at startup when `local` is allowed on a host. A container task uses the docker daemon of the server's environment, so name `DOCKER_HOST` in [`runs.env.allow`](/docs/remote-compute/runs#config) when it is not the default. On a cluster `local` means a task without an image runs in the `toolImage` of a pool, see [Pools](#pools). An [aggregate task](/docs/task/dependencies#aggregate-tasks), which only groups its dependencies, runs on no runtime: it is never refused for `runtimes`. ## One server is one trust level [#one-server-is-one-trust-level] On a host, the process of a run is not a sandbox. It runs as the user of the server, a `local` task reads everything that user can read, and a container task can use the docker daemon the server is configured for. Accounts keep credentials apart (a task only reads the secrets of the accounts its run was admitted for), but they do not keep code apart on one machine. Do not mix production and untrusted agents on one server: run them on different servers, each with the accounts of its own level, and with `local` only on a server whose users you trust with the machine. The storage directory and the other runs of a server are inside what a `local` task can read. The containers of a run with an open network are on a network of the run, not on Docker's default bridge, so they don't reach the containers of other runs or other callers. The volumes of a service are its caller's: the server puts a scope of the caller and the source in front of each name (`pg-data` becomes `hammerkit-ref--pg-data`), so a build file can't open, or with `resetOnChange` remove, the data of somebody else's services. A volume named like a path is refused, since docker would mount that path of the host. The outputs a service gets from the tasks it depends on keep their names, which come from the run's own files. Nothing removes the scoped volumes, and volumes of a server from before the scope are no run's any more: both are listed by `docker volume ls --filter label=app=hammerkit`. Each running run takes one Docker network, and each limited task or service one more, out of the daemon's address pools, which hold about 30 by default. A server with many runs at a time needs larger pools (`default-address-pools` in `daemon.json`). A daemon that runs Windows containers has no such networks: there the open containers of all runs stay on its default network, so it doesn't keep runs apart. What lies beyond the host is not hammerkit's to block on Docker: an open container reaches the LAN, the host's own services through the gateway of its network and the cloud's metadata address (`169.254.169.254`), which hands out the machine's credentials. So `open` means the internet only once the host's firewall says so. On a host that runs nothing but the server, rules like these do it (the run networks are `br-*` bridges, the egress proxies of `allow` items are on `docker0`). They need Docker's `iptables` firewall backend, which has the `DOCKER-USER` chain (not `firewall-backend: nftables`), and a daemon with IPv6 off, its default; with IPv6 on, the same rules are needed in `ip6tables` for `fe80::/10`, `fc00::/7` and `fd00:ec2::/32`. The script can run again, replacing what it set, and the rules stay when Docker restarts. Test them with an open task that has to fail `wget -T 3 http://169.254.169.254/` and still resolve names: ```sh title="firewall of the server's host" # the resolvers the containers use, the only private addresses they reach on port 53: like # Docker, read systemd-resolved's upstream list when /etc/resolv.conf names only its stub RESOLV_CONF=/etc/resolv.conf grep -q '^nameserver 127\.0\.0\.53' "$RESOLV_CONF" && RESOLV_CONF=/run/systemd/resolve/resolv.conf RESOLVERS=$(awk '/^nameserver/ && $2 !~ /^127\./ && !seen[$2]++ { print $2 }' "$RESOLV_CONF") iptables -N HAMMERKIT-RUNS 2>/dev/null || iptables -F HAMMERKIT-RUNS iptables -A HAMMERKIT-RUNS -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN for resolver in $RESOLVERS; do iptables -A HAMMERKIT-RUNS -d "$resolver" -p udp --dport 53 -j RETURN iptables -A HAMMERKIT-RUNS -d "$resolver" -p tcp --dport 53 -j RETURN done iptables -A HAMMERKIT-RUNS -d 10.20.0.15 -j RETURN # an internal registry runs may reach anyway for range in 169.254.0.0/16 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 100.64.0.0/10; do iptables -A HAMMERKIT-RUNS -d "$range" -j DROP done # leaving a network of containers for anywhere else; Docker keeps the networks apart itself for bridge in 'br-+' docker0; do iptables -C DOCKER-USER -i "$bridge" ! -o "$bridge" -j HAMMERKIT-RUNS 2>/dev/null || iptables -I DOCKER-USER -i "$bridge" ! -o "$bridge" -j HAMMERKIT-RUNS # the host itself, through the gateway of the network iptables -C INPUT -i "$bridge" -m conntrack ! --ctstate ESTABLISHED,RELATED -j DROP 2>/dev/null || iptables -I INPUT -i "$bridge" -m conntrack ! --ctstate ESTABLISHED,RELATED -j DROP done ``` Saved as `/usr/local/sbin/hammerkit-firewall.sh`, a unit puts the rules back after a reboot. Start the server after it (`After=hammerkit-firewall.service` in the server's unit), so no run starts before the rules are in: ```txt title="/etc/systemd/system/hammerkit-firewall.service" [Unit] Description=Firewall rules for the containers of hammerkit runs After=docker.service Requires=docker.service [Service] Type=oneshot RemainAfterExit=yes ExecStart=/bin/sh /usr/local/sbin/hammerkit-firewall.sh [Install] WantedBy=multi-user.target ``` `systemctl enable --now hammerkit-firewall` applies them now and on every boot. On a cluster every task is a pod of its own in the caller's namespace, without a token of the cluster. The boundary is the container, the namespace and the network policies the server puts in it (see [Namespaces](#namespaces)), which only hold on a network plugin that enforces `NetworkPolicy`. They keep an open pod off the metadata and private ranges by default, so there `open` means the internet without a firewall of your own (see `deniedEgress` in [Kubernetes](#kubernetes)). Keep the server off the nodes of the pools, so a task that escapes its container does not share a node with the server's token. The server still holds the right to mint the tokens of every account (see [Secrets in the Job](#secrets-in-the-job)), so the advice stands: one server, and one set of accounts, per trust level. ## Capabilities [#capabilities] A container task can add Linux capabilities (`capAdd`, see [Capabilities](/docs/task/container#capabilities)), such as `SYS_ADMIN` for a daemonless image build. A capability is root's power over the kernel the container runs on, which the runs of other callers share, so a server adds one only where its operator allows it, and then only in a sandbox with a kernel of its own: ```yaml title="server.yaml" backend: type: kubernetes capabilities: allow: [SYS_ADMIN] # what a build file may add runtimeClass: gvisor # the sandbox the pods of those tasks run in sandboxes: capabilities: [SYS_ADMIN] # runs on sandboxes add none unless named here ``` ```yaml title="server.yaml" backend: type: host capabilities: allow: [SYS_ADMIN] runtime: runsc # a runtime of the docker daemon: gVisor's ``` | Key | What it does | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `backend.capabilities.allow` | The capabilities a task may add. A build file with a task that adds another is refused when it is published or proposed (`400`); without `capabilities` the server allows none. | | `backend.capabilities.runtimeClass` | `kubernetes` only. The RuntimeClass the pods of tasks that add capabilities run in. | | `backend.capabilities.runtime` | `host` only. The docker runtime the containers of those tasks run in, as named in the daemon's `daemon.json`. | | `sandboxes.capabilities` | What a run on a [sandbox](/docs/remote-compute/sandboxes) may add, out of `allow`. None by default: a sandbox holds code nobody reviewed. | Only the containers of tasks that add capabilities run in the sandbox, the others run as before, and such a task never runs in the [runner of a sandbox](#sandboxes). The sandbox has to be one: hammerkit can't tell whether a RuntimeClass or a docker runtime keeps the capability from the host, so name only one that does, such as gVisor (runsc, which handles the container's system calls in a kernel of its own) or Kata Containers (a virtual machine per pod). gVisor is `gvisor` on GKE (GKE Sandbox, on by default on Autopilot) and installed on the nodes elsewhere; on a docker host `runsc install` adds it to the daemon. Kata is `kata-vm-isolation` with AKS Pod Sandboxing. A task can't ask for `privileged`, on any backend. On a docker host, such a task can't bind files of the run (`src`, file outputs, mounts) yet, see [build files](/docs/remote-compute/build-files): build images with it on a Kubernetes backend. The pod of such a task goes to the nodes of the first [pool](#pools), and on a sandbox to the node of the sandbox's files: those nodes need the runtime. Where only a node pool of its own has it (GKE Standard, AKS), name that pool's labels in the first pool, or make sure the RuntimeClass's `scheduling` sends the pod there. A pod no node can take stays `Pending`, and hammerkit doesn't fail it early yet. Pod Security `baseline` refuses these capabilities in a sandbox too, it does not know about RuntimeClasses. With `capabilities` the server makes the namespaces of new callers enforce `privileged` and audit `baseline`, and an admission policy of yours refuses in them what `baseline` refuses, and capabilities beyond `baseline` to every pod outside the sandbox. The server never changes the level of a namespace it made before: that would take `patch` on every namespace of the cluster. Until you relabel one, the tasks that add capabilities fail in it and say so, the others run. The second policy keeps the namespaces the server makes to its own, so it can't make one that enforces `privileged` outside them. Both need Kubernetes 1.30 or later (ValidatingAdmissionPolicy). Apply them before you set `capabilities`, with the RuntimeClass you name there, the server's ServiceAccount and its `namespacePrefix`: ```yaml title="admission.yaml" apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: { name: hammerkit-capabilities } spec: matchConstraints: namespaceSelector: { matchLabels: { hammerkit.dev/managed: 'true' } } resourceRules: - { apiGroups: [''], apiVersions: ['*'], operations: [CREATE, UPDATE], resources: [pods, pods/ephemeralcontainers] } variables: - name: sandboxed expression: "object.spec.?runtimeClassName.orValue('') == 'gvisor'" # the security context of every container, init and ephemeral ones too - name: contexts expression: >- object.spec.containers.filter(c, has(c.securityContext)).map(c, c.securityContext) + object.spec.?initContainers.orValue([]).filter(c, has(c.securityContext)).map(c, c.securityContext) + object.spec.?ephemeralContainers.orValue([]).filter(c, has(c.securityContext)).map(c, c.securityContext) - name: baseline expression: "['AUDIT_WRITE', 'CHOWN', 'DAC_OVERRIDE', 'FOWNER', 'FSETID', 'KILL', 'MKNOD', 'NET_BIND_SERVICE', 'SETFCAP', 'SETGID', 'SETPCAP', 'SETUID', 'SYS_CHROOT']" - name: seLinuxTypes expression: "['', 'container_t', 'container_init_t', 'container_kvm_t', 'container_engine_t']" - name: sysctls expression: "['kernel.shm_rmid_forced', 'net.ipv4.ip_local_port_range', 'net.ipv4.ip_unprivileged_port_start', 'net.ipv4.tcp_syncookies', 'net.ipv4.ping_group_range', 'net.ipv4.ip_local_reserved_ports', 'net.ipv4.tcp_keepalive_time', 'net.ipv4.tcp_fin_timeout', 'net.ipv4.tcp_keepalive_intvl', 'net.ipv4.tcp_keepalive_probes', 'net.ipv4.tcp_rmem', 'net.ipv4.tcp_wmem']" validations: - expression: "variables.sandboxed || variables.contexts.all(s, s.?capabilities.?add.orValue([]).all(a, a in variables.baseline))" message: capabilities beyond baseline are added in the RuntimeClass gvisor only - expression: "!variables.contexts.exists(s, s.?privileged.orValue(false))" message: no privileged containers - expression: "!object.spec.?hostNetwork.orValue(false) && !object.spec.?hostPID.orValue(false) && !object.spec.?hostIPC.orValue(false)" message: no host namespaces - expression: "!object.spec.?volumes.orValue([]).exists(v, has(v.hostPath))" message: no hostPath volumes - expression: >- !object.spec.containers.exists(c, c.?ports.orValue([]).exists(p, p.?hostPort.orValue(0) != 0)) && !object.spec.?initContainers.orValue([]).exists(c, c.?ports.orValue([]).exists(p, p.?hostPort.orValue(0) != 0)) message: no host ports - expression: "variables.contexts.all(s, s.?procMount.orValue('Default') == 'Default')" message: no unmasked /proc - expression: >- object.spec.?securityContext.?seccompProfile.?type.orValue('') != 'Unconfined' && variables.contexts.all(s, s.?seccompProfile.?type.orValue('') != 'Unconfined') message: no unconfined seccomp - expression: >- object.spec.?securityContext.?appArmorProfile.?type.orValue('') != 'Unconfined' && variables.contexts.all(s, s.?appArmorProfile.?type.orValue('') != 'Unconfined') && object.metadata.?annotations.orValue({}).all(k, !k.startsWith('container.apparmor.security.beta.kubernetes.io/') || object.metadata.annotations[k] == 'runtime/default' || object.metadata.annotations[k].startsWith('localhost/')) message: no unconfined AppArmor - expression: >- object.spec.?securityContext.?seLinuxOptions.?type.orValue('') in variables.seLinuxTypes && object.spec.?securityContext.?seLinuxOptions.?user.orValue('') == '' && object.spec.?securityContext.?seLinuxOptions.?role.orValue('') == '' && variables.contexts.all(s, s.?seLinuxOptions.?type.orValue('') in variables.seLinuxTypes && s.?seLinuxOptions.?user.orValue('') == '' && s.?seLinuxOptions.?role.orValue('') == '') message: no SELinux options beyond the container types - expression: "object.spec.?securityContext.?sysctls.orValue([]).all(s, s.name in variables.sysctls)" message: only the safe sysctls - expression: >- !object.spec.?securityContext.?windowsOptions.?hostProcess.orValue(false) && !variables.contexts.exists(s, s.?windowsOptions.?hostProcess.orValue(false)) message: no Windows host processes --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: { name: hammerkit-capabilities } spec: { policyName: hammerkit-capabilities, validationActions: [Deny] } --- # the server makes the namespaces of its callers only, at the levels it sets apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: { name: hammerkit-namespaces } spec: matchConstraints: resourceRules: - { apiGroups: [''], apiVersions: ['*'], operations: [CREATE, UPDATE], resources: [namespaces] } matchConditions: - { name: server, expression: "request.userInfo.username == 'system:serviceaccount:hammerkit:hammerkit'" } validations: - expression: "object.metadata.name.matches('^hammerkit-[0-9a-f]{16}$')" message: the server makes the namespaces of its callers only - expression: "object.metadata.?labels['hammerkit.dev/managed'].orValue('') == 'true'" message: the namespaces of callers are labelled hammerkit.dev/managed - expression: "object.metadata.?labels['pod-security.kubernetes.io/enforce'].orValue('') in ['baseline', 'privileged']" message: the namespaces of callers enforce Pod Security baseline, or privileged with capabilities --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: { name: hammerkit-namespaces } spec: { policyName: hammerkit-namespaces, validationActions: [Deny] } ``` Then relabel the namespaces the server made before, those of its prefix: ```bash kubectl get namespace -l hammerkit.dev/managed=true -o name | grep '^namespace/hammerkit-' | \ xargs -r -I{} kubectl label {} pod-security.kubernetes.io/enforce=privileged \ pod-security.kubernetes.io/audit=baseline --overwrite ``` When you remove `capabilities` again, set them back to `enforce=baseline` (and remove `audit`) before the server restarts: it runs nothing in a namespace that still enforces `privileged`, and its error names the command. [Building container images](/docs/guides/container-images) walks through a build of a Dockerfile with BuildKit in gVisor, pushed with a token of a service account. ## Kubernetes [#kubernetes] Experimental, like the server. Linux nodes only. ```yaml title="server.yaml" backend: type: kubernetes namespace: hammerkit # the server's own; the namespace of its pod when left out # kubeconfig: /etc/hammerkit/kubeconfig # with context, for a cluster the server is not in # context: prod storageClass: standard-rwo # of the claims of tasks; the cluster's default when left out volumeSize: 1Gi # default, of a task's claim that states no volumeSize maxVolumeSize: 50Gi # the most a task may state; 10Gi (or volumeSize, if more) when left out keepVolumes: 15m # default; how long a task's claim stays for the next run, false for not at all pools: - name: builds platform: { os: linux, arch: amd64 } nodeSelector: { cloud.google.com/gke-nodepool: builds } tolerations: [{ key: builds, operator: Exists, effect: NoSchedule }] toolImage: ghcr.io/corp/build-tools:2026.10 providers: gsm: { image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim } quota: { requests.cpu: '8', requests.memory: 16Gi, persistentvolumeclaims: '20', requests.storage: 100Gi } defaultResources: { requests: { cpu: 250m, memory: 256Mi } } sandboxes: { size: 4Gi, storageClass: sandboxes } # runRole: hammerkit-runs # the ClusterRole the server binds in each caller's namespace # deniedEgress: [169.254.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, fe80::/10, fc00::/7] allowedEgress: [10.20.0.15/32] # an internal registry open tasks may reach anyway networkPolicy: enforced # the network plugin enforces NetworkPolicy; limited tasks need it # openNetworkGate: false # what open pods wait to no longer reach, see Namespaces ``` The server runs as before (it checks the code out, plans the run with the engine in a process of its own and keeps the cache), but every container task of a run becomes a Job and every service a Deployment, in a namespace of the caller. | Key | What it does | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `namespace` | The server's own namespace, where the [credentials of the accounts](#secrets-in-the-job) are. Found on its own when the server runs in a pod. | | `kubeconfig`, `context` | The cluster, for a server outside of it. Without them the server uses its default kubeconfig, else the cluster it runs in. | | `namespacePrefix` | A caller's namespace is this and the first 16 hex digits of the sha-256 of their issuer and subject, `hammerkit-` by default. Two servers on one cluster need a prefix each. | | `storageClass` | The storage class of the claims tasks get. | | `volumeSize`, `maxVolumeSize` | The size of a task's or service's claim when it states no [`volumeSize`](/docs/task/kubernetes#claims) (`1Gi`), and the most one may state: `10Gi` when left out, or `volumeSize` if that is more, since a build file states its own. A task over it fails before its pods start. Each claim counts as its size in a `requests.storage` quota, which bounds what a caller's claims take together. | | `keepVolumes` | How long the claim of a task stays after the run that mounted it last (`15m`), see [Kept claims](#kept-claims). `false` removes it at the end of the run. | | `pools` | Where pods are placed, see [Pools](#pools). One `linux`/`amd64` pool when left out. | | `providers` | Per secret provider, the image its command runs in. Every provider an account reads needs one. | | `quota` | The `ResourceQuota` of each caller's namespace, as Kubernetes writes it (`requests.cpu`, `limits.memory`, `pods`, `persistentvolumeclaims`, `requests.storage`). | | `defaultResources` | `requests` and `limits` (`cpu`, `memory`) for containers without [`resources`](/docs/task#resources), as a `LimitRange`, so they are admitted under a quota that limits compute. | | `sandboxes` | Where [sandboxes](/docs/remote-compute/sandboxes#on-kubernetes) are kept: two claims per sandbox in its caller's namespace, their size and storage class, and the image and resources of the pod that reads and edits its files. | | `runRole` | The ClusterRole the server binds to itself in each caller's namespace, `hammerkit-runs` by default, see [What the server may do](#what-the-server-may-do). | | `deniedEgress` | Address ranges a pod of a run with an open network, or an egress proxy, may not connect to. By default the link-local ranges, where the clouds' metadata endpoints hand out the node's identity (`169.254.0.0/16`, `fe80::/10`), and the private ones (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `100.64.0.0/10`, `fc00::/7`), where the cluster's nodes, pods and services and the networks behind it are. So `open` means the internet. A cluster with pod or service ranges outside of them (GKE's `34.118.224.0/20`) adds those. | | `openNetworkGate` | With `networkPolicy: enforced`, what a pod with an open network waits to no longer reach before it starts: `{ host, port }`, an IPv4 address inside `deniedEgress` that answers without the policy, by default the metadata endpoint `169.254.169.254:80`, or `false` for no wait. See [Namespaces](#namespaces). | | `allowedEgress` | Addresses inside `deniedEgress` they may reach anyway, such as an internal registry, a package mirror, the secret manager a provider reads in a Job, or the database of an `allow` [TCP forward](/docs/task/network). None by default. The pods of the caller's own namespace and DNS are reachable without it. | ### Namespaces [#namespaces] The first run of a caller makes their namespace, labelled `hammerkit.dev/managed`. A namespace of that name without the label is refused, the server runs no task in a namespace it did not make. When a run starts the server puts in it, replacing what was there: * a RoleBinding of `runRole` to itself, its only rights in the namespace; * a `NetworkPolicy` that lets each pod with an open network accept connections from the pods of its namespace only (the services a task needs), so callers can not reach each other's pods; * a `NetworkPolicy` for the pods with an open network and the egress proxies: they reach their namespace, DNS, any address outside `deniedEgress` and the addresses of `allowedEgress`; * a `NetworkPolicy` that lets nothing into an egress proxy, so a proxy takes connections only from the task its own policy names, and from nobody once that policy is gone; * a `NetworkPolicy` that lets nothing into or out of the pods that copy files into and out of claims (labelled `hammerkit.dev/transfer`), which the server reaches through the API only; * `quota` and `defaultResources`; removing one from the config removes it from the namespace. The server does that again for each run, but not within 5 minutes of the last time for the same namespace: a namespace that was deleted and made anew, or a server that restarted, gets it at the next run. Policies add up, so none of them widens a pod with a limited [network](#network) (labelled `hammerkit.dev/network: limited`): its own policy alone decides what it reaches, DNS left out, and what reaches it. An egress proxy takes connections from its task only. Some network plugins apply a policy a moment after a pod started. With `networkPolicy: enforced`, a pod with an open network therefore starts with the init container `hammerkit-network`, which waits until the pod no longer reaches an address the namespace denies it, as a limited pod waits for its own policy (see [Network](/docs/task/network#kubernetes)). By default that is the metadata endpoint, `169.254.169.254:80`. Each open pod starts about a second later for it, the time the gate waits for an answer that doesn't come, and each node pulls the proxy's image once. When the policy doesn't hold within a minute, the pod fails rather than run open, and says so. `openNetworkGate` changes what it waits for: * an address and port inside `deniedEgress` that answers without the policy, such as a node's kubelet (`{ host: 10.0.0.10, port: 10250 }`). Use one where the metadata endpoint doesn't answer at all (bare metal, kind, k3s, AWS without IMDS): there the default gate opens at once and proves nothing. * `false` turns it off, for a cluster whose metadata traffic is redirected before the network plugin sees it, where the gate would never open. When `deniedEgress` doesn't cover the gate's address, or `allowedEgress` lets it in, open pods start without the gate. An allow of the address the cloud redirects metadata to (GKE's `169.254.169.252:988`) isn't seen: turn the gate off there. The policies of a task or service that a [service run](/docs/remote-compute/service-runs) no longer has after a move go once its pods have ended. The namespace enforces the `baseline` [Pod Security Standard](https://kubernetes.io/docs/concepts/security/pod-security-standards/): no privileged pods, host paths or host network (with [capabilities](#capabilities), an admission policy of the operator in its place). Every container of a run runs with `allowPrivilegeEscalation: false` (so `sudo` and other setuid programs do not work), without the `NET_RAW` capability (no `ping`) and with the runtime's seccomp profile, and without a token of the cluster. What a run makes there carries the label `hammerkit.dev/run` and is deleted when the run ends, whether it succeeded or not, and the end of the run waits until its pods and claims are gone: an idle caller costs nothing on the cluster but the [claims it keeps](#kept-claims) for a while. What a run left because the server stopped before its end is deleted when the caller's next run starts: the first run of the caller after the server started, or after a run whose objects could not all be deleted. The runs of one caller execute side by side in their namespace, up to [`runs.limits.running`](/docs/remote-compute/runs#limits-and-restarts), as the runs of different callers do. Several agents of one identity, each on a sandbox of its own, do not wait for each other. On a server the Jobs, Deployments, Services, Secrets and claims of a task are named after its cache id (its first 32 characters) and the run (the first 8 digits of its `hammerkit.dev/run` label), and found by both labels, so two runs of the same task have their own objects and the end of one never removes what the other uses. Kept claims are named after the source they are kept for instead. Two runs that share [kept claims](#kept-claims) execute one after the other: runs of the same branch or tag of a repository, checks of the same repository and runs from a commit hash of it, runs of the same sandbox, or the uploads of one caller (a source of type `upload`), whose kept claims are the caller's. The second waits in the queue, without counting against `runs.maxDuration`. Runs of two branches have kept claims of their own and go side by side. With `keepVolumes: false` nothing is shared and even the runs of one branch run side by side. The runs of a caller share the quota of the namespace, so size it for `runs.limits.running` runs of one caller at once, and for a task with all its services up: a service and the task that needs it hold their room at the same time. Kept claims count once for each branch that ran in the last `keepVolumes`, so set `requests.storage` to bound the disk they take. A pod that does not fit the nodes waits for one, until `runs.maxDuration`. The pod of a task or service over the quota is not created until the quota has room: it says so in its log, with what the quota refused, and waits. A pod that asks for more than the quota allows at all fails at once. After 10 minutes the wait goes on only while a pod that will end holds room in the namespace, such as another task of the run or of another run; when only services hold it, they wait for what waits for them, and the task or service fails. The wait of a task's pod does not count against its [`timeout`](/docs/task#timeouts), and lasts at most that timeout in all. While the quota refuses a task's pod, the run starts no other task that is less urgent than it (see [which ready task runs first](/docs/task/dependencies#which-ready-task-runs-first)), so the room that frees goes to it. A task that needs a service is not held back: its service is up already and holds its room until the task ran. The pods that copy a task's files into its claim and its outputs out of it wait 10 minutes. They ask for 64Mi and are limited to 512Mi of memory, so they take little of a quota sized for the tasks, whatever `defaultResources` says. Each service pod also has a small `debug` container beside the service, with the service's files, for a shell with `kubectl exec`; it asks for 10m CPU and 16Mi and is limited to 100m and 256Mi. A claim over the quota (`persistentvolumeclaims`, `requests.storage`) does not wait. The server first removes released kept claims of the caller to make room, and when that is not enough the task fails with `exceeded quota`. What changes objects of the whole namespace happens for one run at a time: making the namespace and its policies and quota at the start of a run, removing what a stopped server left (the objects of a run that does not execute) and removing kept claims that expired. These take a moment at the start of a run, not the whole run. ### Kept claims [#kept-claims] The claim of a container task stays after the run for `keepVolumes` (`15m`), and each run that mounts it again starts the time anew. The next run of the same source finds the task's outputs in it: when the task's inputs did not change, the task is cached and the tasks that depend on it mount its outputs from there, without restoring them from the cache. A restore copies the outputs into a new claim through a pod, which for a `node_modules` of an `npm ci` takes a while; with the claim kept, a change to code the install does not read goes straight to the compile and test steps. When the inputs did change, the task runs on its kept claim emptied first, as on a new one, so a run's outputs never mix with those of an earlier one. Without a kept claim, a cache hit restores into a new claim as before. A kept claim is the source's, within its [cache](#cache-and-sources): the runs of one branch or tag of a repository share theirs, the checks of a repository and its runs from a commit hash theirs, and a run on a sandbox mounts only those of the same sandbox, never what a run of the repository or of another sandbox kept. The approved runs of a sandbox share claims of their own, per sandbox: they never mount what the sandbox's runs that needed no approval kept, as their cache is apart from theirs. The first run of a new branch finds none and restores the cached outputs, as without kept claims; each active branch takes claims of its own until they expire. Its name carries a hash of the source, and its label `hammerkit.dev/volume-scope` the source. The disk they take is bounded by the time and the quota of the namespace: the server removes a claim once it expired, and when a claim of a run does not fit a `requests.storage` or `persistentvolumeclaims` quota, the kept claims the run does not use make room: those of the caller's other sources that their last run released, and those of its own a task no longer has (one renamed or changed). The claims a run of another source mounts while it executes stay, and the sweep of expired claims leaves all claims of a source while one of its runs executes. Room for a claim is made differently: a run of one branch may remove the released claims of another branch that runs at the same time but has not mounted them yet, and that run's tasks then restore from the cache. Size `requests.storage` for the branches that run at once to keep them warm. After a restart, the server finds the claims it kept before when its first run starts, and removes them as they expire. Services keep no claim; theirs go with the run. ### Pools [#pools] A pool is a set of nodes: `platform` (`linux`, with `amd64` or `arm64`), `nodeSelector` (the labels besides `kubernetes.io/os` and `kubernetes.io/arch`, which come from `platform`) and `tolerations`. Container tasks and services go to the first pool. A task without an image runs in the `toolImage` of the first pool that has one and offers the `platform` the task declares (`platform: { arch: arm64 }` picks an `arm64` pool). A task for macOS or Windows has no pool and fails when the run starts. `runtimes` has to list `local` for such tasks, and at least one pool needs a `toolImage` then. ### Secrets in the Job [#secrets-in-the-job] The server reads no secret of a provider for a task on a cluster: the Job of the task does, with only the credentials of the accounts it uses. Each provider secret gets an init container in the provider's image, which runs the provider's command with the provider's and the account's `env` and writes the value into a volume in memory. The task's container mounts only that volume: a `path:` secret is the file, an `env:` secret is set before the command runs. The credential of an account is set under `accounts..kubernetes` and is put at `/var/run/hammerkit/account/` in the init containers of that account only: ```yaml title="server.yaml" accounts: payments-dev: providers: gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /var/run/hammerkit/account/gcp.json } } kubernetes: serviceAccount: payments-dev # a ServiceAccount in backend.namespace audience: //iam.googleapis.com/projects/123/locations/global/workloadIdentityPools/hk/providers/cluster secret: payments-dev-federation # a Secret in backend.namespace, each key a file ``` | Key | What it does | | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `serviceAccount` | A ServiceAccount of the server's namespace. Right before each Job of the task, the server asks for a token of it (an hour, for `audience`) and puts it at `token`; the Secret that carries it is deleted when the Job is done. A cloud that trusts the cluster's issuer for that ServiceAccount (GCP workload identity federation, AWS `AssumeRoleWithWebIdentity`, Azure federated credentials) exchanges it for the account's credentials. Short-lived, and one trust binding per account whoever the caller is. | | `audience` | The audience of that token; the cluster's default when left out. | | `secret` | A Secret of the server's namespace, each key a file of the directory: the federation config that points at `token`, or a key file where nothing else works (long-lived, so the second choice). | #### Secrets of a task with a limited network [#secrets-of-a-task-with-a-limited-network] A task with `network: none` or an allow list reaches nothing else, and its init containers share its policy. `backend.providers..allow` names the hosts a provider's command reaches: ```yaml title="server.yaml" backend: type: kubernetes networkPolicy: enforced providers: gsm: image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim # Google's STS exchanges the token of the account's ServiceAccount, Secret Manager reads the value allow: [sts.googleapis.com, secretmanager.googleapis.com] runs: secretNetwork: none ``` * The task's egress proxy lets a tunnel to one of those hosts, on port 443, through only when it carries the proxy's token. The token is new for each run of the task, in a Secret of the caller's namespace that the init containers read their proxy variables from (`HTTPS_PROXY=http://hammerkit:@hammerkit-proxy:3128`). The task's own container gets neither the token nor those variables, so its command reaches the provider's hosts as little as any other. The Secret goes with the proxy. * A task with `network: none` gets a proxy for this alone, which allows its own command no host. * The names are exact names or `*.` patterns, without a port; like a pattern of `allow`, they never reach a private address. Name the hosts the command needs, not their domain: `*.googleapis.com` would let it reach every Google API. The provider's command has to read `HTTPS_PROXY`, user and password included; `gcloud` does. A command that ignores it can't read a secret under a limited network. A federation config that impersonates a Google service account (`service_account_impersonation_url`) also needs `iamcredentials.googleapis.com`. * With [`runs.secretNetwork`](/docs/remote-compute/runs#tasks-that-read-secrets) every task that reads a secret has to be limited, so each provider of such a task needs `allow`. On a cluster every account with `providers` needs `kubernetes`, or the server does not start: its commands would otherwise run as whatever identity the node or the provider's image has, the same for every account. The values never reach the server, so it can not mask them in the log. The task's output is masked in the pod instead, with `awk`, line by line, before it leaves the pod: the image of a task that reads such a secret needs `awk` (busybox, alpine, debian and their kin have it), and stdout and stderr arrive as one stream. Encodings of a value (base64, hex) are not masked there. `env:` secrets (variables of `runs.env.allow`) are the server's own and are handed over as before. Not yet on a cluster, and refused when a build file is published: a provider secret on a service, and a provider secret with `cache: true` (its digest would be needed before the Job reads the value). ### Cache and sources [#cache-and-sources] The cache stays on the server, one per source trust as on a host (a repository's runs share it). After a task ran, its outputs are copied from its claim into the cache, and a later run that hits restores them into a new claim before its dependents start, unless the claim is still [kept](#kept-claims). The files of a task's `src` are copied from the server's checkout into its claim. So on a cluster the storage directory of the server (`storage.directory`) belongs on a volume of its own. ### Network [#network] A task's [`network`](/docs/task/network) works on a cluster as in a [Kubernetes environment](/docs/task/network#kubernetes): a limited task's Jobs get a `NetworkPolicy` that lets them reach only the services they need and, under `allow`, their egress proxy, a Deployment on the first pool. A task without an image runs in a container of the [tool image](#pools), so it is limited like a container task: it may name its own `network` on such a server (elsewhere a local task with a limited network is refused when the build file is read), and `runs.network` and `runs.secretNetwork` hold it as well. The policy, the proxy and its Service carry `hammerkit.dev/run` and go with the rest of the run. Pods with an open network are governed by the policy of the caller's namespace, see [Namespaces](#namespaces). The cluster's network plugin has to enforce `NetworkPolicy` (Calico and Cilium do, and recent kind), for `network:` and for keeping callers apart. Without it nothing is isolated. Hammerkit can't tell from the API, so `backend.networkPolicy: enforced` states it; without it a task with a limited network fails instead of running open. With [sandboxes](/docs/remote-compute/sandboxes#on-kubernetes), whose runs are limited by `sandboxes.network`, a config without it is refused unless that is `open`, and so is one with [`endpoints`](/docs/remote-compute/service-runs#on-kubernetes), whose services only the server is to reach. Some apply a policy a moment after the pod started, so a limited pod waits in an init container until its policy holds, and fails after 60 seconds if it never does (see [Kubernetes](/docs/task/network#kubernetes)). A run's policies go only after its pods are gone. A pod with an open network has no such wait, so on those plugins it can reach a `deniedEgress` address in its first moment. A task that reads a provider's secret can have a limited network when the server names the endpoints of the provider: its init containers are in the task's pod, under the same policy, and reach the secret manager through the task's egress proxy (under `none` too). See [Secrets in the Job](#secrets-in-the-job). Without them such a task keeps an open network: a build file that limits it is refused when it is published. ### Sandboxes [#sandboxes] A [sandbox](/docs/remote-compute/sandboxes#on-kubernetes) is two claims in its caller's namespace, next to the namespace's runs: its files, and its git data. A short pod of `backend.sandboxes.image` fetches into them, reads and edits the files and commits what the server pushes; the Jobs of a run on the sandbox mount the files only. The quota of sandboxes is a `ResourceQuota` on their storage class, when they have one of their own. The pod runs as root with only the capabilities to own and change files (the `baseline` Pod Security level allows it, `restricted` does not). A task of a run on a sandbox that mounts all of its files and nothing else runs in the sandbox's runner, a pod `hammerkit-runner-` that stays and takes each command over `pods/exec` (see [sandboxes](/docs/remote-compute/sandboxes#on-kubernetes)). The pod of the files owns it, so it goes with that pod after `sandboxes.idle`, and so does its `NetworkPolicy`, which outlives the runner while the pod of the files stays. The runner carries no `hammerkit.dev/run` label: the end of a run leaves it, and removing what a stopped server left skips it. It counts against the quota like a task's pod, so a namespace with sandboxes needs room for one runner per active sandbox besides the runs. It has the security context, resources and placement of the task's Job, but no token and no service links, and an init container of `busybox:1.37.0-musl` (pinned by digest) that copies the busybox the runner cleans up with: a cluster that pulls only from a registry of its own needs that image there too. When the runner can not be used (the cluster refuses it, it does not start, or another task of the run is in it) the task runs as a Job. ### What the server may do [#what-the-server-may-do] Across the cluster the server makes namespaces and binds `runRole` in the ones it made. With [sandboxes](#sandboxes) it also lists namespaces when it starts, to find the claims of sandboxes in the ones it made. With [service runs](/docs/remote-compute/service-runs#on-kubernetes) it deletes the namespace of each at its end, and lists them to remove the ones a stopped server left. It needs no rights on Secrets or pods outside of them: those come from `runRole`, which it binds in each caller's namespace (Kubernetes lets it bind that role only, by `bind`). In its own namespace it reads the Secrets of the accounts and makes tokens of their ServiceAccounts, by name. The processes of runs use the same identity. `list` on events is how the log of a run that failed gets what the cluster said of its pods (a failed image pull, a pod no node takes); a `runRole` without it still works, and the log says the events could not be read. `list` on `pods.metrics.k8s.io` and, across the cluster, on nodes and `nodes.metrics.k8s.io` are only for what [`remote capacity`](/docs/remote-compute/status#capacity) and the dashboard show of the CPU and memory in use; without them, they say what they could not read. ```yaml title="rbac.yaml" apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: { name: hammerkit-server } rules: - { apiGroups: [''], resources: [namespaces], verbs: [get, list, create, delete] } # list: with sandboxes or service runs, delete: service runs - { apiGroups: [rbac.authorization.k8s.io], resources: [rolebindings], verbs: [get, create, update] } - apiGroups: [rbac.authorization.k8s.io] resources: [clusterroles] verbs: [bind] resourceNames: [hammerkit-runs] # optional, for the capacity: the nodes, what they have and use - { apiGroups: [''], resources: [nodes], verbs: [list] } - { apiGroups: [metrics.k8s.io], resources: [nodes], verbs: [list] } --- # bound by the server in each caller's namespace, never across the cluster apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: { name: hammerkit-runs } rules: - { apiGroups: [''], resources: [resourcequotas, limitranges], verbs: [get, create, update, delete] } - apiGroups: [''] resources: [pods, pods/exec, pods/log, persistentvolumeclaims, secrets, services] verbs: [get, list, create, patch, delete, deletecollection] - { apiGroups: [''], resources: [events], verbs: [list] } # what the cluster said of a run that failed - { apiGroups: [metrics.k8s.io], resources: [pods], verbs: [list] } # what the pods of runs use, for the capacity - apiGroups: [batch] resources: [jobs, jobs/status] verbs: [get, list, create, patch, delete, deletecollection] - { apiGroups: [apps], resources: [deployments], verbs: [get, list, create, patch, delete, deletecollection] } - apiGroups: [networking.k8s.io] resources: [networkpolicies] verbs: [get, list, create, update, patch, delete] --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: { name: hammerkit-accounts, namespace: hammerkit } rules: # the secret and serviceAccount of each account - { apiGroups: [''], resources: [secrets], verbs: [get], resourceNames: [payments-dev-federation] } - { apiGroups: [''], resources: [serviceaccounts/token], verbs: [create], resourceNames: [payments-dev] } ``` Bind `hammerkit-server` with a ClusterRoleBinding and `hammerkit-accounts` with a RoleBinding to the server's ServiceAccount; do not bind `hammerkit-runs` yourself. Making RoleBindings across the cluster still lets the server bind `hammerkit-runs` in a namespace it did not make. To rule that out, admit its RoleBindings only in the namespaces of its own callers, its `namespacePrefix` and 16 hex digits, and not in those of another server on the cluster. The second policy keeps the deletes of namespaces to those of its service runs: ```yaml title="admission.yaml" apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: { name: hammerkit-managed-namespaces } spec: matchConstraints: resourceRules: - { apiGroups: [rbac.authorization.k8s.io], apiVersions: ['*'], operations: [CREATE, UPDATE], resources: [rolebindings] } matchConditions: - { name: server, expression: "request.userInfo.username == 'system:serviceaccount:hammerkit:hammerkit'" } validations: - expression: "namespaceObject.metadata.?labels['hammerkit.dev/managed'].orValue('') == 'true'" - expression: "namespaceObject.metadata.name.matches('^hammerkit-[0-9a-f]{16}$')" --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: { name: hammerkit-managed-namespaces } spec: { policyName: hammerkit-managed-namespaces, validationActions: [Deny] } --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: { name: hammerkit-service-run-namespaces } spec: matchConstraints: resourceRules: - { apiGroups: [''], apiVersions: ['*'], operations: [DELETE], resources: [namespaces] } matchConditions: - { name: server, expression: "request.userInfo.username == 'system:serviceaccount:hammerkit:hammerkit'" } validations: - expression: "oldObject.metadata.?labels['hammerkit.dev/service-run'].hasValue()" - expression: "oldObject.metadata.name.matches('^hammerkit-[0-9a-f]{16}$')" --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicyBinding metadata: { name: hammerkit-service-run-namespaces } spec: { policyName: hammerkit-service-run-namespaces, validationActions: [Deny] } ``` Upgrading a server for [service runs](/docs/remote-compute/service-runs): add `delete` on namespaces to `hammerkit-server` and the second policy. Without them a service run still ends, but its namespace stays: the server removes what runs in it, and says in its log and in the run's what its ClusterRole needs. Not yet: Windows nodes, holding a pod with an open network back until the policy of its namespace holds, a cache in a registry or a bucket, a limited network for a task that reads a provider's secret, and failing a task early whose pod can not start (an image that can not be pulled, no node for its pool). # Build files (/docs/remote-compute/build-files) A run does not bring its own build file. The server keeps the build files that were published to it, and a run says which one to use and which of its tasks to execute. So whoever may run work cannot change what a task does, and what a task does is decided by whoever may publish. ``` POST /v1/definitions ``` ```json { "entry": ".hammerkit.yaml", "files": { ".hammerkit.yaml": "tasks:\n build:\n cmds: [npm run build]\n", "web/.hammerkit.yaml": "..." } } ``` A published build file is a bundle: the file that starts it (`entry`, the usual `.hammerkit.yaml` when left out) and every file it includes, as text, up to 200 files and 1 MiB. It has no name: the server answers with its version, and a run names that. The server reads the bundle the way a run would, in a directory of its own where only the bundle's files exist, so one that does not load is refused when it is published and not when somebody wants to run it (`400`). A bundle may not include anything outside of itself. Includes from git have to be pinned to a commit and come from an `https://` or `ssh://` address, and so do the git includes of the files they include, because the version below can only vouch for what is in the bundle. ## Versions [#versions] Every bundle has a version, the hash of its files: `sha256:` of the JSON `[entry, [[path, sha-256 of the content], ...]]` with the paths sorted. The same files have the same version anywhere, so a client can compute it and compare it with the server's. A version is never changed; publishing other files adds a version, publishing the same files again, by anybody, answers `200` and changes nothing (the first publish answers `201`). The hash is the only name a version has. There is no `latest` and no line of versions one follows the other in: a name would be the publisher's choice, so a new name could always start a new line, and a comparison with the version before would prove nothing. Whoever reviews a version reads all of it, its [plan](#plan-and-accounts) and its files. | Call | What it does | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `POST /v1/definitions` | Publish a bundle. Needs `builds: publish`, else `403`. With `builds: propose`, or when the bundle names accounts the caller does not `use`, it is a [proposal](#proposing-a-version) and answers `202`. The answer carries the `hash`. | | `GET /v1/definitions` | Every version, newest first, with who published (and approved) it and when, its [annotations](#annotations), the tasks and services it has, the `accounts` it uses and whether it was yanked. `?repo=github.com/corp/payments&machine=ci-7` keeps the versions with those annotations. | | `GET /v1/definitions/` | One version, with its `plan` and what each task works on (`inputs`). | | `GET /v1/definitions//files` | The files of one version, to see what would run. | | `POST /v1/definitions/plan` | What publishing the bundle would do, without storing anything. See [Checking before publishing](#checking-before-publishing). | | `POST /v1/definitions//yank` | Pull a version back, see [Yanking](#yanking-a-version). The body is optional: `{ "reason": "..." }`. | | `POST /v1/definitions//unyank` | Let a yanked version go on. Needs `builds: publish`. | Reading needs a token and at least one service account or `builds` level. Versions are kept in the `storage` directory and survive a restart. ## Annotations [#annotations] Without names, a version says where it comes from with annotations the publisher sends next to the bundle, so the versions of one repository or one machine can be found, and cleaned up, later: ```json { "entry": ".hammerkit.yaml", "files": { ".hammerkit.yaml": "..." }, "annotations": { "path": "web/.hammerkit.yaml", "repo": "github.com/corp/payments", "commit": "4be1a07c...", "machine": "ci-runner-7", "user": "runner" } } ``` `remote publish` fills in `path` (the build file's path in its git repository, or on the machine outside of one), `repo` (the `origin` remote as `host/path`, left out when it is not an https or ssh address), `commit`, `machine` and `user`. When a file of the bundle differs from that commit, or is not committed, it adds `dirty=true`: the version has the commit's files plus changes, not the commit's files. Any other key works too: up to 20, lower case letters, digits, `.`, `_` and `-`, each value up to 500 characters. Annotations are not part of the hash, so the same files sent from two machines are one version, and it keeps the annotations of its first publish (a proposal keeps the proposer's when it is approved). The server stores them as they were sent: they identify a version, they prove nothing. Who published it is `publishedBy`, from the token. ## Inputs [#inputs] A version lists, for every task, what it works on, so a client knows which of its files a run needs without planning the build file itself: ```json "inputs": { "build": { "src": ["package.json", "src/**/*.ts", "!src/**/*.spec.ts"], "mounts": [], "deps": [], "needs": [] }, "test": { "src": [], "mounts": [".npm"], "deps": ["build"], "needs": ["db"] } } ``` `src` are the patterns of the task's own `src` as the build file writes them (environment variables are not substituted), `mounts` are the host paths a container task mounts, `deps` the tasks and `needs` the services it waits for. Paths are relative to the root of the bundle, which is also the root of the project's files, and a path outside of it is left out because it cannot be sent. What a task inherits from its dependencies is listed with those. Versions published before this was recorded have no `inputs`. ## Plan and accounts [#plan-and-accounts] A version also carries a `plan`, what each task and service does as the engine reads it, and the `accounts` its secrets are read as. A reviewer or an agent sees `deploy` uses `op-payments-prod` before anything runs: ```json "accounts": ["op-payments-prod"], "plan": { "deploy": { "kind": "task", "file": ".hammerkit.yaml", "runtime": "docker", "image": "node:22", "cmds": ["./deploy.sh"], "command": "./deploy.sh", "network": "open", "deps": [], "needs": [], "accounts": ["op-payments-prod"], "secrets": [{ "from": "gsm:kube-token", "account": "op-payments-prod", "target": "KUBE_TOKEN" }], "serverEnvs": [], "mounts": [], "ports": [], "generates": [], "flags": ["local-task"] } } ``` `command` is the last command of a task (the `cmd` of a service), where a run's [arguments](/docs/remote-compute/runs#arguments) land; `null` when there is none. `serverEnvs` are the variables of the server its `envs` read (`$NAME`, `${NAME}`). `network` is what it reaches besides its needs, as the build file declares it (`open` when it declares nothing). Versions published before these were recorded have neither. `file` is the build file that declares it, as a path in the bundle, or the repository and the path in it for one of a git include. `deps` and `needs` name the tasks and services as the plan does (`services:postgres` for `postgres` of an include). Versions published before these were recorded have no `file`, and name what a task needs as its build file does. `runtime` is `docker`, `local`, or `none` for an [aggregate task](/docs/task/dependencies#aggregate-tasks), which runs nothing. The plan never holds a value of a secret, only where it is read from. `flags` are what a reviewer looks at first: `local-task` (it runs on the host, not in a container), `host-port` (it binds a port of the host), `remote-include` (it includes a file from git) and `capabilities` (it adds the Linux capabilities in `capAdd`, see [Capabilities](/docs/remote-compute/backend#capabilities)). A bundle that names an account the server does not define is refused and the message lists the accounts that exist. ## Publishing and accounts [#publishing-and-accounts] A bundle that uses an account the publisher does not `use` becomes a [proposal](#proposing-a-version) even for `builds: publish`, and is approved by somebody who may publish and who uses the accounts the proposer lacks. So one person can not publish a task that reads the production key for somebody else to run without a second pair of eyes. ## What a build file may not do [#what-a-build-file-may-not-do] A build file is written by whoever publishes it and runs with what the operator gave the server, so a bundle is refused (`400`, naming the file and task) when it * names `environments`: the docker host or Kubernetes cluster is the server's to choose, * has a service for Kubernetes (`selector`, `kubeconfig` or `context`), * names a cache backend other than the server's own (`s3`, `registry`, or a `path`), * mounts a path outside of the bundle into a container, such as the docker socket, `/etc` or a home directory, or has a `src` or `generates` outside of it (the server would copy its own files into the task), * has a service volume whose name is a path (`/var/run/docker.sock:/var/run/docker.sock`), which docker would mount from the host, * has a task on a runtime that `runtimes` does not allow, * has a task that adds a capability (`capAdd`) that [`backend.capabilities`](/docs/remote-compute/backend#capabilities) does not allow, which is any when the server names no sandbox for them, * has a task or service whose `network` reaches more than [`runs.network`](/docs/remote-compute/runs#network) allows, * declares `secretProviders` or `secretAccounts` (keep them in your [host config](/docs/task/secret-providers#on-this-machine-only)), reads a `file:` secret, reads an `env:` secret the server does not list in `runs.env.allow`, or reads a secret as the account that reads repositories (see [Accounts](/docs/remote-compute/accounts) and [Sources](/docs/remote-compute/sources)). * on a [Kubernetes backend](/docs/remote-compute/backend#secrets-in-the-job), reads a provider secret in a service, one with `cache: true`, or one in a task with a limited [`network`](/docs/remote-compute/backend#network). A run checks the same paths again in its checkout, before it starts and before each task or service runs: one whose `src`, `generates` or mount goes through a symbolic link of the checkout fails, since docker and the copy into a cluster would follow the link out of it. On a docker backend, once a container has started, the server also compares each folder and file it binds with the one of the checkout, and a container that sees anything else fails before anything of its image runs: a task's before any of its commands, a service's before its own command or entrypoint. The server checks that with tools of its own, which it puts in a folder of its own in each container (`/.hammerkit-` and a random name), so the image needs nothing for it. They are a file in the `tools` folder of the server's hammerkit directory (`HAMMERKIT_DIR`), bound by its path, so a server that runs in a container needs that folder at the same path on the docker host, as it needs the files of its runs. No `src`, output, mount, volume or secret file can be at `/` or at or in a folder of `/` whose name starts with `.hammerkit`. A task that adds [capabilities](/docs/remote-compute/backend#capabilities) runs in a sandboxed runtime on a docker host, where what the container binds can't be compared, so there it can't bind files of the run: its `src`, file outputs and mounts fail the run (cache volumes are fine). Without this a task with no secrets would still reach what the operator's machine can: a published file would be enough to send a run, or the cache, to a host of its choosing. ## Only part of a build file [#only-part-of-a-build-file] One build file usually holds tasks for the server and tasks only your machine runs: a watcher, a task that mounts the docker socket, a service for local development. Mark those `publish: false`, and the server leaves them out of the version: they are not in its tasks, not in its plan, and not checked against the rules above. ```yaml title=".hammerkit.yaml" includes: infra: { path: ./infra.yaml, publish: false } # everything of the file services: db: image: postgres:16 publish: false tasks: build: image: node:22-alpine cmds: [npm run build] watch: cmds: [npm run watch] publish: false ``` A file left out is not checked for what the list above refuses of a published one: its `environments` (a local docker host or kind cluster), `caches`, Kubernetes services and `secretProviders` stay on your machine. Neither is a Kubernetes service with `publish: false`. A git include with `publish: false` is not fetched by the server, so its `ref` may be a branch. Locally `publish` changes nothing. A published task or service that depends on, needs, extends or runs as `init` one that is left out is refused, naming both: `test needs db, which is not published (publish: false)`. `publish: false` belongs to the include, not to the file: a file that another published file includes too is published through that one, under that one's name. ## Checking before publishing [#checking-before-publishing] ```shell hammerkit remote publish --dry-run ``` The server reads the bundle with its policy as a publish would, and says what publishing it would do: publish it, [propose](#proposing-a-version) it (and which accounts make it a proposal), or that it is published already. A bundle the server would refuse prints why: every problem with its policy, or the first git include it may not fetch. Nothing is stored. Any `builds` level may check, `run` too, so a developer can check a build file before the account that publishes it does. As for a proposal, a caller who may not publish can't make the server fetch a git include: a build file that includes from git (and publishes that include) is checked by somebody with `builds: publish`. A version that is published already is not checked again, the answer says when it is yanked. ``` POST /v1/definitions/plan ``` takes the bundle as `POST /v1/definitions` does and answers its `hash`, `tasks`, `services`, `plan`, `accounts` and `outcome`: `published`, `proposed`, `already published`, or `null` for who may only run (`missing` lists the accounts the caller does not use, `yanked` is set for a yanked version). A build file that is checked, proposed or published reads the variables of runs (`runs.env.allow` and `runs.env.set`) by their names: `${NAME}` stands in for the value of each but `PATH`, `HOME` and `CI`. So a `description` or an `image` that reads one shows `${DEPLOY_REGION}` in the plan, never what a run would see; a run reads the values. A version published by a server of before this keeps the plan it was published with: publish the build file again (with any change, so it gets a new hash) when its `description` or `image` reads a variable of runs. ## Proposing a version [#proposing-a-version] Who defines the work decides what runs, so a new version of a build file can wait for a second pair of eyes. `builds: propose` is the lower level of `publish`, like `request` is the lower level of `use`: ```yaml title="server.yaml" entitlements: - group: { iss: https://login.example.com, name: platform } accounts: { op-payments-prod: use } builds: publish - claims: iss: https://login.example.com match: { client: coding-agent } builds: propose ``` A caller who may only propose sends the same `POST`. The server reads the bundle as it does for a publish and refuses one that does not load, then keeps it as a **proposal** (`202`). A publish also ends up here when the bundle names accounts the publisher does not `use` (`missing` lists them). It is not a version: it can not be run and it is not listed with the versions. Someone who may `publish`, uses the accounts in `missing`, and did not propose it, approves it, and only then it is published, as the proposer's version with the approver recorded as `approvedBy`. That is the version runs refer to by hash, so what was approved is what runs. | Call | What it does | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET /v1/proposals` | The proposals the caller made or may see, newest first, with `state` (`waiting`, `approved`, `rejected`, `expired`), who proposed it, the tasks and services, the `accounts` and the `missing` ones, `plan` and whether the caller `canDecide`. | | `GET /v1/proposals/` | One of them. | | `GET /v1/proposals//files` | Its files, to read what would be published. | | `POST /v1/proposals//approve` | Publish it. The body is optional: `{ "reason": "..." }`. | | `POST /v1/proposals//reject` | Drop it; the reason (up to 500 characters) is kept with it. | A proposal is visible to its proposer and to everyone who may publish; what the caller neither proposed nor may decide does not exist for them (`404`), and one that is decided or expired is `409`. A proposal expires after the `approvals.timeout` like a requested run does, and `approvals.maxWaiting` limits the proposals that wait (`503`). Proposals are kept in memory, unlike runs: after a restart the proposer proposes again. Each proposal and decision is logged with who made it. Whoever decides reads the proposal whole, its plan and its files (`remote show` does both). There is no diff against an earlier version: a proposal follows none. ### No git includes [#no-git-includes] A proposal is read before anybody approved it. A bundle that includes from git would make the server fetch from the network for a caller who may only propose, so a proposal with `includes` or `references` from git is refused (`400`). Whoever may publish can use them. ### Cooldown [#cooldown] After a rejection, the proposer can not propose anything, and nobody that version, for `approvals.cooldown` (default `10m`): `429`, with the time left. A proposal that only ran out of time does not start one. It keeps the people who decide from being asked again and again; like the proposals it is forgotten by a restart. A proposal approves a new version of the build file, not the code its commands run: `npm test` still executes the files a run brings. Production accounts stay behind [`request`](/docs/remote-compute/entitlements#approvals) and `source: [ref]`. ## Yanking a version [#yanking-a-version] A published version is never changed, but one can be pulled back when it should not run any more (it leaks a token, it deploys to the wrong place). `POST /v1/definitions//yank` is open to whoever has `builds: publish`, and to whoever `use`s an account the version works with: the people who answer for what runs as that account do not have to wait for a publisher. The body may carry a `reason`, up to 500 characters. A run that names a yanked version is refused with `409` and the reason, and a run that was accepted before, and still waits for an approval or in the queue, fails when it would start. Runs that already execute are not stopped; cancel them. The version and its files stay on the server for the record, and the version list shows who yanked it and why. Without names, yanking is the only way to stop a version: every published hash stays runnable until it is yanked. `unyank` lets it go on, and needs `builds: publish`. Both are logged with who did it, and a yank survives a restart. From the command line, [`remote publish`](/docs/cli/remote#publishing-a-build-file) publishes or proposes the build file of the working directory and prints its hash, and `remote proposals`, `show`, `approve`, `reject`, `yank` and `unyank` work with the versions. # Checks (/docs/remote-compute/checks) Experimental. The keys, the API and the settings can still change between releases. A CI job that only has to know whether a task passed for the files of its commit does not have to run it: it asks the server with [`hammerkit remote verify`](/docs/cli/remote#verifying-checks). The server answers from what its own runs passed. How to set that up is the guide [Checks that ran before CI](/docs/guides/agents/verified-checks). ## Check runs [#check-runs] Only the server writes checks, from runs it executed, for each task that `succeeded` or was `cached`. A request names tasks and a source, never a result. The runs whose tasks count: | Run | Counts | | ---------------------------------------------------------------------------------------- | ----------------------------------------------------- | | From a ref (`source.type: ref`) without [arguments](/docs/remote-compute/runs#arguments) | yes | | On a sandbox with `check: true`, see [below](#checks-of-a-sandbox) | yes | | On a sandbox without `check` | no: ignored files of the sandbox can make a task pass | | From uploaded files | no | | Any run with arguments | no: it ran something other than the task | A task without a [check key](#check-keys), and a task that failed, leave nothing. A `cached` task counts because the cache found its entry by the same inputs the check key covers. Checks hold only where tasks are isolated from the server: in containers or on a cluster. A [`local`](/docs/remote-compute/runs#config) task runs as the user of the server, and could write the server's records itself. ## Checks of a sandbox [#checks-of-a-sandbox] ```json { "definition": "sha256:9f2c...", "tasks": ["test"], "source": { "type": "sandbox", "id": "sbx_Jp0dY8m2kQ1aZt5x", "check": true } } ``` A check of a [sandbox](/docs/remote-compute/sandboxes) runs on a copy of the files a push of the sandbox would commit: the files git tracks and those `.gitignore` does not leave out, at the run's `tree`. Nothing else of the sandbox is there, so a file the owner wrote where no commit holds it can not make the task pass. What the tasks write stays in the copy, which is removed when the run is over: the sandbox does not change. Otherwise it is a run on the sandbox: it holds the sandbox while it runs, its network and accounts are those of a run on a sandbox. Its cache is the one of the repository's runs from a ref, since it runs on files a commit holds. Two more rules: * The caller needs `repos` for the sandbox's repository, as for a run from a ref, and a branch of it they may push to (`push` in `sandboxes`): `403` without. A check leaves what it made in the repository's cache, which runs from `main` restore from, and so can a caller who pushes a branch and runs it. A caller who may only pull can not. * It takes no arguments: `400`. * It takes a cached result only by the files' checksums, never by their dates (`cache: modify-date` is taken as `checksum`). The agents' [`check` tool](/docs/remote-compute/agents) submits such a run. ## Check keys [#check-keys] A check is found by its key, which both the server and `remote verify` compute from the build file and the files of a checkout, before anything ran in it: * the task's definition, as for its [cache](/docs/task/caching): commands, image, variables, mounts, the definitions of the services it `needs`, and so on, * the sha256 of every file its `src` matches, by path, without `.git`, * the keys of the tasks it depends on, * for each service it needs, the files of the service's `src` and the keys of the tasks the service depends on: an `e2e` that needs `api` gets another key when the source of `api` changes. So the key of a commit's task is the same wherever the commit is checked out, and a check counts for every commit with the same files. It hashes with sha256 throughout, as the cache does, so a file made to share the hash of another does not pass for it. Run `remote verify` in a fresh checkout: a file a task generated there, where the `src` of a task reads, gives another key than the commit has. The build file is read as a server reads it (no `.env`, no placement on a cluster, any secret provider and account). A task has no key when it, a task it depends on, or a task a service it needs depends on, declares no `src` that matches a file, or when it reads a secret with `cache: true`, whose value is part of its cache key. An [aggregate](/docs/task/dependencies#aggregate-tasks) (`deps` and nothing to run) needs no `src` of its own: it has a key when all its dependencies have one. Like any task, it counts as verified only for a run that ran it: a run of the aggregate itself, which costs nothing more when its dependencies already passed. ## Lookup [#lookup] ``` POST /v1/checks/lookup { "repo": "github.com/corp/payments", "keys": [{ "task": "test", "key": "4f0c…" }] } ``` Up to 100 keys. The answer has an entry per key, the newest run that passed it, or `null`: ```json { "checks": [ { "task": "test", "key": "4f0c…", "passed": { "task": "test", "status": "succeeded", "run": "run_Qx3k9ZpL2mW7vN4a", "by": { "iss": "https://accounts.google.com", "sub": "1180…" }, "source": { "type": "sandbox", "sandbox": "sbx_4vJ0mT2cX9qLw1aB", "commit": "4be1a07c…", "tree": "a1f3…" }, "definition": "sha256:9f2c…", "finishedAt": "2026-10-25T09:14:03.120Z", "durationMs": 72000 } } ] } ``` A check that passed with a version of the build file that was [yanked](/docs/remote-compute/build-files) since is `null`, with a `reason`. The caller needs `checks: read`, or a `builds` level, and `repos` that cover the repository, see [Entitlements](/docs/remote-compute/entitlements): `403` otherwise. The answer leaves out the accounts the run used: a CI job's log is often readable by more people than the server. ## Config [#config] ```yaml title="server.yaml" runs: checksKeep: 30d # default ``` Checks are kept in the server's storage directory (`checks/`), one file per repository and key, the newest run's. One older than `runs.checksKeep` is not found any more and is removed within six hours. # Entitlements (/docs/remote-compute/entitlements) An entitlement says which service accounts a caller may use, and what they may do with build files. The server matches the validated claims of the token only, never anything the caller sends along. Nothing is allowed by default. ```yaml title="server.yaml" entitlements: - group: { iss: https://login.example.com, name: platform } accounts: { op-payments-dev: use, op-payments-prod: use } repos: [github.com/corp/*] builds: publish - group: { iss: https://login.example.com, name: eng-payments } accounts: { op-payments-dev: use, op-payments-prod: request } repos: [github.com/corp/payments] builds: run - claims: iss: https://token.actions.githubusercontent.com match: { repository: corp/payments, ref: refs/heads/main } accounts: { op-payments-dev: use } repos: [github.com/corp/payments] builds: publish ``` | Key | What it does | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `subject`, `group` or `claims` | Who the entry is for, exactly one of them, see [Who](#who). | | `accounts` | The [service accounts](/docs/remote-compute/accounts) the caller may `use` or `request`, see [Account levels](#account-levels). | | `repos` | The repositories the caller may name in a run from a git ref, see [Repositories](#repositories). | | `builds` | What the caller may do with build files: `run`, `propose` or `publish`, see [Build file levels](#build-file-levels). | | `checks` | `read`: the caller may look up the [checks](/docs/remote-compute/checks) of the repositories of `repos`, and nothing else, for a CI job that only verifies. A `builds` level allows it too. Like the other levels it combines with the `repos` of every entitlement that matches the caller. | | `sandboxes` | The repositories the caller may keep [sandboxes](/docs/remote-compute/sandboxes) of, one `repo` (`host/path` pattern) per entry with a `pull` or `push` type per branch pattern (`branches`). A list of its own: `repos` is for runs from a ref. | | `endpoints` | The repositories whose [service runs](/docs/remote-compute/service-runs) the caller may open the endpoints of (`open`), or also publish endpoints in their own (`publish`), one `repo` pattern and `level` per entry, see [Who opens an endpoint](/docs/remote-compute/service-runs#who-opens-an-endpoint). | | `observe` | `true`: the caller sees every run of the server whole, who started it, what its tasks run and its log, in [`remote status`](/docs/remote-compute/status), `remote runs --all`, `remote logs`, the dashboard and `GET /v1/runs?scope=all`, and the published build files as who has a `builds` level does. For operators; it stops, extends and approves nothing. | An entry names accounts, a `builds` level, `checks`, `sandboxes`, `endpoints`, `observe` or more of them. `checks` needs `repos`. ## Who [#who] | Who | Matches | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `subject: {iss, sub}` | exactly that person or client; an optional `name` is how the dashboard names its runs when its tokens name nobody, such as a service account's without `--include-email` | | `group: {iss, name}` | every caller of that issuer whose groups claim lists the name | | `claims: {iss, match}` | every token of that issuer whose string claims equal all of `match`, for CI jobs (repository, ref, environment) | A caller gets what all entitlements that match give, and nothing when no entry matches. Every account that an entitlement names has to be defined under `accounts`, a typo stops the server at startup. ## Account levels [#account-levels] `accounts` maps an account to a level: * `use` runs on the account at once and approves the runs others request on it. * `request` lets the caller ask for runs on the account. Such a run waits until somebody who may `use` the account approves it, see [Approvals](#approvals). When several entitlements give a level for the same account, `use` wins. The account that reads the repositories of [`sources`](/docs/remote-compute/sources) is the server's own: no entitlement may name it. ## Build file levels [#build-file-levels] `builds` is what the caller may do with [build files](/docs/remote-compute/build-files), one level that includes the lower ones: | `builds` | May | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `run` | start runs of published build files. Without a level nothing runs, an account alone does not allow it. | | `propose` | also send new versions, which somebody who may publish approves, see [Proposing a version](/docs/remote-compute/build-files#proposing-a-version). | | `publish` | also publish build files. Who defines the work does not have to be who runs it. | A build file has no name, a version is the hash of its files, so the config never lists build files. ## Repositories [#repositories] `repos` lists the repositories (`github.com/corp/payments`, `github.com/corp/*` for one level, `gitlab.corp.com/platform/**` for a group with its subgroups) the caller may name in a run from a git ref; the union of the matching entitlements applies. A caller without `repos` can not start runs from a ref. The server also has to be able to read the repository, see [Sources](/docs/remote-compute/sources). ## What a caller sees [#what-a-caller-sees] Every grant is written to the log with issuer, subject, accounts and the entitlements that matched. A caller with no account and no `builds` level can do nothing on the server, they are not even shown which build files exist. `GET /v1/me` returns the identity and what the caller may use: ```json { "iss": "https://login.example.com", "sub": "4f1c2b", "accounts": [ { "name": "op-payments-dev", "level": "use" }, { "name": "op-payments-prod", "level": "request" } ], "repos": ["github.com/corp/payments"], "builds": "run", "checks": "read", "sandboxes": [], "endpoints": [], "observe": false } ``` `GET /v1/accounts` lists the accounts the caller has a level on, with the operator's `description` and the `sources` the account allows (`upload`, `ref`, `sandbox`, or `upload` and `ref` when the config does not say): ```json { "accounts": [ { "name": "op-payments-dev", "description": "dev cluster", "level": "use", "sources": ["upload", "ref"] }, { "name": "op-payments-prod", "level": "request", "sources": ["ref"] } ] } ``` ## Approvals [#approvals] Some work should not happen because one identity asked for it: a deploy with the production key, a task with access to customer data. Access to a service account comes at two levels for that, see [Account levels](#account-levels). Whoever may **use** an account runs on it directly. Whoever may only **request** it can ask for a run, which waits in state `awaiting-approval` until somebody who may use that account approves it. This suits an agent or a developer who proposes a deployment and a person who confirms it. ```yaml title="server.yaml" entitlements: - group: { iss: https://login.example.com, name: release-managers } accounts: { op-payments-prod: use } - group: { iss: https://login.example.com, name: eng-payments } accounts: { op-payments-dev: use, op-payments-prod: request } builds: run approvals: timeout: 1h # how long a run waits, default 1h maxWaiting: 100 # runs waiting at the same time, default 100 ``` | Key | What it does | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `approvals.timeout` | How long a requested run, or a [proposed version](/docs/remote-compute/build-files#proposing-a-version), waits for its decision before it is rejected or expires. `1h` by default. | | `approvals.maxWaiting` | Runs, and separately proposals, that wait at the same time; more are refused with `503`. `100` by default. | | `approvals.cooldown` | After a proposal was rejected, how long its proposer can not propose that build file again, and nobody that version. `10m` by default. | * The accounts of a run are derived from the secrets of the published, hash-pinned build file, see [Accounts](/docs/remote-compute/accounts). Nothing in the request can change them, so every run that reads a production secret as a requester is held, whatever its tasks are called or labelled, and whoever may [publish](/docs/remote-compute/build-files) cannot lift it. * A run needs an approval for the accounts the caller can only request, the others they use. `deploy` waits for `op-payments-prod` while `test` in the same run does not. * An approver covers every account they may `use`. A run that waits for accounts A and B is released as soon as the approvals cover both: one person who uses both is enough, otherwise two approve, one per account. Each account has its entry in the run's `approvals`, with who approved it. They share one expiry: a run that nobody covers within `timeout` is rejected, and so is one that any covering approver rejects. More than `maxWaiting` waiting runs are refused with `503`. A run that waits keeps waiting over a restart, for the time that is left, see [Limits and restarts](/docs/remote-compute/runs#limits-and-restarts). * A request for an account that nobody may use is refused at once (`409 nobody can approve `) and not left to expire. * A caller approves what they have access to, and nobody approves their own run. A caller who uses none of the accounts that wait sees nothing of the run. Approvers see the published build files, so they can read what they approve, and nothing more. * The approver is shown what to decide on: the tasks and services the run executes with their dependencies as `plan` (commands, image, the secrets and the account each is read as, mounts, ports and the `flags` to look at first, never a value), the accounts, the version, the source and the options. Who uses an account a run needed approved may also read that run (`GET /v1/runs/`, its events and its logs) and cancel it; `GET /v1/runs` stays the caller's own. * What an approval cannot cover is what the code does: pinning commands does not pin the code they run, so an account that matters should also be limited to `source: [ref]` and `refs`. | Call | What it does | | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET /v1/approvals` | The runs waiting for an approval the caller can give, with who asked, the build file version, tasks, accounts, source and options. | | `GET /v1/approvals/` | One of them. | | `POST /v1/approvals//approve` | Approve the accounts the caller uses; the run is queued when all are covered (`already-approved` is `409`). The body is optional: `{ "reason": "..." }`. | | `POST /v1/approvals//reject` | End the run; the reason (up to 500 characters) becomes its `error`. | What a caller may not approve does not exist for them (`404`), and a run that is no longer waiting is `409`. Each call is logged with who decided and the result, and the decision is an event of the run, so the one who asked sees who approved it. `hammerkit remote accounts` shows the caller's levels. The command line has [`remote approvals`, `approve` and `reject`](/docs/cli/remote#approving-runs). ## Changing entitlements [#changing-entitlements] `kill -HUP ` (in a container `docker kill -s HUP `) reads the config file again and takes over its `entitlements`: who may use which account, which repositories and which build files changes for the next request, the server keeps running and no run is touched. Entitlements that name an account or issuer the running server does not have are refused and the old ones stay. Everything else (accounts, issuers, providers, sources, limits) is read at start only, the server says which keys of the file differ and need a restart. As SIGHUP is also what a closed terminal sends, closing the terminal of a server started in it does not stop it: stop it with Ctrl-C or `SIGTERM`. SIGHUP is not available on Windows, where closing the console stops the server. # Overview (/docs/remote-compute) Experimental. The config and the API can still change between releases, see the [remote compute spec](https://github.com/no0dles/hammerkit/blob/master/specs/remote-compute/spec.md). A hammerkit server runs tasks for other people and agents on a machine that has what the work needs: a Mac mini for iOS builds, a Linux box with big CPUs, the credentials to deploy. Callers never get those credentials, and they can not change what a task does: they name a published build file by its hash, the tasks to run and where the code comes from, and the server decides with its config whether they may. ## How it fits together [#how-it-fits-together] 1. **Who calls.** Every call carries an OAuth token of an identity provider the server trusts. The server knows people, CI jobs and agents by the claims of that token only, see [Authentication](/docs/remote-compute/authentication). 2. **What they may do.** [Entitlements](/docs/remote-compute/entitlements) map callers to service accounts (`use` or only `request`), to the repositories they may run from and to what they may do with build files (`run`, `propose` or `publish`). 3. **Which credentials.** [Accounts](/docs/remote-compute/accounts) are the service accounts of the operator, with the secret provider commands that read secrets as them. The secrets of a published build file say which account each is read as, so the account of a run follows from its tasks, never from the request. 4. **What runs.** A run executes tasks of a [published build file](/docs/remote-compute/build-files), pinned to a version, so whoever may run can not change what a task does. 5. **Which code.** The code comes from a git ref of a repository the server reads itself, see [Sources](/docs/remote-compute/sources), from files the caller uploads, or from a [sandbox](/docs/remote-compute/sandboxes): a checkout of the caller's that stays on the server. 6. **Where it runs.** [Runs](/docs/remote-compute/runs) are queued, limited per caller and executed in a process of their own on the [backend](/docs/remote-compute/backend). A [service run](/docs/remote-compute/service-runs) keeps services up for a while instead, with their ports at addresses of their own behind the server's sign-in. ## Starting a server [#starting-a-server] [`hammerkit server`](/docs/cli/server) reads its own config file, so it needs no build file. ```bash hammerkit server --config /etc/hammerkit/server.yaml ``` The server runs until it is interrupted. `HAMMERKIT_SERVER_CONFIG` can stand in for `--config`. ## Setting up a server [#setting-up-a-server] For coding agents on a GKE cluster, [Server on Kubernetes](/docs/guides/agents/kubernetes) does all of this step by step. The pieces, in the order an operator sets them up. 1. **Identity provider.** Register an API audience for the server and list the issuer under `issuers` ([Authentication](/docs/remote-compute/authentication)). Nothing else identifies a caller. 2. **Accounts.** Define one service account per set of credentials a build needs, and the secret provider commands that read them ([Accounts](/docs/remote-compute/accounts)). A production account is limited to `source: [ref]` and the branches it may run from. 3. **Repositories.** List the repositories the server may read under `sources`, with a token that a provider command mints, for example from a GitHub App ([Sources](/docs/remote-compute/sources)). 4. **Entitlements.** Say who may `use` or `request` which account, which repositories they may name and whether they may `run`, `propose` or `publish` build files ([Entitlements](/docs/remote-compute/entitlements)). 5. **Start it.** `hammerkit server --config server.yaml`, behind a TLS proxy or with `tls`. Send `SIGHUP` to read the entitlements again without a restart ([Changing entitlements](/docs/remote-compute/entitlements#changing-entitlements)). A small config for one team on a Linux box, with a GitHub App token and a production account that needs a second person: ```yaml title="server.yaml" listen: { host: 0.0.0.0, port: 8480 } behindTlsProxy: true issuers: - { issuer: https://login.example.com, audience: hammerkit, clientId: hammerkit-cli } secretProviders: gsm: { command: [gcloud, secrets, versions, access, latest, '--secret={{ref}}'] } github-app: { command: [/usr/local/bin/github-app-token, '{{ref}}'] } accounts: scm-read: { providers: { github-app: { env: { APP_KEY_FILE: /etc/hammerkit/app.pem } } } } op-payments-dev: { providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/dev.json } } } } op-payments-prod: providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/prod.json } } } source: [ref] refs: [main] sources: - match: github.com/corp/* secret: { from: 'github-app:payments', account: scm-read } runs: env: { allow: [PATH, DOCKER_HOST] } entitlements: - group: { iss: https://login.example.com, name: platform } accounts: { op-payments-dev: use, op-payments-prod: use } repos: [github.com/corp/*] builds: publish - group: { iss: https://login.example.com, name: eng-payments } accounts: { op-payments-dev: use, op-payments-prod: request } repos: [github.com/corp/payments] builds: run ``` A developer then registers and signs in with [`remote add` and `remote login`](/docs/cli/remote), someone in `platform` publishes the build file with `remote publish`, which prints its hash, and a run is a `POST /v1/runs` naming that hash (`remote run` does not exist yet). A run that reads the production secret waits in `remote approvals` until a member of `platform` approves it. ## Config [#config] The file belongs to whoever operates the server and is reviewed like code. Unknown keys are an error, so a typo is not silently ignored. ```yaml title="server.yaml" listen: host: 127.0.0.1 # default port: 8480 # default tls: cert: /etc/hammerkit/tls.crt key: /etc/hammerkit/tls.key issuers: - issuer: https://login.example.com audience: hammerkit groupsClaim: groups # default clientId: hammerkit-cli scope: openid api://hammerkit/.default - issuer: https://token.actions.githubusercontent.com # CI jobs, matched by claims audience: hammerkit secretProviders: # the commands that read secrets, see Secret providers gsm: command: [gcloud, secrets, versions, access, latest, '--secret={{ref}}'] accounts: # the service accounts a secret can be read as op-payments-dev: description: Payments, test secrets providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/payments-dev.json } } } op-payments-prod: description: Payments, production providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/payments-prod.json } } } source: [ref] # only from reviewed code refs: [main, 'release/*'] # and only from these branches scm-read: # reads the repositories, no entitlement may name it description: Read-only access to source repositories providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/scm-read.json } } } entitlements: - subject: { iss: https://login.example.com, sub: 4f1c2b } accounts: { op-payments-dev: use, op-payments-prod: request } builds: run - group: { iss: https://login.example.com, name: release-managers } accounts: { op-payments-prod: use } builds: publish - claims: iss: https://token.actions.githubusercontent.com match: { repository: corp/payments, ref: refs/heads/main } accounts: { op-payments-dev: use } repos: [github.com/corp/payments] builds: publish sources: # the repositories a run may get its code from - match: github.com/corp/* # an organisation secret: { from: 'gsm:github-readonly-token', account: scm-read } - match: gitlab.corp.com/platform/** # a group with its subgroups secret: { from: 'gsm:gitlab-group-token', account: scm-read, username: oauth2 } runtimes: [docker] # what tasks may run on, docker by default approvals: timeout: 1h # default maxWaiting: 100 # default cooldown: 10m # default, a rejected version can not be proposed again before storage: directory: /var/lib/hammerkit # default ~/.hammerkit/server runs: maxConcurrent: 4 # default keep: 200 # default limits: # per caller (issuer and subject), 429 over it waiting: 10 # default, runs that wait for an approval queued: 20 # default, runs that wait for a free place running: 2 # default, runs that execute at once, the others wait in the queue maxDuration: 1h # default, a longer run is stopped and fails maxLogBytes: 1048576 # default, log output kept per run env: allow: [PATH, DOCKER_HOST] # the only variables of the server a run sees, none by default set: { CI: 'true' } network: open # default, the most any task reaches besides its needs backend: type: host platform: { os: macos, arch: arm64 } # detected when left out ``` | Key | What it does | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `listen` | Interface and port. Port `0` picks a free one. | | `tls` | Certificate and key to serve https. | | `behindTlsProxy` | Set to `true` when a proxy in front terminates TLS. | | `issuers` | The OAuth issuers whose tokens are accepted, see [Authentication](/docs/remote-compute/authentication). | | `secretProviders` | The commands that read secrets, see [Accounts](/docs/remote-compute/accounts#secret-providers). | | `accounts` | The service accounts a secret can be read as, see [Accounts](/docs/remote-compute/accounts). | | `entitlements` | Who may use or request which account, which repositories and what they may do with build files, see [Entitlements](/docs/remote-compute/entitlements). | | `approvals` | How long requested runs and proposals wait, see [Approvals](/docs/remote-compute/entitlements#approvals). | | `sources` | The repositories a run may get its code from, and the credential the server reads and pushes them with, see [Sources](/docs/remote-compute/sources). | | `runs` | Concurrency, limits per caller, durations, the environment and the network ceiling of runs, see [Runs](/docs/remote-compute/runs#config). | | `sandboxes` | The runtimes, network and limits of sandboxes, see [Sandboxes](/docs/remote-compute/sandboxes#config). | | `endpoints` | The domain, certificate and sign-in of the endpoints of service runs, see [Service runs](/docs/remote-compute/service-runs#server-config). | | `runtimes` | What tasks may run on, see [Backend](/docs/remote-compute/backend#runtimes). | | `backend` | Where tasks run and what the machine offers, see [Backend](/docs/remote-compute/backend). | | `storage` | Where the server keeps what must survive a restart: the published build files, the uploaded source files, the sandboxes (`sandboxes/`), the task cache and the records of the runs (`records/`). Runs work in `runs/` below it while they run. | Plain http carries OAuth tokens, so it is accepted on the loopback interface only. Anywhere else the config needs `tls`, or `behindTlsProxy: true`. The server checks the references within the file when it starts: an entitlement has to name a configured issuer and defined accounts, an account only names declared secret providers, and a source reads its token with a provider the account has credentials for. ## Discovery [#discovery] `GET /.well-known/hammerkit` needs no token and returns what the server is, so a client can check it before signing in: ```json { "name": "hammerkit", "version": "1.25.0", "protocol": 1, "issuers": [ { "issuer": "https://login.example.com", "audience": "hammerkit", "clientId": "hammerkit-cli", "scope": "openid api://hammerkit/.default" } ], "backend": { "type": "host", "platform": { "os": "macos", "arch": "arm64" } }, "dashboard": "https://hammerkit.run.example.com/" } ``` `dashboard` is there when the server has a [dashboard](/docs/remote-compute/status#dashboard). `GET /healthz` answers `{"status":"ok"}` for a load balancer or a process manager. Every other path needs a token, except `/` for a browser, which goes to the dashboard. # Runs (/docs/remote-compute/runs) A run is a request to execute tasks of a published [build file](/docs/remote-compute/build-files). The server writes every run down in `storage` (`records/`, readable by the user of the server only), so a restart keeps them with their logs, see [Limits and restarts](#limits-and-restarts). Every call needs a token, and a run can be seen and cancelled only by the caller that submitted it: for anyone else its id is `404`, the same as an id that does not exist. Who decides about it, and who may open a service run's endpoints, see it too; with `observe` a caller sees every run in lists, see [Who sees what](/docs/remote-compute/status#who-sees-what). From the command line, [`remote approvals`](/docs/cli/remote#approving-runs) lists the runs that wait for you. Submitting a run from the command line (`remote run`, `--on`) is not there yet; a run is a `POST /v1/runs`. ## Submitting a run [#submitting-a-run] | Call | What it does | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `POST /v1/runs` | Submit a run. Answers `202` with the run in state `queued`, or `awaiting-approval` when the caller only has the account on [`request`](/docs/remote-compute/entitlements#approvals). | | `GET /v1/runs` | The caller's runs, newest first, without the `results` of their tasks. | | `GET /v1/runs/` | One run: `state` is `awaiting-approval`, `queued`, `running`, `succeeded`, `failed`, `cancelled` or `rejected`; a failed or rejected run has an `error`. `results` says how each task went, and `?wait=30s` answers once the run is over, see [Results and waiting](#results-and-waiting). | | `POST /v1/runs//cancel` | Cancel a run: one that is queued or waits for an approval at once, a running one when its tasks have stopped. | | `POST /v1/runs//retry` | Start a run that is over again, as a new run of yours, see [Retrying a run](#retrying-a-run). | | `GET /v1/runs//events` | The run's events as server-sent events, see [Events](#events). | | `GET /v1/runs//logs` | The run's output as numbered lines, with a tail, a range and a search, see [Logs](#logs). | ```json { "definition": "sha256:9f2c...", "tasks": ["build", "test"], "source": { "type": "ref", "repo": "github.com/corp/payments", "ref": "main" }, "options": { "env": "staging", "cache": "checksum", "timeout": "30m" } } ``` * `definition` is the hash of the published [version](/docs/remote-compute/build-files#versions) to run, as `remote publish` printed it. The run is pinned to it, so what it executes does not change under it. A hash that was not published is `400`, a yanked one `409`. * `tasks` are task names as on the command line, up to 100. A task the build file does not have is `400`. Instead of names, labels in `options.filter` can select the tasks. The tasks named run as one plan, like those a label selects: side by side as far as their dependencies allow, in no particular order, with the [longest remaining chain](/docs/task/dependencies#which-ready-task-runs-first) first. What a task usually takes comes from its last successful runs in the repository. To run one task after another, make it depend on the other. * There is no account in the request. The run uses the accounts of the secrets of the tasks it executes, with their dependencies and the services they need, derived from the pinned version, and the answer lists them as `accounts`. A caller who has no level on one of them gets `403` naming the account, one who may only `request` it gets a run in `awaiting-approval`, and a caller without `builds` level may not run. * `source` says where the code comes from: `{ "type": "ref", "repo": "github.com/corp/payments", "ref": "main" }` for a branch or tag of a repository (or a full commit hash) that the server checks out itself. The server pins the ref to the commit it is at when the run is submitted; with `"commit": "4be1a07c..."` (the full hash, 40 or 64 hex digits) the ref has to be at that commit, else the run is refused with `409` (`main is at 9e1f... now, not at 4be1a07c...`), so a CI job runs the commit it was started for and not one pushed after it. Or `{ "type": "upload", "snapshot": "sha256:..." }` for the caller's working tree, announced and sent beforehand, see [Source files](#source-files), or `{ "type": "sandbox", "id": "sbx_..." }` for a [sandbox](/docs/remote-compute/sandboxes) of the caller's. A snapshot that is unknown, or that still lacks files, and a sandbox that is not the caller's, are `400`. The kinds the accounts of the run list in `source` (and, for `refs`, the branches) are the only ones allowed, anything else is `403`. * `options` are the settings of `hammerkit run` that make sense on a server, all optional: `env`, `filter` and `exclude` (labels), `cache` (`checksum`, `modify-date` or `none`), `cacheReadOnly`, `timeout` (for example `30m`, for tasks without a timeout of their own), `maxTimeout` (for every task, also one with a longer timeout of its own, see [Capping every task](/docs/task#capping-every-task)) and `skipDeps`. A run as a whole is still stopped after the server's `runs.maxDuration`. How many tasks run in parallel and how much of the machine they use stay with the server, and `watch` and the log mode are for terminals only, so they are not accepted. * `options.args` are [arguments](/docs/task/arguments), what `hammerkit run -- ...` passes after `--`. See [Arguments](#arguments). The request names the work, it does not contain it: what the tasks do comes from the published build file, and the files they work on come from `source`. A `ref` is fetched by the server. An `upload` refers to a snapshot of the working tree that is sent separately, so the request stays small however large the project is. The body is JSON up to 64 KiB, and unknown fields are rejected. Each submitted run is logged with who submitted it, the accounts and the source. What `POST /v1/runs` answers, in the order the server checks: | Status | When | | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `202` | The run is accepted, `queued` or `awaiting-approval`. | | `400` | The body is malformed or has unknown fields, the build file or version does not exist, a task is unknown, arguments go to anything but one task that reads no secret or server credential and takes arguments, the repository is not readable by the server or the ref is not one branch or tag, the snapshot is unknown or incomplete, or the sandbox is not the caller's. | | `401` | The token is missing or invalid. | | `403` | The caller has no `builds` level, may not run from that repository, has no level on an account of the run, or the account refuses that source or ref. A run on a sandbox also needs its repository and branch in the `sandboxes` of an entitlement. | | `409` | The version was yanked, was published before accounts were recorded, nobody may approve an account the caller can only request, or the ref is no longer at the `commit` the request names. | | `413` | The caller's [sandboxes](/docs/remote-compute/sandboxes) take more than their limit (runs on a sandbox only). | | `429` | The caller is over their [limits](#limits-and-restarts). | | `501` | The run is from an `upload`, which the executor does not support yet. | | `502` | The server could not reach the repository to resolve the ref. | | `503` | The server is shutting down, or too many runs wait for an approval. | ## Arguments [#arguments] A run can pass arguments to one task, as `hammerkit run test -- src/login.spec.ts -t "login flow"` does locally: ```json { "definition": "sha256:9f2c...", "tasks": ["test"], "source": { "type": "ref", "repo": "github.com/corp/payments", "ref": "main" }, "options": { "args": ["src/login.spec.ts", "-t", "login flow"] } } ``` They are appended to the last command of that task, each quoted for its shell, so shell syntax in an argument stays text. Its dependencies and the services it needs get none. The run's `plan` shows the command as it runs, so the caller sees `npx vitest run 'src/login.spec.ts' '-t' 'login flow'`, and a published version shows the last command of each task as `command`, so a caller knows where the arguments land. * `tasks` names exactly one task; labels can not select it. Several tasks, a service, or only labels with arguments are `400`. * Up to 100 arguments, 16 KiB together, and none with a NUL character. * A run in which any task or service reads a secret, or takes a variable of the server into its `envs` (`$NAME` or `${NAME}` of [`runs.env`](#config)) other than `PATH`, `HOME`, `CI` and the [`HAMMERKIT_RUN_` variables](#executing-a-run), passes no arguments. That covers the task itself, its dependencies and the services it needs: an argument could make the command print what the task holds, and what a dependency wrote into its outputs or a service answers, in a form no mask of secrets knows (`rev token.txt`). The run is `400` naming the item and what it reads, and the executor checks again before it starts. A shell task (`cmds: [sh -c]`) on a server therefore runs without any account or credential of the server, nor next to one. * A task with [`args: false`](/docs/task/arguments#which-tasks-take-arguments) takes none: the run is `400`. With `runs.args: declared` in the [config](#config) the server passes arguments only to the tasks that say `args: true`, any other task is `400` as well. Use it when a published build file holds tasks nobody wrote for arguments: one of them that takes them runs whatever its last command does with what a caller appends, so each would need `args: false`, and a new task that forgets it takes them again. With `declared` the build file names the tasks that take them, typically the [shell task](/docs/remote-compute/agents#the-shell-task) of agents and a test task, and a reviewer reads `args` in the [plan](/docs/remote-compute/build-files) of each. * Such a run never uses an account, so it never waits for an approval. * [`remote approvals`](/docs/cli/remote#approving-runs) still lists the arguments of a run as JSON strings, and `remote show` prints every command whole, never cut, with control and bidi characters written out. * On a server that allows the `local` runtime, arguments to a local task run as commands on the server's machine, as the code of the ref already does. * A server never takes `--set`; variables stay what the published build file says. As [locally](/docs/task/arguments), the task with arguments skips the cache: it reads no cache entry and stores none, and a later run without arguments doesn't take its outputs for a cache hit. Naming a method in `options.cache` (`"cache": "checksum"`) keeps its caching, as `--cache` before `--` does. Its dependencies cache as always. ## Results and waiting [#results-and-waiting] A run says how each of its tasks went in `results`, by task name, from the moment the task starts. A client reads this first, and only reads the [log](#logs) when the last lines are not enough: ```json { "id": "run_4vJ0mT2cX9qLw1aB", "state": "failed", "error": "test failed", "results": { "build": { "status": "succeeded", "startedAt": "2026-10-12T20:47:14.569Z", "finishedAt": "2026-10-12T20:47:14.588Z", "durationMs": 42, "cache": { "hit": false, "causes": ["source changed: src/index.ts"] }, "commands": [{ "startedAt": "2026-10-12T20:47:14.571Z", "finishedAt": "2026-10-12T20:47:14.585Z" }], "lastLog": ["compiled 12 files"] }, "test": { "status": "failed", "startedAt": "2026-10-12T20:47:14.589Z", "finishedAt": "2026-10-12T20:47:14.616Z", "durationMs": 70, "exitCode": 1, "cache": { "hit": false }, "lastLog": ["FAIL src/index.spec.ts", " expected 1 to be 2"] } } } ``` | Field | What it says | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `status` | `running`, `succeeded`, `failed`, `cached` (taken from the cache, nothing ran), `skipped` (a dependency no task of the run needed, because every task depending on it was cached) or `cancelled` (stopped with the run, or never started because another task failed). A task the run names that is checked again as a dependency of a later one keeps what it did the first time. | | `startedAt`, `finishedAt` | When the server heard that the task started and ended. A cached task has no `startedAt`. | | `durationMs` | How long the task took, as the engine measured it. | | `exitCode` | The exit code of the command that failed. | | `error` | Why a task failed when no command exited, for example `timed out after 30m`. Masked like the log, up to 2 KiB. | | `cache` | `hit: true` for a cached task. For a task that ran, `causes` says why it was not taken from the cache, as [`--explain`](/docs/cli/explain) would: `never cached`, `source changed: `, `command changed`, `no src declared (uncacheable)` and so on, at most 20 and then `… and N more`. | | `commands` | When each command of a task that ran started and exited, the first 50, once the task is over: the rest of the task's time went to its container, its caches and, on Kubernetes, the pod's scheduling, image and init containers. Kubernetes gives the times of a pod in seconds. Left out where the runtime does not say, such as for a cached task or a shell in a sandbox. | | `lastLog` | The last 20 lines the task printed, stdout and stderr as they came, the newest last, each cut at 1000 characters. They are kept even when the run's log was [cut off](#limits-and-restarts), so the end of a long log, where the error usually is, is never lost. | What the run did before its tasks is in `phases`, in their order, each with its `startedAt` and `finishedAt`: `source` got its files (the checkout of the commit, or those of the sandbox), `namespace` prepared its place on a Kubernetes cluster, and `plan` started the process of the run and read its build files. The time after `plan` until the first task went to the keys of its tasks, computed from their files. A run of an older server has no `phases`. ```json "phases": [ { "phase": "source", "startedAt": "2026-11-14T07:44:16.766Z", "finishedAt": "2026-11-14T07:44:17.916Z" }, { "phase": "namespace", "startedAt": "2026-11-14T07:44:17.916Z", "finishedAt": "2026-11-14T07:44:18.166Z" }, { "phase": "plan", "startedAt": "2026-11-14T07:44:18.216Z", "finishedAt": "2026-11-14T07:44:19.416Z" } ] ``` `GET /v1/runs/?wait=30s` holds the answer until the run is over (`succeeded`, `failed`, `cancelled` or `rejected`) or the time is up, and then answers the run as it is. A client that can not keep a stream of events open, such as a tool call of an agent, waits with it and asks again while the run is not over. The wait is a duration like `500ms` or `30s` (leave `wait` out not to wait, `0s` is not a wait), at most 50 seconds (a longer one is cut to 50 seconds, so the answer comes before a proxy in front of the server closes the quiet connection, which nginx does after 60); anything else is `400`. A run that waits for an approval answers when the time is up, or when the server stops. ## Retrying a run [#retrying-a-run] `POST /v1/runs//retry` (no body), `hammerkit remote retry ` or the retry button on the run's page in the [dashboard](/docs/remote-compute/status#dashboard) starts a run that is over (succeeded, failed, cancelled or rejected) again. The new run is yours, whoever started the first one, and names it in `retryOf`. It asks for what the first one asked for: the same published build file, the same tasks or services, settings and arguments, from the same source. * **Who may retry it:** who may read the run whole, its owner, who decided about it, and `observe`, since a retry runs its arguments again. `may.retry` says so on the run. * **How it is decided:** as a run you start yourself. Your entitlements say whether you may run from its repository, which accounts you may use, and whether one needs an approval. Nothing carries over from the first run: an approval it got is not yours, so a retry of a run that needed one waits for one again, and the server resolves its secrets as for any run. * **What it runs:** a run from a branch or tag runs the commit the first one ran. When the branch moved on since, the retry is refused with `409` and says so (for example `main moved to since ran , start a new run from main`): the accounts of a branch never run a commit that is no longer on it. A run on a [sandbox](/docs/remote-compute/sandboxes) runs on the files the sandbox holds now, and only its owner may retry it. A run of uploaded files is not retried. * **Which tasks:** all the tasks the first run named, not only the ones that failed. What passed is taken from the cache, so it costs little, and the run is whole again in one place. * **A service run** starts again with its services, name, endpoints and the `ttl` and `sleepTtl` it had, within today's `runs.up.maxTtl`. While a service run of yours with that name is not over, the retry is refused (`409`): a new run of the name would replace it and its volumes, so stop it first. A run that is not over yet is not retried (`409`): stop it first. ## Logs [#logs] `GET /v1/runs//logs` reads the output of a run as numbered lines, as far as the server kept it. Without parameters it answers the first 200 lines: ```json { "lines": [{ "line": 2, "task": "test", "stream": "stdout", "text": " expected 1 to be 2", "match": true }], "total": 214, "next": 3, "truncated": false } ``` | Parameter | What it does | | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `task` | Only the lines of this task. The lines are numbered from 1 over what `task` selects, so a later call with the same `task` reads around a line it found. Without it, every task's lines, and the server's own (`hammerkit`, the checkout). | | `tail` | The last lines, up to 1000. With `grep`, the last lines of what it finds: the matches and their context, counted as lines. | | `from` | The first line to read, by its number. Not together with `tail`. | | `limit` | How many lines one answer has, up to 1000: 200 by default, or as many as `tail` asks for. When more follow, `next` is the `from` that continues; a match just before `from` still shows its lines after. | | `grep` | Only lines that contain this text. It is a fixed string, not a regular expression, so a pattern can not keep the server busy. It can not span lines. | | `i` | `true` to ignore case in `grep`. | | `context` | Lines before and after each match, up to 20, marked `"match": false`. They are lines of the `task` asked for; without `task`, the lines around the match in the whole log, of any task. | `total` counts the lines `task` selects, `truncated` says the log of the run was cut off at `runs.maxLogBytes` and its end is not kept (the [results](#results-and-waiting) still have the last lines). An unknown parameter or a value out of range is `400` and says which, a run of somebody else's is `404`. A read costs the lines it answers, not the length of the log: the server notes where the lines are once, as the log grows. An agent that ran the tests reads `results.test.lastLog` first, then for example `?task=test&grep=FAIL&context=5` to see every failing test with what it printed around it. ## Events [#events] The events carry the sequence number as their `id`, so a client that lost the connection continues with the `Last-Event-ID` header (or `?after=`) and misses nothing. The stream replays what happened and then follows the run until it is over; a finished run's events are replayed and the stream ends. Event types are `state` (`awaiting-approval`, `queued`, `running` and the final state, with the `error` of a failed run), `approval` (who approved or rejected the run), `task` (a task started or ended, with the `state` a [result](#results-and-waiting) has and, once it ended, `durationMs`, `exitCode`, `error` and `causes`) and `log` (a line of a task's output). A run that produces more than 5000 events is cut off with one `truncated` event; the state changes are always kept, and so are the results of the tasks, while their own events are cut off with the log. ## Source files [#source-files] A run from an `upload` works on the caller's files. They are sent in two steps so that only what the server does not have yet travels: the caller announces the tree as a list of files with their digests, the server says which digests it lacks, and the caller sends those. The server takes the files already, but does not execute runs from an upload yet: such a run is refused with `501`. ``` POST /v1/sources ``` ```json { "files": [ { "path": "package.json", "sha256": "9f2c...", "size": 812 }, { "path": "bin/run.sh", "sha256": "41ab...", "size": 233, "mode": "exec" } ] } ``` The answer names the tree, a **snapshot**, and says what is missing: ```json { "snapshot": "sha256:5d1e...", "files": 2, "bytes": 1045, "createdAt": "2026-10-11T11:00:00.000Z", "missing": ["9f2c...", "41ab..."], "complete": false } ``` The snapshot is the `sha256:` of the JSON `[[path, sha-256, mode], ...]` with the paths sorted, the same construction as the version of a build file, so the same files have the same name anywhere. It answers `201` for a new tree and `200` for one the server already knows. Then each missing file is sent on its own: | Call | What it does | | -------------------------------- | ------------------------------------------------------------------------------------------------- | | `POST /v1/sources` | Announce a tree. Up to 20,000 files, 32 MiB per file, 1 GiB per tree. | | `PUT /v1/sources/blobs/` | Send the content of one file as the request body. Answers `201`, or `200` when the server had it. | | `GET /v1/sources` | The trees the caller keeps, what they take and the limit. | | `GET /v1/sources/` | One tree and the digests still missing; `complete` when none is. | | `DELETE /v1/sources/` | Forget a tree and the content no other tree of the caller needs. | A second run after one edited file announces the tree again and sends that file only. A client can also skip the announcement when the snapshot it computes is the one it used last and `GET /v1/sources/` says it is complete. * **Only the caller's.** Trees and content belong to the caller who sent them; another caller's snapshot is `404`. Content is never shared between callers: a server that answers "I already have this file" to anybody would tell them what other people work on. * **Only announced content.** A file is taken only if a tree the caller announced lists its digest, with the size announced, and only if the content has that digest. Anything else is `400`. * **Credentials do not leave.** A tree with a path that looks like a credential is refused with `400` that names it, even when a `src` pattern reaches it: `.env` (not `.env.example`, `.env.sample`, `.env.template`), `.npmrc`, `.netrc`, `.pypirc`, `.git-credentials`, private keys (`id_rsa` and its kin, `*.pem`, `*.p12`, `*.pfx`) and anything below `.git/`, `.ssh/`, `.aws/`, `.gnupg/` or `.kube/`, in any case and under the names Windows also opens (`.SSH`, `.git.`, `GIT~1`). Secrets belong in a secret provider; there is no override. * **Quota.** A caller may keep 4 GiB of files, files still being sent included; a file that would go over is `413`. A caller may keep 1,000 trees, whose announcements take at most 64 MiB together; a tree over either is `413`. Nothing is removed on its own, a caller deletes what they no longer need. * **Who may.** Callers with a `builds` level; others get `403`. An account can further restrict its `source`. Paths are relative to the root of the project, forward slashes, nothing that climbs out; `inputs` of a [build file version](/docs/remote-compute/build-files) say which files a run needs. ## Executing a run [#executing-a-run] The server runs the tasks of an accepted run on its own machine, with the same engine as `hammerkit run`, so the `local` and `docker` runtimes and the task cache work as they do there. Each run executes in a process of its own (`hammerkit server-run`, started by the server): 1. It fetches the ref of the repository, as pinned to its commit when the run was submitted (see [Pinning](/docs/remote-compute/sources#pinning)), into a directory made for the run. A ref that is not a plain branch, tag or commit name is refused. 2. It writes the files of the published version into a directory of the run that no task mounts, and over the checkout, so the tasks see the published build files whatever the repository says. A file that would land in `.git` (in any spelling a case-insensitive file system reads as that) or behind a symbolic link of the checkout is refused. 3. It starts the process of the run in the published files, which it reads every build file from, includes too: a task that writes a build file into the checkout, or a build file of the repository beside an included one, changes nothing of what the run executes. The tasks work in the checkout. It runs the tasks named (or the ones the labels select) as one plan, side by side as far as their dependencies allow, the [longest remaining chain](/docs/task/dependencies#which-ready-task-runs-first) first; once one fails it starts no new task and the run fails. It reports each task and its output as events. Cancelling the run tells the process so on its standard input, the same on every platform, so the engine stops its tasks and removes its containers; the process is killed if it is still running after 15 seconds. On Windows the engine stops a local task by ending its shell, so a program the shell started can keep running; the directory it holds is then left and reported (step 4). 4. It removes the directory, whether the run succeeded, failed or was cancelled. A directory that can't be removed (a file still in use, for example) is left in place and reported in the run's log and the server's log; the run still ends the way it ended. The process starts with an environment the server builds, not with its own: the variables `runs.env.allow` lists, plus `runs.env.set`, and a `HOME` that is an empty directory of the run's alone. So what the engine reads of its process (the docker host, registry logins in the home directory, cloud credentials) is what the operator gave runs and nothing else; a private registry needs `DOCKER_CONFIG` in the allowlist, pointing to a login the operator chose for runs. Tasks do not get the environment of the process either, only their own `envs` and the secrets they declare. The server also sets what the run is from, which a task or service reads into its `envs` like any variable of the process: | Variable | Value | | ----------------------- | --------------------------------------------------------------------------------- | | `HAMMERKIT_RUN_REPO` | the repository, such as `github.com/corp/payments` | | `HAMMERKIT_RUN_REF` | the branch or tag the run was submitted for, as given (not set for a commit hash) | | `HAMMERKIT_RUN_COMMIT` | the commit the run was pinned to (not on a sandbox, whose files may hold changes) | | `HAMMERKIT_RUN_SANDBOX` | the sandbox of a run on a sandbox | ```yaml tasks: preview: envs: BRANCH: ${HAMMERKIT_RUN_REF:-local} cmds: - ./deploy-preview.sh "$BRANCH" ``` So a task that reads a secret, which takes no arguments on a server, still knows which branch it deploys. They hold no credential: a task that reads them takes arguments and keeps its network as before. The server sets them alone: a config that names one in `runs.env` doesn't load. Outside a server they are not set, hence the default above. A ref may hold `/`, `.` and upper case and be up to 200 characters long, which is no valid image tag, DNS label or Kubernetes label. Turn it into one in the command, for example the start of its hash: `printf %s "$BRANCH" | sha256sum | cut -c1-12`. The plan of a run, which its approver reads, is made with the run's values. A value is part of the task's cache key, so a task that reads `HAMMERKIT_RUN_COMMIT` runs again for every commit. A task that reads any of them, or depends on one that does, records no [check](/docs/remote-compute/checks): its result says nothing a CI job's `remote verify`, which has no run, could look up. Keep them out of the tasks CI verifies. The task cache is kept below the server's `storage` directory, one cache for the runs of a repository and one per caller for uploads and for sandboxes. A run from an upload or a sandbox can therefore not leave an entry that a run from a reviewed repository finds. A task is keyed by its definition and its sources, not by where the checkout is. A run fails with the reason when the server does not list the repository, when the ref can not be fetched or has moved, when it comes from an `upload`: uploads are not supported by the executor yet, use `source: ref`. Cancelling a run stops the fetch or the running task. Before it starts, the executor derives the accounts of the run again from the pinned files with the run's environment, and fails the run if it would use one it was not admitted for. The secrets of the tasks are read as the accounts of the server config, by the provider commands, and a secret of any other account is refused when it is fetched. A run on a [sandbox](/docs/remote-compute/sandboxes#running-on-a-sandbox) skips the fetch: the published files are written to a directory of the run, and the process works on the files of the sandbox, which stay when the run is over. The process of a run is not an isolation boundary, see [One server is one trust level](/docs/remote-compute/backend#one-server-is-one-trust-level). ## Limits and restarts [#limits-and-restarts] One caller can not take the machine. Per caller (issuer and subject) the server counts the runs that wait for an approval (`runs.limits.waiting`) and the runs in the queue (`runs.limits.queued`); a submit over either is answered `429` and says which. At most `runs.limits.running` runs of one caller execute at the same time, and the next run in the queue whose caller has a place left starts first, so one caller with many runs does not keep the others waiting. On a Kubernetes backend, runs of one caller that share kept claims also run one at a time, see [Namespaces](/docs/remote-compute/backend#namespaces). [`remote status`](/docs/remote-compute/status) shows the queue, what holds each run in it, and which of your runs count against each limit. A run that executes longer than `runs.maxDuration` is stopped and fails with that reason. The log of a run is cut off at `runs.maxLogBytes` with a `truncated` event; the state changes and the results of the tasks, with their last lines, are always kept. Runs are written down when their state changes. On start the server takes them back: * finished runs keep their state, the results of their tasks, events and logs; * runs that wait for an approval wait on, for the time that was left, and are rejected when it has passed; * runs that were queued or running when the server stopped have failed with it: `the server stopped while the run was running, submit it again`. A graceful stop (SIGTERM) leaves the waiting runs waiting, and cancels the queued and running ones. A [service run](/docs/remote-compute/service-runs) that is up gives its place back: it counts against `runs.up` instead, and it is not kept over a restart either, since its services stop with the server. Changes to `runs` need a restart; only the entitlements are read again on `SIGHUP`, see [Changing entitlements](/docs/remote-compute/entitlements#changing-entitlements). ## Config [#config] ```yaml title="server.yaml" runs: maxConcurrent: 4 # default keep: 200 # default activity: everyone # default, who sees the runs of others limits: # per caller (issuer and subject), 429 over it waiting: 10 # default, runs that wait for an approval queued: 20 # default, runs that wait for a free place running: 2 # default, runs that execute at once, the others wait in the queue maxDuration: 1h # default, a longer run is stopped and fails maxLogBytes: 1048576 # default, log output kept per run env: allow: [PATH, DOCKER_HOST] # the only variables of the server a run sees, none by default set: { CI: 'true' } network: { allow: [registry.npmjs.org] } # open by default secretNetwork: none # open by default, what a task that reads a secret reaches args: declared # any by default, which tasks take arguments up: { ttl: 8h, maxTtl: 72h } # the defaults, sleepTtl unset, see Service runs checksKeep: 30d # default, how long a check is found ``` | Key | What it does | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `maxConcurrent` | Runs that execute at the same time, the others wait in queue. `4` by default. | | `keep` | Finished runs that are remembered, the oldest are dropped first. `200` by default. | | `activity` | Who sees the runs of others, see [Who sees what](/docs/remote-compute/status#who-sees-what): `everyone` (the default), a caller with a `builds` level sees who runs what in the repositories they have access to, where it waits and how long it took, never its arguments, why it failed or its log; `involved` only the runs they decide about or may open the endpoints of. | | `limits` | What one caller may have at a time, see [Limits and restarts](#limits-and-restarts). | | `maxDuration` | A run that executes longer is stopped and fails. `1h` by default. | | `maxLogBytes` | The log of a run that says more is cut off. 1 MiB by default. | | `env.allow` | The variables of the server process that a run, its provider commands and the git fetch can see. Nothing else of the server's environment is passed on, and the list is empty by default. A server that runs tasks needs `PATH` in it, Windows needs its system variables (`SystemRoot`, `ComSpec`, `PATHEXT`, `TEMP`) as well. | | `env.set` | Fixed values that are set in addition. Neither list may name a `HAMMERKIT_RUN_` variable. | | `network` | The most any task or service of a run reaches besides the services it needs: `open` (the default), `none` or `{ allow: [hosts] }`, as the [`network`](/docs/task/network) of a task. See [Network](#network). | | `secretNetwork` | The most a task or service that reads a secret, or a variable of the server other than `PATH`, `HOME` and `CI`, reaches: `open` (the default), `none` or `{ allow: [hosts] }`, within `network`. See [Tasks that read secrets](#tasks-that-read-secrets). | | `args` | Which tasks take [arguments](#arguments): `any` (the default) but those with `args: false`, or `declared`, only those with `args: true`. | | `up` | The time and limits of [service runs](/docs/remote-compute/service-runs#server-config). | | `checksKeep` | How long a [check](/docs/remote-compute/checks) that a run passed is found by `remote verify`. `30d` by default. | ## Network [#network] `runs.network` is a ceiling for every run of the server, as `hammerkit run --network` is for a run on a laptop: ```yaml title="server.yaml" runtimes: [docker] runs: network: { allow: [registry.npmjs.org, '*.githubusercontent.com'] } ``` * A version whose tasks or services ask for more is refused when it is published (or proposed): with the ceiling above, a task without `network` (which is `open`), or one that allows `pypi.org`, can't be published. The error names the task and what it asks for. A wildcard of the server covers the names under it; a name does not cover a wildcard. A name of the server covers a [TCP forward](/docs/task/network#tcp-forwards) to any port of it (`db.corp.internal:5432`); a forward covers only itself, so `db.corp.internal:5432` on the server lets a task forward to that port but not reach the name through the proxy. * So a build file published to such a server names its `network` on every task and service, which is also what a reviewer reads: the [plan](/docs/remote-compute/build-files) of a version shows each one's `network`. * Every run executes under the ceiling as well, and a version published before the ceiling was set, which asks for more, is refused when a run of it is submitted. * The services a task needs are reachable at every level. * Local tasks can't be limited on the server's machine, so a server with a ceiling below `open` allows only the `docker` runtime; `runtimes: [docker, local]` with it is a config error. On a [Kubernetes backend](/docs/remote-compute/backend#network) a local task runs in a pod of a pool's tool image and is limited like a container task, so there it is allowed, and a local task may name its own `network`. * On a Kubernetes backend the ceiling is a NetworkPolicy on every pod, so it needs `backend.networkPolicy: enforced`; without that it is a config error. A task that reads a provider's secret there can be limited only when the server names the provider's endpoints, see [Secrets in the Job](/docs/remote-compute/backend#secrets-in-the-job). ### Tasks that read secrets [#tasks-that-read-secrets] A task that reads a secret runs its tools next to it: `npm publish` with a token, a deploy with a cloud credential. A tool that was changed, a dependency's install script or a compromised release, can send the value anywhere the task reaches. `runs.secretNetwork` holds every task and service that reads a secret, or takes a variable of the server into its `envs` (other than `PATH`, `HOME` and `CI`), to less than the others: ```yaml title="server.yaml" runs: secretNetwork: { allow: [registry.npmjs.org] } ``` ```yaml title=".hammerkit.yaml" tasks: publish: image: node:24-alpine deps: [build] secrets: - { from: 'gsm:npm-token', account: release, env: NPM_TOKEN } network: { allow: [registry.npmjs.org] } cmds: [npm publish] ``` * It is checked as `runs.network` is: a version in which such an item asks for more is refused when it is published, proposed and when a run of it is submitted, naming the item and what it reads. The other tasks keep what `runs.network` allows. * `open`, the default, holds nothing beyond `runs.network`, so existing build files keep working. `none` suits a server whose secret tasks only talk to the services they need. * It limits the task while it holds the value, not what it hands on: a task that writes the secret into its outputs, or a service it needs that has an open network, takes it further. Those are in the build file, which a reviewer reads. * An allow list is no wall around a shared host: a task that may reach `*.googleapis.com` reaches another project's buckets there too. Allow the names a task needs, not their domain. * Local tasks on the server's machine can't be limited, so with `secretNetwork` below `open` a local task that reads a secret is refused, except on a Kubernetes backend. # Sandboxes (/docs/remote-compute/sandboxes) A sandbox is a caller's own checkout of a repository on the server, which runs work on. It is for an agent without a machine of its own, or a person whose laptop should not build: the files stay on the server between runs, so what `npm install`, a formatter or a code generator wrote is there for the next run. It is not for production, see [Accounts](#accounts). The published build file always runs. A sandbox only supplies the files: its own `.hammerkit.yaml` is just a file, and changing it changes nothing until a new version is [proposed and published](/docs/remote-compute/build-files). This page covers creating, listing, moving, deleting, running on sandboxes, reading, searching and editing their files, and pushing their changes, on a host and on a [Kubernetes backend](#on-kubernetes). [Agents](/docs/remote-compute/agents) covers the clients that let a coding agent work on one. ## Who may keep one [#who-may-keep-one] The `sandboxes` of an [entitlement](/docs/remote-compute/entitlements) list the repositories the caller may keep sandboxes of, one `repo` (`host/path` pattern) per entry, and give each branch pattern of it a type: * `pull`: a sandbox may start from a matching branch, and fetch one. * `push`: `pull`, and [pushing](#pushing) to the branch. Without `sandboxes` the caller keeps none. It is a list of its own: `repos` are for runs from a ref only, and a sandbox needs no `repos` entry for its repository. The entries of all of the caller's entitlements whose `repo` covers a repository count, and where several patterns match a branch, the higher type wins. An entry's branches never count for another repository. The server has to read the repository too, the same check as for a [run from a ref](/docs/remote-compute/sources). ```yaml title="server.yaml" entitlements: - subject: { iss: https://login.example.com, sub: agent-7 } accounts: { op-payments-dev: use } repos: [github.com/corp/payments] builds: run sandboxes: - repo: github.com/corp/payments branches: 'agent/**': push # agent/ and everything below it main: pull - repo: github.com/corp/* branches: { main: pull } ``` A branch pattern is a branch name, where a segment may hold `*` (any characters but `/`) and the last one may be `**`, everything below. A sandbox belongs to the caller who created it. For anybody else its id is `404`, the same as an id that does not exist, and it does not show in their list. ## Creating and managing [#creating-and-managing] | Call | What it does | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `POST /v1/sandboxes` | `{ "repo": "github.com/corp/payments", "ref": "main" }`: fetch a branch into a new sandbox. Answers `201` with the sandbox. A sandbox starts from a branch, never from a tag or a commit, and keeps the commit the branch pointed to. | | `GET /v1/sandboxes` | The caller's sandboxes, oldest first, with their `limits` and what they `used`. | | `GET /v1/sandboxes/` | One sandbox. | | `POST /v1/sandboxes//fetch` | `{ "ref": "next" }`: move a sandbox without changes to the commit a branch points to now. | | `POST /v1/sandboxes//push` | `{ \"branch\": \"agent/fix-login\", \"message\": \"Fix the login\" }`: commit the changes and push them, see [Pushing](#pushing). | | `DELETE /v1/sandboxes/` | Cancel the runs on the sandbox, wait for them to stop and remove its files. | ```json { "id": "sbx_Jp0dY8m2kQ1aZt5x", "repo": "github.com/corp/payments", "ref": "main", "commit": "4f1c2a...", "bytes": 18352211, "createdAt": "2026-10-12T21:00:00.000Z", "lastUsedAt": "2026-10-12T21:10:00.000Z", "busy": true, "run": "run_Wm3o0yN2e4cR9pLs" } ``` After a [push](#pushing), `ref` is the branch it pushed to, `commit` what it pushed, and `pushed` lists each branch it pushed with the commit it pushed there last. Creating a sandbox is a fetch, not a run: it runs no task, holds no account and takes no place of the [run limits](/docs/remote-compute/runs#limits-and-restarts). The server reads the repository with the credential of its [sources](/docs/remote-compute/sources), as for a run from a ref, and the request waits until the fetch is done. The checkout is shallow, the commit and nothing before it. A sandbox stays until it is deleted. Nothing expires and nothing is removed on its own, `lastUsedAt` says when a run or a fetch last used it. `fetch` moves a sandbox only while nothing changed its files (files that `.gitignore` leaves out, such as `node_modules`, are not changes and stay) and no run holds it, else it is `409`. While the caller's sandboxes take their limit of bytes, a fetch is `413`, and while all sandboxes take the server's total it is `507`. The creates and fetches of one caller run one at a time, so each sees what the one before it added. A fetch that takes the caller over their limit is kept and measured: their next create, edit that adds bytes, or run on a sandbox is refused until they delete one. | Status | When | | ------ | ----------------------------------------------------------------------------------------------------------------------------------------- | | `400` | The body is malformed, the repository is not `host/path`, the server can not read it, or `ref` is not a branch of it (a tag or a commit). | | `403` | No entry in the `sandboxes` of the caller's entitlements covers the repository, or none of its branch patterns covers the branch. | | `404` | The sandbox does not exist or is somebody else's. | | `409` | A fetch of a sandbox with changes, or one a run holds. | | `413` | The caller is at their limit of sandboxes or bytes. | | `502` | The server could not reach the repository. | | `507` | All sandboxes together take what the server allows. | ## Running on a sandbox [#running-on-a-sandbox] A run names the sandbox as its `source`, the rest of the request is a normal [run](/docs/remote-compute/runs#submitting-a-run): ```json { "definition": "sha256:9f2c...", "tasks": ["test"], "source": { "type": "sandbox", "id": "sbx_Jp0dY8m2kQ1aZt5x" } } ``` * **The published files run on the sandbox.** The server reads the published version from a directory of the run, and its tasks work on the sandbox: a build file published as `ci/build.yaml` works in `ci/` of the sandbox. Nothing of the published version is written into the sandbox. * **One run at a time.** A run holds its sandbox from when it is submitted until it is over, waiting for an approval and in the queue included. A second run on the same sandbox waits in the queue for the first, runs on other sandboxes start next to it within the caller's [limits](/docs/remote-compute/runs#limits-and-restarts) (`runs.limits.running`, 2 by default), on [Kubernetes](/docs/remote-compute/backend#namespaces) too. * **What it started from** is recorded in the run: the `commit` the sandbox was fetched at, the git `tree` of its files when the run was submitted, which together name the changes it held, and whether it held any (`clean`). The run logs the tree again when it starts, since a run before it on the same sandbox may have changed the files. * **An approved run works on a copy.** A run that waited for an [approval](/docs/remote-compute/entitlements#approvals) runs on a copy of the tree it was approved with, as a [check](/docs/remote-compute/checks#checks-of-a-sandbox) does, not on the sandbox: files `.gitignore` leaves out (`node_modules`, outputs) are no part of the tree, an approver does not see them, and they are not in the copy. Its tasks install or build what they need, and what they write stays in the copy. On Kubernetes the copy is uploaded read only, as for a run from a ref: a task writes only into what it `generates`. The run fails when the sandbox's tree is no longer the approved one; submit it again. * **No links out.** The files are written by tasks and by whoever pushed the commit, so the server does not trust them. A `src`, mount or output of a task, or one it inherits from a task it depends on, that leaves the sandbox or goes through a symbolic link fails the run, and so does one the server can not look into. It is checked when the run starts and again before each container is made, once it has started (where what the container binds is compared with the sandbox's files, see [build files](/docs/remote-compute/build-files)) and before outputs are copied: a task that leaves `src -> /` behind does not give the next run the server's disk. * **Containers only.** The tasks of a run on a sandbox may only use the runtimes in `sandboxes.runtimes`, `docker` by default. A local task would run as the user of the server and could read every other sandbox on the machine, so `local` has to be listed there by name, and in `runtimes`. * **Network.** No task or service of a run on a sandbox reaches more than `sandboxes.network`, `none` by default: only the services it needs, nor more than [`runs.network`](/docs/remote-compute/runs#config), which holds for every run. A build file can narrow it, never widen it, see [network limits](/docs/task/network). * **Variables.** An `env:` secret of a run on a sandbox reads only what `sandboxes.env.allow` lists, nothing by default: the code that would read it is nobody's reviewed code. So does a build file: `$NAME` or `${NAME}` in its `envs` may read `PATH`, `HOME`, `CI` and what the server sets about the run, else only a variable `sandboxes.env.allow` lists. A run that reads another is refused with which task reads what. * **Cache.** Runs on sandboxes share a cache per owner, never with runs from a ref: a sandbox holds changes nobody reviewed, and an entry they leave must not reach a reviewed run. An approved run has a cache of its own, apart from the owner's runs that needed no approval. * **Checks.** With `"check": true` in the `source` the run works on a copy of the files a push would commit instead of the sandbox, and what it passes counts for a CI job that verifies the pushed commit, see [Checks of a sandbox](/docs/remote-compute/checks#checks-of-a-sandbox). What a run wrote stays in the sandbox (not what an approved run or a check wrote, they work on a copy) and counts against the owner's bytes, the docker volumes of its outputs included, whatever permissions a task gave its files. While a caller's sandboxes take more than their limit, their runs on sandboxes are refused with `413`. | Status | When | | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `400` | The sandbox does not exist or is somebody else's, or a task reads an `env:` secret that `sandboxes.env.allow` does not list. | | `403` | No entry in the `sandboxes` of the caller's entitlements covers the repository and the branch of the sandbox (any more), a check's caller has no `repos` for the repository, an account of the run does not list `sandbox` in its `source`, or the server runs no tasks on sandboxes (only `runtimes: [local]`, see below). | | `413` | The caller's sandboxes take more than their limit. | ## Files [#files] The files of a sandbox are read, searched and edited through the API, in two ways: * **Typed endpoints** for clients without file tools of their own, such as an MCP server: a tree, numbered reads, ripgrep, batches of edits and the changes. Their parameters follow the tools models are trained on: `offset`/`limit` reads in `cat -n` style, exact string replacement with `old_string`/`new_string`, git diffs and OpenAI's `apply_patch`, and ripgrep's options. * **A raw surface** for agent harnesses that bring their own tools and only swap where they run (pi's tool operations, for example): read bytes, write bytes, stat. | Call | What it does | | ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET /v1/sandboxes//tree` | The files and directories git sees, see [Tree](#tree). | | `GET /v1/sandboxes//files/` | A text file as numbered lines, see [Reading](#reading). | | `GET /v1/sandboxes//files/?raw` | The bytes of a file, with its sha256 as `ETag`. | | `GET /v1/sandboxes//files/?stat` | What a path is: `type` (`file`, `directory`, `symlink`, or `other` for a pipe, socket or device), `size`, `exec`, `modifiedAt`, a link's `target`. | | `HEAD /v1/sandboxes//files/` | The same as headers: `content-length`, `last-modified`, `x-hammerkit-type`, `x-hammerkit-exec`. `404` when there is nothing. | | `PUT /v1/sandboxes//files/` | Write the body as the file's bytes, creating the directories above it. `If-Match: ""` writes only over that content. `201` for a new file. | | `POST /v1/sandboxes//search` | ripgrep, see [Search](#search). | | `POST /v1/sandboxes//edits` | A batch of edits, all or nothing, see [Edits](#edits). | | `GET /v1/sandboxes//changes` | What changed since the commit, see [Changes](#changes). | Paths are relative to the sandbox, with `/` between names and no `.` or `..`; in a URL they are percent-encoded (`files/src/my%20file.ts`). Three rules hold for every endpoint: * **No symbolic link is followed.** A link shows as a link with its `target`; reading, editing or listing through one is `400`. Writing a file where a link is replaces the link. A read checks that the file it opened is the one in the sandbox, so a task that swaps a directory for a link does not lead a read outside of it, and an edit writes into the directories it checked, name by name. A server on Linux needs `/proc` for that and reads and edits nothing without it (`503`). Pipes, sockets and devices are `400` too, and a read takes no more than the file held when it was opened. * **`.git/` and credentials stay out.** What [uploads refuse](/docs/remote-compute/sources) (`.env`, `.npmrc`, keys, `.ssh/`, `.aws/`, `.git/`) is `403` to read or write, and is left out of the tree, the search and the changes, even when the repository commits it. Names match in any case and under the names Windows also opens (`.SSH`, `.git.`, `GIT~1`). The git data of a sandbox is not in its files anyway. * **Runs come first.** Reading one file works at any time, also while a run works on the sandbox (not [on Kubernetes](#on-kubernetes)). Edits, `PUT`, and the tree, search and changes (which walk the files with git and ripgrep) are `409` while a [run holds the sandbox](#running-on-a-sandbox), and the next run waits for a batch of edits to finish. Every file endpoint needs an entry in the `sandboxes` of the caller's entitlements that covers the repository and the branch of the sandbox, like a run on it. What edits add counts against the owner's [bytes](#config): an edit that would go over is `413`. ### Tree [#tree] `GET /v1/sandboxes//tree?path=src&depth=2` lists what git sees below `path` (the whole sandbox without it): committed files and new ones, not what `.gitignore` leaves out, such as `node_modules`. `depth` is how many levels, `1` by default; directories show as entries of their own. With `glob` (`**/*.ts`, matched against the path below `path`) it lists the matching files at any depth instead. ```json { "path": "src", "entries": [ { "path": "src/app.ts", "type": "file", "size": 1180 }, { "path": "src/lib", "type": "directory" }, { "path": "src/lib/db.ts", "type": "file", "size": 2207 } ], "total": 3, "truncated": false, "cursor": null } ``` Every list answers its `total`. A list longer than `limit` (1,000 by default, at most 10,000) is `truncated`, and its `cursor` asks for the next page: `?cursor=`. ### Reading [#reading] `GET /v1/sandboxes//files/src/app.ts?offset=40&limit=20` answers lines 40 to 59 of a text file, numbered like `cat -n`: the line number right aligned in six places, a tab, the line. ```json { "path": "src/app.ts", "sha256": "9f2c...", "size": 1180, "lines": 72, "offset": 40, "content": " 40\texport function main() {\n 41\t return run()\n...", "truncated": true, "nextOffset": 60 } ``` `offset` starts at `1`, `limit` is 2,000 lines by default. A read answers at most 256 KiB, and a line longer than 2,000 characters is cut, saying how long it was. `truncated` and `nextOffset` say where to read on. A binary file or one that is not UTF-8 is `415`: read it with `?raw`. A file larger than 32 MiB is `413` for a numbered read; `?raw` streams it. `sha256` is what an edit's `ifHash` and `PUT`'s `If-Match` compare against, so a change a run made since the read is not overwritten unseen. ### Search [#search] `POST /v1/sandboxes//search` runs [ripgrep](https://github.com/BurntSushi/ripgrep) on the files, with `.gitignore` honoured and hidden files searched. The server needs `rg` on its `PATH`; without it, search is `501`. ```json { "pattern": "createUser\\(", "glob": "*.ts", "output_mode": "content", "C": 2 } ``` | Field | What it does | | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `pattern` | A regular expression in ripgrep's syntax. Its errors come back as ripgrep says them, with `400`. | | `path` | A file or directory to search in, the whole sandbox by default. | | `glob` | One glob or a list, as ripgrep's `--glob` (`*.ts`, `!*.test.ts`). | | `type` | A file type of ripgrep (`ts`, `py`, `rust`). | | `i` | Ignore case. | | `fixed_strings` | The pattern is text, not a regular expression. | | `multiline` | Let the pattern match across lines (`.` matches a line feed too). | | `A`, `B`, `C` | Lines after, before and around each match, for `content`. `-i`, `-A`, `-B` and `-C` work as names too. | | `output_mode` | `files_with_matches` (the default): the paths. `content`: the lines, `{ path, line, text }`, with `context: true` on the lines around a match. `count`: the matches per file. | | `head_limit` | Results per page, 250 by default, at most 5,000. `cursor` asks for the next page. | On Linux, a search in a `path` searches the file or directory the server checked: a link a task swaps in on the way while it runs leads it nowhere else. The `.gitignore` files of the directories above the path still apply. Results come in path order. ripgrep stops counting 20,000 results past the page; `exact` is then `false` and `total` is a lower bound. A search that takes longer than a minute is `504`. ### Edits [#edits] `POST /v1/sandboxes//edits` applies a batch of operations in order. Each one sees what the ones before it did, and the batch is written only when every operation applies: either all of it changes the files or none of it does. `dryRun: true` checks the batch and answers what it would change without writing anything. ```json { "ops": [ { "op": "replace", "path": "src/app.ts", "old_string": "return 1", "new_string": "return 2" }, { "op": "write", "path": "docs/notes.md", "content": "# Notes\n" }, { "op": "move", "path": "old.ts", "to": "src/new.ts" }, { "op": "delete", "path": "tmp/out.log" }, { "op": "patch", "patch": "*** Begin Patch\n*** Update File: README.md\n@@\n-old\n+new\n*** End Patch\n" } ] } ``` | Operation | What it does | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `replace` | Replaces `old_string` with `new_string` in a text file. `old_string` has to match exactly once, whitespace included; when it matches more often, the error names the lines. `replace_all: true` replaces every match. | | `write` | Writes `content` as the whole file, creating it and the directories above it. `encoding: "base64"` for bytes. | | `delete` | Deletes a file or a link. Directories are not deleted. | | `move` | Moves a file to `to`, which may not exist yet. | | `patch` | Applies a patch: a unified diff as `git diff`, `git format-patch` or `diff -u` write it, or the body of OpenAI's `apply_patch` (`*** Begin Patch`). | `replace`, `write`, `delete` and `move` take `ifHash`, the `sha256` of a read: when the file holds something else now, the batch is `412`. The answer lists what changed: ```json { "dryRun": false, "files": [ { "path": "README.md", "change": "modified", "sha256": "1d0e...", "size": 512 }, { "path": "old.ts", "change": "deleted" }, { "path": "src/new.ts", "change": "added", "sha256": "77a1...", "size": 640 } ] } ``` An operation that does not apply is `422`, and the error names it and says what to do: `op 0 (replace): old_string is 2 times in src/app.ts, at lines 12, 40: add lines around it until it is there once, or set replace_all`, with `op` (its index) in the body. How a patch is applied: * **A git diff** applies like `git apply`: each hunk where its lines are, or at the nearest place where they all are when lines above it moved. It may add, delete, rename and copy files and set the executable bit. Binary patches, links and submodules are refused; write binary files with `PUT`. A diff of a file with CRLF line endings applies to it either way, and the lines it adds take the file's line ending. * **An `apply_patch` body** finds each chunk's lines after the one before it, after the line its `@@` names: exactly, then ignoring trailing whitespace, then ignoring whitespace around the lines, as OpenAI's reference implementation does. `*** End of File` ties a chunk to the end of the file, `*** Move to:` renames. The patches of one batch compare at most 20 million lines to place their hunks; a hunk that is not in a large file stops there with `422`. Give it the line numbers of the file as it is, or more lines around the change. A hunk whose line numbers are past the end of the file is looked for from the end; a line number over a billion is `422`. One batch holds at most 256 MiB in memory, the files it reads and the files it would write together, and processes at most 1 GiB of text (each read, scan and result of its operations counted): a batch over either is `413`, send its operations in smaller batches. All batches being planned on the server hold at most 512 MiB together; a batch that would go over is `503`, send it again in a moment. A `replace` whose result would be over 32 MiB is `413` before it is made. While a batch is written, the bytes it adds count against the owner's limit and the server's total, so batches on two sandboxes at once can not pass them together. ### Changes [#changes] `GET /v1/sandboxes//changes` compares the files with the commit the sandbox was fetched at, the way a run sees them: new files count, what `.gitignore` leaves out does not, and neither does a git repository a task made inside the files. Neither does what hammerkit writes next to a task while it runs: a `.hammerkit/` directory at any depth (the last cache entry it resolved, the state of local tasks) is the run's, not a change. The build file `.hammerkit.yaml` is a change like any other file. | `format` | What it answers | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `summary` (the default) | `{ commit, files: [{ path, status, from, additions, deletions }] }`: `added`, `modified`, `deleted`, `renamed` (with `from`), lines added and deleted (none for binary files). | | `diff` | The changes as `git diff` shows them, as `text/x-diff`. | | `patch` | One commit in `git format-patch` form, binary files included, for `git am` on another clone. `message`, `name` and `email` set its message and author (`hammerkit` by default). | A patch is one way work leaves a sandbox: the caller, or a connector on their side, applies it and pushes a branch. The other is a [push](#pushing) by the server. Changes larger than 64 MiB are `413`. | Status | When | | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `400` | The path is not inside the sandbox, goes through a link, or is a directory where a file is needed; the body is malformed; a ripgrep error. | | `403` | The path is on the deny list; or no entry in the `sandboxes` of the caller's entitlements covers the repository and the branch of the sandbox any more. | | `404` | The sandbox, or the file, does not exist. | | `409` | A run holds the sandbox (edits, `PUT`, tree, search, changes); a move or patch onto a file that exists. | | `412` | The file changed since the read that `ifHash` or `If-Match` names. | | `413` | The edits would take more than the owner's bytes, a file is over 32 MiB, a batch holds more than 256 MiB or processes more than 1 GiB, or the changes are over 64 MiB. | | `415` | A numbered read or a text edit of a binary or non-UTF-8 file. | | `422` | An operation of the batch does not apply; `op` names it. | | `501` | Search, on a server without ripgrep. | | `503` | The batches being planned on the server hold what they may together; send it again in a moment. A read or an edit on a Linux server without `/proc`. | ## Pushing [#pushing] `POST /v1/sandboxes//push` commits the changes of a sandbox and pushes the commit to a branch of its repository, with the `secret` of the server's [source](/docs/remote-compute/sources#pushing) for it. The caller's `sandboxes` decide where to: a sandbox only pushes to a branch whose pattern has the type `push` in an entry for its repository, `agent/**` for example, never to a `main` they may only `pull`. ```json { "branch": "agent/fix-login", "message": "Fix the login\n\nThe session expired a minute early." } ``` ```json { "branch": "agent/fix-login", "commit": "9b0e7c...", "sandbox": { "id": "sbx_Jp0dY8m2kQ1aZt5x", "ref": "agent/fix-login", "commit": "9b0e7c...", "pushed": { "agent/fix-login": "9b0e7c..." } } } ``` * **What is committed** is what [`changes`](#changes) shows: new, changed and deleted files, not what `.gitignore` leaves out, nothing of the deny list (an `.env` a task wrote stays on the server) and no `.hammerkit/` directory a run wrote. The commit's parent is the commit the sandbox is at. * **By whom.** The commit's author and committer are the caller, as their token names them (`name`, and `email` unless the token says it is not verified), else their subject at the host of their issuer (`agent-7@login.example.com`). A token without `name` or `preferred_username`, such as a Google service account's ID token, is named by the part of its verified address before the `@` (`deploy-bot` for `deploy-bot@my-project.iam.gserviceaccount.com`), and only without one by its subject. A request can not set the author, and a message with `Signed-off-by`, `Reviewed-by`, `Acked-by` or `Tested-by` is refused: those would attest the commit in somebody else's name. Other trailers, such as `Co-authored-by`, are the caller's text. * **Only new branches, or its own.** The branch must not exist yet, or be one this sandbox pushed and still at the commit it pushed there. A branch somebody else created or pushed to since is `409`: push to a new branch. A sandbox never writes over other work, and it never force-pushes. * **Afterwards** the sandbox is at the pushed commit, on that branch: `changes` shows what is not pushed yet, and the next push to the same branch adds a commit on top of the last one. A sandbox [fetched](#creating-and-managing) to another commit since can not push to the branch any more (`409`), it starts a new one. * Like edits, a push waits for a batch of edits, is `409` while a run holds the sandbox, and needs the repository and the sandbox's branch in the caller's `sandboxes`. The server reads the branch first with its read credential, then pushes with the write credential, telling the remote which commit it expects the branch at (none for a new one). The remote refuses the push when somebody created or moved the branch in between, and that is `409` too. | Status | When | | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `400` | The body is malformed, the message signs off in somebody's name, or the branch is not a name git takes (no part may start with `.` or `-`, or end with `.` or `.lock`). | | `403` | No `push` pattern of the caller's entries for the repository covers the branch, the source has no `secret`, or as for the files, the caller lost the repository or the sandbox's branch. | | `404` | The sandbox does not exist or is somebody else's. | | `409` | Nothing to push, a run holds the sandbox, or the branch is not this sandbox's to push to. | | `502` | The server could not read its push credential (the details are in its log), could not reach the repository, or the remote refused the push for another reason (a rule of the repository, a token without write access, a GitHub App without the `workflows` permission pushing a workflow file): the error carries the reason the remote gave. Also when the push went through but the sandbox could not be moved to it: the branch stays the sandbox's. | Opening a pull request from the branch is left to the caller: hammerkit holds a token that writes branches, not one that reviews them. A pushed branch runs in the repository's CI like any other, with its secrets unless the operator keeps those branches out, see [Sources](/docs/remote-compute/sources#pushing). ## Accounts [#accounts] A sandbox holds changes nobody reviewed, so its runs only use the [accounts](/docs/remote-compute/accounts#restricting-the-code) whose `source` lists `sandbox`. An account without `source` takes runs from an upload or a ref, not from a sandbox. Keep production accounts on `source: [ref]`. ```yaml title="server.yaml" accounts: op-payments-dev: providers: { gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: /etc/hammerkit/creds/payments-dev.json } } } source: [ref, sandbox] ``` ## Config [#config] ```yaml title="server.yaml" sandboxes: runtimes: [docker] # default when runtimes has docker, `local` has to be named (in runtimes too) network: { allow: [registry.npmjs.org, '*.githubusercontent.com'] } # default: none env: { allow: [] } # default, variables of runs.env env: secrets and envs may read capabilities: [] # default, capabilities of backend.capabilities.allow a task may add limits: count: 5 # default, sandboxes per caller bytes: 20Gi # default, bytes of one caller's sandboxes together total: 500Gi # default, all sandboxes on the server ``` | Key | What it does | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `runtimes` | What the tasks of a run on a sandbox may run on, out of `runtimes`. `[docker]` by default when `runtimes` has `docker`. A server with only `runtimes: [local]` starts as before and refuses runs on sandboxes until `local` is named here. | | `network` | The most open network a task or service of such a run gets, written as a task's [`network`](/docs/task/network): `none` (the default), `open`, or `{ allow: [hosts] }` for HTTP(S) to those hosts through the egress proxy. It has to be within `runs.network`, which caps every run. With `local` in `runtimes` it has to be `open`, hammerkit can not limit a local task. | | `env.allow` | The variables of `runs.env.allow` or `runs.env.set` an `env:` secret, and the `envs` of a build file, of a run on a sandbox may read. None by default; `PATH`, `HOME`, `CI` and the variables of what the run is from are always readable by `envs`. | | `capabilities` | The capabilities of [`backend.capabilities.allow`](/docs/remote-compute/backend#capabilities) a task of a run on a sandbox may add (`capAdd`). None by default. | | `limits.count` | Sandboxes one caller may keep. Creating another is `413`. | | `limits.bytes` | What one caller's sandboxes may take together, their git data included. A create that would go over is `413` and leaves nothing behind. | | `total` | What all sandboxes on the server may take. Over it, a create is `507`. | The bytes of a sandbox are measured after it is fetched and after each run on it. A run's result does not wait for that measure, but the owner's next run or create does. The files are kept below `storage` (`sandboxes/`, readable by the user of the server only), with the git data of each sandbox next to its files rather than in them: tasks write to the files, and git on the server never reads a configuration or a hook a task could have written. What a task made unreadable is made readable again before it is measured or deleted, and what a container wrote as another user (a service runs as the user of its image) is given back to the server's user with a short `alpine` container. A restart keeps the sandboxes, also one whose record can't be read: its files stay for the operator to look at. Changes to `sandboxes` need a restart. ## On Kubernetes [#on-kubernetes] With a [Kubernetes backend](/docs/remote-compute/backend#kubernetes) a sandbox is two `PersistentVolumeClaim`s in its caller's namespace, mounted only while something works on it: an idle sandbox costs its storage and nothing else. The API, the limits and the rules above are the same. * **Two claims.** One holds the sandbox's files, which the tasks of its runs mount. The other holds its git data, which only the pod of its files mounts. A link a task makes in the files leads at most to the root of the files' claim, so no task reaches git's configuration or hooks, as on a host. * **Fetch.** The server fetches the commit with the read credential of `sources`, as for a run from a ref, and hands the repository to the pod of the sandbox, which takes it into the claim. The credential never reaches the caller's namespace. * **Files.** Reads, searches, edits, the changes and the bytes of a sandbox run in a short pod that mounts its claims, with the same code as on a host (`hammerkit server-sandbox`), so paths, links and the deny list follow the same rules. The pod has no token and no network (a `NetworkPolicy` denies all of it), runs on the first pool with no capabilities beyond owning and changing files and a read-only root, and stays while requests come or a run works, `backend.sandboxes.idle` after the last one. * **Push.** The pod commits the changes; the server copies the objects of the git data, checks that the commit is one on top of the sandbox's without anything of the deny list, and pushes it with the write credential. The write credential never reaches the caller's namespace either. * **Runs.** The process of a run gets a copy of the files to plan with. Its Jobs mount the files' claim itself where their sources are, read and write, so what a task writes is in the sandbox afterwards, as on a host. An approved run and a check work on their copy instead, as a run from a ref does, and never mount the claim: their `src` and `mounts` are uploaded and read only, they write into what they `generates`, and the claims they keep are not the ones the sandbox's other runs keep. A source outside of the sandbox goes into the task's own claim as before. A source or mount that goes through a link is refused, as on a host. * **Commands.** A task of a run on a sandbox that mounts the whole of the sandbox's files (`mounts: [.]` or `src: [.]`) and nothing else (no volumes, needed services, secrets or `generates`), such as the shell task of an [agent](/docs/remote-compute/agents), runs in the sandbox's runner: one pod next to the pod of the files that stays while that pod does and takes each command over `pods/exec`. A command then starts in about a second, where a Job takes several seconds to get a pod. Before and after each command, and when a command is cancelled, runs out of time or loses its connection, every process in the runner is killed and `/tmp` and `/dev/shm` are emptied, so nothing a command started is left to the next one. The runner has the sandbox's files at `/.hammerkit-sandbox`, and each command first links the directory of its run (where a Job would have the files) there, so paths are the same as in a Job, and the runner the same from one run to the next. Before each command the runner checks that its mounts are still the mounts it was made with; when a command before moved one, the runner goes and the task fails, so run it again to get a new runner. A task that mounts a part of the files only (`src: [src]`) runs as a Job: the folders around its mount would be the runner's own, and what a command left there would be in the next runs. What does that is a static busybox (`busybox:1.37.0-musl`, pinned by digest, which an init container copies in) on a mount the runner can only read, so a command can not replace it, not even as root. Files a command writes elsewhere outside the sandbox, such as the home directory or `/usr/local/bin`, stay for the next commands while the runner does: the cleanup is for processes, not a fresh filesystem. On a node whose kernel lets a process trace another of its user (`kernel.yama.ptrace_scope` 0), a command running as root can still attach to the runner's first process, which the cleanup spares, and leave something running that way. Where every command needs a container of its own, set `runner: false`. The runner has the image, variables, resources, mounts and network of the task: a task that differs in any of them gets a new runner, and the old one goes. A second task of the same run, while the first is in the runner, runs as a Job, and so does a task whose runner the cluster refuses (a full quota) or whose runner does not start (an image without `sh`). A command stopped while the runner starts (cancelled, or a short timeout) leaves it starting for the next command. With `backend.sandboxes.runner: false` every task is a Job. * **Network.** Runs on sandboxes get `sandboxes.network`, `none` by default, which is a `NetworkPolicy` on their pods. So with an entitlement that grants `sandboxes` and a `sandboxes.network` other than `open`, the config needs `backend.networkPolicy: enforced`. Under `{ allow: [hosts] }` a task's [egress proxy](/docs/task/network) stays with the sandbox while the pod of its files runs, so the next run of that task on the sandbox uses it instead of waiting for a new one. Each run lets in its own pods only. A task whose allow list or needed services changed gets a new proxy, and the old one goes. All of them go when the pod of the files does. That is one small pod per limited task of an active sandbox, counted by the namespace's quota. * **One file while a run works**, as on a host: the pod of the files stays while the run works. * **One node.** A claim is usually `ReadWriteOnce`: several pods share it only on one node. The pods that mount a sandbox (the Jobs of its run and the pod of its files) are placed next to each other, so its disk never moves between nodes during a run, which takes seconds each time. The tasks of a run on a sandbox go to the node of the pod of its files, in the first pool; a task on another pool (a local task whose `toolImage` is on a later pool) fails. When a pod of the run finds no room on that node for 10 seconds, the pod of the files makes way, so the run's pod goes where there is room and the disk follows it. * **Kubernetes 1.30 or later.** The server talks to the pod over `pods/exec` with the `v5.channel.k8s.io` protocol, which ends a request's input without closing the connection. ```yaml title="server.yaml" backend: type: kubernetes sandboxes: image: registry.corp/hammerkit@sha256:… # default: no0dles/hammerkit of the server's version size: 4Gi # default, what each claim of a sandbox requests storageClass: sandboxes # default: the class of tasks' claims idle: 1m # default, how long the pod of the files stays runner: true # default, commands of a sandbox go into one pod that stays resources: # default, of the pod of the files requests: { cpu: 100m, memory: 256Mi } limits: { memory: 2Gi } ``` | Key | What it does | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `image` | The image of the pod of a sandbox's files: hammerkit with git and ripgrep, as the published image of the server's version is. It reads and changes every sandbox's files, so pin it by digest. | | `size` | What each of the two claims of a sandbox requests: the most its files, and its git data, can take. No more than `sandboxes.limits.bytes`. | | `storageClass` | The storage class of sandboxes' claims. When it is one of their own, each caller's namespace gets a `ResourceQuota` on that class (`requests.storage` of twice `sandboxes.limits.bytes` and `persistentvolumeclaims` of twice `sandboxes.limits.count`, two claims a sandbox), so the cluster holds the limits too. It can not be the class of tasks' claims, whose claims would count against it. | | `idle` | How long the pod of a sandbox's files stays after the last request. Its runner and egress proxies go with it. | | `runner` | Whether a task that mounts the whole of the sandbox's files, and nothing else, runs in the sandbox's [runner](#on-kubernetes) instead of a Job of its own. | | `resources` | The requests and limits of the pod of a sandbox's files, which runs git and ripgrep over all of them. | The server keeps the record of each sandbox in its storage directory as on a host, and the claims carry the sandbox's id and owner and a label of the server (its namespace prefix and namespace). When the server starts, its own claims without a record are deleted once they are an hour old (younger ones may be a sandbox being created), and a record without claims is dropped. Claims of another server on the cluster, or of this server under another prefix, are left alone, and so are the records of the latter. # Service runs (/docs/remote-compute/service-runs) A service run starts services of a published build file on the server, the way [`hammerkit up`](/docs/cli/up) starts them, and keeps them up until its time ends. Ports of its services can be published as endpoints: each gets an address of its own, such as `https://feature-login-8080-3fa2c1e09b7d.run.example.com`, that the server answers behind its own sign-in and passes on to the service. A person opens it in a browser, an agent reads it with its token. It is for whatever needs the app running for a while: somebody reviews a branch before it is merged, a demo, an agent that checks its change in the real app. The run ends on its own at its `ttl`, so a forgotten one doesn't hold the server, and with a `sleepTtl` its services sleep while nobody looks and wake on the next request. ```txt $ hammerkit remote up web --endpoint web:8080 --ttl 3d --sleep-ttl 30m run_Vd1rX9pQ2mT7wN4a on build: web from feature/login at 4be1a07c... [hammerkit] checked out github.com/corp/shop at 4be1a07c... run_Vd1rX9pQ2mT7wN4a is up until 2026-10-28T10:12:00.000Z, asleep after 30m without a request web:8080 https://feature-login-8080-3fa2c1e09b7d.run.example.com/ extend it with: hammerkit remote extend run_Vd1rX9pQ2mT7wN4a 8h, stop it with: hammerkit remote stop run_Vd1rX9pQ2mT7wN4a ``` ## Starting one [#starting-one] A service run is a run that names `services` instead of `tasks`, with `POST /v1/runs` like any other [run](/docs/remote-compute/runs#submitting-a-run). The services start with what they need and depend on, as `hammerkit up` starts them: tasks that build an image or install packages run first. ```json { "definition": "sha256:9f2c...", "services": ["web"], "endpoints": [{ "service": "web", "port": 8080 }], "ttl": "3d", "sleepTtl": "30m", "name": "feature-login", "source": { "type": "ref", "repo": "github.com/corp/shop", "ref": "feature/login" } } ``` | Field | What it does | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `services` | The services of the build file to start, by name. A run names tasks or services, not both, and takes no [arguments](/docs/remote-compute/runs#arguments) or label filters. | | `endpoints` | Ports to publish, up to 5: each a `service` the run starts and a `port` it declares in `ports` (the container port). Needs an `endpoints` level `publish` for the repository, see [Who opens an endpoint](#who-opens-an-endpoint). | | `ttl` | How long the run lasts once its services are ready, such as `8h` or `3d`, asleep or not; then they are removed. `runs.up.ttl` when left out (`8h`), at most `runs.up.maxTtl` (`72h`). | | `sleepTtl` | How long the services stay up without a request to one of its endpoints, such as `30m`; then they [sleep](#sleeping) until the next request wakes them. `runs.up.sleepTtl` when left out; without either, they never sleep. A run from a ref only. | | `name` | Lower case letters, digits and `-`, at most 40, which the addresses carry. The branch of the source when left out (`feature/login` is `feature-login`). | | `source` | A ref, or a [sandbox](/docs/remote-compute/sandboxes) on a host backend. | The run goes through the states of every run, with two more: `up` once every service it names is ready (its healthcheck passed), and `asleep` while its services sleep. From then on its time counts, and it no longer takes a place of `runs.maxConcurrent` or holds its sandbox: the next run starts, and an agent can go on working. `GET /v1/runs/?wait=30s` answers as soon as the run is up. A run that is up says when, and until when: ```json { "id": "run_Vd1rX9pQ2mT7wN4a", "state": "up", "upAt": "2026-10-25T10:12:00.000Z", "expiresAt": "2026-10-28T10:12:00.000Z", "sleepsAt": "2026-10-25T10:42:00.000Z", "up": { "services": ["web"], "ttl": 259200000, "sleepTtl": 1800000, "name": "feature-login", "endpoints": [ { "service": "web", "port": 8080, "host": "feature-login-8080-3fa2c1e09b7d", "url": "https://feature-login-8080-3fa2c1e09b7d.run.example.com/" } ] } } ``` It ends `succeeded` when its time ran out or somebody stopped it, and `stoppedBy` says why: `ttl`, `replaced` or who stopped it. A service that stops on its own, or never gets ready, fails the run. `maxDuration` limits how long it may take to get up. A run that failed names the services that failed and why in its `error`, such as `web: service web did not pass its healthcheck "curl -f localhost:8080" within 30s of starting`, and its log has the same line under the service's name, after what the service printed. The log stays with the run after its services are gone: `GET /v1/runs//logs`, or [`hammerkit remote logs`](/docs/cli/remote#reading-a-run-again). [`hammerkit remote up`](/docs/cli/remote#service-runs) does this from the command line, and the [agent tools](/docs/remote-compute/agents#service-runs) from a coding agent. ## Addresses [#addresses] The host of an endpoint is the run's `name`, the port and twelve hex digits of who started it: `--.`. So the runs of several branches stand side by side, each at addresses of its own, and two people with a branch of the same name don't take each other's. A run whose address would be one of another run that is not over, of somebody else or of another name, is refused with `409`. A second service run of yours with the same name takes the place of the first, and the addresses stay. Push a fix, start the branch again, and the reviewer reloads the same page. When it starts the same services of the same build file, the first run [moves to the new commit](#moving-to-another-commit): it keeps its id, its time and the data of its services. Otherwise it is replaced: once the new run is up, the old one stops. Either way, until the new commit is up, the old one answers. Each caller has at most `runs.up.limits.up` service runs starting or up at a time (`3` by default); one more is answered `429`. The server takes `runs.up.total` of them (`20`). A run that sleeps gives its place back, so you can start another while yours sleep. It asks for a place again when a request wakes it: with none free it sleeps on, its page says why and reloads every 30 seconds (`503` with `Retry-After` for other clients), and it wakes on the first request after one of your runs stops or sleeps. What sleeps has a limit of its own, `runs.up.limits.asleep` (`10`): with that many of yours asleep, a new run is answered `429` until you stop one (`hammerkit remote stop`) or one ends. Your runs of one name take one place: a new one that replaces a run of that name is not refused for `runs.up.limits.up` or `runs.up.total`, even while both are there (`runs.limits.queued` and `runs.limits.waiting` still count it). Only the newest of a name stays: one that waited for an approval and gets up after a newer one stops again at once. ## Time [#time] The time of a run is its `ttl` from the moment it is first up, asleep or not. Its owner, and whoever may open its endpoints, give it more: `POST /v1/runs//extend` with `{ "ttl": "8h" }` sets the end to that long from now, at most `runs.up.maxTtl`, and never sooner than it was. So does `hammerkit remote extend`, and the status page of every endpoint, `/.hammerkit/status`, which shows the run, its source and until when it stays, with a form to extend it. `POST /v1/runs//cancel` (or `hammerkit remote stop`) takes the services down at once, asleep or not. Its owner may, who decides about it, and whoever may publish endpoints of its repository (`endpoints` level `publish`), who could bring up a run there themselves; who may only open them may extend it, not stop it. An endpoint whose run ended says so, and who to ask, with status `410`. An extend, a stop and a [wake](#sleeping) answer with the run. Who may read it whole (its owner, who decides about it, and `observe`) gets all of it; who may only open or publish its endpoints gets what the list of runs shows them: its state, its end, its services and endpoints and who started it, not its arguments, its plan or the results of its tasks. ### Sleeping [#sleeping] With a `sleepTtl`, a run whose endpoints get no request for that long stops its services and is `asleep`: nothing of it runs, its addresses and its id stay. The next request of somebody who may open an endpoint wakes it: the run starts the services again, from the commit it is at (the one it started at, or its last move, wherever its branch is now), and the page says they are waking up and reloads until they are up (`503` with `Retry-After` for other clients). `POST /v1/runs//wake`, of its owner or somebody who may open its endpoints, wakes it the same way without a request, as the start button of the dashboard does; it answers `409` with the reason when the run is not asleep or can not wake now. Every request and WebSocket keeps it up for its `sleepTtl` from then. The `ttl` goes on counting while it sleeps; at its end the run ends, as one that never slept. A wake runs as the run's owner, whoever opened the endpoint, so it checks the owner's entitlements of the moment (by the groups and claims of their token when they started it): an owner who may no longer run from the repository, use its accounts or publish its endpoints ends the run instead, with the reason as its error. An account they may only request ends it too, nobody is there to approve it; start the run again. A run keeps the volumes of its services while it sleeps: the database of a preview wakes with the rows it had. What a service wrote anywhere else in its container is gone, as on a new run. The volumes go only when the run really ends: at its `ttl`, when it is stopped or replaced, or when it fails. While it sleeps, the status page of its endpoints says so (`asleep with its volumes kept`). A server with `runs.up.keepVolumes: false` keeps nothing: a woken run starts from nothing. A run from a sandbox doesn't sleep; push the sandbox and start the branch to get one that does. The services live with the server: when it stops, they stop. A run that may sleep is asleep then, with its volumes, and the next request after the server is back wakes it, unless its `ttl` ended while the server was down: then it ends as the server starts, gives its place back, and its volumes go. The others fail with `the server stopped while the services were up, start them again`. A server that starts again finds the volumes of its sleeping runs; those of a run that ended or that it has no record of any more go. On a host, the docker volumes of a service run carry a key of the server (by its runs' directory), so a second server on the same docker leaves them alone. Earlier prereleases named them otherwise: a server of this version doesn't sweep those, remove them with `docker volume rm` once their runs ended. ## Moving to another commit [#moving-to-another-commit] `remote up` with the name of a run of yours that is up or asleep, and the same build file version, services, endpoints and `sleepTtl`, moves that run to the commit its ref points at now (or `--commit`), instead of starting a new one. The server checks the move as it checks a new run: you may still run from the repository, with the run's accounts. A run that would need an approval is started anew and replaces the old one. ```txt $ git push && hammerkit remote up api web --endpoint web:4200 --name pr-42 run_Vd1rX9pQ2mT7wN4a on build (up) moves to feature/login at 9e01b77d... [hammerkit] moving from 4be1a07c... to 9e01b77d... [hammerkit] checked out github.com/corp/shop at 9e01b77d... run_Vd1rX9pQ2mT7wN4a is up until 2026-10-28T10:12:00.000Z, asleep after 30m without a request web:4200 https://pr-42-4200-3fa2c1e09b7d.run.example.com/ ``` What moves with it: * **Its id, addresses and time.** The run stays `up`; `--ttl` gives it more time, never less. It takes no other place of `runs.up.limits`. * **The data of its services.** Volumes stay, as across a sleep, with one exception: a service whose image or volumes change is another service to hammerkit, and starts with new, empty volumes (a database from `postgres:16.3` to `16.4` has no rows). The volumes of the old one stay until the run ends and count against its quota. * **What did not change.** Tasks run again only where their files changed, as any run of the commit would. A service whose files, variables, definition or dependencies' outputs did not change goes on running as it is, with the secrets it read: a rotated secret reaches it with a change of the service, or a new run (`remote stop`, then `remote up`). * **A dev server, on Kubernetes.** A service with [`continuous: true`](/docs/service/container#continuous) watches its own files: when only its `src` or `mounts` change, the move writes the changed files into its running pod instead of starting a new one, and it reloads. It serves the files as they arrive, not the commit before until the new one is healthy. A move that does not get up, also because the dev server restarts or stops being ready within 15 seconds of getting them, puts its files back. The run's record lists every commit it moved to (`moves`, with `moving` while a move is under way), and its `source` and the status page of its endpoints show the commit it is at now. A run that is asleep only changes its commit: the next wake starts the services of the new one. A second `remote up` while a move is under way takes its place. How the services get to the new commit depends on the backend: * **On Kubernetes** the run executes again beside itself, in its namespace. A service that changed rolls to a new pod, and its requests go to the old one until the new one passes its healthcheck. What the new commit uploads and builds (a service's `src` and `mounts`, the outputs of the tasks it depends on, such as `dist` or `node_modules`) goes into a folder of its own state in the claim, so the old pod serves the files and builds it started with until then, also when the move fails. A service with volumes of its own (a database) stops before its new pod starts, since two pods would write the same data. Once all services are ready, the run is at the new commit, and what the old commit had that the new one does not (a service whose image changed) goes. A move whose services don't get ready puts every service it rolled back as it was, removes the ones it added, and leaves the run up at the commit it was at; `remote up` says why: `did not move to 9e01b77d...: ...`. The folders of a claim go once no pod of the run mounts them and a later state is built or uploaded. * **On a host** the services stop, their volumes kept, and start again at the new commit: every service restarts, and the endpoints answer that they are starting meanwhile. When the new commit doesn't get up, the services start again at the commit before, with their volumes, and `remote up` says why. A move takes a place of `runs.maxConcurrent` while it runs, and waits for one, and it ends after `runs.maxDuration` as a run would. During a move the pods of the old commit still mount their claims, and a cloud disk attaches to one node at a time. So every pod of a service run that mounts claims (a service's, a Job of a changed task, an upload) has to go to a node where a pod runs that mounts the same claims, while one does; the first one goes anywhere. A full node keeps the pod waiting, and a rollout that makes no progress for 10 minutes fails the move. Unlike the first start, where the tasks run before the services, a move runs the Jobs of its changed tasks and the uploads of its changed services in the run's namespace while all services are up: its quota needs room for them and for one more pod of a rolled service. ## Who opens an endpoint [#who-opens-an-endpoint] The owner of a run opens its endpoints. Others need an `endpoints` level for its repository in an [entitlement](/docs/remote-compute/entitlements): ```yaml title="server.yaml" entitlements: - group: { iss: https://login.example.com, name: eng-shop } endpoints: [{ repo: github.com/corp/shop, level: open }] - subject: { iss: https://login.example.com, sub: shop-agent } builds: run sandboxes: [{ repo: github.com/corp/shop, branches: { 'agent/**': push } }] endpoints: [{ repo: github.com/corp/*, level: publish }] ``` | Level | What it allows | | --------- | ------------------------------------------------------------------------------------------------------ | | `open` | Open the endpoints of the service runs of the repository, whoever started them, and extend those runs. | | `publish` | `open`, and name `endpoints` in one's own service runs of the repository. | Starting a service run needs what any run needs (`builds`, `repos` or `sandboxes`, the accounts it reads); `endpoints` is only about the addresses. Who may open an endpoint is checked on every request with the entitlements of the moment, so an entitlement that is taken away (and [reloaded](/docs/remote-compute/entitlements#changing-entitlements)) closes the endpoints at once, signed in or not. A person signs in with the browser: the endpoint sends them to `login.`, which signs them in at the issuer of `endpoints.login` (the authorization code flow with PKCE), and back to the page they asked for. The sign-in holds for that one host, as a cookie the app never sees, for `endpoints.session` (`1h`) or until the run ends. `/.hammerkit/logout` signs out. Once it ran out, the next page the person opens (a page load: a `GET` that accepts HTML and that the browser makes for a page of its own, not a frame) signs them in again, which the issuer usually does without asking while they are signed in there. The app's own requests get a `401` of the server instead: `fetch` and XHR (htmx and Turbo too), a `POST`, a frame, a new WebSocket. A single-page app left open stops talking to its backend once an hour (the default), until the page is reloaded. Previews of such apps may want a longer `endpoints.session`, at the cost of the next paragraph. The cookie keeps who signed in with their groups and claims of that moment. An entitlement taken away on the server closes the endpoint at once; a group or claim taken away at the issuer counts at the next sign-in, so at the latest `endpoints.session` later. Keep it short where that matters. A WebSocket is checked when it opens and stays open after that, past the end of the sign-in and past an entitlement taken away; and a sleeping run wakes with the groups and claims its owner had when they started it (see [Sleeping](#sleeping)). What a sign-in needs until it comes back from the issuer (the verifier of PKCE, the nonce, the page to go back to, at most 1 KB of it, else without its query) is in a cookie of `login.` that the server signed, not on the server: callers who are not signed in fill no table of sign-ins. A sign-in starts only when the browser goes to it, not for a request a page makes, and each comes back once. It ends in the browser it started in: the endpoint gives that browser a cookie of its own host, which the last step of the sign-in checks, so a link of a sign-in sent to somebody else signs them in as nobody. Anybody else sends a token as the API takes it, in `Hammerkit-Authorization` (not `Authorization`, which stays the app's): ```sh curl -H "Hammerkit-Authorization: Bearer $(agent-token)" https://feature-login-8080-3fa2c1e09b7d.run.example.com/api/health ``` Without a sign-in, a page load in a browser is sent to the login; any other request gets `401`. Signed in as somebody who may not open it, `403`. The endpoints share one site, so a browser sends the cookie of one along when the page of another asks it something. The app of one endpoint is not to act as its visitor on another: a request that a page of another endpoint (or site) makes, the endpoint in a frame of such a page, and a WebSocket whose `Origin` is not the endpoint's own, don't count as signed in. Following a link to an endpoint does. ## What the app gets [#what-the-app-gets] The request as it came, without the cookie of the sign-in and without headers that start with `hammerkit-`, `x-hammerkit-` or `x-forwarded-`, which the server sets itself: | Header | Value | | -------------------- | ---------------------------------------------------------- | | `x-hammerkit-caller` | The issuer and subject of who asked, separated by a space. | | `x-forwarded-host` | The host of the endpoint. | | `x-forwarded-proto` | `https`, or `http` on a server on the loopback interface. | | `x-forwarded-for` | The address the request came from. | WebSockets pass through. The paths below `/.hammerkit/` are the server's, the app never gets them. The app's own sign-in, such as "Sign in with Google", needs its redirect URIs or origins registered at its provider; an address per branch is a new one each time, and Google takes no wildcards. That is not solved yet: a service run of an app with such a login works with a login of its own for review, or one the app turns off there. A service behind an endpoint may not read a secret or a variable of the server on a server whose [`runs.secretNetwork`](/docs/remote-compute/runs#tasks-that-read-secrets) is not `open`: a page is a way out no mask of the log covers. Neither may what it is made from or talks to: its init task and the tasks it depends on, which could leave a value in a file it serves, and the services it needs, on and on, which could hand one over the connection they answer on. The run is refused with what reads what, such as `web needs api, which reads the secret API_KEY`. ## Server config [#server-config] ```yaml title="server.yaml" listen: { host: 0.0.0.0, port: 443 } tls: { cert: /etc/hammerkit/api.crt, key: /etc/hammerkit/api.key } endpoints: domain: run.example.com tls: { cert: /etc/hammerkit/run.crt, key: /etc/hammerkit/run.key } # *.run.example.com login: issuer: https://login.example.com clientId: hammerkit-endpoints clientSecret: { from: gcp:projects/corp/secrets/endpoints-client/versions/latest, account: hammerkit-server } scope: openid email session: 1h # default runs: up: ttl: 8h # default, when a run names none sleepTtl: 30m # unset by default: services never sleep unless the run says maxTtl: 72h # default, the furthest a run or an extend sets the end limits: { up: 3, asleep: 10 } # defaults, per caller starting or up, and asleep; 429 over it total: 20 # default, on the whole server, 503 over it keepVolumes: true # default, a run keeps the volumes of its services while it sleeps quota: { requests.cpu: '4', requests.memory: 8Gi } # Kubernetes only, backend.quota when left out ``` | Key | What it does | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `endpoints.domain` | The domain the hosts of endpoints are below. A wildcard DNS record (`*.run.example.com`) points at the server. Without `endpoints`, a run that names endpoints is refused; service runs without them still run. | | `endpoints.tls` | The wildcard certificate of `*.`, served for those hosts next to `tls`, which the API keeps. Left out behind a TLS proxy, which then holds it: see below. Hammerkit does not get certificates. | | `endpoints.login` | How a person signs in with a browser: an issuer of `issuers`, a client of its own (`clientId`) whose one redirect URI is `https://login./callback`, the `scope` it asks for (`openid`; with `profile email` the dashboard names who signed in, and the owners of their runs, by name). `clientSecret` is for providers whose web clients need one, such as Google: read as an account no entitlement may name, like the secret of a [source](/docs/remote-compute/sources). Without `login`, only tokens open endpoints. | | `endpoints.session` | How long a sign-in at one endpoint holds, at most until its run ends; also how long a group or claim taken away at the issuer still counts. | | `endpoints.dashboard` | The host of the [dashboard](/docs/remote-compute/status#dashboard) below the domain, `hammerkit` by default (`https://hammerkit.run.example.com`); `false` for none. It needs `login`. | | `runs.up` | The time and the limits of service runs, see [Time](#time) and [Addresses](#addresses). | | `runs.up.keepVolumes` | Whether a run keeps the volumes of its services while it sleeps, see [Sleeping](#sleeping). `false` removes them when it sleeps. | The sign-in gets the claims an entitlement matches on from the ID token, so `group` and `claims` entitlements work for people signed in at an endpoint as for the API, as long as the ID token carries those claims (ask for them in `scope`). Behind a proxy (`behindTlsProxy: true`), the proxy holds the certificate of `*.` and sends every host of it to the server, with its `Host` header as it came, and lets WebSockets through (`Upgrade`). On Kubernetes that is one more host of the Ingress or Gateway of the server: ```yaml title="ingress.yaml" spec: tls: [{ hosts: [hammerkit.example.com, '*.run.example.com'], secretName: hammerkit-tls }] rules: - host: '*.run.example.com' http: { paths: [{ path: /, pathType: Prefix, backend: { service: { name: hammerkit, port: { number: 8080 } } } }] } ``` A server on the loopback interface with plain http serves endpoints without certificates, for trying it out, with `http://.:/` addresses. ### On a host [#on-a-host] The ports of endpoints are published on a free port of the loopback interface of the docker host, where only the server reaches them, so the docker daemon has to be on the server's machine. No other port of a service run is published, whatever its build file says: `hammerkit up` would publish them on every interface, where two runs collide. The volumes of a service run are its own (`hammerkit-up--`), not the ones a `hammerkit up` on the same docker uses. They stay while the run sleeps and go at its end; the server removes those of runs it no longer has when it starts. ### On Kubernetes [#on-kubernetes] A service run gets a namespace of its own, with the policies, quota (`runs.up.quota`) and rights of a caller's namespace, and goes with it at the end: the services stay up for hours while the caller's other runs go on in theirs. While the run sleeps, its namespace stays with the claims of its services, so their volumes count against its quota; its Deployments, Jobs, pods, Services and Secrets go. With `runs.up.keepVolumes: false` the namespace goes when it sleeps, and a woken run gets a new one. The server reaches an endpoint at the Service of its service, so it has to run in the cluster (not with a `kubeconfig`), and one more NetworkPolicy of the namespace lets the server's namespace in, to the ports of the endpoints only: a database of the run stays for its pods. That policy is what keeps the other pods of the cluster from reaching a service past the sign-in, so `endpoints` on Kubernetes needs [`backend.networkPolicy: enforced`](/docs/remote-compute/backend), set once the network plugin of the cluster enforces NetworkPolicy (GKE Autopilot and Dataplane V2 do). Any pod in the server's namespace gets that way in too, so run the server alone in its namespace: no runs, sandboxes or other workloads there. Its ClusterRole needs `delete` on namespaces, which an admission policy keeps to the namespaces of its service runs (see [What the server may do](/docs/remote-compute/backend#what-the-server-may-do)). A server without it (an upgrade that didn't add it) says so in its log and in the run's, and removes what runs in the namespace instead, which stays. When the server starts, it removes the namespaces of service runs that none of its runs is in, but those of runs that sleep, from which it removes what still ran. The outputs of the tasks a service depends on (an `install` that makes `node_modules`) are in the claim of the task, which the service's pod mounts. A task that is a hit in the server's [cache](/docs/remote-compute/backend#cache-and-sources) doesn't run: a short pod named after it (`install-…-upload`) takes its outputs from the server into its claim, and then the service starts. A miss runs the task's Job. The claim names the state whose outputs it holds whole, so a run that wakes, or moves to a commit where the task did not change, mounts them as they are: no Job and no upload, also after sleep removed the Jobs. A claim keeps the outputs of the states pods of the run mount and of the latest one; the others go when a state is built or restored. With `runs.up.keepVolumes: false` the claims go when the run sleeps, and a wake uploads them again. What a service prints goes into the log of the run, as on a host: while it starts, and while it is up, until the log of the run reaches its limit. A service that can't become ready fails the run with the reason the cluster gives, at once when waiting won't change it: an image the registry says is not there or not for this cluster, a container that crashed again after the cluster restarted it as often as the service's [`restarts`](/docs/service/container#restarts) allow (twice by default; with its exit code, or that it ran out of memory), or a network policy that did not take hold. A service without a healthcheck counts as ready once its container ran for 10 seconds without a restart. A reason that may still pass, such as a registry that limits pulls for now or a container that can't be created yet (a Secret that is not there yet), fails it once it lasted a minute; until then the log says what it waits for. A pod that waits for a node says so too, with what the scheduler said, and the run goes on waiting up to `maxDuration` for one. Before the namespace goes, the server writes into the log of a failed run what the cluster said: the pods that were not as they should be, and the warning events of the namespace. On a cluster a service run starts from a branch, not from a sandbox, whose claim is in the caller's namespace: push the sandbox to a branch and start that. ## Not covered yet [#not-covered-yet] * The app's own sign-in at a provider that wants its redirect URIs registered, see [What the app gets](#what-the-app-gets). * A service run from a sandbox on Kubernetes. * Services that keep running over a restart of the server: a run that may sleep wakes after it, the others fail. * A run that follows its branch on its own: a move happens on `remote up`, a wake starts the commit the run is at. * Updating the files of a running service in place on a host: every service starts again at the new commit, also one with `continuous: true`. * Keeping the volumes of a run that is replaced (another build file or other services). # Sources (/docs/remote-compute/sources) `sources` lists the repositories a run with `source: ref` may get its code from, and how the server reads each. The caller names a repository and a ref, never a URL, and the server fetches the code itself. ```yaml title="server.yaml" sources: # the repositories a run may get its code from - match: github.com/corp/* # an organisation secret: { from: 'gsm:github-readonly-token', account: scm-read } - match: gitlab.corp.com/platform/** # a group with its subgroups secret: { from: 'gsm:gitlab-group-token', account: scm-read, username: oauth2 } ``` | Key | What it does | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `match` | The repositories the entry covers, as a pattern of `host/path`. The first entry that matches decides. | | `secret.from` | The token, as `provider:reference` of a secret provider. The server reads with it and [sandboxes push](#pushing) with it. Without `secret` the repository is read anonymously and nothing pushes to it. | | `secret.account` | The [account](/docs/remote-compute/accounts) the token is read as. It needs credentials for that provider. | | `secret.username` | The user name of the basic authentication the token is sent with, `x-access-token` by default. A GitLab token is commonly sent as `oauth2`. | ## Which repositories a run may use [#which-repositories-a-run-may-use] A run from a `ref` names a repository and a ref in its request. The repository is written `host/path`, such as `github.com/corp/payments` or `gitlab.corp.com/platform/team/tool`; the server builds the address from it (`https://host/path.git`). A leading `https://` and a trailing `.git` are dropped; credentials, a port or a query are refused with `400`. Two lists decide whether it may be used, and a run needs both: 1. **The caller's `repos`**, the patterns of their entitlements added up ([Entitlements](/docs/remote-compute/entitlements)). A repository outside of them is `403 you may not run from ` at submit, before anything is approved or queued. A caller without `repos` can not start runs from a ref. 2. **The server's `sources`**, the repositories it can read at all. `match` is a pattern (`github.com/corp/payments` exactly, `github.com/corp/*` for one level, `gitlab.corp.com/platform/**` for a group with its subgroups, which covers GitLab subgroups) and the first source that matches decides. A repository none covers is `400 this server can not read `. Both lists use the same patterns, host names and paths are compared in lower case. ## The credential [#the-credential] The credential is a secret of a secret provider, read as a service account of the operator, like the secrets of a task: `secret.from` is `provider:reference`, `account` is the account it is read as, and `username` (default `x-access-token`) is the user name of the basic authentication the token is sent with. A provider command can mint a short-lived token, for example of a [GitHub App](#a-github-app); it is read again after a minute. A source without a `secret` is read anonymously. The token goes to git for that one fetch through the environment of the git process, scoped to the host of the repository, so it is never an argument, never written to `.git/config` or the checkout, never in the environment of a task, and masked in the logs of the run. The account that reads repositories is the server's own: no entitlement may name it and no build file may use it for a secret. ## A GitHub App [#a-github-app] A GitHub App gives the server tokens that belong to no person: each lasts an hour, reaches only the repositories the App is installed on and has only the App's permissions. `hammerkit github-app-token` mints one from the App's private key, so a server needs no script and no other runtime for it: the released image runs it as it is. ```yaml title="server.yaml" secretProviders: github-app: command: [hammerkit, github-app-token, '{{ref}}'] accounts: scm: description: The GitHub App of the server providers: github-app: env: GITHUB_APP_ID: '123456' GITHUB_APP_KEY_SECRET: projects/ci-project/secrets/github-app-key/versions/latest sources: - match: github.com/corp/payments secret: { from: 'github-app:corp/payments', account: scm } # this repository only - match: github.com/corp/* secret: { from: 'github-app:corp', account: scm } # all the installation reaches ``` The reference is `owner/repo` for a token limited to that one repository, or `owner` for one that reaches every repository the App's installation on that owner reaches. The command prints the token and nothing else. It reads its settings from the account's `env`, because a provider command sees nothing else of the server's environment: | Variable | What it does | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GITHUB_APP_ID` | The App's ID (or its client ID). Required. | | `GITHUB_APP_KEY_SECRET` | The App's private key in Google Secret Manager, as a version: `projects//secrets//versions/`. | | `GITHUB_APP_KEY_FILE` | The App's private key as a file, such as a mounted Kubernetes Secret, for a server outside Google Cloud. Set this or `GITHUB_APP_KEY_SECRET`, not both. | | `GITHUB_API_URL` | The API of a GitHub Enterprise Server, such as `https://github.corp.com/api/v3`. `https://api.github.com` by default. | | `GOOGLE_APPLICATION_CREDENTIALS` | With `GITHUB_APP_KEY_SECRET`: a credential configuration of workload identity federation (`type: external_account`) whose `credential_source` is a file, such as a projected Kubernetes ServiceAccount token. Without it the key is read as the identity of the metadata server (Compute Engine, Cloud Run, GKE with Workload Identity). A key of a Google service account is not taken. | The identity that reads the key needs `roles/secretmanager.secretAccessor` on that secret, and nothing else of Google's. Give the App the permissions the server needs: `Contents: Read-only` to read, `Contents: Read and write` when [sandboxes push](#pushing), and no `Workflows` permission, so a push that changes `.github/workflows` is refused by GitHub. An error says which step failed (the key, Google, Secret Manager, the installation, the token) and what the service answered, never the key or a token. The command writes it to stderr, and the server puts it in the error of the run or sandbox that needed the token. ## Pinning [#pinning] The server resolves the ref when the run is submitted, with `git ls-remote`: `refs/heads/`, else `refs/tags/` (an annotated tag to its commit). A name that is both a branch and a tag, or neither, is `400`, a repository it can not reach is `502`. The run stores the repository, the ref and the `commit`, and the approver sees them. The executor fetches the ref and fails the run if it no longer points to that commit, so a branch that moved after the approval does not run other code than the one approved. A 40-digit commit hash is taken as it is; an account with `refs` refuses one, because it names no branch. ## Pushing [#pushing] The server pushes the changes of [sandboxes](/docs/remote-compute/sandboxes#pushing) with the `secret` of the source, the same token it reads the repository with. A repository the server reads anonymously is not pushed to. Who may push to which branches is not the source's to say: the `sandboxes` of an [entitlement](/docs/remote-compute/entitlements) give a branch pattern the type `push`. ```yaml title="server.yaml" sources: - match: github.com/corp/* secret: { from: 'gsm:github-token', account: scm } # reads, and pushes the sandboxes entitlements: - subject: { iss: https://idp.corp, sub: 4f1c... } sandboxes: - repo: github.com/corp/payments branches: main: pull 'agent/**': push # agent/fix-login, agent/team/x; not main ``` A branch pattern is written like an account's `refs`: a name (`agent/fix`), `*` for a part of one level (`agent/*`) and `**` as the last level for everything below (`agent/**`). Keep the `push` patterns to branches that nobody else writes to and that no deployment runs from: a sandbox holds changes nobody reviewed, the push is how they get to a review. For a push the token needs write access to the repository's contents; a server that never pushes can keep a read-only one. As for a read, the token goes to one git process through its environment, is masked, and the account is the server's own: no entitlement may name it and no build file may use it. The pushing git process gets nothing else to sign in with: no global or system git configuration, no credential helper, no `.netrc` of a home directory and no ssh agent, so a push is always the service account's and never the identity of whoever runs the server. `hammerkit github-app-token` mints a short-lived token of a [GitHub App](#a-github-app) with write access to the repository's contents. A token can not be limited to branches on the host's side, so protect the branches people work on (`main`, releases) in the repository as well: a pattern that is too wide is then caught twice. **Pushed branches run in your CI.** Most repositories run CI on a push to any branch, with the repository's secrets: GitHub Actions `on: push`, GitLab pipelines, Buildkite. A branch a sandbox pushed holds code nobody reviewed, including the build scripts and tests CI runs, and the workflow files themselves when the token may write them. Before you give a pattern the type `push`: * keep the push patterns out of CI that holds secrets (`branches-ignore: ['agent/**']`), or put the jobs with secrets behind a review (environments with required reviewers, protected variables); * give the token no permission over workflow files (a GitHub App without `workflows: write`), so a push that changes them is refused by the host. # Status (/docs/remote-compute/status) A server runs a few runs at a time and queues the rest. Everything on this page is in the [dashboard](#dashboard) of a server with endpoints, too. When a run of yours does not start, `hammerkit remote status` says why: every place is taken, you are at your own limit of running runs, another run works on its sandbox, or it waits for an approval. Each of your limits lists the runs that count against it, so you know which one to stop. ```txt $ hammerkit remote status build: signed in as alice places: 4 of 4 taken (runs.maxConcurrent) run_a8Zt0pQw3mVx7yLc test, lint since 2026-11-07T10:01:12.000Z, 6m 2s run_c2Lk9sDe4rTy6uIo build since 2026-11-07T10:05:40.000Z, 1m 34s run_Hn3bV6cX9zL2kJ5g e2e of bob since 2026-11-07T10:02:03.000Z, 5m 11s 1 run of others queue: 2 1. run_f1Gh5jKl8zXc3vBn deploy waiting 2m 10s waits: your limit of running runs, 2 of 2 (runs.limits.running) 2. run_Pq8wE2rT5yU1iO4p lint of bob waiting 40s waits: no free place, 4 of 4 taken (runs.maxConcurrent) waiting for an approval: 1 of at most 100 run_k9Qw2eRt5yUi8oPa e2e needs op-payments-prod until 2026-11-07T11:42:00.000Z service runs: 2 of 20 places taken on the server run_Vd1rX9pQ2mT7wN4a up feature-login: up web until 2026-11-09T10:12:00.000Z, sleeps at 2026-11-07T10:42:00.000Z web:8080 https://feature-login-8080-3fa2c1e09b7d.run.example.com run_Ws8tY3uI6oP1aS4d asleep feature-cart: up web until 2026-11-08T09:00:00.000Z your limits: running 2 of 2 run_a8Zt0pQw3mVx7yLc, run_c2Lk9sDe4rTy6uIo full queued 1 of 20 run_f1Gh5jKl8zXc3vBn approval 1 of 10 run_k9Qw2eRt5yUi8oPa services 1 of 3 run_Vd1rX9pQ2mT7wN4a asleep 1 of 10 run_Ws8tY3uI6oP1aS4d capacity: 3 nodes with 12 CPU, 45 GiB, 4.1 CPU, 19 GiB in use runs: 14 pods, requested 5.25 CPU, 11 GiB, in use 2.8 CPU, 7.4 GiB ``` `--json` prints the overview as the server answers it, for a script or an agent. ## The queue [#the-queue] The queue is in the order the server starts runs. When a place frees up, the first run in it whose owner has a place of their own left starts, so one caller with many runs does not keep the others waiting (see [Limits and restarts](/docs/remote-compute/runs#limits-and-restarts)). Each run in the queue says how long it has waited and what holds it, the first that applies: | Waits for | What it means | What helps | | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | | your limit of running runs | You have `runs.limits.running` runs executing. A free place would not start it. | Wait for one of yours, or stop it with `remote stop`. | | its sandbox | Another run works on the same [sandbox](/docs/remote-compute/sandboxes), which runs one at a time. | Wait for that run. | | a run that holds what it needs | On Kubernetes, runs of one caller that share kept claims run one at a time, see [Namespaces](/docs/remote-compute/backend#namespaces). | Wait for that run. | | no free place | Every place of `runs.maxConcurrent` is taken. | Wait; the operator may raise `runs.maxConcurrent`. | A run that waits for an [approval](/docs/remote-compute/entitlements#approvals) is not in the queue yet: it is listed under `waiting for an approval` with the accounts it still needs and when it is rejected without an answer. Its wait in the queue counts from its approval. `places` counts the places taken. A service run takes one while it starts and gives it back once it is up. A line `N taken by no running run` means the server counts a place that no run holds, which is a fault worth reporting with the output of `remote status --json`. ## Your limits [#your-limits] | Line | Counts | Its limit | | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | | `running` | Your runs that execute, service runs while they start | `runs.limits.running` | | `queued` | Your runs in the queue, and service runs that are waking | `runs.limits.queued` | | `approval` | Your runs that wait for an approval | `runs.limits.waiting` | | `services` | Your service runs that are starting or up, one place per name: a newer run of a name replaces the older one; one that sleeps takes none and asks for one when it wakes | `runs.up.limits.up` | | `asleep` | Your service runs that sleep, one per name | `runs.up.limits.asleep` | A submit over `queued` or `approval` is answered `429` and says which; over `services` or `asleep` too. A run that ended counts against nothing. ## Service runs and past runs [#service-runs-and-past-runs] `hammerkit remote services` lists the service runs that are starting, up or asleep: yours, and those of others you may see, with their addresses (where you may open them) and when they end or sleep. `--ended` adds the ones that ended, as long as the server keeps them. `hammerkit remote runs` lists your runs, newest first, as many as the server keeps (`runs.keep`, 200 by default), each with how long it waited for a place and how long it ran (a service run: until it was up). A queued run says where it stands and why it waits. | Option | What it does | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `--state ` | Only runs in these states, comma separated: `awaiting-approval`, `queued`, `running`, `up`, `asleep`, `succeeded`, `failed`, `cancelled`, `rejected`. | | `--services`, `--tasks` | Only service runs, or only runs of tasks. | | `--all` | Also the runs of others you may see, each with its owner, see [Who sees what](#who-sees-what). | | `--limit ` | Show more than 20. | Read the log of one with [`remote logs`](/docs/cli/remote#reading-a-run-again): what its tasks printed, what its services printed under their names, and the server's own lines under `hammerkit`. ## Past tasks [#past-tasks] `hammerkit remote tasks` sums up, per task and repository, the runs the server keeps: how often it ran, failed and came from the cache, what it took the last time, the median and the 90th percentile of the times it executed (a task from the cache does not count), and how long its runs waited for a place. `--mine` counts only your runs, `--since 7d` only the runs of the last 7 days; `--since` and `--until` also take a time, such as `2026-11-01T08:00:00Z`. `--json` prints it as the server answers it. ```txt $ hammerkit remote tasks test github.com/corp/shop last succeeded 2026-11-07T10:12:40.000Z took 2m 41s 37 runs, 3 failed, 12 cached median 2m 30s, 90% 3m 5s, waited 12s e2e github.com/corp/shop last failed 2026-11-07T09:58:02.000Z took 8m 9 runs, 2 failed median 7m, 90% 9m, waited 1m 40s ``` The run page of the dashboard shows the same median beside each task, as `Usually`. ## Capacity [#capacity] `hammerkit remote capacity` shows what the machines of the server have and what the runs take of them, CPU in cores and memory: ```txt $ hammerkit remote capacity capacity: 3 nodes with 12 CPU, 45 GiB, 4.1 CPU, 19 GiB in use runs: 14 pods, requested 5.25 CPU, 11 GiB, in use 2.8 CPU, 7.4 GiB hk-3fa2c1e09b7d4a55 caller of alice 6 pods requested 2.5 CPU, 5 GiB of 8 CPU, 16 GiB in use 1.2 CPU, 3.1 GiB hk-91c0d7be5e2f3a18 service run of bob run_Vd1rX9pQ2mT7wN4a 2 pods requested 0.5 CPU, 1 GiB of 2 CPU, 4 GiB in use 0.1 CPU, 600 MiB ``` On **Kubernetes** a line per namespace of the server, a caller's or a service run's: what its pods requested of the scheduler, the quota the server set on it (`backend.quota`, `runs.up.quota`) and what the pods use now. Requests and the quota need nothing more. What pods use comes from the metrics API (metrics-server, which GKE, EKS and AKS run), and the nodes from listing them; give the server these rules for both, or the output says what it could not read and shows the rest. What the cluster answered is shown to an `observe` entitlement; others are told that some of it could not be read. A read that takes longer than 3 seconds is not waited for: the next look shows it. ```yaml title="rbac.yaml" # in hammerkit-runs, bound by the server in its namespaces: what the pods use - { apiGroups: [metrics.k8s.io], resources: [pods], verbs: [list] } # in hammerkit-server, optional: the nodes, what they have and use - { apiGroups: [''], resources: [nodes], verbs: [list] } - { apiGroups: [metrics.k8s.io], resources: [nodes], verbs: [list] } ``` On a **host**, the machine of its docker daemon: its CPUs and memory, the memory in use and the load when the daemon runs on the server's own machine, and what the containers of runs use. The server reads it at most every 15 seconds. Everyone with an entitlement sees the totals; whose a namespace is, its owner and `observe` see. ## Who sees what [#who-sees-what] Everyone sees their own runs in full. What a caller sees of the runs of others depends on `runs.activity` in the server's config: * `everyone` (the default): a caller with a `builds` level sees the runs of others in the repositories they have access to, those of their `repos` and of their `sandboxes`: whose a run is, what it runs (its tasks or services), its source, its state, where it waits, how long it waited and ran, and how long each task took. Not its arguments, what its tasks run, why it failed, its log, or the addresses of its endpoints. Who may only open endpoints or read checks (a preview reviewer, a CI job) sees none of it, and a run of uploaded files, which has no repository, is for its owner and who is involved. * `involved`: a caller sees only the runs of others they are involved in, below. Whatever `runs.activity` says, a caller also sees: * the runs that wait for an approval of an account they `use`: they decide about them, and read them whole, with their plan and their log; * the service runs of a repository whose endpoints they may open (an `endpoints` level), with their addresses; * every run whole, with its log, with `observe: true` on one of their entitlements. A run a caller may not see counts in the totals and stands in the queue as `a run of somebody else`, with no id, owner or task; a run that holds one of yours is named only when you may see it. `observe` is for the people who run the server and need to see why the queue is stuck or why a run failed: ```yaml title="server.yaml" entitlements: - group: { iss: https://login.example.com, name: platform } observe: true ``` It only shows. It stops, extends and approves nothing, and runs nothing. ## Dashboard [#dashboard] The same in a browser: a server with [endpoints](/docs/remote-compute/service-runs#server-config) and a browser sign-in (`endpoints.login`) serves a dashboard at `https://hammerkit.`, for example `https://hammerkit.run.example.com`. Opening the address of the server itself in a browser goes there too, and `hammerkit remote dashboard` prints it. The overview of the dashboard: the queue with why each run waits, the capacity and the service runs | Page | What it shows | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Overview | What `remote status` prints: the queue with how long each run has waited and why (only while runs wait), the capacity: CPU and memory as requested of the total, and its active resources, the runs (their tasks or services, repository and owner) and sandboxes that take the most first, with what they requested and use (on Kubernetes), and in a few words why a total or what runs use is not known. The service runs are on Services. | | Services | The service runs that are starting, up or asleep, then those that ended in the last day, by repository as on Tasks, narrowed by text (service, ref, build file, person) and by state, source and user, as on Runs. A run shows its state mark (a click opens the run), its services, each a link to its endpoint where you may open it, with its state below; its branch and commit with its build file below; and when it started and by whom (when it sleeps and ends on hover). Where you may, a button keeps a run up for longer, a triangle wakes one that sleeps, and a square stops it. | | Tasks | What `remote tasks` prints, by repository, over the last hour, day, 7 days (by default), 30 days, a range from and to a time, or all the runs the server keeps, with the build file each task ran from the last time (a link to its page and its hash) and how often it came from the cache and succeeded, sortable by each column, and narrowed as you type by a task, a build file or a repository, and by source with the filter below the search, as on Runs. | | Runs | The runs of everybody you may see, newest first, over the same periods as Tasks (7 days by default), with their state, how long they took in all (what they waited for a place and ran on hover), and when and by whom they were created, narrowed as you type by a task, a service, a branch, a person or an id, and by state, source and task with the filters below the search. A run that waits for an account you use is marked **your approval**. | | A run | Its state, and **Approve** or **Reject** (with a reason) when it waits for an account you use; a retry button (an arrow in a circle) on a run that is over, which starts it again as a new run of yours and opens it, see [Retrying a run](/docs/remote-compute/runs#retrying-a-run); its source, build file and tasks; its owner, when it was created and its approvals; where it stands in the queue, how long it waited and ran; a graph of its tasks and services as a build file shows it, each in the color of how it went, with how many of its tasks came from the cache; a timeline of where its time went since it was submitted: a line of the run itself with its [phases](/docs/remote-compute/runs#results-and-waiting) (the queue, the checkout, the namespace, the plan, and the cache check, where the keys of the tasks are computed from their files), then a line per task and service with its steps in colors: what it waited for (the queue for a task that depends on nothing, else its dependencies), the setup of its pod (scheduled, image pulled, caches restored and stored) and when its commands ran (in the color of how it ended), a task taken from the cache a marker at the moment it was, with the totals of each above, each moment counted once when tasks run side by side; the services it brings up, then its tasks in the order they run, those still to come too, each with how long it took and the tasks and services it waits for, and what each does (its image, commands, dependencies and secrets, from the run or, for who may see build files, from its build file). A click on a task in the graph opens it in a panel beside the page: how it went and why it was not cached, what it does whole (its commands, sources, what it generates, its secrets, accounts, network, what it runs after and what runs after it, each a click away), and for who may read the run whole its log. Who may read it whole also sees its arguments, why the tasks failed, and, in a **Log** tab next to **Command** when a task or service opens below it, the last 200 lines of its log (one that failed opens by itself, one that runs keeps following). | | Log | All of the log of a run, a thousand lines a page, of all or of one task or service, a text to look for with the lines around it, and **download** as a text file. | | Build files | The versions of build files by repository (as the `repo` annotation of the publish names it), each with its state: published, yanked, waiting for somebody who may publish (marked **your approval** when that is you), or rejected or expired in the last day; with their path, commit, tasks, and who published or proposed them when. A proposed version opens to the graph of its tasks and services, as a build file does (with its toggles per build file), and every file, with **Publish** or **Reject** for who may; nobody decides about their own, as with [proposals](/docs/cli/remote#proposing-a-version). | | A build file | One published version, in three tabs. **Info**: its path in the repository, hash, who published it, its repository and commit, accounts; for a version that names its repository, what the runs of its tasks in the last 7 days add up to (task runs, how many came from the cache, how many passed, the time they ran); and a table of its tasks and services with the build file that declares each, what each waits for and, from those runs, how often it ran, came from the cache and passed and its median time. **Graph**: its services in rows on top and its tasks below, laid out from left to right in the order they run with an arrow from what a task depends on to it; the arrows to a service show while it or a task that needs it is hovered or opened, a number on a task says how many services it needs, each box how often it came from the cache, and a dot what to look at, such as a local task or a host port. Below it, a toggle per build file hides the tasks and services it declares (a version published before the plan named the build files groups them by the prefix of their include). **Files**: its files as they are. A click on a task or service in the table or the graph opens what it does whole in a panel beside the page, with its runs of the last 7 days. For who may see build files (a `builds` level, an account, or `observe`). | | Sandboxes | Your [sandboxes](/docs/remote-compute/sandboxes) by repository, with how many you have and the space they take against your limits, the branch and commit of each, what it pushed, its size, when it was last used, and the run that works on it now, and **Delete** (once asked: a run on it stops, what was not pushed is gone). Nobody else sees them. | | A sandbox | Its repository, branch, commit, size and run, and what changed in its files since the commit: each file with the lines added and removed, and the diff, and **Delete**. The page does not change or push a sandbox. | What changes is read again every 10 seconds, without reloading the page: a reason being typed for a reject stays. Nothing more needs setting up: the dashboard signs people in the way an endpoint does, through `login.`, with the redirect URI already registered, and sets a cookie for its own host only. Who signs in needs an entitlement on the server, and sees what [Who sees what](#who-sees-what) says, checked with the entitlements of the moment on every page. The groups and claims of a person are the ones of their sign-in, for as long as the session holds (`endpoints.session`, 8 hours by default): somebody taken out of a group at the issuer keeps what the group gave them to look at and to stop or extend until they sign in again. **Approve**, **Publish**, **Reject** and the retry button take a sign-in of the last 15 minutes (a retry starts a run with the accounts your groups give you): an older one shows *Sign in again*, which goes through `login.` and back with the groups of now (at an issuer that still knows the person, without a password). A decision is logged with its reason, as one through the API. **Stop** is for the owner and who decides about a run, **Extend** for the owner of a service run and who may open its endpoints. **Retry** is for who may read the run whole and may run from its repository (a run on a sandbox: its owner); the new run is theirs, decided with their entitlements as a run they start. An `observe` entitlement alone sees every run and changes none. The dashboard is an app of the server (React), which reads and does everything through the [API](#api) as the person who signed in: its host passes `/api/v1/...` on to `/v1/...` for what the pages read and the buttons do, and refuses the rest (starting runs other than a retry, publishing, sandboxes). So it shows what the command line shows, with the same rules. Its pages load nothing from anywhere else, run no inline script, and no other site may frame them. What the buttons send carries a token of the person and of their sign-in, which ends with it; a request from another site, or from an endpoint of the same domain, is refused. To switch the dashboard off, set `endpoints.dashboard: false`; to put it at another host, name it there (`dashboard: status` for `https://status.run.example.com`). `--json` of `remote status` and the API below give the same to a script. ## API [#api] `GET /v1/overview` answers what `remote status` prints. A caller with no entitlement is answered `403`. ```json { "you": { "iss": "https://login.example.com", "sub": "alice", "observe": false }, "places": { "max": 4, "used": 4, "runs": [{ "id": "run_a8Zt0pQw3mVx7yLc", "state": "running", "tasks": ["test", "lint"] }], "others": 1 }, "queue": [ { "position": 1, "mine": true, "run": { "id": "run_f1Gh5jKl8zXc3vBn", "state": "queued", "tasks": ["deploy"], "queuedAt": "2026-11-07T10:05:02.000Z" }, "waitsFor": { "type": "own-limit", "running": 2, "limit": 2 } }, { "position": 2, "mine": false, "waitsFor": { "type": "place", "used": 4, "max": 4 } } ], "approvals": { "waiting": 1, "max": 100, "runs": [] }, "services": { "open": 3, "max": 20, "runs": [] }, "limits": { "waiting": { "used": 0, "limit": 10, "runs": [] }, "queued": { "used": 1, "limit": 20, "runs": ["run_f1Gh5jKl8zXc3vBn"] }, "running": { "used": 2, "limit": 2, "runs": ["run_a8Zt0pQw3mVx7yLc", "run_c2Lk9sDe4rTy6uIo"] }, "services": { "used": 0, "limit": 3, "runs": [] } } } ``` Runs are shown as `GET /v1/runs` lists them (shortened here): with `queuedAt`, `startedAt`, `finishedAt` and, per task, `timings` (`status`, `durationMs`, `cached`); a run of somebody else with its `owner`, without `options`, `plan` and `error`. `waitsFor.type` is `own-limit`, `sandbox` or `held` (with the `run` that holds it, when you may see it), `place`, or `starting` for a run that starts with the next look at the queue. `GET /v1/runs` takes the same filters as `remote runs`: | Parameter | What it does | | --------- | ---------------------------------------------------------------------------------------------------------------------------- | | `scope` | `mine` (the default), `visible` (also the runs of others you may see) or `all` (every run, needs `observe`, else `403`). | | `state` | One or more states, comma separated. | | `kind` | `tasks` or `services`. | | `id` | One run, as the list shows it: a run of somebody else that `GET /v1/runs/` keeps for its owner and who decides about it. | A queued run, in the list and in `GET /v1/runs/`, carries where it stands: `"queue": { "position": 1, "waitsFor": { "type": "place", "used": 4, "max": 4 } }`. With `observe`, `GET /v1/runs/` and `GET /v1/runs//logs` answer for every run. A run carries what you may do with it, so a page shows only the buttons that work: `"may": { "stop": true, "extend": true, "decide": true }`, with what you may not left out (`stop` for its owner and who decides about it, `extend` for a service run that was up and who may open it, `decide` for a run that waits for an account you use, `retry` for a run that is over that you read whole and may run from where it ran). A run of somebody else that `GET /v1/runs/` answers you whole (you decide about it, or `observe`) is listed with `"readable": true`. A listed run names where in its repository its build file is (`entry`), next to its version (`definition`). `GET /v1/runs//logs` names what the log can be narrowed to with `task`: `"names": ["build", "web", "hammerkit"]`, where `hammerkit` is the server's own lines. `GET /v1/tasks` answers what `remote tasks` prints, over the runs you may see (`scope=mine` for your own) and created in a period (`since` and `until`, each back from now such as `7d` or a time such as `2026-11-01T08:00:00Z`; `remote tasks --since --until`): `{ "tasks": [{ "repo", "task", "runs", "succeeded", "failed", "cached", "last", "medianMs", "p90Ms", "totalMs", "medianWaitMs" }], "oldest" }`, where `oldest` is when the oldest run it counts was created, and `last` is how the task ran the last time (`status`, `at`, `durationMs`) and from which build file: its version (`definition`) and its path in the repository (`entry`). `GET /v1/capacity` answers what `remote capacity` prints: `backend`, `machines` (`count`, `allocatable`, `used`, `load`), `runs` (`requested`, `used`, `pods` or `containers`), `groups` (per namespace: `name`, `kind`, `owner`, `run`, `pods`, `requested`, `quota`, `used`), `uses` (on Kubernetes, per run or sandbox, the most used first: `kind` `run`, `sandbox` or `other`, `run`, `tasks`, `up`, `repo`, `owner`, `pods`, `requested`, `used`; of the runs of others only what they take, but to an `observe` entitlement), `problems` and `unknown` (`total` and `used`: why the total of the machines or what runs use could not be read, in a few words, such as `may not list nodes` or `no metrics API`), CPU in cores and memory in bytes. # Container service (/docs/service/container) Container services run in the same network as the other container tasks. By default, they are started when a task needs them and stopped once no remaining task needs them. ## Differences from Docker Compose [#differences-from-docker-compose] Container services are similar to [Docker Compose](https://docs.docker.com/compose/compose-file/) services, but are better integrated with your tasks. They lack many of Compose's features, but a few aspects still make them appealing. ### Seamless integration [#seamless-integration] Hammerkit is aware of the task dependency tree and can make decisions about the lifetime of a service. Depending on your setup, this can reduce CPU and memory usage. It keeps things simple and removes the need for another tool that has to be started, and waited for, before you can run your task. ### Reuse [#reuse] Services can be included or referenced like tasks. This makes it possible to reuse services and their state across multiple build files and projects — for example, to run a single database for several projects and save CPU and memory. ### Store/Restore [#storerestore] The [store/restore](/docs/cli/store-restore) commands work for services as well, exporting and importing their volume data. This can be used to seed a database, or to back up and restore project data. ## Ports [#ports] Services can expose ports to the host machine. This is useful for debugging or for connecting to a service from your machine. Host ports are published when the service runs through [`hammerkit up`](/docs/cli/up), and during `hammerkit run` only when a [local task](/docs/task) needs the service (it reaches services through the host). Container tasks reach a service over its link, so a `run` that only has container tasks publishes nothing, and two runs on one machine never collide on a host port. Container tasks don't need published ports. They share a network with the services and can reach all of their ports. On Kubernetes, a service without `ports` is reached on every port of its pod; one with `ports` is reached through its Service, on the listed ports only. ```yaml services: postgres: image: postgres:16-alpine ports: - 5432:5432 ``` ## Shell [#shell] A service's `cmd` and its healthcheck `cmd` run in exec form by default: the command line is split into arguments and handed to the container directly, and the `cmd` becomes the arguments of the image's entrypoint. Set `shell` to run both through a shell instead, like a container task's [execution shell](/docs/task/container#execution-shell): ```yaml services: status: image: alpine:3.21 shell: /bin/sh cmd: mkdir -p /srv && printf 'HTTP/1.0 200 OK\r\n\r\nready on %s\n' "$(uname -s)" > /srv/response && while true; do nc -l -p 8080 < /srv/response; done healthcheck: cmd: grep -q "ready on" /srv/response && nc -z 127.0.0.1 8080 ``` With `shell`, the service starts as ` -c ""` in place of the image entrypoint, on Docker and Kubernetes alike, and the healthcheck runs as ` -c ""`. The shell has to exist in the image. Without a `cmd`, the image's own entrypoint and command are kept. ## Healthcheck [#healthcheck] A healthcheck checks whether a service is ready to be used. A task only starts once every service it needs is running and, if a healthcheck is defined, has passed it. A healthcheck requires a command that can be used to test the readiness of a service. The command is executed inside the service container. To pass the healthcheck the command needs to return with an exit code of `0`. The following example contains a Postgres service. It uses the `pg_isready` executable in a healthcheck, which ensures the Postgres server is ready to accept connections. ```yaml services: postgres: image: postgres:16-alpine healthcheck: cmd: "pg_isready -U postgres" ``` Without a healthcheck, a task may start before the service is ready. The `cmd` runs in exec form: it is split into arguments (quotes group one argument), with no shell in between — so `&&`, pipes and `$(...)` need the service's [`shell`](#shell). ### Timing and failure [#timing-and-failure] The healthcheck has two fields: `cmd` and an optional `timeout`. There are intentionally no `interval` or `retries` knobs: * hammerkit runs `cmd` inside the service container roughly **once a second**, starting one second after the container starts, and treats the service as ready the first time it exits `0`; * the check runs as the **user of the container's main process** (PID 1), like a Kubernetes exec probe runs as the container's user. Images whose entrypoint drops privileges (`exec su-exec rabbitmq …`) are checked as that user, so a check can't create files the service then cannot read — a RabbitMQ CLI run as root would create the Erlang cookie the server is locked out of; * each individual check has a short execution timeout (a couple of seconds), so a hanging check doesn't block forever — it just counts as "not ready yet" and is retried; * a service that doesn't pass within **`timeout` (default 20s)** of its container starting fails, with the healthcheck command in the error, and so does every task that needs it — instead of the run waiting until you cancel it. The clock starts once the container runs, so pulling the image doesn't count. Raise it for slow starters, per service or for every service without one through `HAMMERKIT_HEALTHCHECK_TIMEOUT`: ```yaml services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:8.18.1 healthcheck: cmd: curl -sf http://localhost:9200/_cluster/health timeout: 90s ``` On [Kubernetes](/docs/task/kubernetes), the same `cmd` is translated into readiness and liveness probes on the deployment, and `timeout` bounds the wait for a ready replica from each start of its container. A startup probe with the same `cmd` gives a container its `timeout` to pass it once before the liveness probe can restart it. A service without a healthcheck counts as up once its container ran for 10 seconds without a restart, so one that exits right after it started fails `up`. ### Restarts on Kubernetes [#restarts] On Kubernetes the cluster restarts a service's container that exits, with a back-off of 10s, 20s, 40s and so on between starts. While hammerkit waits for the service to become ready, it fails the service with how the container exited once it crashed again after `restarts` restarts: by default 2, so at the third crash, about 35 seconds in. Raise it (up to 10) for a service that may crash a few times while what it needs starts, or set `0` to fail at the first crash. A healthcheck's `timeout` counts from each start of the container, so the back-off between two starts doesn't use it up: ```yaml services: api: image: node:22-alpine cmd: node server.js restarts: 5 ``` On Docker a service whose container exits fails at once, whatever `restarts` says. On Kubernetes hammerkit follows what the service's container prints, also of an instance that crashed before it was seen. A line ends at a line feed or a CR (a progress bar's updates are lines of their own); a line longer than 64 KiB is cut there, and says so. ## Working directory [#working-directory] A service runs in the build file's directory, like a task: its `src`, `mounts` and the outputs of its `deps` are mounted there, so a command such as `node server.js` finds its files. Images that start a process relative to their own `WORKDIR` — an entrypoint calling `./dist/index.js`, or `npm start` in `/app` — need that directory back. Set `workdir` to an absolute path inside the container: ```yaml services: core: image: registry.example.com/core:6 workdir: /app/packages/core ``` `workdir` applies to the Docker and Kubernetes runtimes alike and is part of the service's identity; services that don't declare it keep their id. ## A service that watches its files [#continuous] A dev server that rebuilds on its own when its files change (`ng serve`, `vite`, `nodemon`) is a service with `continuous: true`: ```yaml services: web: image: node:24-alpine continuous: true deps: [install] src: - src - angular.json cmd: npx ng serve --host 0.0.0.0 ports: - 4200 ``` On your machine its `src` and `mounts` are mounted from your files, so it sees every save, and `hammerkit up --watch` leaves it running instead of restarting it. In a [service run](/docs/remote-compute/service-runs#moving-to-another-commit) on Kubernetes, a move to a commit that changes only its `src` or `mounts` writes the changed files into the pod that runs, and removes the ones that are gone: the pod stays, and the dev server reloads. If it then restarts (even once, whatever its `restarts` allows) or stops being ready within 15 seconds, the move fails and the files of the commit before are put back. A path of its `src` or `mounts` that the new commit doesn't have is emptied rather than removed, since the pod mounts it. Its `deps` work as for any service. When the output of one changes (a new `node_modules`), or its image, command or variables do, it gets a new pod. The files reach the pod through its claim; a watcher that relies on file events needs the claim on a disk of the node (the default of most clusters), and polling on a network file system. ## Volumes [#volumes] Services start and stop depending on what tasks need. To persist data across restarts, use volumes. ```yaml services: postgres: image: postgres:16-alpine volumes: - "postgres-db:/var/lib/postgresql/data" ``` A volume name is a Docker volume, the same for every checkout on that Docker host. Where several checkouts share one host — CI jobs on one runner — name it after one of the service's `envs`, so each job gets its own: ```yaml services: postgres: image: postgres:16-alpine envs: DATA_PREFIX: ${CI_JOB_ID:-dev} volumes: - "${DATA_PREFIX}-postgres:/var/lib/postgresql/data" ``` ## Init [#init] Setup that has to happen once the service is up — create buckets, initiate a replica set, assign roles in an identity provider — goes into a task the service names as its `init`. The task runs after the service passed its healthcheck, and the service counts as ready only once it succeeded: whatever else needs the service starts against a set-up service. A failing or timed-out init fails the run. ```yaml services: keycloak: image: quay.io/keycloak/keycloak:25.0.6 healthcheck: cmd: … init: keycloak-roles tasks: keycloak-roles: image: curlimages/curl:8.10.1 mounts: - ./assign-roles.sh:/assign-roles.sh:ro envs: KEYCLOAK_URL: http://keycloak:8080 secrets: - from: env:KEYCLOAK_ADMIN_PASSWORD env: KEYCLOAK_ADMIN_PASSWORD cmds: - sh /assign-roles.sh ``` `init` names a task like `deps` does (`name` or `prefix:name`), so the init has everything a task has: `deps`, `envs`, [`secrets`](/docs/task/secrets), `mounts`, `extend` for one setup shared by several services, or a local task. The task needs the service under the service's name without listing it; it may list other services it needs as well. A service of an [include](/docs/build-file/includes) is reached by its name in its own file (`db`, not `services:db`), or by the `name:` the task lists it under (`needs: [{ name: keycloak, service: idp }]`). * **It runs each time the service starts in a run**, never from the cache. A service kept running by `up --daemon` already ran its init and is reused without running it again; name the task (`hammerkit run keycloak-roles`) to run it against the running service anyway. Asked for by name while the service is not running, the service starts for it and the init runs once. * **Its `timeout`** defaults to 5 minutes (`HAMMERKIT_INIT_TIMEOUT` changes it). * **A changed init task recreates the service** on the next `up` or `run`, so the init runs again. * A service with an `init` is always awaited, also under `up --daemon --wait start`. ## Hostname [#hostname] A service's container uses the service name as its hostname, so the service resolves its own name: software that advertises its own address, such as a MongoDB replica-set member `mongodb:27017`, works without knowing the container's IP. Other containers reach it by the name they list it under in `needs`. ## Secrets [#secrets] Credentials reach a service through `secrets`, as an environment variable or a read-only file, without being written to the build file or hammerkit's logs; its `init` gets them too. See [secrets](/docs/task/secrets). ## Resources [#resources] `resources` limits a service's container like a [task's](/docs/task#resources), and `cpus`/`memory` or `requests`/`limits` work the same. A service holds its requests for as long as it runs, so the tasks of the run share what is left, and `up` fails up front when the services together request more than the host has (`--skip-resource-check` starts anyway). On Kubernetes the deployment's container gets the same requests and limits. A running service whose resources changed is recreated. ```yaml services: postgres: image: postgres:16-alpine resources: cpus: 500m memory: 1Gi ``` ## Network [#network] `network` limits what the service reaches, like a [task's](/docs/task/network): `none` for nothing but the services it needs, `{ allow: [hosts] }` for those hosts through an egress proxy, `open` (the default) for everything. The tasks and services that need it reach it whatever its level. The healthcheck checks the service itself, so it runs without the proxy variables. A running service whose network changed is recreated. ```yaml services: payments: image: node:24-alpine network: allow: [api.stripe.com] ``` ## Mounts [#mounts] Mounts can be used to pass local files into the container. They are meant for local configuration and settings files. Append `:ro` to mount one read-only (`./postgres.conf:/etc/postgresql/postgresql.conf:ro`); volumes take `:ro` too (`secrets:/run/secrets:ro`, or `readOnly: true` in the object form). Mounts are not recommended for frequently changing files. Syncing large directories on macOS/Windows uses a lot of CPU. ```yaml services: postgres: image: postgres:16-alpine mounts: - "./postgres.conf:/etc/postgresql/postgresql.conf" ``` # Service (/docs/service) Services help tasks do their work — for example, a database for your application or an API for your integration tests. Services can run in a container or be forwarded from a Kubernetes cluster. Each service a task needs is reachable on the container network by the service's name. Local tasks can also use needs, but hammerkit does not provide DNS resolution for them. Instead, the connection details of each needed service are passed to the task as environment variables (see [environment hints](#environment-hints-for-local-tasks)). ## Container [#container] Like [container tasks](/docs/task/container), container services need an image to run. Tasks declare which services they need, and hammerkit ensures those services are running before the task executes. Use [healthchecks](/docs/service/container#healthcheck) to ensure that a service is running and ready to be used. A password reaches the service as a [secret](/docs/task/secrets), never written in the build file. ```yaml services: postgres: image: postgres envs: POSTGRES_USER: api POSTGRES_DB: api secrets: - from: env:DB_PASSWORD # never a password in the build file env: POSTGRES_PASSWORD healthcheck: cmd: "pg_isready -U api" ports: - 5432 tasks: api: image: node:alpine needs: [postgres] cmds: - node index.js ``` ## Kubernetes [#kubernetes] [Kubernetes services](/docs/service/kubernetes) let tasks use resources from local and remote Kubernetes clusters. This makes it easy to integrate with resources from development or staging environments, and to debug or troubleshoot your application with data from other environments. ```yaml services: postgres-staging: context: staging ports: - 5432:5432 selector: type: deployment name: postgres tasks: api-staging: image: node:alpine needs: [postgres-staging] cmds: - node index.js ``` ## Connecting from a container task [#connecting-from-a-container-task] A container task shares a network with its needed services and reaches each one by its **service name** as the hostname, on the service's **container port**. You don't publish a port for this — `ports` are only needed to reach a service from your host. For the `postgres` service above (exposing `5432`), a container task connects with a URL like `postgres://api@postgres:5432/api` — `postgres` is the service name and `5432` the container port. Keep that string in a shared top-level `envs` block so the service and the task agree on it: ```yaml title=".hammerkit.yaml" envs: DATABASE_URL: postgres://api@postgres:5432/api services: postgres: image: postgres:16-alpine envs: POSTGRES_USER: api POSTGRES_DB: api POSTGRES_HOST_AUTH_METHOD: trust healthcheck: cmd: "pg_isready -U api" ports: - 5432 tasks: migrate: image: node:24-alpine needs: [postgres] envs: DATABASE_URL: $DATABASE_URL cmds: - node migrate.js ``` ## Environment hints for local tasks [#environment-hints-for-local-tasks] A local task (a task without an `image`) is not part of that container network, so hammerkit instead injects the connection details as environment variables. For every needed service the following variables are available, where `NAME` is the uppercased service name: * `HAMMERKIT__HOST` - host the service is reachable on. * `HAMMERKIT__PORT` - the primary published port. * `HAMMERKIT__PORT_` - the published port for a specific container port. The suffix is always the numeric container port (ports cannot be named), so this is mainly useful for services that expose more than one port. For a service named `postgres` exposing port `5432` a local task receives `HAMMERKIT_POSTGRES_HOST`, `HAMMERKIT_POSTGRES_PORT` and `HAMMERKIT_POSTGRES_PORT_5432`. ## Service dependencies [#service-dependencies] Like tasks, services can declare `deps` and `needs`. Use `deps` when a service requires a task to run first (for example building an image input), and `needs` when a service depends on another service. ```yaml services: api: image: node:alpine deps: [install] needs: [postgres] cmd: node index.js postgres: image: postgres:16-alpine ports: [5432] tasks: install: image: node:alpine cmds: - npm ci ``` ## Starting and stopping services [#starting-and-stopping-services] Services start and stop automatically based on the needs of the tasks you run. They can also be controlled directly with the CLI: * `hammerkit up` starts the services. * `hammerkit down` stops the services. This is useful to keep shared services such as a database running across multiple task runs. A service's container is recreated when it starts, so data written inside it is lost on the next start unless you attach a [volume](/docs/service/container#volumes). Declare a volume to persist a database's data — or to seed/back it up with [store / restore](/docs/cli/store-restore). # Kubernetes service (/docs/service/kubernetes) Kubernetes services give your tasks access to resources in Kubernetes clusters. This lets you share resources on a remote machine or run tasks against test, staging or production environments. To run your own tasks and services **on** a cluster instead of forwarding existing resources, see [running on Kubernetes](/docs/task/kubernetes). ## Cluster config lives in the environment [#cluster-config-lives-in-the-environment] The cluster connection — `context`, `kubeconfig` and the default `namespace` — is declared once in an [environment](/docs/task/kubernetes#declaring-an-environment), not on every service. A Kubernetes service then only needs the `selector` and the `ports` to forward, and is pointed at a cluster by selecting an environment with `--env`. ```yaml title=".hammerkit.yaml" environments: local: kubernetes: context: docker-desktop prod: kubernetes: context: gke_my-project_europe-west1_my-cluster kubeconfig: ./prod-kube.yaml namespace: databases services: postgres: ports: - 5432:5432 selector: type: deployment name: postgres ``` ```bash hammerkit my-task --env prod # forwards postgres from the prod cluster hammerkit my-task --env local # same service, local cluster ``` The same build file now targets dev/prod/local clusters without repeating the cluster wiring on each service. When an environment named `default` exists it is used without passing `--env`. Port-forwarding no longer requires the `kubectl` binary to be installed. Hammerkit forwards ports directly through the Kubernetes client. ## Service fields [#service-fields] | Field | Required | Description | | ------------ | -------- | ----------------------------------------------------------------------------------------------- | | `selector` | yes | Resource `type` (`deployment`, `service` or `pod`) and `name` to forward from. | | `ports` | yes | Port mappings (`hostPort:containerPort`) to forward to localhost. | | `namespace` | no | Overrides the environment namespace; defaults to the environment's, then `default`. | | `context` | no | Overrides the environment context. Rarely needed — prefer the environment. | | `kubeconfig` | no | Overrides the environment kubeconfig. Defaults to the environment's, then `$HOME/.kube/config`. | ## Overriding per service [#overriding-per-service] A service can override the inherited namespace (or, for a mixed-cluster setup, the context/kubeconfig) while still picking up the rest from the environment: ```yaml environments: prod: kubernetes: context: gke_my-project_europe-west1_my-cluster namespace: databases services: postgres: namespace: orders # override just the namespace ports: - 5432:5432 selector: type: deployment name: postgres ``` ## Cluster providers [#cluster-providers] Hammerkit talks to a cluster through your kubeconfig, so every provider works the same way — the only difference is the `context` name (and optionally a dedicated `kubeconfig` file). The context is whatever the provider's auth tooling writes into your kubeconfig. Put it in the environment once; services stay provider-agnostic. ### GKE (Google Kubernetes Engine) [#gke-google-kubernetes-engine] `gcloud container clusters get-credentials ` writes a context named `gke___`: ```yaml environments: prod: kubernetes: context: gke_my-project_europe-west1_my-cluster namespace: databases ``` ### EKS (Amazon) [#eks-amazon] `aws eks update-kubeconfig --name ` writes an ARN-style context: ```yaml environments: prod: kubernetes: context: arn:aws:eks:eu-central-1:123456789012:cluster/my-cluster namespace: databases ``` ### AKS (Azure) [#aks-azure] `az aks get-credentials --name ` writes a context named after the cluster. Point `kubeconfig` at a dedicated file when you don't want to use the default one (for example a kubeconfig provided by CI): ```yaml environments: prod: kubernetes: context: my-cluster kubeconfig: ./aks-config.yaml namespace: databases ``` With the cluster declared in the environment, the same service definition works across all of them: ```yaml services: postgres: ports: - 5432:5432 selector: type: deployment name: postgres ``` # Arguments (/docs/task/arguments) Narrowing a run to one test file, one test name or one flag doesn't need a task per variation. Whatever follows `--` on the command line is appended to the **last command** of the task you run, as `npm run test -- ...` does: ```yaml tasks: test: image: node:24-alpine deps: [install] src: [src, package.json] cmds: - npm run generate - npx vitest run ``` ```bash hammerkit run test -- src/login.spec.ts -t "login flow" ``` The last command runs as `npx vitest run 'src/login.spec.ts' '-t' 'login flow'`; `npm run generate` runs unchanged. The build file declares nothing for this, unless it wants to refuse them, see [Which tasks take arguments](#which-tasks-take-arguments). ## One argument stays one argument [#one-argument-stays-one-argument] Each argument is quoted for the shell the command runs in, so shell syntax inside it is text: ```bash hammerkit run test -- 'a; rm -rf /' ``` The task receives one argument, `a; rm -rf /`, and no second command runs. The quoting follows the task's [execution shell](/docs/task/container#execution-shell): `sh`, `bash`, `zsh`, `dash`, `ash`, `ksh` and `mksh` take single quotes, as does PowerShell, which runs local tasks on Windows. A task whose shell hammerkit can't quote for (`fish`, `cmd.exe`) refuses arguments rather than guess. Two runtimes read the command line once more, and hammerkit accounts for both: * On [Kubernetes](/docs/task/kubernetes) the kubelet expands `$(NAME)` and `$$` before the shell runs, so each `$` in an argument is passed as `$$` and arrives as typed. * On Windows, Windows PowerShell builds a new command line for the program it starts, and starts `.cmd` and `.bat` shims (`yarn.cmd`, most of `node_modules/.bin`) through `cmd.exe`. A local task there refuses an empty argument, one with `"`, `&`, `|`, `<`, `>`, `^`, `%`, `!` or a line break, and one with a space that ends in `\`, rather than let it split or run. To hand over a whole pipeline, make the last command a shell that takes it: ```yaml tasks: sh: image: node:24-alpine mounts: [.] args: true cmds: - sh -c ``` ```bash hammerkit run sh -- 'npm test 2>&1 | tail -20' ``` ## Which tasks take arguments [#which-tasks-take-arguments] Every task takes arguments unless it says otherwise. A task whose last command must run as written says `args: false`: ```yaml tasks: install: image: node:24-alpine src: [package.json, package-lock.json] generates: [node_modules] args: false cmds: - npm ci --ignore-scripts ``` ```bash hammerkit run install -- --foreground-scripts ``` The run fails before anything starts: `task install takes no arguments after -- (args: false)`. Without it, `--foreground-scripts` would land after `--ignore-scripts`, and an argument such as `-- --ignore-scripts=false` could undo what the command says. The same goes for a service's `cmd` with [`up`](#services). `args: true` says a task is meant to take them, as the shell task above. Locally it changes nothing, but a [server](/docs/remote-compute/runs#arguments) with `runs.args: declared` passes arguments only to the tasks that say it, so a task nobody wrote for arguments never gets them, and nobody has to remember `args: false` on every new task. A task that [extends](/docs/task/extending) another one takes its `args` unless it sets its own. [`ls --json`](/docs/cli/ls#json) shows `args` as the build file says it, `null` when it says nothing. ## Caching and dependencies [#caching-and-dependencies] A run with arguments is a one-off, so the task skips the [cache](/docs/task/caching): it always runs, stores no cache entry, and doesn't read its sources to compute a key. Its outputs replace the ones the task wrote before, so the next run without arguments isn't a cache hit either: it runs again, or restores its outputs from the cache. To cache a variant you run often, name a cache method on the command line: ```bash hammerkit run test --cache checksum -- -t a ``` The arguments are part of the cache key, so `-- -t a` then hits only its own entry. Running a task without arguments (or with an empty `--`) keeps the cache entries it had before. Dependencies and needed services never get the arguments. In the example, `install` runs as declared and keeps its own cache entry. ## Services [#services] [`up`](/docs/cli/up) appends the arguments to the `cmd` of the one service it starts: ```yaml services: api: image: node:24-alpine cmd: node server.js labels: { app: api } ``` ```bash hammerkit up --filter app=api -- --inspect ``` The service starts as `node server.js --inspect`. Without a [`shell`](/docs/service/container#shell) the `cmd` is an argument list, so the arguments are added to it as they are; with one, they are quoted for it like a task's. A service without `cmd`, and a Kubernetes service, take no arguments. ## Rules [#rules] * Arguments go to exactly one task or service. `run` with a filter that selects two tasks fails before anything runs, naming them; `run` counts tasks only, `up` counts tasks and services. * Only `run` and `up` take arguments; other commands refuse them. * Everything after the first `--` is an argument, including `--set` and `--file`, which are hammerkit's own options only before it. [`--set`](/docs/build-file/environment-variables) is unchanged and stays the way to fill a `$NAME` the build file references. * [`ls --json`](/docs/cli/ls#json) shows each task's last command and each service's `cmd`, so a script or an agent can tell where its arguments land. A version published to a server shows it as `command` in its plan. * A run on a [server](/docs/remote-compute/runs#arguments) takes them as `options.args`, for one task in a run where nothing reads a secret or a credential of the server, and under `runs.args: declared` only for a task with `args: true`. # Caching (/docs/task/caching) Every task should declare the source files it requires. Based on those files, hammerkit checks whether anything changed since the last run that requires the task's commands to run again. If all source files are unchanged, the task is skipped. A task can only be skipped if all of its dependencies can be skipped as well; otherwise the result could be inconsistent. When a dependency changes, every task depending on it runs again. ### What gets cached, and what invalidates it [#what-gets-cached-and-what-invalidates-it] The cached **result** is the task's `generates` output (plus the per-task state hammerkit keeps under `.hammerkit`). That is what gets reused — and what [`store` / `restore`](/docs/cli/store-restore) and remote [backends](/docs/build-file/caches) move between machines. A task's cache key has two parts: * the task's **definition**: `cmds` (with any [arguments](/docs/task/arguments) after `--`), `image` (or the host OS for a local task), the CPU architecture, `envs`, `mounts`, `shell`, `src` and `generates` paths, the working directory, which [secrets](/docs/task/secrets#caching) it reads and where from, the definitions of its **dependencies**, and the definitions of the **services** it `needs` (image, command, variables, ports, healthcheck, and what they depend on), and * its **state**: the declared **`src`** files (their sha256 content checksum by default, or their modification dates with `modify-date`; on a server a symbolic link by where it points) combined with the state of its **dependencies** and of the services it needs (their `src` and the state of the tasks they depend on), recursively. So a task is re-run when its definition changes, when its `src` changes, or when anything it depends on changes — its dependencies' sources *or* their definitions, and the services an `e2e` task tests against. Use [`hammerkit explain`](/docs/cli/explain) to see which of these caused a rebuild. Hammerkit remembers which run of a task produced the outputs in its folders. When the same task runs with other [`--set`](/docs/build-file/environment-variables) values or [arguments](/docs/task/arguments) in between, those outputs are the other run's, so the next run is not a local hit: it runs, or restores its own outputs from the cache backend. Changing a task's `description`, `labels` or `timeout` does not invalidate it. A task's `src` may point at what a dependency `generates` (an `e2e` task reading `dist`, say). Those files are not hashed again: they are represented by the dependency's definition and state. So the task has the same key on a clean checkout, where `dist` isn't built yet, as in a workspace where it is — which is what lets a clean CI checkout reuse what an agent pushed. A cache hit restores the task's outputs where they are expected: a container task's exported directories and file outputs on the host, the rest in its volumes. An archive restores only into the output it was stored for: an entry of it beside that path is skipped, and so is a hard link to a file beside it. On a server a symbolic link that points out of the output is skipped too. Deleting an output (`rm -rf dist`) makes the task restore it from a cache backend, or run again. The architecture is part of the key because outputs often contain native binaries (esbuild, cypress, native node modules), and Docker pulls images for the host's architecture. An arm64 laptop and an x64 CI runner therefore don't share cache entries. An `image` is identified by its name, not its content. A tag such as `node:24-alpine` moves when it is republished, and hammerkit keeps using results built with the old image. Pin images by digest (`node:24-alpine@sha256:…`) where that matters. All paths in the key are relative to the project root (the git root, or the directory of the main build file), so the same commit has the same keys in every checkout — on a laptop, a CI runner or an agent sandbox. That is what lets a [remote cache](/docs/build-file/caches) be shared between machines. A task **without `src`** of its own can't be proven up to date, so it runs every time — and so does every task that depends on it. Its dependencies' `src` doesn't count: it says nothing about the files the task's own commands read. Declare `src` on everything you want cached. An [aggregate task](/docs/task/dependencies#aggregate-tasks) is the exception: it runs no command, so it is up to date when all its dependencies are. The same applies when every `src` entry of a task matches no file (a typo, a moved directory). Each `src` entry that matches no file is reported as a warning. ### Define source files [#define-source-files] The following example declares `package.json` and `package-lock.json` as source files. Before each run, the checksums of those two files are compared with the checksums from the last successful run; if they are equal, the task is skipped. ```yaml title=".hammerkit.yaml" tasks: install: src: - package.json - package-lock.json cmds: - npm install ``` ### Define source folders [#define-source-folders] The next example declares the `src` folder as the task source. To check whether the task can be skipped, the entire folder is traversed recursively. ```yaml title=".hammerkit.yaml" tasks: build: src: - src cmds: - tsc -b ``` Keep in mind that the recursive traversal can be expensive on huge folders. Avoid using, for example, a `node_modules` folder as a source. ### Define glob sources [#define-glob-sources] This example defines a glob pattern `src/**/*.ts` as the task source. This is similar to a folder source, but filters, for example, by file extension: every `.ts` file under `src`, in any subdirectory. ```yaml title=".hammerkit.yaml" tasks: build: src: - src/**/*.ts cmds: - tsc -b ``` Globs are matched against paths relative to the task's directory, and may use environment variables (`$DIR/**/*.ts`). Several patterns are supported. Hammerkit matches globs with [minimatch](https://github.com/isaacs/minimatch) (the matcher behind node-glob); see its docs for all details. A quick summary: * `*` Matches 0 or more characters in a single path portion. * `?` Matches 1 character. * `[...]` Matches a range of characters, similar to a RegExp range. If the first character of the range is `!` or `^` then it matches any character not in the range. * `!(pattern|pattern|pattern)` Matches anything that does not match any of the patterns provided. * `?(pattern|pattern|pattern)` Matches zero or one occurrence of the patterns provided. * `+(pattern|pattern|pattern)` Matches one or more occurrences of the patterns provided. * `*(a|b|c)` Matches zero or more occurrences of the patterns provided. * `@(pattern|pat*|pat?erN)` Matches exactly one of the patterns provided. * `**` If a "globstar" is alone in a path portion, then it matches zero or more directories and subdirectories searching for matches. * `{a,b}` Matches either of the comma-separated alternatives. A glob starting with `**` walks the whole task directory, including `node_modules`. Start it at the directory that holds the sources (`src/**/*.ts`) to keep that cheap. ### checksum vs. modify-date [#checksum-vs-modify-date] `checksum` is the default method, both locally and in CI. It compares the **content** of the `src` files, so a result built on one machine is correctly reused on another — including a fresh CI runner. Prefer `checksum` for anything shared with CI. `modify-date` compares file **modification times**, which a `git clone` resets to "now" on every fresh checkout — so on a clean CI runner a `modify-date` task looks changed every time and never hits the cache. `modify-date` is only a local optimization for very large source folders where hashing is expensive; it is not portable across machines. ### Selecting a cache method [#selecting-a-cache-method] By default a task is skipped based on the content checksum of its source files. The `cache` field can change the method per task. The shorthand form takes one of `checksum`, `modify-date` or `none`. A task with `none` always runs, and its own run doesn't read its sources at all, so a `src: ['**']` over a large `node_modules` costs nothing. A task that depends on it still hashes them for its own key. In `--watch` they are read, to rerun the task on a change. ```yaml title=".hammerkit.yaml" tasks: build: cache: modify-date src: - src cmds: - tsc -b ``` ### Changing the default for all tasks [#changing-the-default-for-all-tasks] A task without a `cache` field uses the built-in `default` cache, and its method is the global default rather than something declared on the task. There are two ways to change that default for every such task. **Per run - the `--cache` flag.** Pass `--cache ` to set the method for all tasks that did not declare their own `cache`. The choices are `checksum`, `modify-date` and `none`. The default is `checksum` both locally and in CI, so a result cached on one machine is reused on the other. See [execute](/docs/cli/execute) for the full option reference. ```bash title="terminal" hammerkit build --cache checksum ``` **Persistently - redeclare `caches.default`.** To change the default cache *backend* for every task (for example to share results through a remote bucket), redeclare the built-in `default` cache in the build file. See [caches](/docs/build-file/caches#built-in-caches). An explicit `cache` on a task always wins. Both the shorthand (`cache: modify-date`) and the object form are left untouched by `--cache`, so only tasks that did not opt in follow the global default. ### Remote caches & backends [#remote-caches--backends] A task can reference a named [cache](/docs/build-file/caches) with a remote backend, so its result can be shared between machines and CI runs. Reference a cache by name and optionally override its method. ```yaml title=".hammerkit.yaml" caches: remote: method: checksum backend: type: s3 bucket: my-build-cache region: eu-central-1 tasks: build: cache: name: remote # use the "remote" cache declared above method: checksum # optional, overrides the cache method src: - src generates: - dist cmds: - tsc -b ``` When the cache has a remote backend, hammerkit pulls the result before the task runs if it is missing locally, and pushes the result after a successful run. The built-in backends are `local`, `s3` and `registry`; the `s3` backend also covers any S3-compatible store such as MinIO, Cloudflare R2 and Google Cloud Storage, and the `registry` backend any OCI container registry (GHCR, Docker Hub, ECR, …). See [caches](/docs/build-file/caches) for the full backend reference, the R2/GCS examples and the pull/push behavior. To keep the network out of the build, leave tasks on the machine-local `default` cache and move entries explicitly with [`cache pull` / `cache push`](/docs/cli/cache); see [agents, workspaces and CI](/docs/guides/agents-and-ci). ### Limitations: what the cache can't see [#limitations-what-the-cache-cant-see] Hammerkit assumes a task is a **function of its declared inputs**: the same definition, the same `src` contents and the same dependencies produce the same outputs. A cache hit replays the outputs of the first run with those inputs. When a task's result depends on anything else, the cache can serve a stale result — so these are the places to look when you set up a build. #### Inputs hammerkit doesn't track [#inputs-hammerkit-doesnt-track] | Situation | What can go wrong | What to do | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | **A task reads files it doesn't declare.** A local task can read anything on disk. A container task sees its `src`, its `mounts` and its dependencies' sources and outputs — and a glob makes the whole directory it starts in visible (`src/**/*.ts` mounts all of `src`, but only `.ts` files are hashed). | Changing such a file (a `.css` the build bundles, a config one level up) leaves the task cached. | Declare every file and folder a task reads. Use a folder (`src`) rather than a narrowing glob when the task reads more than the glob matches. | | **Inputs passed through `mounts`.** A mount is part of the key only by its path, never its contents. | A changed file behind a mount is not noticed. | Use `mounts` only for caches and tooling (`~/.npm`, a docker socket); put inputs in `src`. | | **Host tools of a local task.** A local task's key includes the operating system and CPU architecture, but not the versions of Node, Go, a compiler or anything else on the host's `PATH`. | Upgrading a tool on one machine reuses results built with the old one; two developers with different versions share entries. | Use container tasks for anything you cache or share. For a local task, add the tool's version file (`.nvmrc`, `.tool-versions`) to `src`. | | **Image tags that move.** An `image` is identified by its name, not its content. | `node:24-alpine` is republished; hammerkit keeps reusing results built with the old image. | Pin images by digest (`node:24-alpine@sha256:…`) where it matters, and bump the digest deliberately. | | **The network.** Installs without a lockfile, `apt-get update`, downloads of "latest", calls to live APIs. | The output depends on the day it was built, and a cache hit pins whatever the first run fetched. | Make the inputs explicit: lockfiles in `src`, pinned versions in commands and images. Give a task that must fetch fresh data `cache: none`. | | **Time, randomness and version stamps.** Build dates, random seeds, versions from `git describe`, `$GITHUB_SHA` or a build number. | Read through a declared env, the value changes on every commit and the task never hits. Read from `.git` or the clock, it is invisible and a stale stamp is served. | Keep stamping out of cached tasks: put it in a small final task with `cache: none` that reads the cached outputs. | #### Results hammerkit doesn't check [#results-hammerkit-doesnt-check] * **Outputs are checked for existence, not content.** Deleting an output (`rm -rf dist`) makes the task restore or rebuild it, but editing a file inside `dist` by hand is not noticed. Run `hammerkit clean` after manual changes. * **Dependency outputs are trusted.** A task reading what a dependency generates is keyed by the dependency's definition and sources, not by those files. A dependency that breaks one of the rules above passes the problem on. * **Non-deterministic tasks.** If the same inputs can produce different outputs (timestamps inside archives, test order, flaky tests), a cache hit freezes one of them. A flaky test that passed once stays "passed" until its inputs change. * **`modify-date` only sees timestamps.** Content that changes while the modification time is preserved (`cp -p`, `rsync -t`, extracting an archive) is not noticed. Keep the default `checksum` for anything shared. * **Trust.** A shared cache hit means "someone with write access ran this". Decide who may push — see [who may write to the cache](/docs/guides/agents-and-ci#who-may-write-to-the-cache). #### Changes that rebuild more than needed [#changes-that-rebuild-more-than-needed] These never serve a stale result, but cost a rebuild: * **Env values are part of the key**, including tokens: rotating `NPM_TOKEN` rebuilds every task that declares it. Declare credentials only on the tasks that use them. * **Build-file `envs` apply to every task in the file**, so changing one rebuilds them all. Put a value on the tasks that use it when only those should rebuild. * **The CPU architecture is part of the key.** An arm64 laptop and an x64 CI runner don't share entries. #### Checking your build [#checking-your-build] * Run the build twice: the second run should report every task as cached. A task that rebuilds without a change reads something that differs between runs — [`hammerkit explain`](/docs/cli/explain) names it. * Change one source file: only the tasks reading it, and the tasks depending on them, should rebuild (`hammerkit run --dry-run` shows the plan without running). * Audit regularly, for example nightly: build once with the cache and once with `--cache none` on a clean checkout, and compare test results or output checksums. A difference points at an input a task doesn't declare. # Container (/docs/task/container) Every task can run inside a container. All you need is to set an `image` property on your task. ```yaml title=".hammerkit.yaml" tasks: install: image: node:24-alpine cmds: - npm install ``` This example runs `npm install` inside a container with the image `node:24-alpine`. However, the container has no access to your local files, so nothing gets installed. To give it access to your project files, use sources, generates and mounts. ### Adding source files/folders [#adding-source-filesfolders] All source files and folders are mounted when the container starts and can be accessed inside it. ```yaml title=".hammerkit.yaml" tasks: install: image: node:24-alpine src: - package.json - package-lock.json cmds: - npm install ``` ### Adding output files/folders [#adding-output-filesfolders] The installed node_modules are written to the container's file system. If those files and folders are needed after the task has finished, declare them under `generates`. ```yaml tasks: install: image: node:24-alpine src: - package.json - package-lock.json generates: - node_modules cmds: - npm install ``` ### Adding mounts [#adding-mounts] For other files and folders that are neither sources nor generates, use mounts. They can be relative, absolute or relative to your home directory. ```yaml title=".hammerkit.yaml" tasks: install: image: node:24-alpine src: - package.json - package-lock.json generates: - node_modules mounts: - relative/path - /some/absolute/path - .npm:/root/.npm cmds: - npm install ``` A mount is written `host[:container]`: * `path` — a single path mounts the host path to the **same** path inside the container. * `host:container` — mounts the host path on the left to the container path on the right. * `host:container:ro` — the same, read-only for the container (`:rw` is the default). Any other third part is an error. Host paths can be relative (resolved against the build file's directory), absolute, or in your home directory with `~/` (`~/.npm:/root/.npm`). `$HOME/` means the home directory too, unless the task declares a `HOME` in its `envs`, which then takes its place. A mount in the home directory names its container path (`~/.npm` alone is an error). Relative container paths resolve against the task's working directory. `$PWD/` at the start of a host path is also your **home directory**, not the project, for compatibility with older build files. [`validate`](/docs/cli/validate) warns about it. Write `~/` for the home directory, or a relative path for the project. On a [server](/docs/remote-compute), a host path outside the published files is refused, and a home-directory path is named as such in the error. A host path that does not exist yet is created before the container starts: an empty file when its name has an extension (`config.json`), else a directory. When hammerkit can't create a directory, for example because it runs in a container of its own whose file system is not the Docker host's, Docker creates it on its host instead. A missing file it can't create is an error, since Docker would put a directory there. ### Working directory [#working-directory] A container task runs with its working directory set to the build file's directory (mirrored inside the container), and your sources are mounted there. So relative paths in `cmds` (`node_modules/.bin/tsc`, `./script.sh`) behave just like they would when running on the host. The sources and generates of all dependencies are mounted into the container automatically. This avoids long lists of mounts and keeps tasks consistent. ### Execution shell [#execution-shell] By default, each command runs in a `sh` shell. Override it with a `shell` property on the task. ```yaml title=".hammerkit.yaml" tasks: shell_example: image: ubuntu shell: bash cmds: - echo $RANDOM ``` ### File permission [#file-permission] On Linux hosts a container task runs as your `uid:gid`, so files it generates on mounted paths come back owned by you rather than by `root`. That user has no home directory in the image, so hammerkit sets `HOME=/tmp` for the task (unless its `envs` set `HOME`): tools like npm, Go, Maven and the .NET SDK keep their caches under it. To avoid permission conflicts on the mount points themselves, hammerkit `chown`s the working directory, the sources, the generated directories and the volumes to your `uid:gid` before the task runs. A `mounts` entry keeps its owner: it is a path of your machine, such as the docker socket, whose owner is not hammerkit's to change. A mounted path the task writes to has to be writable for you already. The blast radius is small and deliberate: the `chown` is **not** recursive, so it only re-owns those top-level directories — files already inside an image (for example pre-baked, root-owned content) are left untouched. On macOS and Windows the Docker Desktop VM handles the uid mapping, so this step is skipped entirely. ### Capabilities [#capabilities] `capAdd` adds Linux capabilities to the task's container, on top of the runtime's default set. A tool that mounts file systems, for example, needs `SYS_ADMIN`: ```yaml title=".hammerkit.yaml" tasks: mounts: image: alpine:3.22 capAdd: [SYS_ADMIN] cmds: - mount -t tmpfs tmpfs /mnt && touch /mnt/file ``` To build a Dockerfile, use [`build`](/docs/task/image-builds) instead: hammerkit adds what BuildKit needs, wherever the task runs. A capability is named without the `CAP_` prefix, as in `capabilities(7)`: `SYS_ADMIN`, `NET_ADMIN`. `ALL` is not one, a task lists what it needs. On Docker they are `--cap-add`, on Kubernetes the container's `securityContext.capabilities.add`. A capability holds for root only, so a task with `capAdd` runs its commands as the user of its image, not as your `uid:gid` (see File permission above): files it writes into the project are that user's. Capabilities are not part of the task's cache key. On a host that uses AppArmor, such as Ubuntu, Docker's default profile still refuses mounts whatever the capabilities, so a build that mounts needs a sandboxed runtime there. A [server](/docs/remote-compute/backend#capabilities) adds a capability only where its operator allows it, and then only in such a sandbox: see [Building container images](/docs/guides/container-images). # Dependencies (/docs/task/dependencies) Every task can have a list of dependencies. They run before the task, and if any of them fails, all pending tasks are aborted. Dependencies can be chained as deep as needed, as long as there is no cycle. ```yaml title=".hammerkit.yaml" tasks: install: cmds: - npm install build: deps: [install] cmds: - tsc -b publish: deps: [build] cmds: - npm publish ``` ## Dependencies run in parallel [#dependencies-run-in-parallel] Dependencies that don't depend on each other run **concurrently**, up to the `--concurrency` worker count (default `4`). In the example above `install` must finish before `build`, but if two branches of the graph are independent they execute at the same time. A dependency that fails stops the whole run; see [parallelism and exit codes](/docs/cli#parallelism) in the CLI reference. ## Which ready task runs first [#which-ready-task-runs-first] When more tasks are ready than there are free workers (or, with [resource requests](/docs/task#resources), than there is room), hammerkit starts the one on the **longest remaining chain** first. A task's chain is its own expected duration plus that of the longest line of tasks that wait for it, through the services that need it too. The tasks at the start of a long chain, such as the `install` and `build` before a slow `e2e`, go before a short task nothing waits for, such as `lint`, which still fits in later. The expected duration of a task is the median of the last 5 times it ran on this machine, kept in `durations/` of hammerkit's directory (`~/.hammerkit` on Linux, `~/Library/Application Support/hammerkit` on macOS, `HAMMERKIT_DIR` when set). On a [server](/docs/remote-compute/runs), it is the median of its last 9 successful runs in the same repository. A task that never ran counts as the median of the others, and with no history at all every task counts the same, so the chain is counted in tasks. A task that only groups others, and one already taken from the cache, count nothing. Tasks that are equally urgent start in the order they became ready. ## Dependencies of cached tasks [#dependencies-of-cached-tasks] A dependency only runs when a task that needs it has to run. If every task depending on it is a cache hit, it is **skipped**: when `e2e` is cached, `hammerkit run e2e` doesn't rebuild the app `e2e` was tested against. Skipped tasks show up as `skipped` in the [build summary](/docs/cli/execute#build-summary). Tasks you ask for — by name, or selected with a [label filter](/docs/labels) — always run (or restore from the cache); only tasks pulled in purely as dependencies are skipped. A service's dependency runs once the service starts, so when every task needing the service is a cache hit, neither the service nor its dependencies run. Under `up` and in watch mode everything runs. A task that has to run restores its whole dependency chain, not only its direct dependencies. With `install` ← `build` ← `test`, a changed `test` runs against a cached `build`, and `install` is still restored from the cache (shown as `cached`) because `test` may need its outputs, for example `node_modules`, at run time. A skipped dependency leaves no outputs in the workspace. If you need them — for example an exported `dist` — request that task too, or pass `--no-skip-deps` to run the whole dependency graph as in earlier versions. ## Aggregate tasks [#aggregate-tasks] A task with `deps` and nothing else to do groups other tasks under one name: no `image`, no `cmds`, no `src`, no `generates`, no `secrets` and no `needs`. ```yaml title=".hammerkit.yaml" tasks: build:all: deps: [build:api, build:web] test:unit: image: node:24-alpine deps: [build:all] src: [test] cmds: - npm test ``` An aggregate runs nothing, so it needs no runtime and no image, and writes nothing to its directory: * a task that depends on it gets what its dependencies generate (`node_modules`, `dist`), as if it depended on them directly; * it is up to date when all its dependencies are, so a task that depends on it can be a cache hit, and it has a [check key](/docs/remote-compute/checks#check-keys) when they do. Asked for by name, it still checks its dependencies and restores their outputs, and is reported `cached` when all of them were; * a [server](/docs/remote-compute/backend#runtimes) runs it whatever `runtimes` allows, and `--network` does not need to limit it. A task with `deps` and its own `cmds` or `src` but no `image` is a local task as before. Referenced and included tasks are addressed with the `prefix:task` syntax in `deps`, for example `deps: [npm:install]`. See [references](/docs/build-file/references) and [includes](/docs/build-file/includes). Make sure to set up correct [caching](/docs/task/caching) to speed up your execution and prevent unnecessary work. # Extending (/docs/task/extending) Extending tasks avoids a lot of duplicated task definitions in bigger projects and monorepos. With the `extend` property, a task uses another task's configuration as a template. The following sections and examples use the two tasks in a `build.tsc.yaml` file, shown below. The [catalog](/docs/catalog/tsc) has a maintained `tsc` entry that you can include instead of copying. ```yaml title="build.tsc.yaml" tasks: install: src: - package.json - package-lock.json generates: - node_modules cmds: - npm ci build: deps: [ install ] src: - tsconfig.json - src cmds: - node_modules/.bin/tsc generates: - dist ``` ### Extend a task [#extend-a-task] Extending `tsc:build` uses all of its properties as a base. If nothing else is defined, the result is an exact copy that runs in the working directory of the current build file. ```yaml title=".hammerkit.yaml" tasks: build: extend: tsc:build includes: tsc: ./build.tsc.yaml ``` ### Override properties in an extend [#override-properties-in-an-extend] An extending task can override any property, whether the base defines it or not. Single values such as `image`, `shell`, `cache` and `timeout` are replaced by the extending task's own value. Lists and maps are **added to** the base's instead: | Property | With `extend` | | ------------------------------------ | --------------------------------------------------------- | | `image`, `shell`, `cache`, `timeout` | The task's own value replaces the base's. | | `cmds`, `src`, `generates`, `mounts` | The base's entries come first, the task's own follow. | | `deps`, `needs` | Both are used. | | `envs`, `labels` | Merged. The task's own value wins for a name both define. | The `envs` of the base also include the top-level `envs` of the build file it comes from, so a base that reads `$NODE_VERSION` keeps its default when you extend it. ```yaml title=".hammerkit.yaml" tasks: build: extend: tsc:build src: - ../../tsconfig.base.json includes: tsc: ./build.tsc.yaml ``` Here `build` runs the base's commands against the base's sources plus `../../tsconfig.base.json`. ### Container properties without an image [#container-properties-without-an-image] A task that extends a container task takes the base's image, so it may set `mounts`, `volumeSize` and `capAdd` without an `image` of its own: ```yaml title=".hammerkit.yaml" tasks: build: extend: base:node mounts: - .git cmds: - git log includes: base: ./build.base.yaml ``` When neither the task nor its base has an image, they are an error: `task build sets mounts, but neither it nor its base has an image`. A base's `volumeSize`, `capAdd`, `resources` and `secrets` are not passed on; set them on the extending task when it needs them. ### Reset inherited values [#reset-inherited-values] To take nothing from the base for a property, name it in `reset`. The task's own value for that property still applies. In this example the dependencies are cleared, so the task runs without depending on `install`: ```yaml title=".hammerkit.yaml" tasks: build: extend: tsc:build reset: [deps] includes: tsc: ./build.tsc.yaml ``` `reset` accepts `deps`, `needs`, `src`, `generates`, `cmds`, `mounts`, `envs` and `labels`. It is an error without `extend`. An empty `deps: []` does not clear the base's dependencies, because the lists are added together. ### Naming the base task [#naming-the-base-task] `extend` uses the same `:` addressing as [dependencies](/docs/task/dependencies): the part before the colon is the [include](/docs/build-file/includes) (or [reference](/docs/build-file/references)) prefix, and the part after it is the task name in that file. `extend: tsc:build` means "the `build` task from the file included as `tsc`". To extend a task in the *same* file, use its bare name (`extend: build-base`). # Image builds (/docs/task/image-builds) A task with `build` builds a Dockerfile, like `docker build`. Hammerkit runs the build with [BuildKit](https://github.com/moby/buildkit), the builder behind `docker build`, in a container of its own. The same task builds on your dev machine, in CI and on a [hammerkit server](/docs/remote-compute), where no task gets a Docker daemon. ```yaml title=".hammerkit.yaml" tasks: image: description: build the app image and push it build: context: app # docker build app tags: ghcr.io/acme/app:latest # --tag push: true # --push ``` That is the whole task: hammerkit writes its image and its command. A task with `build` has no `image`, `cmds` or `shell` of its own. ## The fields [#the-fields] | Field | Default | What it does | | ------------ | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `context` | `.` | The folder sent to the build, relative to the build file. | | `dockerfile` | `Dockerfile` in the context | The Dockerfile, relative to the build file. It may be outside the context. | | `tags` | none | The names of the image with their tags, one or a list, as `--tag`. Without tags the build only checks that the Dockerfile builds. | | `push` | `false` | Pushes the image to the registry of each tag. Needs `tags`. | | `cache` | none | An image in a registry, such as `ghcr.io/acme/app:buildcache`, that the build reads its layers from first and writes them to after. A build in a new place (a CI runner, a pod) then starts from the layers of the last one. | | `digest` | none | A file the task writes `@sha256:…` to: the image it pushed, by its digest. Needs `tags`. | | `login` | none | Sign-ins to registries, see [Signing in](#signing-in-to-a-registry). | | `insecure` | `false` | The registries of the tags and the cache speak plain HTTP, as a `registry:2` on a network of your own does. | | `platforms` | the machine's | The platforms of the image, one or a list such as `linux/amd64` and `linux/arm64`, as `docker build --platform`. See [Several platforms](#several-platforms). | `$NAME` and `${NAME}` take the value of an [environment variable](/docs/build-file/environment-variables) in every field but `login.registry` and `platforms`, as in `image` of a container task: ```yaml envs: REGISTRY: europe-west4-docker.pkg.dev/acme/apps tasks: api: build: dockerfile: api/Dockerfile tags: $REGISTRY/api:main cache: $REGISTRY/api:buildcache ``` The task takes the other fields of every task: `deps`, `needs`, `src`, `generates`, `envs`, `secrets`, `account`, `network`, `resources`, `timeout`, `cache`, `labels`, `runner`, `publish`, and `volumeSize` of a container task. Its `envs` and `secrets` can't set `PATH`, `LD_*`, `BUILDKIT*`, `BUILDCTL*` or `HAMMERKIT_REGISTRY_*`: hammerkit runs BuildKit's programs, not ones a variable names. It doesn't take `extend`, and no task can extend it. It takes no [arguments](/docs/task/arguments) after `--`. ## Sources and caching [#sources-and-caching] Like `docker build`, the task reads its context. When the task has no `src`, its `src` is the context, and the Dockerfile where that is outside of it. A context of `.` is the whole folder, so any change in it builds the image again. List `src` to name what the Dockerfile copies instead, and the task builds again only when one of them changes: ```yaml tasks: api: build: dockerfile: api/Dockerfile # the context is ., the repository tags: ghcr.io/acme/api:main push: true src: # what api/Dockerfile copies - api/Dockerfile - .dockerignore - package.json - package-lock.json - api/src ``` With `src`, only those paths are in the task's container, so list the Dockerfile and `.dockerignore` too. A `COPY` of a path that isn't in `src` fails the build with `not found`. A context, or a folder in it, that a task in `deps` generates is read from that task's output: the build gets what the dependency made there, such as a `dist` it compiled or a Dockerfile it wrote, as any task that depends on it does. ```yaml tasks: dist: image: node:24-alpine src: [src, package.json, package-lock.json] generates: [dist] cmds: [npm ci, npm run build] image: deps: [dist] build: tags: ghcr.io/acme/app:main # the Dockerfile copies dist/ push: true src: [Dockerfile, .dockerignore] ``` A task whose sources didn't change is a [cache](/docs/task/caching) hit and doesn't run: the image of those sources is in the registry already. Set `cache: none` on the task to build on every run. ## Deploying the image by its digest [#deploying-the-image-by-its-digest] `digest` writes the name of the image with the digest the registry has for it, for a task that deploys exactly that image. The file is an output of the build, so a task that depends on it reads it, in its container too, and a cache hit brings it back: ```yaml tasks: image: build: tags: europe-west4-docker.pkg.dev/acme/apps/api:main push: true digest: .images/api # europe-west4-docker.pkg.dev/acme/apps/api@sha256:… deploy: image: gcr.io/google.com/cloudsdktool/google-cloud-cli:slim deps: [image] cache: none cmds: - gcloud run services update api --region europe-west4 --image "$(cat .images/api)" ``` The digest names the image even after `main` moves on to a newer one. Without `push` it is the digest of the image BuildKit built, which no registry has. A build with `cache: none` pushes a new image on every run, and a new image has a new digest. A task that reads the file of such a task therefore always runs too, as if its cache were off: a cache hit would bring back what it made from an older digest. ## Signing in to a registry [#signing-in-to-a-registry] A push, and a `cache` in a private registry, need a sign-in. Each entry of `login` names the registry, the user and the password, which is a [secret](/docs/task/secrets): `from` and `account` as there, never the password itself. ```yaml tasks: image: build: tags: ghcr.io/acme/app:latest push: true login: - registry: ghcr.io user: acme-ci password: { from: 'env:GITHUB_TOKEN' } # the workflow's, with packages: write ``` The password is a variable of the task's container, masked in its logs, and the build reads it from a Docker config the task writes for itself: neither your own `docker login` nor the one of the machine is used. What the user is depends on the registry: | Registry | `registry` | `user` | `password` | | ------------------------ | ------------------------------------------ | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | GitHub (GHCR) | `ghcr.io` | any name | the `GITHUB_TOKEN` of a workflow with `packages: write`, or an installation token of a GitHub App, never a personal token | | Google Artifact Registry | `-docker.pkg.dev` | `oauth2accesstoken` | an access token of a service account (`gcloud auth print-access-token`) | | Amazon ECR | `.dkr.ecr..amazonaws.com` | `AWS` | `aws ecr get-login-password` | | Azure Container Registry | `.azurecr.io` | `00000000-0000-0000-0000-000000000000` | `az acr login --name --expose-token --query accessToken -o tsv` | | Docker Hub | `docker.io` | your organization's name | an organization access token, never one of your own Docker ID | The tokens of Google, AWS and Azure expire within hours: print them with a [secret provider](/docs/task/secret-providers), read as a service account, so no key is in a file. On a server that is the [account](/docs/remote-compute/accounts) of the operator, as in [Building container images](/docs/guides/container-images#3-an-account-that-may-push). ## The network [#the-network] A build reaches what its task reaches. With a [`network`](/docs/task/network) allow list the build goes through the task's egress proxy: hammerkit hands the proxy to the `RUN` steps of the Dockerfile too, as the `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` build arguments. These don't change the image or its layers. List what the build reaches: the registry it pushes to, the one of each `FROM`, and what the `RUN` steps download. ```yaml tasks: image: build: tags: ghcr.io/acme/app:latest push: true network: allow: - ghcr.io - pkg-containers.githubusercontent.com # where ghcr.io keeps the layers - registry-1.docker.io # FROM node:24-alpine - auth.docker.io - production.cloudflare.docker.com # Docker Hub's layers, from its CDNs or S3 - production.cloudfront.docker.com - docker-images-prod.s3.dualstack.us-east-1.amazonaws.com - dl-cdn.alpinelinux.org # RUN apk add - registry.npmjs.org # RUN npm ci ``` A registry the task [needs](/docs/task/needs) as a service, such as a `registry:2`, it reaches by the service's name, as any task does. ## Several platforms [#several-platforms] With `platforms`, one build makes an image for each platform and pushes them under the same tags, as a list that `docker pull` picks the platform of the machine from: ```yaml tasks: image: build: tags: ghcr.io/acme/app:main push: true platforms: - linux/amd64 - linux/arm64 ``` A `RUN` step for a platform other than the machine's runs in the QEMU emulator that comes with BuildKit's image, so neither your dev machine nor a server needs an emulator of its own. An emulated `RUN` step is several times slower than a native one. Where the steps compile, let the compiler of the machine's platform build for the other one instead: `FROM --platform=$BUILDPLATFORM` for the build stage, and `$TARGETOS` and `$TARGETARCH` for the compiler, as in `GOOS=$TARGETOS GOARCH=$TARGETARCH go build`. Only the stage the image ends up with then runs for each platform. `digest` gets the digest of the list, which a deployment pulls the right platform from. `cache` keeps the layers of every platform. ## Where it runs [#where-it-runs] BuildKit mounts the layers it builds, which a container can only do with the capability `SYS_ADMIN`. * **On a dev machine and in CI**, hammerkit runs BuildKit's container privileged, as `docker buildx` runs its own BuildKit. That has the rights of the Docker daemon, which you have already. Its layers stay in a volume of the task, so the next build of the task starts from them; `hammerkit clean` removes it. The image is pushed, not loaded into your Docker: build with `tags` and `push`, then `docker pull` it to run it. * **On a server with a Kubernetes backend**, the task adds `SYS_ADMIN`, which the server grants only in a sandboxed runtime such as gVisor, where the capability reaches the sandbox's kernel and not the machine's. Its operator allows it once in [`backend.capabilities`](/docs/remote-compute/backend#capabilities). The layers are in memory: give the task `resources` with enough `memory` for them, and `cache` so a build starts from the layers of the last one. [Building container images](/docs/guides/container-images) sets a server up for it. A server on a docker host (`backend: { type: host }`) doesn't build images yet: it refuses a task with `build` at publish, see [the guide](/docs/guides/container-images#on-a-docker-host). * **In a [Kubernetes environment](/docs/task/kubernetes) of your build file**, without a server, the pod of the task gets `SYS_ADMIN` and nothing more, which BuildKit's `RUN` steps don't run with outside a sandbox. Build images there on a server, or with Docker. The BuildKit image is pinned (`moby/buildkit:v0.25.1`), so a build file builds with the same BuildKit everywhere. ## When it fails [#when-it-fails] | What you see | What it means | | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `push needs tags` or `digest needs tags` | `push` and `digest` need the names of the image. | | `build tag "…" is no image name` | A tag, after the variables are in it, has a space, a comma or a quote. | | `PATH, … can't be set on a task with build` | `envs` and `secrets` of a build task don't name `PATH`, `LD_*`, `BUILDKIT*`, `BUILDCTL*` or `HAMMERKIT_REGISTRY_*`, which would change the programs BuildKit runs or the passwords of its logins. | | `failed to compute cache key: … not found` | The Dockerfile copies a path that isn't in the task's `src`. | | `401 Unauthorized` or `403 Forbidden` on the push | The `login` is missing, names another registry than the tag, or its password can't push there. | | `RUN` steps can't download (`temporary error` from `apk`, `ECONNREFUSED` from `npm`) | The host isn't in the task's `network` (and, on a server, its `runs.secretNetwork`). | | `builds an image, which a server builds only with a kubernetes backend for now` | The server runs tasks on a docker host, which doesn't build images yet. Build on a server on Kubernetes, or in CI. | | `adds the capability SYS_ADMIN, and this server allows none` | The server has no `backend.capabilities`, see [the guide](/docs/guides/container-images#2-let-the-server-add-sys_admin-there). | # Task (/docs/task) Tasks are the pieces of work needed to build or develop your project. A minimal task contains just a list of commands. ```yaml tasks: example: cmds: - echo "minimal example" ``` **Local vs. container — the one rule to remember.** If a task declares an `image`, its commands run **inside that container**. If it has no `image`, they run **directly on your host** using the tools installed there. Everything else (sources, outputs, caching, dependencies) works the same in both cases. Containers are the recommended default: they remove the need to install build tools on every machine and make a build behave identically on your laptop and in CI. See [run a task in a container](/docs/task/container). ## Anatomy of a task [#anatomy-of-a-task] A task is built from a small, fixed set of fields: | Field | Purpose | | ----------- | --------------------------------------------------------------------------------------------------------- | | `cmds` | The commands to run, in order. [Arguments](/docs/task/arguments) after `--` are appended to the last one. | | `image` | Run the commands inside this container image. Omit to run on the host. | | `src` | Input files/folders. Used for [caching](/docs/task/caching) — unchanged sources let the task be skipped. | | `generates` | Output files/folders the task produces. This is what gets cached, stored and restored. | | `deps` | Other tasks that must run first (see [dependencies](/docs/task/dependencies)). | | `needs` | Services that must be running first (see [needs](/docs/task/needs)). | | `mounts` | Extra paths to make available to a container task (see [container](/docs/task/container)). | | `envs` | [Environment variables](/docs/build-file/environment-variables) for the commands. | | `labels` | Group/filter tasks (see [labels](/docs/labels)). | | `cache` | The [caching](/docs/task/caching) method/backend for this task. | | `timeout` | Maximum execution time, e.g. `10m` (see [timeouts](#timeouts)). | | `resources` | CPU and memory limits of the container (see [resources](#resources)). | | `network` | What the container reaches besides its needs (see [network](/docs/task/network)). | | `runner` | The server the task has to run on (see [runner](#runner)). | ## Source files (`src`) [#source-files-src] Tasks that depend on input files should declare them under `src`. Hammerkit detects whether the `src` files have changed compared to previous runs and skips execution if they are unchanged. ```yaml tasks: build: description: "run typescript build" src: - tsconfig.json - src cmds: - tsc ``` ## Generated files (`generates`) [#generated-files-generates] Tasks that produce output files should declare them under `generates`. Hammerkit archives generated files, so build outputs can be cached, stored and restored. ```yaml tasks: build: description: "run typescript build" src: - tsconfig.json - src generates: - dist cmds: - tsc ``` ### Exporting generated files [#exporting-generated-files] By default generated files of a container task stay inside the container volume. Mark a generate with `export: true` to copy it back into your workspace after the task ran. This is useful when another tool outside hammerkit needs the produced files, for example a build artifact you want to inspect or publish. ```yaml tasks: build: image: node:alpine generates: - path: dist export: true cmds: - tsc -b ``` An exported output is copied back only when the task succeeds. Test reports, screenshots and videos are most useful when a test fails: mark those `export: always` and hammerkit copies them to the workspace even when the task exits non-zero. A failed task is never cached, so nothing else changes. ```yaml tasks: e2e: image: cypress/included:13.15.0 generates: - path: cypress/videos export: always cmds: - cypress run ``` ### Outputs start empty [#outputs-start-empty] When a task runs, its outputs start empty: hammerkit deletes what an earlier run left in them before the task's commands run. A file a deleted source used to produce, or a partial result of a failed run, would otherwise end up in the next result and in the cache. An output that is also a source of the task, or contains one, is left alone: a `package-lock.json` that `npm install` both reads and writes keeps its content. Set `resetOnChange: false` on an output to keep its content between runs, for a tool that builds incrementally on top of its previous output: ```yaml tasks: build: image: rust:1.90-slim src: [Cargo.toml, Cargo.lock, src] generates: - path: target resetOnChange: false # cargo reuses what it compiled before cmds: - cargo build --release ``` Whatever such an output keeps becomes part of the next result, so prefer a [mount](/docs/task/container#adding-mounts) for a tool's cache: a mount is never part of a task's result. ## Timeouts [#timeouts] A task that hangs — a deadlocked test, a stuck network call — would otherwise block the run until your CI provider kills the job. Set a `timeout` and hammerkit aborts the task once it runs longer, fails it with `timed out after `, and stops the run like any other failure. ```yaml tasks: e2e: image: cypress/included:13.15.0 timeout: 15m cmds: - cypress run ``` Durations are written as a number plus a unit: `ms`, `s`, `m`, `h` or `d`, and can be combined (`1h30m`). A timed-out task is cleaned up the same way as a cancelled one — its container or Kubernetes job is removed — and never writes a cache entry. The timeout does not affect the task's cache key. To give every task a default, pass `--timeout` to [`run`](/docs/cli/execute) or [`up`](/docs/cli/up); a task's own `timeout` takes precedence. ```bash hammerkit run --timeout 30m ``` ### Capping every task [#capping-every-task] `--max-timeout` limits every task, also one with a longer `timeout` of its own: each task runs at most the shorter of the two, and a task without a timeout (and without `--timeout`) runs at most `--max-timeout`. It never lets a task run longer than its own `timeout`. ```bash hammerkit run --max-timeout 2m ``` On a server it is the run option `maxTimeout`, also `remote run --max-timeout`. The [agent clients](/docs/remote-compute/agents) send the timeout an agent gives a command this way, so a shell task with `timeout: 10m` stops a command after the 30 seconds the agent asked for. The time counts from when a task starts, its container or pod start included, not while the run waits in the queue or for the task's dependencies. On Kubernetes the time a task's pod waits for room under the quota of its namespace does not count either: the timeout stops while the quota refuses the pod and goes on once it is created. The waits of a task add up, and may last as long as its timeout in all: a task with `timeout: 30m` waits up to 30 minutes for room, then runs up to 30 minutes. Once it waited that long it fails with `gave up waiting for room under the quota of the namespace after 30m, its timeout`, not with `timed out`. The quota's own rule applies besides: after 10 minutes a pod waits on only while a pod that will end holds the room (see [the backend's namespaces](/docs/remote-compute/backend#namespaces)). Only the wait of the task's own pod is left out: the pods that copy its files into and out of its claim wait up to 10 minutes for room, and that wait counts against the timeout. ## Resources [#resources] A heavy build step can take every core and all the memory of the machine it runs on. Declare `resources` and hammerkit limits the task's container to them, and plans the run by them. ```yaml tasks: build: image: node:22 resources: cpus: 2 memory: 4Gi cmds: - npm run build ``` Both fields are optional and use Kubernetes notation: `cpus` as cores (`2`, `0.5`) or millicores (`500m`), `memory` with a `Ki`/`Mi`/`Gi`/`Ti` suffix (at least `6Mi`). `cpus` and `memory` set the limit and request the same. To set them apart, use `requests` and `limits`; a request must not exceed its limit, and a request without a limit is not capped: ```yaml tasks: build: image: node:22 resources: requests: { cpus: 500m, memory: 1Gi } limits: { cpus: 2, memory: 4Gi } cmds: - npm run build ``` On Docker the limits become the container's CPU and memory limits, and a memory request below its limit a soft memory reservation. On Kubernetes the job container gets the same `requests` and `limits`. A task that exceeds its memory limit is killed by the runtime and fails. `cpus` above what the Docker host has runs the container without a CPU limit, with a warning, as such a limit would limit nothing. On Kubernetes a request beyond what any node offers leaves the pod Pending. A local task can declare `resources` too, but nothing limits it; [`validate`](/docs/cli/validate) and `run --dry-run` say so. Resources are not part of the task's cache key: changing them does not rebuild the task. ### Planning the run [#planning-the-run] The requests also plan the run on the machine hammerkit runs on, for local tasks and for containers on Docker (on Kubernetes the cluster places pods by their requests): * **Before anything starts**, hammerkit checks that every task fits, together with the services it needs. A task whose requests plus those of its services exceed the host's CPUs or memory fails the run up front, naming the task, instead of hanging or being killed halfway. With `up`, all services must fit together. `--skip-resource-check` starts anyway. * **While it runs**, a task starts only when its requests fit next to everything running, the services included. When every task and service on the host requests both `cpus` and `memory`, the run is not capped at a worker count (`--concurrency`, default 4): as many tasks run in parallel as the requests allow. Otherwise `--concurrency` still caps the run, and tasks with requests also wait for room. A task that alone exceeds the host runs once nothing else does. The capacity is the Docker host's when containers run there, else this machine's. `HAMMERKIT_CPUS` and `HAMMERKIT_MEMORY` (e.g. `8`, `16Gi`) set it, for a CI runner that shares its machine, for example. ## Runner [#runner] Experimental. A task can name its server already, but running on one arrives in a follow-up release. Until then a run that resolves to a server stops with an error instead of running locally. `runner` names a server registered with [`hammerkit remote add`](/docs/cli/remote), for a task that has to run somewhere else, such as a Mac mini for an Xcode build. Set it on a task or as a default for the whole build file; `extend` inherits it. ```yaml runner: linux-pool # default of the tasks of this file tasks: ios: runner: mac-mini cmds: - xcodebuild -scheme app ``` The build file holds a name only. The URL and the OAuth issuer live in the host config, so a project can't send your source code to a server you never registered. Where a run happens, highest first: 1. `--on` / `--local` (follow-up release) 2. the `HAMMERKIT_ON` environment variable (`local`, `auto` or a server name) 3. the `runner` of the tasks in the run 4. `run.on` of the host config, set with `hammerkit remote use` 5. the local machine `auto` stands for the tasks' `runner`, else local. A run goes to a single server, so tasks that name different servers fail the run, and a name that is not registered is an error: hammerkit never quietly runs a task somewhere else than you asked. `local` and `auto` can't be used as a `runner`. # Running on Kubernetes (/docs/task/kubernetes) Since `1.6.0` hammerkit can execute the same build file either on the local Docker daemon or on a Kubernetes cluster. This is done through an **environment**: tasks run as jobs on the cluster, container services run as deployments, and the result of each task is cached the same way as locally. This is different from [Kubernetes services](/docs/service/kubernetes), which only forward existing resources from a cluster into a local task. Here your own tasks and services run on the cluster. ## Declaring an environment [#declaring-an-environment] Environments are declared in a top-level `environments:` block. Each environment targets either `kubernetes` or `docker`. ```yaml title=".hammerkit.yaml" environments: default: kubernetes: context: docker-desktop services: postgres: image: postgres:16-alpine envs: POSTGRES_USER: postgres POSTGRES_DB: demo secrets: - from: env:DB_PASSWORD env: POSTGRES_PASSWORD ports: - 5432:5432 healthcheck: cmd: "pg_isready -U postgres" tasks: api: image: node:24-alpine needs: [postgres] cmds: - node index.js ``` ### Kubernetes target [#kubernetes-target] | Field | Required | Description | | --------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `context` | yes | The kube context (cluster + user) the tasks run in. | | `namespace` | no | Namespace the jobs and deployments are created in. | | `kubeconfig` | no | Path to a kubeconfig file. Defaults to `$HOME/.kube/config`. | | `httpRoutes` | no | Gateway API routes that expose services, see [HTTP routes](#http-routes). | | `networkPolicy` | no | `enforced`: the cluster's network plugin enforces NetworkPolicy, which a task or service with a limited [network](/docs/task/network#kubernetes) needs. | | `volumeSize` | no | The size of the [claim](#claims) of a task or service that states no `volumeSize` of its own, `1Gi` when left out. | The same `context`, `kubeconfig` and `namespace` also supply the cluster connection for port-forward [Kubernetes services](/docs/service/kubernetes): declare the cluster once here and select it with `--env`, rather than repeating it on each service. ### Docker target [#docker-target] | Field | Required | Description | | ------ | -------- | ------------------------------------------------- | | `host` | no | Address of a remote Docker daemon to run against. | ## Claims [#claims] On Kubernetes each container task and service gets one `PersistentVolumeClaim` (`ReadWriteOnce`, the cluster's default storage class). It holds the files of its `src`, of the `src` of its dependencies (as Docker mounts them) and of its `mounts`, uploaded before it starts, and what it `generates`, which the tasks that depend on it mount from there. It stays until `hammerkit clean`, so the next run finds the outputs it already made. On a server, the claim of a task stays for a while after the run, for the next run of the same source, see [Kept claims](/docs/remote-compute/backend#kept-claims). So a task whose dependencies have a `src` gets a claim, and an upload before it starts, even when it has no `src`, `mounts` or `generates` of its own (a `check` that only reads what `install` made and its sources). Each such claim counts against the storage and pod quota of the namespace and adds the time of an upload pod to the task. Where a dependency's `src` covers a large part of the project, a narrower `src` keeps that small. Each path has a folder of its own in the claim, named after its last segment. When two paths of a task end alike, such as `node_modules` and `api/node_modules`, the least deep one keeps that name and the others' folders also carry a hash of their path in the project. Earlier versions put such paths in one folder, so a task with paths that end alike has a new id since: the first run after the update runs it again, in a new claim, instead of reusing the mixed outputs it made before (kept or cached). A service keeps its claim and its id, so a service with two volumes that end alike should have its claim removed once after the update (`hammerkit clean`), since the first of them still holds what both wrote. A path of `generates` with an extension, such as `lcov.info`, is a file, as on Docker: it is in the claim as an empty file before the task starts, and the task writes into it where it is. A tool that replaces the file instead of writing into it, by moving another one over it or by removing it first (BusyBox's `cp` in an Alpine image does), can't do that with a file mounted on its own, as on Docker: write into it (`cat a > b`), or let such a task generate the folder the file is in instead. hammerkit tells a file by its extension, so a folder with a dot in its name, such as `x.egg-info`, counts as a file too: name the folder it is in. Making the file takes an upload pod before the task, also when it has no `src`. Earlier versions made a folder of such a path, which a kept claim may still hold: it is replaced by the file before the task runs. A task's `src` is mounted read only. A path mounted inside one of its folders, such as a `generates: [pkg/coverage/lcov.info]` next to `src: [pkg]`, gets its place in the uploaded folder before the task starts, since the container runtime can't make it there itself. A `src` inside a folder of another `src`, such as `src: [docs, docs/Dockerfile]`, is in that folder's upload and mount, as on Docker, and is not mounted on its own. hammerkit tells a `src` or `mounts` path that is there by what it is on disk, so a file without an extension, such as `Dockerfile`, counts as a file. The claim requests `1Gi`, or the environment's `volumeSize`. A task that writes more (an `npm ci`, a build of many images) states its own: ```yaml tasks: install: image: node:24-alpine src: [package.json, package-lock.json] generates: [node_modules] volumeSize: 5Gi cmds: - npm ci ``` A cloud disk is the size it requests, so a task that outgrows it fails with `No space left on device`; a local cluster's host path storage is not limited. A larger `volumeSize` grows a claim an earlier run made, where its storage class allows volume expansion. Where it does not, the run fails and says so: remove the claim with `hammerkit clean`. A smaller one leaves a claim as it is, a claim never shrinks. ## Selecting an environment [#selecting-an-environment] Pass `--env ` to run against a declared environment. Without it, hammerkit uses the local Docker daemon. ```bash hammerkit api --env default ``` The `--env` option is available on the `run`, [`store` and `restore`](/docs/cli/store-restore) commands, so cache results can be stored from and restored to a cluster. ## Healthchecks [#healthchecks] A container service [healthcheck](/docs/service/container#healthcheck) is translated into readiness, liveness and startup probes on the Kubernetes deployment, so dependent tasks only start once the service is ready. A service without one counts as ready once its container ran for 10 seconds without a restart (the deployment's `minReadySeconds`), so a service that crashes right after it started is not taken as up. ## HTTP routes [#http-routes] An HTTP route exposes a service of the environment under a hostname, for example to reach a deployed application from outside the cluster. Hammerkit creates a Gateway API [`HTTPRoute`](https://gateway-api.sigs.k8s.io/) per entry, attached to a `Gateway` in the cluster, and removes it again with `hammerkit down`. ```yaml environments: staging: kubernetes: context: staging-cluster namespace: my-app httpRoutes: - host: api.example.com service: api servicePort: 3000 path: / gateway: web # parent Gateway in this namespace # gatewayNamespace: gateways # set if the Gateway lives elsewhere ``` | Field | Required | Description | | ------------------ | -------- | ---------------------------------------------------------- | | `host` | yes | Hostname the route responds to. | | `service` | yes | Name of the service to route to. | | `gateway` | yes | Name of the parent `Gateway`. | | `gatewayNamespace` | no | Namespace of the parent `Gateway`, if not the route's own. | | `servicePort` | no | Port of the service to route to. | | `path` | no | Path prefix that is routed to the service. | This requires the [Gateway API CRDs](https://gateway-api.sigs.k8s.io/guides/) and a `Gateway` resource to be installed in the cluster. Hammerkit no longer creates `Ingress` resources: the Gateway API is their successor, and an `ingresses` list in a build file is rejected with a pointer to `httpRoutes`. # Needs (/docs/task/needs) `needs` connect a task to the [services](/docs/service) it depends on — a database, a queue, an API. When you run a task, hammerkit starts every service it needs, waits until each is ready, runs the task, and stops the services again once nothing else needs them. ```yaml services: postgres: image: postgres:16-alpine healthcheck: cmd: "pg_isready -U postgres" ports: - 5432 tasks: api: description: "start api" image: node:24-alpine needs: [postgres] cmds: - node index.js ``` ## Readiness [#readiness] If a service defines a [`healthcheck`](/docs/service/container#healthcheck), hammerkit runs that command inside the service container and only starts the dependent task once it exits `0`. The check is polled about once a second until it passes or its `timeout` (default `20s`) runs out; there are no `interval`/`retries` knobs. A check that never passes fails the service and the run, so make the command something that genuinely reports readiness (like `pg_isready`). Without a healthcheck the task may start the moment the container starts, before the service inside it is actually accepting connections. ## Connecting to the service [#connecting-to-the-service] How a task reaches its service depends on whether the task runs in a container: * **Container task** (has an `image`): it shares a network with the service and reaches it by the **service name** as the hostname, on the service's **container port**. For the example above, connect to `postgres:5432`. * **Local task** (no `image`): it isn't on that network, so hammerkit injects the connection details as environment variables instead — `HAMMERKIT_POSTGRES_HOST` and `HAMMERKIT_POSTGRES_PORT`. See [environment hints for local tasks](/docs/service#environment-hints-for-local-tasks). A robust pattern that works for both is to keep the connection in a shared top-level `envs` block and reference it from the service and the task: ```yaml title=".hammerkit.yaml" envs: POSTGRES_USER: api POSTGRES_DB: api DATABASE_URL: postgres://api@postgres:5432/api services: postgres: image: postgres:16-alpine envs: POSTGRES_USER: api POSTGRES_DB: api POSTGRES_HOST_AUTH_METHOD: trust healthcheck: cmd: "pg_isready -U api" ports: - 5432 tasks: migrate: image: node:24-alpine needs: [postgres] envs: DATABASE_URL: $DATABASE_URL cmds: - node migrate.js ``` ## Naming a need [#naming-a-need] A bare `needs: [postgres]` uses the service under its own name. A service of an included or referenced file has a prefixed name, such as `db:postgres`. A host name can't contain a colon, so the task reaches it as `db-postgres`, with dashes in place of the colons. To use another alias, such as plain `postgres`, use the object form: ```yaml tasks: api: image: node:24-alpine needs: - service: db:postgres # the service to depend on name: postgres # the hostname/alias the task uses cmds: - node index.js ``` Two needs of one task can't share a name: a build file that would reach two services under the same name is an error that asks for a `name:` on one of them. See [services](/docs/service) for defining services, lifecycle and the [connection-string details](/docs/service#environment-hints-for-local-tasks). # Network (/docs/task/network) A container task reaches whatever the machine it runs on reaches. That is fine for most builds, but not for a step that should not need the network (a test suite), or for code you don't fully trust (an install script, an agent's change). `network` limits what a task or service reaches besides the services it needs. ```yaml tasks: test: image: node:24-alpine network: none # nothing at all cmds: [npm test] e2e: image: node:24-alpine needs: [postgres] network: none # postgres, and nothing else cmds: [npm run e2e] install: image: node:24-alpine network: allow: [registry.npmjs.org, '*.githubusercontent.com'] cmds: [npm ci] migrate: image: node:24-alpine network: allow: ['db.example.com:5432'] # raw TCP to one host and port cmds: [npm run migrate] deploy: image: node:24-alpine network: open # the default cmds: [npm run deploy] services: postgres: image: postgres:16-alpine healthcheck: cmd: pg_isready -U postgres ``` | Level | Reaches, besides its `needs` | | ---------------- | --------------------------------------------------------------------------------------------------- | | `open` (default) | Everything the host reaches, as without `network`. | | `none` | Nothing. | | `allow: [hosts]` | HTTP and HTTPS to the listed hosts, and raw TCP to a listed host and port, through an egress proxy. | The services in `needs` are reachable at every level, under the names the task needs them by: a task lists a service because it uses it. Under a limit, those names must be host names (`db`, `api-v2`), never patterns: the egress proxy passes a needed service by exactly its name. `network` works the same on [container services](/docs/service/container#network), and the tasks and services that need a limited service still reach it. `network` is not part of the [cache key](/docs/task/caching): it decides whether a task can work, not what a successful one produces. Ports a service publishes on the host stay published. A limit holds or the run fails: where hammerkit can't make it hold (an older Docker, Windows containers, a cluster that doesn't enforce NetworkPolicy), a task or service with a limited network is an error before it starts, never a run without limits. ## Allowed hosts [#allowed-hosts] A host is a name (`registry.npmjs.org`) or `*.` and a domain for every name under it (`*.githubusercontent.com` allows `raw.githubusercontent.com`, not `githubusercontent.com` itself). Names only, no IP addresses (nor names a resolver reads as one, such as `0x7f000001` or `127.1`), and any port of an allowed host is allowed through the proxy. A name with a port (`db.example.com:5432`) is a [TCP forward](#tcp-forwards). The proxy resolves an allowed name itself and connects to the address it checked. A name a wildcard allows must not resolve to the host, a private network or the cluster (loopback, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16` and the like), so a domain whose DNS someone else controls can't point into your network; a name you list exactly may. No name may resolve to a link-local address, where cloud metadata services live. Under `allow`, hammerkit starts an egress proxy next to the task and sets `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` (and their lowercase spellings) for it, with the needed services in `NO_PROXY`. Package managers, `curl`, `git` and most HTTP clients read these variables; Node's built-in `fetch` only with `NODE_USE_ENV_PROXY=1` (Node 22.21 and 24.5 on). The proxy passes requests and `CONNECT` tunnels to an allowed host and answers everything else with `403 Forbidden`. Each refused host shows up once in the task's log, so a failing install says why: ``` warn network: refused example.com ``` The proxy is the only way out. A tool that ignores the proxy variables finds no route at all, and it fails closed: when the proxy is gone, nothing gets out. A tool that ignores `NO_PROXY` (busybox `wget`, say) still reaches the needed services, through the proxy. BusyBox `wget`, the `wget` of Alpine images, also asks the proxy for an `https://` URL instead of opening a `CONNECT` tunnel; TLS stays between the client and the host, so the proxy doesn't serve that, and the log says so: use `curl` or GNU `wget` (`apk add wget`) for an https host. A service's healthcheck checks the service itself, so it runs without the proxy variables. ## TCP forwards [#tcp-forwards] Databases, caches and most other clients that don't speak HTTP ignore the proxy variables, so they can't use an allowed host. For them, an entry names a host and a port: ```yaml tasks: migrate: image: node:24-alpine network: allow: ['db.example.com:5432', 'cache.example.com:6379'] cmds: [npm run migrate] ``` The proxy listens on that port and forwards each connection, as raw TCP, to that port of that host, and the task reaches the proxy under the host's own name: `psql -h db.example.com` connects as it would without a limit, and TLS stays between the client and the host. The proxy resolves the name itself and checks every address, as for a name listed exactly. Nothing else of that host is reachable without the proxy variables; with them, a `CONNECT` to that host and port passes too. A connection to another port of that name finds nothing listening and is refused right away, so it doesn't show up among the refused hosts in the log. A forward names one host, never a `*.` pattern, and two hosts can't share a port in one allow list, since the proxy listens on it for one of them. Port `3128` is the proxy's own, and a forward can't use the name of a service the task needs. A database the task runs itself still belongs in its `needs` as a [service](/docs/service): those are reached directly, on any port. On a [server](/docs/remote-compute/backend#kubernetes), a host or forward with a private address (`10.x`, `192.168.x` and the like) is reachable only when the operator lists it in `allowedEgress`, or in the firewall of a Docker host: the server keeps its runs off the networks behind it unless told otherwise. ## A ceiling for the run [#a-ceiling-for-the-run] `--network` on [`run`](/docs/cli/execute) and [`up`](/docs/cli/up) sets the most any task or service of the run reaches, whatever the build file says. It is how a run of code you didn't write gets limited without changing the build file: ```bash hammerkit run test --network none hammerkit run install --network allow:registry.npmjs.org,*.npmjs.com hammerkit run migrate --network allow:db.example.com:5432 ``` The levels are ordered `none` \< `allow` \< `open`, and the ceiling clamps a more open setting down to it: under `--network none` every task reaches its needed services and nothing else. Under an `allow:` ceiling, an `open` task gets the ceiling's hosts, and a task with its own allow list keeps the hosts both lists allow. A host without a port allows its forwards (`db.example.com` and `*.example.com` both keep `db.example.com:5432`), while a forward only allows itself. A name that one of the two lists allows only through a pattern counts as a pattern: under `*.example.com`, the task's `db.example.com:5432` must not resolve to a private address. Two forwards on one port, one from each list, are an error. A service started by an earlier `up` is recreated when the ceiling changes its network. A [server](/docs/remote-compute/runs#network) sets the same ceiling for all of its runs with `runs.network`, and refuses to publish a build file that asks for more. With [`runs.secretNetwork`](/docs/remote-compute/runs#tasks-that-read-secrets) it sets a lower one for the tasks that read secrets. ## Local tasks [#local-tasks] Hammerkit can't limit the network of a local task (one without an `image`), which runs on the host. `network` other than `open` on a local task is an error when the build file is read, and so is a `--network` ceiling below `open` with a local task in the run. Run the task in a container to limit it. On a [server with a Kubernetes backend](/docs/remote-compute/backend#network) a local task runs in a pod of a tool image, and there it may be limited. ## Docker [#docker] Limits need Linux containers on Docker 28 or later. A limited task or service gets an internal Docker network of its own, with no route out and no address on the host (`com.docker.network.bridge.inhibit_ipv4`), so it doesn't reach the host's own services either. The first service it needs joins that network under the name it needs it by; each further one gets an internal network of its own that only the task or service (and its proxy) joins, so the services it needs don't reach each other through it. A task with `network: none` that needs no service gets no network at all, only the loopback interface. Each of these networks takes an address range of the Docker daemon, whose default pools hold about 30. A host that runs many limited tasks with several needs at once can run out of them; smaller `default-address-pools` (such as `/24`s) in the daemon's `daemon.json` make room for more, and the error says so. The container that copies a task's or service's outputs into and out of its volumes for the [cache](/docs/task/caching) runs its image, so it has no network at all, whatever the task's `network`. Under `allow`, the task or service and its proxy share a second internal network that nothing else joins, and the proxy listens there only: the services it needs can't use its allow list. The host of each TCP forward is an entry in the container's `/etc/hosts` that points at the proxy there. The proxy is also on the default bridge, to reach the allowed hosts. It is a small Node.js script run in a stock `node:24-alpine` container, pinned by digest, so there is no hammerkit image to pull; it runs unprivileged, on a read-only root. A limited service's ports are published by a relay container next to it, which passes each connection on to the service; the service itself stays without a route out. The networks, proxy and relay are removed with the task or service: a service started with `up --daemon` keeps them until `down`, and [`clean`](/docs/cli/clean) removes those of a run that was killed. Their names carry the run, so two runs of the same task never share them. ## Kubernetes [#kubernetes] On a [Kubernetes environment](/docs/task/kubernetes), a limited task's jobs and a limited service's pods get a `NetworkPolicy` that allows egress to the pods of the services they need and, under `allow`, to their proxy, which runs as its own Deployment and Service, with a port for each TCP forward. Everything else is denied, DNS included: the needed services, the proxy and the hosts of TCP forwards are host aliases of the pod. The policy limits ingress too: a task takes no connections, a service takes them from the tasks and services that need it, and from anywhere when one of the environment's `httpRoutes` points at it. The proxy takes connections from its own task's or service's pods only. A service's healthcheck probe runs through `env` to empty the proxy variables, so its image needs an `env` binary. A `NetworkPolicy` only takes effect when the cluster's network plugin enforces it (Calico, Cilium, and recent versions of kind's kindnet do; some plain setups don't), and hammerkit can't tell from the API. So the environment states it: ```yaml environments: cluster: kubernetes: context: ci networkPolicy: enforced ``` Without it, a task or service with a limited network is an error. Some plugins, kindnet among them, apply a policy a moment after a pod started. So a limited pod starts with an init container, `hammerkit-network`, that waits until the pod no longer reaches the cluster DNS, which its policy denies; the task's or service's own containers start after it. When the policy doesn't take hold within a minute, the pod fails rather than run without limits. The init container runs from the proxy's image. On a [server](/docs/remote-compute/backend#namespaces), a pod with an open network is limited too, by the policies of its namespace, and has the same init container when the server states `networkPolicy: enforced`: it waits until the pod no longer reaches the clouds' metadata endpoint, `169.254.169.254`, which those policies deny by default. On a cluster where that address doesn't answer, it opens at once; the server's `openNetworkGate` points it at another denied address or turns it off (see [Namespaces](/docs/remote-compute/backend#namespaces)). The pod that copies files into and out of a task's or service's volume claims mounts its outputs where the task wrote them, so it runs nothing from its image's paths: an init container copies a pinned, static busybox into a folder of its own, mounted read-only, and every command runs that busybox. No output may be in or over that folder. On a server, that pod has no network at all: the policies of its task or service leave it out. A task's policy and proxy belong to its run, so concurrent runs of the same task keep their own. They are removed only once the run's pods have ended, so a cancelled or timed-out job doesn't keep running without limits while it terminates. A task's refused hosts show up in its log; a service outlives the run on a cluster, so its refused hosts are in the log of its proxy, the Deployment named `hammerkit-proxy-`. # Secret providers (/docs/task/secret-providers) A [secret](/docs/task/secrets) can be read from a secret manager such as Google Secret Manager or 1Password. Hammerkit does not know any manager: a **provider** is a command that prints the value, and an **account** says which service account it runs as. A company declares both once and every project reuses them. ```yaml tasks: test: image: node:22-alpine secrets: - from: gsm:test-db-password # provider gsm, reference test-db-password env: DB_PASSWORD cmds: - npm test ``` `from: :` names the provider and what to fetch. The text after the first `:` is the reference, passed to the provider as it is. ## Providers [#providers] A provider is a command as a list of arguments. It is never run through a shell, and `{{ref}}` is replaced by the secret's reference. ```yaml envs: GCP_PROJECT: company-secrets-prod # an input, replaced with `with:` when included secretProviders: gsm: command: [gcloud, secrets, versions, access, latest, '--secret={{ref}}', '--project=${GCP_PROJECT}'] op: command: [op, read, --no-newline, '{{ref}}'] ``` | Field | Description | | --------- | ----------------------------------------------------------------------------------------------------------------------- | | `command` | The command and its arguments. `${NAME}` is a variable of the file's `envs`, else of the environment hammerkit runs in. | | `env` | Static environment variables of the command. | | `timeout` | How long the command may run, `30s` when omitted. | The recipes put the reference in `--secret={{ref}}`, so a reference can never become an option of the CLI, and a reference that starts with `-` is rejected. The output is the value as it is, without trimming: a key file keeps its trailing newline, which is why 1Password is read with `--no-newline`. A command that fails, is not found, runs too long or prints nothing fails the task before it starts, naming the secret, the provider and the account. ## Accounts [#accounts] An account is a service account: the environment that makes a provider's command act as it. Each value points at a credential the machine running hammerkit holds, so no credential is ever written to a build file. ```yaml secretAccounts: test: # used when nothing else is named default: true providers: gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: '${GSM_TEST_CREDENTIALS}' } } op: { env: { OP_SERVICE_ACCOUNT_TOKEN: '${OP_TEST_TOKEN}' } } deploy: # production: only machines that deploy hold this credential providers: gsm: { env: { CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: '${GSM_DEPLOY_CREDENTIALS}' } } ``` A secret is read as its own `account`, else the `account` of its task or service, else the account marked `default: true`: ```yaml tasks: deploy: account: deploy # every provider secret of this task image: deployer:1 secrets: - from: gsm:prod-db-password env: DB_PASSWORD - from: op:op://Prod/api/token path: /run/secrets/api-token account: test # this one is read as `test` ``` An unknown provider or account, an account without credentials for the secret's provider, or no account where one is needed fails when the build file is read. Every task of the file is checked then, so it stops a task that reads no secret, too. The message names the task, the secret, the account and the file to declare it in, with what to add: ``` task cli reads secret /run/secrets/gcloud.json from gsm:ci-key as account operator (the account: of the secret), but no account operator is declared (declared: tests in /home/me/.hammerkit/config.yaml). An account says which credential a secret is read with, and each machine declares its own in its host config, /home/me/.hammerkit/config.yaml. Add operator there: secretAccounts: operator: providers: gsm: env: CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE: ${OPERATOR_CREDENTIALS} Or read the secret as one of the declared accounts with account: on the secret. Every task of the build file is checked when it is read, so this stops other tasks too. See https://hammerkit.dev/docs/task/secret-providers#accounts ``` The variables of the snippet are the ones another account passes to the same provider. ### Only a service account can authenticate [#only-a-service-account-can-authenticate] The command runs with `PATH`, the provider's `env` and the account's `env`, and nothing else of your shell. A personal `gcloud auth login` or the 1Password app is never picked up, and when a `${VARIABLE}` of the account is not set, the secret fails with `account test, provider op: OP_TEST_TOKEN is not set` and the command does not run. Several accounts can coexist on one machine under different variable names, so a deploy task and a test task use different identities in one run. | Manager | Credential the account passes | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1Password | `OP_SERVICE_ACCOUNT_TOKEN` of a [service account](https://developer.1password.com/docs/service-accounts/use-with-1password-cli/). Do not pass `OP_CONNECT_*`, it takes precedence. | | Google Secret Manager | `CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE`, a service-account key file or a workload identity federation config file. `google-github-actions/auth` exports it. | Prefer short-lived credentials (workload identity federation, a token your CI mints per job) over long-lived key files. If `gcloud` or `op` need a configuration directory, point `CLOUDSDK_CONFIG` or `OP_CONFIG_DIR` at one in the provider's `env`. ## Sharing providers and accounts [#sharing-providers-and-accounts] Providers and accounts are names shared by all build files of a build, like [caches](/docs/build-file/caches). Keep them in one file the platform team owns and [include](/docs/build-file/includes) it: ```yaml includes: company: git: https://github.com/acme/company-secrets.git ref: 3f9c2e1a8b7d4c5e6f0a1b2c3d4e5f60718293a4 # a full commit SHA with: GCP_PROJECT: company-secrets-prod ``` * A provider or account declared in a **git include requires `ref` to be a full commit SHA.** A tag or branch can be moved, and a moved file would run with your service accounts' credentials. * A name declared in two files is an error naming both files. * `env` and `file` are built in and cannot be provider names. ## On this machine only [#on-this-machine-only] A build file that a [server](/docs/remote-compute/build-files) runs may not declare providers or accounts: the server reads secrets with its own. To use the same build file locally, declare them in the host config, `config.yaml` in hammerkit's directory (`~/.hammerkit` on Linux, `~/Library/Application Support/hammerkit` on macOS, `%APPDATA%\hammerkit` on Windows; `HAMMERKIT_CONFIG` names another file), and keep only the secrets in the build file: ```yaml title="config.yaml" secretProviders: op: command: [op, read, 'op://{{ref}}'] secretAccounts: dev: default: true providers: op: env: OP_SERVICE_ACCOUNT_TOKEN: ${OP_DEV_TOKEN} ``` ```yaml title=".hammerkit.yaml" tasks: test: image: node:22-alpine secrets: - from: op:dev/db/password env: DB_PASSWORD cmds: [npm test] ``` A local run uses the providers and accounts of the host config and of the build files together. The machine that runs a task says how it reads secrets: a provider or account the host config declares replaces the one of the same name in a build file, and a default account in the host config replaces the build files' default. So a branch that still declares its providers loads, and uses yours. (Two build files that declare the same name are an error.) An account of the host config may name a provider that only some projects declare: that is checked when a secret uses it, and a project that reads no secret loads whatever the host config says. `${NAME}` in the host config is read from the environment of the process. A server never reads a host config, it has its own [accounts](/docs/remote-compute/accounts). ## Values [#values] Each value is fetched once per run for a provider, account and reference, however many tasks use it, which matters for rate-limited services (a 1Password service account is limited per token). It is kept in memory, never written anywhere, and fetched again by the next run. `ls` and `validate` never fetch. The value is masked in hammerkit's logs like any other secret (see [Logs](/docs/task/secrets#logs)). The secret manager's own log shows the service account; with `--verbose` hammerkit logs `secret db-password via gsm/test` for each fetch, so a read can be traced to the run. ## Caching [#caching] As for any secret, a provider secret does not affect the cache by default. With `cache: true` the task id holds a salted digest of the fetched value, so a rotated value, or a new `latest` version in Google Secret Manager, gives a new id. ```yaml tasks: build: image: alpine:3.21 secrets: - from: gsm:license-key env: LICENSE_KEY cache: true cmds: - ./build.sh ``` Commands that need task ids (`exec`, `up`, `explain`, `cache push`/`pull`/`ls`, `clean`) fetch these secrets first, once, for the tasks in scope, so they need the account's credential too; a machine without it fails with the unset-variable error instead of computing another id. The digest depends on the value, not on who read it: two accounts that see the same value give the same id on every machine. The digest is stored in the cache entry's description, and remote caches carry it. Use `cache: true` for high-entropy or non-sensitive values (a license key, a toggle), never for a short password: a digest of a short value can be guessed. ## What a secret can still do [#what-a-secret-can-still-do] A task that receives a secret can print or send it, and so can the code it runs, such as the postinstall script of a dependency. Give secrets only to the tasks that need them, never to install or build steps. Where hammerkit itself runs in a container (a CI image), the provider's CLI has to be in that image. # Secrets (/docs/task/secrets) A credential — an API token, a registry password, a service-account key — reaches a task or service through `secrets`. The build file only names where the value comes from; the value itself is read when the task or service starts, never written to the build file, the cache or hammerkit's logs. ```yaml tasks: publish: image: node:22-alpine secrets: - from: env:NPM_TOKEN # host environment variable env: NPM_TOKEN # → environment variable in the container - from: file:~/.config/gcloud/key.json path: /run/secrets/gcloud.json # → read-only file in the container cmds: - npm publish ``` Each secret has a source and exactly one target: | Field | Description | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `from` | `env:NAME` — a variable of the environment hammerkit runs in, or one given with [`--set`](/docs/cli); `file:path` — a host file, relative to the build file or `~/…`; `provider:ref` — a value read from a [secret manager](/docs/task/secret-providers). | | `env` | Inject the value as this environment variable. | | `path` | Mount the value as a read-only file at this path in the container. | | `cache` | `true` makes the value part of the task's cache key (see below). Default `false`. | | `account` | For a `provider:ref` source, the service account it is read as, see [secret providers](/docs/task/secret-providers). | Services take the same `secrets`. A service's [`init`](/docs/service/container#init) is a task and declares the secrets it needs itself. ```yaml services: postgres: image: postgres:17-alpine secrets: - from: env:DB_PASSWORD env: POSTGRES_PASSWORD healthcheck: cmd: pg_isready -U postgres ``` A secret manager such as Google Secret Manager or 1Password is a `provider:ref` source, read as a service account; see [secret providers](/docs/task/secret-providers). ## When the value is read [#when-the-value-is-read] Planning a run never reads a secret, so `ls`, `validate` and runs of other tasks work without it. The value is read right before the container starts; when its source is missing the task fails before anything runs, naming the secret: `secret NPM_TOKEN: environment variable NPM_TOKEN is not set`. ## Logs [#logs] Hammerkit masks every secret value it has read as `***` in all it logs, including the output it streams from tasks and services. Each line of a multi-line value (a key file) is masked on its own, and so are the common encodings of a value: base64 (also when it sits inside a longer base64 string, and the URL-safe form), hex, URL-encoded and JSON-escaped. Output is masked line by line, a line once it ends, however its output arrived (in several writes of a command or frames of docker); a line of more than 64 KiB is masked in parts of that length. The values belong to one run, so a process that serves several runs never masks one run's values in another's output. Values shorter than four characters are not masked, and the base64 and hex forms of a value only when they are at least eight characters long: masking shorter strings would mangle unrelated output. Masking is a safety net, not a boundary: a task can still print a value reversed, split or encrypted. A secret given as an `env` target is visible to anyone who can `docker inspect` the container, as with any environment variable; a `path` target is not. ## Caching [#caching] By default the value of a secret does not affect the cache: rotating a token does not rebuild everything that uses it. Which secrets a task declares, and where from, does: the task id holds each secret's name and source (a variable, a provider reference and account, or `file` without its path, which differs between machines), never its value. When a secret determines the output — a license key that changes what gets built, say — set `cache: true`. The cache key then holds a salted digest of the value, never the value itself, and a new value means a new key. ```yaml tasks: build: image: alpine:3.21 secrets: - from: env:LICENSE_KEY env: LICENSE_KEY cache: true cmds: - ./build.sh ``` A `provider:ref` source takes `cache: true` too; the commands that need task ids then fetch the value first, see [secret providers](/docs/task/secret-providers#caching). ## Runtimes [#runtimes] * **Docker**: `env` targets are set on the container. A `file:` source is bind-mounted read-only. An `env:` source with a `path` target is written to an owner-only file in hammerkit's data directory, mounted read-only and removed when the container is. The file is readable by the host user only; an image whose process runs as another user (a database dropping to its own user, say) and needs to read it should get the secret as an `env` target, or from a `file:` source whose permissions you set. * **Kubernetes**: the values go into a `Secret` of the task or service, which the pod references (`secretKeyRef` for `env` targets, a read-only `subPath` mount for `path` targets). A task's Secret is deleted once it has run, a service's when it stops. * **Local tasks** take `env` targets only; a `path` target on a local task is an error. A daemon service started with [`up`](/docs/cli/up) keeps its secrets until `down`. A changed secret declaration recreates it on the next `up`; a rotated value does not — run `hammerkit down` first. # Watching (/docs/task/watching) Tasks with source folders and files can be watched and restarted when they change. This is useful, for example, to run API servers that restart when the server code changes. ```yaml tasks: api: src: - src cmds: - node -r ts-node/register src/index.ts ``` ```bash hammerkit api --watch ``` In watch mode hammerkit watches the `src` of every task in the run. When a file changes it re-runs the affected task — and, because it knows the dependency graph, any tasks downstream of it — reusing cached results for everything unchanged. A short debounce coalesces bursts of saves into a single restart. Combine `--watch` with services (via `needs`, or [`hammerkit up --watch`](/docs/cli/up)) to keep a database running while your api restarts on every edit; the [development workflow guide](/docs/guides/development-workflow) puts these together. ## Tools that watch their files [#tools-that-watch-their-files] Some tools watch their files themselves and rebuild incrementally, which is faster than hammerkit restarting them: `ng serve`, `vite`, `nodemon`. They run until you stop them, so they are a [service](/docs/service/container#continuous) with `continuous: true`, not a task. `hammerkit up --watch` starts them once and leaves them running while their files change. ```yaml services: serve: image: node:24 continuous: true src: - src - angular.json cmd: npx ng serve --host 0.0.0.0 ports: - 4200 ``` Before hammerkit 2, a task could be marked `continuous` too. Such a build file now fails to parse: make the task a service, with its `cmds` as its `cmd`. # .NET (/docs/tutorials/dotnet) This tutorial restores the NuGet packages of a small .NET solution once, then runs `dotnet test` and `dotnet publish` against them without network access, each in an `mcr.microsoft.com/dotnet/sdk:10.0` container and each skipped when its inputs haven't changed. You need [hammerkit and Docker](/docs/installation); the .NET SDK doesn't have to be installed on your machine. The finished project is [`examples/tutorial-dotnet`](https://github.com/no0dles/hammerkit/tree/master/examples/tutorial-dotnet). ## The project [#the-project] A console app and an xUnit test project: ```txt tutorial-dotnet/ ├── Directory.Build.props ├── src/Greet/ │ ├── Greet.csproj │ ├── packages.lock.json │ ├── Greeter.cs │ └── Program.cs └── tests/Greet.Tests/ ├── Greet.Tests.csproj ├── packages.lock.json └── GreeterTests.cs ``` `Directory.Build.props` applies to both projects. It sets up three things the build file relies on: ```xml title="Directory.Build.props" true $(MSBuildThisFileDirectory).nuget/packages true ``` * **`UseArtifactsOutput`** writes `bin` and `obj` to `artifacts/` instead of into each project folder. The project folders are the tasks' sources, and they're mounted read-write: build output written into them would change the tasks' own inputs, and the next run would miss the cache. * **`RestorePackagesPath`** restores packages into `.nuget/packages` in the repository instead of `~/.nuget/packages`, where a task can keep them as its output. * **`RestorePackagesWithLockFile`** writes `packages.lock.json` next to each project: the full list of packages with their hashes, transitive ones included. Commit them; they're what the restore is cached against. ```xml title="tests/Greet.Tests/Greet.Tests.csproj" net10.0 enable enable false ``` ```cs title="tests/Greet.Tests/GreeterTests.cs" using Greet; using Xunit; namespace Greet.Tests; public class GreeterTests { [Fact] public void GreetsByName() { Assert.Equal("Hello, hammerkit!", Greeter.Hello("hammerkit")); } } ``` ## 1. Restore packages [#1-restore-packages] A task names the image it runs in, the files it reads (`src`) and the files it produces (`generates`). The `restore` task reads only the project and lock files: ```yaml title=".hammerkit.yaml" tasks: restore: description: download the locked NuGet packages image: mcr.microsoft.com/dotnet/sdk:10.0 src: - Directory.Build.props - src/Greet/Greet.csproj - src/Greet/packages.lock.json - tests/Greet.Tests/Greet.Tests.csproj - tests/Greet.Tests/packages.lock.json generates: [.nuget] cmds: - dotnet restore tests/Greet.Tests --locked-mode ``` ```bash hammerkit restore ``` Restoring the test project restores the app it references too. Run it again and hammerkit skips it: no project or lock file changed. Editing a `.cs` file doesn't invalidate it either, since `restore` doesn't read them. `--locked-mode` makes the restore fail instead of quietly updating a lock file that no longer matches its project. ## 2. Test and publish [#2-test-and-publish] Both tasks depend on `restore`, which mounts its `.nuget` into their containers: ```yaml title=".hammerkit.yaml" tasks: test: description: run the unit tests image: mcr.microsoft.com/dotnet/sdk:10.0 deps: [restore] src: [src, tests] cmds: - dotnet test tests/Greet.Tests --disable-build-servers publish: description: publish the app image: mcr.microsoft.com/dotnet/sdk:10.0 deps: [restore] src: [src] generates: - path: out export: true cmds: - dotnet publish src/Greet --configuration Release --output out --disable-build-servers ``` Neither lists `Directory.Build.props`: a task also sees the sources of the tasks it depends on, and a change to them reruns it. `dotnet test` and `dotnet publish` restore again before they build, but every package is already in `.nuget/packages`, so that step finishes locally without the network. `--disable-build-servers` stops the SDK from leaving its compiler server running after the command. Hammerkit keeps a task's container around for the next run, and the server would keep running in it for nothing. A container task's outputs stay in a volume hammerkit manages; `export: true` also copies `out` into your project. The published app runs with `dotnet out/Greet.dll`. `test` and `publish` don't depend on each other, so they run in parallel. ## 3. One task for CI [#3-one-task-for-ci] A task without `cmds` groups other tasks. `ci` names everything a change has to pass. The complete build file: ```yaml title=".hammerkit.yaml" tasks: restore: description: download the locked NuGet packages image: mcr.microsoft.com/dotnet/sdk:10.0 src: - Directory.Build.props - src/Greet/Greet.csproj - src/Greet/packages.lock.json - tests/Greet.Tests/Greet.Tests.csproj - tests/Greet.Tests/packages.lock.json generates: [.nuget] cmds: - dotnet restore tests/Greet.Tests --locked-mode test: description: run the unit tests image: mcr.microsoft.com/dotnet/sdk:10.0 deps: [restore] src: [src, tests] cmds: - dotnet test tests/Greet.Tests --disable-build-servers publish: description: publish the app image: mcr.microsoft.com/dotnet/sdk:10.0 deps: [restore] src: [src] generates: - path: out export: true cmds: - dotnet publish src/Greet --configuration Release --output out --disable-build-servers ci: description: everything CI checks deps: [test, publish] ``` ```bash hammerkit run ci ``` ```txt Summary: ci executed 7.3s publish executed 5.7s restore executed 3.7s test executed 7.3s 4 executed, 0 cached (0% cache hit), 7.3s total ``` Run it again without changing anything: ```txt Summary: ci cached 15ms publish cached 0ms restore skipped 0ms test cached 0ms 0 executed, 3 cached, 1 skipped (100% cache hit), 35ms total ``` `restore` is skipped: every task that needs it was a cache hit, so the packages aren't needed either. Edit a file in `src` and `test` and `publish` run again, against the packages `restore` already has. Ignore what hammerkit and the SDK write into the project: ```txt title=".gitignore" .hammerkit .nuget artifacts out ``` ## Run it in CI [#run-it-in-ci] CI runs the same command: [install hammerkit](/docs/installation#in-ci) and run `hammerkit run ci`. A fresh runner starts with an empty cache; to reuse what your laptop or the previous run already built, share the cache as described in [caching in CI](/docs/guides/ci-caching). ## Next steps [#next-steps] * Tests that need a database: start it as a [service](/docs/service) and let the task [`need`](/docs/task/needs) it, see [services & networking](/docs/guides/services-networking). * Several projects in one repository: [monorepos](/docs/guides/monorepo). * A task reruns and you don't know why: [`hammerkit explain`](/docs/cli/explain). # Go (/docs/tutorials/go) This tutorial runs `go vet`, `go test` and `go build` for a small Go module in a `golang:1.25-alpine` container, each skipped when its inputs haven't changed. You need [hammerkit and Docker](/docs/installation); Go doesn't have to be installed on your machine. The finished project is [`examples/tutorial-go`](https://github.com/no0dles/hammerkit/tree/master/examples/tutorial-go). ## The project [#the-project] ```txt tutorial-go/ ├── go.mod ├── go.sum ├── main.go └── greet/ ├── greet.go └── greet_test.go ``` The tests use one third-party module, `go-cmp`: ```go title="go.mod" module example.com/greet go 1.25 require github.com/google/go-cmp v0.7.0 ``` ```go title="greet/greet.go" package greet import "fmt" // Hello returns the greeting for name. func Hello(name string) string { return fmt.Sprintf("Hello, %s!", name) } ``` ```go title="greet/greet_test.go" package greet import ( "testing" "github.com/google/go-cmp/cmp" ) func TestHello(t *testing.T) { if diff := cmp.Diff("Hello, hammerkit!", Hello("hammerkit")); diff != "" { t.Errorf("Hello() mismatch (-want +got):\n%s", diff) } } ``` ## 1. Test [#1-test] A task names the image it runs in and the files it reads (`src`). The `test` task reads the module files and all Go sources: ```yaml title=".hammerkit.yaml" envs: # keep the mounted module cache deletable (Go makes it read-only by default) GOFLAGS: -modcacherw tasks: test: description: vet and test every package image: golang:1.25-alpine src: [go.mod, go.sum, main.go, greet] mounts: [.cache/go-mod:/go/pkg/mod] cmds: - go vet ./... - go test ./... ``` ```bash hammerkit test ``` Run it again and hammerkit skips it: nothing in its `src` changed. Go downloads modules into its module cache, `/go/pkg/mod` in this image. That folder isn't part of the project, and it doesn't need to be an input: `go.sum` already pins every module's content. So instead of a task output it's a [mount](/docs/task/container#adding-mounts): `.cache/go-mod` in the project is mounted at `/go/pkg/mod`, and a module is downloaded once, not by every task run. ## 2. Build [#2-build] The `build` task compiles the binary into `bin`: ```yaml title=".hammerkit.yaml" tasks: build: description: compile the greet binary image: golang:1.25-alpine src: [go.mod, go.sum, main.go, greet] mounts: [.cache/go-mod:/go/pkg/mod] generates: - path: bin export: true cmds: - go build -o bin/greet . ``` A container task's outputs (`generates`) stay in a volume hammerkit manages, ready for tasks that depend on it. `export: true` also copies `bin` into your project, so you can run `bin/greet` yourself. The binary is built for Linux, the container's OS; set `GOOS` in the task's `envs` to build for another one. `test` and `build` don't depend on each other, so they run in parallel. ## 3. One task for CI [#3-one-task-for-ci] A task without `cmds` groups other tasks. `ci` names everything a change has to pass. The complete build file: ```yaml title=".hammerkit.yaml" envs: # keep the mounted module cache deletable (Go makes it read-only by default) GOFLAGS: -modcacherw tasks: test: description: vet and test every package image: golang:1.25-alpine src: [go.mod, go.sum, main.go, greet] mounts: [.cache/go-mod:/go/pkg/mod] cmds: - go vet ./... - go test ./... build: description: compile the greet binary image: golang:1.25-alpine src: [go.mod, go.sum, main.go, greet] mounts: [.cache/go-mod:/go/pkg/mod] generates: - path: bin export: true cmds: - go build -o bin/greet . ci: description: everything CI checks deps: [test, build] ``` ```bash hammerkit run ci ``` ```txt Summary: build executed 3.4s ci executed 4.6s test executed 4.5s 3 executed, 0 cached (0% cache hit), 4.6s total ``` Run it again without changing anything: ```txt Summary: build cached 0ms ci cached 12ms test cached 0ms 0 executed, 3 cached (100% cache hit), 28ms total ``` Ignore what hammerkit and Go write into the project: ```txt title=".gitignore" .hammerkit .cache bin ``` ## Run it in CI [#run-it-in-ci] CI runs the same command: [install hammerkit](/docs/installation#in-ci) and run `hammerkit run ci`. A fresh runner starts with an empty cache; to reuse what your laptop or the previous run already built, share the cache as described in [caching in CI](/docs/guides/ci-caching). ## Next steps [#next-steps] * Tests that need a database: start it as a [service](/docs/service) and let the task [`need`](/docs/task/needs) it, see [services & networking](/docs/guides/services-networking). * Several modules in one repository: [monorepos](/docs/guides/monorepo). * A task reruns and you don't know why: [`hammerkit explain`](/docs/cli/explain). # Tutorials (/docs/tutorials) Each tutorial takes a small `greet` project from an empty build file to a cached pipeline that CI runs with one command. The steps are the same in every language: 1. Pick an image that pins the toolchain, so nothing needs to be installed on the host. 2. Declare what each task reads (`src`) and produces (`generates`). 3. Order the tasks with `deps`. 4. Group everything CI checks in one `ci` task. 5. Run it twice: the second run executes nothing. TypeScript with npm, `tsc` and `node --test`. A virtualenv, pytest and ruff. Modules, `go vet`, `go test` and a binary. Cargo, tests and a release binary. Maven, JUnit and a jar. NuGet, xUnit and a published app. ## Dependencies first [#dependencies-first] Every tutorial except Go starts with a task that only downloads dependencies. It reads nothing but the manifest and the lockfile, so it runs again only when a dependency changes, and the tasks after it build against what it downloaded: | Language | Image | Download task | Keeps | | -------- | ----------------------------------- | --------------------------------- | -------------- | | Node.js | `node:24-alpine` | `npm ci` | `node_modules` | | Python | `python:3.13-slim` | `pip install -r requirements.txt` | `.venv` | | Rust | `rust:1.90-slim` | `cargo fetch` | `.cargo-home` | | Java | `maven:3.9-eclipse-temurin-21` | `mvn dependency:go-offline` | `.m2` | | .NET | `mcr.microsoft.com/dotnet/sdk:10.0` | `dotnet restore` | `.nuget` | A task's outputs have to be inside the project, so where a package manager downloads into the home directory by default, the tutorial points it at a folder in the project. Go is the exception: its module cache can only be set to an absolute path, which differs between your laptop and CI. There the module cache is [mounted](/docs/task/container#adding-mounts) instead, and every task downloads what's missing from it. ## Tested like the code [#tested-like-the-code] Every tutorial's project is in the repository's [`examples/`](https://github.com/no0dles/hammerkit/tree/master/examples) folder (`tutorial-node`, `tutorial-python`, …) and hammerkit's integration tests run it: `ci` builds from scratch, then a second run has to come entirely from the cache. The build files in the tutorials are those files. # Java (/docs/tutorials/java) This tutorial downloads the dependencies and plugins of a small Maven project once, then runs `mvn test` and `mvn package` against them offline, each in a `maven:3.9-eclipse-temurin-21` container and each skipped when its inputs haven't changed. You need [hammerkit and Docker](/docs/installation); neither a JDK nor Maven has to be installed on your machine. The finished project is [`examples/tutorial-java`](https://github.com/no0dles/hammerkit/tree/master/examples/tutorial-java). ## The project [#the-project] ```txt tutorial-java/ ├── pom.xml └── src/ ├── main/java/dev/hammerkit/greet/Greet.java └── test/java/dev/hammerkit/greet/GreetTest.java ``` ```xml title="pom.xml" 4.0.0 dev.hammerkit greet 1.0.0 21 UTF-8 org.junit.jupiter junit-jupiter 5.13.4 test org.junit.platform junit-platform-launcher 1.13.4 test maven-dependency-plugin 3.9.0 maven-surefire-plugin 3.5.4 org.apache.maven.surefire surefire-junit-platform 3.5.4 ``` Besides JUnit, the POM pins three things so that everything a build needs can be downloaded up front. Surefire, Maven's test runner, normally downloads its JUnit provider and picks a JUnit launcher only when the tests start. Declaring both moves them into the POM. `go-offline` downloads dependencies declared on plugins only from `maven-dependency-plugin` 3.9.0 on, so the POM pins that version too. ```java title="src/main/java/dev/hammerkit/greet/Greet.java" package dev.hammerkit.greet; public class Greet { public static String hello(String name) { return "Hello, " + name + "!"; } public static void main(String[] args) { System.out.println(hello(args.length > 0 ? args[0] : "world")); } } ``` ```java title="src/test/java/dev/hammerkit/greet/GreetTest.java" package dev.hammerkit.greet; import static org.junit.jupiter.api.Assertions.assertEquals; import org.junit.jupiter.api.Test; class GreetTest { @Test void greetsByName() { assertEquals("Hello, hammerkit!", Greet.hello("hammerkit")); } } ``` ## 1. Download dependencies [#1-download-dependencies] A task names the image it runs in, the files it reads (`src`) and the files it produces (`generates`). The `deps` task downloads every dependency and plugin the POM names into a local repository in the project: ```yaml title=".hammerkit.yaml" envs: # flags for every mvn call: no progress noise, and the local repository in # the project, where the deps task keeps it as its output MAVEN_ARGS: --batch-mode -Dmaven.repo.local=.m2 tasks: deps: description: download dependencies and plugins image: maven:3.9-eclipse-temurin-21 src: [pom.xml] generates: [.m2] cmds: - mvn dependency:go-offline ``` ```bash hammerkit deps ``` Run it again and hammerkit skips it: `pom.xml` is unchanged. Only a dependency change downloads again. `MAVEN_ARGS` is read by every `mvn` call. It moves the local repository from `~/.m2/repository` into the project, where `deps` can keep it as a task output. ## 2. Test and package [#2-test-and-package] Both tasks depend on `deps`, which mounts its `.m2` into their containers: ```yaml title=".hammerkit.yaml" tasks: test: description: compile and run the unit tests image: maven:3.9-eclipse-temurin-21 deps: [deps] src: [src] cmds: - mvn --offline test package: description: build the jar image: maven:3.9-eclipse-temurin-21 deps: [deps] src: [src] generates: - path: target export: true cmds: - mvn --offline package -DskipTests ``` Neither lists `pom.xml`: a task also sees the sources of the tasks it depends on, and a change to them reruns it, so editing the POM reruns `deps`, `test` and `package`. `--offline` keeps Maven from reaching the network, so an artifact missing from `deps`' output fails the build instead of being downloaded on the side. A container task's outputs stay in a volume hammerkit manages; `export: true` also copies `target` into your project, so the jar is at `target/greet-1.0.0.jar`. `-DskipTests` doesn't skip any checking: `test` runs the tests, and `ci` below requires both. `test` and `package` don't depend on each other, so they run in parallel. ## 3. One task for CI [#3-one-task-for-ci] A task without `cmds` groups other tasks. `ci` names everything a change has to pass. The complete build file: ```yaml title=".hammerkit.yaml" envs: # flags for every mvn call: no progress noise, and the local repository in # the project, where the deps task keeps it as its output MAVEN_ARGS: --batch-mode -Dmaven.repo.local=.m2 tasks: deps: description: download dependencies and plugins image: maven:3.9-eclipse-temurin-21 src: [pom.xml] generates: [.m2] cmds: - mvn dependency:go-offline test: description: compile and run the unit tests image: maven:3.9-eclipse-temurin-21 deps: [deps] src: [src] cmds: - mvn --offline test package: description: build the jar image: maven:3.9-eclipse-temurin-21 deps: [deps] src: [src] generates: - path: target export: true cmds: - mvn --offline package -DskipTests ci: description: everything CI checks deps: [test, package] ``` ```bash hammerkit run ci ``` ```txt Summary: ci executed 34.1s deps executed 31.8s package executed 33.9s test executed 34.1s 4 executed, 0 cached (0% cache hit), 34.2s total ``` Run it again without changing anything: ```txt Summary: ci cached 17ms deps skipped 0ms package cached 0ms test cached 0ms 0 executed, 3 cached, 1 skipped (100% cache hit), 34ms total ``` `deps` is skipped: every task that needs it was a cache hit, so the repository isn't needed either. Edit a file in `src` and `test` and `package` run again, against the artifacts `deps` already has. Ignore what hammerkit and Maven write into the project: ```txt title=".gitignore" .hammerkit .m2 target ``` ## Run it in CI [#run-it-in-ci] CI runs the same command: [install hammerkit](/docs/installation#in-ci) and run `hammerkit run ci`. A fresh runner starts with an empty cache; to reuse what your laptop or the previous run already built, share the cache as described in [caching in CI](/docs/guides/ci-caching). ## Next steps [#next-steps] * Tests that need a database: start it as a [service](/docs/service) and let the task [`need`](/docs/task/needs) it, see [services & networking](/docs/guides/services-networking). * Several modules in one repository: [monorepos](/docs/guides/monorepo). * A task reruns and you don't know why: [`hammerkit explain`](/docs/cli/explain). # Node.js (/docs/tutorials/node) This tutorial builds a small TypeScript library with `npm ci`, `tsc` and `node --test`, each in a `node:24-alpine` container and each skipped when its inputs haven't changed. You need [hammerkit and Docker](/docs/installation); Node doesn't have to be installed on your machine. The finished project is [`examples/tutorial-node`](https://github.com/no0dles/hammerkit/tree/master/examples/tutorial-node). ## The project [#the-project] ```txt tutorial-node/ ├── package.json ├── package-lock.json ├── tsconfig.json ├── src/greet.ts └── test/greet.test.js ``` ```json title="package.json" { "name": "greet", "private": true, "type": "module", "scripts": { "build": "tsc", "test": "node --test" }, "devDependencies": { "typescript": "5.9.3" } } ``` ```ts title="src/greet.ts" export function hello(name: string): string { return `Hello, ${name}!` } ``` The tests import the compiled code from `dist`: ```js title="test/greet.test.js" import { test } from 'node:test' import assert from 'node:assert/strict' import { hello } from '../dist/greet.js' test('greets by name', () => { assert.equal(hello('hammerkit'), 'Hello, hammerkit!') }) ``` ## 1. Install dependencies [#1-install-dependencies] A task names the image it runs in, the files it reads (`src`) and the files it produces (`generates`). Create `.hammerkit.yaml` with an `install` task: ```yaml title=".hammerkit.yaml" tasks: install: description: install the locked dependencies image: node:24-alpine src: [package.json, package-lock.json] generates: [node_modules] cmds: - npm ci ``` ```bash hammerkit install ``` `npm ci` runs in the container and `node_modules` is kept as the task's output. Run it again and hammerkit skips it: `package.json` and `package-lock.json` are unchanged, so there's nothing to install. ## 2. Build [#2-build] The `build` task compiles `src`. `deps: [install]` runs `install` first (or reuses its cached result) and mounts its `node_modules` into the build container: ```yaml title=".hammerkit.yaml" tasks: build: description: compile src to dist image: node:24-alpine deps: [install] src: [src, tsconfig.json] generates: - path: dist export: true cmds: - npm run build ``` A container task's outputs stay in a volume hammerkit manages, ready for the tasks that depend on it. `export: true` also copies `dist` into your project, so you can use the compiled code outside hammerkit. Edit `src/greet.ts` and run `hammerkit build`: `install` is cached and only `build` runs. ## 3. Test [#3-test] The tests read `test` and the `dist` that `build` generates: ```yaml title=".hammerkit.yaml" tasks: test: description: test the compiled code image: node:24-alpine deps: [build] src: [test] cmds: - npm test ``` `test` only declares its own folder in `src`. Its dependency's sources are part of its cache key anyway, so a change in `src/` reruns `build` and then `test`, while a change in `test/` reruns only `test`. ## 4. One task for CI [#4-one-task-for-ci] A task without `cmds` groups other tasks. `ci` names everything a change has to pass. The complete build file: ```yaml title=".hammerkit.yaml" tasks: install: description: install the locked dependencies image: node:24-alpine src: [package.json, package-lock.json] generates: [node_modules] cmds: - npm ci build: description: compile src to dist image: node:24-alpine deps: [install] src: [src, tsconfig.json] generates: - path: dist export: true cmds: - npm run build test: description: test the compiled code image: node:24-alpine deps: [build] src: [test] cmds: - npm test ci: description: everything CI checks deps: [build, test] ``` ```bash hammerkit run ci ``` ```txt Summary: build executed 2.4s ci executed 3.1s install executed 1.2s test executed 3.1s 4 executed, 0 cached (0% cache hit), 3.1s total ``` Run it again without changing anything: ```txt Summary: build cached 1ms ci cached 14ms install skipped 0ms test cached 0ms 0 executed, 3 cached, 1 skipped (100% cache hit), 30ms total ``` `install` is skipped: every task that needs it was a cache hit, so its output isn't needed either. Ignore what hammerkit and the tools write into the project: ```txt title=".gitignore" .hammerkit node_modules dist ``` ## Run it in CI [#run-it-in-ci] CI runs the same command: [install hammerkit](/docs/installation#in-ci) and run `hammerkit run ci`. A fresh runner starts with an empty cache; to reuse what your laptop or the previous run already built, share the cache as described in [caching in CI](/docs/guides/ci-caching). ## Next steps [#next-steps] * Tests that need a database: start it as a [service](/docs/service) and let the task [`need`](/docs/task/needs) it, see [services & networking](/docs/guides/services-networking). * Several packages in one repository: [monorepos](/docs/guides/monorepo). * A task reruns and you don't know why: [`hammerkit explain`](/docs/cli/explain). # Python (/docs/tutorials/python) This tutorial runs a small Python package's checks, ruff and pytest, in a `python:3.13-slim` container, with the virtualenv built once and reused until the requirements change. You need [hammerkit and Docker](/docs/installation); Python doesn't have to be installed on your machine. The finished project is [`examples/tutorial-python`](https://github.com/no0dles/hammerkit/tree/master/examples/tutorial-python). ## The project [#the-project] ```txt tutorial-python/ ├── requirements.txt ├── pyproject.toml ├── greet/__init__.py └── tests/test_greet.py ``` Every package is pinned, including the ones pytest pulls in, so the file determines exactly what gets installed: ```txt title="requirements.txt" iniconfig==2.3.0 packaging==26.3 pluggy==1.6.0 Pygments==2.21.0 pytest==9.1.1 ruff==0.16.10 ``` ```python title="greet/__init__.py" def hello(name: str) -> str: return f"Hello, {name}!" ``` ```python title="tests/test_greet.py" from greet import hello def test_greets_by_name(): assert hello("hammerkit") == "Hello, hammerkit!" ``` ## 1. Create the virtualenv [#1-create-the-virtualenv] A task names the image it runs in, the files it reads (`src`) and the files it produces (`generates`). The `install` task builds `.venv` from the requirements: ```yaml title=".hammerkit.yaml" tasks: install: description: create the virtualenv from the pinned requirements image: python:3.13-slim src: [requirements.txt] generates: [.venv] cmds: - python -m venv .venv - .venv/bin/python -m pip install --no-cache-dir -r requirements.txt ``` ```bash hammerkit install ``` Run it again and hammerkit skips it: `requirements.txt` is unchanged. Only a dependency change rebuilds the virtualenv. Call tools as `.venv/bin/python -m ` rather than `.venv/bin/`. The scripts in `.venv/bin` hard-code the absolute path the venv was created at, so they break when a cached `.venv` is restored into a checkout at another path, like a CI runner's. `python -m` works from anywhere. ## 2. Lint and test [#2-lint-and-test] Both tasks depend on `install`, which mounts its `.venv` into their containers, and read the package, the tests and the tool configuration: ```yaml title=".hammerkit.yaml" envs: # keep Python from writing __pycache__ into the mounted sources PYTHONDONTWRITEBYTECODE: '1' tasks: lint: description: lint with ruff image: python:3.13-slim deps: [install] src: [pyproject.toml, greet, tests] cmds: - .venv/bin/python -m ruff check . test: description: run the tests image: python:3.13-slim deps: [install] src: [pyproject.toml, greet, tests] cmds: - .venv/bin/python -m pytest ``` Sources are mounted read-write. Without `PYTHONDONTWRITEBYTECODE`, Python writes `__pycache__` folders into `greet` and `tests`, which changes the tasks' own inputs, and the next run misses the cache. `lint` and `test` don't depend on each other, so they run in parallel. ## 3. One task for CI [#3-one-task-for-ci] A task without `cmds` groups other tasks. `ci` names everything a change has to pass. The complete build file: ```yaml title=".hammerkit.yaml" envs: # keep Python from writing __pycache__ into the mounted sources PYTHONDONTWRITEBYTECODE: '1' tasks: install: description: create the virtualenv from the pinned requirements image: python:3.13-slim src: [requirements.txt] generates: [.venv] cmds: - python -m venv .venv - .venv/bin/python -m pip install --no-cache-dir -r requirements.txt lint: description: lint with ruff image: python:3.13-slim deps: [install] src: [pyproject.toml, greet, tests] cmds: - .venv/bin/python -m ruff check . test: description: run the tests image: python:3.13-slim deps: [install] src: [pyproject.toml, greet, tests] cmds: - .venv/bin/python -m pytest ci: description: everything CI checks deps: [lint, test] ``` ```bash hammerkit run ci ``` ```txt Summary: ci executed 6.9s install executed 6.1s lint executed 6.6s test executed 6.9s 4 executed, 0 cached (0% cache hit), 7.0s total ``` Run it again without changing anything: ```txt Summary: ci cached 19ms install skipped 0ms lint cached 0ms test cached 0ms 0 executed, 3 cached, 1 skipped (100% cache hit), 45ms total ``` `install` is skipped: every task that needs it was a cache hit, so the virtualenv isn't needed either. Edit a file in `greet` and only `lint` and `test` run again. Ignore what hammerkit and the tools write into the project: ```txt title=".gitignore" .hammerkit .venv ``` ## Run it in CI [#run-it-in-ci] CI runs the same command: [install hammerkit](/docs/installation#in-ci) and run `hammerkit run ci`. A fresh runner starts with an empty cache; to reuse what your laptop or the previous run already built, share the cache as described in [caching in CI](/docs/guides/ci-caching). ## Next steps [#next-steps] * Tests that need a database: start it as a [service](/docs/service) and let the task [`need`](/docs/task/needs) it, see [services & networking](/docs/guides/services-networking). * A task reruns and you don't know why: [`hammerkit explain`](/docs/cli/explain). # Rust (/docs/tutorials/rust) This tutorial fetches the crates of a small Cargo project once, then runs `cargo test` and `cargo build --release` against them offline, each in a `rust:1.90-slim` container and each skipped when its inputs haven't changed. You need [hammerkit and Docker](/docs/installation); Rust doesn't have to be installed on your machine. The finished project is [`examples/tutorial-rust`](https://github.com/no0dles/hammerkit/tree/master/examples/tutorial-rust). ## The project [#the-project] ```txt tutorial-rust/ ├── Cargo.toml ├── Cargo.lock └── src/ ├── lib.rs └── main.rs ``` The tests use one third-party crate, `pretty_assertions`. The manifest names the library and the binary explicitly; the next step explains why. ```toml title="Cargo.toml" [package] name = "greet" version = "0.1.0" edition = "2021" # Named explicitly so `cargo fetch` works from Cargo.toml and Cargo.lock # alone, before the sources are there. [lib] path = "src/lib.rs" [[bin]] name = "greet" path = "src/main.rs" [dev-dependencies] pretty_assertions = "1" ``` ```rust title="src/lib.rs" pub fn hello(name: &str) -> String { format!("Hello, {name}!") } #[cfg(test)] mod tests { use super::hello; use pretty_assertions::assert_eq; #[test] fn greets_by_name() { assert_eq!(hello("hammerkit"), "Hello, hammerkit!"); } } ``` ## 1. Fetch dependencies [#1-fetch-dependencies] A task names the image it runs in, the files it reads (`src`) and the files it produces (`generates`). The `fetch` task downloads every crate `Cargo.lock` pins: ```yaml title=".hammerkit.yaml" envs: # Cargo keeps downloaded crates in CARGO_HOME. In the project, the fetch # task can keep them as its output. CARGO_HOME: .cargo-home tasks: fetch: description: download the locked crates image: rust:1.90-slim src: [Cargo.toml, Cargo.lock] generates: [.cargo-home] cmds: - cargo fetch --locked ``` ```bash hammerkit fetch ``` Run it again and hammerkit skips it: `Cargo.toml` and `Cargo.lock` are unchanged. Only a dependency change downloads again. `fetch` reads only the manifest and the lockfile, so editing a source file doesn't invalidate it. That's why `Cargo.toml` names its targets: without the sources, Cargo can't discover `src/lib.rs` and `src/main.rs` on its own and refuses to fetch. `--locked` makes Cargo fail instead of quietly updating `Cargo.lock`. ## 2. Test and build [#2-test-and-build] Both tasks depend on `fetch`, which mounts its `.cargo-home` into their containers: ```yaml title=".hammerkit.yaml" tasks: test: description: run the unit tests image: rust:1.90-slim deps: [fetch] src: [src] cmds: - cargo test --offline --locked build: description: compile the release binary image: rust:1.90-slim deps: [fetch] src: [src] generates: - path: bin export: true cmds: - cargo build --release --offline --locked - install -D target/release/greet bin/greet ``` Neither lists `Cargo.toml` or `Cargo.lock`: a task also sees the sources of the tasks it depends on, and a change to them reruns it, so editing the manifest reruns `fetch`, `test` and `build`. `--offline` keeps Cargo from reaching the network, so a crate missing from `fetch`'s output fails the build instead of being downloaded on the side. A container task's outputs stay in a volume hammerkit manages, ready for the tasks that depend on it. `export: true` also copies `bin` into your project. Declaring the whole `target` folder would work too, but `target` holds every intermediate build file; `bin` holds only what you ship. The binary is built for Linux, the container's OS. `test` and `build` don't depend on each other, so they run in parallel. ## 3. One task for CI [#3-one-task-for-ci] A task without `cmds` groups other tasks. `ci` names everything a change has to pass. The complete build file: ```yaml title=".hammerkit.yaml" envs: # Cargo keeps downloaded crates in CARGO_HOME. In the project, the fetch # task can keep them as its output. CARGO_HOME: .cargo-home tasks: fetch: description: download the locked crates image: rust:1.90-slim src: [Cargo.toml, Cargo.lock] generates: [.cargo-home] cmds: - cargo fetch --locked test: description: run the unit tests image: rust:1.90-slim deps: [fetch] src: [src] cmds: - cargo test --offline --locked build: description: compile the release binary image: rust:1.90-slim deps: [fetch] src: [src] generates: - path: bin export: true cmds: - cargo build --release --offline --locked - install -D target/release/greet bin/greet ci: description: everything CI checks deps: [test, build] ``` ```bash hammerkit run ci ``` ```txt Summary: build executed 1.6s ci executed 1.8s fetch executed 944ms test executed 1.8s 4 executed, 0 cached (0% cache hit), 1.8s total ``` Run it again without changing anything: ```txt Summary: build cached 0ms ci cached 15ms fetch skipped 0ms test cached 0ms 0 executed, 3 cached, 1 skipped (100% cache hit), 33ms total ``` `fetch` is skipped: every task that needs it was a cache hit, so the crates aren't needed either. Edit a file in `src` and `test` and `build` run again, against the crates `fetch` already has. Ignore what hammerkit and Cargo write into the project: ```txt title=".gitignore" .hammerkit .cargo-home bin target ``` ## Run it in CI [#run-it-in-ci] CI runs the same command: [install hammerkit](/docs/installation#in-ci) and run `hammerkit run ci`. A fresh runner starts with an empty cache; to reuse what your laptop or the previous run already built, share the cache as described in [caching in CI](/docs/guides/ci-caching). ## Next steps [#next-steps] * Tests that need a database: start it as a [service](/docs/service) and let the task [`need`](/docs/task/needs) it, see [services & networking](/docs/guides/services-networking). * A task reruns and you don't know why: [`hammerkit explain`](/docs/cli/explain). # Why hammerkit (/docs/why-hammerkit) Most projects already have *some* way to run builds: a `Makefile`, npm scripts, a `Taskfile`, or a pile of CI YAML. Hammerkit exists because those tend to drift apart from each other and from what runs in CI. Its job is to be **one build definition that runs the same way everywhere**. ## The five reasons that matter [#the-five-reasons-that-matter] ### 1. One config for local *and* CI [#1-one-config-for-local-and-ci] A hammerkit build file is the single source of truth. The command you run on your laptop — `hammerkit build` — is the exact command CI runs. There is no second pipeline definition to keep in sync, and no "works on my machine / breaks in CI" gap, because it is literally the same build. ### 2. No repeated, tool-specific configs [#2-no-repeated-tool-specific-configs] A typical repo accumulates a `Makefile`, npm scripts, a Dockerfile per step, and a CI workflow that re-encodes all of it. Hammerkit replaces that sprawl: one file declares the tasks, their inputs, their outputs, and the image each runs in. Reducing config duplication is a feature, not a side effect. ### 3. Runs on every platform [#3-runs-on-every-platform] There is no coupling to GitHub Actions, GitLab CI, or any single provider. The same build runs anywhere a container does — your macOS laptop, a colleague's Linux box, a Windows machine with WSL, or any CI runner. Move providers without rewriting your build. ### 4. Containers make builds repeatable [#4-containers-make-builds-repeatable] When a task declares an `image`, its tools come from that image — not from whatever happens to be installed on the machine. That means you don't have to install Node, the .NET SDK, Helm, and the AWS CLI on every developer's laptop and every runner; you just need a container engine. Each task can even use a *different* image, so you're not maintaining one giant "build everything" image. ### 5. Effort is shared through caching [#5-effort-is-shared-through-caching] Hammerkit knows the `src` (inputs) and `generates` (outputs) of every task. So it can: * **skip** a task when its inputs are unchanged, and * **share** the cached result between machines and CI runs through a pluggable cache backend (local directory, an S3-compatible bucket or any OCI registry). The bigger the project, the more this pays off: a `git clone` on a fresh runner can restore outputs another machine already produced instead of rebuilding them. That saves wall-clock time *and* compute cost — see [caches](/docs/build-file/caches) and [task caching](/docs/task/caching). ## Compared to other tools [#compared-to-other-tools] | Capability | Make | npm scripts | Taskfile | Earthly | **hammerkit** | | ---------------------------------------------- | ---------- | ----------- | ---------- | ------- | ----------------------------------- | | Same definition runs locally **and** in CI | ⚠️ manual | ⚠️ manual | ⚠️ manual | ✅ | ✅ | | Runs steps in containers | ❌ | ❌ | ❌ | ✅ | ✅ (per-task `image`) | | Also runs directly on the host (no container) | ✅ | ✅ | ✅ | ❌ | ✅ (omit `image`) | | Tracks declared inputs **and** outputs | ⚠️ targets | ❌ | ⚠️ sources | ✅ | ✅ (`src` / `generates`) | | Skips unchanged work | ⚠️ mtime | ❌ | ✅ checksum | ✅ | ✅ (checksum default) | | Shares cache across machines / CI | ❌ | ❌ | ❌ | ✅ | ✅ (local, S3 and registry backends) | | Runs independent work in parallel | ⚠️ `-j` | ❌ | ✅ | ✅ | ✅ | | Built-in services (e.g. a database) for a task | ❌ | ❌ | ❌ | ⚠️ | ✅ (`needs`) | | Provider-agnostic (no CI lock-in) | ✅ | ✅ | ✅ | ✅ | ✅ | > ⚠️ = possible but only with manual wiring or partial support. ## The principles behind it [#the-principles-behind-it] These are the values hammerkit is designed around: * **Local-first.** A build tool that only runs in CI is broken; hammerkit must be usable on a developer's laptop. * **Platform-agnostic.** No coupling to a single CI provider — the same build runs anywhere a container does. * **One config, many tools.** A single build file replaces N tool-specific configs. * **Caching reduces waste, not just time.** Smart, distributed caching saves compute — that matters for cost and footprint, not only speed. * **Stable contracts.** The CLI and build-file schema are contracts; breaking them is a deliberate, versioned decision. Ready to try it? Head to [getting started](/docs/getting-started), then the [tutorial](/docs/tutorials) for your language.