ci

ci is the HWS continuous-integration service: it plans workflow files from your repository into jobs, queues them on NATS JetStream, runs each job in a fresh heyvm microVM on one of your runner hosts, and serves a dashboard and machine API over the results.

What it is

ci is a single Rust process (ci/) with no agent to install on build machines. A machine becomes a runner by running heyvmd and joining a heyvm network; ci discovers those hosts through the heyvm control plane, opens an authenticated tunnel to each daemon, and drives builds on them.

The workflow format is GitHub Actions-shaped (jobs, needs, if, strategy.matrix, steps with run or uses, ${{ }} expressions), with two differences: a job's uses: names the heyvm network and host it runs on, and a job's vm: block describes the machine to boot.

You trigger runs with git submit, a Git subcommand that sends an exact revision to ci. There is no webhook integration with a Git host.

Architecture

From submit to result

1. Submit. git submit picks a commit that origin already has, computes a binary patch from it to the tree you want built, collects the workflow YAML from that tree, and POSTs a git-patch descriptor to /api/submit. No repository archive is uploaded.
2. Plan. ci selects workflow files whose on: includes submit, applies each file's branch and path filters, expands matrices, resolves each job's network, and stores the expanded job plans and the source descriptor in Postgres in one transaction. Each selected workflow file becomes one run.
3. Queue. Jobs whose needs are satisfied move from pending to queued and are published to JetStream. Postgres commits first; if the publish fails the job returns to pending and the scheduler retries.
4. Claim. A consumer for the job's subject claims the job atomically in Postgres (recording the owning ci process), which starts the job's clocks.
5. Prepare and boot. If the job uses vm.build, the runner's daemon builds or reuses the image. ci then creates a VM of the declared size on the chosen host.
6. Checkout. The guest fetches the pinned base revision from the repository URL, applies the patch, and refuses to continue unless the resulting Git tree matches the submitted tree exactly.
7. Steps. Each step runs through the daemon's exec-operation API as sh -lc. Output is masked for secrets and appended to shared log storage as it arrives; the dashboard streams it live.
8. Cleanup. After the last step ci captures the VM console, records the outcome, and hands the VM to durable cleanup, which stops and deletes it and only then forgets it.
9. Advance. Finishing a job re-runs the scheduler for its run, queuing dependents or skipping them.

Queue subjects

Jobs are sharded so that several ci processes can consume without overlapping:

<prefix>.job.r.<runner_id>     pinned to one host
<prefix>.job.n.<network_id>    any eligible host in that network
<prefix>.evt.<run_id>.<job_key|run>   status events (outbox publisher)

<prefix> is CI_NATS_SUBJECT_PREFIX (default ci). Two installations sharing one NATS server need different prefixes.

A running job extends its acknowledgement every 20 seconds, so ack_wait is 60 seconds regardless of job length and a dead dispatcher releases its job in about a minute. A job whose delivery fails before it is claimed is retried after 60 seconds, 5 minutes, then 15 minutes; the fourth pre-claim failure fails the job. A host's own subject is consumed one job at a time; a network subject fans out to up to four concurrent jobs across runners.

Placement

uses: on a job decides where it runs:

For an unpinned job, ci reads each candidate daemon's GET /capabilities (to skip hosts that cannot run the job's driver) and /storage, and picks the online host with the most free disk. A host must have room for the job's data disk, twice the declared image size, and 5 GiB of headroom; a host whose free space cannot be measured is excluded. Ties go to the lower runner ID.

The network for a job without a network in uses: resolves in this order, and is stamped into the stored job plan at submit time so a redelivery runs where it was scheduled:

1. the job's uses: <network>/…
2. the app-lb workflow object's network
3. the registered repository's assigned network (set on /repos)
4. the first entry of CI_NETWORK, or the account's default network when CI_NETWORK=*

A submit that resolves to a network this ci instance does not serve is refused with the served list in the error.

A pinned job waits for its host. If the host is offline, the job stays queued and fails after CI_RUNNER_WAIT_SECS. Set fallback: any on the job to let it run on any host in the network instead.

VMs are disposable

Treat every job as getting a fresh VM. With vm.reuse false (or after guest filesystem corruption) ci destroys the VM as soon as the job ends. With the default vm.reuse: true it stops the VM and parks it, but the idle sweep runs with a zero-second window and destroys parked VMs on its next tick, so a later job only lands on one if it is claimed in that gap. There is no warm VM pool to plan around. What persists between jobs on a host is the image: a vm.build image is named by the hash of its Dockerfile, build context and size_mb, built once per host, and reused by every later job that names the same inputs. Unused built images are evicted later through heyvmd's ownership-checked eviction route, never while a runner has active work.

A job's data disk (disk_size_gb) is therefore new and empty in every job. Treat it as scratch space, not a cache.

Size classes

vm.size_class takes the heyvm sizes:

Pick on memory. A microVM has no swap, so a build that needs more is OOM-killed rather than slowed down; CARGO_BUILD_JOBS in the job's env trades wall-clock time for memory headroom. ci reads back what the daemon actually allocated. A VM reported smaller than the declared class is resized once, and the job fails with both sizes named if it is still too small. heyvmd admits a VM only if the declared memory of its running VMs fits the host, so large classes are harder to place.

Native runners

A job with runs-on: instead of uses:/vm: goes to a native runner whose advertised labels include every label listed:

jobs:
  mac:
    runs-on: [macos, macos-intel, x86_64-apple-darwin]
    steps:
      - run: cargo build --release

The agent is the native-runner-agent binary built from ci/:

cargo build --locked --release --manifest-path ci/Cargo.toml --bin native-runner-agent

Profiles advertise fixed labels: mac-intel advertises namespace-profile-mac-build, macos, macos-intel, x86_64-apple-darwin; windows-x64 advertises blacksmith-2vcpu-windows-2025, windows, windows-x64, x86_64-pc-windows-msvc.

Native jobs support shell steps, conditions, env, step outputs, working directories, timeouts, continue-on-error, cancellation, ci/upload-artifact, and ci/checkout-release. Other built-in actions fail; put them in a dependent Linux job. Logs arrive with the completion report rather than streaming. Native jobs run your code directly on the host with no sandbox. The /api/native/ routes must be in app-lb public_paths; ci authenticates them itself. Protocol details are in NATIVE_RUNNERS.md.

Running it

Requirements

• heyvmd on each build host, joined to a heyvm network (heyvm network add-host). The daemon must require authentication (JWT_SECRET); ci refuses a runner that answers unauthenticated.
• Postgres. Migrations are compiled into the binary and applied at every startup under an advisory lock.
• NATS with JetStream enabled (nats-server -js -sd /var/lib/ci-js). Do not use the NATS system account; JetStream cannot be enabled on it.
• For image builds (vm.build), each runner host needs docker, mke2fs, and fakeroot when heyvmd does not run as root.

Start

CI_HEYO_API_KEY=… \
CI_NETWORK=prod-runners \
CI_DATABASE_URL=postgres://ci:…@db/ci \
CI_WEBHOOK_SECRET=$(openssl rand -hex 32) \
CI_NATS_URL=nats://127.0.0.1:4222 \
ci

Configuration is environment-only. Every value is validated before ci binds a port or dials NATS; a bad or missing value exits non-zero with a message naming the variable (ci: refusing to start — …). The startup log prints one summary line of every resolved non-secret setting.

For a single machine with no heyvm cloud account, point ci at a local daemon:

CI_LOCAL_RUNNER=1 CI_ALLOW_UNAUTHENTICATED_RUNNERS=true CI_NETWORK=local \
CI_HEYO_API_KEY=unused CI_DATABASE_URL=postgres://…/ci CI_WEBHOOK_SECRET=0123456789abcdef ci

CI_LOCAL_RUNNER=1 uses the SDK's default local daemon URL; any other value is taken as the daemon's base URL. The runner is named local.

A supervisor program template is in ci/deploy/supervisor/ci.conf, and an app-lb static deployment for fronting the dashboard is in ci/deploy/ci.json.

Offline commands

Configuration

Required:

Service:

Runners and timing:

NATS:

Set at most one of the four NATS credential forms; naming two is a startup error. A credential embedded in the URL still works but logs a warning.

Artifacts and reports:

When CI_S3_BUCKET is set, ci also uploads a per-job debug report (job identity, outcome, step logs, VM console) to <prefix>/<run>/<job>/debug-<attempt>-<sandbox>.json before deleting the VM, regardless of the artifact sink. Use a private bucket.

Integrations:

Release, deployment and host-maintenance variables (CI_RELEASE_POLICIES, CI_HOST_APP_LB_TARGETS, CI_HOST_MAINTENANCE_TARGETS, CI_HOST_HEYVM_BOOTSTRAP_TARGETS, CI_CONTROLLER_*, CI_APPLICATION_*, CI_EXPECTED_SHA) are operator configuration for the built-in release actions. They are documented with those actions in the ci README.

Fronting it with app-lb

app-lb's sign-in gate admits browsers only, so the machine routes must be listed in the deployment's public_paths; each authenticates its own requests:

"public_paths": [
  { "path": "/healthz", "scope": "public" },
  { "path": "/api/submit", "scope": "public" },
  { "path": "/api/native/", "scope": "public" },
  { "path": "/api/stream/", "scope": "public" },
  { "path": "/api/runs/", "scope": "public" },
  { "path": "/__ui/", "scope": "public" }
]

Keep /repos, /maintenance, /networks/*/join and the /vms actions behind the gate. ci reads the forwarded identity headers (x-auth-request-user, -email, -name), which are trustworthy only on a gated deployment, and keeps its own admin list seeded from CI_ADMIN_EMAILS. See app-lb auth.

Submitting with git submit

Install and configure

ci/install-git-submit.sh          # installs git-submit into ~/.local/bin

Register the repository on the dashboard's /repos page and mint a token. The page shows the token once, with the two lines to run in your clone:

git config ci.endpoint https://ci.us2.heyo.work
git config ci.token    cis_…

CI_SUBMIT_TIMEOUT (default 300 seconds) bounds the upload. The client needs git, curl, base64, python3, and openssl for the shared-secret path.

Usage

git submit                     # submit HEAD
git submit --dry-run           # show what would be sent
git submit --dirty             # include uncommitted tracked changes
git submit --ref <rev>         # submit another commit
git submit pr59                # submit the exact head of GitHub PR #59
git submit --only app-lb       # run only this workflow (repeatable)
git submit --workflow <id>     # attribute the run to a named workflow object
git submit --submit-empty      # submit even when the tree equals the default branch
git submit --version
git submit --upgrade           # reinstall from the release channel

The client prints one run URL per started workflow, plus a submission URL when a release workflow was selected, and any warnings (for example, a pinned host that is offline).

What gets sent:

• A pinned base and a patch. The client asks origin which heads it advertises, picks the nearest published ancestor of the revision you are submitting, and sends a binary full-index patch from that base to your target tree. A pushed commit is its own base with an empty patch. Local commits and --dirty work as long as some ancestor is published; an unpublished root-only repository cannot be submitted. --archive has been removed.
• Workflow YAML from the target tree (not your worktree): .ci/workflows/*.yml and *.yaml by default, at most 256 KiB per file and 1 MiB total.
• The changed-path list for the full diff from the default branch, which drives paths: filters and changed(). If the diff cannot be computed (root commit, base unavailable, more than 5000 paths), the change set is unknown and every path filter matches.

The runner, not ci, does the checkout: it fetches the base from the repository URL, applies the patch, and verifies the tree. Private repositories therefore need a checkout credential in heyosecret (CI_GIT_AUTH_TOKEN or GITHUB_TOKEN in the workflow's secret scope). The submit token is not repository read access.

pr<number> fetches refs/pull/<number>/head into a temporary object store without touching your branch, index or worktree. It cannot be combined with --ref or --dirty.

--only <selector> runs just the named workflow files. A selector is the file path, its basename with or without extension, or the workflow's name:, matched case-insensitively. A named workflow runs even if its branches/paths filters would have declined, and its run is created with an unknown change set so job-level changed() conditions also pass. A selector that matches nothing fails the submit.

Repository tokens

A repository token is minted per registration, stored as a SHA-256 digest, shown once, and revocable on its own. A submit whose payload names a different repository than its token is refused. Pausing a registration refuses its tokens without deleting them. CI_WEBHOOK_SECRET works for any repository and cannot be revoked per repository; set CI_REQUIRE_REPO_TOKEN=true once every repository has a token.

A registration can also set a workflow glob override and an assigned network.

Workflow objects

With CI_APP_LB_URL set, ci polls app-lb for workflow objects, each pointing at a repository, a ref, a path glob and a network:

heyctl create workflow build \
  --repo git@github.com:me/app.git \
  --network prod-runners \
  --path '.ci/workflows/*.yml'
heyctl get workflows

Objects match a submit by repository URL (git@github.com:me/app.git and https://github.com/me/app are the same). Several objects may name one repository; each produces its own runs. There is no in-place edit; delete and re-create with the same ID. See heyctl.

Workflow reference

name: build
on:
  submit:
    paths: ['api/**', 'Cargo.lock']

jobs:
  build:
    uses: prod-runners                     # any eligible host in this network
    vm:
      driver: firecracker
      build:
        dockerfile: .ci/image/Dockerfile
      size_class: large
      disk_size_gb: 20
    strategy:
      matrix:
        target: [x86_64, aarch64]
    env:
      CI_ENVIRONMENT: prod                 # selects the secret scope
    timeout-minutes: 75
    outputs:
      version: ${{ steps.stamp.outputs.version }}
    steps:
      - id: stamp
        run: echo "version=$(git describe --always)" >> "$CI_OUTPUT"
      - name: Build
        working-directory: /workspace/api
        run: cargo build --release --locked --target ${{ matrix.target }}
        timeout-minutes: 60
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
      - uses: ci/upload-artifact
        with:
          name: api-${{ matrix.target }}
          path: dist

  deploy:
    needs: [build]
    if: ${{ needs.build.result == 'success' && ci.branch == 'main' }}
    steps:
      - run: echo "deploying ${{ needs.build.outputs.version }}"

Every level rejects unknown keys (stpes:, timeout_minutes:) with a parse error naming the job, except inside vm:, where an unknown key is currently ignored. Run ci --check-workflows before you submit.

Top level

on: submit filters

on:
  submit:
    branches: [main, 'release/*']     # or branches-ignore
    paths: ['packages/api/**']        # or paths-ignore

paths builds when any changed path matches; paths-ignore skips only when every changed path matches. branches and branches-ignore cannot be combined, nor can paths and paths-ignore, and a leading ! is refused. An unknown change set matches every filter. A submit where every workflow declines is a success with no runs, and git submit prints each reason.

Job keys

vm:

vm.build runs docker build, docker export, then mke2fs on the runner, so Dockerfile features all work but OCI metadata does not survive: ENV, CMD and ENTRYPOINT are dropped. Put environment a step needs in /etc/profile.d with a RUN (steps run under sh -lc, a login shell). The image must have an /init.sh that prints HEYVM_READY; .ci/image/ci/init.sh is a working example. The first job on a host that needs an image builds it and later jobs wait for that one build.

Steps

Every step gets CI=true, CI_JOB (job ID) and CI_JOB_KEY (matrix-expanded key). Write name=value lines to $CI_OUTPUT to set step outputs. The checkout lands in /workspace and is wiped on every checkout; paths in ci/upload-artifact and cache_key_files are relative to the repository root.

Expressions

${{ }} is substituted in run, env, with, if and job outputs. Contexts:

Functions: contains, startsWith, endsWith, format, join, toJSON, fromJSON, success, failure, cancelled, always, and changed(pattern, …).

changed() uses the same glob matcher as paths: and is true when any changed file matches any argument, or when the change set is unknown. Use it to gate jobs in a monorepo:

jobs:
  api:
    if: ${{ changed('packages/api/**', 'Cargo.lock') }}

An explicit if: replaces the default "all dependencies succeeded" rule, which is how if: ${{ always() }} runs a cleanup job after a failure.

Secrets and variables

ci resolves secrets once per job from heyosecret under the prefix:

ci/<workflow>/<environment>/<NAME>

• <workflow> is the app-lb workflow object's ID when one matched; otherwise the registered repository's name (by default the last segment of its URL); otherwise the submitted repository's directory name. It is exposed as ${{ ci.workflow }}.
• <environment> is the job's env.CI_ENVIRONMENT, defaulting to default.
• Characters outside [A-Za-z0-9-_.:@] are replaced with -.

An entry tagged public in heyosecret becomes vars.<NAME> and is shown in plain text. Everything else becomes secrets.<NAME> and is masked (***) in logs before they are stored or streamed; values shorter than four characters are not masked. The heyosecret token itself never reaches a build. The --secrets-prefix field on a workflow object is stored but not currently applied.

Store the checkout credential for a private repository as CI_GIT_AUTH_TOKEN (or GITHUB_TOKEN) in the same scope.

Artifacts

Uploading

- uses: ci/upload-artifact
  with:
    name: app                  # required
    path: dist                 # required, relative to the repository root
    description: App binary for x86_64-linux   # artifacts sink only
    public: true               # artifacts sink only
    alias: app-live            # artifacts sink only

The step packs path into a tar.gz in the guest. With the artifacts sink, the guest pushes the tarball straight to the store with curl -T (using CI_ARTIFACT_GUEST_URL or CI_ARTIFACT_URL); if that fails, ci reads it out through the exec channel in chunks, which is much slower on Firecracker. Both directions are checked by SHA-256.

In the artifacts store each upload is tagged:

ci-<workflow>-<run>-<job_key>-<name>

truncated to 64 characters. Run IDs are fixed-width hex (%012x-%08x, time then sequence), so sorting these tags lexicographically sorts them by time.

• description is stored as a label beside the digest, shown in the store and by installers' --list.
• public: true marks the blob public (PUT /public/{digest}), so {store}/blobs/{digest} downloads with no credential. The link is printed in the step log as [ci] public link: … and shown on the run page. Tags and listings still need the key.
• alias moves a second, stable tag onto this upload, so a deployment can follow the newest green build by name. Aliases cannot start with ci-, and a failure to set one fails the step.

Each upload writes a ci.artifact.published.v1 event. A published artifact is not a release approval: a later step can still fail.

Downloading in a later job

jobs:
  package:
    needs: [build]
    steps:
      - uses: ci/download-artifact
        with:
          name: app
          path: app.tar.gz        # the uploaded tar.gz, not unpacked
          job: build              # only when several producers used this name
      - run: tar -xzf app.tar.gz

The producer must be in needs and must have succeeded. Size and SHA-256 are verified. Cross-job files must go through artifacts; each job's VM is deleted when it finishes. In a coordinated release, with.workflow names a validation workflow path from the same submission.

Dashboard and HTTP API

Pages

A no consumer flag on an online host in the Queue column means jobs are routed to a subject nothing reads: the host is not in a network this instance serves, or its consumer is not bound.

Cancelling marks the run and all unfinished jobs cancelled. A running step cannot be aborted in the guest; ci stops waiting within seconds, records it cancelled, and cleans up the VM while the command finishes or times out. A re-run is always a new run (rerun_of links them). Re-run failed jobs carries successful jobs and their outputs over and schedules the rest.

Machine API

These routes authenticate with the repository token (Authorization: Bearer cis_…) scoped to the run's repository, or with an HMAC of the request path signed by CI_WEBHOOK_SECRET. A run from another repository answers 404.

The /api/native/* routes serve native runners (bearer CI_NATIVE_RUNNER_SECRET), and /api/lifecycle/* serves managed-platform lifecycle calls (bearer CI_APPLICATION_LIFECYCLE_TOKEN).

curl -fsS -H "Authorization: Bearer $(git config ci.token)" \
  "https://ci.us2.heyo.work/api/runs/$RUN/logs?failed_only=true&tail=4096"

Multiple instances, drain and maintenance

Several ci processes can serve the same installation, including one per region, as long as they share the same Postgres database and the same NATS server and subject prefix. Every registered process may execute work; there is no leader. A job is claimed atomically by one process, which owns its VM and cleanup until the cleanup is confirmed. If a process dies, its unclaimed queue messages are redelivered to another; work it had already claimed is not taken over automatically and needs reconciliation.

Maintenance routes are admin-only (behind the app-lb gate, never in public_paths), and each request applies only to the process that receives it. Send x-ci-target-boot: <boot id> to have any other process reject the request with 409. ci --inspect-executor lists boot IDs.

Draining one ci process

POST /maintenance/{operation-uuid}/pause      stop admitting new jobs on this process
GET  /maintenance                             this process's outstanding work
POST /maintenance/{operation-uuid}/quiesce    succeeds once its jobs and cleanups are done
POST /maintenance/{operation-uuid}/resume     reopen admission for the same operation

Unclaimed deliveries go back to NATS for the other process to take. Running jobs finish and clean up normally.

Draining a runner host

Pausing a ci process does not stop the other process from placing jobs on a host. To empty a host, use the runner drain on either process:

POST /maintenance/runners/{runner-id}/{operation-uuid}/pause
GET  /maintenance/runners/{runner-id}          running jobs, host work, cleanup, drained
POST /maintenance/runners/{runner-id}/{operation-uuid}/resume

Use the canonical runner ID, not its display name. The drain is stored in Postgres and survives restarts. Jobs already running finish on that host. New and queued unpinned jobs go to another healthy host in the same network; jobs pinned with uses: <network>/<host> (without fallback: any) and jobs targeting an existing VM stay pinned and wait. The host is drained when admission is closed and running jobs, outstanding host work and cleanups are all zero. Every ci process sharing the host must run a version that implements runner drain.

Releases and deployments

A repository may define one workflow with on: release. On a full git submit, ci creates the selected validation runs and a release coordinator run together; the coordinator waits until every selected validation succeeds and then runs its jobs. --only, reruns and partial selections never run the coordinator. The coordinator must start with a merge job containing only ci/merge-release, and every deployment job must depend on it.

Built-in release and deployment actions:

Sequence regional stages with needs so a failure in one region stops the next. Deployment actions record an operation ID before calling the remote API and reconcile that same operation after a restart or lost response rather than submitting another. Their inputs, the operator-owned target mappings they require, and recovery endpoints are specified in the ci README. For the platform-level view of regional rollouts, see multi-region and orchestrator.

This repository's release pipeline

Each installable component has one workflow file in .ci/workflows/, with one job named release that builds, tests, and uploads a tarball with ci/upload-artifact:

Every artifact also carries REVISION or BUILD-INFO and a SHA256SUMS file. Each workflow's paths: filter lists its crate, its path dependencies, its own workflow file and its image directory under .ci/image/, so a commit only rebuilds what it touches. regional-release.yml is the on: release coordinator that merges and then rolls validated artifacts out to us3, then eu1, then replaces the ci controller.

Publishing for installation is a separate step run by someone with the store's API key:

1. The workflows upload to the artifacts store under ci-<workflow>-<run>-release-<name> tags.
2. .ci/publish-releases.sh resolves each app's newest tag to a blob digest, marks that blob public, and writes one <app>.json manifest per app plus index.json, install-apps.sh and bootstrap-host.sh.
3. --push-tag releases-live uploads that directory as one public bundle and moves the tag; the releases site deployment serves it at get.us2.heyo.work.

ART_API_KEY=… sh .ci/publish-releases.sh --all \
  --from-url https://get.us2.heyo.work --push-tag releases-live
heyctl pull releases

The installer then downloads public blobs with no credential and verifies the digest and SHA256SUMS. To install the binaries, see installation.

Troubleshooting

For offline validation of every workflow file:

ci --check-workflows .ci/workflows/*.yml

This checks parsing and planning only, not that the build commands work.