Artifacts

The artifacts store (art) is a content-addressed blob store built for ext4: it holds VM images, site bundles, Dockerfile build inputs, and workspace snapshots, and serves them to app-lb and heyvm over the CLI or HTTP.

What it is

A blob's only name is the sha256 of its bytes. Tags, manifests, and labels all point at digests. The project ships as:

• a Rust library (artifacts),
• the art CLI, which owns every destructive operation (rm, untag, gc, heyvm sparsify),
• the art serve HTTP daemon, which exposes content (upload, download, tags, manifests, labels) but never lifecycle.

How it fits into HWS:

• heyvm base images. art heyvm import takes heyvm's Firecracker images into the store, and art heyvm materialize produces a writable rootfs from one while copying only live data.
• app-lb deployments. A deployment's artifact block pulls a rootfs or site bundle from a store; its build.store points at a Dockerfile manifest to build; its vm.workspace block snapshots a workspace into a store. See app-lb.
• heyctl. heyctl artifact (alias heyctl art) logs in to a store and pushes images and Dockerfiles. See heyctl.
• CI. The CI service stores build artifacts in a store. See ci.

The store is Linux-only. It depends on O_TMPFILE, FALLOC_FL_PUNCH_HOLE, copy_file_range, and statx, and fails to build elsewhere.

How it uses ext4

• Hardlinks are references. A read-only materialization is a hardlink, so it costs no bytes or time. The blob's st_nlink is the reference count, maintained by the kernel; the store keeps no counter and needs no crash recovery.
• Sparse storage. ext4 has no reflink, and heyvm images are fully allocated on disk but mostly zeros inside. --squash scans for zero runs and punches holes, so a 20 GiB image with ~600 MiB of real data takes ~600 MiB on disk. The digest still covers the full logical content, so it matches sha256sum of the original file.
• Atomic, deduplicating inserts. Bytes are written to an unnamed O_TMPFILE, synced, and linked into place under their digest. If the name already exists, that is a deduplication hit.
• Free-space guard. Writes that would leave less than ART_MIN_FREE_BYTES free on the filesystem are refused (exit code 3, HTTP 507).
• Immutable blobs. Blobs are mode 0444. A writable materialization is always a private copy.

The store must be owned by the user who reads it. With fs.protected_hardlinks=1 (the Linux default), only the owner can hardlink a 0444 file; a shared store silently falls back to copying for everyone else.

Concepts

A ref is a tag or a digest. When you ask for a blob by a tag that names a manifest, the store steps through it: a manifest with one entry resolves to that entry; a manifest with several is an error that lists them (except art heyvm materialize, which picks the rootfs.ext4 entry).

Manifest kinds used today:

Garbage collection marks every blob reachable from a tag (directly or through a manifest), then deletes blobs that are unreachable, have st_nlink == 1 (no outstanding materializations), and are older than the minimum age. An untagged manifest does not keep its blobs alive. Labels and public markers for deleted content are removed in the same pass.

Install

cargo build --release --locked --manifest-path artifacts/Cargo.toml
sudo install -m0755 artifacts/target/release/art /usr/local/bin/art
art init                    # creates the store under $ART_ROOT or ~/.artifacts

The HTTP daemon is behind the default-on daemon cargo feature and the S3 client behind the default-on s3 feature. Build with --no-default-features for the library alone, or --no-default-features --features daemon for a daemon with no TLS stack (the remote tier then works only against ART_REMOTE_DIR).

Configuration

Every value resolves flag, then environment variable, then default.

Store (all commands)

Daemon (art serve)

ART_ADMIN_PASSWORD, ART_DASHBOARD_OPEN, and ART_DASHBOARD_GATE are mutually exclusive; setting two of them is a startup error. With none set, the dashboard is not mounted at all.

Global store (daemon and art s3, art repo)

Set ART_S3_BUCKET and the daemon becomes a regional cache in front of the bucket. See Global store.

CLI reference

Global flags: --root, --min-free-bytes, --json (machine-readable output on every command).

Exit codes: 0 success, 1 failure, 2 bad usage (invalid digest or tag), 3 out of space.

Content

Names and access

Global store

Removal

Dockerfiles

A Dockerfile manifest is a build input: the recipe plus the files it copies. The store doesn't build it; app-lb fetches it and runs heyvm mvm build.

art dockerfile put ./Dockerfile --context ./app --tag web-rootfs \
    --image-name web --size-mb 4096

The context is packed deterministically into one blob, so re-pushing an unchanged tree uploads nothing. Nothing is excluded: there is no .dockerignore handling. Point --context at a clean directory, or pack the archive yourself.

heyvm integration

Typical flow on a heyvm host:

art heyvm sparsify            # reclaim space in the existing images first
art heyvm import --all        # take them into the store
art heyvm materialize debian-hermes /tmp/rootfs.ext4
art gc

Run sparsify before import on a nearly full disk: the images are fully allocated until they are sparsified. sparsify refuses to punch holes in a file with more than one link, because that would change a running VM's disk.

HTTP API (art serve)

Uploads and downloads stream, so large images never sit in memory. Small request bodies are capped at 2 MiB; blob uploads are bounded only by the free-space guard.

Blob responses carry ETag (the digest), Cache-Control: public, max-age=31536000, immutable, and x-art-allocated (bytes actually on disk).

Namespaced references work raw or percent-encoded: /tags/heyo/postgres:16 and /tags/heyo%2Fpostgres%3A16 are the same tag.

Error statuses: 400 bad digest, tag, or label; 401 bad or missing key; 403 read-only mode; 404 not found; 409 digest mismatch, ambiguous manifest, or a manifest/tag naming content the global store does not hold; 412 a lost If-Match; 503 the global store is unreachable (writes only); 507 out of space.

There is no gc and no materialize route. Garbage collection stays in the CLI, and a remote materialization is just GET /blobs/{digest}.

Authentication

With ART_API_KEY set, every route except /healthz requires the key as Authorization: Bearer <key> or X-Api-Key: <key>. The comparison is constant-time.

Anonymous requests (no credential at all) may GET/HEAD exactly what a public repository needs to be pulled:

• /tags/{tag} and /manifests/{tag} for a tag in a public repository;
• /manifests/{digest} and /blobs/{digest} for anything such a tag reaches;
• /blobs/{digest} for a blob marked public on its own;
• /repos (filtered to public repositories) and /repos/{repo} for a public one.

Everything else — other listings, private repositories, every write — still needs the key. A request that presents a wrong key is rejected even for something public. Only key holders can make a repository public.

art public web-bundle                    # or: curl -XPUT -H "x-api-key: $KEY" $URL/public/web-bundle
curl -O https://art.example.com/blobs/<digest>   # no credential needed

Example: push and tag over HTTP

D=$(sha256sum rootfs.ext4 | cut -d' ' -f1)
curl -T rootfs.ext4 -H "x-api-key: $KEY" "$URL/blobs/$D"
curl -XPUT -H "x-api-key: $KEY" --data "$D" "$URL/tags/web-v2"

heyctl artifact push does this for you and stores the key in a saved registry.

Global store

With ART_S3_BUCKET set, the bucket is the system of record and each regional art serve is a cache in front of it. Every region sees the same tags, and the bucket alone is enough to rebuild any of them — it is the backup.

• Writes go to the bucket first. PUT /blobs answers only once the blob is in the bucket. A manifest is refused (409) unless the bucket holds every blob it names; a tag is refused unless the bucket holds what it points at. The bucket never holds a pointer to nothing.
• Blobs and manifests read through. A miss is fetched, checked against its digest and kept. Concurrent misses on one blob share one download. A corrupt object is never cached.
• Tags revalidate. A tag older than ART_TAG_TTL is checked with If-None-Match, so a tag moved in us3 is seen in eu1 within the TTL. If the bucket is unreachable, the cached tag is served and the failure logged: an image a region already holds keeps booting during an S3 outage. Writes fail with 503.
• Mutable state is mirrored. Every ART_SYNC_INTERVAL the daemon lists the bucket's tags, labels, public markers and repositories and updates its local copies, so listings, the dashboard and the hub show the global state. The mirror only removes local objects it copied from the bucket; a tag that exists only in one region is left alone and logged until art s3 backfill publishes it.
• The disk is a cache. Past ART_CACHE_MAX_BYTES, or below ART_CACHE_MIN_FREE_BYTES free, blobs the bucket holds and nothing has materialized are evicted, least recently used first. A blob only on this disk is never evicted.
• Garbage collection is art s3 gc, never the daemon. It holds a lease object (locks/gc) taken with a conditional write, so two regions cannot sweep at once, and keeps anything younger than --min-age (default 24h) so another region's half-finished push survives. art gc on a regional store only trims its cache.

The daemon refuses to start unless the bucket honours conditional writes (If-None-Match: *): tag compare-and-swap and the GC lease depend on it. AWS S3, R2 and MinIO all do.

Bucket requirements: versioning on (tag history and undelete for free) and a lifecycle rule that aborts incomplete multipart uploads after one day.

Bucket layout

<prefix>blobs/<aa>/<64-hex>.asp   the blob, artsparse-encoded
<prefix>manifests/<aa>/<64-hex>   canonical JSON, byte-identical to a local store's
<prefix>labels/<aa>/<64-hex>      label JSON
<prefix>public/<aa>/<64-hex>      empty marker: blob downloads anonymously
<prefix>tags/<name>               "<digest>\n" (a namespaced tag keeps its '/')
<prefix>repos/<repo>.json         {"public", "description", "updated"}
<prefix>locks/gc                  the GC lease while a sweep runs

artsparse v1 keeps a sparse rootfs small in the bucket (a 64 MiB image with 12 MiB of data is a 12 MiB object): the four bytes ASP1, a little-endian u32 header length, a JSON header {"size": <logical length>, "extents": [[offset, length], ...]}, then each extent's bytes in order. Everything outside the extents is zeros. The blob's digest covers the full logical stream, holes included, so a reader proves what it rebuilt by re-hashing.

Moving an existing store into the bucket

For a store already serving traffic, follow the migration runbook: its data must reach the bucket from a copy of the running VM's disk before anything replaces the VM. The core of it:

art s3 backfill --dry-run     # what would be uploaded, and any tag conflicts
art s3 backfill               # safe to re-run; tags go last
art s3 verify                 # every tag reaches content the bucket holds

Turning on ART_S3_BUCKET before backfilling is safe: local-only tags keep resolving and are reported until published, and a write that names content only this store holds — a push whose blob the cache already had, a tag on an old manifest — publishes that content to the bucket first.

Public hub

ART_HUB=1 serves a catalog of public repositories at /hub, readable by anyone: each repository's tags, sizes, entries, and copyable pull commands. It shows nothing anonymous pulls do not already allow, and a private repository is a 404, not a 403.

Publish (key holders only):

heyctl artifact push postgres.ext4 --tag heyo/postgres:18 --public
# or, for an existing tag:
art repo public heyo/postgres
art repo describe heyo/postgres "PostgreSQL 18 on Debian, for heyvm"

Pull (anyone):

heyctl artifact pull hub.heyo.work/heyo/postgres:18 --dest ./postgres.ext4
curl -fsSL https://hub.heyo.work/manifests/heyo/postgres:18

An app-lb deployment pulls with no auth:

"artifact": { "store": "https://hub.heyo.work", "ref": "heyo/postgres:18" }

The heyo/* images on the hub (alpine, debian, ubuntu, postgres) are built from images/.

Dashboard

A read-only, server-rendered dashboard (no JavaScript, no external assets) lives at /dashboard: an overview with capacity and usage, a blob list comparing stored and logical size, a manifest list, and detail pages showing which tags and manifests reference a blob. Deleting, tagging, and GC stay in the CLI.

The dashboard credentials are separate from ART_API_KEY. The API key does not open the dashboard, and the dashboard login does not open the API.

Running as a microVM behind app-lb

artifacts/Dockerfile and init.sh package the daemon as a heyvm image. Build from the repository root, because the build needs the shared ui/ directory:

heyvm mvm build --local-only -f artifacts/Dockerfile -n artifacts --size-mb 768
art heyvm sparsify artifacts

In an app-lb build block this is "dockerfile": "artifacts/Dockerfile" with "context": ".". The image build runs a smoke test that art serve starts and answers /healthz.

init.sh brings up networking, mounts /dev/vdb at /workspace, starts sshd, and prints HEYVM_READY. It does not start the daemon; start_command does, so that env_vars (including ART_API_KEY) reach it.

Put ART_ROOT inside /workspace. The rootfs is recopied from the base image on every cold boot, so a store on the rootfs loses every blob on restart.

Example deployments:

Keep a store deployment at exactly one replica (min_replicas: 1, max_replicas: 1). Each VM has its own disk, so multiple replicas are independent stores and round-robin would return 200 or 404 depending on which one answered.

How app-lb uses a store

app-lb addresses a store in one of two forms, and picks the transport from the spelling:

A local store is much faster (holes are skipped, bundles are hardlinked); a URL is what lets one store serve many hosts.

Pulling a rootfs or site (artifact)

"artifact": {
  "store": "http://127.0.0.1:8080",
  "ref": "web-v2",
  "grow_gb": 4,
  "auth": { "secret": "art", "key": "api_key" }
}

The digest is verified before the bytes are used, and the image file is named after it, so a re-pull of content already on disk is skipped. Trigger a pull with heyctl pull <deployment> [--ref REF] [--wait].

Building from a Dockerfile manifest (build.store)

A deployment's build.store (mutually exclusive with build.repo) names a store holding a heyvm.dockerfile.v1 manifest. Push one with art dockerfile put or heyctl artifact push-dockerfile, then point the deployment at it with heyctl set build --store <url> --ref <tag>.

Workspace snapshots (vm.workspace)

A deployment's vm.workspace block keeps a writable directory across VM replacement. When a replica retires, app-lb captures the workspace, stores it as a tar.gz blob, and moves a tag to it; the next replica is seeded from that snapshot.

On the app-lb host the working copies live under /var/lib/app-lb/workspaces/<deployment>/. app-lb refuses to move the tag if the store holds a snapshot this host has never seen, which keeps history linear. Use a dedicated tag per deployment; never share a workspace tag between two deployments. See app-lb for the full workspace lifecycle.

On-disk layout

$ART_ROOT/
  .store.lock              flock: shared for a commit, exclusive for gc
  blobs/<aa>/<64-hex>      mode 0444, immutable
  manifests/<aa>/<64-hex>  canonical JSON, addressed by its own sha256
  labels/<aa>/<64-hex>     name and description for a blob or manifest
  tags/<name>              one digest and a newline; a namespaced tag's '/' is '~'
  repos/<repo>.json        repository metadata; '/' is '~'
  remote-state.json        ETags of what the mirror copied from the global store
  tmp/                     incoming files on the fallback insert path

<aa> is the first two hex characters of the digest.

Common operations

art usage                              # how much space the store is saving
art ls --tags
art stat debian-hermes                 # links > 1 means VMs are using it
art verify --all                       # re-hash everything
art gc --dry-run                       # see what would be removed
art gc --min-age 2h
curl -s -H "x-api-key: $KEY" "$URL/usage"

Troubleshooting