heyctl

heyctl is a kubectl-style command-line client, and a Rust client library, for the app-lb admin API: deployments, VM pools, secrets, tokens, auth providers, builds, pulls, artifact stores, and the telemetry the obs plugin collects. The crate is published on crates.io as hws, the Heyo Web Services SDK; the binary it installs is heyctl.

What it is

app-lb is the HWS load balancer and autoscaler. Everything it does is driven through its admin API (default http://127.0.0.1:9090), and heyctl is the CLI for that API. Its verbs follow kubectl's: you apply declarative specs, use imperative helpers (create, scale, set) that write those specs for you, and read them back with get, describe and top.

It is a separate crate from app-lb (app-lb/heyctl), so installing it does not pull in pingora, openssl or the ACME stack. It shares only the wire format with the server. The same crate is also a library: build it with default-features = false to get the typed client without clap or a terminal.

A deployment is one of three kinds, and many commands only apply to one of them:

When the server rejects a command for a deployment kind, heyctl passes the server's reason through.

Install

Installer

curl -fsSL https://heyo.computer/heyctl/install.sh | sh

The script is app-lb/heyctl/install.sh. It reads a version manifest (<site>/heyctl/versions.json), downloads the matching blob anonymously from the artifact store, verifies it against its sha256 digest and against the SHA256SUMS inside the tarball, and installs heyctl into ~/.local/bin.

Pass flags through the pipe with sh -s --. Without the -s --, sh reads the flags as its own:

curl -fsSL https://heyo.computer/heyctl/install.sh | sh -s -- --prefix /usr/local
curl -fsSL https://heyo.computer/heyctl/install.sh | sh -s -- --list
curl -fsSL https://heyo.computer/heyctl/install.sh | HEYCTL_VERSION=0.2.0 sh

The installer never sends an Authorization header. The artifact store serves public blobs only to anonymous requests, so an ART_API_KEY in your environment is ignored on purpose. Redirects may not downgrade from HTTPS to HTTP.

It detects linux/darwin and x86_64/aarch64, and tells you which platforms the manifest actually offers if yours is missing.

From source

heyctl is the hws package in the app-lb Cargo workspace:

cargo install hws                    # from crates.io
# or, from this repository:
cd app-lb
cargo build --release -p hws
install -m 0755 target/release/heyctl ~/.local/bin/

As a library (the Heyo Web Services SDK):

[dependencies]
hws = { version = "0.2", default-features = false }

The library is async. The blocking feature adds hws::blocking::Client, which is what the CLI uses. The crate README has a quick start that creates a workload, reads its telemetry and rolls it out with a namespace token; examples/namespace_workload.rs is the same program. cargo doc -p hws --no-default-features --open builds the API docs; the changelog lists what changed between releases.

Connecting

With no config file and no flags, heyctl talks to http://127.0.0.1:9090, app-lb's default admin listener. A local app-lb needs no setup.

The admin listener is plaintext HTTP on loopback by default. To reach a remote one, either tunnel it:

ssh -L 9090:127.0.0.1:9090 lb-host
heyctl --server 127.0.0.1:9090 get deployments

or front the admin listener with an app-lb TLS deployment (see examples/app-lb-admin.json) and log in to its HTTPS name. --insecure-skip-tls-verify accepts a self-signed certificate on an endpoint you control.

Do not point a context at a hostname behind a Google sign-in gate. heyctl cannot complete an OAuth flow, so every command fails with a 401 whatever credentials you store. Tunnel to the admin listener instead, or see Putting the dashboard behind Google.

Credentials, contexts and the config file

The config file

Contexts (server plus credentials) and artifact-store registries live in one JSON file:

The file is written mode 0600 and its directory 0700. It holds passwords, tokens and API keys in plaintext. Don't print it, paste it, or commit it. heyctl config view redacts secrets unless you pass --show-secrets; heyctl config path prints the location. An empty file is treated as no file.

To keep credentials out of the file, store a command instead of the value (--password-command, --token-command, --api-key-command), or verify without storing (--no-store-password, --no-store-key) and supply the value through the environment.

How app-lb authenticates heyctl

app-lb accepts three credentials on its admin API. See app-lb auth for the server side.

A namespace-scoped Heyo API key is used against Cloud's namespace door, https://<cloud>/namespaces/<ns>/lb, and reaches only that namespace at the tier it was minted with (view or admin).

Heyo account login and regional visibility

For a platform administrator account, use --email, not --user (gateway Basic auth):

heyctl login --server https://admin.heyo.work --email sam@heyo.computer
heyctl get deployments --fleet
heyctl get deployments --fleet --namespace default -o json

Login prompts for the account password and uses the same HTTPS /login exchange as the dashboard. It saves only the expiring session token in the existing permission-restricted config. Run login again when it expires; no password is retained for unattended refresh. Redirects and insecure TLS are not allowed.

--fleet reads the gateway's configured fleet, showing each deployment's gateway, region, health and unavailable-region errors. JSON/YAML preserves the server's full response, including truncation and error fields. It does not discover servers from DNS, change write targets, or fail over the selected admin endpoint. Without --fleet, existing reads and all deployment writes remain regional. Use an explicit regional context for host maintenance; fleet observation does not turn apply or restart into a coordinated release.

Precedence

For each of the password, token and artifact API key, heyctl uses the first of:

1. the flag or env var (--password/HEYCTL_PASSWORD, --token/HEYCTL_TOKEN, --api-key/HEYCTL_ART_API_KEY)
2. the context's stored *_command, run through sh -c on each request
3. the context's stored value

A token outranks a user/password on the same context. The username defaults to admin; login never prompts for it, so pass --user if the server sets APP_LB_DASHBOARD_USER to something else. A wrong username gives the same 401 as a wrong password.

Global options

These work on every command, before or after the subcommand.

-o json and -o yaml print the server's payload unmodified, so they round-trip into apply. -o name prints deployment/<id> lines for xargs.

Scripts and CI can skip the config file entirely:

HEYCTL_SERVER=https://cloud.example.com/namespaces/team-a/lb \
HEYCTL_TOKEN="$HEYO_KEY" heyctl get deployments

login, logout, whoami

heyctl login --server 127.0.0.1:9090                       # prompts for a password if the server wants one
heyctl login --server https://lb-admin.example.com --user admin
heyctl login --server lb.example.com:9090 --password-command 'pass show app-lb/admin'
heyctl login --server lb.example.com:9090 --no-store-password   # then export HEYCTL_PASSWORD
heyctl login --server https://cloud.example.com/namespaces/team-a/lb --token-stdin <<< "$HEYO_KEY"
heyctl whoami
heyctl logout --keep-context                               # drop the stored password only

login probes the server, verifies the credentials against whatever is actually gated, and saves a context (named after the server's host unless you pass --name). A token login verifies with GET /deployments, since Cloud's namespace door has no /healthz to probe.

whoami prints the config file, context, server, user, where the password comes from, whether the server is reachable, which routes require auth, and whether the deployment API and /metrics are allowed for this identity. It is the first thing to run when you get a 401. It also reports the current artifact registry.

config

--context NAME on any command uses a context once without switching.

Command reference

Resource names take kubectl's forms: deployments, deployment web, deployment/web, deploy web, or a bare web where the kind is unambiguous. Kinds and aliases:

Reading

heyctl get deployments -o wide          # adds MIN MAX WARM TARGET BACKEND SOURCE AUTH
heyctl get deployment/web -o yaml
heyctl get vms -d web
heyctl get secrets -n team-a            # ids and key names, never values
heyctl get jobs -d web                  # builds, pulls, updates and mount pulls, newest first
heyctl get job job-3f2a1c8e             # one job, with its log
heyctl get disks -d sb-7f3a9c
heyctl get auth-providers -n team-a
heyctl describe deployment web
heyctl top vms -w

Creating and applying

create deployment flags, by group:

--path-prefix is forwarded unchanged; app-lb does not strip it. --disk-gb is the only guest storage that survives a VM stop, because the root filesystem is recopied from the image on every boot.

Secret values passed as KEY=VALUE arguments are visible in ps and shell history. Prefer --from-file, --from-env or --from-stdin.

Editing in place

Every set command is a read-modify-write of the whole spec (PUT /deployments/:id). heyctl edits the server's JSON rather than its own struct, so fields it does not know about survive. All set commands take --dry-run.

Passing any --command, --env, --secret-env, --allow-domain, --allow-email or --public-path replaces that whole list. Use heyctl edit for incremental changes.

set auth --public-path writes a bare-string entry, which app-lb reads as scope admin: the path skips Google sign-in but requires an admin-tier app-token. To make a path fully open (a health check, a webhook), write {"path": "/healthz", "scope": "public"} with heyctl edit or apply. See public paths.

set build and set artifact are mutually exclusive on one deployment, because both rewrite vm.image.

Jobs: build, pull, update, mounts

These start an asynchronous job on the app-lb host. Without --wait they return the job id straight away; follow it with heyctl get job <id>. One job runs per deployment at a time; a second is refused, not queued. A failed job makes heyctl exit non-zero after printing the tail of the log.

pull --ref <digest> pins exact bytes without changing the stored spec, which is how you roll back. A tag follows wherever it is moved. apply and edit start a mount pull automatically when a mount has no tree on the host.

An update whose commands succeed but whose upstreams never come back is reported as a failure. The commands run as app-lb's user.

Scaling and lifecycle

--replicas N pins min = max = N. Give --min/--max again to hand control back to the autoscaler.

--idle-action retain stops an idle VM instead of destroying it, and a later request or exec resumes it. Only the /workspace data disk (--disk-gb) persists across that stop.

Cordon state is durable across app-lb restarts. A drain is refused when no other healthy upstream would remain, unless you pass --force. On timeout the upstream stays cordoned.

delete vm recycles a VM: the autoscaler replaces it if the policy still wants the capacity. Use scale to shrink. delete secret is refused while a deployment references the secret; --force deletes it anyway. Certificates, jobs and disks cannot be deleted with delete.

Getting inside a VM

Both go through app-lb, not the heyvm daemon, so they work wherever the admin API does. Both start a VM if the deployment has none running (up to its cold_start_timeout_secs), unless you pass --no-wake. An open shell counts as in-flight work, so the pool will not scale to zero under it. These are the only way into a deployment created with --no-route.

Tokens

App-tokens are app-lb's own scoped, revocable bearer credentials. See app-lb auth for the scope model.

mint writes the secret to stdout and everything else to stderr, so capturing it is safe:

APP_LB_TOKEN=$(heyctl token mint ci --admin admin --all-deployments -q)

A token scoped to specific deployments cannot mint tokens, so it cannot widen itself.

Plugins

enable/disable are the operator's fleet-wide switch. Per-namespace plugins (obs) also have to be installed in a namespace before they do anything there. Where -n is optional, it defaults to the namespace the token is confined to.

Telemetry

Both read app-obs through app-lb's obs plugin, so they need it installed in the namespace (heyctl plugins install obs -n NS) and nothing beyond a namespace token. top without -n still shows the LB's live counters.

Artifact stores

An artifact store (art serve) is a separate service from app-lb, so heyctl artifact (aliases art, registry) keeps its own saved registries in the same config file. --context never retargets an artifact command.

push --image NAME resolves ~/.heyo/images/firecracker/<name>.ext4 (or under $MVM_DATA_DIR), where heyvm mvm build writes images. The tag defaults to the file name without .ext4. push-dockerfile's tag defaults to the Dockerfile's directory name. The build context is packed as-is, with no .dockerignore handling, so point it at a clean directory.

Shell completion

heyctl completion bash > /etc/bash_completion.d/heyctl
heyctl completion zsh  > ~/.zfunc/_heyctl

fish, elvish and powershell are also supported.

Common workflows

Deploy a spec

# web.yaml
id: web
routes: [{ host: web.example.com }]
vm:
  driver: firecracker
  image: nginx-fc
  port: 80
  size_class: mini
scaling: { min_replicas: 1, max_replicas: 4 }
health: { path: /healthz }

heyctl apply -f web.yaml --dry-run   # print what would be sent
heyctl apply -f web.yaml
heyctl rollout status web
heyctl describe deployment web

For field names, generate a spec with heyctl create deployment ... --dry-run or read an existing one with heyctl get deployment web -o yaml. Copying between load balancers is a pipe:

heyctl get deployment web -o json | heyctl --context staging apply -f -

The imperative equivalent:

heyctl create deployment web --host web.example.com --image nginx-fc --port 80 \
  --size mini --min 1 --max 4 --health-path /healthz

Build from a Dockerfile

heyctl create secret github --from-stdin token < ~/.github-pat
heyctl set build web --repo https://github.com/acme/web.git --ref main --secret github
heyctl build web --logs

Each build produces an image named <deployment>-<short sha>, so describe and get -o wide show which commit is running.

Push and pull an image

Build the image once, push it to a store, and let each app-lb host pull it:

heyctl artifact login https://art.us2.heyo.work --api-key-stdin < ~/.art-key
heyctl artifact push --image web-v2

heyctl create secret art --from-stdin api_key < ~/.art-key
heyctl set artifact web --store https://art.us2.heyo.work --ref web-v2 --secret art/api_key
heyctl pull web --wait
heyctl get jobs -d web

A store root on the app-lb host (--store /srv/artifacts) is much cheaper than a URL: the image is materialised locally and sparse regions are skipped. Re-pulling unchanged bytes skips the transfer but still rolls the pool.

Follow logs

Application logs come from the obs plugin, once it is installed in the namespace:

heyctl plugins install obs -n team-a
heyctl logs web -n team-a --since 15m --level error
heyctl top -n team-a

Job output comes with the job:

heyctl build web --logs        # stream build output, then wait
heyctl pull web --logs
heyctl update app-obs --logs
heyctl get job job-3f2a1c8e    # status plus log tail of any job

Without the plugin, run a command in the guest (heyctl exec web -- tail -n 100 /var/log/heyvm-start.log). Guest start-command errors live in that file, not in app-obs.

Scale

heyctl scale web --min 1 --max 8 --warm 2 --target-concurrency 20
heyctl scale web --replicas 3                  # pin
heyctl scale web --scale-to-zero-after 600
heyctl restart web --wait                      # rolling recycle
heyctl top

Agent sandbox

heyctl create deployment sb-7f3a9c --no-route --port 8080 --size medium --disk-gb 8
heyctl scale sb-7f3a9c --idle-action retain
heyctl exec sb-7f3a9c --cwd /workspace -- git status
heyctl shell sb-7f3a9c
heyctl set route sb-7f3a9c --host sb-7f3a9c.example.com   # expose it later
heyctl set route sb-7f3a9c --none                         # withdraw it again

Static upstream maintenance

heyctl drain stage us1.internal:8080 --reason 'kernel upgrade' --timeout 300
# ... maintenance ...
heyctl uncordon stage us1.internal:8080
heyctl get vms -d stage     # shows Draining separately from health

Work in a namespace

heyctl create namespace team-a --description "Acme"            # fleet admin
heyctl create deployment api -n team-a --host api.example.com --port 8080
heyctl create secret db -n team-a --from-env url=DATABASE_URL
heyctl token mint team-a-ci --admin admin --namespace team-a -q
heyctl get deployments -n team-a
heyctl plugins install obs -n team-a                             # telemetry for every app in it

A token minted with --namespace and no --deployment reaches every deployment in that namespace and nothing outside it. Secrets and auth providers are resolved per namespace: get secrets -n team-a is the only way to name one when ids collide across namespaces.

Exit codes

Troubleshooting

For the full spec format see app-lb and the app-lb README. The heyctl source README, app-lb/heyctl/README.md, has longer notes on each command.