Orchestrator

The orchestrator is the HWS control plane that records what should be running where. It provisions sandboxes for CI jobs, deploys and rolls long-running services, publishes the service discovery sets app-lb routes from, and runs region-by-region rollouts with durable, restart-safe progress.

What it is

The orchestrator (orchestrator/) is one Rust binary backed by PostgreSQL. Each request persists desired state and returns. Background reconcilers then converge that state by calling the systems that actually move VMs and traffic.

It depends on the following:

   CI ──(JWT)──► /orchestration/resources/*  ─┐
                                              │   ┌──────────────┐     ┌────────────┐
 operator/CI ─(internal key)─► /services/* ───┼──►│ orchestrator │────►│   Cloud    │──► heyvm (VMs)
                                              │   │  + Postgres  │     └────────────┘
                                              │   └──────┬───────┘
             HeyoSecret ◄── envRefs / tokens ─┘          │ discovery (versioned endpoints)
                                                         ▼
                                           app-lb (one or more per region)
                                           polls discovery, reports drain status

Running it

Locally

cp orchestrator/.env.example orchestrator/.env   # then edit; see the configuration notes below
cargo run --locked --manifest-path orchestrator/Cargo.toml --bin orchestrator
curl -si localhost:4446/health

At startup the orchestrator does the following, in order:

1. Loads .env from the crate directory, falling back to .env in the working directory.
2. Loads the configuration.
3. Connects to PostgreSQL and applies the SQL files in migrations/.
4. Fetches backend capabilities from ORCHESTRATOR_BACKEND_API_URL + /capabilities if that is set.
5. Starts four reconcilers: workflow steps, delayed retirement, regional rollouts and application updates.

The migrations directory is resolved from ORCHESTRATOR_MIGRATIONS_DIR, then ./migrations, then the compile-time crate directory.

GET /health returns {"status":"ok", "deploymentGitSha", "deploymentPrNumber", "deploymentId"}. The body fields come from HEYO_ORCHESTRATOR_DEPLOYMENT_* environment variables and are diagnostic only. The x-heyo-revision response header holds the build-time HEYO_BUILD_GIT_SHA, or unknown for unstamped builds. Use the header, not the body, to tell which binary answered.

Packaging

A healthy /health does not prove that deployments work. Before you accept deployment traffic, check that CLOUD_INTERNAL_URL reaches a Cloud that accepts the internal key and can allocate a backend in this region. The orchestrator also reads Cloud's deployment-status table directly, so Cloud and the orchestrator must share a database.

Configuration

Configuration is layered as follows:

1. Built-in defaults.
2. A TOML file. The path is HEYO_ORCHESTRATOR_CONFIG_PATH, else HEYO_CONFIG_PATH, else ~/.heyo/orchestrator/orchestrator.toml. The file is used if it exists.
3. An ORCHESTRATOR_-prefixed environment layer. Its keys are split on _. This works for nested keys such as ORCHESTRATOR_NATS_ENABLED (nats.enabled). It does not work for field names that contain an underscore: ORCHESTRATOR_AGENT_MODEL becomes agent.model, not agent_model.
4. Explicitly read environment variables, listed below. These fill a value only when layers 1–3 left it empty.

Any setting not in the environment table can only be set in the TOML file. This covers the agent provider and model settings, the db_* pool settings, target_os and the driver lists. The ORCHESTRATOR_AGENT_*, ORCHESTRATOR_DB_* and ORCHESTRATOR_TARGET_* lines in .env.example and the service workflow have no effect. So do ORCHESTRATOR_DATABASE_URL, ORCHESTRATOR_JWT_SECRET, ORCHESTRATOR_INTERNAL_API_KEY and ORCHESTRATOR_CLOUD_INTERNAL_URL. Use the unprefixed names in the table.

Environment variables

TOML-only settings

A minimal file:

database_url = "postgres://orchestrator:change-me@127.0.0.1:5432/orchestrator"
jwt_secret = "change-me"
internal_api_key = "change-me"
heyosecret_url = "https://heyosecret.example.com"
discovery_routed_services = "example"

[nats]
enabled = false

Authentication

The discovery endpoint uses the internal key. The app-lb secret you register as a discovery reader must therefore hold the orchestrator's internal API key.

API reference

Resource sandboxes (CI)

Services

Agent workflows

Workflow steps run through adapters: ai.discovery, ai.planning, repo.summary, repo.key_files, repo.framework_detection, repo.patch, repo.verify, heyo.deploy_preflight, heyo.deploy and heyo.healthcheck. AI steps never have side effects. Deterministic adapters apply patches, run verification and deploy. Steps are claimed with row locks, so two orchestrator instances cannot run the same step. See the design notes for the state machine.

Service deployment spec

POST /orchestration/services/deployments takes an app-lb-style snake_case document. Unknown fields and unsupported features are rejected; they are not ignored. The same format lives in .heyo/services/*.json.

{
  "id": "example",
  "user_id": "heyo-system",
  "vm": {
    "driver": "libvirt",
    "image": "ubuntu:24.04",
    "port": 8080,
    "start_command": "cd /workspace && exec ./start.sh",
    "env_vars": {"PORT": "8080"},
    "env_from": [{"secret": "example", "key": "database-url", "as": "DATABASE_URL"}]
  },
  "routes": [{"host": "example.example.com"}],
  "health": {"path": "/health", "timeout_secs": 5},
  "deploy": {"archive_id": "...", "region": "EU", "async": true, "drain_seconds": 30}
}

The orchestrator resolves env_from into envRefs of the form NAME=heyosecret://<path>@active. It reads the values from HeyoSecret just before it asks Cloud for the VM. Resolved values are never stored. Each deployment's creation recipe, including the exact secret versions, is recorded for replay.

Deployment behaviour

• Single candidate (default). The previous healthy deployment keeps serving while the candidate converges. The candidate and the app-lb route must stay healthy for 10 seconds before cutover. The previous deployment is then drained and, if requested, stopped or deleted.
• Rolling replicas. This mode applies to services listed in ORCHESTRATOR_DISCOVERY_ROUTED_SERVICES. The request must include the service's stable route and scaling. deploy.replica_regions must have exactly one entry per replica. The orchestrator publishes one health-gated candidate to discovery, drains one old replica, and repeats. A failed candidate leaves the remaining healthy set serving.
• Retirement is delayed and fenced. An old replica is stopped only after its drain has expired, its owning rollout is still current and successful, and a per-service PostgreSQL advisory lock is held. A later rollout that reactivates a previous deployment cancels that deployment's pending retirement.
• Environment binding. An optional deployment_environment scopes Cloud placement. It is bound to the service ID permanently, so use separate service IDs for staging and production.

Discovery and app-lb

For a discovery-routed service, app-lb holds a static deployment whose discovery block points at this orchestrator. app-lb replaces its upstream list from each accepted snapshot and keeps the last good set when the orchestrator is unreachable.

{"id": "example", "routes": [{"host": "example.example.com"}], "upstreams": [],
 "discovery": {"service_id": "example",
   "source": {"url": "https://orchestrator.example.com/orchestration/services/example/discovery",
              "auth": {"secret": "discovery-reader", "key": "token"}}}}

discovery-reader is an app-lb secret. Give it the orchestrator's internal key as its value, using HeyoSecret as the source of truth. See HeyoSecret.

Observers are how the orchestrator proves that a drain happened. List every app-lb instance that can admit traffic for a service. Each URL must address one instance, not a load balancer in front of several.

[[discovery_observers]]
service_id = "example"
region = "eu1"
deployment_id = "example"                           # app-lb deployment id
base_url = "https://eu1-app-lb-admin.example.com"   # that app-lb's admin API
token_secret_path = "platform/eu1-app-lb-admin"     # HeyoSecret path of an app-lb admin token
# Optional host-managed ingress bootstrap:
ingress_url = "https://eu1-ingress.example.com"
discovery_url = "https://orchestrator.example.com/orchestration/services/example/discovery"
discovery_token_secret = "discovery-reader"         # app-lb secret id, not a value
# Hierarchical (regional-v1) only:
# gateway_id = "eu1-a"
# regional_peer_token_secret = "gateway-peer"

The controller calls GET /deployments/{deployment_id}/discovery-status on each observer. A drain gate passes only when every observer reports the persisted discovery version, matching serving membership, and zero in-flight requests to withdrawn peers. Elapsed time alone never counts as a drain.

When ingress_url, discovery_url and discovery_token_secret are set, the first deployment of a new service registers the discovery route on each app-lb. Registration is create-only (If-None-Match: *) and is read back to confirm it. The orchestrator never replaces an existing route.

External service bindings let the orchestrator adopt an application that app-lb already runs, such as CI, without creating a VM:

[[external_service_bindings]]
service_id = "ci"
authority = "https://app-lb-admin.example.com/"     # credential-free origin
namespace = "default"
region = "eu1"
deployment_id = "ci-eu1"
health_origin = "https://ci.example.com/"
token_secret_path = "platform/app-lb-admin"         # HeyoSecret path
lifecycle_token_secret_path = "ci/lifecycle-token"  # HeyoSecret path

Relationship to HeyoSecret

The orchestrator is a HeyoSecret client, through the heyosecret-client crate. It reads secrets for:

• envRefs and env_from on service and resource deployments
• token_secret_path for each observer and external binding
• lifecycle tokens
• git credentials

It never writes secrets. HeyoSecret has no per-path authorization, so the orchestrator is the policy layer. A deployment names paths, and the orchestrator decides which paths to resolve and passes only the values to Cloud.

HeyoSecret is itself deployed by the orchestrator in the managed setup. Its update path reads heyosecret/database-url and heyosecret/master-key from the running instance.

Operations

Check which binary is serving.

curl -sI https://orchestrator.example.com/health | grep -i x-heyo-revision

Watch a regional rollout.

curl -s -H "Authorization: Bearer $ORCHESTRATOR_INTERNAL_API_KEY" \
  https://orchestrator.example.com/orchestration/services/regional-rollouts/$OP | jq '.status, .phase, .items[-1]'

Run the tests.

cargo test --locked --manifest-path orchestrator/Cargo.toml --bin orchestrator
# PostgreSQL-backed suites need a disposable database:
ORCHESTRATOR_TEST_DATABASE_URL=postgres://postgres@127.0.0.1:54329/postgres \
  cargo test --locked --manifest-path orchestrator/Cargo.toml -- --include-ignored

The two-gateway drain fixture, two_real_gateways_drain_through_authenticated_durable_barriers, also needs APP_LB_TEST_BINARY set to an app-lb binary built with --features reqwest/rustls-tls-native-roots.

Database recovery. Never edit rollout, retirement or discovery rows while deployments or cleanup are running. Stop them first. A database trigger cannot recall a Cloud deletion that is already in flight.

Troubleshooting