pg-fc: Postgres on microVMs

pg-fc runs one Firecracker microVM per Postgres database behind a single pooler endpoint on :6432, stopping idle databases and moving cold ones to cheaper storage tiers until the next client connects.

What it is

pg-fc has two parts:

The pooler drives VMs through the local heyvm daemon (heyvmd) HTTP API, the same one app-lb uses. It also serves an optional admin dashboard and JSON API. app-lb's pgfc plugin reads that API (see app-lb and the plugin reference).

client ──► pg-vm-pool :6432 ──► pg-<schema> VM :5432   (one VM per database)
                │                    │
                │                    └── /workspace data disk (per schema)
                ├── heyvmd API :34099 (create / start / stop / resize VMs)
                └── dashboard + JSON API (optional, PG_VM_POOL_DASHBOARD_LISTEN)

The lifecycle of one schema:

Every tier except warm is opt-in. With no tier variables set, pg-fc only stops idle VMs.

Requirements

Linux with KVM only.

Build

Guest image

The pooler boots VMs from the heyvm image named by PG_VM_POOL_IMAGE (default pg). Build it with heyvm:

cd pg-fc
heyvm mvm build --local-only -f Dockerfile.pg18 --name pg

Dockerfile defaults to PostgreSQL 16 (ARG PG_MAJOR=16). Dockerfile.pg18 builds PostgreSQL 18. Pick one major for every host that shares an S3 bucket: a data directory initialized by one major can't be opened by the other, so a host running a different major can't restore the other hosts' archives. See the major-mismatch runbook.

To build a raw ext4 rootfs for use without heyvm:

PG_MAJOR=18 ./build-rootfs.sh pg-rootfs.ext4 2G

build-rootfs.sh defaults to PostgreSQL 18 and checks that the requested major's postgres binary is present in the output.

Pooler

cargo build --release --locked --manifest-path pg-fc/Cargo.toml
# binary: pg-fc/target/release/pg-vm-pool

The binary takes no flags. Configure it with environment variables. Log verbosity uses RUST_LOG (default info,pg_vm_pool=info). Unknown PG_VM_POOL_* variables log a warning at startup.

How the guest behaves

• init.sh is PID 1. It mounts the data disk (/dev/vdb; override with the kernel arg pgdata_dev=), formats it on first boot, runs initdb into /workspace/pgdata, and starts Postgres on 0.0.0.0:5432.
• The data disk is thin-provisioned. The guest formats a small filesystem (2 GB, kernel arg pgdata_init_mb= overrides) and grows it online with resize2fs as the database grows, up to the device size.
• Postgres settings are regenerated at every boot from the VM's actual RAM, vCPUs and disk into $PGDATA/heyvm-tuning.conf. Put manual overrides in postgresql.conf, which is read after it and wins.
• The profile is single-tenant: wal_level=minimal (unless replication is on), synchronous_commit=off, strict memory overcommit, a swapfile on the data disk, and temp_file_limit at a quarter of the disk. The pooler runs CHECKPOINT before every idle stop, so a stop does not lose acknowledged commits.
• Guest auth is trust. Access control is the pooler's job (see Client authentication).
• The rootfs is recreated from the image on every boot. Anything changed only inside a running guest's rootfs does not survive a restart.

Running the pooler

With heyvmd running and the pg image built:

target/release/pg-vm-pool        # listens on 127.0.0.1:6432

# The dbname selects the VM, creating it on first use:
psql "host=127.0.0.1 port=6432 user=postgres dbname=tenant1"   # -> VM pg-tenant1
psql "host=127.0.0.1 port=6432 user=postgres dbname=tenant2"   # -> VM pg-tenant2

The schema-to-VM binding is stored in PG_VM_POOL_STATE_FILE (default ~/.heyo/pg-vm-pool/registry.tsv). This file is the only link between a schema and its VM and disk. If you lose it, every existing schema looks brand new. Back it up, and never run two poolers against the same state directory.

Connection behaviour:

• A connection holds one Postgres backend slot for its whole life. When a VM is full, new clients wait at the pooler for PG_VM_POOL_ADMIT_TIMEOUT_SECS instead of getting too many clients.
• Both legs of every splice use TCP keepalive (and TCP_USER_TIMEOUT on Linux), so a client that vanishes without a FIN releases its slot.
• If Postgres has died inside a VM that still reports running, the next connect detects it and restarts the VM. A Postgres that is recovering (57P03) is left alone.
• By default the pooler dials each VM's guest IP directly over the host tap (PG_VM_POOL_DIRECT_CONNECT). It falls back to an iroh tunnel if the daemon reports no guest IP.

As a systemd service

deploy/pg-fc@.service is a templated unit:

Set PG_VM_POOL_STATE_FILE explicitly inside the state directory. The working directory alone does not move the registry, which otherwise defaults under $HOME.

sudo install -m 0644 pg-fc/deploy/pg-fc@.service /etc/systemd/system/
sudo systemd-analyze verify /etc/systemd/system/pg-fc@.service
sudo systemctl daemon-reload
sudo systemctl enable --now pg-fc@node-a

The unit does not install heyvm, build the image, open ports or set up replication. On a shared host, leave eviction, orphan sweeping and disk reclamation off until you have decided which process owns which disks.

As a supervisord program

deploy/supervisor/pg-vm-pool.conf is an example program with the full density stack enabled (orphan sweep, periodic reclaim, compacted tier, S3 archive tier, pressure eviction). Its header comment lists the one-time host setup it needs. Paths in it are host-specific, so adapt them before you use it.

Change settings by editing environment= and running supervisorctl reread && supervisorctl update pg-vm-pool. A plain restart does not reload environment=. Double-quote values that contain commas.

deploy/provision-pooler-host.sh builds a bare Ubuntu machine into a pooler host (storage, heyvm, image, sudoers pin, supervisor). It is idempotent. Its only destructive step, RAID creation, requires RAID_CREATE=yes and refuses non-blank devices. It does not configure a firewall.

Upgrading

Upgrading the pooler binary is stop, replace, start. Never start a second copy alongside the running one: two poolers over one state directory both act on the same registry and VMs, and neither knows about the other's sessions.

1. Build or download the new pg-vm-pool and check its digest.
2. Stop the service (systemctl stop pg-fc@<node> or supervisorctl stop pg-vm-pool) and confirm the old PID has exited.
3. Replace the binary in place atomically (write a temporary file, then rename it).
4. Start the service and verify: the running /proc/<pid>/exe has the new digest, the registry path it uses is unchanged, and a SELECT 1 through :6432 succeeds.
5. If verification fails, stop it, restore the previous binary, start it and verify again.

deploy/replace_pooler.py automates exactly this sequence with a journaled receipt and automatic rollback. deploy/rollout_poolers.py runs it across several hosts as app-lb update jobs, preflighting every target before replacing any. Clients connected during the upgrade are disconnected. VMs keep running and are re-adopted when the pooler starts.

Upgrade the guest image separately, and keep the same PostgreSQL major. Replication features need the current init.sh. Test the new image on a disposable data disk before pointing a serving pooler at it.

Configuration

Every variable is optional. Values are read at startup. A few can also be changed at runtime (see Runtime configuration).

Core

Timeouts and admission

Idle reaping

A schema's first connect creates its VM and gets the long timeout. Later reconnects are cheap restarts and get the short one. When the host is slow, restarts stop being "fast", so the reaper backs off on its own.

Disk growth

Warm spares

Offload tiers

The compacted, frozen and S3 tiers share one pacer. It dispatches one job at a time while the host is quiet and yields to queued client bring-ups and reclaim passes.

S3 settings:

The guest streams dump bytes to and from S3 with presigned URLs, so the secret key never leaves the pooler. Empty databases (no user relations) are never uploaded.

Disk pressure, reclaim and orphans

At startup the pooler warns if PG_VM_POOL_RUN_DIR contains no sb-<id>/ directories. A wrong run dir silently disables every disk-reclaiming feature, so check this warning first when disk use doesn't fall.

TLS and dashboard

Client authentication

With PG_VM_POOL_PASSWORD unset, anyone who can reach PG_VM_POOL_LISTEN is proxied to a VM, and any database name they send creates a new VM. That is only acceptable on loopback.

Set PG_VM_POOL_PASSWORD and the pooler sends each client an AuthenticationCleartextPassword challenge before dialing any VM. A wrong password gets 28P01. Because the challenge is cleartext, enable TLS whenever the listener is not loopback. The pooler logs a warning if it isn't.

TLS

With TLS off, the pooler answers SSLRequest with N and clients continue in plaintext. With TLS on, clients that request TLS get it, and plaintext clients are still accepted. TLS terminates at the pooler. The pooler-to-VM hop is plaintext over the host-local tap.

PG_VM_POOL_TLS_CERT=/etc/pg-fc/tls/current/cert.pem \
PG_VM_POOL_TLS_KEY=/etc/pg-fc/tls/current/key.pem \
target/release/pg-vm-pool

The cert files are re-read before each handshake when they change, so renewals need no restart. If Traefik on the same host owns the certificate, deploy/sync-traefik-cert.py <acme.json> <hostname> <out-dir> exports and validates it into <out-dir>/current/. Run it periodically.

Dedicated databases

PG_VM_POOL_PASSWORD is a shared credential that can create unlimited databases. To hand a credential to an application or customer, provision a dedicated database instead. It has its own role and password, can open only its own database (other names get 42501), and cannot create VMs.

Provisioning uses the dashboard listener, so PG_VM_POOL_DASHBOARD_LISTEN must be set:

# username defaults to the database name; password is generated if omitted
curl -u admin:$DASH_PASS -X POST http://127.0.0.1:34199/api/databases \
     -H 'content-type: application/json' -d '{"database":"acme"}'
# -> 201 {"database":"acme","username":"acme","password":"…","status":"provisioning",…}

curl -u admin:$DASH_PASS http://127.0.0.1:34199/api/databases              # list, no passwords
curl -u admin:$DASH_PASS -X DELETE http://127.0.0.1:34199/api/databases/acme  # revoke

The client then connects normally:

psql "host=pg.example.com port=6432 user=acme dbname=acme sslmode=require"

• The password is shown once. It is stored in cleartext in PG_VM_POOL_DEDICATED_FILE.
• Names must be lowercase letters, digits and underscores, start with a letter, and be at most 63 bytes. The pg_ and spare prefixes and Postgres catalog names are rejected, as is any name already in use as an ordinary schema.
• The role is NOSUPERUSER NOCREATEDB NOCREATEROLE, owns its database, and is recreated on every bring-up, so it survives dump restores.
• Revoking removes only the credential. The VM and data remain, and the name reverts to ordinary schema routing.

Client guidance

A pooler connection can take much longer than a normal Postgres connect, because the first byte may wait for a VM to be created, started or restored.

Recommendations:

• Set an explicit connect timeout that is longer than your worst expected cold start (for example connect_timeout=60 in libpq, or connectionTimeoutMillis in node-postgres). Some drivers, including node-postgres Pool, have no connect timeout by default, so a connect to a slow thaw hangs silently.
• Don't tie liveness to the first database connect. If your app blocks startup on the database and its platform health check (for example app-lb's) expires before a thaw finishes, the app boot-loops without logging anything useful. Start serving, then connect, or give the health check a window longer than a cold start.
• Retry on these codes:

• Reconnect after idle. A stopped VM drops its sessions. Use pool validation (SELECT 1 on checkout) or short idle lifetimes rather than holding connections for hours.
• Use keep-alive schemas for latency-critical databases. Listing them in PG_VM_POOL_KEEPALIVE_SCHEMAS keeps them warm at the cost of their RAM.

Cross-host replication

A dedicated database on one pg-fc node can be replicated continuously to a peer node. Both nodes need:

PG_VM_POOL_REPLICATION=1
PG_VM_POOL_NODE_NAME=node-a                   # different on each node
PG_VM_POOL_DASHBOARD_LISTEN=0.0.0.0:34199     # the peer calls this API
PG_VM_POOL_DASHBOARD_USER=admin
PG_VM_POOL_DASHBOARD_PASSWORD=…

A node acting as primary also needs a reachable listener, an advertised address and TLS:

PG_VM_POOL_LISTEN=0.0.0.0:6432
PG_VM_POOL_ADVERTISE_PG_HOST=203.0.113.10     # what the PEER's guests dial
PG_VM_POOL_TLS_CERT=/path/fullchain.pem
PG_VM_POOL_TLS_KEY=/path/privkey.pem

Then register the peer and start a pairing from the primary:

curl -u admin:$DASH_PASS -X POST http://127.0.0.1:34199/api/peers \
  -H 'content-type: application/json' \
  -d '{"name":"node_b","base_url":"https://b.example:34199","user":"admin",
       "password":"…","pg_host":"198.51.100.20","pg_port":6432}'

curl -u admin:$DASH_PASS -X POST http://127.0.0.1:34199/api/replication \
  -H 'content-type: application/json' -d '{"database":"acme","peer":"node_b"}'

curl -u admin:$DASH_PASS http://127.0.0.1:34199/api/replication/acme   # state and lag

The default mode is logical replication: a publication on the primary, and a subscription on the replica that connects through the primary's ordinary pooler listener using a separate REPLICATION login. Things to know:

• DDL, sequence values and large objects are not replicated. After a new table is created on both nodes, pick it up with refresh. Promote re-seeds sequences.
• Tables with no primary key need REPLICA IDENTITY FULL.
• A database in a live pairing is pinned on both nodes: it is never idle-stopped, compacted, frozen, archived or evicted. It costs RAM and disk on both hosts permanently.
• An abandoned slot retains WAL until max_slot_wal_keep_size invalidates it. If a replica is gone for good, detach the pairing. Don't just delete the record.
• Peering is full trust. Each node stores the other's dashboard admin password.
• There is no automatic failover. promote (on the replica) and detach (on the primary) are operator actions.

For planned switchovers, the API also offers durable source fences (fence, fence-selective, unfence) and a physical-replication handoff (physical-prepare, physical, physical-handoff, physical-reseed, physical-standby-bind). The guest side of the physical path is /usr/local/bin/pg-fc-physical (physical.sh). These are controller primitives with strict preconditions. Read the replication section of the component README before using them.

Dashboard and JSON API

Set PG_VM_POOL_DASHBOARD_LISTEN (and credentials) to serve a server-rendered dashboard from inside the pooler process. The dashboard can stop, resize and delete every VM on the host. Keep it on loopback or a private address, and always set Basic auth.

The JSON API sits on the same listener and uses the same auth. It is keyed by schema name, not sandbox id, because a schema outlives its VMs. Wire types are in the pg-fc-api crate (pg-fc/api/).

Runtime configuration

These settings can change without a restart: idle_timeout_secs, idle_timeout_fast_secs, warm_spares, compact_after_secs, freeze_after_secs and archive_after_secs.

curl -u admin:$DASH_PASS http://127.0.0.1:34199/api/config
curl -u admin:$DASH_PASS -X PUT http://127.0.0.1:34199/api/config \
  -H 'content-type: application/json' -d '{"idle_timeout_secs": 300}'

Overrides persist to runtime-config.json next to the registry and apply over the environment at boot. GET shows each value's source (override, env or default). A tier that was off at boot can't be turned on this way: the request returns 400 until you set the environment variable and restart.

Webhook alerts

From /monitoring, add rules on host CPU %, memory %, disk %, or the heyvmd health check (the threshold is consecutive failed probes). The evaluator POSTs JSON once when a rule triggers and once when it resolves:

{"source":"pg-vm-pool","host":"pool-1","rule_id":"…","metric":"disk",
 "state":"triggered","threshold_pct":90.0,"value_pct":93.4,"detail":"/"}

Common operations

Reclaim disk space

Data disks are sparse files, but without discard passthrough they only grow. Four tools recover space, from least to most destructive:

To run reclaim from the pooler, pin the exact command in sudoers and set PG_VM_POOL_RECLAIM_CMD to the same string:

# /etc/sudoers.d/pg-vm-pool  (0440)
pooler ALL=(root) NOPASSWD: /opt/pg-fc/reclaim-disks.sh /srv/heyvm/run --shrink --prune-swap

PG_VM_POOL_RECLAIM_CMD="sudo -n /opt/pg-fc/reclaim-disks.sh /srv/heyvm/run --shrink --prune-swap"

Install the script at a root-owned path. Verify the pin as the pooler user with sudo -n -l <exact command>. sudoers compares arguments byte for byte, and a mismatch fails every run while only logging a failed reclaim once an hour.

Recover a full disk

If the host disk is full, offloads that boot a VM make it worse. emergency-drain.sh stops the pooler and only runs steps that free space without first consuming any. disk-audit.sh is read-only and reconciles the run dir, heyvmd and the registry.

Change VM size

Resize a schema from its dashboard page or with POST /api/schemas/{schema}/resize. The new size applies on the VM's next boot. PG_VM_POOL_SIZE_CLASS sets the size for new VMs only.

Troubleshooting

Testing

cargo test --locked --manifest-path pg-fc/Cargo.toml

# end-to-end against a running pooler and heyvmd (from pg-fc/)
cd pg-fc
cargo run --release --example e2e
cargo run --release --example e2e_concurrent

# cold-start cost against a synthetic fleet (in-process heyvmd stub)
cargo test --release loadtest -- --ignored --nocapture --test-threads=1

examples/e2e_replication.rs needs two nodes. See its header for the variables it reads.

See also

• app-lb: the pgfc plugin surfaces this dashboard in app-lb
• multi-region
• Component README: design notes on every tier, the offload pacer, reclaim locking and physical handoff