Quoro documentation

Quoro is a self-hosted tool that runs one microVM per application on Cloud Hypervisor and KVM, and deploys, exposes and observes those applications from the same interface. It is built to be operated by AI agents through scoped, time-limited grants, with every action recorded in an append-only, hash-chained audit log.

Quoro is experimental. Read the roadmap for what each milestone delivers before you rely on it.

Where to start

• Install: Installing Quoro covers the one-command installer, its roles (primary and node), what it verifies and the host layout it creates.
• Concepts: the product requirements explain what Quoro is for and who it is for; the glossary defines the terms used everywhere else.
• Architecture: how the pieces fit: control plane, node agents, engine, ingress, state replication and the agent loop.
• Contracts: the interfaces the components are built against: the HTTP API, events, the database schema, the node-agent protocol, the permission model and the audit log format.
• Security: the threat model and how to report a vulnerability.
• Decisions: every accepted design decision is an architecture decision record.

This documentation is built from the Markdown in the repository’s docs/ directory with mdBook. The site at quoro.dev is published from GitHub Pages each time a release is published, so it matches the latest release, not the main branch.

Quoro installer

One command installs Quoro on a Linux host: the static binary, the Cloud Hypervisor runtime it runs apps on, the cloudflared connector it serves app hostnames through, two systemd units, the data directories and the config file.

curl -fsSL https://raw.githubusercontent.com/OmarAlex24/quoro/main/installer/install.sh | sudo sh

Requirements: Linux on x86_64 (x86_64-unknown-linux-musl is the only published target for now), root, systemd running, and curl, tar, gzip, sha256sum, install, mktemp, cmp, systemctl and grep on the host; a primary also needs useradd for its system account. The engine also needs mkfs.ext4 (e2fsprogs), ip (iproute2) and nft (nftables), and Git builds need git, which the node agent runs to fetch the app’s commit: when one is missing the installer runs apt-get update and then installs the four packages from apt without recommends; without apt-get it stops and names the missing tool. /dev/kvm is required at run time: the installer warns when it is absent (the node agent then refuses to start the engine), so the files can still be inspected on a machine without virtualisation. The installer stops and lists anything else missing.

The script is POSIX sh, and every statement that touches the host lives inside a function with a single main "$@" at the end: a truncated download does nothing.

Environment variables

Example, pinning a version:

curl -fsSL https://raw.githubusercontent.com/OmarAlex24/quoro/main/installer/install.sh \
  | sudo QUORO_VERSION=0.1.0 sh

Roles

QUORO_ROLE decides what the host is (ADR 0089):

• primary (default) runs the control plane, quoro.service: the API, the panel and the node sessions, as the unprivileged system user quoro-api with an empty capability bounding set. The same host also runs its own node agent, so it deploys apps too. Nothing else is needed: the primary’s agent enrolls itself through a join file the control plane writes, so there is no token.
• node runs only the node agent, quoro-node.service, as root. It runs the microVMs for one primary and serves no API and no panel. It belongs to exactly one primary and is enrolled during the install.

A node install needs a join token. Create one on the primary, in the panel or with POST /api/v1/nodes/join-tokens, and pass it with the primary’s URL and TLS fingerprint:

curl -fsSL https://raw.githubusercontent.com/OmarAlex24/quoro/main/installer/install.sh \
  | sudo QUORO_ROLE=node \
         QUORO_JOIN_URL=https://<primary>:8443 \
         QUORO_JOIN_TOKEN=<token> \
         QUORO_JOIN_FINGERPRINT=sha256:<hex> sh

The three values are checked before anything is written, so a missing one fails before the host is touched. A join token is single use and expires, so create a fresh one for every attempt. A host that is already enrolled needs no token: re-running the installer with QUORO_ROLE=node finds its identity and skips the enrollment.

A node install refuses a host that already runs a control plane (it finds /etc/systemd/system/quoro.service or /var/lib/quoro/quoro.db): turning that host into a node would orphan the database, its key files and its panel. It stops without deleting anything; uninstall the control plane first.

What it verifies, and what it trusts

1. The script itself arrives over TLS from raw.githubusercontent.com; curl -fsSL fails on an HTTP error and refuses plain HTTP.

2. The tarball against SHA256SUMS from the same release. This proves the download is the published bytes (integrity), nothing more. The check happens before anything is extracted, and a missing line for the tarball is a hard failure.

3. The tarball against its GitHub artifact attestation: a keyless Sigstore bundle (quoro-<version>-x86_64-unknown-linux-musl.tar.gz.sigstore.jsonl) verified with

   gh attestation verify <tarball> --bundle <bundle> \
     --repo OmarAlex24/quoro \
     --signer-workflow OmarAlex24/quoro/.github/workflows/release.yml \
     --source-ref refs/tags/v<version> \
     --deny-self-hosted-runners
   

   That pins the repository, the signing workflow, the tag and the runner kind. Verification uses a GitHub CLI downloaded from github.com/cli/cli whose sha256 is pinned in install.sh (quoro-install: prints when it fetches it). The pinned CLI runs with an empty config dir and with GH_TOKEN / GITHUB_TOKEN unset, so a token in the environment cannot change the result.

4. The engine runtime against the pins in the tarball. The tarball also carries pins.txt (crates/engine/pins.txt): one line per artifact with its pinned sha256. The installer reads the lines for this architecture, requires the cloud-hypervisor and guest-kernel pins, and stages each file only if its sha256 matches. Files come from QUORO_INSTALL_FROM first, then from an existing /usr/local/lib/quoro, then from the URL in the pin; a local candidate that does not match is ignored with a warning and the next source is tried. A download that does not match is fatal. Upstream publishes no signatures for these artifacts, so the integrity anchor is the chain TLS to GitHub → attested tarball → pins.txt → sha256, and the engine re-verifies the same pins on every start.

5. The tunnel connector against its own pins in the tarball. Nodes behind NAT serve app hostnames through Cloudflare Tunnel, which the node agent runs as a cloudflared child process (ADR 0024, ADR 0055). The tarball carries ingress-pins.txt (crates/ingress/pins.txt) in the same format, one cloudflared line per architecture. The installer reads the line for this architecture, requires it to be cloudflared, requires its URL to be under https://github.com/cloudflare/cloudflared/releases/download/, and stages the file with the same rules as (4): QUORO_INSTALL_FROM, then an existing /usr/local/lib/quoro, then the pinned URL; a local candidate that does not match is ignored with a warning, a download that does not match is fatal. The integrity chain is the same one: TLS to GitHub → attested tarball → ingress-pins.txt → sha256. A tarball from an older release has no ingress-pins.txt; the installer then warns and installs without a connector, because only the tunnel needs it.

The repository, signer workflow, tag ref and GitHub CLI version/sha256 are constants in the script. No environment variable overrides them. The one exception is QUORO_INSTALL_DRY_RUN_REF, which replaces only the attested source ref, requires local artifacts and cannot be combined with the insecure flag; it exists so a dry-run build can be verified before a release, and the attestation check is otherwise unchanged. Changing any other constant is a reviewed change (ADR 0027).

Any failure in (2), (3), (4) or (5) aborts with

signature verification failed, nothing was installed

and no file is written outside the temporary directory. Host packages are installed only after every check passed.

Host layout created (ADR 0054, ADR 0089)

/var/lib/quoro and /var/lib/quoro-node are siblings, never nested, so each service owns its own tree and the unprivileged control plane cannot read the agent’s state. A quoro user left by an older installer is no longer used; the installer logs it and leaves it alone.

The config is a starting point, not a secret. On a primary:

# Quoro configuration (ADR 0054 host layout, ADR 0089 privilege split).
# Edit this file and restart quoro: systemctl restart quoro quoro-node

# One TLS listener for API, panel and MCP. Ingress keeps 80/443.
listen = "0.0.0.0:8443"

# SQLite database, cluster key files (0600) and the local join file.
data_dir = "/var/lib/quoro"

# Engine: "cloud-hypervisor" runs one microVM per app; "fake" runs nothing
# and exists for development only.
engine = "cloud-hypervisor"

# Base URL nodes dial, for example "https://quoro.example.com:8443".
# Unset, it defaults to the Host header of the request that created the
# join token, which is right for most installs.
# public_url = "https://<this host>:8443"

# Engine settings. The values below are the defaults, and the block is
# commented out; uncomment a line only to change it. Guest DNS defaults
# to this host's non-loopback resolvers; set dns to override them.
# [cloud_hypervisor]
# artifacts_dir = "/usr/local/lib/quoro"
# init_binary = "/usr/local/lib/quoro/quoro-init"
# state_dir = "/var/lib/quoro-node/engine"
# bridge = "quoro0"
# subnet = "10.42.0.0/24"
# dns = ["192.0.2.53"]

[node]
# The node agent runs as quoro-node.service, separate from the
# unprivileged control plane (ADR 0089).
local = false

On a node:

# Quoro configuration (ADR 0089 privilege split).
# This host is a node: it runs microVMs for its primary and runs no
# control plane. Edit this file and restart with:
#   systemctl restart quoro-node

# Engine: "cloud-hypervisor" runs one microVM per app; "fake" runs nothing
# and exists for development only.
engine = "cloud-hypervisor"

# Engine settings. The values below are the defaults, and the block is
# commented out; uncomment a line only to change it. Guest DNS defaults
# to this host's non-loopback resolvers; set dns to override them.
# [cloud_hypervisor]
# artifacts_dir = "/usr/local/lib/quoro"
# init_binary = "/usr/local/lib/quoro/quoro-init"
# state_dir = "/var/lib/quoro-node/engine"
# bridge = "quoro0"
# subnet = "10.42.0.0/24"
# dns = ["192.0.2.53"]

[node]
# The identity, local store and engine state of the agent on this host.
# data_dir = "/var/lib/quoro-node"

An existing config is never rewritten with a new one. When it has no engine line, the installer logs that Quoro will run the fake engine and how to switch. On a primary, a config with no [node] table gets [node] and local = false appended, so the control plane stops trying to run the node agent itself; a config whose [node] table has no local = false is left alone with a warning, because quoro serve would then run the agent unprivileged and refuse.

After installing, the script prints the binary’s quoro version output. On a primary it also prints a one-time setup token from quoro setup-token (single use, expires after one hour, needed by POST /api/v1/setup, which the panel serves at https://<host>:8443/ with a self-signed certificate) and this host’s TLS fingerprint, read from journalctl -u quoro. The token is printed to the terminal only; it is never written to a file. Get a fresh one any time with:

sudo quoro setup-token

On a node it prints the node id from /var/lib/quoro-node/identity.json; the node then shows as connected on the primary’s Nodes page.

Back up the cluster key files under /var/lib/quoro separately from the database (ADR 0050): a database copy does not contain them, and losing the data-encryption key loses every stored secret.

Privileges (ADR 0089)

Two processes, two privilege levels, and neither can borrow the other’s:

• quoro.service (the control plane: API, panel and node sessions) runs as quoro-api with an empty capability bounding set and AmbientCapabilities= empty, so no capability can be gained; NoNewPrivileges=yes, ProtectSystem=strict and /var/lib/quoro as its only writable path. Port 8443 needs no capability. It signs desired state and never touches a hypervisor, a bridge or a firewall.
• quoro-node.service (the node agent) runs as root, because the engine creates the bridge, TAP devices and nftables rules and converts images with ownership preserved. It is the only privileged Quoro process.

The installer never hands /var/lib/quoro to a person: if a quoro-api account already exists with a uid of 1000 or above, it stops before writing anything.

Hosts installed by the M1 installer have /var/lib/quoro owned by root. The installer stops quoro.service before it changes that ownership, gives every file to quoro-api except engine/, and leaves engine/ owned by root: it is the M1 engine state, which the control plane must not read. The node agent keeps its engine state in /var/lib/quoro-node/engine from then on, and VMs that M1 created are not adopted.

Idempotency

Re-running the installer is safe:

• quoro.toml is created only when absent; an existing config is kept as is and reported as kept existing /etc/quoro/quoro.toml. On a primary the only edit is the appended [node] table when it is missing.
• The binary, the units and the engine runtime files are compared with cmp -s before replacement; when unchanged they are reinstalled identically and the services are left running. The connector at /usr/local/lib/quoro/cloudflared is compared the same way.
• /var/lib/quoro-node is created or confirmed as root:root 0700, never recreated. /var/lib/quoro is the same for quoro-api:quoro-api 0700, and the ownership change is skipped when the directory already has that owner.
• A node that is already enrolled is not enrolled again, so no join token is needed on a re-run.
• Each unit is restarted only when the binary changed, its unit changed, an engine runtime file changed, the connector changed, or it is not active; the installer then waits up to 20 s for it to stay active for five consecutive seconds (so a refusal right after start is caught) and fails with that unit’s journalctl -u <unit> -n 50 hint otherwise. On a primary quoro.service starts before quoro-node.service, because the control plane writes the join file its own agent enrolls with.

Virtualisation and the engine

The engine needs /dev/kvm and hardware virtualisation (vmx/svm), and it writes /proc/sys/net/ipv4/ip_forward. It runs inside quoro-node.service, which is root because it manages a bridge, TAP devices and nftables rules; the unit keeps hardening that the engine tolerates (NoNewPrivileges, ProtectHome, PrivateTmp, ProtectSystem=full). When the host cannot run the engine, quoro node exits 78 and the unit stays failed (RestartPreventExitStatus=78) instead of restarting in a loop; read journalctl -u quoro-node. KillMode=process is deliberate: stopping or restarting the agent does not stop the VMs, and the next agent adopts them from its engine state.

Uninstall

A primary runs both units, a node only the agent. Stop and remove what the host has:

sudo systemctl disable --now quoro-node
sudo systemctl disable --now quoro          # primary only
sudo rm -f /etc/systemd/system/quoro-node.service
sudo rm -f /etc/systemd/system/quoro.service  # primary only
sudo systemctl daemon-reload
sudo rm /usr/local/bin/quoro
sudo rm -rf /usr/local/lib/quoro

systemctl stop quoro-node leaves every VM running on purpose (KillMode=process): stopping the agent is not stopping the workloads, and the next agent adopts them. To stop the VMs as well, stop them from the primary first, or stop the hypervisors directly:

sudo pkill -f /usr/local/lib/quoro/cloud-hypervisor

The engine leaves its network devices behind; remove them too:

sudo ip link delete quoro0
sudo nft delete table ip quoro_quoro0

The table is named quoro_<bridge>, so quoro0 (the default bridge) gives quoro_quoro0.

/var/lib/quoro-node (identity, credential, private key and engine state) and /etc/quoro (config) are kept on purpose on both roles, and /var/lib/quoro (database and cluster key files) is kept on a primary: remove them deliberately if that is what you want, and back up the key files under /var/lib/quoro first. Stop the VMs before deleting a node’s directory, otherwise the hypervisors keep running without their state:

sudo pkill -f /usr/local/lib/quoro/cloud-hypervisor
sudo rm -rf /var/lib/quoro-node

A node that is removed this way holds a credential the primary still knows about; remove the node on the primary too. On a primary the installer’s system account goes with it:

sudo userdel quoro-api

Hosts installed by an older installer also have a quoro system user that this installer no longer creates or uses; remove it, after the services are stopped, with:

sudo userdel quoro

Quoro: Product Requirements (v2)

Status: draft for review. Source of truth: the decision ledger; where this document and an accepted decision disagree, the decision wins.

1. Summary

Quoro is one self-hosted tool that unifies a hypervisor with application deployment [D-0002]. One engine, built on Cloud Hypervisor and KVM, runs one microVM per app, and the same interface deploys, exposes and observes those apps: push-to-deploy, domains, TLS, logs [D-0002, D-0008]. The agent is not an add-on: agents are first-class operators through MCP and events from day one, reading the same state as the panel, acting through the same API and being logged the same way [D-0002].

Quoro starts as a self-use and experimenter tool: anyone can install it on a box they have and experiment [D-0003]. Enterprise and managed offerings come later [D-0003].

The core is AGPL-3.0. Enterprise code, when it exists, lives in ee/ under a separate license, and a CLA is required from the first external contributor [D-0004].

2. Problem

A self-hoster who wants both virtual machines and application deploys today stacks two tools: Proxmox for the hypervisor, then a Debian VM inside it running Dokploy or Coolify for the apps [D-0002]. The result is two dashboards, two mental models and two upgrade paths, and neither of them was designed with agents in mind [D-0002].

The tools that do cover both layers are Kubernetes-heavy and aimed at hosting providers rather than at one person with a few machines: Harvester with Rancher, and Cozystack [D-0002, F-0006]. They solve a different problem at a different cost.

What is missing is a single tool that owns the VM layer and the deploy layer, keeps the operational model small, and treats an AI agent as an ordinary, auditable operator rather than as an external script with root [D-0002, D-0029, D-0031].

3. Users and personas

Primary user now: the self-use / experimenter. Someone who has a machine or a VPS, installs Quoro, and tries to run a real app on it [D-0003].

Three personas follow from that [D-0003]:

(a) Indie developer / advanced homelabber. Runs a home server, often behind NAT, and wants deploys, domains and TLS without opening ports or fighting certificates. For this user the ingress answer is Cloudflare Tunnel [D-0024].

(b) Small startup on its own hardware or VPS. Has a public IP and wants a Railway/Vercel-like experience on infrastructure it controls [D-0003]. For this user the ingress answer is a reverse proxy with ACME [D-0024]. They care about reliability: app replicas for stateless apps, a warm copy of stateful data, and an honest statement of what each policy costs [D-0020, D-0021].

(c) Team already operating with AI agents. Already has agents doing operational work and wants those agents to act on infrastructure with scoped permissions, human approval for destructive steps, and a complete audit trail [D-0029, D-0030, D-0031].

Enterprise and managed customers are explicitly later [D-0003]. Nothing in the MVP is designed around them.

4. Goals

• One interface for VMs and app deploys. Create and manage VMs, and deploy services with push-to-deploy, domains, TLS and logs, from the same place [D-0002].
• Agents as first-class operators from day one. MCP and events are part of the product, not a later integration; the agent reads the same state, uses the same API and is logged the same way as a human [D-0002, D-0029, D-0030, D-0031, D-0032].
• Apps keep running when the control plane dies. Nodes are autonomous, store their desired state locally and reconcile it; they dial out to the control plane and never the reverse [D-0010, D-0011].
• One binary, one-command install. API, panel and engine ship in the same binary; the installer is a single command, and POST /api/v1/setup requires a one-time setup token [D-0012, D-0054, D-0033].
• Works across VPS from different providers over the internet. Independent nodes, not a tightly coupled cluster that needs low-latency links [D-0010].
• Honest, explicit reliability trade-offs per app. Each app carries an HA policy with its cost shown, and the system states when it cannot promise something [D-0020, D-0021].

5. Non-goals (for now)

The following are out of scope and each of them could eat the project [D-0034]:

• Live migration of running VMs [D-0034].
• Distributed storage such as Ceph [D-0034].
• Complex networking: SDN, VLANs [D-0034].
• Multi-tenancy with billing [D-0034].
• Writing our own VMM [D-0034, D-0007].

Also out of scope for now:

• Docker on nodes. Apps run in microVMs, not containers on the host [D-0008].
• Running our own image registry. A curated catalog in the repo points at pinned upstream images instead [D-0026].
• A container fallback engine in the MVP. That is a later Engine implementation [D-0009].
• Automatic failover before M6 [D-0016, D-0033].
• Persistent app data before M5a: local, non-replicated volumes ship in M5a [D-0052]; until then a deployed app has no persistent disk.

6. Product principles

Agent-operable. The agent is one more operator. It reads the same state as the panel, acts through the same service layer and API, and appears in the same audit log. MCP has no separate path and no separate permissions model [D-0002, D-0031].

Least privilege, never forever. Every agent action runs inside a grant: a scope, a resource selector and a TTL, optionally a maximum number of uses. Destructive or irreversible actions are off by default and need a human just-in-time approval that issues a short-lived extra grant [D-0030].

Everything auditable. The audit log is append-only and hash-chained, can be copied off-site to object storage with object lock, which ships in M5a [D-0031, D-0053], and records who acted, on whose behalf, triggered by what, through which tool or endpoint, with which parameters, with which result, and with the agent’s stated justification [D-0031].

Minimal, auditable dependencies. Rust for everything that touches infrastructure, with every dependency justified [D-0056]. We write our own engine on Cloud Hypervisor and KVM rather than depending on Incus or writing a VMM; the engine sits behind an Engine trait so other backends stay possible [D-0007].

Reliability honesty. The data plane never depends on the panel [D-0011]. HA needs at least three votes, and Quoro says so instead of pretending a two-node setup is safe [D-0017]. The panel shows the cost of each HA policy [D-0020] and an N+1 capacity indicator [D-0021]. Failback is never automatic [D-0022].

Self-hosted for real. The static panel can be hosted anywhere, but the stateful API stays self-hosted [D-0012]. There is no vendor lock-in: DNS, LB and object storage sit behind provider interfaces, and Cloudflare is the first implementation, not the only one [D-0025].

Supply-chain safety. Binaries are signed and builds reproducible, because the installer is a pipe to a shell and the tool controls infrastructure [D-0027]. The catalog pins upstream images with verifiable signatures [D-0026], and its format, owner, location and signature scheme are decided at M4 planning, before the first template ships [D-0070].

7. Feature set mapped to milestones

Every milestone below is exactly as recorded in the roadmap decision; nothing is moved between milestones [D-0033].

HA and backups are out of the MVP, but the MVP architecture prepares them [D-0033]. For example, self-fencing is part of the node-agent protocol from M0 so it is not redesigned later [D-0019].

M3 tags v0.1.0 as an experimental preview with signed builds, not for production; M4 complete is the first general release and the broad announcement [D-0064]. The domain and trademark checks (Q-0008) must be done before the M3 tag [D-0064].

Definitions of Done for each milestone live in docs/product/roadmap.md.

8. The AI-native loop

The product is built around one loop: detect, notify, wake, act [D-0032].

Detect. Each node agent plus health checks report state and metrics [D-0032]. Because telemetry stays on the node and is queried through the primary on demand, detection does not depend on shipping logs to a central place [D-0014].

Notify. Webhooks are configurable per event. Examples from the decision: VM down, build failed, disk full, certificate expiring [D-0032].

Wake. The webhook starts an agent, for example one built on the Claude Agent SDK, with Quoro’s MCP server and the incident context attached [D-0032]. The reference runner lives in examples/agent-runner/, outside the quoro binary: it verifies the webhook signature, holds the agent’s own key and acts through MCP within the incident grant [D-0049]. The agent is a registered identity with a name, a model and an owner [D-0029].

Act. The agent can do only what its grant allows. A destructive step requires human approval in the panel, after which the agent receives a short-lived extra grant [D-0030]. Every call lands in the audit log through the same service layer a human uses [D-0031].

Worked example

The demo that explains the product in one sentence: “I told an agent the app was down, it restarted it, and it is in the audit log” [D-0033].

Grant: app:restart on project=blog, TTL one hour [D-0030]. Note that this is an incident-scoped grant: when an event woke the agent it received only the scopes needed, and the grant expires on its own [D-0030].

The resulting audit entry contains [D-0031]:

Revoking the grant locks the agent out within seconds, and because the log is hash-chained any later tampering with the entry is evident [D-0030, D-0031].

9. Reliability, HA and the witness

9.1 Architecture recap

Nodes are independent and orchestrated by Quoro, not clustered Incus-style, so that a deployment can span VPS from different providers over the internet [D-0010]. Nodes are autonomous and dial out to the control plane; each stores its desired state locally and reconciles it, so apps keep running and self-restart if the control plane disappears [D-0011].

Every node runs the same binary containing API, panel and engine. Roles are node, witness and edge, and they are combinable [D-0012]. Exactly one node is the primary: the single source of truth for state. The other nodes are replica nodes: they hold a copy, serve reads and forward writes to the primary [D-0012]. State moves as a continuous, append-only, numbered, hash-chained log; a replica node asks for everything after N and verifies the chain, which makes tampering evident [D-0013].

Telemetry — app logs and metrics — stays on each node and is not replicated; the panel queries it through the primary on demand, with optional archive to object storage [D-0014].

9.2 Leader election roadmap

Single node in the MVP; manual promotion of a replica node to primary ships in M5b, alongside leases and fencing [D-0016, D-0033, D-0067]; automatic failover with Raft and at least three votes later, in M6 [D-0016, D-0033]. Manual promotion means a command or a button, or an agent doing it with human approval [D-0016]. Automatic election with only two nodes causes split-brain, because after a network cut both sides can believe they are primary [D-0016].

9.3 Why three votes, and the witness

Majority voting needs an odd number of voters. With two nodes Quoro tells the user that automatic failover is not safe, and offers a witness [D-0017]. The witness only breaks ties and stores no data [D-0017].

Witness options [D-0017]:

1. Self-hosted lightweight daemon, free, on any machine in a different failure domain. The failure domain matters: a home PC sitting next to node A fails together with node A and is not a witness.
2. Optional external lease arbiter: a Cloudflare Durable Object (single-threaded actor with strongly consistent storage and alarms; needs a Cloudflare account and a deployed Worker, and fencing tokens are still required) or S3/R2 conditional writes (atomic If-Match/If-None-Match PUT with one winner, 1 write/s per key, no TTL and fencing tokens required; see docs/research/durable-objects.md and docs/research/r2-conditional-writes.md).

Free-witness rule. The self-hosted witness is free and first-class [D-0017]. A user with two nodes, a self-hosted witness and nothing else must be able to run a correct HA setup, and the panel must not hint otherwise.

9.4 Edge as the third voter

The edge/LB can be the third voter by combining the witness and edge roles in the same binary [D-0018]. It sits in a third location, sees both nodes, and decides where traffic goes, so its health view is the one users actually experience [D-0018]. The natural shape is two VPS plus one small LB VPS [D-0018].

Caveat: a single edge is a traffic single point of failure [D-0018]. Mitigations are Cloudflare as the traffic edge with the small VPS acting only as witness, or two edges with DNS failover [D-0018].

9.5 Fencing

Fencing is mandatory and lease-based, and the node agent self-fences from M0 [D-0019]. A lease is a node’s right to run a stateful app. expiring leases (warm/hot) end when they are not renewed in time; local leases (none/cold) end only by an explicit revoke or an operator declaring the node lost [D-0047]. When the arbiter is unreachable an expiring lease first becomes uncertain (keep running, start nothing new) and only then expired (self-fence); see docs/contracts/node-agent-protocol.md §8. The reason is concrete: starting a stateful app on node B while it still runs, isolated, on node A produces two diverging copies and therefore data corruption [D-0019]. The same lease mechanism serves both primary election and per-app failover, which is why it has to be in the node-agent protocol from the beginning instead of being retrofitted later [D-0019].

9.6 HA policies

Each app carries one policy, and the panel shows what it costs [D-0020].

Replication is asynchronous by default; synchronous is optional for nearby nodes and adds latency to every write [D-0020].

Suggested defaults [D-0020]: stateless apps get app replicas behind the edge LB when there are at least two nodes; stateful apps get warm. For stateless apps, the edge LB stops sending traffic to the dead node, so the user sees zero downtime [D-0020].

The warm RTO is the microVM boot time. Cloud Hypervisor’s own claim of a boot “under 100 ms” is unbenchmarked and stays a Hypothesis (see docs/research/cloud-hypervisor.md); the design assumes fast boot makes warm nearly as good as hot [D-0008].

The N+1 capacity indicator answers a different question in plain language: “if node A dies, do its apps fit on the others?” For example, “yes”, or “no, you are missing 6 GB of RAM” [D-0021]. No self-hosted tool does this well today, and it is an ideal signal for an agent to watch [D-0021].

9.7 Sync engine

The sync engine is ours: control-plane log replication plus a content-addressed chunk engine for volumes [D-0023]. A volume is snapshotted, split into content-addressed chunks — deduplicated and encrypted — and only new chunks are shipped; to another node this is warm replication, to S3 or R2 it is a cold backup [D-0023]. The first implementation uses incremental snapshots every few seconds, so RPO equals the interval [D-0023].

Databases are the exception: they use their native replication orchestrated by Quoro — Postgres streaming for Postgres — offered as catalog templates, because block-level copies of a running database are only crash-consistent [D-0023]. SQLite databases and catalog templates are backed up by Quoro’s own chunk engine with client-side encryption, taken from consistent snapshots (the SQLite backup API or VACUUM INTO), not by Litestream, whose 0.3 line is unmaintained; Litestream with SSE-C may be added later as an optional backend for a lower RPO (see docs/research/rust-api-and-mcp-tooling.md) [D-0069].

The snapshot primitive underneath the chunk engine is, for now, reflink snapshots plus a full content-defined chunk scan for M5a cold backups; a spike comparing lvm-thin and btrfs before M5b chooses the primitive for warm, and the Engine trait’s snapshot_volume hooks keep it swappable [D-0062].

9.8 Failback and planned switchover

Failback is never automatic or immediate [D-0022]. When node A comes back it does not have the data B collected while A was down, and A may also hold writes B never received, so there are two histories [D-0022].

The sequence [D-0022]:

1. A returns fenced, checks its lease, and starts nothing stateful.
2. A rejoins following B, which is now the source of truth: for control-plane state A becomes a replica node, and for each stateful app A holds the warm copy while B is the lease holder.
3. A’s divergent writes are quarantined in a separate snapshot and recorded in the audit log. They are never silently dropped.
4. Incremental resync from the last common point; A becomes warm.
5. Default: no failback. The roles simply swap and B stays the lease holder.
6. If A is preferred, the operator runs a planned switchover (“relevo inteligente”): wait for replication lag near zero, wait for a low-traffic window chosen by the user or proposed by an agent with approval, freeze writes for seconds, run the final sync, move the lease, shift traffic at the LB, drain connections.

A planned switchover loses no data. Data loss of seconds happens only in surprise failures [D-0022]. For stateless apps the LB slow-starts traffic back to A [D-0022].

10. Ingress and providers

Two ingress paths cover the two main users [D-0024]: a reverse proxy with ACME for hosts with a public IP, and Cloudflare Tunnel for homelabs behind NAT, which avoids opening ports and fighting TLS.

The proxy is an embedded Rust reverse proxy — hyper, rustls and an ACME client crate — behind the ingress interface that the Cloudflare Tunnel backend also implements; Caddy driven through its admin API stays as a fallback backend if the M2 spike shows ACME is too costly [D-0059]. The maturity of the ACME crate is a research task before M2 [D-0059]. Both sit behind a common interface [D-0024].

Multiple cloudflared replicas on the same tunnel give host-level failover only; whether they expose any app health check is a Hypothesis to be confirmed, and per-node failover needs one tunnel per node behind a load balancer. Cloudflare Load Balancing is a paid add-on from $5/month, its HTTPS monitors with the app hostname test the app, there are no TCP monitors for tunnels, and an LB sees all replicas of one tunnel as one endpoint; see docs/research/cloudflare-lb-and-tunnel.md [D-0024].

Providers sit behind interfaces: DNS, load balancing and S3-compatible storage, with Cloudflare as the first implementation and not the only one [D-0025]. Backups can target any S3-compatible store, including R2, B2 and MinIO [D-0025].

11. Platform requirements

The MVP requires a KVM-capable host: bare metal, or a VPS that exposes nested virtualization [D-0009]. Many VPS do not expose nested virtualization, which is exactly why the requirement is stated up front rather than discovered at install time [D-0009]. A container fallback is a later Engine implementation and is more work with weaker isolation, so it is not a way around the requirement [D-0009].

Nested virtualization is the exception on cheap VPS: it is not available on Hetzner Cloud, Hostinger, Contabo VPS or netcup VPS, while it is available (some opt-in) on AWS 7i/8i families, GCP Intel, Azure Dv3/Dv4, OCI Intel VMs, OVHcloud Public Cloud, Contabo VDS, Vultr VX1 and DigitalOcean. Bare metal is the reliable path, and the installer must always probe /dev/kvm with a real microVM boot rather than trusting the provider’s documentation; see docs/research/vps-nested-virt.md.

Cloud Hypervisor is v54.0 under Apache-2.0 AND BSD-3-Clause, driven through an HTTP/JSON API over a UNIX socket, and the host kernel must be 6.8 or newer; snapshots hold RAM and device state but not disks, with no incremental API, virtio-fs needs an external virtiofsd, and consoles are text-only (see docs/research/cloud-hypervisor.md). The console is therefore a text console over the microVM’s serial or virtio-console, shown with xterm.js (Cloud Hypervisor has no VNC or framebuffer); consoles are text-only and there is no noVNC in the stack [D-0056].

12. Managed offering

An optional managed offering may come later [D-0003]; nothing in it is designed yet and no feature in this document depends on it. The self-hosted witness is free and first-class [D-0017].

13. Competitive landscape

Approximate stacks; verify before any pitch [F-0006].

Proxmox and Incus MCPs exist, but they are small community projects [F-0006].

Where Quoro sits: it covers both layers — hypervisor and app deploy — without Kubernetes, and it is agent-native by design rather than by integration [D-0002, F-0006]. The closest shape is “Proxmox plus Dokploy in one tool”, with a safety model for agents built into the core [D-0002, D-0029].

14. Risks

15. Open questions

All of the following are open. This document does not choose between the options.

The ingress proxy, Git build method, OCI-to-rootfs conversion and init, snapshot primitive, node mesh, first public release and manual promotion of a replica node are decided in D-0059 to D-0064 and D-0067; docs-site timing in D-0065, the HA policy costs in D-0068, encrypted SQLite and template backups in D-0069, and when the curated catalog is decided in D-0070.

Realtime transport is decided: SSE carries one-way streams (event stream, live logs, deployment progress) and WebSockets carry bidirectional sessions (terminal, VM console); see docs/architecture/architecture.md §13 [D-0057].

Answered verification questions

16. References

• docs/product/roadmap.md — milestones and Definitions of Done.
• docs/glossary.md — terms such as primary, replica node, app replica, lease holder, HA policies and node roles.
• Decision ledger (private) — the source of truth for every id cited here.

Quoro roadmap

Status: draft for review. Source: D-0033 (milestones), D-0034 (out of scope), D-0036 (done = runs on the lab).

This turns D-0033 into a gated plan and adds no features: where another decision explains how an item behaves, the decision is cited next to the item, and every claim that rests on an unverified external fact is either answered in docs/research/ or marked as a hypothesis.

Release callout: first public release (D-0064). M3 tags v0.1.0 as an experimental preview with signed builds, not for production; M4 complete is the first general release and the broad announcement [D-0064]. The domain and trademark checks (Q-0008) come before the M3 tag [D-0064].

1. How to read this roadmap

Gating rules

• A milestone starts only after the previous milestone’s demo ran on the lab and the maintainer approved it [D-0036].
• “Done” means it runs on the lab, not that it compiles [D-0036].
• Contracts (OpenAPI, DB schema, event schema, node-agent protocol, audit-log format, permission model, Engine trait) freeze at the end of M0. After that, a contract change is a standalone PR, never mixed into feature work [D-0036].
• Every PR closes a GitHub issue, and issues plus milestones are the task source of truth [D-0036].
• Every Definition of Done below is a functional check a person can run over SSH on the lab VMs. No performance figures appear here because the lab is nested virtualization and is not valid for benchmarks.

The lab

The test lab is three nested-virtualization Debian 13 VMs (lab-1, lab-2 and lab-3) on a single physical host, so they share one failure domain; it is not valid for performance benchmarks. Each VM is 2 vCPU / 4 GB RAM / 30 GB on a local disk, with CPU type host and nested KVM. Every agent and contributor tests over SSH.

All three VMs sit on one physical host and therefore share a failure domain. The lab can simulate separate failure domains by stopping VMs or cutting the network between them; the real requirement that a witness sits in a different failure domain than the nodes [D-0017] is documented by the lab, not proven by it.

Workstreams

After M0 the work splits into workstreams, each in its own crate or package to keep merges small: engine (Cloud Hypervisor, OCI to rootfs, disks and snapshots, networking, console), control-plane (Axum, auth, services, OpenAPI), panel (React against a mock generated from OpenAPI), node-agent (protocol, local reconciliation, self-fencing), ingress (proxy plus Cloudflare Tunnel behind one interface), mcp (rmcp over the same service layer as the API), and installer/releases/docs site (see docs/architecture/architecture.md §17). The later workstreams are the sync/chunk engine and witness/edge (see docs/architecture/architecture.md §17).

Where HA sits

High availability and backups are not part of the MVP, but the MVP architecture prepares them [D-0033]: the node-agent protocol carries self-fencing from M0 so it is not redesigned later [D-0019], the control plane replicates an append-only numbered hash-chained log from the start [D-0013], and one sync engine serves both node replication and backups [D-0023].

2. Overview

3. Milestones

M0 — Contracts

Scope

• Repository, CI and cargo workspace, plus the OpenAPI spec, DB schema and event schema [D-0033].
• Node-agent protocol, including self-fencing [D-0019]. A lease is a node’s right to run a stateful app. expiring leases (warm/hot) end when they are not renewed in time; local leases (none/cold) end only by an explicit revoke or an operator declaring the node lost [D-0047]. When the arbiter is unreachable an expiring lease first becomes uncertain (keep running, start nothing new) and only then expired (self-fence); see docs/contracts/node-agent-protocol.md §8.
• The spike “boot a microVM from an OCI image” [D-0033], on lab-1, using Cloud Hypervisor and KVM [D-0007].
• The bases documents: PRD, ADRs, architecture, contracts, threat model, roadmap, workstream briefs, glossary, OSS files [D-0040].
• The Engine trait, so other backends can be added later without rewriting the control plane [D-0007].

External facts behind this milestone are verified in the research files. Cloud Hypervisor: v54.0, Apache-2.0 AND BSD-3-Clause, an HTTP API over a UNIX socket, snapshots that hold RAM and device state but not disks, virtio-fs that needs an external virtiofsd, text consoles only (serial, virtio-console; no VNC or framebuffer) and a host kernel of 6.8 or later (docs/research/cloud-hypervisor.md); boot under 100 ms remains a Hypothesis. The OCI-to-rootfs conversion and the minimal init are ours: a minimal Rust init and our own OCI-to-ext4 pipeline, with layer paths resolved inside the rootfs (openat2 with RESOLVE_IN_ROOT), zstd layers, and config delivered through a config disk or the kernel command line [D-0061]. utoipa 6 with orval or openapi-ts generates usable client code from the spec, with one known orval break (docs/research/rust-api-and-mcp-tooling.md).

Definition of Done

• On lab-1, a clean clone builds the whole workspace and the CI validation script (OpenAPI, DB schema, event schema, node-agent protocol) exits 0. The same validation runs in CI.
• Without KVM, two tests pass against an in-process primary and a fake engine: revoking a local lease fences its VM, and a test-only short expiring lease expires and fences within the bound in node-agent-protocol.md §8.4 (both tests are required by §8.7).
• The node’s lease state machine implements held -> uncertain -> expired and revoked as in node-agent-protocol.md §8.4; uncertain never stops a stateful app.
• On lab-1, the spike boots an OCI image converted to a root disk inside a Cloud Hypervisor microVM and the guest’s serial console prints the expected marker [D-0033].
• On lab-1, the contracts directory contains OpenAPI, DB schema, event schema, node-agent protocol, audit-log format, permission model and Engine trait, each listed in the contracts index [D-0040].
• Every contract element tied to an accepted decision is written as decided; the only remaining open items are listed in the contracts README and none blocks the freeze.

Demo script

1. On lab-1, run the workspace build and the contract validation. Expected: exit code 0 with each schema reported as valid.
2. On lab-1, run the OCI spike: convert the image, start the microVM, attach to the serial console. Expected: the guest boots and prints the marker line.
3. Run the lease tests from the node-agent protocol. Expected: both pass — revoking a local lease fences its VM, and a test-only short expiring lease expires and fences within the bound in node-agent-protocol.md §8.4.
4. Show the contracts index and confirm that contracts freeze at the end of M0. Expected: the maintainer approves the freeze.

Dependencies

• Nothing precedes M0. The rootfs pipeline (mkfs.ext4 -d plus an OCI registry client) is verified by the M0 spike.
• Q-0003 (rootfs conversion and minimal init) is answered by the M0 spike: our own conversion pipeline and init, validated there [D-0061].
• Realtime transport, lease modes, text-only consoles, the OpenAPI source of truth, provider credentials, cluster keys and the host layout with the one-time setup token are decided [D-0057, D-0047, D-0056, D-0048, D-0051, D-0050, D-0054]. Custody of the cluster keys on promotion stays open and is decided with promotion in M5b [D-0067]; it does not block the freeze.
• Contracts freeze at the end of M0; any later change is a standalone PR [D-0036].

Explicitly out of scope

• Any deploy flow: no node agent talking to a control plane, no domain, no TLS. That is M2.
• Any MCP work: agents appear in M3.
• CI running on a self-hosted runner: deferred, because a self-hosted runner would let fork PRs on a public repository run arbitrary code on the project’s infrastructure.

M1 — Skeleton

Scope

• One-command installer [D-0033].
• Engine lifecycle: create, start, stop and delete a VM [D-0033].
• Basic panel [D-0033].
• Signed release binaries and a reproducible build, because the installer is curl | sh and the tool controls infrastructure [D-0027].
• The panel is built against a mock generated from the frozen OpenAPI spec (see docs/architecture/architecture.md §17).

Definition of Done

• On a freshly reinstalled lab-1, the one-command install completes and quoro answers a version command.
• On lab-1, creating a VM through the API returns an id, starting it makes it appear as running, and stopping then deleting it leaves no VM behind.
• On lab-1, the same create/start/stop/delete sequence works from the panel in a browser, with the state shown in the list after each step.
• On lab-2, the release binary’s signature verifies against the published signer identity and the Sigstore trust root [D-0027, D-0081].
• On lab-2, rebuilding from the same source produces a byte-identical artifact to the published one [D-0027].

Demo script

1. Reinstall lab-1 from the base image, log in over SSH, run the one-command installer. Expected: it finishes without manual steps and prints the version.
2. Open the panel, create a VM, start it, watch it become running. Expected: the list shows the new state.
3. Stop the VM and delete it. Expected: the list is empty again.
4. On lab-2, verify the signature of the downloaded binary and rebuild from source. Expected: signature valid, hashes match.

Dependencies

• M0 contracts, frozen.
• D-0027 shapes the release step.
• Cloud Hypervisor’s HTTP API over a UNIX socket is the engine’s start/stop path (docs/research/cloud-hypervisor.md).

Explicitly out of scope

• Deploying an application image with a domain and TLS: M2.
• Live logs: M2.
• Any HA behaviour: M5a onwards.

M2 — Deploy

Scope

• Node agent: it stores desired state locally and reconciles it, dialing out to the control plane and never being dialed into [D-0011].
• Deploy an OCI image with a domain and TLS [D-0033].
• Live logs in the panel, streamed over SSE [D-0033, D-0057].
• Ingress: a reverse proxy with ACME for hosts that have a public IP, or Cloudflare Tunnel for homelabs behind NAT, both behind one interface [D-0024]. The lab has no public IP, so ACME HTTP-01 cannot run there; lab TLS goes through Cloudflare Tunnel, with ACME exercised separately through DNS-01 [D-0055].
• The control plane stays the single writer; on a single node that node is the control-plane primary [D-0012].

The ingress backend is an embedded Rust reverse proxy — hyper, rustls and an ACME client crate — behind the ingress interface that the Cloudflare Tunnel backend also implements; Caddy driven through its admin API stays as a fallback backend if the M2 spike shows ACME is too costly [D-0059]. The maturity of the ACME crate is a research task before M2 (Hypothesis until it is verified) [D-0059]. The node mesh is not needed here: the control-plane session is TLS dial-out (D-0011), and the mesh is first needed in M5a for cross-node replica traffic [D-0063]. Cloudflare Tunnel behaviour: the load balancer sees all replicas of one tunnel as one endpoint and cloudflared replicas have no app health check, so per-node failover needs one tunnel per node behind a load balancer (docs/research/cloudflare-lb-and-tunnel.md).

Definition of Done

• Roles: lab-1 runs the control plane (control-plane primary); lab-2 runs a node agent that dials out to lab-1 [D-0011, D-0012].
• On lab-2, deploying an image through the panel yields an app that answers HTTP 200 on its domain.
• On lab-2, the app is served over TLS for its domain through Cloudflare Tunnel with a test domain the maintainer controls [D-0024, D-0055].
• On the lab, the ACME path issues a certificate through DNS-01 against a test zone (separate ingress task) [D-0055].
• On lab-2, the panel’s live log view shows new lines within the same session while the app writes to stdout; live logs are streamed over SSE [D-0057].
• Stop the control plane on lab-1; the app on lab-2 keeps answering HTTP 200 and the node agent keeps reporting its desired state locally [D-0011].
• On lab-2, kill the app’s process while the control plane on lab-1 is still stopped; the node agent restarts it and the domain answers 200 again [D-0011].

Demo script

1. On lab-2, install Quoro and let the node agent connect out to the control plane on lab-1. Expected: the node appears as connected in the panel.
2. Deploy an OCI image with a domain assigned; then open live logs and trigger a request. Expected: the app answers 200 on that domain with a valid certificate, and the request line appears in the log view.
3. Stop the control plane on lab-1. Expected: the app on lab-2 still answers 200.
4. Kill the app process on lab-2. Expected: the node agent restarts it and the domain answers 200 again.
5. Start the control plane again and check the node list. Expected: no duplicate app.

Dependencies

• M1 lifecycle and installer.
• An ACME/TLS crate spike before M2 (obtain and renew one certificate).
• D-0011 (dial-out and local reconciliation) and D-0024 (ingress).
• Provider credentials (Cloudflare API and tunnel tokens, DNS zones, S3 keys) are entered and stored through the human-only providers API (/api/v1/providers, scope provider:manage) with encrypted secrets [D-0051].
• Needed before or during M2: an ACME crate maturity check, with the Caddy admin-API backend as the fallback [D-0059].

Explicitly out of scope

• Agents and MCP: M3.
• Several nodes acting as replica nodes of the control plane: the manual promotion of a replica node arrives in M5 (D-0016), and automatic election is M6.
• Git builds: M4.

M3 — AI-native

Scope

• MCP server exposing read plus restart and redeploy [D-0033], built on rmcp over the same service layer as the API (see docs/architecture/architecture.md §17).
• Agents as first-class identities, separate from humans, recording who, on whose behalf and why [D-0029].
• Grants with scope, resource selector, TTL and optional maximum uses; incident-scoped grants that auto-expire; just-in-time approval for destructive actions [D-0030].
• Audit log, append-only and hash-chained, written by the same path as the API; destructive actions and refusals included [D-0031].
• Event webhooks per event, so a webhook can wake an agent with incident context [D-0032].
• Documentation stays Markdown in the repository; the docs-site tooling is picked during M3 planning together with the v0.1.0 preview [D-0065].

rmcp 3.8 is the official Rust MCP SDK, with three semver-major releases between March and July 2026, so the MCP surface is pinned to a checked version (docs/research/rust-api-and-mcp-tooling.md).

Release callout (D-0064). This milestone ends with the v0.1.0 tag: an experimental preview with signed builds, not for production [D-0064]. M4 complete is the first general release and the broad announcement [D-0064]. The domain and trademark checks (Q-0008) must be done before this tag [D-0064].

Definition of Done

• On lab-1, tell an agent the app on lab-2 is down; the agent restarts it through MCP, the app answers 200, and the audit log holds an entry naming the agent, the human who authorized it and the reason [D-0033, D-0029].
• On lab-1, an agent holding a read-only grant attempts a restart and is refused; the refusal appears in the audit log [D-0030, D-0031].
• On lab-1, an agent whose grant TTL has passed is refused on the next call, with the reason shown as grant expired [D-0030].
• On lab-1, revoke an agent’s grant in the panel; the agent’s next call is refused [D-0030].
• On lab-1, an agent attempts a delete without approval; instead of executing, the panel shows a pending approval request, and approving it lets the action run under a short-lived extra grant [D-0030].
• On lab-1, the audit chain verifies from the first entry to the last; after one entry is edited in the local copy, verification reports the exact entry as broken [D-0013, D-0031].
• On lab-1, an event webhook fires on an app-down event and an agent is woken with the incident context [D-0032].
• On lab-1, the reference agent runner in examples/agent-runner/ receives the wake webhook, verifies its signature, mints an access token with its own agent key and acts through MCP within the incident grant, passing incident_id and justification [D-0049, D-0032].

Demo script

1. On lab-2, stop the app’s process and disable the node agent’s restart for the demo, so the app stays down; then wake the agent through the app-down webhook. Expected: the agent reads state over MCP and restarts the app.
2. Load the app’s domain. Expected: HTTP 200.
3. Open the audit log in the panel. Expected: one entry with agent name, model, owner, on-behalf-of human, triggering event, tool, parameters and result [D-0029, D-0031].
4. Try the same restart with a read-only grant and again with an expired grant. Expected: both refused and both audited.
5. Ask the agent to delete the app. Expected: a pending approval request appears instead of a deletion.
6. Run audit verification, then edit one audit line in a copy and verify again. Expected: clean pass, then a reported break at that entry.

Dependencies

• M2 (a working deploy and node agent to operate on).
• D-0029 to D-0032 define the model; D-0013 supplies the replicated log the audit chain rides on.
• Axum SSE and WebSocket routes verified before the M3 event stream.
• The agent runner that receives the wake webhook and starts an agent against Quoro’s MCP endpoint is a reference example in examples/agent-runner/, outside the quoro binary and owned by the MCP workstream [D-0049]. It verifies Quoro-Signature on the delivery, reads the event, WebhookWake.agent_id and incident_id, holds that agent’s own agent key configured by the operator at the runner (Quoro never sends credentials in a webhook), mints a 5-minute access token through POST /api/v1/auth/agent-token, and acts through Quoro’s MCP endpoint within the incident grant, passing incident_id and justification on every mutating tool [D-0049].
• Realtime transport is decided: SSE for one-way streams (event stream, live logs, deployment progress) and WebSockets for bidirectional sessions (terminal, VM console) [D-0057].

Explicitly out of scope

• Metrics dashboards and the web terminal: M4.
• Off-site copy of the audit log to S3/R2 with object lock [D-0031]: ships in M5a with the S3-compatible path [D-0053].
• Agents approving their own destructive actions: never allowed by design [D-0030].

M4 — Full experience

Scope

• Git builds, starting with Dockerfile-only builds [D-0033, D-0060]. A BuildKit daemon runs inside an ephemeral builder microVM (Hypothesis: not yet tried under Cloud Hypervisor) and its output goes to a node-local OCI layout, which then enters the normal OCI-to-rootfs path [D-0060]. Railpack and Nixpacks are re-verified before M4 (Hypothesis: Nixpacks is in maintenance and Railpack is its successor), and a zero-config builder may be added later as a second Builder implementation [D-0060].
• Metrics per app and per node, surfaced in the panel; telemetry and app logs stay on each node, only state is replicated [D-0014].
• Web terminal into the panel [D-0033].
• VM console for a microVM: a text console over the microVM’s serial or virtio-console socket, shown with xterm.js (docs/research/cloud-hypervisor.md: no VNC or framebuffer); consoles are text-only and there is no noVNC in the stack [D-0056].
• Decide the curated catalog’s format, owner, location and signature scheme at M4 planning, before the first template ships [D-0070].

Release callout (D-0064). M4 complete is the first general release and the broad announcement [D-0064]; the experimental v0.1.0 preview is tagged at the end of M3. This milestone still needs its demo and approval before M5a [D-0036].

Definition of Done

• On lab-2, point an app at a Git repository with a Dockerfile; the build runs, produces an image and the app answers 200 on its domain.
• On lab-2, a build that fails shows the failing step and its log output in the panel.
• On lab-1, the panel shows at least CPU and memory for lab-2 and for one app on it, refreshed without a page reload [D-0014].
• On lab-2, open the web terminal, run a command and see its output; the session is recorded in the audit log [D-0031].
• On lab-2, open the VM console of a running microVM and see the guest’s boot output on the serial console.

Demo script

1. Create an app from a Git repository with a Dockerfile. Expected: the build runs and the app answers 200.
2. Push a commit that breaks the build. Expected: the panel shows the failing step and its log output.
3. Open the metrics view for lab-2 and for the app. Expected: CPU and memory values appear and update.
4. Open the web terminal, run uname -a in the app’s context, then open the VM console. Expected: the terminal output appears and is in the audit log, and the console shows guest boot or login output.

Dependencies

• M3 (audit and grants, so terminal and console access are attributable).
• D-0014 constrains where telemetry lives.
• Standalone contract PRs for the web terminal, VM console and metrics endpoints, merged before the feature work (D-0036).
• Needed before or during M4: Dockerfile-only builds, with Railpack and Nixpacks re-verified before M4 [D-0060].
• This milestone is the first general release [D-0064].

Explicitly out of scope

• App replicas and edge load balancing: M5a.
• Any backup or restore UI: M5a.
• A docs site: documentation stays Markdown in the repo until the first public release, and the docs-site tooling is picked during M3 planning together with the v0.1.0 preview (Q-0009, D-0065). It belongs to the installer/releases/docs-site workstream, not to a product milestone (see docs/architecture/architecture.md §17).

M5a — Stateless HA and backups

Scope

• Stateless app replicas: extra running instances of a stateless app, placed on several nodes and put behind the edge load balancer, so the edge stops sending traffic to a dead node [D-0020].
• Edge load balancing, with the edge as its own role of the same binary and able to combine with witness [D-0012, D-0018].
• Backups to S3 or R2, at the cold policy: backups only [D-0020, D-0023].
• The content-addressed chunk engine, used for backups first [D-0023].
• Backups of the control-plane SQLite database and of catalog templates go through that same chunk engine with client-side encryption, taken from consistent snapshots (the SQLite backup API or VACUUM INTO), not through Litestream [D-0069].
• HA policies are named none, cold, warm and hot, and the panel shows what each costs [D-0020].
• Local, non-replicated persistent volumes: create, attach to an app, survive redeploy [D-0052].
• Off-site copy of the audit log to an S3-compatible bucket with object lock [D-0031, D-0053].

Cloudflare Load Balancing is a paid add-on from $5/month; an HTTPS monitor on the app hostname tests the app; the load balancer sees all replicas of one tunnel as one endpoint; cloudflared replicas have no documented app health check (Hypothesis); per-node failover needs one tunnel per node behind a load balancer (docs/research/cloudflare-lb-and-tunnel.md).

D-0016 places manual promotion of a replica node to primary in M5, and it ships in M5b alongside leases and fencing [D-0016, D-0067]. Promotion is manual: a command or a button, or an agent doing it with human approval [D-0016].

Definition of Done

• With lab-1 and lab-2 as nodes and lab-3 as the edge, a stateless app runs two app replicas, one on each node, and the edge sends traffic to both.
• Power off lab-1; the app still answers 200 on its domain through the edge on lab-3, served by the app replica on lab-2 [D-0020].
• On lab-1, a volume is created and attached to an app; data written to it survives a redeploy [D-0052].
• With lab-1 back, take a cold backup of a stateful app running on lab-1 to an S3-compatible bucket, then restore it onto lab-2 and confirm the restored app serves the data that was present at backup time [D-0020, D-0023].
• On lab-1, the panel shows the cold policy’s cost for the app, and switching the policy changes the shown cost [D-0020].
• On lab-3, an edge health check removes lab-1 from rotation after lab-1 is powered off, and adds it back after lab-1 returns.
• On lab-1, one audit segment is exported with object lock to a bucket; an overwrite or delete of it is refused; truncating the tail of the local audit log is detected against the exported copy [D-0031, D-0053].

Demo script

1. Set up lab-1 and lab-2 as nodes and lab-3 as edge.
2. Deploy a stateless app with two app replicas and request its domain several times. Expected: the panel lists both replicas on different nodes, and responses come from both (each instance identifies itself).
3. Power off lab-1. Expected: the domain keeps answering 200, requests land on lab-2 only.
4. Power lab-1 back on and wait for the node to reconnect. Expected: the edge sends traffic to it again.
5. Take a cold backup of a stateful app to the bucket, then restore it on lab-2. Expected: the restored app serves the backed-up data.
6. Create a volume, attach it to an app and write a file to it; redeploy the app. Expected: the file is still there after the redeploy [D-0052].
7. Export one audit segment with object lock to the bucket, then try to overwrite or delete it. Expected: both are refused; truncating the tail of the local audit log is detected against the exported copy [D-0031, D-0053].

Dependencies

• M4.
• D-0020 (policies), D-0023 (chunk engine), D-0018 (edge as voter, relevant only from M5b), D-0012 (roles in one binary).
• Briefs for the sync/chunk engine and the witness/edge workstreams are written and approved before M5a starts.
• Local, non-replicated volumes ship in M5a [D-0052]; a standalone post-freeze contract PR adds the volume endpoints before M5a.
• A standalone contract PR adds the HA policy cost and N+1 capacity fields.
• The node mesh is kernel WireGuard configured by Quoro over netlink (no wg-quick), with a per-node keypair and peers plus allowed IPs in the desired state; boringtun in userspace is only a fallback and our own implementation is rejected [D-0063]. It is first needed in M5a for cross-node replica traffic; the control-plane session is TLS dial-out (D-0011), so M2 needs no mesh, and a publicly reachable peer acts as the relay hub for NAT [D-0063].
• The snapshot primitive the chunk engine reads from: reflink snapshots plus a full content-defined chunk scan for the M5a cold backups [D-0062].
• The off-site audit-log copy [D-0031] ships in M5a with the S3-compatible path [D-0053].
• Object lock in compliance mode must be verified per provider (R2, S3, B2, MinIO) before M5a (research task).
• Object storage choice: S3-compatible; R2 offers an atomic conditional PUT at 1 write/s per key with no TTL, and is only needed as a lease arbiter from M5b (docs/research/r2-conditional-writes.md).

Explicitly out of scope

• warm and hot: M5b and M6.
• Witness, leases and fencing for stateful apps: M5b.
• Automatic failover of the control plane: M6.
• Distributed storage such as Ceph [D-0034].

M5b — Witness, leases and warm replication

Scope

• Witness role: a lightweight daemon that only breaks ties and stores no data, in a different failure domain than the nodes [D-0017].
• Quoro reports that HA needs a third vote when only two nodes exist, and offers a witness [D-0017].
• Leases and lease-based fencing [D-0019]. A lease is a node’s right to run a stateful app. expiring leases (warm/hot) end when they are not renewed in time; local leases (none/cold) end only by an explicit revoke or an operator declaring the node lost [D-0047]. When the arbiter is unreachable an expiring lease first becomes uncertain (keep running, start nothing new) and only then expired (self-fence); see docs/contracts/node-agent-protocol.md §8.
• Replicated volumes at the warm policy: data continuously replicated, image pulled, instance stopped [D-0020], shipped by the content-addressed chunk engine [D-0023].
• The N+1 capacity indicator: “if node A dies, do its apps fit on the others?” [D-0021].
• Failback behaviour: a returning node rejoins fenced, starts nothing stateful, quarantines its divergent writes as a snapshot recorded in the audit log, resyncs incrementally, and by default no failback happens [D-0022].
• A planned switchover is available when the user prefers the original node: wait for lag near zero, wait for a low-traffic window, freeze writes, final sync, move the lease, shift traffic, drain connections; it loses no data [D-0022].
• The edge can also be the third voter, as the witness+edge combination of the same binary [D-0018].

External arbiters are optional and their behaviour is verified. R2 conditional writes as a lease arbiter: an atomic conditional PUT, 1 write/s per key, no TTL, and fencing tokens are still required (docs/research/r2-conditional-writes.md). Cloudflare Durable Objects as the arbiter: fits on the Workers Free plan, needs a Cloudflare account and a Worker, and fencing tokens are still required (docs/research/durable-objects.md). The snapshot primitive under the chunk engine is reflink snapshots plus a full content-defined chunk scan for the M5a cold backups; a spike comparing lvm-thin and btrfs chooses the primitive for warm before M5b, and the Engine trait’s snapshot_volume hooks keep it swappable [D-0062].

Failure domains: all three lab VMs run on one physical host. The lab therefore simulates separate failure domains by stopping VMs or cutting the network between them. The real requirement that the witness sits in a different failure domain [D-0017] is documented here, not proven by the lab.

Definition of Done

• Roles: lab-2 is the control-plane primary and a node; lab-1 is a node hosting the stateful app; lab-3 is added as witness during the test [D-0012, D-0017].
• With only lab-1 and lab-2 as nodes, Quoro reports that HA needs a third vote and offers to add a witness [D-0017].
• Add lab-3 as witness; Quoro then reports three votes, and lab-3 stores no application data.
• A stateful app runs at warm on lab-1 with its data shipped to lab-2; the panel shows the last shipped snapshot on lab-2.
• Cut lab-1 off from lab-2 and lab-3; lab-1 self-fences its stateful apps and they stop on lab-1 [D-0019].
• While lab-1 still holds the app’s lease, Quoro refuses to start the warm copy on lab-2 [D-0019].
• After lab-1’s lease is lost, an operator (or an agent with human approval) starts the warm copy on lab-2, and it contains the data of the last shipped snapshot [D-0016, D-0020, D-0023]. Automatic per-app failover is M6.
• Bring lab-1 back; it starts nothing stateful, its copy of the app becomes the warm copy following lab-2 (lab-2 is now the lease holder and source of truth), and its divergent writes appear as a quarantined snapshot recorded in the audit log [D-0022].
• Write a known record to the app, then run a planned switchover back to lab-1; it completes, the app serves from lab-1, and the record is present [D-0022].
• The N+1 indicator answers “if lab-1 dies, do its apps fit?” with yes, or with no plus the missing resources [D-0021].
• An operator promotes a replica node to primary; the promotion is refused while the old primary may still hold expiring leases unless the operator acknowledges the fence [D-0067].
• Before expiring leases ship, the concurrent-fence and mid-fence crash tests pass [D-0075].
• A held expiring lease survives an agent restart within the same host boot [D-0072].

Demo script

1. Start with lab-1 and lab-2 as nodes only and open the HA view. Expected: Quoro says a third vote is missing and offers a witness [D-0017]. Add lab-3 as witness. Expected: Quoro reports three votes and lab-3 shows no volumes.
2. Deploy a stateful app at warm on lab-1. Expected: the panel shows the data being shipped and lab-2 holding the last shipped snapshot.
3. Cut the network between lab-1 and the other two. Expected: lab-1’s app stops itself (self-fence). Try to start the warm copy on lab-2 before the lease is lost: refused. After the lease is lost, start it: it serves the last shipped data.
4. Reconnect lab-1. Expected: it starts nothing stateful, its copy becomes the warm copy following lab-2, and a quarantined snapshot of its divergent writes is in the audit log [D-0022].
5. Start a planned switchover to lab-1 in the panel, then open the N+1 indicator and ask “if lab-1 dies?”. Expected: the switchover completes with no data loss and the app serves from lab-1; the indicator answers yes, or no plus the missing resources [D-0021].

Dependencies

• M5a (edge, app replicas, chunk engine, bucket).
• D-0017, D-0018, D-0019, D-0020, D-0021, D-0022, D-0023.
• A standalone contract PR adds the HA policy cost and N+1 capacity fields.
• Needed before or during M5b: the lvm-thin versus btrfs spike that chooses the warm snapshot primitive [D-0062].
• This milestone does not deliver automatic promotion of the control plane; that is M6 [D-0016].

Explicitly out of scope

• Automatic control-plane election: M6.
• The hot policy: M6.
• Automatic failback: never, it is always planned and approved [D-0022].
• Automatic per-app failover: M6 [D-0033].

M6 — Automatic failover and hot

Scope

• Automatic election of the control-plane primary with a majority, using Raft semantics; automatic failover only with at least three votes [D-0016].
• The hot policy: the instance runs on another node and replicates, so a node loss is covered by a running copy [D-0020].
• Per-app failover rides on the same lease mechanism as the election [D-0019].
• Failback remains never automatic [D-0022].

All three lab VMs are on one physical host. The lab simulates a split by cutting the network between VMs and a node loss by stopping a VM. The requirement that voters sit in different failure domains [D-0017] is documented, not proven, by the lab.

Definition of Done

• With lab-1, lab-2 and lab-3 as voters, powering off the primary node triggers an automatic election and a new primary takes writes without any human action [D-0016].
• Cut the network into a minority partition; the minority side never acts as primary, and no node in it accepts writes [D-0016].
• In the majority partition, writes still succeed and the audit chain still verifies after the partition heals [D-0013, D-0031].
• A hot app keeps answering 200 on its domain across the loss of the node that was serving it, with a running copy on another node [D-0020].
• Bring the failed node back; it returns as a replica node and no automatic failback happens; a planned switchover is the only way to move the workload back [D-0022].

Demo script

1. Run three voters: lab-1, lab-2, lab-3. Note the current primary.
2. Power off the primary. Expected: a new primary is elected automatically, the panel shows it without a manual step, and a change made through the panel succeeds on it.
3. Cut the network so one node is alone. Expected: that node refuses writes and does not claim to be primary.
4. Deploy a hot app and stop the node serving it. Expected: the domain keeps answering 200 from the running copy. Power the stopped node back on. Expected: it rejoins as a replica node, nothing moves automatically, and a planned switchover is available in the panel [D-0022].

Dependencies

• M5b (witness, leases, replication).
• D-0016, D-0019, D-0020, D-0022.
• Needed before or during M6: snapshot consistency of hot volumes with the chosen primitive.

Explicitly out of scope

• Managed hosting: Later.
• Synchronous replication as the default: async is the default, synchronous is optional for nearby nodes [D-0020].

4. Later

• An optional managed offering may come later [D-0003]; nothing in this roadmap depends on it. The self-hosted witness is free and first-class [D-0017].
• A container-fallback Engine implementation is a later backend behind the Engine trait, because many VPS do not expose nested virtualization [D-0009]. It is not part of the MVP.
• A self-hosted GitHub Actions runner stays deferred, because a self-hosted runner would let fork PRs on a public repository run arbitrary code on the project’s infrastructure.

5. Out of scope for now

The following are explicitly excluded, because each could eat the project [D-0034]:

• Live migration [D-0034].
• Distributed storage such as Ceph [D-0034].
• Complex networking: SDN and VLANs [D-0034].
• Multi-tenant operation with billing [D-0034].
• Writing our own VMM; Quoro depends on KVM and Cloud Hypervisor instead [D-0007, D-0034].

6. Open questions affecting the roadmap

Open questions are open. This table lists them with options where the ledger gives options, and never picks one.

Decided and no longer open: the ingress proxy [D-0059], the Git build method [D-0060], the OCI-to-rootfs conversion and init [D-0061], the snapshot primitive [D-0062], the node mesh [D-0063], the first public release [D-0064], manual promotion of a replica node [D-0067], docs-site timing [D-0065], encrypted SQLite and template backups [D-0069] and when the curated catalog is decided [D-0070].

Answered verification questions

Every “to verify” claim is researched before it enters a document; the answered ones above cite their research file, and unconfirmed ones stay marked as hypotheses [D-0040].

Glossary

Terms used across Quoro’s documents. Bracketed ids point to the decision ledger; where a definition and an accepted decision disagree, the decision wins.

ACME: The protocol the ingress uses to obtain and renew TLS certificates automatically for hosts that expose a public IP. [D-0024]

ADR: Architecture decision record, one per accepted decision; an accepted ADR is law and changing it requires a new ADR approved by the maintainer. [D-0036, D-0040]

agent: An AI actor with its own identity (name, model, owner), separate from humans; every action records who acted, on behalf of whom, and why. [D-0029]

AI-native loop: The cycle detect (node agent and health checks report state) -> notify (configurable webhooks per event) -> wake (a webhook starts an agent with Quoro’s MCP and incident context) -> act (only within the agent’s grant). [D-0032]

app replica: An extra running instance of a stateless app on another node, behind the edge load balancer, so the edge can stop sending traffic to a dead node and keep the app available. [D-0020]

approval request: A request an agent files for a destructive or irreversible action, which a human approves in the panel; approval releases a short-lived extra grant. Also: JIT approval. [D-0030]

audit log: An append-only, hash-chained record of every action, holding who, on behalf of whom, the triggering event, the tool or endpoint, the parameters, the result and the agent’s stated justification. It can be copied off-site to S3/R2 with object lock, and MCP writes through the same service layer and the same log. [D-0031]

catalog: Curated templates kept in the repository, pointing to pinned upstream images with verifiable signatures; Quoro runs no image registry of its own for now. Its format, owner, location and signature scheme are decided at M4 planning, before the first template ships. Also: template. [D-0026, D-0070]

change log: The numbered, append-only, hash-chained sequence of state changes that a replica node asks for (“give me everything after N”) and verifies by hash. Also: state log. [D-0013, D-0015]

chunk engine: The volume sync mechanism: a snapshot is split into content-addressed, deduplicated, encrypted chunks and only new chunks are shipped. Shipping to another node is warm replication; shipping to S3/R2 is cold backup. [D-0023]

Cloudflare Tunnel: The ingress option for homelabs behind NAT, where cloudflared dials out to Cloudflare so no port has to be opened. Load Balancing is a paid add-on from $5/month and an LB sees all replicas of one tunnel as one endpoint; cloudflared replicas give host-level failover only, per-node failover needs one tunnel per node behind an LB, and whether replicas expose any app health check is a Hypothesis (see docs/research/cloudflare-lb-and-tunnel.md). [D-0024]

Cloud Hypervisor: The Rust virtual machine monitor, driven through its HTTP/JSON API over a UNIX socket, that Quoro’s engine builds on together with KVM. It is v54.0 under Apache-2.0 AND BSD-3-Clause; snapshots hold RAM and device state but not disks, with no incremental API; virtio-fs needs an external virtiofsd; consoles are text-only (serial, virtio-console), with no VNC or GPU; the host kernel must be 6.8 or newer (see docs/research/cloud-hypervisor.md; its “under 100 ms” boot claim is unbenchmarked and stays a Hypothesis). [D-0007]

cold: HA policy where other nodes hold backups only, so RPO is the time since the last backup and RTO is a restore. No copy of the app runs elsewhere, so a failure means restoring from backup. Its cost is backup storage and restore time (D-0068). [D-0020]

console: A terminal in the panel for a microVM, text-only, rendered with xterm.js; there is no graphical console and no noVNC. [D-0056]

contract: A frozen interface: OpenAPI spec, event schema, DB schema, node-agent protocol, audit-log format, permission model, Engine trait. Contracts freeze after M0 and any change is a standalone PR reviewed on its own. [D-0033, D-0036, D-0040]

control plane: The stateful layer (API and state) that holds Quoro’s state and accepts writes through the primary; if it dies, apps keep running and only changes are blocked. [D-0010, D-0011]

data plane: The apps and node agents that keep running and reconciling their local desired state when the panel or the control plane is unreachable. [D-0010, D-0011]

desired state: The locally stored description of what a node should run, written by the control plane and applied by the node agent without needing the control plane to be reachable. [D-0011]

dial out: The rule that a node opens the connection to the control plane and never the reverse, so the control plane can live anywhere and nodes work behind NAT. [D-0011]

edge: Node role that is the traffic entry point and load balancer; it can serve as the third voter, and a single edge is a traffic single point of failure. [D-0012, D-0018]

Engine trait: The boundary behind which the Cloud Hypervisor engine sits, so other backends such as Firecracker or containers can be added later. [D-0007, D-0009]

external lease arbiter: An optional off-cluster service that grants leases, either a Cloudflare Durable Object (single-threaded actor with strongly consistent storage and alarms; needs a Cloudflare account and a deployed Worker, and fencing tokens are still required) or S3/R2 conditional writes (atomic If-Match/If-None-Match PUT with one winner, 1 write/s per key, no TTL and fencing tokens required; not portable, since AWS S3 and MinIO have it while Backblaze B2 appears to lack it, a Hypothesis; see docs/research/durable-objects.md and docs/research/r2-conditional-writes.md). [D-0017]

failback: Returning an app to its previous node after a failure; it is never automatic or immediate, and the default is no failback, with roles simply swapped. [D-0022]

failure domain: A set of machines that fail together, for example a home PC next to node A; a witness must sit outside the nodes’ failure domain. [D-0017]

fencing: A node’s mandatory, lease-based act of stopping its own stateful apps when it loses its lease — for an expiring lease only once it is expired, not while it is uncertain; the node agent must self-fence from M0 so the protocol is not redesigned later. [D-0019]

grant: A permission envelope made of scope, resource selector, TTL and optionally a maximum number of uses, for example app:restart on project=blog for one hour. Tokens are short-lived and refreshed against the grant, and revoking the grant locks the agent out within seconds. [D-0030]

HA policy: The per-app reliability choice among none, cold, warm and hot, with its cost shown in the panel. Replication is asynchronous by default (synchronous is optional for nearby nodes and adds latency to every write), and suggested defaults are app replicas behind the load balancer for stateless apps on two or more nodes and warm for stateful apps. [D-0020]

hot: HA policy where the app runs and replicates on other nodes, giving near-zero RPO and RTO at the cost of disk, RAM and CPU. [D-0020]

incident grant: A grant with just the scopes needed for the incident that woke an agent, issued when an event wakes it and expiring automatically. [D-0030]

ingress: The entry layer for app traffic: a reverse proxy with ACME for hosts with a public IP, Cloudflare Tunnel for hosts behind NAT, both behind a common interface; the proxy is an embedded Rust proxy (hyper + rustls with an ACME client crate), with Caddy driven through its admin API kept as a fallback backend (D-0059). [D-0024, D-0059]

JIT approval: See approval request. [D-0030]

lab: The test lab is three nested-virtualization Debian 13 VMs (lab-1, lab-2 and lab-3) on a single physical host, so they share one failure domain; it is not valid for performance benchmarks. [D-0036]

lease: A lease is a node’s right to run a stateful app. expiring leases (warm/hot, from M5b) end when they are not renewed in time; local leases (none/cold) never expire when the arbiter is unreachable and end only by lease_revoke or an operator declaring the node lost. When the arbiter is unreachable an expiring lease first becomes uncertain (keep running, start nothing new) and only then expired (self-fence); see docs/contracts/node-agent-protocol.md §8. [D-0019, D-0047]

lease holder: The node currently holding an app’s lease; only it may run that stateful app. [D-0019]

majority: See quorum. [D-0016, D-0017]

manual promotion: The M5b action (D-0067) that promotes a replica node to primary, either as a command or button or by an agent with human approval. [D-0016, D-0067]

MCP: Model Context Protocol server exposing Quoro to agents; it is first-class from day one and uses the same service layer and audit log as the HTTP API. [D-0002, D-0031]

mesh: The WireGuard network connecting nodes across providers, configured by Quoro over netlink against kernel WireGuard with no wg-quick, per-node keypairs and peers in desired state (D-0063). [D-0010, D-0063]

microVM: The unit of app isolation, one per app, booted with a minimal init; nodes run no Docker. [D-0008]

milestone: A roadmap stage (M0, M1, M2, M3, M4, M5a, M5b, M6, then Later) that ends with a demo on the lab and the maintainer’s approval before the next one starts. [D-0033, D-0036]

minimal init: The small init process that boots inside a microVM in place of a full distribution init; Quoro writes its own minimal Rust init (D-0061). [D-0008, D-0061]

N+1 capacity indicator: The panel answer to “if node A dies, do its apps fit on the others?”, given as yes or as no plus what is missing, for example 6 GB of RAM. [D-0021]

node: (1) A machine running the Quoro binary; (2) the node role, as opposed to witness or edge. [D-0012]

node agent: The per-node component that stores desired state locally, reconciles it, dials out to the control plane and self-fences when it loses its lease (for an expiring lease, only once it is expired). [D-0011, D-0019]

none: HA policy where nothing runs on other nodes, so no resources are reserved elsewhere (D-0068) and no RPO or RTO is promised for the app. [D-0020]

OCI image: The container image a deploy is built from; Quoro converts it into a root disk and boots it in a microVM, through its own OCI-to-ext4 pipeline with layer paths resolved inside the rootfs (D-0061). [D-0008, D-0061]

planned switchover: A deliberate handover of an app to another node: wait for replication lag near zero and a low-traffic window, freeze writes briefly, do a final sync, move the lease, shift traffic at the load balancer and drain connections. It loses no data, unlike a surprise failure, which can lose seconds. Also: relevo, “relevo inteligente”. [D-0022]

primary: The one control-plane node that is the single source of truth; all writes go through it. [D-0012]

provider: The credentials of one external provider, stored as a resource with a prv_ id and one kind (dns, tunnel, lb or object_store). The secret is write-only, encrypted with the cluster data-encryption key, never returned by a read and redacted from audit parameters; managing providers needs the human-only scope provider:manage, and there is no MCP tool for it. The credential for the off-site audit copy is a separate put-only object_store provider. [D-0051]

provider interface: The pluggable boundary for DNS, load balancing and S3-compatible storage, with Cloudflare as the first implementation rather than the only one. [D-0025]

quarantined snapshot: The separate snapshot in which a returning node’s divergent writes are kept and recorded in the audit log, so they are never silently dropped. [D-0022]

quorum: The majority of votes needed for automatic decisions; HA requires at least three votes and an odd number of voters. Also: majority, vote. [D-0016, D-0017]

Raft: The consensus algorithm planned for automatic failover with a majority of at least three votes, in M6. [D-0016]

reconcile: See desired state. [D-0011]

relevo: See planned switchover. [D-0022]

replica: (1) replica node: a node that holds a copy of control-plane state, serves reads and forwards writes to the primary; (2) app replica: an extra running instance of a stateless app behind the edge load balancer. A replica node is not an app replica. [D-0012, D-0020]

reproducible build: A build that can be rebuilt bit for bit from its sources; reproducible builds and signed binaries are mandatory because the installer is curl | sh. Also: signed binary. [D-0027]

role: What a node does in the cluster (node, witness, edge), combinable in the same binary. [D-0012]

rootfs: The root disk a microVM boots from, produced from an OCI image at deploy time. [D-0008]

RPO: Recovery point objective, the amount of data an app may lose on failure; each HA policy sets it, from “since the last backup” for cold to near zero for hot. [D-0020]

RTO: Recovery time objective, how long an app takes to serve again after failure; each HA policy sets it, from a restore for cold to a microVM boot for warm and near zero for hot. [D-0020]

scope: The verb inside a grant, for example read, restart, redeploy or delete. [D-0030]

self-fencing: See fencing. [D-0019]

setup token: A one-time secret required in the Quoro-Setup-Token header by POST /api/v1/setup, which claims the owner on first run. It is single use and expires after one hour; a missing or invalid token answers 401 unauthenticated. [D-0054]

signed binary: See reproducible build. [D-0027]

split-brain: The state where both sides of a network cut become primary and diverge, which is why automatic election with only two nodes is not allowed. [D-0016]

stateful app: An app that keeps data of its own, so it may run only on its lease holder and is fenced off when the lease is lost. [D-0019, D-0020]

stateless app: An app that keeps no local state, so it can run as app replicas on several nodes behind the edge load balancer for zero downtime. [D-0020]

state log: See change log. [D-0013, D-0015]

sync engine: Quoro’s own replication layer, made of control-plane log replication and the chunk engine for volumes; databases instead use their native replication, offered as catalog templates (Litestream 0.5 has no client-side encryption, so SQLite backups use Quoro’s own chunk engine from consistent snapshots instead, D-0069; a Litestream-based lower-RPO backend may be added later as an option). [D-0023, D-0069]

telemetry: App logs and metrics, which stay on each node and are queried through the primary on demand, with optional archive to S3/R2; only state is replicated. [D-0014]

template: See catalog. [D-0026]

TTL: Time to live, the expiry attached to a grant after which it no longer authorizes anything. [D-0030]

vote: See quorum. [D-0016, D-0017]

voter: See quorum. [D-0016, D-0017]

warm: HA policy where data is continuously replicated and the image is pulled on another node but the microVM stays stopped, giving RPO in seconds and RTO equal to a microVM boot at the cost of disk. [D-0020]

warm copy: The stopped, continuously replicated copy of a stateful app on another node under the warm policy; it may start only once the previous lease holder has lost the app’s lease. [D-0019, D-0020]

witness: Node role that only breaks ties and stores no data, run as a self-hosted daemon in a failure domain different from the nodes. The self-hosted witness is free and first-class [D-0017].

workstream: A parallel line of work after M0, each in its own crate or package and owning its paths: engine, control-plane, panel, node-agent, ingress, mcp, installer/releases/docs site, and later sync/chunk engine and witness/edge (see docs/architecture/architecture.md §17). [D-0036]

Quoro architecture

Status: M0 draft for review. Where an accepted decision settles a choice, the text cites its D-id; a recommendation is marked pending maintainer decision only where its question is still open.

Quoro is one self-hosted binary that unifies a hypervisor with application deployment: it runs one microVM per app on Cloud Hypervisor and KVM, and the same interface deploys, exposes and observes those apps. This document describes how the pieces fit together: the roles of the binary and the control-plane statuses, the split between control plane and data plane, how nodes stay autonomous when the control plane is unreachable, how state is replicated through a hash-chained change log, where telemetry lives, the engine layer, networking, ingress, providers, identities and grants, the realtime transport (D-0057), the high-availability groundwork laid in the MVP, the key flows, the confirmation of the baseline stack (D-0056, superseding D-0028), the workspace layout, and every open question that this document depends on.

1. Scope and reading guide

This document is the architecture reference named in D-0040 and the M0 architecture deliverable of D-0033. It covers component boundaries, process model, data flow, failure behaviour and the interfaces between workstreams. It does not restate contracts: those are separate files that freeze at the end of M0 (D-0036).

In scope:

• The components inside the single quoro binary and which of them run under which role and control-plane status (D-0012).
• The control-plane and data-plane split, and what still works when the control plane is down (D-0010, D-0011).
• State storage, the change log and replication (D-0013, D-0015).
• Telemetry locality (D-0014), the engine layer (D-0007, D-0008, D-0009), networking (D-0010, D-0034), ingress (D-0024) and provider interfaces (D-0025).
• Identities, grants, audit and the AI-native loop (D-0029 to D-0032).
• The realtime transport (D-0057), the HA groundwork (D-0016 to D-0023, D-0033), the key flows, the stack confirmation (D-0056, superseding D-0028) and the workspace layout.

Out of scope: product requirements (see docs/product/roadmap.md), the threat model (see ../architecture/threat-model.md), and the contents of each contract, which are linked but not copied.

Contracts this document depends on

Vocabulary is fixed by ../contracts/README.md; terms follow ../glossary.md. Every name used here for roles, statuses, ids, scopes, event types, errors, headers, channels, node-agent messages and tables comes from those two files.

Labelling convention

Every design statement below carries one of these labels:

• Decision (D-xxxx): accepted and binding. An accepted decision is law; changing one requires a new decision approved by the maintainer (D-0036).
• Finding (F-xxxx, or a docs/research/<file>.md path): a sourced fact about external software, gathered before it entered a document (D-0040).
• Pending maintainer decision: a recommendation or an undecided design choice. Written as “recommended, pending maintainer decision”; it is not a decision and must not be implemented as one.
• Open (Q-xxxx): an unanswered question. Options are listed and the recommendation, if any, is marked pending maintainer decision; the question stays open.
• Hypothesis: a claim about external software, or about behaviour this project has not yet run, that is not established by the findings. Each one is collected in section 19 with what would verify it.

Anything stated without one of these labels and without a citation is a Hypothesis.

2. System overview

There is one binary, quoro, on every machine (D-0012). The binary bundles the components below; which of them run depends on the role (node, witness, edge) and, for a node, on its control-plane status (primary, replica) (D-0012).

   humans (panel, session cookie)        agents (MCP, access token)
                 \                              /
                  \                            /
            +------------------------------------------+
            |   primary  (role: node)                  |
            |   HTTP API  ----  MCP server  (/mcp)     |
            |       |                |                 |
            |   service layer (auth, grants, audit)    |
            |       |                |                 |
            |   state store (SQLite: change_log +      |
            |   derived tables)   audit log  scheduler |
            |   event bus  webhook dispatcher          |
            +------------------------------------------+
               ^   ^   ^                   |
   dial-out    |   |   |                   | providers (D-0025)
   WebSocket   |   |   |                   v
   sessions    |   |   |            +----------------------+
   (TLS, node  |   |   |            |  DNS   LB   object  |
    opens them)|   |   |            |  provider  store    |
               |   |   |            +----------------------+
     +---------+   |   +---------+
     |             |             |
+----------+  +----------+  +----------+
|  node    |  |  node    |  |  node    |
| agent    |  | agent    |  | agent    |
| engine   |  | engine   |  | engine   |
| ingress  |  | ingress  |  | ingress  |
| telemetry|  | telemetry|  | telemetry|
+----+-----+  +----+-----+  +----+-----+
     |             |             |
  Cloud Hypervisor microVMs (one per app, KVM)

  - - - - - - - - - - - - - - - - - - - - - - - - -
  witness  (M5b: tie-break vote, no data)
  edge     (M5a: traffic entry, load balancer;
            can combine with witness, D-0018)
  - - - - - - - - - - - - - - - - - - - - - - - - -

The arrows into the primary are inbound sessions, not inbound connections: each node agent opened that session outward (D-0011). Nothing in the design dials into a node.

Process model

There is one OS process per machine, not one per component (D-0012). Inside it:

The process model follows from two constraints. Nothing that can block a request may run on the request path, because a deploy that waits on a registry pull must not make the panel unresponsive. And nothing that runs a guest may share an address space with the control plane, because a compromised app must not reach Quoro’s memory (D-0008).

Privileges are split the same way (ADR 0089, which replaces the M1 root service of ADR 0088). The control plane (quoro serve, unit quoro.service) runs as the quoro-api system user with an empty capability bounding set, NoNewPrivileges and a read-only system except its data directory /var/lib/quoro: it signs desired state and never touches a hypervisor, a bridge or a firewall. The node agent (quoro node, unit quoro-node.service) is the only privileged process: it runs the engine as root, keeps its state in /var/lib/quoro-node, and dials the control plane like any other node, over loopback on a primary. Its unit uses KillMode=process, so restarting the agent leaves the VMMs running and the next agent adopts them. A development host may still run both in one process with [node] local = true, which then needs root.

Host layout, ports and CLI

Decided (D-0054):

• One listener. One HTTPS listener, default port 8443 and configurable, serves the API, the panel and MCP. ingress owns 80 and 443 on the same host, so the control plane never competes with a customer’s domain for those ports.
• Certificate. The primary’s certificate is self-signed at install, and the join command carries its SHA-256 fingerprint, which is the pin model of section 5. Rotating that certificate therefore invalidates every outstanding join command. An operator may later replace it with an ACME certificate.
• Paths. Control-plane data lives in /var/lib/quoro, node-agent data in /var/lib/quoro-node (siblings, never nested), configuration in /etc/quoro/quoro.toml (TOML), and logs go to journald.
• CLI. quoro serve --role node|witness|edge, quoro version, quoro join <url> --token --fingerprint, quoro node, quoro mcp stdio, quoro setup-token. quoro join only enrolls; quoro node runs the node agent.
• First run. The installer prints a one-time setup token that POST /api/v1/setup requires in the Quoro-Setup-Token header: single use, valid for one hour. The primary keeps only the token’s SHA-256 hash and its expiry in the data directory. quoro setup-token issues a fresh token, invalidating the previous unused one, and refuses once an owner exists.

Alternatives not chosen: ACME for the listener from install, which needs a public name and fails behind NAT; and binding /setup to localhost, which breaks remote installs.

3. Roles and control-plane status

Roles are what a machine does; statuses are what a node’s control-plane component is. They are different axes (D-0012).

Roles (combinable, D-0012, D-0018):

Combining roles is a deployment choice, not a code branch: a machine that is witness+edge runs the witness vote and the load balancer from the same binary (D-0018). A machine can be node and edge at once, which is what a two-machine cluster with a load balancer on one of them looks like.

A single edge is a traffic single point of failure (D-0018). With the edge also a voter, losing it removes both the traffic path and one vote. D-0018’s mitigations are Cloudflare as the traffic edge with the small VPS acting only as witness, or two edges with DNS failover. Verify in M5b that the chosen mitigation keeps both traffic and quorum when the edge dies.

Control-plane status of a node:

What exists per milestone:

• M1 to M4 have exactly one primary, which is also a node, plus nodes that run the node agent and dial out to it (D-0016, D-0033).
• Witness, edge and leases with expiry arrive with M5 (D-0033): witness in M5b, edge in M5a.
• Replica nodes serve reads and forward writes to the primary (D-0012). Turning that serving on requires promoting a replica to primary; manual promotion ships in M5b, alongside leases and fencing, and refuses while the old primary may still hold expiring leases unless an operator acknowledges the fence (D-0067).

Design choice, stated as such: from M2 every enrolled node already follows the change log over its session and keeps a verified copy, so a later manual promotion needs no protocol change (D-0013). The M2 node agent therefore implements the follow path even though nothing else in M2 uses a replica. This costs storage on each node and buys the ability to promote without a redesign.

4. Control plane and data plane

The control plane is the API, the state store and the audit log on the primary: the layer that decides what should run and records who asked for it. The data plane is the node agents, the engine, the ingress and the apps themselves: the layer that actually runs things (D-0010, D-0011, D-0012).

The principle is that the two fail independently. If the control plane dies, apps keep running and the node agents keep restarting them; only changes are blocked (D-0010, D-0011, D-0012).

One row depends on the ingress backend (D-0024): a reverse proxy on a public-IP host renews certificates locally with ACME, so renewal continues while the primary is down; a host behind a Cloudflare Tunnel needs nothing renewed locally because the tunnel terminates TLS at the provider.

The data plane never infers intent. A node with no session to the primary applies the last desired state it persisted and nothing else. This is what makes “the control plane is down” a change freeze rather than an outage (D-0011).

Failure behaviour by component

The third column is as important as the second. Every failure mode that could be “helpfully” automated is left manual until the milestone that owns it, because the alternative in a partition is two writers (D-0016, D-0019).

5. Nodes: autonomy and dial-out

Enrollment is initiated by a human: node:enroll creates a one-time join token, and the node redeems it to receive its identity (D-0011; scope and prefix from ../contracts/README.md). The node generates its own keypair; the private key never leaves the machine.

The sequence:

1. A human with node:enroll calls POST /api/v1/nodes/join-tokens and gets a one-time njt_ token (1 h default) and a join command.
2. The join command carries the primary URL, the token and the SHA-256 fingerprint of the primary’s TLS certificate, which the node pins: a certificate whose digest differs is refused, so a self-signed certificate is safe to accept.
3. The node generates an Ed25519 keypair, keeps the private key locally, and calls POST /api/v1/nodes/enroll with the token, its name, its public key, its roles, its capacity and the KVM probe result.
4. The primary marks the token used (it is one-time), records the node with status pending, emits node.enrolled, and returns node_id, cluster_id, primary_url, a node credential (shown once, stored hashed) and the cluster public key that signs desired state.
5. The node stores these and opens the session with GET /api/v1/nodes/session using the credential; hello and welcome follow on that session.
6. pending becomes connected when the first session starts, and node.connected is emitted.

Reconnects reuse steps 5 and 6 only and never re-enroll: a lost credential is recovered by removing and re-enrolling the node by a human. That is deliberate: enrollment and reconnect must not be two implementations of the session. The flow is fixed by ../contracts/node-agent-protocol.md section 2.

After enrollment the node opens a long-lived WebSocket-over-TLS session to the primary and keeps it open. The direction is fixed: the node dials out, the control plane never dials in (D-0011). That is what lets the control plane live anywhere — including behind NAT itself — and lets nodes work from networks that accept no inbound connections.

On its side the node keeps:

• Desired state in a local SQLite store, written from desired_state messages and persisted before it is applied (D-0011).
• A reconciler that compares desired state with observed state continuously, not only on message arrival. This is what makes a node converge after a crash, a reboot or a long disconnect.
• Health checks per instance, whose results become status_report messages and, on failure, event messages (D-0032).
• Crash restart with backoff: a VM that exits without being asked to stop is restarted locally (D-0011). Restarts are requested by bumping restart_generation in desired state, so a repeated request is idempotent and a disconnected node still performs the restart it was last told to perform.
• An event buffer, so events raised while disconnected are replayed on reconnect; the control plane deduplicates them by their event ids.

Desired state carries a monotonic rev. A node applies a desired_state only if its rev is newer than the one it holds, which makes re-delivery harmless and makes the “everything after N” catch-up in section 6 well defined.

Message types on the session, named exactly as in ../contracts/README.md and specified in ../contracts/node-agent-protocol.md:

The session is a WebSocket. That is a transport choice for a machine-to-machine protocol and is independent of the panel realtime split in section 13 (D-0057).

6. State: SQLite and the change log

Control-plane state lives in SQLite on the primary (D-0015). SQLite is chosen because “one binary, one command” rules out a database server, and because the MVP has a single writer anyway (D-0015).

Every state change is one numbered, hash-chained entry in the change log, appended in the same SQLite transaction that updates the state tables (D-0013, D-0015). The state tables are derived: they can be rebuilt by replaying the log from genesis. That property is what makes replication, verification and recovery one mechanism instead of three.

Change-log entry fields (schema in ../contracts/db-schema.sql):

A replica asks “everything after N” and verifies the chain as it applies entries (D-0013). Verification is the same check the audit log uses for its own chain (D-0031), so tamper evidence on state comes for free.

How replication runs:

1. The replica sends its last verified seq N when it connects, and after that as it advances.
2. The primary streams entries after N in order, each with its hash and prev_hash.
3. The replica verifies each hash before applying it, and stops at the first mismatch rather than applying past it. A mismatch is an incident, not a retry.
4. Entries are applied in the same order and in transactions, so a crash mid-batch leaves a consistent state at some seq rather than a torn one.
5. The replica records its verified seq, which is what it reports on reconnect.

Because state tables are derived, recovery from a bug in a state table’s logic is “fix the code, replay the log”, not “restore a backup and lose the tail” (D-0015). The log is the durable artifact; the tables are a materialised view of it.

term is part of the entry from M0 even though it is constant at 1 while there is one primary: it is prepared for the election work of D-0016 and cannot be added later without rewriting history.

Some tables are local and are not replicated: sessions, access_tokens and webhook_deliveries (marked local in ../contracts/db-schema.sql). api_tokens and agent_keys are replicated, but only as hashes, so a replica holds no usable credential. Two reasons:

• Secrets. Session tokens and agent access tokens are the local secrets. Copying them to every replica node multiplies the places they can leak from, and a replica does not need them to serve reads on behalf of a caller who authenticated to that replica.
• Churn. Webhook delivery attempts change on every retry and are of interest only to the node that dispatched them. Replicating them would put the noisiest table on the replication path.

Backing up the primary’s database off-site is not in the MVP (D-0033), but the design has to name a path. Litestream was suggested in D-0015; Litestream 0.5 has no client-side encryption, which conflicts with the encrypted-backup expectation of D-0023 (F-0040). D-0069 settles the default: SQLite and catalog-template backups go through Quoro’s own chunk engine with client-side encryption, taken from consistent snapshots via the SQLite backup API or VACUUM INTO. Litestream is not used. The rejected and optional paths, kept for the record:

Litestream with SSE-C may be added later as an optional backend for a lower RPO (D-0069).

Cluster keys

Decided (D-0050), three keys:

• Cluster signing key (Ed25519). Signs desired_state and lease messages; nodes pin its public half at enrollment, so losing or rotating it is a cluster-wide re-pin.
• Cluster data-encryption key (DEK, 256-bit XChaCha20-Poly1305). Encrypts every *_enc column; a *_enc column stores the key id, the nonce and the ciphertext, so a row records which key wrote it.
• Node-local key. Encrypts a node’s local copy of the secrets for the apps placed on it.

Storage and rotation: the keys are generated at first start and stored as 0600 files in /var/lib/quoro, outside the SQLite file, and backed up separately, because losing the data-encryption key loses every secret. The signing key rotates through the node-agent message signing_key_rotate, signed by the old key. The data-encryption key rotates by re-encrypting rows under a new key id. Custody of both keys when a replica is promoted is decided together with manual promotion in M5b (D-0067); the open part is whether the keys are replicated wrapped to each replica’s node key or re-keyed with an operator-confirmed re-pin.

7. Telemetry locality

App logs and metrics stay on the node (D-0014). Shipping them all to the primary would make the primary the bottleneck and the single point of failure for observability, which is exactly where it is least affordable.

Logs. The minimal init forwards the app’s stdout and stderr over the microVM’s virtio-console to the node agent, which writes size-capped local files (F-0034 covers which console types Cloud Hypervisor offers). The panel and MCP read logs through the primary, which opens a stream_open on the node’s session on demand and relays stream_data back; the primary never stores app logs (D-0014). A bounded pull of app logs is a command of kind logs_query answered by command_result; a live view is stream_open (kind app_logs), stream_data, and stream_end / stream_close. log_entries and log_ack are not app-log reads: they are change-log follow messages (section 6).

Metrics (M4). Per-VM counters come from the engine, which reads them from Cloud Hypervisor (docs/research/cloud-hypervisor.md); host metrics come from the host. They are sampled and kept on the node, and the panel reads them through the primary on the same on-demand path as logs (D-0014).

Optional archiving of telemetry to S3-compatible storage arrives later (D-0014) and is not assigned to a milestone.

If the node is unreachable when a read is attempted, the API returns node_unreachable (504), which says what happened: the state is known to the primary, the telemetry is not reachable right now (../contracts/README.md).

Log pipeline and retention

• The minimal init is the app’s parent process inside the microVM and forwards stdout and stderr over the virtio-console (F-0034). Line boundaries are preserved; partial lines are flushed after a short timeout so a chatty app cannot hold the last line forever.
• The node agent writes size-capped local files and rotates them. The cap is what keeps a log flood from filling the node’s disk, which would take the node down and its apps with it.
• Reads are on demand. A pull of a bounded range is a command of kind logs_query answered by command_result; a live view uses stream_open (kind app_logs), stream_data and stream_end / stream_close on the node’s existing session, so a log view needs no inbound connection to the node (D-0011, D-0014).
• The primary relays and forgets. It holds no copy after the stream ends, which is why the primary’s disk usage does not scale with app traffic (D-0014).
• Archiving to S3-compatible storage is optional and later (D-0014). When it exists, it is a node-side upload from the node that holds the logs, not a transfer through the primary.

8. Engine layer

The engine sits behind an Engine trait (D-0007, D-0009), specified in ../contracts/engine-trait.md. The trait is the only way the node agent touches virtualization: create, start, stop, delete, and — for M5 — snapshot and volume diff hooks. Cloud Hypervisor is the first implementation; a container runtime is a possible later implementation (D-0009); a fake in-memory engine ships for tests on machines without KVM.

Cloud Hypervisor specifics:

• One VMM process per microVM, driven over its HTTP/JSON API on a UNIX socket (docs/research/cloud-hypervisor.md). The engine owns the process lifecycle; a crashed VMM surfaces as vm.crashed in the node’s event stream.
• License: Apache-2.0 AND BSD-3-Clause (F-0036). Cloud Hypervisor is Apache-2.0 AND BSD-3-Clause; compatibility with an AGPL-3.0 Quoro that runs it as a separate process is a Hypothesis (section 19).
• Host requirements: a KVM-capable host (D-0009) and host kernel >= 6.8 (F-0036).
• Nested virtualization: rare on cheap VPS. Hetzner Cloud, Hostinger, Contabo VPS and netcup VPS state that they do not support it (F-0044, docs/research/vps-nested-virt.md). The installer therefore probes /dev/kvm, the vmx/svm CPU flag and a real microVM boot instead of trusting provider marketing. A host that fails the probe is refused during install rather than at first deploy.
• Image path: OCI image -> root disk -> boot with a minimal init (D-0008). Quoro ships its own minimal Rust init and its own OCI-to-ext4 pipeline (D-0061), validated by the M0 spike (spikes/oci-microvm); the pipeline resolves layer paths inside the rootfs with openat2 and RESOLVE_IN_ROOT, and also covers zstd layers and config delivery through a config disk or the kernel command line. The guest kernel is built by Quoro and pinned reproducibly (D-0027).
• Consoles are text-only. Cloud Hypervisor has no VNC, virtual GPU or framebuffer device (F-0034). The microVM console is the serial or virtio-console UNIX socket, relayed by the node agent, so consoles are xterm.js over serial or virtio-console only.
• Disks and snapshots. Cloud Hypervisor snapshots hold guest RAM and device state, not disk contents, and there is no incremental or dirty-bitmap API (F-0035). The chunk engine (D-0023) therefore needs its own disk snapshot primitive. The interim primitive is reflink snapshots plus a full content-defined chunk scan for M5a cold backups, and a spike comparing lvm-thin with btrfs before M5b picks the primitive for warm backups at few-second intervals (D-0062); the Engine trait’s snapshot_volume hooks keep it swappable.
• virtio-fs needs an external virtiofsd (docs/research/cloud-hypervisor.md). The engine therefore does not depend on it: apps boot from a disk image, and virtio-fs is at most an optimisation for a later milestone.
• No live migration (D-0034). Moving a running microVM between nodes is not a goal; moving an app is a stop/start with a lease, not a migration.

The engine is the component with the largest surface for a mistake that costs data, which is why its contract is frozen at M0 and why the fake engine exists: engine behaviour must be testable without KVM, and CI must be able to run the node agent’s reconciliation logic on an ordinary machine.

9. Networking

Networking is deliberately simple: no SDN overlay, no VLANs, no distributed switch (D-0034). Each node is its own L2 island with a small amount of local plumbing, and cross-node traffic is carried by a mesh VPN (D-0010).

Per node:

• A Linux bridge that carries the node’s microVMs and the ingress.
• One TAP device per microVM, created by the engine and attached to the bridge.
• A private subnet per node, carved from one cluster-wide range. No two nodes hand out the same addresses, which keeps routes unambiguous when traffic crosses the mesh.
• nftables rules in two groups: egress NAT, so a microVM can reach the internet through the node’s address; and isolation between microVMs, so no microVM reaches another project’s microVMs by default. Flows into a microVM exist only from the ingress on the same bridge.
• Ingress forwarding to the microVM’s private address. The ingress is the only component that talks to the public side.

The bridge, the subnet carving and the address plan are this document’s proposal for the engine workstream, not a decision: D-0011 and D-0008 fix that microVMs exist and are isolated, but not how the host network is laid out. Two properties are requirements rather than taste: a microVM must not reach another project’s microVMs by default, and the address plan must survive a node being added or removed without renumbering existing microVMs.

A worked proposal for the address plan: one cluster range (for example 10.42.0.0/16), one /24 per node, the bridge on the first address of that /24, and microVMs allocated upward from it. The point of writing it down is that the mesh needs stable peer addresses, and a per-node /24 gives them without a central allocator after enrollment.

Two rules keep this from drifting. Addresses are recorded in state, so a reinstall of a node does not hand out an address that a recorded volume or VM still refers to. And isolation is a default that a project can widen for itself, never one that another project can widen for it.

Node mesh. Nodes are independent machines, often at different providers (D-0010), so cross-node traffic — edge-to-replica traffic in M5a, node-to-node volume replication for warm in M5b, the chunk engine, and any future node-to-node call — runs over WireGuard (D-0010). The mesh is kernel WireGuard configured by Quoro over netlink, with no wg-quick, per-node keypairs and peers plus allowed IPs in desired state; userspace boringtun is only a fallback, and an own implementation is rejected for crypto risk (D-0063). It is first needed in M5a, for cross-node replica traffic, and M2 does not need it; the control-plane session does not depend on it, because that is TLS dial-out (D-0011), so a single-node M2 works with no mesh at all.

Two nodes behind NAT cannot reach each other directly. The mesh therefore assumes at least one publicly reachable peer — a public node or the edge — as a relay hub, or falls back to the dial-out path (D-0063). This is a limitation of the mesh, not of the control plane.

10. Ingress

Ingress is one Ingress interface with two backends (D-0024):

The reverse proxy is an embedded Rust proxy — hyper + rustls + an ACME client crate — behind the same interface, which keeps one binary and full control of health-aware routing and slow start at the cost of owning the ACME edge cases (D-0059). Caddy driven through its admin API stays a fallback backend, to be used if the M2 spike shows that owning ACME is too costly; crate maturity is a research task before M2.

The route table is derived from desired state: each domain maps to the app instances that may serve it (D-0024). It is written by the control plane and applied by the node, so a route change is a state change with a change-log entry, and a node that cannot reach the primary keeps serving the last applied table (section 4).

Findings to design around (docs/research/cloudflare-lb-and-tunnel.md):

• cloudflared replicas of one tunnel are identical copies, routed to the nearest replica, with no app health check.
• The load balancer treats all replicas of one tunnel as a single endpoint, and has no TCP monitors for tunnel origins (F-0046).
• An HTTPS monitor whose Host is the app hostname does check the app through the tunnel; it is proxied by cloudflared to the app (F-0046, which corrects the broader reading of F-0043).

The consequence is a design rule: per-node failover needs one tunnel per node behind a load balancer, not several replicas of one tunnel. Replicas of one tunnel fail over the connector, not the node: if the node behind the tunnel dies, every replica of that tunnel dies with it, and the load balancer cannot see the difference because it sees one endpoint. One tunnel per node turns each node into a distinct endpoint that a monitor can mark down.

Load balancing with health monitors is a paid add-on of the tunnel provider (docs/research/cloudflare-lb-and-tunnel.md); the tunnel backend without it gives connector redundancy only, which is why the tunnel path is documented as best-effort until M5a.

Certificates live on the node that serves them (D-0024). There is no central certificate store to synchronise and no certificate secret in the replicated state: an ACME certificate stays in the node’s own storage, and a tunnel host has no local certificate at all.

11. Providers

Anything Quoro asks of an external service goes through a provider interface (D-0025). Cloudflare is the first implementation, not the only one: the interface exists so the community does not read Quoro as vendor lock-in.

ObjectStore probes capabilities at setup instead of assuming them:

• Conditional PUT (If-Match / If-None-Match). Not portable: R2 and AWS S3 support it, MinIO AIStor documents it, Backblaze B2 apparently does not (F-0038). Anything that depends on it — the conditional-write lease arbiter in section 14 — must be probed per provider at setup.
• Object lock. Not yet researched: whether a given provider supports it, and in which mode, is a Hypothesis per provider. The off-site audit copy depends on it (D-0031) and is gated by an object-lock probe per provider (D-0053); a research task verifies compliance-mode object lock for R2, S3, B2 and MinIO before M5a.
• SSE-C. Presence and behaviour are recorded by the probe, because an optional Litestream backend with SSE-C would depend on it (D-0069, F-0040).

R2 specifics: one write per second per key, and no conditional delete (F-0037). Those two limits rule out using a single object as a high-frequency lease register and rule out a delete-as-lock-release pattern; any arbiter built on R2 must renew at a rate below the write limit and must treat lease release as an overwrite, not a delete.

Backups target any S3-compatible store — R2, B2, MinIO (D-0025) — because a self-hosted tool that only backs up to one vendor is not self-hosted in the part that matters when a machine dies.

Provider credentials

Provider API tokens, tunnel tokens and S3 keys reach Quoro through a human-only providers resource (D-0051):

The resource needs the human-only scope provider:manage. Entries live in the replicated providers table, with the secret in secret_enc under the data-encryption key (D-0050). Secrets are write-only and are redacted from audit parameters, so neither a read call nor an audit record returns one. The audit-copy credential is a separate, put-only object_store entry, so backup credentials cannot delete the audit copy.

12. Identities, grants, audit and the AI-native loop

Identities

Humans (usr_, roles owner, operator, viewer) and agents (agt_: name, model, owner) are separate identities (D-0029). They are separate because the questions are different: for a human, “who”; for an agent, “which agent, on whose behalf, and because of what” (D-0029, D-0031). Role bundles fix which scopes a human has; the full mapping is in ../contracts/README.md and ../contracts/permission-model.md.

These are kept apart on purpose:

The split is what makes revocation immediate: the access token carries identity, not permission, and permission is read from the grants on every call. A token that is still valid but whose grant is gone authorizes nothing (D-0030). It is also why access tokens and sessions are local and unreplicated (section 6): they are short-lived secrets, and a replica does not need them to answer a read for a caller who authenticated to that replica. The token and key tables that do replicate carry hashes only, never a usable secret.

Grants

Agents hold an agent key (akey_) that authorizes nothing by itself. They exchange it for a short-lived access token (atk_), and every request is checked against the agent’s active grants at request time, so revoking a grant locks the agent out on its next call rather than when a token happens to expire (D-0030).

A grant (grt_) is: scopes + resource selector + TTL + optional max uses (D-0030). Selectors are a set of projects, apps and nodes, matched by “any”; {"all": true} matches everything and {} matches nothing (../contracts/README.md).

Three properties of the model matter architecturally:

• Incident grants. When an event wakes an agent, the grant issued with it carries only the scopes the binding names, selects only the affected resource, and expires with the incident or its TTL (D-0030, D-0032). The agent cannot widen it.
• Destructive scopes need JIT approval. The agent files an approval request (apr_), a human approves in the panel, and a short-lived extra grant is issued (D-0030). Approvals are checked in the service layer, not in the MCP layer, so no channel can bypass them.
• Human-only scopes are never grantable. node:enroll, node:remove, agent:write, grant:write, approval:decide, webhook:write and user:write cannot appear in an agent’s grant, so an agent cannot widen its own power, approve its own request, or create another agent (../contracts/README.md).

Refusals are part of the contract, not an afterthought: a refused request carries forbidden with a reason (scope_missing, selector_mismatch, grant_expired, grant_revoked, grant_exhausted, human_only, agent_disabled), and an action that needs approval carries approval_required with the scope and target so the agent can file the request itself (../contracts/README.md).

Audit

The audit log is a separate append-only hash chain, with each record appended in the same transaction as the action it records (D-0031). Refusals are recorded too: a denied restart is as interesting as a granted one, and the M3 demo relies on it (D-0033). A record holds who, on behalf of whom, the triggering event, the tool or endpoint, the parameters, the result and the agent’s justification (D-0031); the format and its verification are fixed in ../contracts/audit-log-format.md. The log can be copied off-site to object storage with object lock, out of reach of whoever operates the server (D-0031).

The log is a chain, not a table with a timestamp column: each record includes the previous record’s hash, so removing or editing an entry breaks every later link. audit.chain_broken is the event that says a verification found one (D-0031; ../contracts/README.md).

The channel of every action is recorded: api, panel, mcp, cli, system (../contracts/README.md). That is what makes “the agent and the human are logged the same way” a property of the design rather than a promise (D-0002).

MCP

The MCP server is rmcp’s Streamable HTTP transport mounted on the same Axum server at /mcp, plus a stdio mode through the CLI (F-0041). It calls the same service layer as the HTTP API, with the same scopes and the same audit log; there is no second path (D-0031). Authentication is the same bearer access token as the API, and every mutating tool takes a required justification and an optional incident_id, the MCP equivalents of the Quoro-Justification and Quoro-Incident headers (../contracts/README.md).

The tool set is the one fixed in ../contracts/README.md: reads for cluster, nodes, projects, apps, deployments, logs, events, incidents, approvals and audit, plus restart_app, redeploy_app, delete_app, resolve_incident, request_approval and get_approval. Each tool maps one-to-one onto an endpoint and needs the same scope.

The AI-native loop

D-0032 defines a four-step loop — detect, notify, wake, act. Each step has a named mechanism here; the incident sits between detection and notification.

1. Detect. Node agents run health checks and emit event messages; the control plane also derives detections from state (node.disconnected, certificate.expiring).
2. Event. The control plane records the event, which can be one of the types in ../contracts/README.md — app.down for a health check past its threshold, vm.crashed for an unexplained exit, node.disk_low, deployment.failed.
3. Incident. An event that a webhook binding is interested in opens an incident (inc_), and the incident scopes the grant the agent will get (D-0030, D-0032).
4. Wake. The webhook dispatcher delivers a signed event to the configured endpoint (Quoro-Signature, five-minute freshness window, retries with webhook_deliveries records). The receiver — an agent runner — starts the agent with Quoro’s MCP endpoint and the incident context.
5. Act. The agent acts under its incident grant. Anything destructive stops at an approval request and waits for a human (D-0030).

Events are defined in ../contracts/events.schema.json. The loop is the M3 deliverable (D-0033).

Agent runner. The wake step needs a runner that receives the webhook delivery, verifies Quoro-Signature, reads WebhookWake.agent_id and the incident id from the payload, holds that agent’s own agent key (akey_) configured by the operator at the runner (Quoro never sends credentials in a webhook), mints a 5-minute access token via POST /api/v1/auth/agent-token, and acts through Quoro’s MCP endpoint within the incident grant, passing incident_id and justification on every mutating tool. The runner is a reference example in examples/agent-runner/, outside the quoro binary, owned by the mcp workstream and delivered in M3 (D-0049); the runner contract is the one in this section.

13. Realtime transport (D-0057)

D-0057 decides the split, superseding D-0044: SSE for one-way streams and WebSockets for bidirectional sessions:

The reasons for SSE on the read paths: it is plain HTTP, so it passes reverse proxies and Cloudflare without upgrade handling; it reconnects automatically; and Last-Event-ID maps directly onto the seq cursor of events and logs, which gives a resume point for free. The reason for WebSockets on the interactive paths is that a terminal needs both directions on one connection.

The claim that Axum supports both natively — an sse module and a ws feature — is a Hypothesis; it was not re-verified in the research stage. The cost of using two mechanisms is one more client-side code path; the benefit is that the common read-only case never needs a WebSocket.

14. High availability: prepared for in the MVP

HA and backups are out of the MVP but prepared for (D-0033). “Prepared for” is a specific claim: each item below is a field, message or path that exists in M0 to M4 and is exercised in M5 or M6, not a design that would have to be retrofitted.

Lease modes

Two modes (D-0047), because D-0019 (self-fencing from M0) and D-0011 (apps keep running when the control plane is gone) would otherwise conflict during M1-M4, when there is only one control plane:

• local: issued for HA policy none or cold, where no other copy can start. It never expires when the arbiter is unreachable; it ends only by an explicit lease_revoke (move, delete, planned switchover) or when an operator declares the node lost, which requires a manual fence acknowledgement before a cold backup is restored elsewhere. That keeps D-0011 and the M2 demo true: stopping the control plane does not stop apps.
• expiring: issued for warm and hot from M5b. Duration-based, measured on the node’s monotonic clock from when the renewal was sent, minus a safety margin. The state path is held -> uncertain -> expired -> self-fence, and a lease is also revoked explicitly.

Fencing tokens are carried on every write to shared storage (D-0047), not only in the lease messages.

The distinction between two lease states is the core safety rule:

• uncertain means “arbiter unreachable”: the node keeps running what it runs and starts nothing new. It has not lost the lease; it cannot prove it still holds it.
• expired means “lease lost”: the node self-fences, stopping its stateful VMs (D-0019).

Conflating the two produces one of two failures: fencing on every network blip, or continuing to run a stateful app that another node has already been allowed to start. Treating them as separate states is what lets a partition be survivable on one side and safe on the other.

For an expiring lease the deadline is valid_until = t_send + D - M, where t_send is the node’s monotonic time when it sent the renewal the current grant answers, D is the grant’s duration_ms and M = max(1000 ms, D / 10) (../contracts/node-agent-protocol.md section 8.4).

Two properties matter in the table. Recovery from uncertain back to held is silent, because nothing was given up; recovery from expired is not, because the app was stopped and the fencing token moved on. And no transition ever goes directly from uncertain to starting something, which is the whole point of the state.

The authority never grants the lease to another holder before D + 2M after its last grant, so the old holder’s fencing completes first (../contracts/node-agent-protocol.md section 8.4).

Fencing tokens increase on every ownership change and stay the same across renewals (D-0073); they travel on every write to shared storage — chunk store, object store, replication stream — and receivers reject stale tokens (D-0019). A token is what makes “the old holder is definitely not writing” enforceable by the storage layer rather than by good behaviour.

Lease arbiter options

Both external arbiters still require fencing tokens (D-0019), and conditional-write support must be probed per provider at setup (F-0038). Neither is required for the self-hosted path: the witness is free and first-class (D-0017), and the external arbiters are what a managed offering can add.

Failback

Failback is never automatic and never immediate (D-0022). The six steps of the planned sequence:

1. The returning node rejoins fenced, checks its leases and starts nothing stateful.
2. It joins as a replica of the current lease holder, which is the source of truth.
3. Its divergent writes are kept in a quarantined snapshot and recorded in the audit log, never silently dropped.
4. Incremental resync runs from the last common point, and the node becomes the warm copy.
5. By default there is no failback: the roles simply swap.
6. If the user prefers the original node, a planned switchover waits for replication lag near zero and a low-traffic window, freezes writes for seconds, does a final sync, moves the lease, shifts traffic at the load balancer and drains connections.

A planned switchover loses no data. Only a surprise failure can lose seconds, and the amount is bounded by the replication interval (D-0020, D-0022). Stateless apps need none of this: the load balancer slow-starts traffic back to the returning node (D-0022).

15. Key flows

Each flow is written as the sequence of calls, records and messages it produces, using the exact names from ../contracts/README.md. “Change-log entry” means one numbered, hash-chained entry appended in the same transaction as the state table update (D-0013, D-0015); “audit record” means one entry in the audit chain, appended in the same transaction as the action (D-0031).

Four rules hold in every flow:

• Authenticate, then authorize, then act. The scope check happens in the service layer, against the grants as they are at that instant (D-0030). No channel is exempt.
• One transaction, one record. The state change, its change-log entry and its audit record commit together, so a crash cannot leave an action without a record or a record without an action (D-0013, D-0031).
• Idempotent by key. Every creating POST carries an Idempotency-Key; retries return the first result instead of creating a second object (../contracts/README.md).
• Correlatable end to end. Every request has a req_ id, echoed as Quoro-Request-Id; event ids evt_ and incident ids inc_ carry the thread from detection to action (D-0029, D-0031).

(a) Deploy (M2)

1. The caller sends POST /api/v1/apps/{app_id}/deployments with scope app:deploy, an Idempotency-Key, and — if it is an agent — Quoro-Justification. The channel is api, panel or mcp depending on the caller, and is recorded.
2. Authorization checks the caller’s identity, the scope against the app’s project through the grant’s selector, and whether a destructive scope needs approval (it does not; app:deploy is not destructive). A refusal is audited with its forbidden reason.
3. Change-log entries: the new deployments row with status pending, and the app’s current deployment pointer. Audit record: who, scope, parameters, result.
4. Event deployment.started.
5. The scheduler chooses a node and builds that node’s desired state at rev N+1, which now contains the new instance, its image reference and digest, and its domain.
6. The primary sends desired_state with rev N+1 on the node’s session.
7. The node persists rev N+1 before applying it (D-0011), then prepares the image: pull, verify the digest, build the rootfs, cache by digest so a redeploy of the same image is free.
8. The node creates and starts the microVM. The deployment status moves preparing, then starting; the VM’s observed status is a separate enum and follows its own transitions. A stateful app starts only while its lease is held (D-0019); otherwise the deployment stops here.
9. The health check passes, the node sends status_report and event vm.started.
10. The primary records deployment.succeeded, applies the route to the route table, and obtains a certificate through ACME or registers the tunnel route (D-0024).
11. The previous deployment becomes superseded and its microVM is removed. For a stateless app the new instance starts before the old one stops; for a stateful app the old one stops first, because there is a single writer (D-0019).
12. Any failure in steps 7 to 10 ends as deployment.failed with the failing phase, which is one of pull, rootfs, boot or health (the deployment phase field, not a status), and the node keeps the previous instance running where it still can.

(b) Restart by an agent (the M3 demo)

1. A node health check fails past its threshold, and the node sends event app.down.
2. The primary records the event and opens an incident inc_, emitting incident.opened (D-0032).
3. The webhook binding that matches the event delivers a signed event to its endpoint: headers Quoro-Event-Id, Quoro-Event-Type, Quoro-Delivery-Id and Quoro-Signature; a receiver rejects a signature older than five minutes.
4. An incident grant is issued and grant.created is emitted. Its scopes come from the binding, its selector is the affected app, and its TTL comes from the binding (D-0030).
5. The agent runner starts the agent with the incident context and Quoro’s MCP endpoint at /mcp.
6. The agent exchanges its agent key (akey_) for a short-lived access token (atk_). The key alone authorizes nothing (D-0030).
7. The agent reads state and logs through MCP: get_app, get_app_logs (scope app:logs), list_incidents and get_incident (scope incident:read).
8. The agent calls restart_app with a justification and the incident id (scope app:restart, endpoint POST /api/v1/apps/{app_id}/restart).
9. The service layer checks the grant: scope, selector, TTL, uses remaining, and agent_disabled. A refusal would be recorded with its reason (scope_missing, selector_mismatch, grant_expired, grant_revoked, grant_exhausted).
10. Change-log entry: the app’s restart_generation is bumped. Audit record: channel mcp, the agent, the human who configured the binding as “on behalf of”, the triggering event, the tool, the parameters, the result and the justification (D-0029, D-0031).
11. Event app.restarted.
12. The primary sends the new desired_state; the node restarts the microVM and reports status_report when it is running again.
13. Event app.up when the health check passes again.
14. The incident is resolved by the agent (resolve_incident, scope incident:resolve) or by a human; incident.resolved is emitted, and the incident grant ends with the incident or when its TTL passes.

(c) Node loss

1. The primary notices the session drop or missed heartbeats and emits node.disconnected. The node’s status becomes disconnected.
2. An incident can be opened and an agent woken, exactly as in (b), with scopes limited to the affected node and apps.
3. The node itself keeps running and reconciling its last desired state; local leases stay held, because a local lease does not expire while the arbiter is unreachable (D-0047). It redials with backoff and jitter.
4. Nothing is started elsewhere automatically before M6 (D-0016, D-0033). The panel shows the app as degraded and the N+1 indicator answers whether its apps would fit on the remaining nodes (D-0021).
5. To restore a stateful app elsewhere from a cold backup (M5a), an operator must first declare the node lost, as part of the local lease mode (D-0047), and acknowledge the manual fence. That revokes its leases and increments the fencing token, so a node that comes back cannot keep writing (D-0019).
6. From M5b, with expiring leases: the node’s renewals start failing, it goes uncertain (keep running, start nothing new), then expired, self-fences its stateful apps, and buffers node.fenced for replay on reconnect; if a forced stop cannot confirm a VM stopped, it emits node.fence_failed instead, marks itself failed, and the lease is not granted elsewhere until an operator acknowledges the fence (D-0074).
7. A warm copy may start only after the old lease has expired on the arbiter’s side plus the safety margin (D-0019). That ordering, not the detection time, is what prevents two writers.

(d) Node return

1. The node redials and sends hello carrying its applied desired-state rev, its change-log position and the leases it believes it holds.
2. The primary sends lease_revoke for any lease that is no longer the node’s, before anything else.
3. The node stops those stateful VMs first, before applying anything else (D-0022).
4. Divergent writes, if any, are kept as a quarantined snapshot and recorded in the audit log (D-0022; M5b). Nothing is discarded silently.
5. The primary sends the full desired_state; the node reconciles, and no duplicate app appears — this is the M2 demonstration that reconciliation is idempotent (D-0011).
6. Change-log follow resumes from the node’s last verified seq (D-0013).
7. There is no automatic failback: a planned switchover is a separate, explicit action (D-0022).

16. Stack confirmation (D-0056, superseding D-0028)

D-0056 fixes the baseline stack, keeping the same items D-0028 listed and replacing the graphical console with text-only consoles. This section confirms it item by item, with the refinements the research forced; D-0057 adds the realtime split.

Two of the “confirmed” rows carry a caveat about the dependency rather than about the choice. rmcp’s release cadence means the MCP layer must be isolated behind the service-layer boundary so a minor-version upgrade is a contained change (D-0048). utoipa’s empty-description behaviour (F-0042) means the API contract has a lint rule, not just a convention: every response must carry a non-empty description. Every dependency in the list must still be justified individually (D-0056).

Dependency justification

D-0056 requires that every dependency be justified, so each one above carries a reason rather than a precedent:

Not on the list: bollard, because there is no Docker on nodes (D-0008); noVNC, because consoles are text-only (D-0056, F-0034); a container runtime, because a container Engine is a later implementation (D-0009).

17. Workspace layout

This is the planned M0 cargo workspace layout. It follows the rule that each workstream owns its paths (D-0036), one crate or package per workstream, so that parallel work rarely touches the same file.

Two notes on the table. docs/contracts/ is not owned by a workstream after M0: it is frozen, and a change there is a standalone pull request that touches only the contract (D-0036). And crates/proto/ is separate from crates/node-agent/ so that the control plane can depend on the message types without depending on the agent implementation, which is what keeps the protocol contract reviewable on its own.

Build and checks

The workspace exists to make the contracts checkable in CI, not only readable:

• xtask validates the contracts: the OpenAPI document parses, every response has a description, the event schema validates its examples, the SQL schema runs in a fresh in-memory SQLite database (../contracts/README.md).
• xtask also generates the utoipa document and diffs it against the frozen openapi.yaml, failing on any semantic difference (D-0048).
• The panel builds against a client that orval generates from the frozen spec, so a panel change that needs a new field needs a contract change first (D-0048; D-0036 for the contract-change rule).
• Releases are signed and reproducible, because the installer is curl | sh (D-0027).

The dependency direction is one-way: crates/quoro depends on every role crate, crates/control-plane and crates/node-agent both depend on crates/types and crates/proto, and crates/engine depends on neither. That is what allows the engine and the control plane to be developed and tested in parallel, which is the reason parallel workstreams can run at the same time at all (D-0036).

Release and verification

D-0027 sets the requirements, and this document restates them so the installer workstream has one list:

• Every release binary is signed.
• The curl | sh installer verifies the signature against a published key before it runs anything.
• Builds are reproducible, including the panel’s JavaScript bundle embedded with rust-embed.
• Provenance is published.

The scheme is not chosen — for example minisign, or Sigstore plus SLSA provenance — and the installer workstream proposes it before M1, pending maintainer decision.

18. Open questions and recommendations

Every open item this document depends on. None of them is decided here.

Decided since the first draft, and therefore no longer listed above:

• Ingress proxy: D-0059.
• Git builds: D-0060.
• Image path, rootfs conversion and minimal init: D-0061.
• Disk snapshot primitive: D-0062.
• Node mesh: D-0063.
• Release milestones: D-0064.
• Manual promotion: D-0067.
• SQLite and catalog-template backups: D-0069 (answers Q-0026; a Litestream backend with SSE-C for a lower RPO may be added later as an option).
• Curated catalog: format, owner, location and signature scheme are decided at M4 planning, before the first template ships: D-0070 (answers Q-0060).
• HA policy costs for none and cold: D-0068 (answers Q-0025).
• Docs-site timing: D-0065 (answers Q-0009).
• Prerequisites for shipping expiring leases in M5b: D-0075.
• Realtime transport: D-0057 (supersedes D-0044).
• Graphical console dropped from the baseline stack: D-0056 (supersedes D-0028).
• Lease modes local and expiring: D-0047.
• API source of truth, generator and pinning: D-0048.
• Agent runner: D-0049.
• Cluster key algorithms, storage and rotation: D-0050.
• Provider credentials and the providers resource: D-0051.
• Node-local volumes in M5a: D-0052.
• Off-site audit copy in M5a: D-0053.
• Host layout, CLI and setup token: D-0054.
• Lab TLS for the M2 demonstration: D-0055.

Custody of the cluster keys on promotion is the only open row. Q-0024 fixed when promotion ships (M5b, D-0067) but not how a promoted replica obtains the keys; until promotion exists the keys do not leave the primary.

Q-0001 to Q-0006, Q-0009, Q-0024, Q-0025, Q-0026 and Q-0060 are decided (D-0059 to D-0065, D-0067 to D-0070) and are therefore absent from the table.

19. Hypotheses to verify

Every claim marked Hypothesis in this document, and what would settle it.

• Axum supports SSE and WebSockets natively. Used in section 13. Verify by building one text/event-stream route and one WebSocket route in the current Axum version and running them in CI; settle before the M3 event stream ships.
• An object store can support object lock on every provider we target. Used in section 11 for the off-site audit copy. Verify per provider with a real bucket: create a locked object, try to overwrite and delete it, and record the result.
• The available Rust ACME and TLS crates are mature enough to own the proxy path. Used in section 10 and D-0059. Verify with a spike before M2: obtain and renew one certificate for a real domain through the embedded proxy.
• An encrypting S3 proxy or SSE-C is operationally acceptable as a backup path. Used in section 6, options 1 and 4, and D-0069’s optional Litestream backend. Verify by restoring a primary snapshot from the bucket in a drill, not by reading documentation.
• BuildKit inside a builder microVM works under Cloud Hypervisor. Used in section 16 and D-0060. Verify with an M4 spike that runs one Dockerfile build end to end and produces an OCI layout.
• Railpack/Nixpacks are viable zero-config builders, with Nixpacks in maintenance and Railpack as its successor. Used in D-0060, which re-verifies them before M4. Verify before M4 by building one real repository with each.
• Running Cloud Hypervisor as a separate process keeps the AGPL-3.0 and Apache-2.0/BSD-3-Clause licenses cleanly separated. Used in section 2. This is a legal reading, not a reviewed conclusion. Verify with a license review before the first public release, which is the M3 preview (D-0064): a lawyer confirms that process separation is enough, or the distribution model changes.
• A crash-consistent block copy of a running database is not sufficient, so native replication is needed. Used in section 14 and D-0023’s exception for databases. Verify by restoring a chunk-engine copy of a running database and measuring what it contains.
• A local lease that never expires does not let two writers exist. Used in section 14 and D-0047. Verify with the M5b partition test: with the control plane unreachable, confirm that no other node can be told to start the stateful app.
• The witness can be small enough to run on any cheap machine in a different failure domain. Used in section 3 and D-0017, which puts the witness in a different failure domain from the nodes. Verify in M5b by running it on a third machine and cutting each link in turn.
• Decision to verify, D-0018: a single edge is a traffic single point of failure. With the edge also a voter, losing it removes both the traffic path and one vote. D-0018’s mitigations are Cloudflare as the traffic edge with the small VPS acting only as witness, or two edges with DNS failover. Verify in M5b that the chosen mitigation keeps both traffic and quorum when the edge dies.
• One tunnel per node with an HTTPS monitor on the app hostname gives per-node failover. Used in section 10. Verify in M5a by powering off a node behind its own tunnel and confirming the load balancer removes that endpoint and keeps serving from the other.
• The load-balancer health check interval is short enough for the RTO the warm policy promises. Used in section 14. Verify by measuring detection-to-recovery time in M5b.
• mkfs.ext4 -d plus a Rust OCI registry client is a workable rootfs pipeline. Used in section 8 and D-0061. Verify in the M0 spike by booting a microVM from a converted image.
• virtio-fs is only an optimisation and never a requirement. Used in section 8. Verify by confirming the M2 deploy path boots from a disk image with no virtiofsd present.

Quoro contracts

Status: M0 draft for review.

These files are the interfaces that Quoro’s workstreams build against in parallel. They are written before any code (D-0040) and are the M0 deliverable named in D-0033.

Contracts freeze at the end of M0. After the freeze, a contract change is a standalone pull request that touches only the contract and is reviewed on its own; it is never mixed into feature work (D-0036). If an implementation needs something a contract does not allow, stop and open an issue proposing the change.

Index

The architecture these contracts implement is described in ../architecture/architecture.md; threats and mitigations are in ../architecture/threat-model.md. Terms follow ../glossary.md.

Validation

Two checks run on every contract change.

• scripts/check_contracts.py (Python 3 standard library only; run python3 scripts/check_contracts.py): events.schema.json parses as JSON, its event-type list equals the Event types table, and every example’s type is in that list and carries the schema’s required top-level keys; db-schema.sql loads into an in-memory SQLite database, the migrations in migrations/ apply in order on top of it with schema_version ending at one plus their count, and its tables equal the Database tables list; in openapi.yaml (checked line by line, without a YAML parser) the document declares OpenAPI 3.1, every operation has x-quoro-scope with a scope from the table or a special value, every response has a description or a $ref, every x-quoro-mcp-tool matches the MCP tools table (endpoint and scope), and every event type is mentioned; every backticked scope in the markdown contracts exists in the Scopes table; the node-agent message names listed below equal the message headings of node-agent-protocol.md section 5, and every JSON example block in that file parses.
• Full semantic validation of openapi.yaml against OpenAPI 3.1 and of events.schema.json against JSON Schema 2020-12 needs third-party validators; it is not in the standard-library script and is added to CI with the cargo workspace in M0.

openapi.yaml is hand-written and is the frozen source of truth for the HTTP API (D-0048). The implementation’s utoipa output is generated and diffed against it in CI, and the panel client is generated from it with orval.

Shared vocabulary

Every contract uses exactly the names below. If a contract needs a name that is not here, add it here in the same pull request.

Roles and statuses

• Roles of the single quoro binary (D-0012): node, witness, edge. They combine (for example witness+edge, D-0018).
• Control-plane status of a node: primary (single writer and source of truth) or replica (holds a copy of the change log, serves reads, forwards writes) (D-0012). Primary and replica are statuses, not roles.
• The witness role stores no data (D-0017). In M1-M4 the cluster has exactly one primary and no witness (D-0016).

Identifiers

All ids are strings <prefix>_<ULID> (ULID in Crockford base32, 26 characters), for example app_01JC9Z6Q5W8N3K4T7R2M1XHB0D. Ids are never reused.

Change-log entries, audit records and events also carry a numeric, gap-free seq per log.

Node-agent protocol messages carry msg_ ids. They are message-local (they pair a request with its answer inside one session), are never stored, and are not resource ids.

Keys

The cluster has three keys (D-0050):

• cluster signing key: Ed25519. It signs desired_state and lease messages, and nodes pin it at enrollment. Rotation uses the node-agent message signing_key_rotate, signed by the old key.
• cluster data-encryption key: 256-bit key for XChaCha20-Poly1305. It encrypts every *_enc column; a *_enc BLOB stores the key id, then the 24-byte nonce, then the ciphertext with its tag, so that entries encrypted under different keys can coexist and are selected by key id, which is how rotation works.
• node-local key: encrypts a node’s local copy of secrets and never leaves the node.

The cluster keys are generated at the primary’s first start and stored as 0600 files under /var/lib/quoro, outside SQLite. They are backed up separately from the database: losing the data-encryption key loses every secret it protects. Custody of the cluster keys on promotion is open (Q-0024): the candidate is to replicate both keys wrapped to each replica’s node key, against re-keying on promotion with an operator-confirmed re-pin.

Resource relationships

• A project contains apps. A VM may belong to an app (app_id set) or stand alone (M1 lifecycle, app_id null). Standalone VMs belong to a project too.
• An app has deployments; the current deployment’s image runs as one or more VMs (instances) placed on nodes. In M1-M4 an app has one instance.
• An app is stateful or not. Stateful apps have volumes and may run only on their lease holder (D-0019).
• An app has an HA policy: none, cold, warm, hot (D-0020). Only none is implemented before M5a; the field exists from M0.
• A domain belongs to one app and has a TLS mode: acme (public-IP hosts) or tunnel (Cloudflare Tunnel) (D-0024).
• An app either deploys images given by reference or has a Git source (source): then its images come from builds of that source, and each successful build starts a deployment with trigger: build (ADR 0117). A build runs on the node that runs the app (ADR 0118).
• An interactive session is a web terminal into an app’s VM or a read-only console of a VM (ADR 0114). It lives on the primary while it is open and leaves two audit records, its open and its session.end (ADR 0115).

Built images

The image a build produces never leaves the node that built it (ADR 0118), so it has no registry reference. Every contract names it with the reference

build.quoro.invalid/<app_id>:<build_id>

and the build’s manifest digest as its digest, for example build.quoro.invalid/app_01JC9Z7B3N5Q7S9Z1W3Y5A7C9E:bld_01JC9ZA2B3C4D5E6F7G8H9J0KM with sha256:.... The host is under the reserved .invalid top-level domain (RFC 6761), so the reference parses as an ordinary OCI reference but can never be pulled: a node prepares it from the build’s OCI layout and never contacts a registry for it. App.image, Deployment.image, vms.image and desired_state spec.vms[].image all carry it verbatim.

Status enums

• Node status: pending (enrolled, never connected), connected, disconnected, fenced, removed.
• VM desired_status: running, stopped. VM observed_status: creating, stopped, starting, running, stopping, crashed, deleted, unknown.
• App status: deploying, running, degraded, down, stopped.
• Deployment status: pending, preparing (pulling image, building rootfs), starting, succeeded, failed, superseded.
• Build status: queued (waiting for the node), preparing (builder VM and source checkout), building, exporting (writing the OCI layout), importing (turning the layout into a root disk), succeeded, failed, cancelled (ADR 0117). The last three are terminal.
• Domain cert_status: pending, issued, renewing, failed, not_applicable (tunnel mode).
• Grant status: active, expired, revoked, exhausted.
• Approval status: pending, approved, denied, expired.
• Incident status: open, resolved.
• Lease mode: local (HA policy none/cold: never expires when the arbiter is unreachable; ends by lease_revoke or an operator declaring the node lost) or expiring (HA policy warm/hot, from M5b: held -> uncertain -> expired, self-fence at expired; revoked explicitly) (D-0047). Lease state as seen by a node: held, uncertain, expired, revoked. See node-agent-protocol.md.
• Webhook delivery status: pending, delivered, failed.

Scopes

A scope is <resource>:<verb> (D-0030). The complete list:

• destructive scopes need JIT approval when an agent uses them: the agent files an approval request, a human approves it, and a short-lived extra grant is issued (D-0030).
• human-only scopes can never appear in an agent’s grant, so an agent cannot widen its own power or approve its own requests.
• The wildcard app:* style is not allowed in grants; scopes are listed explicitly.
• Human roles are fixed scope bundles: owner (every scope), operator (every scope except user:write), viewer (every :read scope plus app:logs). user:read is a :read scope, so every role holds it. app:terminal and vm:console are not :read scopes, so a viewer holds neither (ADR 0114).
• Builds and metrics add no scope of their own: a build is authorized as a deployment of its app (app:deploy), its record as app data (app:read), its log as app output (app:logs); node metrics need node:read and app metrics app:read.

Resource selectors

A grant’s selector says which resources its scopes apply to. JSON form:

{ "projects": ["blog"], "apps": ["app_01J..."], "nodes": ["nod_01J..."] }

A resource matches if it matches any listed entry (projects by slug, others by id); an app matches projects through its project. {"all": true} matches everything. An empty selector {} matches nothing. String form for CLI and docs: project=blog, app=app_01J..., node=nod_01J..., all.

Event types

Event type is <subject>.<what> in lowercase with underscores. The complete list:

Errors

HTTP errors use application/problem+json (RFC 9457) with an extra code field. code values: unauthenticated (401), forbidden (403), approval_required (403), not_found (404), conflict (409), validation_failed (422), rate_limited (429), not_primary (503, write reached a replica that cannot forward), node_unreachable (504, on-demand telemetry could not reach the node), internal (500).

For forbidden, the field reason is one of: scope_missing, selector_mismatch, grant_expired, grant_revoked, grant_exhausted, human_only, agent_disabled. For approval_required, the body carries scope and the target so the agent can file an approval request.

Request headers

• Authorization: Bearer <token> (human API token or agent access token); the panel uses a session cookie.
• Idempotency-Key on every POST that creates something.
• Quoro-Justification: the agent’s stated reason for the call, required on every mutating call by an agent, recorded in the audit log (D-0031).
• Quoro-Setup-Token: the one-time setup token, required by POST /api/v1/setup only (D-0054).
• Quoro-Incident: the incident id the call belongs to, when an agent acts on an incident.
• Last-Event-ID on SSE streams (GET /api/v1/events/stream, GET /api/v1/apps/{app_id}/logs/stream, GET /api/v1/builds/{build_id}/logs/stream): resume after the given seq. The metrics streams (GET /api/v1/nodes/{node_id}/metrics/stream, GET /api/v1/apps/{app_id}/metrics/stream) carry no resumable position and ignore it. One-way streams use SSE; bidirectional sessions use WebSockets (D-0057): the web terminal and the VM console (GET /api/v1/interactive-sessions/{session_id}/ws).
• Every response carries Quoro-Request-Id (req_...).
• Webhook deliveries carry Quoro-Event-Id, Quoro-Event-Type, Quoro-Delivery-Id and Quoro-Signature: t=<unix seconds>,v1=<hex HMAC-SHA256 of "<t>.<raw body>" with the webhook secret>. Receivers reject a signature older than 5 minutes.

Operation annotations

Every operation in openapi.yaml carries x-quoro-scope: one scope from the table above, or a list when the request needs several, or one of four special values: public (no authentication, security: []), authenticated (any valid human or agent identity, no scope), human (any authenticated human identity, session or personal API token; an agent caller is refused with 403 forbidden, reason: human_only), node (node credential; only the node-agent session and enrollment). Operations that an MCP tool also exposes carry x-quoro-mcp-tool with the tool name.

MCP tools

M3 exposes these tools (D-0033: read plus restart and redeploy; delete exists so the approval flow can be exercised). Each calls the same service operation as the listed endpoint and needs the same scope (D-0031).

Every mutating tool takes a required justification string and an optional incident_id, the MCP equivalents of the Quoro-Justification and Quoro-Incident headers.

M4 adds no MCP tool. Builds, metrics and console sessions are reachable by agents through the HTTP API with a grant; the web terminal is human-only (ADR 0114); a later tool is a contract change of its own.

Channels

Every action reaches the same service layer through one of these channels, recorded in the audit log as channel: api (HTTP API), panel (the web panel, which uses the API with a session), mcp (MCP server, D-0031), cli (the quoro command), system (Quoro itself: reconciliation, schedulers, incident automation). MCP tools map one-to-one onto service operations and require the same scope as the matching endpoint.

Node-agent message types

Defined in node-agent-protocol.md. Node to control plane: hello, heartbeat, status_report, event, lease_renew, log_ack, stream_data, stream_end, command_result, dns_challenge. Control plane to node: welcome, desired_state, lease_grant, lease_revoke, log_entries, stream_open, stream_close, stream_input, stream_ack, command, signing_key_rotate, dns_challenge_result, error. Either side: ping, pong.

Database tables

Defined in db-schema.sql and changed only by the numbered files in migrations/: meta, change_log, nodes, node_join_tokens, node_credentials, node_tunnels, projects, vms, apps, app_env, deployments, builds, domains, volumes, leases, users, api_tokens, sessions, agents, agent_keys, access_tokens, grants, approvals, incidents, events, webhooks, webhook_deliveries, providers, audit_log, audit_exports, interactive_sessions. Tables marked local in that file are not replicated through the change log.

Open items before the freeze

Only key custody on promotion is still open. It does not block the freeze: promotion arrives in M5b and the key files and message shapes it needs already exist. The image pipeline and init are decided (D-0061), and so is the M5a snapshot primitive (D-0062), with an lvm-thin vs btrfs spike before M5b. Every other element formerly listed here is decided (D-0047, D-0048, D-0050, D-0051, D-0052, D-0053, D-0054, D-0056, D-0057, D-0061, D-0062, D-0067, D-0071, D-0072, D-0073, D-0074).

Permission model

Status: M0 draft for review.

This contract fixes how Quoro decides whether a caller may perform an operation. It implements agents as first-class identities (D-0029), grants made of a scope, a resource selector, a TTL and an optional maximum number of uses (D-0030), one authorization path shared by the HTTP API and MCP (D-0031), and the incident-grant and just-in-time approval flows (D-0030, D-0032). The M3 scope it must satisfy is the M3 Definition of Done in ../product/roadmap.md (D-0033).

Contracts freeze at the end of M0; after the freeze a change to this file is a standalone pull request (D-0036). If an implementation needs something this contract does not allow, stop and open an issue proposing the change (D-0036).

Vocabulary, ids, scopes, reasons, roles, channels and MCP tool names are defined once in README.md, where the tool list is the table in section MCP tools, and are used here unchanged. The audit record this contract produces is specified in audit-log-format.md.

1. Purpose and principles

Quoro is operated by humans and by AI agents over the same state (D-0002). This contract exists so that an agent can be given real power over infrastructure without that power being open-ended, unnamed or permanent (D-0030).

The principles, each of which the rest of this file turns into a rule:

1. An agent is an identity, not a shared password. It has a name, a model and an owning human, and every action records who acted, on behalf of whom and why (D-0029).
2. Authority is a grant, never a property of a key. An agent key authorizes nothing by itself; it can only mint short-lived access tokens, and those tokens carry no scopes. What an agent may do is read from its grants on every call (D-0030).
3. Every grant is bounded in time. A TTL is mandatory; there is no grant without an expiry (D-0030).
4. Every grant is bounded in space. A grant names the resources its scopes apply to through a selector; an empty selector matches nothing (D-0030).
5. Least privilege by construction. Agents hold the scopes an incident or a task needs, nothing more; incident grants are issued with exactly the scopes the webhook declared (D-0030, D-0032).
6. Destructive actions are off by default. An agent that needs one asks a human, and the human’s approval produces a separate, single-use grant for that one call (D-0030).
7. One gate for every channel. The API, the panel, the CLI, MCP and Quoro’s own automation reach the same service layer and the same authorization check; MCP has no second path (D-0031).
8. Refusals are evidence. A refused call is audited with the reason, not dropped (D-0031).

The threat this model addresses is a capable agent acting with more authority than the situation warrants, and nobody being able to reconstruct afterwards what it did and why (D-0029, D-0031). Threats and their mitigations are listed in ../architecture/threat-model.md.

1.1 Channels

Every action arrives through one of the channels in the vocabulary: api, panel, mcp, cli, system (README.md, “Channels”). The channel is recorded in the audit record but does not change authorization: the same identity with the same grant gets the same answer through any of them (D-0031).

A system action is taken by Quoro itself, for example reconciliation, a scheduler, incident automation or the audit exporter. It is not a caller with a credential; it is an internal actor whose actions are audited like any other (D-0029).

2. Identities and credentials

There are four identity types: user, agent, node and system (README.md, “Identifiers”; the audit record uses the same four values in actor_type). They are separate: an agent is not a human session with a different name (D-0029).

Rules on credentials:

• Secrets are shown once. A node join token, a personal API token, an agent key and a webhook secret are returned by the call that creates them and never again; only a hash (or, for the webhook secret, an encrypted value) is stored (README.md, “Identifiers”; D-0030).
• Personal API tokens (tok_) can be listed, created and revoked only by a human; an agent’s access token is refused with human_only, so an agent can never obtain a credential that escapes its grants (D-0029, D-0030).
• Provider credentials are managed only by humans. Providers hold the secrets of external services (DNS, tunnel, load balancer, object storage), so every operation needs provider:manage, which is human-only: an agent is refused with human_only (D-0051). A provider secret is write-only, encrypted with the cluster data-encryption key, never returned by a read and redacted from audit parameters (D-0050, D-0051).
• Access tokens are opaque. An access token is 32 random bytes encoded as an atk_-prefixed string. It is not a JWT and carries no claims: scopes, selectors and expiry of authority live in the grant rows, not in the token (D-0030).
• Access tokens expire in 5 minutes and are minted against an agent key on demand. A long-running agent re-mints; nothing about the re-mint widens its authority, because the grants are read per call (D-0030).
• An expired or unknown access token is unauthenticated (401). The caller re-mints from its agent key; the agent key itself is never sent to an ordinary endpoint.
• Disabling an agent invalidates its access tokens immediately. agents.disabled is read as part of the identity lookup on every request, before any grant is consulted, so the next call after disabled is set answers forbidden with reason agent_disabled (D-0029, D-0030).
• Revoking an agent key stops new access tokens (agent_keys.revoked_at); already-minted access tokens die at their 5-minute expiry, and any grant check they reach still applies (D-0030).
• The node credential never authorizes an API call. It authenticates the node-agent session and enrollment only; every operation annotated node in openapi.yaml is reachable with it and nothing else is (README.md, “Operation annotations”; D-0011). A node therefore cannot read the audit log, change a grant or act as a human.
• A human’s session cookie is not a bearer token. It is accepted only by the panel origin and never by the MCP endpoint; an MCP client authenticates with an access token (D-0031).

3. Human roles

A human’s authority is their role. Roles are fixed bundles of scopes, not grant rows: a human holds every scope in their bundle for as long as the role is theirs (README.md, “Scopes”).

Consequences of the bundles:

• A viewer can read apps, deployments, nodes, events, incidents, grants, approvals and the audit log, and can stream app logs, but cannot mutate anything (README.md, “Scopes”).
• An operator cannot create or change human users; that is user:write and owner-only (README.md, “Scopes”).
• Every role holds user:read, so an operator or viewer can list users and turn a user id (owner_user_id, on_behalf_of, decided_by, created_by) into a name. user:read is human-only: agents never list users.
• app:terminal and vm:console are in the owner and operator bundles and not in viewer, because neither is a :read scope (ADR 0114). A viewer can read a build’s log (app:logs) but cannot start a build.
• Human roles are not grantable to agents. An agent never receives a “role”; it receives grants (D-0029, D-0030).
• role is a column on the user (users.role), so changing a role changes every subsequent decision for that human with no re-issue step (D-0029).

A human may hold personal API tokens; a token inherits the human’s role at call time, so demoting or disabling the human takes effect on the next call (D-0029).

Own password. Every human may change their own password, whatever their role, with POST /api/v1/auth/password (openapi.yaml, changePassword; ADR 0135). It is not a scope: it needs no user:write, acts only on the caller and proves the current password. It is accepted only with the panel session cookie; a request with an Authorization header is unauthenticated, so a personal API token cannot turn into a new password and an agent never reaches it. A successful change ends the user’s other panel sessions; the calling session stays valid and personal API tokens are not touched (revoke them with DELETE /api/v1/tokens/{token_id}). Interactive sessions opened from an ended panel session close with close_reason credential_expired at their next re-check (section 7.5). Changing another user’s password stays updateUser with user:write.

4. Scopes

A scope is <resource>:<verb> and is the unit of authority in both human roles and agent grants (D-0030). The complete list, with the flags this contract depends on:

4.1 Destructive scopes

Four scopes are destructive: node:remove, project:delete, vm:delete and app:delete (README.md, “Scopes”). They remove data or capacity and cannot be undone by repeating a call.

This contract fixes their M3 behaviour:

• A standard grant can never contain a destructive scope. Attempting to create one is validation_failed at the grant endpoint.
• An incident grant can never contain a destructive scope. webhook.wake.grant_scopes is validated when the webhook is saved, so an incident can never hand an agent the power to delete (D-0032).
• An agent reaches a destructive scope only through an approval grant, that is a grant with kind approval, created by a human’s decision on an approval request (D-0030). Section 9 defines the flow.
• A destructive call with no matching approval grant answers approval_required (403), not forbidden: the agent is not told “never”, it is told “ask first” (README.md, “Errors”; D-0030).

Making destructive scopes grantable in standing grants would be a change to this contract and therefore to an accepted decision (D-0030 says destructive actions are off by default), not an implementation detail (D-0036).

node:remove is destructive and human-only, so no agent can delete a node at all: an approval grant can carry a destructive scope, never a human-only one (README.md, “Scopes”).

4.2 Human-only scopes

Eleven scopes are human-only: node:enroll, node:remove, node:fence_ack, app:terminal, agent:write, grant:write, approval:decide, webhook:write, provider:manage, user:read and user:write (README.md, “Scopes”).

• A grant whose scopes contains a human-only scope is refused at creation with validation_failed, and a hypothetical row that contained one is refused at call time with reason human_only (README.md, “Errors”).
• The set is chosen so an agent can never widen its own power (agent:write, grant:write, webhook:write), approve its own request (approval:decide), take over cluster membership (node:enroll, node:remove), declare a possibly running VM stopped so its lease moves (node:fence_ack, D-0079) or hold external credentials (provider:manage) (D-0030, D-0051). user:read is human-only so an agent never reads the list of human identities and their emails. app:terminal is human-only in M4 because a shell in the app’s VM reads every secret the app holds and acts outside single audited calls (ADR 0114); a later, non-interactive exec scope reachable only through an approval grant is a separate decision.
• Because approval:decide is human-only, “an agent approves its own destructive request” is not a policy that could be misconfigured; it is unrepresentable (D-0030).

4.3 Wildcards

Grants list scopes explicitly. app:*, * and any other wildcard form are rejected with validation_failed (README.md, “Scopes”). The only wildcard in the model is the selector form {"all": true}, which is a resource selector, not a scope (section 5).

5. Resource selectors

A grant’s selector says which resources its scopes apply to. JSON form:

{ "projects": ["blog"], "apps": ["app_01J..."], "nodes": ["nod_01J..."] }

and the string form used by the CLI and by documentation: project=blog, app=app_01J..., node=nod_01J..., all (README.md, “Resource selectors”).

Matching rules:

• A resource matches if it matches any listed entry. Entries are combined with OR, not AND.
• Projects are matched by slug, never by prj_ id; apps and nodes are matched by id (README.md, “Resource selectors”).
• An app matches projects through its own project_id’s slug, so project=blog covers every app in the blog project.
• {"all": true} matches everything and is the only way to say “the whole cluster”.
• {} (the empty selector) matches nothing. A grant with an empty selector is valid to store and useless to use; it is refused at call time with selector_mismatch.

Matching per target type:

5.1 Cluster-level scopes

cluster:read, event:read and audit:read have no resource of their own:

• They are matched by any non-empty selector. A grant carrying event:read with selector {"projects":["blog"]} may read the event stream, but only records whose subject or project belongs to blog (constraint below).
• Results are filtered, never refused. A list call under a selector returns the matching items; it does not answer forbidden because some rows in the table are out of scope. Filtered lists are the normal outcome for agents holding narrow grants (D-0030).
• Events are filtered by subject (and by project_id when set).
• Audit records are filtered by target_type/target_id and project_id, and by actor_id when the selector names apps or nodes the actor belongs to. A record with no target (for example a cluster-level write) is visible only to a selector of {"all": true}.
• A grant whose selector is {} reads nothing under these scopes, consistently with section 5.

The filtering above is a read-path rule; it does not create a second permission check. An implementation that returns everything to a narrowly scoped agent and relies on the panel to hide rows violates this contract (D-0030).

6. Grants

A grant is the only carrier of agent authority (D-0030). Its fields:

The field names match the shared entity model and the grants table in db-schema.sql.

6.1 Kinds

standard is the everyday grant: a human lends an agent a few scopes on a few resources for a shift or a task (D-0030). incident and approval are the two automatic kinds, defined in sections 8 and 9 (D-0030, D-0032).

6.2 Lifecycle

status moves in one direction:

active ──TTL passed──────────▶ expired
   │
   ├──── revoked by a human──▶ revoked
   │
   └──── last use consumed────▶ exhausted

• A grant is created active (event grant.created).
• The transition to expired, revoked or exhausted emits grant.expired or grant.revoked; grant.expired carries reason ttl or max_uses in its data (README.md, “Event types”).
• revoked and expired are terminal; there is no re-activation of a grant id. A new grant is a new row with a new id (README.md, “Identifiers”: ids are never reused).
• exhausted is computed from uses >= max_uses, so a grant with max_uses 1 that authorized one call is exhausted even if its TTL has not passed.

6.3 TTL

expires_at is mandatory on every grant. There is no default of “forever” and no way to omit it: the grant endpoint answers validation_failed without it (D-0030).

• The TTL is absolute from created_at; it is not refreshed by use. An agent that needs longer asks for a longer grant (D-0030).
• Expiry is evaluated against the control plane’s clock at decision time, not against any node’s clock.
• A grant at or past expires_at is expired and authorizes nothing, even for a read (D-0030).

6.4 Maximum uses

• max_uses counts authorized mutating calls, that is calls that pass authorization and are executed with result ok.
• Reads do not consume uses. An agent can poll get_app under a one-use grant as often as it likes (D-0030).
• A refused call does not consume a use; a failed call (result error after authorization) does not consume a use either, because the use is incremented with the action in the same transaction (section 7.3) and an action that did not commit did not happen.
• max_uses of 1 is the shape of every approval grant (section 9).

6.5 Revocation

• A human with grant:write revokes any grant; the system revokes incident grants when their incident resolves (section 8).
• Revocation is checked on every request, not cached and not attached to the token. The next call after revocation is therefore refused with grant_revoked, which is how “locks the agent out within seconds” (D-0030) holds: the bound is the next request, not a refresh interval.
• Revoking a grant does not revoke the agent key or delete audit records; it removes the authority the grant lent (D-0030).
• Revocation emits grant.revoked and is itself audited.

6.6 Issuing grants

POST /api/v1/grants requires grant:write and is human-only. The service refuses with validation_failed:

• a scope that does not exist;
• a wildcard scope;
• a human-only scope;
• a destructive scope on a standard grant (section 4.1);
• a missing expires_at, an expires_at in the past, or a max_uses below 1;
• a selector that is not the documented shape.

The created grant emits grant.created and is audited with the issuing human as actor (created_by_type user) (D-0030, D-0031).

7. Authorization

Authorization is one function called by the service layer. The API, MCP, the panel, the CLI and Quoro’s own automation call it; none of them has a bypass (D-0031).

7.1 Inputs

authorize(
  identity,        -- resolved from the credential: {type, id, role?, disabled?, model?}
  operation,       -- endpoint name or MCP tool name, e.g. "restart_app"
  required_scopes, -- one scope or a list, from x-quoro-scope
  target,          -- {type, id} or null for list calls
  params,          -- the request body/query as the caller sent it
  channel,         -- api | panel | mcp | cli | system
  justification,   -- agent's stated reason (required for mutating agent calls)
  incident_id,     -- optional, from Quoro-Incident
  approval_id      -- optional, only for destructive calls
) -> Decision { allow, grant?, reason?, approval? }

operation is the endpoint or MCP tool name from openapi.yaml; required_scopes comes from the operation’s x-quoro-scope, so a tool exposed over MCP inherits exactly the scope of its endpoint (README.md, “Operation annotations”; D-0031).

7.2 Decision procedure

1. resolve identity
   - unknown/expired credential            -> 401 unauthenticated
   - node credential on a non-`node` op    -> 403 scope_missing
   - system identity                       -> allow (internal actor, audited)

2. if identity.type == "user":
     if required_scopes ⊆ role_bundle(identity.role) -> allow
     else -> 403 scope_missing

3. if identity.type == "agent":
     a. if human_only ∩ required_scopes ≠ ∅         -> 403 human_only
     b. if identity.disabled                        -> 403 agent_disabled
     c. candidates = grants where
          agent_id == identity.id
          AND status == active
          AND expires_at > now
          AND (max_uses is null OR uses < max_uses)
          AND required_scopes ⊆ scopes
     d. if candidates is empty :
          if destructive ∩ required_scopes ≠ ∅        -> 403 approval_required
          else -> 403 scope_missing
     e. candidates = candidates where selector matches `target`
        (for list calls: filter the result rows by selector, never refuse)
     f. if candidates is empty                       -> 403 selector_mismatch
     g. if destructive ∩ required_scopes ≠ ∅ :
          keep only grants with kind == "approval"
            whose approval_id's target == target
            and whose approval_id's params_hash == sha256_jcs(params)
          if none                                   -> 403 approval_required
     h. grant = candidate with the earliest expires_at
        (ties broken by earliest created_at, then by lowest id)
     i. mutating call: in one transaction
          re-read grant FOR UPDATE; re-check expiry, revocation, uses, status
          perform the action
          uses += 1
          if max_uses is not null and uses == max_uses -> status = exhausted, emit grant.expired(reason=max_uses)
          append the audit record with result "ok" and change_seq of the action
          commit
        read call: no use is consumed, no transaction beyond the read
     j. allow, carry grant.id into the audit record

Step (a) before step (c) is what makes human_only the answer even when a malformed grant row exists and even for a disabled agent. Step (b) before the grant query is what makes disabling an agent immediate.

7.3 The transaction

An authorized mutating call writes three things atomically: the state change (through the change log, D-0013), the increment of uses and the audit record (D-0031). Either all three exist or none does. This is why “the action exists but its audit record does not” is not a state the system can reach (see audit-log-format.md).

The transaction re-reads the grant after step (h) so that a revocation or expiry that lands between selection and commit still refuses the call. The bound on “within seconds” (D-0030) is then stricter than seconds: it is the next commit.

7.4 Refusal reasons

Every refusal maps to an HTTP application/problem+json body with code and, for forbidden, a reason:

Order. When several reasons could apply, the answer is the first that applies in this order: human_only, agent_disabled, scope_missing, selector_mismatch, grant_expired, grant_revoked, grant_exhausted. The order is part of the contract because the M3 demo prints the reason (docs/product/roadmap.md, M3 Definition of Done): an agent that reads grant_expired learns something different from an agent that reads selector_mismatch.

Implementation note: the candidate query in step (c) filters on status and expiry for speed, so reasons grant_expired, grant_revoked and grant_exhausted are produced by a second, unfiltered lookup over the agent’s grants when step (d) or (f) would otherwise answer scope_missing or selector_mismatch. The unfiltered lookup is only consulted to choose the reason; it never authorizes.

approval_required body. It carries scope (the single destructive scope) and target (type, id), so the agent can file an approval request without guessing (README.md, “Errors”). It may also carry operation and params_hash as convenience; the approval endpoint recomputes both from the request body it receives.

Every refusal is audited with result refused and result_detail set to the reason (approval_required included). A refusal names no change_seq because nothing changed (D-0031).

Authentication failures are not refusals. An unauthenticated (401) call has no identity to attribute, so it is written to the server log, not the audit chain (D-0031; section 11).

7.5 Interactive sessions

A web terminal or a VM console is opened by one authorized call and then stays open for minutes or hours over a WebSocket (ADR 0057, ADR 0114). The open is authorized like any call: app:terminal for POST /api/v1/apps/{app_id}/terminal-sessions (human-only, so an agent is refused with human_only at step 3a), vm:console for POST /api/v1/vms/{vm_id}/console-sessions (a human’s role, or an agent’s grant whose selector covers the VM). The answer is a single-use ticket valid for 30 seconds; the WebSocket upgrade presents the ticket instead of a credential (openapi.yaml, connectInteractiveSession).

Because one authorization would otherwise cover hours of activity, the primary re-checks an open session and closes it, with the close_reason in brackets, when:

An agent access token lives 5 minutes (section 2), so an agent’s console session ends at that token’s expiry unless the agent opens a new one; console sessions are meant for short reads of boot output, not for permanent tails. The re-check consumes no grant use: only the open is a call. Every close, for any reason, writes the session.end audit record (audit-log-format.md, section 3.3).

7.6 Session transcripts

A terminal session’s transcript holds every byte typed and printed, so it holds whatever secret appeared on the screen (ADR 0115). GET /api/v1/interactive-sessions/{session_id}/transcript therefore needs audit:read and a human caller: an agent holding audit:read is refused with human_only, because an agent may not read through a transcript what it may not obtain through a terminal (ADR 0114). Every transcript fetch is audited, also when a human makes it (audit-log-format.md, section 3).

8. Incident grants and the AI-native loop

D-0032 defines the loop: detect, notify, wake, act. This section fixes the permissions half of the wake and act steps.

8.1 The webhook’s wake configuration

A webhook may carry a wake object (shared entity model; events.schema.json):

• grant_scopes is validated when the webhook is saved: a human-only scope or a destructive scope receives validation_failed. A webhook therefore cannot be used to route around section 4.1 (D-0032).
• webhook:write is human-only, so the agent that will be woken cannot configure its own future grant (README.md, “Scopes”).
• created_by is the human who saved the webhook and becomes on_behalf_of of every incident grant the webhook causes, which is how “on whose behalf” is answered for a wake nobody was present for (D-0029).
• wake can be changed with PATCH /api/v1/webhooks/{webhook_id} but not removed. To stop a webhook from waking an agent, delete the webhook and create it again without wake; the new webhook gets a new signing secret.

8.2 When the grant is issued

1. A detection (node agent report, health check, certificate state) opens an incident with subject {type, id} (D-0032).
2. incident.opened is emitted.
3. For each enabled webhook whose event_types matches and whose selector covers the incident’s subject, the system delivers the event and, if the webhook has wake, issues an incident grant:
   • kind incident;
   • agent_id = wake.agent_id;
   • scopes = wake.grant_scopes, copied verbatim;
   • selector = the incident subject only — an app id, a node id or a project slug, never a broader selector;
   • expires_at = now + wake.grant_ttl_s;
   • max_uses = wake.grant_max_uses;
   • incident_id = the incident;
   • on_behalf_of = webhook.created_by;
   • created_by_type system, reason naming the incident and the webhook.
4. grant.created is emitted and the issuance is audited with actor_type system.
5. The delivery body is the incident.opened event with data.wake = {agent_id, grant_id} added: the woken agent and the incident grant just issued (events.schema.json, data_incident_opened). The signature covers it like the rest of the body, and every retry of that delivery carries the same data.wake. A delivery whose webhook issued no grant (no wake, or the incident was resolved before the delivery) carries no data.wake. The event log itself never carries it.

8.3 When the grant ends

• Revoked by the system when the incident resolves. POST /api/v1/incidents/{incident_id}/resolve revokes every incident grant with that incident_id, emitting grant.revoked (D-0030, D-0032).
• Otherwise it ends by TTL or by max_uses, like any grant (section 6).
• A resolved incident does not delete its grants; they stay visible as revoked so the audit trail can be read later (D-0031).

8.4 Worked example

The D-0030 example, in this contract’s terms: an app on one node goes down.

• Detection opens inc_01J... with subject { "type": "app", "id": "app_01J..." }.
• A webhook with wake = { "agent_id": "agt_01J...", "grant_scopes": ["app:read", "app:logs", "app:restart", "incident:read", "incident:resolve", "approval:create"], "grant_ttl_s": 3600 } fires.
• The agent receives grt_01J...: those six scopes on {"apps":["app_01J..."]}, one hour.
• The agent calls get_app, then restart_app with a justification naming the incident. Both pass; only the restart consumes a use, and max_uses is null so the hour is the only bound.
• The app comes back; the agent calls resolve_incident (it holds incident:resolve because the webhook declared it; without it, the agent reports and a human resolves). On resolve, the grant becomes revoked.

A grant for one hour on one app, and nothing else, is the whole point: the agent could not have restarted a different app, read a project it was not woken for, or deleted anything (D-0030).

9. Just-in-time approval for destructive actions

D-0030 requires that destructive actions be off by default and reachable only after a human approves a specific call. The flow, with the exact endpoints and events:

1. The agent attempts the destructive call. For example DELETE /api/v1/apps/{app_id} (scope app:delete) or the delete_app MCP tool (D-0031, README.md “MCP tools”).
2. The service refuses with approval_required (403) carrying scope and target. Nothing is deleted, and the refusal is audited with result refused, result_detail approval_required (D-0030).
3. The agent files a request: POST /api/v1/approvals with approval:create, body {scope, target, operation, params, justification, incident_id?}. The service computes params_hash as the SHA-256 hex of the RFC 8785 canonical serialization of params, stores status pending and expires_at = now + 1 hour, and emits approval.requested (D-0030).
4. A human decides in the panel with approval:decide, which is human-only: the panel shows the operation, the target, the parameters and the agent’s justification, and the human approves or denies (D-0030; README.md “Scopes”).
5. On approve the system, in one transaction:
   • sets status approved, decided_by, decided_at, optional decision_note;
   • creates an approval grant: kind approval, scopes = exactly the one requested scope, selector = that target only, max_uses 1, expires_at = now + 10 minutes, approval_id set, on_behalf_of = the deciding human;
   • emits approval.approved and grant.created;
   • audits the decision with actor_type user.
6. The agent repeats the same call with the same params. Authorization matches the approval grant by target and params_hash; the call executes; uses reaches 1; the grant becomes exhausted and emits grant.expired with reason max_uses (section 6.2). The audit record carries approval_id and grant_id (D-0030, D-0031).

The same call with different params is a different call: params_hash differs, no approval grant matches, and the answer is approval_required again. This is what stops “approve a harmless restart, then send a delete” (D-0030).

Denial and expiry:

• Denied: POST /api/v1/approvals/{approval_id}/deny with approval:decide and a note; status denied, approval.denied, no grant. The agent’s retry answers approval_required (D-0030).
• Expired: a pending request past its expires_at (default 1 hour) becomes expired and emits approval.expired; no grant is created and a late decision answers conflict (409) (D-0030).
• Already decided: a second decision on the same request answers conflict (409); approval is not idempotent in the “make it work twice” sense, it is idempotent in the “cannot double-issue a grant” sense.

Who may approve:

• Any human holding approval:decide: owner and operator (README.md, “Scopes”).
• Never the agent that filed it, never another agent, and never a caller reaching the endpoint through MCP: approval:decide is human-only, so an MCP client cannot call the decide operation at all (D-0030).
• The deciding human is recorded as decided_by and as the approval grant’s on_behalf_of, so the audit record of the destructive call names the agent that did it and the human who allowed it (D-0029).

What approval does not do: it does not widen the agent’s other grants, it does not create a reusable permission, and it does not pre-approve a class of future calls (D-0030).

10. MCP mapping

M3 exposes the tools listed in README.md (D-0033: read plus restart and redeploy; delete exists so the approval flow can be exercised). The tool list is the table in README.md, section MCP tools. Each calls the same service operation as the listed endpoint and requires the same scope (D-0031).

Rules that follow from D-0031 and are not optional:

• Every mutating tool takes a required justification string and an optional incident_id. They are the MCP equivalents of the Quoro-Justification and Quoro-Incident headers (README.md, “MCP tools”; D-0031). A mutating tool call without justification is validation_failed, not an unaudited success.
• MCP clients authenticate with the same bearer access token as the API: Authorization: Bearer atk_.... There is no MCP-specific identity, and a human session cookie is not accepted by the MCP endpoint (D-0031).
• approval:decide has no MCP tool. The absence is deliberate: an agent cannot approve anything through MCP (D-0030).
• Tool and endpoint annotations must agree. Each operation in openapi.yaml carries x-quoro-scope and, when a tool exposes it, x-quoro-mcp-tool; a tool whose annotation disagrees with the README table fails the contract check.
• The MCP server is a thin adapter. It validates the tool arguments, calls the service operation and returns its result; it never queries the database directly and never constructs an audit record (D-0031).
• rmcp is pinned to an exact minor version, because the SDK shipped three semver-major releases between March and July 2026 and upgrades are scheduled work rather than a range bump (F-0041; docs/research/rust-api-and-mcp-tooling.md). The transport is Streamable HTTP mounted as a Tower service; no session state is assumed, since recent spec revisions remove protocol sessions (F-0041).
• Inbound token validation is Quoro’s own middleware, not something the SDK is documented to provide: server-side OAuth middleware is not documented in rmcp, so the bearer check is implemented on the Axum side (F-0041; docs/research/rust-api-and-mcp-tooling.md, labelled a Hypothesis there for the absence of documentation).

11. Accountability

Every action, allowed or refused, records the same accountability fields (D-0029, D-0031). The full record is specified in audit-log-format.md; this section fixes what each identity type puts in them.

Per identity type:

Two consequences worth stating plainly:

• An agent action always names three parties: the agent, its model and the human whose authority it borrowed (D-0029).
• A system action that a webhook caused still names the human who created the webhook, through the incident grant and trigger_event_id. There is no unattributed automation (D-0029, D-0032).

A human’s read calls are not recorded; an agent’s reads are (D-0031). That asymmetry is deliberate: the log exists to make agent behaviour reconstructable, and the volume of human browsing would bury it (see audit-log-format.md, “What is recorded”).

12. Rate limits

• Rate limits are enforced per identity: a human token and an agent access token each get their own bucket, and sharing an agent key does not pool two agents into one bucket (D-0029).
• Exceeding a limit answers rate_limited (429) in application/problem+json.
• The values are implementation defaults, not contract. This contract fixes only that the limit is per identity and that the answer is rate_limited; an implementation may choose the numbers and change them without a contract change (D-0030).
• A rate-limited call is audited as refused with result_detail rate_limited when the identity is known, so a burst that an agent produced is visible in the log (D-0031).
• POST /api/v1/auth/password is limited per user, counting attempts with a wrong current password, so a stolen session cannot be used to guess the password (ADR 0135).

13. Worked examples

The scenarios cover the M3 Definition of Done in ../product/roadmap.md (D-0033). “Audit result_detail” is the field on the record this call produces.

Two rows are worth reading together. Row 7 and row 8 are the same call: the second one succeeds only because a human approved exactly that call with exactly those params. Row 9 shows what happens when the approval is not there any more — the agent does not inherit a permanent right to delete from having once been approved (D-0030).

14. What this contract does not decide

• Whether destructive scopes may ever go into standing grants. This contract says no for M3 (D-0030: destructive actions are off by default). Changing it needs a decision, not a patch (D-0036).
• The exact numeric rate limits. Implementation defaults (section 12).
• How long a human session lives beyond the 24 h absolute expiry, and whether refresh tokens are added. The contract fixes that a session expires absolutely; anything more is a later decision (D-0029).
• Grant approval by two humans (four-eyes). Not in D-0030 and not in this contract.
• An external identity provider. Quoro authenticates its own humans with a password checked against an Argon2id hash and with its own tokens; no SSO path exists in M0-M6 (D-0029; users.password_hash in db-schema.sql).
• The off-site copy of the audit log, which the audit contract fixes in shape. It ships in M5a with the S3-compatible path, after an object-lock probe (D-0053; audit-log-format.md).

Audit log format

Status: M0 draft for review.

This contract fixes the shape of the audit log, how records chain, how the chain is verified and how a copy is written off-site. It implements D-0031: the audit log is append-only, hash-chained, optionally copied to S3-compatible storage with object lock, and MCP writes through the same path as the HTTP API. It also implements the “everything must be reconstructable” requirement of D-0029 and rides on the replicated change log of D-0013.

The permission rules that decide whether a call may happen are in permission-model.md. This file assumes that decision has been made and describes what is written once it has.

Contracts freeze at the end of M0; a change to this file after the freeze is a standalone pull request (D-0036).

1. Purpose and principles

An operator of a self-hosted cluster has to be able to answer, months later: which agent restarted which app, on whose authority, because of which event, with what parameters, and did it work (D-0029, D-0031). The audit log is the only place that answer exists. It is not a debugging log and not a metrics stream; it is the accountability record of every action taken against the cluster.

The principles:

1. Append-only. Records are inserted, never updated and never deleted (D-0031). There is no update endpoint, no delete endpoint and no retention job in M0-M6 (section 9).
2. One record per action, in the action’s transaction. A record is written in the same SQLite transaction as the change it describes, so an action without its record cannot exist (D-0031; D-0013).
3. Hash-chained. Each record names the hash of its predecessor, so a modified or reordered record is detectable (D-0031; D-0013).
4. Same path for every channel. API, panel, CLI, MCP and Quoro’s own automation write through one service layer and therefore one log (D-0031).
5. Copyable out of reach. An optional off-site copy with object lock exists so that the person operating the server cannot silently rewrite the record (D-0031; section 8).
6. Records are not events. The audit log and the event stream are different logs with different consumers (section 10).

2. Record format

An audit record has no prefixed id; it is identified by its seq, which is gap-free from 1 within the cluster’s audit log (shared entity model; db-schema.sql, table audit_log).

Absent optional fields are omitted from the JSON, not set to null. This matters because the hash covers the serialized object: null and “absent” would otherwise be two encodings of the same record with different hashes (section 5).

2.1 Example: the M3 demo restart, by an agent, through MCP

An app on a node went down, an incident opened, the webhook woke an agent, and the agent restarted the app through the restart_app MCP tool (D-0033 M3 demo; D-0032).

{
  "seq": 1041,
  "time": "2026-11-02T15:42:07Z",
  "actor_type": "agent",
  "actor_id": "agt_01JC9Z6Q5W8N3K4T7R2M1XHB0D",
  "agent_model": "example-model-id",
  "on_behalf_of": "usr_01JC9Z6Q5W8N3K4T7R2M1XHB0E",
  "channel": "mcp",
  "operation": "restart_app",
  "scope": "app:restart",
  "target_type": "app",
  "target_id": "app_01JC9Z6Q5W8N3K4T7R2M1XHB0F",
  "project_id": "prj_01JC9Z6Q5W8N3K4T7R2M1XHB0G",
  "params": { "app_id": "app_01JC9Z6Q5W8N3K4T7R2M1XHB0F" },
  "justification": "app down per incident inc_01JC9Z6Q5W8N3K4T7R2M1XHB0H; restarting its instance",
  "trigger_event_id": "evt_01JC9Z6Q5W8N3K4T7R2M1XHB0J",
  "incident_id": "inc_01JC9Z6Q5W8N3K4T7R2M1XHB0H",
  "grant_id": "grt_01JC9Z6Q5W8N3K4T7R2M1XHB0K",
  "request_id": "req_01JC9Z6Q5W8N3K4T7R2M1XHB0M",
  "result": "ok",
  "change_seq": 4077,
  "prev_hash": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "hash": "a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90"
}

Reading it: the actor is an agent (agt_...), its model is recorded, it acted on behalf of usr_..., the channel was mcp, the operation was the restart_app tool under app:restart, the incident and the triggering event are both named, the grant that allowed it is named, the agent said why, the result was ok, and change-log entry 4077 is the restart itself (D-0029, D-0031).

The two hash values above are illustrative placeholders, not computed hashes; the prev_hash is drawn from a fixed pattern only so the example is readable.

2.2 Example: a refusal

An agent holding a read-only grant attempted delete_app; the call was refused before anything happened (D-0030, D-0031).

{
  "seq": 1042,
  "time": "2026-11-02T15:43:11Z",
  "actor_type": "agent",
  "actor_id": "agt_01JC9Z6Q5W8N3K4T7R2M1XHB0D",
  "agent_model": "example-model-id",
  "on_behalf_of": "usr_01JC9Z6Q5W8N3K4T7R2M1XHB0E",
  "channel": "mcp",
  "operation": "delete_app",
  "scope": "app:delete",
  "target_type": "app",
  "target_id": "app_01JC9Z6Q5W8N3K4T7R2M1XHB0F",
  "project_id": "prj_01JC9Z6Q5W8N3K4T7R2M1XHB0G",
  "params": { "app_id": "app_01JC9Z6Q5W8N3K4T7R2M1XHB0F" },
  "justification": "asked to remove the app; checking whether that is allowed",
  "grant_id": "grt_01JC9Z6Q5W8N3K4T7R2M1XHB0N",
  "request_id": "req_01JC9Z6Q5W8N3K4T7R2M1XHB0P",
  "result": "refused",
  "result_detail": "approval_required",
  "prev_hash": "a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90",
  "hash": "f0e1d2c3b4a5968778695a4b3c2d1e0ff0e1d2c3b4a5968778695a4b3c2d1e0f"
}

A refusal has no change_seq, because nothing changed. result_detail carries the reason from the permission contract’s refusal table, so the M3 demo can print “grant expired” or “approval required” straight from the audit record (docs/product/roadmap.md, M3 Definition of Done; D-0030).

2.3 Example: an approval decision by a human

The same agent filed an approval request, and a human approved it in the panel. The approval created a grant, so the record has a change_seq (D-0030, D-0029).

{
  "seq": 1043,
  "time": "2026-11-02T15:47:52Z",
  "actor_type": "user",
  "actor_id": "usr_01JC9Z6Q5W8N3K4T7R2M1XHB0E",
  "channel": "panel",
  "operation": "POST /api/v1/approvals/{approval_id}/approve",
  "scope": "approval:decide",
  "target_type": "approval",
  "target_id": "apr_01JC9Z6Q5W8N3K4T7R2M1XHB0Q",
  "project_id": "prj_01JC9Z6Q5W8N3K4T7R2M1XHB0G",
  "params": { "approval_id": "apr_01JC9Z6Q5W8N3K4T7R2M1XHB0Q", "decision": "approved", "decision_note": "confirmed with the app owner" },
  "approval_id": "apr_01JC9Z6Q5W8N3K4T7R2M1XHB0Q",
  "request_id": "req_01JC9Z6Q5W8N3K4T7R2M1XHB0R",
  "result": "ok",
  "change_seq": 4078,
  "prev_hash": "f0e1d2c3b4a5968778695a4b3c2d1e0ff0e1d2c3b4a5968778695a4b3c2d1e0f",
  "hash": "6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c"
}

A human record has no agent_model and no on_behalf_of (the human acted for themselves), and no justification (the decision_note is in params). The grant produced by this decision is a separate change-log entry, and the audit record for the destructive call that follows names both approval_id and grant_id (permission-model.md, section 9).

2.4 Reading the log

• GET /api/v1/audit requires audit:read, takes limit (default 50, max 500), cursor and after_seq, and returns { "items": [...], "next_cursor": ... } (shared entity model, “Pagination and lists”).
• Records are returned in seq order, oldest first (D-0031).
• An agent’s selector filters the rows it may see; the list is filtered, never refused (permission-model.md, section 5.1; D-0030).
• There is no endpoint that updates or deletes a record. GET /api/v1/audit and POST /api/v1/audit/verify are the only audit operations in openapi.yaml (D-0031).

3. What is recorded

Each call or action in the rows marked “yes” below produces exactly one record (D-0031):

Why the split falls where it does:

• Agents are recorded even on reads because an agent’s reading pattern is part of the risk the log exists for: what it looked at before it acted is evidence (D-0029, D-0031). The M3 demo reconstructs a whole incident from these records.
• Human reads are not recorded because the volume would bury the signal; a human’s mutating calls and their approval decisions are recorded (D-0031).
• Authentication failures have no identity, so there is nobody to attribute the record to. They are counted in the server log and in metrics, not in the chain (D-0031).

System records exist so that “the system did it” is not a blank space in the story: an incident grant that appears from nowhere, or a lease revoked because a node lost it, would otherwise be an unexplained state change (D-0029; D-0013). Node enrollment and removal are recorded because they change who is in the cluster (D-0011, D-0031).

3.1 Atomic with the change

An authorized mutating call writes, in one SQLite transaction:

1. the change-log entry (the append-only state change, D-0013);
2. the audit record, with change_seq set to that entry’s seq;
3. any grant bookkeeping, such as uses += 1 (permission-model.md, section 7.3).

If the transaction rolls back, none of the three exists. An audit record with a change_seq therefore always points at a change that happened, and a change-log entry produced by an API or MCP call always has an audit record pointing at it (D-0031).

change_seq is absent for calls that changed nothing: refusals, reads, and errors before any write.

3.2 Replication of audit records

The audit chain is separate from the change-log chain, but audit records travel on the change log: each audit record is also replicated to replica nodes as a change-log entry of entity_type audit (D-0013). A replica node therefore verifies two chains, and the audit chain reaches replicas only through a verified change-log stream (D-0013; node-agent-protocol.md).

The audit_log table itself is the primary’s copy; it is not a local table (see the “local” markings in db-schema.sql).

3.3 Interactive sessions (M4)

A web terminal or VM console session lasts minutes, and one record per call does not describe it (ADR 0115). A session therefore leaves exactly two records, the open and the end, paired by the session id.

The open is the normal record of POST /api/v1/apps/{app_id}/terminal-sessions (scope app:terminal, target_type app) or POST /api/v1/vms/{vm_id}/console-sessions (scope vm:console, target_type vm). Its params are the request body plus the values the primary resolved: session_id (the trm_ id the call minted, so the end record can be paired with it), vm_id, and for a terminal rows, cols, command (the argv that runs, after the default is applied) and user (absent when the app’s own user applies); for a console tail_bytes. The single-use ticket is never in params. A refused or failed open writes its record and no session exists.

The end is a record the primary writes when an opened session closes for any reason, including a ticket that was never used and a primary restart that lost the session (interactive_sessions in db-schema.sql migration 0005 lets it close those at start-up):

Because session.end is chained like every record, a transcript edited after the session ended no longer matches transcript_sha256, and the edit is evident (ADR 0031, ADR 0115).

Transcripts. A terminal session’s transcript is recorded by the node that runs it, never sent to the primary unless a human fetches it, and never part of the audit log, the change log or the off-site copy (ADR 0014, ADR 0115):

• Format: asciicast v2 (newline-delimited JSON): a header line {"version": 2, "width": <cols>, "height": <rows>, "timestamp": <unix seconds>, "env": {"TERM": ...}}, then one event line per chunk, [<seconds since start>, "i" | "o" | "r", <data>]: i input, o output, r resize as "<cols>x<rows>". Bytes that are not valid UTF-8 are written with U+FFFD replacement in <data>; the hash covers the file as written.
• Cap: 16 MiB per session. When the next chunk would pass the cap the node ends the session with close_reason transcript_limit, so no part of a session is unrecorded.
• Console sessions have no transcript. They are read-only and show the VM’s own serial output, which the node already keeps in its console ring (ADR 0116); their session.end carries no transcript fields.
• Retention: 30 days on the node, and at most 1 GiB of transcripts per node, the oldest deleted first. Deleting the app or its VM does not delete them; removing the node or purging its data does. A deleted transcript leaves its session.end record, whose hash then has nothing to verify.
• Access: GET /api/v1/interactive-sessions/{session_id}/transcript, which needs audit:read and a human caller (permission-model.md, section 7.6). The primary relays the file from the node over the node session, on demand; it answers node_unreachable (504) when the node is not connected and not_found (404) when the node no longer has the file. Every fetch is recorded, with target_type vm, the session’s VM as target_id and params {session_id}.

3.4 Password changes (M4)

POST /api/v1/auth/password (ADR 0135, openapi.yaml, changePassword) writes one record per call that reached an identity, with the user as actor and as target (target_type user, target_id the user’s id), no scope and channel panel:

params carries both keys, redacted (section 4): {"current_password": "[redacted]", "new_password": "[redacted]"}. The wrong_password records are the trail of someone guessing a password with a stolen session. A request without a session cookie, or with an Authorization header, has no identity to attribute and is not recorded (section 7).

4. Redaction

Requests can carry secrets. The audit record stores the request parameters, so it must not become a second secret store (D-0031).

The redaction rule:

• The values of the keys value, values, secret, password, current_password, new_password, token and key are replaced by the string "[redacted]", at any depth of the params object.
• Every environment variable value is replaced by "[redacted]", whatever the key is called. Environment variable names are kept, because a name is not a secret and the record of which variable changed is the useful part (D-0031; openapi.yaml, app_env).
• Everything else is stored as sent.

Build arguments (source.build_args[].value in openapi.yaml) are not secrets, since they end up in the image’s history, but they sit under the key value and are therefore redacted like every other value. Build secrets are passed by name only (the names of app environment variables), so their values never reach a request.

Provider credentials are covered by the secret key: the requests that carry them (POST /api/v1/providers and POST /api/v1/providers/{provider_id}/rotate in openapi.yaml) put the credential under secret, so it is redacted like any other secret (D-0051).

This list is part of the contract: adding a key to it is a contract change, because a verifier or a consumer that expects an unredacted field would otherwise silently see a different record.

{ "name": "DATABASE_URL", "value": "[redacted]" }

Redaction happens before serialization and therefore before hashing: the hash covers the redacted record, which is what is stored and what the chain verifies (section 5). This is deliberate — the record must be verifiable by anyone who can read it, including an auditor holding only the exported copy.

A parameter that is a secret but whose key is not on the list is an implementation bug to fix in the redaction list, not a reason to hash the unredacted value (D-0031).

5. The hash chain

Each record’s hash is:

hash = lowercase_hex( SHA-256( JCS( record_object_without_hash ) ) )

where JCS is the JSON Canonicalization Scheme (RFC 8785) and record_object_without_hash contains every field of the record except hash, with absent optional fields omitted rather than null (shared entity model, “Audit record”; the change log uses the same construction, D-0013).

• prev_hash of seq 1 is 64 zeros ("0000000000000000000000000000000000000000000000000000000000000000").
• For seq > 1, prev_hash equals the hash of seq - 1.
• seq is gap-free from 1; there is no hole and no duplicate (D-0031).

Because the serialization is canonical, a record can be re-hashed by any implementation that has the record’s JSON: field order, whitespace and number formatting do not affect the result (RFC 8785, cited by the shared entity model).

5.1 Worked chain (illustrative)

Two records, with hashes shortened for reading. The real values are 64 hex characters.

To verify record 2:

1. Take the JSON object of record 2 without hash.
2. Canonicalize it with RFC 8785.
3. SHA-256 it; the lowercase hex must equal hash.
4. Check that prev_hash equals record 1’s hash.
5. Check that seq is 2, one more than record 1’s seq.

Editing any field of record 2 changes H2; editing any field of record 1 changes H1, which breaks record 2’s prev_hash as well. That second effect is the point of the chain: one edited record makes every later record fail.

The H1/H2 values above are placeholders, not computed hashes.

5.2 What the chain is separate from

• The change-log chain (D-0013) chains state changes; the audit chain chains records about actions. They have independent seq spaces and independent head hashes, and a verifier must not mix them.
• The two are joined per record by change_seq, which points from an audit record to the change it produced, not the other way round (section 3.1).

6. Verification

POST /api/v1/audit/verify requires audit:read and re-walks the chain. The algorithm:

prev = 64 zeros if first_seq == 1, else the hash of record first_seq - 1
expect_seq = first_seq
checked = 0
for record in audit_log ordered by seq, from first_seq to last_seq:
    if record.seq != expect_seq          -> broken: reason "seq_gap", broken_at_seq = record.seq
    h = sha256_hex(jcs(record without hash))
    if h != record.hash                  -> broken: reason "hash_mismatch", broken_at_seq = record.seq
    if record.prev_hash != prev          -> broken: reason "prev_mismatch", broken_at_seq = record.seq
    prev = record.hash
    expect_seq += 1
    checked += 1

The result object:

{
  "ok": true,
  "checked": 1043,
  "first_seq": 1,
  "last_seq": 1043,
  "head_hash": "a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90"
}

and on failure:

{
  "ok": false,
  "checked": 517,
  "first_seq": 1,
  "last_seq": 1043,
  "head_hash": "a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f90",
  "broken_at_seq": 518,
  "reason": "hash_mismatch"
}

On failure the service emits audit.chain_broken, the only audit-related event type (README.md, “Event types”; section 10). The verification run itself is audited: a record with operation POST /api/v1/audit/verify, result ok or error, and the outcome in result_detail when it failed.

This matches the M3 Definition of Done: after one entry is edited in the local copy, verification reports that exact entry as broken (docs/product/roadmap.md, M3 Definition of Done; D-0013, D-0031).

6.1 Offline verification

quoro audit verify --file <jsonl> verifies an exported file with no server, no database and no network:

• the file is JSON Lines, one record per line, in seq order (the same format the off-site copy uses, section 8);
• the first record’s prev_hash is checked against the 64-zero seed unless a starting prev_hash is supplied for a segment that does not begin at seq 1;
• the same three checks apply: recomputed hash, prev_hash link, seq continuity;
• it exits non-zero and names the failing seq and reason when the file is broken.

This is the tool an auditor runs against a downloaded segment without trusting the server that produced it (D-0031).

7. Limits, stated honestly

A hash chain is a tamper-evident structure, not a tamper-proof one. It detects:

• a modified record;
• a reordered record;
• an inserted record in the middle;
• a deleted record in the middle (the following record’s prev_hash no longer matches).

It does not detect:

• Truncation of the tail. Deleting the last N records leaves a chain that verifies perfectly, because nothing after them points back. Only a head hash recorded somewhere else catches this.
• A complete rewrite. Someone with write access to the database and the code can recompute every hash from seq 1 and produce a consistent, valid chain.

The only defence against both is that the head is known outside the server that produced it: the off-site copy (section 8) plus anchoring the head hash somewhere the operator does not control (D-0031). This is precisely why D-0031 asks for a copy out of reach of whoever operates the server, and why the design does not claim that the local chain alone is enough.

Two further honest limits:

• The chain proves internal consistency, not that the recorded parameters and results are true; a lying service writes a consistent lie.
• It proves nothing about calls that never reached the audit gate. Authentication failures are not recorded (section 3), so a burst of failed logins is not visible in the chain.

8. Off-site copy

The off-site copy ships in M5a with the S3-compatible path (D-0053), gated by an object-lock probe per provider. Object-lock support per provider is unverified (Hypothesis) until the research task before M5a and the probe.

8.1 Segments

• Records are exported as canonical JSON Lines: one audit record per line, serialized with RFC 8785, in seq order, redacted as stored (section 4).

• A segment is closed every 1,000 records or 5 minutes, whichever comes first.

• The object key is:

  <prefix>/<cluster_id>/audit/<first_seq 20-digit zero-padded>-<last_seq 20-digit>.jsonl
  

• Alongside each segment, a small head object is written:

  <prefix>/<cluster_id>/audit/head-<last_seq 20-digit>.json
  

  containing { "last_seq": ..., "head_hash": "...", "time": "..." }.

• Each object is written once. Segment keys are unique (the seq range is in the key), so an export never overwrites a previous segment.

• Each object is written with an object-lock retention in compliance mode, so the store itself refuses deletion or overwrite until the retention passes (D-0031).

• The exporter’s credentials are limited to put: no delete, no overwrite of a locked object. A credential that can delete defeats the purpose of the copy (D-0031).

• Progress is recorded in audit_exports (target, last_exported_seq, object_lock, updated_at), so a restarted exporter resumes after the last closed segment (shared entity model; db-schema.sql).

8.2 Provider caveats

• Object lock and conditional writes are not portable across S3-compatible providers (F-0038; docs/research/r2-conditional-writes.md). The provider adapter probes the capability at setup, and refuses to report the copy as “locked” if the probe fails: audit_exports.object_lock stays false and the panel says so, rather than promising a lock the store does not enforce.
• Whether a given provider implements object lock in the form this contract needs is Hypothesis, not covered by docs/research/; F-0038 covers conditional writes only, and object-lock support per provider must be probed the same way.
• R2 limits concurrent writes to the same key to one per second (F-0037). Segment keys are unique and each object is written once, so the limit does not bind this design. The head object has one key per last_seq, also unique, so it does not bind either.
• A failed put is retried; a segment is only recorded as exported in audit_exports after the put is acknowledged. After a crash between put and bookkeeping, the exporter lists the keys after last_exported_seq before cutting a new segment and adopts an existing segment (and its head object) whose content verifies, instead of writing again: a locked object cannot be overwritten, and a re-cut range would overlap.

8.3 What the copy does and does not solve

It solves tail truncation and full rewrite on the server, because the segments and the head hashes live somewhere the operator of the server cannot rewrite (D-0031). It does not solve a compromised provider account, and it does not make the local log immutable. The honest statement is: the local chain plus an out-of-reach copy make silent rewriting detectable by anyone who compares the two heads (section 7).

9. Retention

• Audit records are never deleted in M0-M6. There is no retention job, no vacuum of audit_log and no export-then-prune step (D-0031).
• A retention policy would be a decision of its own, and this contract does not anticipate its shape. It would have to reconcile “append-only” with deletion and say what happens to the chain when a middle segment is removed (D-0031, D-0036).
• Growth is bounded in practice by the fact that only mutating calls, agent reads and system actions are recorded, not human reads (section 3). That is an expectation, not a limit this contract fixes.

10. Relationship to events

• Audit records are not events. They are not in events, they are not delivered by webhooks and they do not appear in the event stream. A consumer that wants to react to agent activity uses the audit API with audit:read (D-0031; events.schema.json).
• audit.chain_broken is the only audit event type. It is emitted when verification fails and is the one way the audit log reaches the event stream (README.md, “Event types”).
• A mutating call may produce both an audit record and a domain event, for example a restart that produces an app.restarted event and an audit record with change_seq. They are different records for different consumers, and neither replaces the other (D-0031, D-0032).

Engine trait

Status: M0 draft for review. Contracts freeze at the end of M0 (D-0036); after the freeze a change to this file is a standalone contract pull request, never part of feature work.

This file fixes the boundary behind which a virtual machine monitor sits. Cloud Hypervisor with KVM is the first implementation (D-0007), and the boundary exists so that other backends — Firecracker, a container runtime — can be added later without touching the rest of the system (D-0007, D-0009). The MVP requires a host with KVM; a container fallback is a later implementation of the same trait, not a mode of the first one (D-0009).

The node agent is the only caller. Nothing else in Quoro — not the API, the panel, MCP or the CLI — talks to the engine directly; they change desired state, and the node agent turns that into engine calls (node-agent-protocol.md). The engine therefore has no authentication, no tenancy and no policy of its own: it does what the node agent asks on the node it runs on, and the node agent is the component that checks leases, tokens and desired state first.

This file is a contract about behaviour, not about file layout. Names of modules, crates and private helpers may change; the types, method names, semantics and error variants below may not, except by a standalone contract pull request.

1. Shape of the boundary

The async methods are declared as fn ... -> impl Future<Output = ...> + Send, so a caller generic over E: Engine can tokio::spawn an engine call (D-0071); backends still implement them with async fn. The trait is not object-safe, which is why the node agent selects a backend through an enum (enum Backend { CloudHypervisor(ChEngine), Fake(FakeEngine) }) rather than Box<dyn Engine>; a need for dyn would be met by boxed futures with otherwise identical signatures.

2. The trait

#![allow(unused)]
fn main() {
// Rust 2024 edition (D-0071).
use std::fmt;
use std::future::Future;
use std::net::Ipv4Addr;
use std::path::PathBuf;
use std::pin::Pin;
use std::time::{Duration, SystemTime};

use futures_core::Stream;
use tokio::io::{AsyncRead, AsyncWrite};

/// vm_<ULID>, the same identity the control plane uses (README.md).
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub struct VmId(pub String);

/// vol_<ULID>; volumes exist as a shape from M5 (README.md).
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub struct VolumeId(pub String);

/// "sha256:<64 hex>".
#[derive(Clone, Debug, PartialEq, Eq, Hash)]
pub struct ImageDigest(pub String);

/// A MAC address, six bytes in the order the guest sees them.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
pub struct MacAddr(pub [u8; 6]);
}

#![allow(unused)]
fn main() {
/// An image as requested, and the digest it resolved to.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ImageRef {
    pub reference: String,
    pub digest: ImageDigest,
}

/// An OCI image layout on the node, as a builder wrote it. Added in M4.
/// Section 3.17.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct OciLayout {
    /// Directory or tarball of the layout (OCI image-layout 1.0).
    pub path: PathBuf,
    /// Digest of the image manifest the layout's index names.
    pub manifest_digest: ImageDigest,
}

/// A rootfs that is ready to boot. Cached by digest.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct PreparedImage {
    pub digest: ImageDigest,
    pub rootfs_path: PathBuf,
    pub size_bytes: u64,
    pub config: ImageConfig,
}

/// What the image says about how to run it, read from its OCI config.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ImageConfig {
    pub entrypoint: Vec<String>,
    pub cmd: Vec<String>,
    pub env: Vec<(String, String)>,
    pub user: Option<String>,
    pub workdir: Option<String>,
    pub exposed_ports: Vec<u16>,
}
}

#![allow(unused)]
fn main() {
/// Everything needed to create one microVM.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct VmSpec {
    pub id: VmId,
    pub vcpus: u8,
    pub memory_mb: u32,
    /// Digest of a PreparedImage; the rootfs is mounted read-only.
    pub rootfs: ImageDigest,
    /// Size of the writable data disk.
    pub disk_gb: u32,
    pub volumes: Vec<VolumeAttach>,
    pub net: NetAttach,
    pub init: InitConfig,
    /// Where the app's tagged output goes (section 3.13). From M4 the node
    /// agent sets `VirtioConsole` for apps and builder VMs alike.
    pub console: ConsoleMode,
    /// The serial port (`ttyS0`): kernel and init diagnostics. From M4 the node
    /// agent sets `Socket`, and the engine is its only client (section 3.19,
    /// ADR 0116). Added in M4.
    pub serial: SerialMode,
    /// The virtio-vsock device that carries exec sessions (section 3.18,
    /// ADR 0113). `None` means no device. Added in M4.
    pub vsock: Option<VsockAttach>,
    /// Extra writable disks that outlive the VM, identified by serial. Only
    /// builder VMs use them (ADR 0118); app VMs carry none. Added in M4.
    pub work_disks: Vec<WorkDisk>,
}

impl VmSpec {
    /// The canonical form used to decide whether two specs are the same.
    ///
    /// Section 3.6 compares the same `id` by the canonical form of `VmSpec`,
    /// so `env` is sorted by `(name, value)` and `volumes` by
    /// `(volume_id, mount_path)`. Everything else keeps its order: it is
    /// either a scalar or an ordered list where order is meaningful.
    #[must_use]
    pub fn canonical(&self) -> VmSpec {
        let mut canonical = self.clone();
        canonical
            .init
            .env
            .sort_by(|a, b| a.0.cmp(&b.0).then_with(|| a.1.cmp(&b.1)));
        canonical.volumes.sort_by(|a, b| {
            a.volume_id
                .0
                .cmp(&b.volume_id.0)
                .then_with(|| a.mount_path.cmp(&b.mount_path))
        });
        canonical
    }
}

/// A volume attached to a VM at a mount path inside the guest.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct VolumeAttach {
    pub volume_id: VolumeId,
    pub mount_path: String,
}

/// The VM's network attachment on the node bridge.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct NetAttach {
    pub ip: Ipv4Addr,
    pub gateway: Ipv4Addr,
    pub mac: MacAddr,
    /// Isolation group of the VM. VMs on the same bridge that carry the same
    /// group reach each other; VMs in different groups, and a VM with `None`,
    /// reach no other VM. The node agent sets it to the VM's project id
    /// (`prj_`). Section 3.3.
    pub isolation_group: Option<String>,
    /// Egress only: the VM reaches the outside through the gateway and nothing
    /// on the node, neither another VM nor the host (section 3.3). Builder VMs
    /// set it, with `isolation_group: None` (ADR 0118). Added in M4.
    pub egress_only: bool,
}

/// The app's command line and identity, as the minimal init must apply it.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct InitConfig {
    pub entrypoint: Vec<String>,
    pub cmd: Vec<String>,
    pub env: Vec<(String, String)>,
    pub user: Option<String>,
    pub workdir: Option<String>,
}

/// How (and whether) a console device is attached.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum ConsoleMode {
    /// No console device.
    Off,
    /// virtio-console to this UNIX socket; carries tagged stdout/stderr lines.
    VirtioConsole { socket: PathBuf },
    /// Serial console to this UNIX socket; interactive text, one client at a time.
    /// Superseded in M4 by `VmSpec::serial` (section 3.14); a spec that sets it
    /// together with a `serial` other than `Off` is `Conflict`.
    Serial { socket: PathBuf },
}

/// How the serial port (`ttyS0`) is attached. Added in M4. Section 3.19.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum SerialMode {
    /// No serial device.
    Off,
    /// Serial output appended to this file (the M1-M3 behaviour).
    File { path: PathBuf },
    /// Serial port on this UNIX socket; the engine connects to it as the only
    /// client from `create` on and keeps a node-local ring of its output.
    Socket { path: PathBuf },
}

/// A virtio-vsock device in Cloud Hypervisor's hybrid mode: the host end is a
/// UNIX socket in the VM's runtime directory. Added in M4. Section 3.18.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct VsockAttach {
    /// Guest context id. Each VM has its own vsock namespace, so every VM uses 3.
    pub cid: u32,
    pub socket: PathBuf,
}

/// A writable disk that the engine attaches but does not own: `delete` keeps
/// it, and the node agent removes it. Added in M4. Section 3.6.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct WorkDisk {
    pub path: PathBuf,
    /// Size of the disk when the engine has to create it.
    pub size_gb: u32,
    /// virtio-blk serial the guest finds it by, for example `quoro-work`.
    pub serial: String,
}
}

#![allow(unused)]
fn main() {
/// How a stop is requested.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum StopMode {
    /// Ask the guest to stop and wait at most this long.
    Graceful { timeout: Duration },
    /// Stop now, without cooperation from the guest.
    Force,
}

/// Mirrors the VM observed_status enum in README.md one to one.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum VmState {
    Creating,
    Stopped,
    Starting,
    Running,
    Stopping,
    Crashed,
    Deleted,
    Unknown,
}

/// One VM's observed state.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct VmInfo {
    pub id: VmId,
    pub state: VmState,
    /// Present while a VMM process exists for this VM.
    pub pid: Option<u32>,
    pub started_at: Option<SystemTime>,
    pub exit_code: Option<i32>,
}

/// A lifecycle event for any VM on the node. Section 3.12.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum EngineEvent {
    Started { id: VmId },
    Exited { id: VmId, code: i32 },
    Crashed { id: VmId, reason: String },
}

/// One tagged line of app output.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct OutputLine {
    pub stream: OutputStream,
    pub time: SystemTime,
    pub line: String,
}

/// Which of the app's two output streams a line came from.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum OutputStream {
    Stdout,
    Stderr,
}

/// Cumulative counters since VM start. Section 3.15.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct VmCounters {
    pub cpu_time_ms: u64,
    pub memory_rss_bytes: u64,
    pub net_rx_bytes: u64,
    pub net_tx_bytes: u64,
    pub block_read_bytes: u64,
    pub block_write_bytes: u64,
}

/// What this backend and build support. Section 3.1.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct Capabilities {
    pub engine: String,  // "cloud-hypervisor"
    pub version: String, // the pinned release
    pub kvm: bool,
    pub text_console: bool,      // true
    pub graphical_console: bool, // false
    pub vm_snapshot: bool,
    pub volume_snapshot: bool,
    /// Whether `exec` works on this backend and build (section 3.18). Added in M4.
    pub exec: bool,
}

/// The node network a VM is attached to. Section 3.3.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct NodeNetwork {
    pub bridge: String,
    pub subnet: String, // CIDR, for example "10.42.1.0/24"
    pub gateway: Ipv4Addr,
    /// DNS servers handed to the guests created on this network, in order.
    /// Taken from the node's configuration; empty means guests get no
    /// resolver. Not part of the network's identity. Section 3.3.
    pub dns: Vec<Ipv4Addr>,
}

/// A point-in-time volume snapshot descriptor. Section 3.16.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct VolumeSnapshot {
    pub volume_id: VolumeId,
    pub snapshot_id: String,
    pub created_at: SystemTime,
    pub size_bytes: u64,
    /// Node-local location of the snapshot; not part of any API.
    pub location: PathBuf,
}

/// Every method is total: it either succeeds or returns one of these.
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum EngineError {
    NotFound,
    Conflict(String),
    Unsupported(String),
    ResourceExhausted(String),
    Image(String),
    Hypervisor(String),
    Io(String),
    Timeout,
}

impl fmt::Display for EngineError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        // one short lowercase phrase per variant
        ...
    }
}

impl std::error::Error for EngineError {}
}

#![allow(unused)]
fn main() {
/// A stream of every lifecycle event on the node.
pub type EventStream = Pin<Box<dyn Stream<Item = EngineEvent> + Send>>;

/// A stream of one VM's tagged output lines.
pub type LineStream = Pin<Box<dyn Stream<Item = OutputLine> + Send>>;

/// Bidirectional byte stream on a text console (serial or virtio-console).
pub trait ConsoleIo: AsyncRead + AsyncWrite + Send + Unpin {}

/// Any async byte stream is a console stream; tokio's duplex and UNIX socket
/// streams satisfy this without an adapter.
impl<T: AsyncRead + AsyncWrite + Send + Unpin + ?Sized> ConsoleIo for T {}

/// One end of a text console connection.
pub type ConsoleStream = Pin<Box<dyn ConsoleIo>>;

/// Raw bytes in arrival order, for example a VM's serial output. Added in M4.
pub type ByteStream = Pin<Box<dyn Stream<Item = Vec<u8>> + Send>>;

/// A program to run in a running VM, on a PTY. Added in M4. Section 3.18.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ExecRequest {
    /// The argv; `argv[0]` without `/` is searched in the app's `PATH`.
    pub argv: Vec<String>,
    /// A user name or uid, optionally `:` and a group name or gid; `None` is the app's own user.
    pub user: Option<String>,
    pub rows: u16,
    pub cols: u16,
}

/// One exec session. Added in M4. Section 3.18.
pub struct ExecSession {
    /// The PTY: reads are the program's output, writes are its input. Dropping
    /// or shutting it down ends the session (`SIGHUP` to its process group).
    pub io: ConsoleStream,
    /// Resize requests as `(rows, cols)`.
    pub resize: tokio::sync::mpsc::Sender<(u16, u16)>,
    /// Resolves once with the program's exit code, `128 + signal` when a
    /// signal ended it; an error when the session ended without one.
    pub exit: tokio::sync::oneshot::Receiver<i32>,
}

/// The boundary behind which a virtual machine monitor sits.
///
/// The async methods return `impl Future<Output = _> + Send`, so a caller
/// generic over `E: Engine` can spawn an engine call on the tokio runtime
/// (D-0071). A backend still implements them with `async fn`. The trait is not
/// object-safe, which is why the node agent selects a backend through an enum;
/// a `dyn` need would be met by boxed futures with otherwise identical
/// signatures.
pub trait Engine: Send + Sync {
    // Host and node preparation
    fn capabilities(&self) -> impl Future<Output = Capabilities> + Send;
    fn probe_host(&self) -> impl Future<Output = Result<HostProbe, EngineError>> + Send;
    fn ensure_network(
        &self,
        net: &NodeNetwork,
    ) -> impl Future<Output = Result<(), EngineError>> + Send;

    // Images
    fn prepare_image(
        &self,
        image: &ImageRef,
    ) -> impl Future<Output = Result<PreparedImage, EngineError>> + Send;
    fn remove_image(
        &self,
        digest: &ImageDigest,
    ) -> impl Future<Output = Result<(), EngineError>> + Send;
    /// Added in M4; section 3.17.
    fn prepare_layout(
        &self,
        _layout: &OciLayout,
    ) -> impl Future<Output = Result<PreparedImage, EngineError>> + Send {
        async {
            Err(EngineError::Unsupported(
                "local OCI layouts are not implemented by this engine".into(),
            ))
        }
    }

    // Lifecycle
    fn create(&self, spec: &VmSpec) -> impl Future<Output = Result<VmInfo, EngineError>> + Send;
    fn start(&self, id: &VmId) -> impl Future<Output = Result<VmInfo, EngineError>> + Send;
    fn stop(
        &self,
        id: &VmId,
        mode: StopMode,
    ) -> impl Future<Output = Result<VmInfo, EngineError>> + Send;
    fn delete(&self, id: &VmId) -> impl Future<Output = Result<(), EngineError>> + Send;
    fn inspect(&self, id: &VmId) -> impl Future<Output = Result<VmInfo, EngineError>> + Send;
    fn list(&self) -> impl Future<Output = Result<Vec<VmInfo>, EngineError>> + Send;

    // Observation
    fn events(&self) -> EventStream;
    fn output(&self, id: &VmId) -> Result<LineStream, EngineError>;
    fn console(&self, id: &VmId) -> Result<ConsoleStream, EngineError>;
    fn counters(&self, id: &VmId) -> impl Future<Output = Result<VmCounters, EngineError>> + Send;

    // Interactive access (M4; default implementations return Unsupported)
    /// Section 3.18.
    fn exec(
        &self,
        _id: &VmId,
        _request: &ExecRequest,
    ) -> impl Future<Output = Result<ExecSession, EngineError>> + Send {
        async { Err(EngineError::Unsupported("exec is not implemented by this engine".into())) }
    }
    /// Section 3.19.
    fn serial_output(
        &self,
        _id: &VmId,
        _tail_bytes: Option<u64>,
    ) -> Result<ByteStream, EngineError> {
        Err(EngineError::Unsupported(
            "serial output is not implemented by this engine".into(),
        ))
    }

    // Volumes (M5; default implementations return Unsupported)
    fn snapshot_volume(
        &self,
        _id: &VolumeId,
    ) -> impl Future<Output = Result<VolumeSnapshot, EngineError>> + Send {
        async {
            Err(EngineError::Unsupported(
                "volume snapshots are not implemented by this engine".into(),
            ))
        }
    }
    fn delete_volume_snapshot(
        &self,
        _snap: &VolumeSnapshot,
    ) -> impl Future<Output = Result<(), EngineError>> + Send {
        async {
            Err(EngineError::Unsupported(
                "volume snapshots are not implemented by this engine".into(),
            ))
        }
    }
}

/// The result of the host check in section 3.2.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct HostProbe {
    pub kvm_available: bool,
    pub kernel_release: String,
    /// True when the kernel is recent enough for the backend (6.8 or later for
    /// Cloud Hypervisor v54, section 3.2).
    pub kernel_supported: bool,
    pub cpu_vmx: bool,
    pub cpu_svm: bool,
    pub nested: bool,
}
}

The derives, MacAddr, the blanket ConsoleIo implementation, Display and Error for EngineError and VmSpec::canonical (section 3.6) are contract items (D-0071), added additively during M0. events, output, console and serial_output are not async because they only build a stream; the first item is produced by the stream itself.

M4 adds, additively: VmSpec::serial, VmSpec::vsock and VmSpec::work_disks with SerialMode, VsockAttach and WorkDisk; NetAttach::egress_only; Capabilities::exec; OciLayout, ByteStream, ExecRequest and ExecSession; and the methods prepare_layout, exec and serial_output, whose default implementations return Unsupported, so a backend that does not offer them still compiles. Code that builds a VmSpec, NetAttach or Capabilities literal sets the new fields to SerialMode::File (the M1-M3 serial log), None, an empty list, false and false, which keeps the M1-M3 behaviour. VmSpec::canonical keeps work_disks in order: their serials are unique, so the order carries no meaning, but the node agent always builds them in the same order.

3. Method semantics

Every method is total in the sense that it either succeeds or returns one EngineError; no method panics on bad input, and no method returns a partially created VM on success. The rules below are this contract’s definitions; each one that follows from an accepted decision, a research finding or an open question cites it inline.

3.1 capabilities

Reports what this backend and build support. It never fails: a missing binary or an unusable host is reported as kvm: false, not as an error, because the node agent must be able to report a node’s real state without a working engine. engine is the backend name, version the pinned build, and graphical_console is false for Cloud Hypervisor: it has no VNC, SPICE, GPU or framebuffer device, and its consoles are text only (F-0034). text_console is true. exec reports whether exec (section 3.18) works: the backend attaches vsock devices and the guest init it ships listens for exec sessions. vm_snapshot reports whether the backend can snapshot VM memory and device state; that capability is not used by Quoro before M5 and never as a volume backup (F-0035).

3.2 probe_host

Checks the machine rather than a VM:

• /dev/kvm exists and is writable;
• the kernel release is at least 6.8, which Cloud Hypervisor v54 requires (F-0036);
• the CPU exposes vmx or svm;
• nesting is visible when the machine is itself a guest.

The procedure follows docs/research/vps-nested-virt.md. A successful probe is necessary but not sufficient: /dev/kvm can exist while a guest still fails to start, so the node agent treats the first real boot as the confirmation and reports a failed boot as a deployment failure in the boot phase.

3.3 ensure_network

Brings the node network to the requested NodeNetwork: the bridge exists, carries the gateway address, and forwards app traffic. Quoro owns the networking layer — bridges, TAP devices, addresses, firewall rules — rather than delegating it (D-0007). Precondition: subnet and gateway are consistent. Postcondition: a VM with an address inside subnet can reach the gateway. Idempotent: calling it twice with the same values changes nothing. An existing bridge with a different subnet is Conflict, never a silent reconfiguration.

The identity of a network is bridge, subnet and gateway. dns is not part of it: a call that repeats the identity with a different dns is Ok and changes the servers given to VMs created afterwards; a VM that already exists keeps the servers it was created with until it is recreated. The node agent fills dns from the node’s own configuration, so guest resolvers are a property of the node, not of the control plane.

Isolation groups. Every VM on the bridge is isolated from every other VM on it, except from the VMs that carry the same NetAttach::isolation_group (D-0084). Two VMs with the same group exchange traffic over the bridge in both directions; two VMs with different groups, or a VM whose group is None, exchange nothing — neither frames on the bridge nor packets routed through the gateway. Every VM still reaches the gateway and, through it, the outside, and the host still opens connections to every VM, because the ingress runs on the host. The node agent sets the group to the VM’s project_id, so the VMs of one project reach each other and VMs of different projects do not. The rule is enforced by the engine and holds across VM creates, deletes and an agent restart; the group is part of the VM’s spec, so it is compared by create (section 3.6) like every other field.

Egress only. A VM whose NetAttach::egress_only is true reaches the outside through the gateway and nothing else on the node: no other VM, whatever its group; no address of the host (the node API, the ingress listeners, the node’s own services), except the DNS servers in NodeNetwork::dns on port 53; and no link-local address (169.254.0.0/16). The host may still open connections to it. Builder VMs run untrusted Dockerfiles with outbound network to fetch bases and dependencies, and this is their boundary (ADR 0118). egress_only with an isolation_group other than None is Conflict.

3.4 prepare_image

Pulls the OCI image, verifies that the pulled content matches ImageRef::digest, converts it into a rootfs image, records the extracted ImageConfig, and caches the result by digest.

• Precondition: reference and digest are well formed.
• Postcondition: PreparedImage::rootfs_path exists and boots; a second call with the same digest returns the cached value without network access.
• Idempotent by digest. A digest mismatch between the reference and the pulled content is Image.
• A local OCI layout written by a builder takes prepare_layout instead (section 3.17); both fill the same cache.
• The conversion approach is decided (D-0061): Quoro’s own OCI-to-ext4 pipeline, with a Rust registry client and mkfs.ext4 -d, validated by the M0 spike in spikes/oci-microvm. Three gaps must be closed before the pipeline ships: resolve layer paths inside the rootfs with openat2 RESOLVE_IN_ROOT instead of refusing absolute-symlink parents, support zstd layers, and deliver the configuration through a config disk or the kernel command line instead of baking it into the rootfs.

3.5 remove_image

Removes a cached prepared rootfs by digest. Deleting a VM must not remove an image another VM uses, so removal is a reference-counted operation or is driven by the node agent after it has computed the desired set. Missing digest is Ok. Idempotent. Rootfs images are per-digest artifacts of the deployment pipeline (D-0008, D-0033), so disposing of them is the engine’s cleanup, not part of any API delete.

3.6 create

Creates the VM and its runtime state without starting it. One microVM per app instance, booted from a rootfs built from the OCI image (D-0008), with the lifecycle split that M1 requires (D-0033). Postcondition on success: a VMM process exists (or the backend’s equivalent), the VM is inspectable, and state is Stopped once creation finishes — Creating is only observable while the call is in flight.

• The same id with an equal spec returns Ok and the existing VmInfo.
• The same id with a different spec is Conflict. Equality is compared on the canonical form of VmSpec, VmSpec::canonical(), which sorts env by (name, value) and volumes by (volume_id, mount_path); everything else keeps its order, so field order and the ordering of those two lists must be normalized before comparison.
• Resources are reserved at create, so ResourceExhausted is returned here rather than at start.
• Precondition: rootfs is the digest of a prepared image, net is inside a network already ensured by ensure_network, and volumes are attached to this node.
• serial: Socket makes the engine connect to the serial socket as soon as it exists, before boot, so the ring of section 3.19 holds the whole boot. vsock attaches the device; its socket lives in the VM’s runtime directory, which only root can enter.
• Each WorkDisk is attached as a writable virtio-blk disk with its serial. A missing file is created as an empty ext4 filesystem of size_gb; an existing file is attached as it is. Two VMs that name the same work-disk path at once is Conflict. The engine never removes a work disk (section 3.9).

3.7 start

Boots a created or stopped VM. Postcondition on success: state is Running and EngineEvent::Started has been or will be emitted. A boot is fast enough that a stopped warm copy is a practical failover target (D-0008, D-0020); the boot-time figure is still a marketing claim, not a measured number (docs/research/cloud-hypervisor.md).

• Starting a running VM returns Ok and does not reboot it. Restarts are expressed as a spec change by the node agent (restart_generation in node-agent-protocol.md), so the engine never has to guess intent.
• Starting an unknown id is NotFound. Starting a VM whose kernel or rootfs is missing is Image or Io.

3.8 stop

Stops a VM.

• Graceful { timeout } asks the guest to exit — the power-button path, which the minimal init turns into SIGTERM to the app (section 7) — and waits at most timeout. On expiry it falls back to forced stop. The call returns Ok only when the VM is stopped.
• Force stops without cooperation from the guest. If the VMM does not confirm the stop (its API does not answer, or the process does not exit within the call), the backend escalates to SIGKILL on the VMM process and reaps it. Force returns Ok only when the VMM process is gone; an error from Force means the VM may still be running, and the node agent must treat it that way (node-agent-protocol.md section 8.5, D-0074).
• Stopping a stopped VM returns Ok. Stopping an unknown id is NotFound.
• Postcondition: state is Stopped, exit_code reports the app’s code when the guest reported one, and EngineEvent::Exited has been or will be emitted.

3.9 delete

Removes the VM and its own disks. Missing id returns Ok, so a retried delete converges. Volumes are never removed by delete: they belong to the app, not to the instance, and are deleted through the API with app:delete (D-0033, README.md). Postcondition: the VMM process is gone, the root and data disks are removed, and inspect returns NotFound. Work disks are kept too: the node agent owns them and removes them with the app (ADR 0118). The serial ring of section 3.19 is removed with the runtime directory. Precondition: the VM is stopped; a call on a running VM is Conflict.

3.10 inspect

Returns one VM’s VmInfo. Unknown id is NotFound. This is the only method that observes lifecycle state; it never changes it. It is the engine-level source of the observed_status a node reports (D-0032 detect step), and it never reports a state outside the enum in README.md.

3.11 list

Returns every VM the engine knows about on this node, including VMs created by a previous agent process. This is how the node agent rediscovers its world after a restart: VMMs are separate processes and keep running, so list must find them from the runtime directories on disk and the backend’s API, not from in-memory state (D-0011). A VM whose runtime directory exists but whose process is gone is reported as Crashed with the last known exit code, or as Stopped when the exit followed a requested stop. list never starts, restarts or deletes anything.

3.12 events

A stream of EngineEvent for every VM on the node. The node agent maps them to the event types in README.md — a Crashed becomes vm.crashed, a requested stop becomes vm.stopped, a successful boot becomes vm.started (D-0032). The stream exists for the lifetime of the engine instance; a slow consumer must not block the engine, so the implementation buffers and drops with a recorded counter rather than applying backpressure to the VMM.

3.13 output

A stream of OutputLine for one VM: the app’s stdout and stderr, tagged by stream, produced by the minimal init and forwarded over the virtio-console. This feeds local logs, which stay on the node and are queried on demand (D-0014). A second concurrent call for the same VM is Conflict when the backend can serve only one reader; NotFound when the VM has no console configured (ConsoleMode::Off).

3.14 console

A bidirectional byte stream on the text console: a human types into a terminal in the panel, and the bytes reach the guest’s serial console (F-0034; consoles are text-only, xterm.js over serial or virtio-console, D-0056). Used from M4 (D-0033). Text only; there is no graphical console to attach to (F-0034, D-0056). The Cloud Hypervisor console socket accepts one client at a time, so a second concurrent call is Conflict (docs/research/cloud-hypervisor.md).

Superseded in M4. A second connection to a Cloud Hypervisor serial socket silently takes the port away from the current client, and nothing in the guest reads the serial port, so a direct, bidirectional connection per viewer is unsafe and useless (ADR 0116). The M4 console is read-only and comes from serial_output (section 3.19); the web terminal comes from exec (section 3.18). console stays for ConsoleMode::Serial specs and for FakeEngine; on a VM whose serial is Socket it is Conflict. Nothing in the node agent calls it from M4.

3.15 counters

Returns the VM’s counters for metrics (M4, D-0033). Unknown id is NotFound. Values are cumulative since VM start; the node agent computes rates. Counters are best-effort: a backend that cannot report one of the groups returns zeros for it rather than failing the call.

The M4 metrics use two of them (ADR 0014, node-agent-protocol.md sections 5.15 and 5.17): cpu_time_ms and memory_rss_bytes, which the Cloud Hypervisor backend reads from the VMM process on the host (/proc/<pid>/stat and /proc/<pid>/status), not from the backend’s counter document, so they need no guest agent. The node agent samples them every 5 seconds.

3.16 snapshot_volume and delete_volume_snapshot

The point-in-time disk primitive the chunk engine needs (D-0023). Both are Unsupported in the default implementation, and only the M5 code path calls them.

• The primitive for M5a is decided (D-0062): reflink snapshots with a full content-defined chunk scan for cold backups. Before M5b, a spike comparing lvm-thin and btrfs chooses the primitive for warm volumes.
• They must not be implemented with Cloud Hypervisor VM snapshots. A CH snapshot is a paused VM’s config.json, memory-ranges and state.json; it contains guest RAM and device state but no disk contents, and CH exposes no dirty bitmap (F-0035). A volume snapshot is about the disk.
• If implemented with a filesystem primitive, the call takes no VMM lock: it snapshots the volume of a stopped or running VM as the chosen primitive allows, and the chunk engine handles crash consistency (D-0023).

3.17 prepare_layout

Turns a local OCI image layout, as a builder VM wrote it to its output disk, into a prepared rootfs, the way prepare_image does for a pulled image (ADR 0117, ADR 0118). Added in M4.

• Precondition: path is a readable OCI image layout whose index names manifest_digest: a directory, or a tarball of one, which may be followed by zero padding up to the end of the file (a raw output disk, node-agent-protocol.md section 6.6).
• A tarball is read in user space as untrusted input: only regular files named oci-layout, index.json and blobs/sha256/<64 hex> are taken, every other entry (a link, a device, a path with .., a duplicate name) makes the layout Image, and the bytes read are bounded by the node’s image limit.
• The engine verifies every blob against its digest, checks that the manifest’s digest is manifest_digest, applies the layers with the same pipeline as prepare_image (D-0061), extracts ImageConfig, and caches the result by manifest_digest in the same cache. create then boots it with rootfs set to that digest, and remove_image removes it.
• Idempotent by digest: a second call with a digest already cached returns the cached value without reading the layout.
• A missing or malformed layout, a blob that does not match its digest, a media type the pipeline does not support, or a layout larger than the node’s image limit is Image. Nothing is pulled: a layout that references a blob it does not contain is Image, not a registry fetch.
• The engine never deletes the layout; the node agent does, after the call returns.

3.18 exec

Starts a program on a PTY inside a running VM, in the app’s root filesystem, with the app’s environment and working directory, and returns its byte stream (ADR 0113). Added in M4; the web terminal is its only caller.

• The engine connects to the VM’s vsock socket with Cloud Hypervisor’s hybrid handshake (CONNECT <port>\n, answered OK <port>\n) on the init’s exec port (section 6), and sends the open request. The guest init runs the program and relays bytes; the engine relays them to ExecSession::io.
• argv is non-empty, holds no NUL, and is at most 64 items of at most 4096 bytes. rows and cols are 1 to 1000. Otherwise Conflict.
• An unknown id is NotFound. A VM that is not Running, has no vsock, or whose init does not listen (a VM started before its node got the M4 init) is Conflict. A missing program is not an error of the call: the session starts and exit resolves with 127.
• Many sessions per VM may be open at once. Each is independent: closing one does not touch another, and the app keeps running.
• Back-pressure: when the reader of io stops reading, the engine stops reading from the vsock connection, which stops the program at its next write. A session never buffers without bound.
• The guest is untrusted. The engine bounds every frame it reads from the guest (section 6) and treats output as bytes, never parsing it.
• The vsock socket accepts connections from the host only; the node never creates a <socket>_<port> listener, so the guest cannot open a connection to the host over vsock.

3.19 serial_output

A read-only stream of a VM’s serial output: first up to tail_bytes of the newest bytes the node kept (all of it when None), then live output until the VM is deleted or the caller drops the stream (ADR 0116). Added in M4.

• With serial: Socket, the engine is the socket’s only client, from create on. It appends what it reads to a node-local ring in the VM’s runtime directory: two segments of 512 KiB, the older one dropped when the newer one fills, so the ring holds at most 1 MiB and survives VMM restarts, which keeps the previous boot’s output after a crash loop. While the node agent restarts, the VMM keeps up to 1 MiB of output and replays it when the engine reconnects.
• Any number of streams may be open on one VM; each one reads the ring and then follows, so viewers never connect to the socket.
• An unknown id is NotFound. A VM whose serial is not Socket is Conflict.
• Input is never written to the serial port: nothing in the guest reads it.

4. Concurrency

• Operations on one VmId are serialized by the caller. The node agent holds one lock per VM, so it never issues start and delete for the same id concurrently. The engine does not promise ordering for concurrent calls on one id and may return Conflict.
• The engine must be safe to call concurrently for different VMs, on many VMs at once. This is what lets one node run several apps and replace them independently.
• capabilities, probe_host, ensure_network, prepare_image, remove_image and list are safe concurrently with any VM operation. ensure_network and prepare_image are internally serialized by the implementation where the underlying operation is not reentrant.
• No call blocks the runtime for longer than an I/O wait: image preparation and boot are awaited, never run on the reactor thread.

5. Cloud Hypervisor implementation

One cloud-hypervisor process per VM with its own --api-socket, because the VM’s lifetime must not be the agent’s lifetime (D-0011) and because a per-VM socket is the unit the engine can rebuild after a restart (docs/research/cloud-hypervisor.md). The API is HTTP/JSON over a UNIX socket, with reads as GET and mutations as PUT; the engine does not need a Cloud Hypervisor SDK, only an HTTP client over a UnixStream (docs/research/cloud-hypervisor.md).