M4 builds spike: BuildKit in a builder microVM, Railpack and Nixpacks, and the Builder contract

• Question: can a BuildKit daemon run inside an ephemeral microVM under Cloud Hypervisor with Quoro’s kernel and init, and produce an OCI layout the node feeds into the OCI-to-rootfs path (ADR 0060, ADR 0061)? Are Railpack and Nixpacks still viable zero-config builders? What shape should the Builder interface and the M4 contract take?
• Researched: 2026-11-03, on lab-3 (nested virtualization, 2 vCPU / 4 GB), with the pinned Cloud Hypervisor v54.0, the pinned guest kernel (ch-release-v7.2.8-20261009), the initramfs the node agent built from crates/init, and an unmodified M3 node agent running next to the spike.
• Status: answered. Prototype code stayed outside the repository and was deleted from the lab afterwards.

Short answer

1. BuildKit in a builder microVM: works, with four additions. buildkitd v0.34.2 (rootful, runc executor, overlayfs snapshotter) ran as the app under our init and built every Dockerfile tried: a tiny Alpine image, a Node app, a multi-stage Go app cloned from Git by BuildKit itself, and a deliberately failing build. buildctl --output type=oci wrote an OCI layout tarball whose layers are all tar+gzip and whose config carries one diff_id per layer. A scratch importer that reuses the engine’s own layer::apply_layer, ext4::build and cache::write_entry turned four of those layouts into root disks, and each booted under our init and answered HTTP 200. No kernel change was needed. The four additions: (a) a local-layout source in ImageStore (today it only pulls from a registry); (b) a fourth disk for the BuildKit state, the build context and the output, because the guest root is an overlay and the engine rejects extra disks until M5a; (c) mounting cgroup2 on /sys/fs/cgroup, which our init does not do; (d) a way for logs and the result to leave the VM, which the existing tagged serial output already covers.
2. Railpack: viable second Builder; Nixpacks: no. Nixpacks’ README states it is in maintenance mode and points to Railpack. Its last release, v1.41.0, came out on 2025-10-24, and it builds by shelling out to docker build. Railpack (MIT, v0.43.0 on 2026-10-19, releases every few days) ships a BuildKit gateway frontend: a JSON plan replaces the Dockerfile, the frontend turns it into LLB, and no Dockerfile is generated. It ran inside the same builder microVM with no change to the VM and produced an image that booted and answered 200. Both projects are MIT, which is on the ADR 0080 allowlist. Neither is linked into Quoro: they run inside the builder VM as separate binaries and images.
3. Recommendation: a node-agent Builder trait with one implementation per frontend (Dockerfile first, Railpack later). It takes a source, a build spec and limits, and returns a stream of build events and, on success, an OCI layout on the node. Builds run on the node that will run the app, in an ephemeral builder microVM created through the Engine trait. Logs leave the VM as tagged lines on the serial console, as app logs already do, and reach the panel over SSE through the existing stream_open path. The contract details and the risks are in Implications for Quoro and Open risks.

Every number below comes from nested virtualization on one shared physical host. The roadmap says the lab is not valid for benchmarks, so read the figures as orders of magnitude.

Findings

1. How the spike was run

The spike drove cloud-hypervisor directly with the same boot payload and disk layout the engine uses (crates/engine/src/ch/vmconfig.rs). The production node agent kept running its own VM the whole time, and Quoro was never stopped.

The config disk’s argv was /bin/sh -c '<find the disk whose serial is quoro-work, mount it on /mnt/work, exec /bin/sh /mnt/work/run.sh>'. The environment was the image’s PATH plus BUILDKIT_SETUP_CGROUPV2_ROOT=1, which the BuildKit image sets so that buildkitd moves itself out of the root cgroup. run.sh mounts cgroup2, starts buildkitd --root /mnt/work/buildkit --oci-worker-net=host, waits for buildctl debug workers, runs one buildctl build, prints the result and exits. Our init then reports the exit code in the status region and powers the VM off, and the cloud-hypervisor process exits.

2. What the guest needs

The init passes /proc/self/status with Seccomp: 0 to the app. The VM is the isolation boundary here (ADR 0008). Whether runc applies BuildKit’s default seccomp profile inside the VM was not checked.

3. Measurements

Cold means a fresh work disk: empty BuildKit cache, base images pulled. Warm means the same work disk on a new VM with an unchanged context. “Change” means one source file edited. Times are wall-clock for the whole VM, from cloud-hypervisor start to its exit. The guest phase columns are /proc/uptime markers, so boot means kernel plus init up to the app’s first line.

The fixed cost of an ephemeral builder VM is about 0.3 s to boot and 0.2 to 0.4 s for buildkitd to come up. Cold builds are dominated by downloads and compilation. A warm cache on a persistent work disk brings typical rebuilds down to 2 to 4 s. In an earlier variant of the tiny image, a warm cached rebuild produced the same manifest digest as the cold build, because BuildKit reused the cached export.

Import into the engine’s cache with the scratch importer (verify every blob digest, apply the layers with openat2(RESOLVE_IN_ROOT), mkfs.ext4 -d, write image.json):

The app VMs used 1 vCPU and 512 MiB, our init, and the config the engine would derive from image.json (entrypoint plus cmd, env, workdir). An earlier tiny image failed with quoro-init: FATAL entrypoint "httpd" not found, because Alpine 3.22’s busybox has no httpd applet. The init’s failed-boot diagnostic surfaced the mistake as designed. The Dockerfile then added busybox-extras.

4. Logs and the failing step

• Transport: works with the existing path. buildctl --progress=plain writes to the app’s stdout and stderr. Our init tags every line (out or err ) on the serial console, interleaved with quoro-init: diagnostics, exactly as for app logs (engine-trait.md section 6). Nothing else is needed to get the log out of the VM. Every step line carries BuildKit’s vertex number (#6 ...). Confirmed.

• Failing step: available in two forms. The plain output ends with the failing vertex, its last log lines and the Dockerfile excerpt:

  #6 [3/4] RUN echo "about to fail" && ls /does-not-exist
  #6 0.187 ls: /does-not-exist: No such file or directory
  #6 ERROR: process "/bin/sh -c echo \"about to fail\" && ls /does-not-exist" did not complete successfully: exit code: 1
  Dockerfile:3
  --------------------
     2 |     RUN echo "step one ok"
     3 | >>> RUN echo "about to fail" && ls /does-not-exist
     4 |     RUN echo "never reached"
  --------------------
  

  With --progress=rawjson each status update is one JSON object, so the failing step is machine-readable: the vertex’s digest, name ([3/4] RUN ...), started, completed and error, plus per-vertex logs entries. The Dockerfile excerpt is still printed as text. Confirmed.

• Result: buildctl exits 1, the app exits 1, and the init writes exit=1 to the status region, so the node sees a failed build without parsing the log. --metadata-file gives the manifest digest (containerimage.digest) on success. Confirmed.

• Output: the shared work disk works. The OCI tarball, the metadata file and the BuildKit cache sit on the work disk, which the node reads after the VM exits. One wrinkle: the init powers off without unmounting that disk, so the host had to mount it with noload (journal not replayed). The builder’s run.sh, or the init, must unmount it, or the node must run e2fsck -p before reading it. Confirmed.

• vsock and virtio-fs were not needed. vsock would allow a structured channel later, but adds a guest agent. virtio-fs needs a virtiofsd per share (see cloud-hypervisor.md). Hypothesis, not tested.

5. OCI layout compatibility with the engine pipeline

6. Rootless and daemonless

• Rootful inside the VM is the recommended mode. Rootless BuildKit protects the host from the daemon. Here the microVM already does that (ADR 0008), and rootless would add user-namespace mapping, rootlesskit and the --oci-worker-no-process-sandbox trade-offs for no gain. Rootless was not tested.
• Daemonless works. The image ships buildctl-daemonless.sh. The spike’s run.sh follows the same pattern (start buildkitd, wait, build, stop), so the builder VM needs no long-lived daemon. Confirmed.
• Licenses of what runs inside the builder VM: BuildKit Apache-2.0, runc Apache-2.0 (GitHub repository metadata). They are separate binaries in a separate image, not Rust dependencies, so cargo-deny does not see them. They should still be recorded the way the Cloud Hypervisor binary is. Confirmed (licenses).
• Supply chain: BuildKit’s v0.34.2 release publishes provenance.json, sbom.json and a sigstore.json bundle per asset, which fits ADR 0027 better than an unpinned moby/buildkit:latest would. The builder image must be pinned by digest the way pins.txt pins Cloud Hypervisor and the kernel. Confirmed (assets exist). Whether the image itself carries attestations was not checked.

7. Railpack and Nixpacks (re-verification for ADR 0060)

Railpack fits ADR 0060’s “second Builder implementation” with no change to the builder VM: it is a different frontend and a prepare step in front of the same buildctl call. The ADR 0060 hypothesis that Nixpacks is in maintenance and Railpack is its successor is now confirmed.

Versions checked

Implications for Quoro

• ADR 0060: the hypothesis holds. Dockerfile builds by BuildKit inside an ephemeral builder microVM work under Cloud Hypervisor with our kernel and init, and the output feeds the OCI-to-rootfs path once ImageStore gets a local-layout source. Railpack is confirmed as Nixpacks’ successor and runs in the same VM.

• Where builds run. On the node that will run the app, in an ephemeral builder microVM created through the Engine trait, one VM per build. The layout is produced and imported on the same node, so no registry is needed and nothing is pushed anywhere. That matches ADR 0008 (no Docker on nodes) and ADR 0014 (logs stay on the node). The cost is that every node needs builder capacity (about 2 vCPU, 1 to 2 GiB and a cache disk while a build runs), and a build cache is per node.

• Engine contract gaps (an additive Engine-trait change, standalone PR per ADR 0036):

  1. prepare_image from a local OCI layout: an ImageSource::Layout { path } next to the registry reference, or a separate import_layout method, both cached by manifest digest.
  2. An extra writable disk on VmSpec for the build, identified by serial (for example quoro-work). VolumeAttach is rejected by the Cloud Hypervisor backend until M5a (“volumes ship in M5a”), so either a dedicated “scratch disk” field or an early, node-local subset of volumes.
  3. Either cgroup2 mounted by the init (a small init change that is harmless for apps) or a documented builder entrypoint that mounts it. Mounting it in the init is simpler and also helps app images that expect it.
  4. Unmount, or syncfs plus remount read-only, extra disks before power-off, so the node can read the work disk without journal recovery.

• Builder interface (node agent). A trait the node agent owns, not the Engine. It uses the Engine to run the VM:

  #![allow(unused)]
  fn main() {
  pub trait Builder {
      /// Runs one build; events stream while it runs.
      fn build(&self, spec: &BuildSpec) -> impl Future<Output = Result<BuildHandle, BuildError>> + Send;
  }
  pub struct BuildSpec {
      pub build_id: BuildId,            // bld_<ULID>
      pub source: BuildSource,          // Git { url, ref_, commit } | Upload { archive }
      pub frontend: Frontend,           // Dockerfile { path, target } | Railpack (later)
      pub context_dir: String,          // subdirectory of the repo
      pub build_args: Vec<(String, String)>,
      pub secrets: Vec<SecretRef>,      // names; values decrypted on the node
      pub limits: BuildLimits,          // vcpus, memory_mb, timeout, cache_gb, output_gb
      pub cache_key: String,            // per app: isolates caches
  }
  pub enum BuildEvent { Phase(BuildPhase), Line { stream, vertex: Option<u32>, text }, Step(StepStatus) }
  pub struct BuildOutput { pub layout: PathBuf, pub manifest_digest: ImageDigest }
  pub struct BuildFailure { pub phase: BuildPhase, pub step: Option<FailedStep>, pub excerpt: Vec<String> }
  pub struct FailedStep { pub vertex: u32, pub name: String, pub error: String, pub dockerfile_line: Option<u32> }
  }

  Implementations: DockerfileBuilder (M4) and RailpackBuilder (later). Both run the same builder VM and differ only in the frontend options and a prepare step. A FakeBuilder for tests mirrors FakeEngine.

• Git source. BuildKit clones https://...git#ref contexts itself, tested with a public repository, which avoids a host-side clone. A host-side clone onto the work disk is still the better default: the node resolves the ref to a commit SHA and records it, Quoro handles Git credentials (deploy keys, tokens) on the node instead of inside the build, and the same checkout feeds Railpack’s prepare. Private-repository authentication is out of this spike.

• Build records and states (control plane, replicated). builds with id, app_id, source (URL, ref, resolved commit), frontend, status (queued, preparing (VM and checkout), building, exporting, importing, succeeded, failed, cancelled), phase on failure, failed_step (vertex, name, error, Dockerfile line), image_digest, timings, created_by, trigger (api, push, redeploy). A successful build starts a deployment with trigger: build, which openapi.yaml already lists, as it already lists build as a deployment phase.

• Build logs over SSE (ADR 0057, ADR 0014). The log stays on the node, next to the build’s work directory, as tagged lines parsed into BuildEvents. The panel reads it through a new stream_open kind build_logs (params build_id, tail) and an SSE endpoint GET /api/v1/builds/{id}/logs/stream, plus a paginated GET .../logs for finished builds. Only the status and the failing step (a few lines) are replicated, so the panel shows the failure even when the node is down. Use --progress=rawjson and keep the plain lines as the human log: rawjson gives the failing vertex structurally and plain gives the Dockerfile excerpt.

• Cache and limits. One persistent work disk per app per node (BuildKit cache plus output), sized by cache_gb, pruned with buildctl prune --keep-storage at the end of each build, and deleted with the app. Per build: vCPU and memory of the builder VM, a wall-clock timeout (on expiry, stop the VM and mark the build failed with phase building), an output-size cap, and one concurrent build per node by default. Builds count against the node’s capacity like any VM.

• Audit and grants. Starting, cancelling and reading the logs of a build are API operations with scopes (build:write and build:read, or app:deploy and app:logs), recorded in the audit log like deployments (ADR 0031).

Open risks and unknowns

• Untrusted Dockerfiles on a production node. The VM is the boundary, but the builder VM has outbound network to fetch bases and dependencies. It should get its own isolation group (ADR 0084) with egress only, no route to app VMs or the node API.
• Secrets in builds. Build args end up in image history. BuildKit --secret needs the values inside the VM, so the contract must say how they get there (config disk, work disk, or a later channel) and that they are never logged.
• Disk pressure. Railpack’s cold cache was 2.6 GB for one small app. Per-app caches on every node can fill a small homelab disk, so the cap and prune policy must be in the contract, and the node must watch its free space.
• Reproducibility. Manifests carry org.opencontainers.image.created, so rebuilding after a cache prune changes the digest. SOURCE_DATE_EPOCH can be set if build reproducibility is wanted (ADR 0027 covers Quoro’s own artifacts, not app images).
• Base-image rate limits. Every node pulls bases from Docker Hub anonymously, with no shared mirror. Registry credentials or a pull-through cache are later items.
• Architecture. Only linux/amd64 was built. Multi-platform builds (QEMU binfmt inside the VM, buildkit-qemu-* ships in the image) were not tried.
• Builder image updates. The builder rootfs is about 300 MB and must be pinned, verified and refreshed like the kernel. The node prepares it with the normal prepare_image path, since it is an ordinary OCI image.
• Nested-virt numbers only. Real hosts should be faster. The relative costs (0.3 s boot, cold downloads dominating, 2 to 4 s warm rebuilds) are what the plan can rely on.
• Not exercised: rootless BuildKit, seccomp inside the VM, private Git and private registries, cancellation mid-build (expected to be “stop the VM”), concurrent builds on one node, and Nixpacks’ --out Dockerfile path.

Sources

• BuildKit v0.34.2 release and assets: https://github.com/moby/buildkit/releases/tag/v0.34.2 (accessed 2026-11-03)
• BuildKit license (Apache-2.0): https://github.com/moby/buildkit (repository metadata, accessed 2026-11-03)
• runc license (Apache-2.0): https://github.com/opencontainers/runc (repository metadata, accessed 2026-11-03)
• Nixpacks README with the maintenance-mode notice: https://github.com/railwayapp/nixpacks (accessed 2026-11-03)
• Nixpacks releases: https://github.com/railwayapp/nixpacks/releases (accessed 2026-11-03)
• Nixpacks Docker builder: https://github.com/railwayapp/nixpacks/blob/main/src/nixpacks/builder/docker/docker_image_builder.rs (accessed 2026-11-03)
• Nixpacks CLI (build, --out): https://nixpacks.com/docs/cli (accessed 2026-11-03)
• Railpack README and license: https://github.com/railwayapp/railpack (accessed 2026-11-03)
• Railpack releases: https://github.com/railwayapp/railpack/releases (accessed 2026-11-03)
• Railpack in production (frontend, plan, buildctl invocation): https://railpack.com/platforms/running-railpack-in-production (accessed 2026-11-03)
• Railpack frontend image versions: https://github.com/orgs/railwayapp/packages/container/package/railpack-frontend (accessed 2026-11-03)
• Engine code read for the pipeline: crates/engine/src/image/ (mod.rs, registry.rs, layer.rs, oci.rs, cache.rs, ext4.rs), crates/engine/src/ch/vmconfig.rs, crates/engine/src/ch/engine.rs
• Init behaviour: crates/init/README.md, crates/init/src/guest/mounts.rs, crates/init/src/guest/console.rs