The core problem Oxide faces is unsexy but brutal: how do you test rack-scale infrastructure when every finished rack ships to a paying customer? Their answer, Voxel (Virtual OXidE Labs), is a full software emulation of an Oxide rack — sleds, switches, service processors, root of trust, BGP networking — running on a single machine with no special test builds. The Prime Directive, as Bryan Cantrill frames it, is that labs must use stock Oxide software without hack or modification. Voxel runs on commodity hardware. The author's home setup is a Minisforum MS-A2 with a 32-thread AMD Ryzen 9 9955HX and 96GB RAM. On that, six sleds spin up in about 10 minutes — either as one rack or a multirack configuration with separate management planes. Each sled lives in a Propolis VM alongside emulated switches and infrastructure, all wired by Oxide's falcon topology suite using simnet and VNIC pairs. The genuinely hard engineering is in the microcontroller emulation. Oxide racks embed a Service Processor (SP) and Root of Trust (RoT) in every sled — Cortex-M7 and Cortex-M33 devices running Hubris firmware. Rather than stubbing these out with happy-path simulators, the team built sp-emu, an open-source ARM emulator for illumos and Linux that faithfully runs real Hubris firmware. Each device runs in a single thread, emulating instructions, time, and interrupts. The time problem is particularly gnarly: Hubris schedules off a 1ms SysTick, but emulated hardware is slower than silicon. Their solution converts wall time into instruction counts, firing tick exceptions based on the kernel's own reload value. Networking fidelity comes from SoftNPU, Oxide's software P4 switch, combined with Debian VMs running FRR for BGP (numbered and unnumbered) or static routing. This means Voxel can replicate diverse customer networking topologies — the kind of configuration variance that breaks things in production but is impossible to stockpile in physical hardware. The image creation pipeline is elegant. Since Oxide sleds are only ever a 'gimlet' (or 'cosmo') or a 'scrimlet' (switch-wired sled), Voxel creates a VM, installs the Omicron control plane, snapshots it before configuration, and reuses that ZFS snapshot to stamp out racks. The control plane installs either from source via cargo or from a TUF (The Update Framework) repo packaged by their CI tool Buildomat. No special test releases, no divergence from what ships. There are acknowledged gaps. The sprockets PKI trust material isn't yet sourced from the RoT at manufacturing time the way production racks do it. But the team calls this out explicitly and frames it as non-blocking for runtime testing — honest engineering tradeoffs rather than swept-under-the-rug compromises. What makes this significant beyond Oxide's own CI pipeline is the pattern it establishes: high-fidelity emulation of vertically integrated hardware stacks, open-sourced at the hardest layers (sp-emu is public), running stock production code. For an industry that routinely ships undertested firmware on embedded management controllers, this is a structural challenge to the norm.
