Emulating to the point of running stock firmware is where you cross the line into lying
Voxel stuffed an entire Oxide rack into a mini PC in twelve weeks, but the real bar isn't getting it to boot — it's running unmodified main software on stock firmware. That's the line between emulation and self-deception.
The video won't play here. Listen to the audio instead:
The argument · tap a timestamp to hear it
The test environment is always twenty-seventh in line
Steve Karam says anyone who's done software development or IT knows the test environment usually isn't second fiddle — it's twenty-seventh fiddle, ahead only of the education and enablement environment and documentation. And when your product is an entire rack, the problem is especially hard: the rack itself, the fabric, the sleds, the switch zone, the SP, the root of trust — every layer needs one. So Oxide has multiple fidelity tiers: dog food racks, the racklet mini rack, single-instance Omicron (either simulated or physical running on Helios) — but it was missing a middle tier that's easier than a racklet and higher fidelity than a single chassis.
— Steve KaramFour by two compressed everything into boot time
Voxel's predecessor was called four by two, and it used a pile of shell scripts to define what the rack should look like, bringing up a virtual rack at boot with cargo bay, shared disks, networking and emulated networking. It did work — IPv6 and unnumbered BGP were first tested on four by two. But Steve points out the fatal flaw: doing everything at boot time is a recipe for disaster — boot took forty minutes, it was hard to self-heal, and "you know it's going to crash in the last two minutes of that fortieth minute." So Ryan Goodfellow proposed a rewrite and came up with the name Voxel: Virtual Oxide Emulation Labs.
— Steve KaramVoxel splits build and deploy into two steps
The biggest difference between Voxel and four by two is the separation of build and deploy: give it an Omicron commit, it clones, compiles, produces a perfectly formed sled image and snapshots it, then reuses that snapshot to play every sled and gimlet. The other big change is rewriting everything in Rust, shedding shell scripts as much as possible. Because one snapshot has to play six roles, someone has to tell it what to be — Voxel has a built-in agent written in Rust, shipped down as each sled boots, telling it "you're the switch," "you're sled 1," "you're sled 2."
— Steve KaramLLMs are great at enumerating registers, bad at judging threads
Steve says LLMs are very strong at enumerating registers and finding the bits and pieces that make up a complete system, because nobody can hold an entire rack in their head; with RFDs and design docs as context, the results get much clearer. But elsewhere it confidently reaches wrong conclusions: he once had the root of trust and the SP share a single thread for communication, and the LLM said that was absolutely not a problem — after splitting them, performance improved 500x, and the rack could finally bring both up at once.
— Steve KaramDon't modify the main software — that's Voxel's bottom line
Steve says Voxel has a creed it tries hard to hold to: don't modify the main software. When he worked on multi rack, he stuffed all kinds of hacks into Voxel to make Voxel and Omicron pretend multi-rack was already supported — for instance, creating a port that doesn't exist in Omicron at all but magically exists in Voxel. Robert said on the company channel that you shouldn't create things that don't actually exist in Omicron. Steve admits he was right: it's putting the cart before the horse, and it's the trap you easily fall into when doing emulation — you start rewriting the foundation, and then you think "why not."
— Steve KaramSimulation returns expected values; emulation exposes the truth
Steve distinguishes simulation from emulation: SP SIM is a set of lightweight stubs that respond to the sled like a real SP and can complete virtual rack initialization and come online, but it isn't a microcontroller. That's classic simulation — it returns expected responses, mostly walks the happy path, and from a testing standpoint that's exactly the problem. He gave Claude a weekend task to write an STM32 H753 microcontroller emulator, because he couldn't find an existing Rust option. It ran the first time but could barely do anything, and after another two or three hours of tuning, it successfully flashed a minimal hubris archive and the machine came up.
— Steve KaramPolling-style initialization kills emulation performance
Oxide's initialization is very pull heavy: services poll for updates at a nearly constant frequency. That's terrible for an emulated service processor, because staying in a wait state means spinning the thread, which completely destroys emulation's already-modest performance. And a real SP needs a whole pile of instructions: QSPI host boot flash, SP rot communicating with the root of trust, ignition registers, and so on. Steve's approach was to repeatedly start the Voxel lab, watch why RSS wouldn't come up, find it stuck recording temperature measurements or the like, and then add the minimal instruction set and minimal sensor set one at a time.
— Steve KaramRunning stock firmware is where you cross the line into lying
Steve gives his standard for "when is it good enough": Voxel plus SP EMU can run a complete multi-node or multi-rack lab inside a single Helios chassis, using unmodified stock Omicron (whether from source or from the tough repo), running stock hubris images — the sidecar and Gimlet images are the ones that ship with the product, with no modifications. He says that when emulation works with your shipping product, that's the boundary of the lie. Ben Stoltz also added Boodleby (root of trust secure boot) and Glasgow Bridge, so it can do tracing with humility and even run Oxide's native system update to flash firmware.
— Steve KaramIn their own words · checked verbatim
you are building like a production quality product used by Oxide to build more production quality software. It's not a step down from everything else we're doing, right?
Andrew Stone0:00
Like, without testing, without like actually using the real dendrite software and like passing through the right stuff, like, you don't have a demo, right? You're just kinda lying.
Andrew Stone5:12
this other tooling had become existentially important to the company because this is how we were able to deliver as many racks as we were
Rain Paharia15:20
It's hard to self heal. If something breaks, you know know damn well it's gonna break in the last, like, two minutes of the forty minute process.
Steve Karam17:00
an LLM will confidently tell you that, no. This absolutely is not a problem. You can have the root of trust and SP sharing a thread and communicating that way
Steve Karam24:05
I don't feel like we should be creating things that aren't actually in Omicron.
Steve Karam26:00
And to me, that is kind of the barrier of lies when they when the emulation can work with your release product.
Steve Karam44:00
Figures
| four by two boot time | 40 minutes | 17:00 |
| Debugging time after Claude wrote the STM32 H753 emulator | 2-3 hours | 30:00 |
| Voxel development duration | about 12 weeks (started late May/June) | 8:44 |
| Number of commits on the Voxel Proto branch | 70-plus | 8:44 |
| Performance gain after splitting the SP and root of trust threads | 500x | 24:05 |
| commissioning API timeline | two weeks | 15:20 |
| Thread count of Steve Karam's minisforum mini PC | 32 threads | 0:00 |
| Node rack scale runnable on that machine | six to eight nodes | 0:00 |
Glossary
- Voxel / Virtual Oxide Emulation Labs
- Oxide's rack-level emulation environment, using a snapshot to play every sled and gimlet.
- four by two
- Voxel's predecessor, which used shell scripts to bring up a virtual rack at boot time.
- simulation vs emulation
- The former returns expected responses and walks the happy path; the latter models real hardware behavior and exposes the truth.
- SP EMU
- A service processor emulator that models the STM32 microcontroller so stock hubris firmware can genuinely run.
- hubris
- Oxide's embedded operating system, running on the SP and the root of trust.
- commissioning API
- Wicket's versioned official API, used for rack initialization and recovery-style updates.
How to listen
Engineers and CTOs working at the hardware-software boundary, on emulation environments, or on internal toolchains — especially teams stuck on the question of how much fidelity is enough.
If rack hardware details don't interest you, you can skip the technical stretch around 38:00 on polling and register modeling.