OpenPXE/docs/architecture.md

OpenPXE architecture

Protocol stack

Client firmware PXE ROM
        │
        │ DHCPDISCOVER (UDP/67 broadcast, option 60 "PXEClient", option 93 arch)
        ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  OpenPXE                                                               │
│                                                                         │
│    ┌──────────────┐     ┌──────────────┐      ┌──────────────────────┐  │
│    │ DHCP proxy   │     │ TFTP server  │      │  HTTP server (axum)  │  │
│    │ :67, :4011   │     │ :69          │      │  :80                 │  │
│    │  (dhcproto)  │     │  (custom)    │      │                      │  │
│    └──────┬───────┘     └──────┬───────┘      └──────┬───────────────┘  │
│           │                    │                     │                  │
│           └────────────────────┼─────────────────────┘                  │
│                                │                                        │
│                     ┌──────────▼────────────┐                           │
│                     │  IsoStore (on disk)   │                           │
│                     │  + ClientRegistry     │                           │
│                     └───────────────────────┘                           │
└─────────────────────────────────────────────────────────────────────────┘
        │
        ▼ reply with option 60, option 66 tftp-server, option 67 bootfile
Client firmware PXE ROM
        │
        │ TFTP RRQ: snponly.efi (or undionly.kpxe for BIOS)
        ▼
Client runs iPXE
        │
        │ DHCPDISCOVER with option 77 "iPXE"
        ▼
OpenPXE sees user-class "iPXE" → replies with HTTP URL: /boot.ipxe
        │
        │ HTTP GET /boot.ipxe  (iPXE menu, auto-generated from IsoStore)
        ▼
User picks entry; iPXE chains /boot/<id>.ipxe
        │
        │ HTTP GET kernel + initrd (or wimboot + WIM files)
        ▼
Kernel boots with distro-specific args pointing back at /iso/<id>.iso

Crate layout

Crate	Responsibility
openpxe-core	Shared types: Config, ClientArch, FirmwareClass, ClientRegistry
openpxe-ipxe-assets	Embeds bundled iPXE binaries via rust-embed
openpxe-iso-store	On-disk ISO store, introspection, boot-entry generation
openpxe-dhcp-proxy	UDP listener + dhcproto reply builder; pure decide() unit-testable
openpxe-tftp	RFC 1350 + OACK (blksize / tsize / windowsize). Serves only embedded assets — no filesystem
openpxe-http-api	axum router: web UI, API, iPXE script generation, ISO streaming
openpxe-webui	Single index.html served as static string
openpxe (bin)	Wires everything together, runs the three servers concurrently

Key decisions and why

DHCP proxy only, not a full DHCP server

Proxy mode (RFC 4578) replies with boot parameters (siaddr, option 66, 67) but never sets yiaddr — it does not lease IPs. The client merges proxy replies with its normal DHCP lease from the network's existing DHCP server.

This sidesteps needing CAP_NET_RAW or AF_PACKET. A full DHCP server has to craft Ethernet frames to a client that doesn't yet have an IP — that requires raw sockets and elevated privileges. A proxy replies on UDP to a client that already has an IP (or will shortly, from the other DHCP server), so plain SOCK_DGRAM is enough.

Two-stage iPXE chain

1. Firmware PXE ROM sends DHCPDISCOVER with option 60 = PXEClient, option 93 = arch.
2. OpenPXE replies with TFTP server + arch-specific iPXE binary (undionly.kpxe for Legacy BIOS, snponly.efi for x86_64 UEFI, etc.).
3. Client TFTPs the iPXE binary and runs it.
4. iPXE does its own DHCP, setting option 77 (user-class) to iPXE.
5. OpenPXE detects the user-class and this time replies with an HTTP URL in option 67 pointing at /boot.ipxe.
6. iPXE fetches and executes that script, which chains the selected OS.

The split is essential because firmware PXE ROMs only speak TFTP; they don't do HTTP. iPXE adds HTTP (and a lot more), and is small enough to fit in the TFTP hop.

TFTP option negotiation is mandatory

Default 512-byte blocks means an ~1 MiB iPXE binary is ~2000 packets. On anything less than pristine wired Ethernet this is unusably slow or outright fails. We negotiate:

• blksize (RFC 2348): up to ~1468 bytes for Ethernet MTU
• tsize (RFC 2349): file size; some PXE ROMs require it present
• windowsize (RFC 7440): 8–16 gives order-of-magnitude throughput gains

Do not touch the client OS trust store

Whatever we do for Windows, we never:

• ship drivers signed with test/development certificates
• instruct users to enable bcdedit /set testsigning on
• install any certificate into the target's root/trust store

iVentoy's httpdisk.sys approach broke this rule. OpenPXE doesn't.

Linux ISO boot uses kernel+initrd extraction, not sanboot

Loading a full ISO into RAM via memdisk or sanboot fails:

• Above ~1–2 GiB, RAM emulation is too slow or too large.
• Most modern distros can't find the emulated CD device from the initramfs.

We instead extract vmlinuz + initrd at upload time (well, look up their locations — we serve them via ISO9660 byte-range lookup, no extraction on disk) and pass distro-specific kernel args that point the installer back at the HTTP-served ISO.

No iPXE UX

iPXE is an implementation detail. The web UI only accepts .iso uploads and shows distros, not .ipxe scripts or boot targets. If someone who knows iPXE wants to peek, they can curl /boot.ipxe — that's fine. But the UI never surfaces it.

Unit tests

Run with cargo test --workspace --lib. Current coverage:

• ClientArch alias handling (0x0009 → x86_64 UEFI)
• FirmwareClass classification (iPXE wins over PXEClient echo)
• Raw option-93 extraction from the wire
• TFTP RRQ parsing with options
• HTTP Range header parsing (full, open-ended, suffix, explicit)
• Distro family detection from volume label
• ISO id slugification

Phase 3 hardening

A code-review pass after Phase 2 turned up 16 issues across the protocol stack and container surface. Everything P0/P1/P2 is addressed in this release:

Protocol reliability (review P1 #10/#12):

• TFTP retransmit loop rewritten with explicit window tracking so UEFI SNP clients no longer hang on files that end mid-window. Covered by 4 new unit tests (plan_window_* in crates/tftp).
• DHCP proxy now honors the broadcast-flag bit (RFC 2131 §4.1) when choosing unicast vs broadcast reply destination.

Container posture (review P1 #2/#13/#14/#15):

• Multi-arch container (linux/amd64 + linux/arm64) via buildx.
• New entrypoint (deploy/docker/entrypoint.sh) chowns data dirs as root, then drops to uid 10001 via gosu — fixes the "bind-mount comes up root-owned" problem that breaks ISO upload on standard Docker hosts.
• /healthz and /readyz split from /api/status — readyz fails if no iPXE binaries are bundled, giving K8s probes a real signal.
• Startup aborts with a clear error if OPENPXE_PUBLIC_IP can't be auto-detected (no more silent 127.0.0.1 advertisement).
• scripts/fetch-ipxe.sh fails non-zero if zero binaries download; the Dockerfile uses arch-scoped paths (x86_64-efi/snponly.efi etc.).

Windows boot plumbing (new):

• openpxe-iso-store::smb::SmbManager supervises smbd on the Windows toggle: start → spawn + write smb.conf; reconcile → SIGHUP on share changes; stop → SIGTERM. SmbState surfaced to the UI for visibility.
• extract_windows_iso shells out to 7z (or bsdtar fallback) to unpack the ISO tree into smb_dir/<slug>/ as the SMB share root.
• WimPatcher (from Phase 2) is still in place for boot.wim injection.
• Guardrail: flipping the Windows toggle in /api/settings is rejected with a 400 if wimboot isn't bundled.

iVentoy/Bootimus parity (P2):

• ISO sizes in menu labels ([ 4376 MB]), iVentoy format.
• Reboot Computer + Exit and continue BIOS boot in Tools menu.
• Number-key hotkeys (1..9) on boot entries, letter hotkeys on tools.
• Clients tab cross-joins the deployment queue so an operator sees "in queue #2" or "assigned: ubuntu-linux" status inline.

Developer ergonomics (new):

• openpxe seed --from <path> CLI to import ISOs from a directory. Same pipeline as web upload (slug, sha256, introspection, boot-entry).
• docker-compose.yml with openpxe (host network, real PXE) and openpxe-dev (published ports, DHCP disabled, for API testing).

API cleanliness:

• All timestamps now serialized as RFC 3339 strings (the time crate's default 9-tuple broke browser Date parsing).
• Queue poll retains assignment until the operator releases it; if the client's chain fails, it reuses the assignment instead of falling back to the menu.

Phase 4 — UI restructure + remote storage

The web UI was rebuilt around six tabs (Dashboard / Network / Queue / Storage / Terminal / About) inspired by the iVentoy layout the user attached and Netbox Labs's compact-card pattern. The old hierarchical "Monitoring / Content / Configuration" sidebar grouping is gone — every tab is one click from the brand bar.

NFS share manager (crates/iso-store/src/nfs.rs):

• Operators add a remote share via Storage → NFS shares; OpenPXE mounts it under <work_dir>/nfs/<id>/ and walks it for *.iso files.
• Each ISO found is registered with IsoStore::register_external using a new IsoSource::Nfs { mount_id, relative_path } variant. The store resolves these to disk lazily via set_nfs_root, so adding NFS required exactly one new field on IsoMeta (with a serde default for forward compatibility with old meta.json files).
• Versions: vers=3 and vers=4.1 only. v3 also gets nolock since appliances commonly disable lockd. All mounts use soft,timeo=100 so a dead server surfaces as a UI error rather than wedging iPXE.
• State persists to <work_dir>/nfs.json. On startup the manager re-attempts every spec; failures are logged per-mount and surfaced in the UI rather than blocking startup.
• Container requirement: mount.nfs binary plus CAP_SYS_ADMIN. The default Dockerfile bundles nfs-common. OpenShift operators must swap to a more permissive SCC or use a CSI driver.

Live log + operator terminal (crates/core/src/log_bus.rs, crates/http-api/src/{log_stream,terminal}.rs):

• A LogBusLayer feeds every tracing event into a 500-line ring buffer plus a tokio::sync::broadcast channel.
• /api/log/stream is an SSE endpoint that emits the recent buffer followed by live updates. Slow clients see a lagged event rather than dropping the stream.
• /api/terminal accepts a single command line and dispatches to a whitelist (status, isos, clients, queue {list,assign,release}, nfs {list,mount,unmount,scan}, smb {status,start,stop,reload}, log {clear,tail}). Output is mirrored onto the LogBus so reading the live tail tells the same story as scrolling the terminal pane.
• The whitelist exists deliberately — exposing a raw shell to the web would be an RCE endpoint.

Network tab:

• /api/network exposes auto-detected nic_name, subnet_mask, and gateway (parsed from ip route / ip addr at startup). These are read-only by design — silently changing the public IP on a hot UI would break PXE for every client mid-boot.
• The only writable network field is dns_server, an optional informational hint stored in Settings. OpenPXE does not run a DNS server; the field exists so operators don't have to dig out the upstream DNS at 3 AM.

Bootimus parity nicks (Bootimus v0.1.55 → v0.1.62):

• Storage table tints amber for ISOs that won't boot with current settings (Windows ISO when Windows is disabled, Linux ISO with no detected kernel + over the sanboot size threshold). Each row carries an inline reason — same affordance as Bootimus's "Image Properties" warning.
• Dashboard surfaces a "Images that won't boot" panel reusing the same predicate, so the operator sees the problem before they pick the ISO in the queue.
• Streaming uploads are already in place via axum multipart; the v0.1.62 fix to "502 on big upload" doesn't apply.

Phase 5 — pre-beta hardening

Per-MAC host bindings (crates/core/src/host_bindings.rs):

• New HostBindings registry maps a MAC → preferred BootEntry::id (or one of the reserved menu shortcuts (_local, _queue``, _tools_menu`).
• Persisted to <work_dir>/hosts.json. Like SettingsStore, in-memory is authoritative — disk corruption falls back to empty rather than failing startup.
• Inspired by Tinkerbell smee's MAC-prepended URL pattern. The DHCP reply now embeds ?mac=${mac} in the boot.ipxe URL; iPXE substitutes the literal MAC client-side, so the HTTP layer can short-circuit past the menu when a binding exists.
• /api/hosts GET / POST / DELETE drives the Hosts tab.

Prometheus metrics (crates/core/src/metrics.rs):

• Lock-free AtomicU64-backed counters + gauges. No prometheus / metrics-rs dep — they bring a registry, runtime, and complexity we don't need for a fixed set of metric families.
• Counters: DHCP replies (per arch label), DHCP declined, TFTP transfers (per status label), TFTP bytes, HTTP requests (per route label).
• Gauges: ISO count, client count, queue count, queue-imaging count, NFS active mounts, uptime, build info.
• Exposed as plain Prometheus text at /metrics.

Code cleanup pass: clippy --workspace --all-targets is now warning-free. Replaced format!()-into-String with std::fmt::Write::write!, switched manual reverse comparators to Reverse, fixed map_or(false, …) → is_some_and, and a handful of other idiom fixes.

UI overhaul for the v0.2.0 pre-beta milestone:

• Light + dark themes via :root[data-theme=light] token swap. Toggled by a top-right button or the T key. Persisted in localStorage; pre-paint inline script avoids dark→light flash.
• New SVG logos: a refined anvil (logo.svg) and a SMIL-animated anvil-forge.svg (rising sparks + pulsing underglow). Pure SVG — no GIFs, no CSS keyframes for the sparks.
• "Forge progress" widget on the Dashboard and Queue: animated anvil paired with a linear-gradient(warn → accent) progress bar with a moving sheen. Goes idle (greyscale, no sheen) at zero imaging load.
• Loader replaced "Loading…" text with the same anvil.
• Sidebar gains a Hosts tab.

Windows boot validation:

• New integration test synthesizes an ISO9660 with the SOURCES\BOOT.WIM sentinel, uploads it, and asserts:
  1. introspection labels it windows_pe with has_boot_wim=true,
  2. the boot entry is BootKind::Wimboot with all five canonical files (bootmgr, bootmgr.efi, bcd, boot.sdi, boot.wim),
  3. the rendered iPXE script chains wimboot with initrd --name entries for each, and
  4. no trust-store strings appear: bcdedit, testsigning, certutil, httpdisk, test-signed are all explicitly forbidden in the rendered output.
• WinPE bootstrap (startnet.cmd) now picks up Bootimus v0.1.58 fixes: explicit net start Workstation before net use, surfaces errors instead of blind retries.

Test count: 66 → up from 56 in v0.1.0.

What's deferred to Phase 6

• Full ISO9660 + Joliet + Rock Ridge parser (current lookup is plain ISO9660 — Debian ISOs with Rock Ridge extensions may miss some paths).
• Real-hardware Windows boot validation (plumbing tested; no MS ISO pushed through the full pipeline yet).
• Real-hardware NFS validation (mount manager tested; no real NAS pushed through the full pipeline yet).
• UEFI HTTP Boot protocol (option 60 = HTTPClient) untested on real firmware.
• Raspberry Pi netboot quirks (option 43 vendor-specific, per-MAC prefixes).
• Multi-replica / daemonset deployment model (single replica for now).
• Pure-Rust SMB server (replace smbd) — slim image, no Samba.
• Auto-install / autounattend file library (Bootimus v0.1.58 pattern).
• Per-client / per-group menus (Bootimus v0.1.16 pattern).
• Real-hardware integration: at minimum a Linux ISO booted on a real BIOS box, a Windows ISO booted via wimboot on a real UEFI box, and a Pi 4 booting from an NFS-mounted Raspberry Pi OS ISO.
• Distro profile manifest (Bootimus v0.1.27 pattern) — currently introspection logic is hard-coded; could become data-driven so an operator can add a new distro profile from the UI without rebuilding.
• Wake-on-LAN trigger (Bootimus v0.1.16 pattern) — power-on a host then imaging starts unattended via a per-MAC binding.
• Syslog receiver (smee feature) — capture client-side install syslog for diagnostic visibility.
• IPv6 PXE / DHCPv6 — currently IPv4 only.