OpenPXE/docs/architecture.md

PXEForge architecture

Protocol stack

Client firmware PXE ROM
        │
        │ DHCPDISCOVER (UDP/67 broadcast, option 60 "PXEClient", option 93 arch)
        ▼
┌─────────────────────────────────────────────────────────────────────────┐
│  PXEForge                                                               │
│                                                                         │
│    ┌──────────────┐     ┌──────────────┐      ┌──────────────────────┐  │
│    │ DHCP proxy   │     │ TFTP server  │      │  HTTP server (axum)  │  │
│    │ :67, :4011   │     │ :69          │      │  :80                 │  │
│    │  (dhcproto)  │     │  (custom)    │      │                      │  │
│    └──────┬───────┘     └──────┬───────┘      └──────┬───────────────┘  │
│           │                    │                     │                  │
│           └────────────────────┼─────────────────────┘                  │
│                                │                                        │
│                     ┌──────────▼────────────┐                           │
│                     │  IsoStore (on disk)   │                           │
│                     │  + ClientRegistry     │                           │
│                     └───────────────────────┘                           │
└─────────────────────────────────────────────────────────────────────────┘
        │
        ▼ reply with option 60, option 66 tftp-server, option 67 bootfile
Client firmware PXE ROM
        │
        │ TFTP RRQ: snponly.efi (or undionly.kpxe for BIOS)
        ▼
Client runs iPXE
        │
        │ DHCPDISCOVER with option 77 "iPXE"
        ▼
PXEForge sees user-class "iPXE" → replies with HTTP URL: /boot.ipxe
        │
        │ HTTP GET /boot.ipxe  (iPXE menu, auto-generated from IsoStore)
        ▼
User picks entry; iPXE chains /boot/<id>.ipxe
        │
        │ HTTP GET kernel + initrd (or wimboot + WIM files)
        ▼
Kernel boots with distro-specific args pointing back at /iso/<id>.iso

Crate layout

Crate	Responsibility
pxeforge-core	Shared types: Config, ClientArch, FirmwareClass, ClientRegistry
pxeforge-ipxe-assets	Embeds bundled iPXE binaries via rust-embed
pxeforge-iso-store	On-disk ISO store, introspection, boot-entry generation
pxeforge-dhcp-proxy	UDP listener + dhcproto reply builder; pure decide() unit-testable
pxeforge-tftp	RFC 1350 + OACK (blksize / tsize / windowsize). Serves only embedded assets — no filesystem
pxeforge-http-api	axum router: web UI, API, iPXE script generation, ISO streaming
pxeforge-webui	Single index.html served as static string
pxeforge (bin)	Wires everything together, runs the three servers concurrently

Key decisions and why

DHCP proxy only, not a full DHCP server

Proxy mode (RFC 4578) replies with boot parameters (siaddr, option 66, 67) but never sets yiaddr — it does not lease IPs. The client merges proxy replies with its normal DHCP lease from the network's existing DHCP server.

This sidesteps needing CAP_NET_RAW or AF_PACKET. A full DHCP server has to craft Ethernet frames to a client that doesn't yet have an IP — that requires raw sockets and elevated privileges. A proxy replies on UDP to a client that already has an IP (or will shortly, from the other DHCP server), so plain SOCK_DGRAM is enough.

Two-stage iPXE chain

1. Firmware PXE ROM sends DHCPDISCOVER with option 60 = PXEClient, option 93 = arch.
2. PXEForge replies with TFTP server + arch-specific iPXE binary (undionly.kpxe for Legacy BIOS, snponly.efi for x86_64 UEFI, etc.).
3. Client TFTPs the iPXE binary and runs it.
4. iPXE does its own DHCP, setting option 77 (user-class) to iPXE.
5. PXEForge detects the user-class and this time replies with an HTTP URL in option 67 pointing at /boot.ipxe.
6. iPXE fetches and executes that script, which chains the selected OS.

The split is essential because firmware PXE ROMs only speak TFTP; they don't do HTTP. iPXE adds HTTP (and a lot more), and is small enough to fit in the TFTP hop.

TFTP option negotiation is mandatory

Default 512-byte blocks means an ~1 MiB iPXE binary is ~2000 packets. On anything less than pristine wired Ethernet this is unusably slow or outright fails. We negotiate:

• blksize (RFC 2348): up to ~1468 bytes for Ethernet MTU
• tsize (RFC 2349): file size; some PXE ROMs require it present
• windowsize (RFC 7440): 8–16 gives order-of-magnitude throughput gains

Do not touch the client OS trust store

Whatever we do for Windows, we never:

• ship drivers signed with test/development certificates
• instruct users to enable bcdedit /set testsigning on
• install any certificate into the target's root/trust store

iVentoy's httpdisk.sys approach broke this rule. PXEForge doesn't.

Linux ISO boot uses kernel+initrd extraction, not sanboot

Loading a full ISO into RAM via memdisk or sanboot fails:

• Above ~1–2 GiB, RAM emulation is too slow or too large.
• Most modern distros can't find the emulated CD device from the initramfs.

We instead extract vmlinuz + initrd at upload time (well, look up their locations — we serve them via ISO9660 byte-range lookup, no extraction on disk) and pass distro-specific kernel args that point the installer back at the HTTP-served ISO.

No iPXE UX

iPXE is an implementation detail. The web UI only accepts .iso uploads and shows distros, not .ipxe scripts or boot targets. If someone who knows iPXE wants to peek, they can curl /boot.ipxe — that's fine. But the UI never surfaces it.

Unit tests

Run with cargo test --workspace --lib. Current coverage:

• ClientArch alias handling (0x0009 → x86_64 UEFI)
• FirmwareClass classification (iPXE wins over PXEClient echo)
• Raw option-93 extraction from the wire
• TFTP RRQ parsing with options
• HTTP Range header parsing (full, open-ended, suffix, explicit)
• Distro family detection from volume label
• ISO id slugification

Phase 3 hardening

A code-review pass after Phase 2 turned up 16 issues across the protocol stack and container surface. Everything P0/P1/P2 is addressed in this release:

Protocol reliability (review P1 #10/#12):

• TFTP retransmit loop rewritten with explicit window tracking so UEFI SNP clients no longer hang on files that end mid-window. Covered by 4 new unit tests (plan_window_* in crates/tftp).
• DHCP proxy now honors the broadcast-flag bit (RFC 2131 §4.1) when choosing unicast vs broadcast reply destination.

Container posture (review P1 #2/#13/#14/#15):

• Multi-arch container (linux/amd64 + linux/arm64) via buildx.
• New entrypoint (deploy/docker/entrypoint.sh) chowns data dirs as root, then drops to uid 10001 via gosu — fixes the "bind-mount comes up root-owned" problem that breaks ISO upload on standard Docker hosts.
• /healthz and /readyz split from /api/status — readyz fails if no iPXE binaries are bundled, giving K8s probes a real signal.
• Startup aborts with a clear error if PXEFORGE_PUBLIC_IP can't be auto-detected (no more silent 127.0.0.1 advertisement).
• scripts/fetch-ipxe.sh fails non-zero if zero binaries download; the Dockerfile uses arch-scoped paths (x86_64-efi/snponly.efi etc.).

Windows boot plumbing (new):

• pxeforge-iso-store::smb::SmbManager supervises smbd on the Windows toggle: start → spawn + write smb.conf; reconcile → SIGHUP on share changes; stop → SIGTERM. SmbState surfaced to the UI for visibility.
• extract_windows_iso shells out to 7z (or bsdtar fallback) to unpack the ISO tree into smb_dir/<slug>/ as the SMB share root.
• WimPatcher (from Phase 2) is still in place for boot.wim injection.
• Guardrail: flipping the Windows toggle in /api/settings is rejected with a 400 if wimboot isn't bundled.

iVentoy/Bootimus parity (P2):

• ISO sizes in menu labels ([ 4376 MB]), iVentoy format.
• Reboot Computer + Exit and continue BIOS boot in Tools menu.
• Number-key hotkeys (1..9) on boot entries, letter hotkeys on tools.
• Clients tab cross-joins the gate queue so an operator sees "at gate #2" or "assigned: ubuntu-linux" status inline.

Developer ergonomics (new):

• pxeforge seed --from <path> CLI to import ISOs from a directory. Same pipeline as web upload (slug, sha256, introspection, boot-entry).
• docker-compose.yml with pxeforge (host network, real PXE) and pxeforge-dev (published ports, DHCP disabled, for API testing).

API cleanliness:

• All timestamps now serialized as RFC 3339 strings (the time crate's default 9-tuple broke browser Date parsing).
• Gate poll retains assignment until the operator releases it; if the client's chain fails, it reuses the assignment instead of falling back to the menu.

Phase 4 — UI restructure + remote storage

The web UI was rebuilt around six tabs (Dashboard / Network / Forge Gate / Storage / Terminal / About) inspired by the iVentoy layout the user attached and Netbox Labs's compact-card pattern. The old hierarchical "Monitoring / Content / Configuration" sidebar grouping is gone — every tab is one click from the brand bar.

NFS share manager (crates/iso-store/src/nfs.rs):

• Operators add a remote share via Storage → NFS shares; PXEForge mounts it under <work_dir>/nfs/<id>/ and walks it for *.iso files.
• Each ISO found is registered with IsoStore::register_external using a new IsoSource::Nfs { mount_id, relative_path } variant. The store resolves these to disk lazily via set_nfs_root, so adding NFS required exactly one new field on IsoMeta (with a serde default for forward compatibility with old meta.json files).
• Versions: vers=3 and vers=4.1 only. v3 also gets nolock since appliances commonly disable lockd. All mounts use soft,timeo=100 so a dead server surfaces as a UI error rather than wedging iPXE.
• State persists to <work_dir>/nfs.json. On startup the manager re-attempts every spec; failures are logged per-mount and surfaced in the UI rather than blocking startup.
• Container requirement: mount.nfs binary plus CAP_SYS_ADMIN. The default Dockerfile bundles nfs-common. OpenShift operators must swap to a more permissive SCC or use a CSI driver.

Live log + operator terminal (crates/core/src/log_bus.rs, crates/http-api/src/{log_stream,terminal}.rs):

• A LogBusLayer feeds every tracing event into a 500-line ring buffer plus a tokio::sync::broadcast channel.
• /api/log/stream is an SSE endpoint that emits the recent buffer followed by live updates. Slow clients see a lagged event rather than dropping the stream.
• /api/terminal accepts a single command line and dispatches to a whitelist (status, isos, clients, gate {list,assign,release}, nfs {list,mount,unmount,scan}, smb {status,start,stop,reload}, log {clear,tail}). Output is mirrored onto the LogBus so reading the live tail tells the same story as scrolling the terminal pane.
• The whitelist exists deliberately — exposing a raw shell to the web would be an RCE endpoint.

Network tab:

• /api/network exposes auto-detected nic_name, subnet_mask, and gateway (parsed from ip route / ip addr at startup). These are read-only by design — silently changing the public IP on a hot UI would break PXE for every client mid-boot.
• The only writable network field is dns_server, an optional informational hint stored in Settings. PXEForge does not run a DNS server; the field exists so operators don't have to dig out the upstream DNS at 3 AM.

Bootimus parity nicks (Bootimus v0.1.55 → v0.1.62):

• Storage table tints amber for ISOs that won't boot with current settings (Windows ISO when Windows is disabled, Linux ISO with no detected kernel + over the sanboot size threshold). Each row carries an inline reason — same affordance as Bootimus's "Image Properties" warning.
• Dashboard surfaces a "Images that won't boot" panel reusing the same predicate, so the operator sees the problem before they pick the ISO in the gate.
• Streaming uploads are already in place via axum multipart; the v0.1.62 fix to "502 on big upload" doesn't apply.

What's deferred to Phase 5

• Full ISO9660 + Joliet + Rock Ridge parser (current lookup is plain ISO9660 — Debian ISOs with Rock Ridge extensions may miss some paths).
• Real-hardware Windows boot validation (plumbing tested; no MS ISO pushed through the full pipeline yet).
• Real-hardware NFS validation (mount manager tested; no real NAS pushed through the full pipeline yet).
• UEFI HTTP Boot protocol (option 60 = HTTPClient) untested on real firmware.
• Raspberry Pi netboot quirks (option 43 vendor-specific, per-MAC prefixes).
• Multi-replica / daemonset deployment model (single replica for now).
• Pure-Rust SMB server (replace smbd) — slim image, no Samba.
• Auto-install / autounattend file library (Bootimus v0.1.58 pattern).
• Per-client / per-group menus (Bootimus v0.1.16 pattern).