345 lines
18 KiB
Markdown
345 lines
18 KiB
Markdown
# OpenPXE architecture
|
||
|
||
## Protocol stack
|
||
|
||
```
|
||
Client firmware PXE ROM
|
||
│
|
||
│ DHCPDISCOVER (UDP/67 broadcast, option 60 "PXEClient", option 93 arch)
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ OpenPXE │
|
||
│ │
|
||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
|
||
│ │ DHCP proxy │ │ TFTP server │ │ HTTP server (axum) │ │
|
||
│ │ :67, :4011 │ │ :69 │ │ :80 │ │
|
||
│ │ (dhcproto) │ │ (custom) │ │ │ │
|
||
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────────────┘ │
|
||
│ │ │ │ │
|
||
│ └────────────────────┼─────────────────────┘ │
|
||
│ │ │
|
||
│ ┌──────────▼────────────┐ │
|
||
│ │ IsoStore (on disk) │ │
|
||
│ │ + ClientRegistry │ │
|
||
│ └───────────────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
│
|
||
▼ reply with option 60, option 66 tftp-server, option 67 bootfile
|
||
Client firmware PXE ROM
|
||
│
|
||
│ TFTP RRQ: snponly.efi (or undionly.kpxe for BIOS)
|
||
▼
|
||
Client runs iPXE
|
||
│
|
||
│ DHCPDISCOVER with option 77 "iPXE"
|
||
▼
|
||
OpenPXE sees user-class "iPXE" → replies with HTTP URL: /boot.ipxe
|
||
│
|
||
│ HTTP GET /boot.ipxe (iPXE menu, auto-generated from IsoStore)
|
||
▼
|
||
User picks entry; iPXE chains /boot/<id>.ipxe
|
||
│
|
||
│ HTTP GET kernel + initrd (or wimboot + WIM files)
|
||
▼
|
||
Kernel boots with distro-specific args pointing back at /iso/<id>.iso
|
||
```
|
||
|
||
## Crate layout
|
||
|
||
| Crate | Responsibility |
|
||
|---------------------|-------------------------------------------------------------------|
|
||
| `openpxe-core` | Shared types: `Config`, `ClientArch`, `FirmwareClass`, `ClientRegistry` |
|
||
| `openpxe-ipxe-assets` | Embeds bundled iPXE binaries via `rust-embed` |
|
||
| `openpxe-iso-store` | On-disk ISO store, introspection, boot-entry generation |
|
||
| `openpxe-dhcp-proxy` | UDP listener + `dhcproto` reply builder; pure `decide()` unit-testable |
|
||
| `openpxe-tftp` | RFC 1350 + OACK (blksize / tsize / windowsize). Serves only embedded assets — no filesystem |
|
||
| `openpxe-http-api` | `axum` router: web UI, API, iPXE script generation, ISO streaming |
|
||
| `openpxe-webui` | Single `index.html` served as static string |
|
||
| `openpxe` (bin) | Wires everything together, runs the three servers concurrently |
|
||
|
||
## Key decisions and why
|
||
|
||
### DHCP proxy only, not a full DHCP server
|
||
|
||
Proxy mode (RFC 4578) replies with boot parameters (`siaddr`, option 66, 67)
|
||
but never sets `yiaddr` — it does not lease IPs. The client merges proxy
|
||
replies with its normal DHCP lease from the network's existing DHCP server.
|
||
|
||
This sidesteps needing `CAP_NET_RAW` or AF_PACKET. A full DHCP server has
|
||
to craft Ethernet frames to a client that doesn't yet have an IP — that
|
||
requires raw sockets and elevated privileges. A proxy replies on UDP to a
|
||
client that already has an IP (or will shortly, from the other DHCP server),
|
||
so plain `SOCK_DGRAM` is enough.
|
||
|
||
### Two-stage iPXE chain
|
||
|
||
1. Firmware PXE ROM sends DHCPDISCOVER with option 60 = `PXEClient`,
|
||
option 93 = arch.
|
||
2. OpenPXE replies with TFTP server + arch-specific iPXE binary
|
||
(`undionly.kpxe` for Legacy BIOS, `snponly.efi` for x86_64 UEFI, etc.).
|
||
3. Client TFTPs the iPXE binary and runs it.
|
||
4. iPXE does its own DHCP, setting option 77 (user-class) to `iPXE`.
|
||
5. OpenPXE detects the user-class and this time replies with an HTTP URL
|
||
in option 67 pointing at `/boot.ipxe`.
|
||
6. iPXE fetches and executes that script, which chains the selected OS.
|
||
|
||
The split is essential because firmware PXE ROMs only speak TFTP; they
|
||
don't do HTTP. iPXE adds HTTP (and a lot more), and is small enough to fit
|
||
in the TFTP hop.
|
||
|
||
### TFTP option negotiation is mandatory
|
||
|
||
Default 512-byte blocks means an ~1 MiB iPXE binary is ~2000 packets.
|
||
On anything less than pristine wired Ethernet this is unusably slow or
|
||
outright fails. We negotiate:
|
||
|
||
- `blksize` (RFC 2348): up to ~1468 bytes for Ethernet MTU
|
||
- `tsize` (RFC 2349): file size; some PXE ROMs require it present
|
||
- `windowsize` (RFC 7440): 8–16 gives order-of-magnitude throughput gains
|
||
|
||
### Do not touch the client OS trust store
|
||
|
||
Whatever we do for Windows, we never:
|
||
|
||
- ship drivers signed with test/development certificates
|
||
- instruct users to enable `bcdedit /set testsigning on`
|
||
- install any certificate into the target's root/trust store
|
||
|
||
iVentoy's `httpdisk.sys` approach broke this rule. OpenPXE doesn't.
|
||
|
||
### Linux ISO boot uses kernel+initrd extraction, not sanboot
|
||
|
||
Loading a full ISO into RAM via `memdisk` or `sanboot` fails:
|
||
- Above ~1–2 GiB, RAM emulation is too slow or too large.
|
||
- Most modern distros can't find the emulated CD device from the initramfs.
|
||
|
||
We instead extract `vmlinuz` + `initrd` at upload time (well, look up their
|
||
locations — we serve them via ISO9660 byte-range lookup, no extraction on
|
||
disk) and pass distro-specific kernel args that point the installer back at
|
||
the HTTP-served ISO.
|
||
|
||
### No iPXE UX
|
||
|
||
iPXE is an implementation detail. The web UI only accepts `.iso` uploads
|
||
and shows distros, not `.ipxe` scripts or boot targets. If someone who
|
||
knows iPXE wants to peek, they can `curl /boot.ipxe` — that's fine. But
|
||
the UI never surfaces it.
|
||
|
||
## Unit tests
|
||
|
||
Run with `cargo test --workspace --lib`. Current coverage:
|
||
|
||
- `ClientArch` alias handling (0x0009 → x86_64 UEFI)
|
||
- `FirmwareClass` classification (iPXE wins over PXEClient echo)
|
||
- Raw option-93 extraction from the wire
|
||
- TFTP RRQ parsing with options
|
||
- HTTP Range header parsing (full, open-ended, suffix, explicit)
|
||
- Distro family detection from volume label
|
||
- ISO id slugification
|
||
|
||
## Phase 3 hardening
|
||
|
||
A code-review pass after Phase 2 turned up 16 issues across the protocol
|
||
stack and container surface. Everything P0/P1/P2 is addressed in this
|
||
release:
|
||
|
||
**Protocol reliability** (review P1 #10/#12):
|
||
- TFTP retransmit loop rewritten with explicit window tracking so UEFI SNP
|
||
clients no longer hang on files that end mid-window. Covered by 4 new
|
||
unit tests (`plan_window_*` in `crates/tftp`).
|
||
- DHCP proxy now honors the broadcast-flag bit (RFC 2131 §4.1) when
|
||
choosing unicast vs broadcast reply destination.
|
||
|
||
**Container posture** (review P1 #2/#13/#14/#15):
|
||
- Multi-arch container (linux/amd64 + linux/arm64) via `buildx`.
|
||
- New entrypoint (`deploy/docker/entrypoint.sh`) chowns data dirs as root,
|
||
then drops to uid 10001 via `gosu` — fixes the "bind-mount comes up
|
||
root-owned" problem that breaks ISO upload on standard Docker hosts.
|
||
- `/healthz` and `/readyz` split from `/api/status` — readyz fails if no
|
||
iPXE binaries are bundled, giving K8s probes a real signal.
|
||
- Startup aborts with a clear error if `OPENPXE_PUBLIC_IP` can't be
|
||
auto-detected (no more silent `127.0.0.1` advertisement).
|
||
- `scripts/fetch-ipxe.sh` fails non-zero if zero binaries download; the
|
||
Dockerfile uses arch-scoped paths (`x86_64-efi/snponly.efi` etc.).
|
||
|
||
**Windows boot plumbing** (new):
|
||
- `openpxe-iso-store::smb::SmbManager` supervises `smbd` on the Windows
|
||
toggle: `start` → spawn + write `smb.conf`; `reconcile` → SIGHUP on
|
||
share changes; `stop` → SIGTERM. `SmbState` surfaced to the UI for
|
||
visibility.
|
||
- `extract_windows_iso` shells out to `7z` (or `bsdtar` fallback) to
|
||
unpack the ISO tree into `smb_dir/<slug>/` as the SMB share root.
|
||
- `WimPatcher` (from Phase 2) is still in place for `boot.wim` injection.
|
||
- Guardrail: flipping the Windows toggle in `/api/settings` is rejected
|
||
with a 400 if `wimboot` isn't bundled.
|
||
|
||
**iVentoy/Bootimus parity** (P2):
|
||
- ISO sizes in menu labels (`[ 4376 MB]`), iVentoy format.
|
||
- `Reboot Computer` + `Exit and continue BIOS boot` in Tools menu.
|
||
- Number-key hotkeys (1..9) on boot entries, letter hotkeys on tools.
|
||
- Clients tab cross-joins the deployment queue so an operator sees "in queue #2"
|
||
or "assigned: ubuntu-linux" status inline.
|
||
|
||
**Developer ergonomics** (new):
|
||
- `openpxe seed --from <path>` CLI to import ISOs from a directory.
|
||
Same pipeline as web upload (slug, sha256, introspection, boot-entry).
|
||
- `docker-compose.yml` with `openpxe` (host network, real PXE) and
|
||
`openpxe-dev` (published ports, DHCP disabled, for API testing).
|
||
|
||
**API cleanliness**:
|
||
- All timestamps now serialized as RFC 3339 strings (the `time` crate's
|
||
default 9-tuple broke browser `Date` parsing).
|
||
- Queue poll retains assignment until the operator releases it; if the
|
||
client's chain fails, it reuses the assignment instead of falling back
|
||
to the menu.
|
||
|
||
## Phase 4 — UI restructure + remote storage
|
||
|
||
The web UI was rebuilt around six tabs (Dashboard / Network / Queue /
|
||
Storage / Terminal / About) inspired by the iVentoy layout the user
|
||
attached and Netbox Labs's compact-card pattern. The old hierarchical
|
||
"Monitoring / Content / Configuration" sidebar grouping is gone — every
|
||
tab is one click from the brand bar.
|
||
|
||
**NFS share manager** (`crates/iso-store/src/nfs.rs`):
|
||
- Operators add a remote share via Storage → NFS shares; OpenPXE mounts
|
||
it under `<work_dir>/nfs/<id>/` and walks it for `*.iso` files.
|
||
- Each ISO found is registered with `IsoStore::register_external` using
|
||
a new `IsoSource::Nfs { mount_id, relative_path }` variant. The store
|
||
resolves these to disk lazily via `set_nfs_root`, so adding NFS
|
||
required exactly one new field on `IsoMeta` (with a serde default for
|
||
forward compatibility with old `meta.json` files).
|
||
- Versions: `vers=3` and `vers=4.1` only. v3 also gets `nolock` since
|
||
appliances commonly disable `lockd`. All mounts use `soft,timeo=100`
|
||
so a dead server surfaces as a UI error rather than wedging iPXE.
|
||
- State persists to `<work_dir>/nfs.json`. On startup the manager
|
||
re-attempts every spec; failures are logged per-mount and surfaced in
|
||
the UI rather than blocking startup.
|
||
- Container requirement: `mount.nfs` binary plus `CAP_SYS_ADMIN`. The
|
||
default Dockerfile bundles `nfs-common`. OpenShift operators must
|
||
swap to a more permissive SCC or use a CSI driver.
|
||
|
||
**Live log + operator terminal** (`crates/core/src/log_bus.rs`,
|
||
`crates/http-api/src/{log_stream,terminal}.rs`):
|
||
- A `LogBusLayer` feeds every tracing event into a 500-line ring buffer
|
||
plus a `tokio::sync::broadcast` channel.
|
||
- `/api/log/stream` is an SSE endpoint that emits the recent buffer
|
||
followed by live updates. Slow clients see a `lagged` event rather
|
||
than dropping the stream.
|
||
- `/api/terminal` accepts a single command line and dispatches to a
|
||
whitelist (`status`, `isos`, `clients`, `queue {list,assign,release}`,
|
||
`nfs {list,mount,unmount,scan}`, `smb {status,start,stop,reload}`,
|
||
`log {clear,tail}`). Output is mirrored onto the LogBus so reading the
|
||
live tail tells the same story as scrolling the terminal pane.
|
||
- The whitelist exists deliberately — exposing a raw shell to the web
|
||
would be an RCE endpoint.
|
||
|
||
**Network tab**:
|
||
- `/api/network` exposes auto-detected `nic_name`, `subnet_mask`, and
|
||
`gateway` (parsed from `ip route` / `ip addr` at startup). These are
|
||
read-only by design — silently changing the public IP on a hot UI
|
||
would break PXE for every client mid-boot.
|
||
- The only writable network field is `dns_server`, an optional
|
||
informational hint stored in `Settings`. OpenPXE does not run a DNS
|
||
server; the field exists so operators don't have to dig out the
|
||
upstream DNS at 3 AM.
|
||
|
||
**Bootimus parity nicks** (Bootimus v0.1.55 → v0.1.62):
|
||
- Storage table tints amber for ISOs that won't boot with current
|
||
settings (Windows ISO when Windows is disabled, Linux ISO with no
|
||
detected kernel + over the sanboot size threshold). Each row carries
|
||
an inline reason — same affordance as Bootimus's "Image Properties"
|
||
warning.
|
||
- Dashboard surfaces a "Images that won't boot" panel reusing the same
|
||
predicate, so the operator sees the problem before they pick the ISO
|
||
in the queue.
|
||
- Streaming uploads are already in place via axum multipart; the v0.1.62
|
||
fix to "502 on big upload" doesn't apply.
|
||
|
||
## Phase 5 — pre-beta hardening
|
||
|
||
**Per-MAC host bindings** (`crates/core/src/host_bindings.rs`):
|
||
- New `HostBindings` registry maps a MAC → preferred `BootEntry::id`
|
||
(or one of the reserved menu shortcuts (`_local`, `_queue``,
|
||
`_tools_menu`).
|
||
- Persisted to `<work_dir>/hosts.json`. Like `SettingsStore`, in-memory
|
||
is authoritative — disk corruption falls back to empty rather than
|
||
failing startup.
|
||
- Inspired by Tinkerbell `smee`'s MAC-prepended URL pattern. The DHCP
|
||
reply now embeds `?mac=${mac}` in the boot.ipxe URL; iPXE substitutes
|
||
the literal MAC client-side, so the HTTP layer can short-circuit
|
||
past the menu when a binding exists.
|
||
- `/api/hosts` GET / POST / DELETE drives the **Hosts** tab.
|
||
|
||
**Prometheus metrics** (`crates/core/src/metrics.rs`):
|
||
- Lock-free `AtomicU64`-backed counters + gauges. No `prometheus` /
|
||
`metrics-rs` dep — they bring a registry, runtime, and complexity
|
||
we don't need for a fixed set of metric families.
|
||
- Counters: DHCP replies (per arch label), DHCP declined, TFTP
|
||
transfers (per status label), TFTP bytes, HTTP requests (per route
|
||
label).
|
||
- Gauges: ISO count, client count, queue count, queue-imaging count,
|
||
NFS active mounts, uptime, build info.
|
||
- Exposed as plain Prometheus text at `/metrics`.
|
||
|
||
**Code cleanup pass**: clippy `--workspace --all-targets` is now
|
||
warning-free. Replaced `format!()`-into-`String` with
|
||
`std::fmt::Write::write!`, switched manual reverse comparators to
|
||
`Reverse`, fixed `map_or(false, …)` → `is_some_and`, and a handful of
|
||
other idiom fixes.
|
||
|
||
**UI overhaul** for the v0.2.0 pre-beta milestone:
|
||
- Light + dark themes via `:root[data-theme=light]` token swap.
|
||
Toggled by a top-right button or the `T` key. Persisted in
|
||
localStorage; pre-paint inline script avoids dark→light flash.
|
||
- New SVG logos: a refined anvil (`logo.svg`) and a SMIL-animated
|
||
`anvil-forge.svg` (rising sparks + pulsing underglow). Pure SVG —
|
||
no GIFs, no CSS keyframes for the sparks.
|
||
- "Forge progress" widget on the Dashboard and Queue: animated
|
||
anvil paired with a `linear-gradient(warn → accent)` progress bar
|
||
with a moving sheen. Goes idle (greyscale, no sheen) at zero
|
||
imaging load.
|
||
- Loader replaced "Loading…" text with the same anvil.
|
||
- Sidebar gains a **Hosts** tab.
|
||
|
||
**Windows boot validation**:
|
||
- New integration test synthesizes an ISO9660 with the `SOURCES\BOOT.WIM`
|
||
sentinel, uploads it, and asserts:
|
||
1. introspection labels it `windows_pe` with `has_boot_wim=true`,
|
||
2. the boot entry is `BootKind::Wimboot` with all five canonical
|
||
files (`bootmgr`, `bootmgr.efi`, `bcd`, `boot.sdi`, `boot.wim`),
|
||
3. the rendered iPXE script chains wimboot with `initrd --name`
|
||
entries for each, and
|
||
4. **no** trust-store strings appear: `bcdedit`, `testsigning`,
|
||
`certutil`, `httpdisk`, `test-signed` are all explicitly
|
||
forbidden in the rendered output.
|
||
- WinPE bootstrap (`startnet.cmd`) now picks up Bootimus v0.1.58
|
||
fixes: explicit `net start Workstation` before `net use`, surfaces
|
||
errors instead of blind retries.
|
||
|
||
**Test count**: 66 → up from 56 in v0.1.0.
|
||
|
||
## What's deferred to Phase 6
|
||
|
||
- Full ISO9660 + Joliet + Rock Ridge parser (current lookup is plain ISO9660 — Debian ISOs with Rock Ridge extensions may miss some paths).
|
||
- Real-hardware Windows boot validation (plumbing tested; no MS ISO pushed through the full pipeline yet).
|
||
- Real-hardware NFS validation (mount manager tested; no real NAS pushed
|
||
through the full pipeline yet).
|
||
- UEFI HTTP Boot protocol (option 60 = `HTTPClient`) untested on real firmware.
|
||
- Raspberry Pi netboot quirks (option 43 vendor-specific, per-MAC prefixes).
|
||
- Multi-replica / daemonset deployment model (single replica for now).
|
||
- Pure-Rust SMB server (replace smbd) — slim image, no Samba.
|
||
- Auto-install / autounattend file library (Bootimus v0.1.58 pattern).
|
||
- Per-client / per-group menus (Bootimus v0.1.16 pattern).
|
||
- Real-hardware integration: at minimum a Linux ISO booted on a real
|
||
BIOS box, a Windows ISO booted via wimboot on a real UEFI box, and a
|
||
Pi 4 booting from an NFS-mounted Raspberry Pi OS ISO.
|
||
- Distro profile manifest (Bootimus v0.1.27 pattern) — currently
|
||
introspection logic is hard-coded; could become data-driven so an
|
||
operator can add a new distro profile from the UI without rebuilding.
|
||
- Wake-on-LAN trigger (Bootimus v0.1.16 pattern) — power-on a host then
|
||
imaging starts unattended via a per-MAC binding.
|
||
- Syslog receiver (smee feature) — capture client-side install syslog
|
||
for diagnostic visibility.
|
||
- IPv6 PXE / DHCPv6 — currently IPv4 only.
|