Miles Ward
49d0b00a8a
This is the bulk pre-beta cleanup pass. Bumps the workspace to 0.2.0.
Test count is 56 -> 66 (+10), clippy is fully clean across the
workspace (was several dozen warnings).
## New features
**Per-MAC host bindings** (Tinkerbell smee pattern). New
`HostBindings` registry maps a MAC -> preferred boot target, persisted
to <work_dir>/hosts.json. The DHCP reply now embeds `?mac=${mac}` in
the boot.ipxe URL; iPXE substitutes the literal MAC client-side, so
the HTTP layer can short-circuit straight to the bound target instead
of rendering the menu. Reserved menu shortcuts (`_local`, `_gate`,
`_tools_menu`) are valid targets too. New /api/hosts CRUD + a Hosts
tab in the sidebar.
**Prometheus `/metrics`** endpoint. Tiny lock-free implementation —
just AtomicU64s and a Display impl, no `prometheus` / `metrics-rs`
dep. Counters: DHCP replies (per arch label), DHCP declined, TFTP
transfers (per status), TFTP bytes, HTTP requests (per route).
Gauges: ISO count, client count, gate count, gate-imaging, NFS active
mounts, uptime, build info. Plain text exposition format,
text/plain;version=0.0.4 content-type, no auth (all metric values are
non-sensitive counts).
**Light + dark themes**. CSS tokens on `:root` and
`:root[data-theme=light]`, swap by toggle button (top-right) or `T`
hotkey. Persisted in localStorage; pre-paint inline script avoids
dark<->light flash. Light palette designed against the Netbox Labs
reference screenshot — near-white surfaces, soft grey dividers,
accent unchanged for brand consistency. Terminal pane stays dark in
both themes (it's a console, that's the right read).
**Animated SVG logo + forge widget**. New `logo.svg` is a refined
silver/grey anvil. New `anvil-forge.svg` adds rising sparks and a
pulsing underglow via SMIL — pure SVG, no GIF, no JS animation loop.
Used:
- in the **forge progress** widget on Dashboard + Forge Gate, paired
with a `linear-gradient(warn -> accent)` bar with a moving sheen;
goes idle (greyscale, no sheen) at zero imaging load
- in the page-load `<div class=loader>` that replaces the old
"Loading..." text
## Code cleanup pass
`cargo clippy --workspace --all-targets` is now warning-free. Spot
fixes across the tree:
- `format!()`-into-`String` -> `std::fmt::Write::write!`
- manual reverse comparators -> `Reverse`
- `map_or(false, ...)` -> `is_some_and`
- redundant closures -> method references
- `r#"..."#` raw strings without `"` -> `r"..."`
- `std::io::Error::new(Other, ...)` -> `Error::other`
- `as i32` on `c.id()` -> `cast_signed()`
- merged identical match arms
## Windows workflow validation
New integration test synthesizes an ISO9660 with the SOURCES\\BOOT.WIM
sentinel, uploads it, asserts:
1. introspection labels it `windows_pe` with has_boot_wim=true,
2. the boot entry is `BootKind::Wimboot` with all five canonical
files (bootmgr, bootmgr.efi, bcd, boot.sdi, boot.wim),
3. the rendered iPXE script chains wimboot with `initrd --name`
entries for each file, and
4. NO trust-store strings appear in the rendered output: bcdedit,
testsigning, certutil, httpdisk, and test-signed are all
explicitly forbidden as a hard guarantee.
WinPE bootstrap (startnet.cmd) picks up the Bootimus v0.1.58 lessons:
explicit `net start Workstation` before `net use` to avoid the SMB
client lazy-init race, and surfaces errors instead of blind retries.
## Docs
architecture.md gains a "Phase 5" section explaining the host-bindings
+ metrics + theming + Windows-test work, plus a refreshed "deferred
to Phase 6" list (real-hardware integration, autounattend library,
distro profile manifest, WoL trigger, syslog receiver, IPv6).
README updates the status line, the "what it does" list, and adds
the new Hosts/Terminal tab names.
345 lines
18 KiB
Markdown
345 lines
18 KiB
Markdown
# PXEForge architecture
|
||
|
||
## Protocol stack
|
||
|
||
```
|
||
Client firmware PXE ROM
|
||
│
|
||
│ DHCPDISCOVER (UDP/67 broadcast, option 60 "PXEClient", option 93 arch)
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ PXEForge │
|
||
│ │
|
||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
|
||
│ │ DHCP proxy │ │ TFTP server │ │ HTTP server (axum) │ │
|
||
│ │ :67, :4011 │ │ :69 │ │ :80 │ │
|
||
│ │ (dhcproto) │ │ (custom) │ │ │ │
|
||
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────────────┘ │
|
||
│ │ │ │ │
|
||
│ └────────────────────┼─────────────────────┘ │
|
||
│ │ │
|
||
│ ┌──────────▼────────────┐ │
|
||
│ │ IsoStore (on disk) │ │
|
||
│ │ + ClientRegistry │ │
|
||
│ └───────────────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
│
|
||
▼ reply with option 60, option 66 tftp-server, option 67 bootfile
|
||
Client firmware PXE ROM
|
||
│
|
||
│ TFTP RRQ: snponly.efi (or undionly.kpxe for BIOS)
|
||
▼
|
||
Client runs iPXE
|
||
│
|
||
│ DHCPDISCOVER with option 77 "iPXE"
|
||
▼
|
||
PXEForge sees user-class "iPXE" → replies with HTTP URL: /boot.ipxe
|
||
│
|
||
│ HTTP GET /boot.ipxe (iPXE menu, auto-generated from IsoStore)
|
||
▼
|
||
User picks entry; iPXE chains /boot/<id>.ipxe
|
||
│
|
||
│ HTTP GET kernel + initrd (or wimboot + WIM files)
|
||
▼
|
||
Kernel boots with distro-specific args pointing back at /iso/<id>.iso
|
||
```
|
||
|
||
## Crate layout
|
||
|
||
| Crate | Responsibility |
|
||
|---------------------|-------------------------------------------------------------------|
|
||
| `pxeforge-core` | Shared types: `Config`, `ClientArch`, `FirmwareClass`, `ClientRegistry` |
|
||
| `pxeforge-ipxe-assets` | Embeds bundled iPXE binaries via `rust-embed` |
|
||
| `pxeforge-iso-store` | On-disk ISO store, introspection, boot-entry generation |
|
||
| `pxeforge-dhcp-proxy` | UDP listener + `dhcproto` reply builder; pure `decide()` unit-testable |
|
||
| `pxeforge-tftp` | RFC 1350 + OACK (blksize / tsize / windowsize). Serves only embedded assets — no filesystem |
|
||
| `pxeforge-http-api` | `axum` router: web UI, API, iPXE script generation, ISO streaming |
|
||
| `pxeforge-webui` | Single `index.html` served as static string |
|
||
| `pxeforge` (bin) | Wires everything together, runs the three servers concurrently |
|
||
|
||
## Key decisions and why
|
||
|
||
### DHCP proxy only, not a full DHCP server
|
||
|
||
Proxy mode (RFC 4578) replies with boot parameters (`siaddr`, option 66, 67)
|
||
but never sets `yiaddr` — it does not lease IPs. The client merges proxy
|
||
replies with its normal DHCP lease from the network's existing DHCP server.
|
||
|
||
This sidesteps needing `CAP_NET_RAW` or AF_PACKET. A full DHCP server has
|
||
to craft Ethernet frames to a client that doesn't yet have an IP — that
|
||
requires raw sockets and elevated privileges. A proxy replies on UDP to a
|
||
client that already has an IP (or will shortly, from the other DHCP server),
|
||
so plain `SOCK_DGRAM` is enough.
|
||
|
||
### Two-stage iPXE chain
|
||
|
||
1. Firmware PXE ROM sends DHCPDISCOVER with option 60 = `PXEClient`,
|
||
option 93 = arch.
|
||
2. PXEForge replies with TFTP server + arch-specific iPXE binary
|
||
(`undionly.kpxe` for Legacy BIOS, `snponly.efi` for x86_64 UEFI, etc.).
|
||
3. Client TFTPs the iPXE binary and runs it.
|
||
4. iPXE does its own DHCP, setting option 77 (user-class) to `iPXE`.
|
||
5. PXEForge detects the user-class and this time replies with an HTTP URL
|
||
in option 67 pointing at `/boot.ipxe`.
|
||
6. iPXE fetches and executes that script, which chains the selected OS.
|
||
|
||
The split is essential because firmware PXE ROMs only speak TFTP; they
|
||
don't do HTTP. iPXE adds HTTP (and a lot more), and is small enough to fit
|
||
in the TFTP hop.
|
||
|
||
### TFTP option negotiation is mandatory
|
||
|
||
Default 512-byte blocks means an ~1 MiB iPXE binary is ~2000 packets.
|
||
On anything less than pristine wired Ethernet this is unusably slow or
|
||
outright fails. We negotiate:
|
||
|
||
- `blksize` (RFC 2348): up to ~1468 bytes for Ethernet MTU
|
||
- `tsize` (RFC 2349): file size; some PXE ROMs require it present
|
||
- `windowsize` (RFC 7440): 8–16 gives order-of-magnitude throughput gains
|
||
|
||
### Do not touch the client OS trust store
|
||
|
||
Whatever we do for Windows, we never:
|
||
|
||
- ship drivers signed with test/development certificates
|
||
- instruct users to enable `bcdedit /set testsigning on`
|
||
- install any certificate into the target's root/trust store
|
||
|
||
iVentoy's `httpdisk.sys` approach broke this rule. PXEForge doesn't.
|
||
|
||
### Linux ISO boot uses kernel+initrd extraction, not sanboot
|
||
|
||
Loading a full ISO into RAM via `memdisk` or `sanboot` fails:
|
||
- Above ~1–2 GiB, RAM emulation is too slow or too large.
|
||
- Most modern distros can't find the emulated CD device from the initramfs.
|
||
|
||
We instead extract `vmlinuz` + `initrd` at upload time (well, look up their
|
||
locations — we serve them via ISO9660 byte-range lookup, no extraction on
|
||
disk) and pass distro-specific kernel args that point the installer back at
|
||
the HTTP-served ISO.
|
||
|
||
### No iPXE UX
|
||
|
||
iPXE is an implementation detail. The web UI only accepts `.iso` uploads
|
||
and shows distros, not `.ipxe` scripts or boot targets. If someone who
|
||
knows iPXE wants to peek, they can `curl /boot.ipxe` — that's fine. But
|
||
the UI never surfaces it.
|
||
|
||
## Unit tests
|
||
|
||
Run with `cargo test --workspace --lib`. Current coverage:
|
||
|
||
- `ClientArch` alias handling (0x0009 → x86_64 UEFI)
|
||
- `FirmwareClass` classification (iPXE wins over PXEClient echo)
|
||
- Raw option-93 extraction from the wire
|
||
- TFTP RRQ parsing with options
|
||
- HTTP Range header parsing (full, open-ended, suffix, explicit)
|
||
- Distro family detection from volume label
|
||
- ISO id slugification
|
||
|
||
## Phase 3 hardening
|
||
|
||
A code-review pass after Phase 2 turned up 16 issues across the protocol
|
||
stack and container surface. Everything P0/P1/P2 is addressed in this
|
||
release:
|
||
|
||
**Protocol reliability** (review P1 #10/#12):
|
||
- TFTP retransmit loop rewritten with explicit window tracking so UEFI SNP
|
||
clients no longer hang on files that end mid-window. Covered by 4 new
|
||
unit tests (`plan_window_*` in `crates/tftp`).
|
||
- DHCP proxy now honors the broadcast-flag bit (RFC 2131 §4.1) when
|
||
choosing unicast vs broadcast reply destination.
|
||
|
||
**Container posture** (review P1 #2/#13/#14/#15):
|
||
- Multi-arch container (linux/amd64 + linux/arm64) via `buildx`.
|
||
- New entrypoint (`deploy/docker/entrypoint.sh`) chowns data dirs as root,
|
||
then drops to uid 10001 via `gosu` — fixes the "bind-mount comes up
|
||
root-owned" problem that breaks ISO upload on standard Docker hosts.
|
||
- `/healthz` and `/readyz` split from `/api/status` — readyz fails if no
|
||
iPXE binaries are bundled, giving K8s probes a real signal.
|
||
- Startup aborts with a clear error if `PXEFORGE_PUBLIC_IP` can't be
|
||
auto-detected (no more silent `127.0.0.1` advertisement).
|
||
- `scripts/fetch-ipxe.sh` fails non-zero if zero binaries download; the
|
||
Dockerfile uses arch-scoped paths (`x86_64-efi/snponly.efi` etc.).
|
||
|
||
**Windows boot plumbing** (new):
|
||
- `pxeforge-iso-store::smb::SmbManager` supervises `smbd` on the Windows
|
||
toggle: `start` → spawn + write `smb.conf`; `reconcile` → SIGHUP on
|
||
share changes; `stop` → SIGTERM. `SmbState` surfaced to the UI for
|
||
visibility.
|
||
- `extract_windows_iso` shells out to `7z` (or `bsdtar` fallback) to
|
||
unpack the ISO tree into `smb_dir/<slug>/` as the SMB share root.
|
||
- `WimPatcher` (from Phase 2) is still in place for `boot.wim` injection.
|
||
- Guardrail: flipping the Windows toggle in `/api/settings` is rejected
|
||
with a 400 if `wimboot` isn't bundled.
|
||
|
||
**iVentoy/Bootimus parity** (P2):
|
||
- ISO sizes in menu labels (`[ 4376 MB]`), iVentoy format.
|
||
- `Reboot Computer` + `Exit and continue BIOS boot` in Tools menu.
|
||
- Number-key hotkeys (1..9) on boot entries, letter hotkeys on tools.
|
||
- Clients tab cross-joins the gate queue so an operator sees "at gate #2"
|
||
or "assigned: ubuntu-linux" status inline.
|
||
|
||
**Developer ergonomics** (new):
|
||
- `pxeforge seed --from <path>` CLI to import ISOs from a directory.
|
||
Same pipeline as web upload (slug, sha256, introspection, boot-entry).
|
||
- `docker-compose.yml` with `pxeforge` (host network, real PXE) and
|
||
`pxeforge-dev` (published ports, DHCP disabled, for API testing).
|
||
|
||
**API cleanliness**:
|
||
- All timestamps now serialized as RFC 3339 strings (the `time` crate's
|
||
default 9-tuple broke browser `Date` parsing).
|
||
- Gate poll retains assignment until the operator releases it; if the
|
||
client's chain fails, it reuses the assignment instead of falling back
|
||
to the menu.
|
||
|
||
## Phase 4 — UI restructure + remote storage
|
||
|
||
The web UI was rebuilt around six tabs (Dashboard / Network / Forge Gate /
|
||
Storage / Terminal / About) inspired by the iVentoy layout the user
|
||
attached and Netbox Labs's compact-card pattern. The old hierarchical
|
||
"Monitoring / Content / Configuration" sidebar grouping is gone — every
|
||
tab is one click from the brand bar.
|
||
|
||
**NFS share manager** (`crates/iso-store/src/nfs.rs`):
|
||
- Operators add a remote share via Storage → NFS shares; PXEForge mounts
|
||
it under `<work_dir>/nfs/<id>/` and walks it for `*.iso` files.
|
||
- Each ISO found is registered with `IsoStore::register_external` using
|
||
a new `IsoSource::Nfs { mount_id, relative_path }` variant. The store
|
||
resolves these to disk lazily via `set_nfs_root`, so adding NFS
|
||
required exactly one new field on `IsoMeta` (with a serde default for
|
||
forward compatibility with old `meta.json` files).
|
||
- Versions: `vers=3` and `vers=4.1` only. v3 also gets `nolock` since
|
||
appliances commonly disable `lockd`. All mounts use `soft,timeo=100`
|
||
so a dead server surfaces as a UI error rather than wedging iPXE.
|
||
- State persists to `<work_dir>/nfs.json`. On startup the manager
|
||
re-attempts every spec; failures are logged per-mount and surfaced in
|
||
the UI rather than blocking startup.
|
||
- Container requirement: `mount.nfs` binary plus `CAP_SYS_ADMIN`. The
|
||
default Dockerfile bundles `nfs-common`. OpenShift operators must
|
||
swap to a more permissive SCC or use a CSI driver.
|
||
|
||
**Live log + operator terminal** (`crates/core/src/log_bus.rs`,
|
||
`crates/http-api/src/{log_stream,terminal}.rs`):
|
||
- A `LogBusLayer` feeds every tracing event into a 500-line ring buffer
|
||
plus a `tokio::sync::broadcast` channel.
|
||
- `/api/log/stream` is an SSE endpoint that emits the recent buffer
|
||
followed by live updates. Slow clients see a `lagged` event rather
|
||
than dropping the stream.
|
||
- `/api/terminal` accepts a single command line and dispatches to a
|
||
whitelist (`status`, `isos`, `clients`, `gate {list,assign,release}`,
|
||
`nfs {list,mount,unmount,scan}`, `smb {status,start,stop,reload}`,
|
||
`log {clear,tail}`). Output is mirrored onto the LogBus so reading the
|
||
live tail tells the same story as scrolling the terminal pane.
|
||
- The whitelist exists deliberately — exposing a raw shell to the web
|
||
would be an RCE endpoint.
|
||
|
||
**Network tab**:
|
||
- `/api/network` exposes auto-detected `nic_name`, `subnet_mask`, and
|
||
`gateway` (parsed from `ip route` / `ip addr` at startup). These are
|
||
read-only by design — silently changing the public IP on a hot UI
|
||
would break PXE for every client mid-boot.
|
||
- The only writable network field is `dns_server`, an optional
|
||
informational hint stored in `Settings`. PXEForge does not run a DNS
|
||
server; the field exists so operators don't have to dig out the
|
||
upstream DNS at 3 AM.
|
||
|
||
**Bootimus parity nicks** (Bootimus v0.1.55 → v0.1.62):
|
||
- Storage table tints amber for ISOs that won't boot with current
|
||
settings (Windows ISO when Windows is disabled, Linux ISO with no
|
||
detected kernel + over the sanboot size threshold). Each row carries
|
||
an inline reason — same affordance as Bootimus's "Image Properties"
|
||
warning.
|
||
- Dashboard surfaces a "Images that won't boot" panel reusing the same
|
||
predicate, so the operator sees the problem before they pick the ISO
|
||
in the gate.
|
||
- Streaming uploads are already in place via axum multipart; the v0.1.62
|
||
fix to "502 on big upload" doesn't apply.
|
||
|
||
## Phase 5 — pre-beta hardening
|
||
|
||
**Per-MAC host bindings** (`crates/core/src/host_bindings.rs`):
|
||
- New `HostBindings` registry maps a MAC → preferred `BootEntry::id`
|
||
(or one of the reserved menu shortcuts `_local`, `_gate`,
|
||
`_tools_menu`).
|
||
- Persisted to `<work_dir>/hosts.json`. Like `SettingsStore`, in-memory
|
||
is authoritative — disk corruption falls back to empty rather than
|
||
failing startup.
|
||
- Inspired by Tinkerbell `smee`'s MAC-prepended URL pattern. The DHCP
|
||
reply now embeds `?mac=${mac}` in the boot.ipxe URL; iPXE substitutes
|
||
the literal MAC client-side, so the HTTP layer can short-circuit
|
||
past the menu when a binding exists.
|
||
- `/api/hosts` GET / POST / DELETE drives the **Hosts** tab.
|
||
|
||
**Prometheus metrics** (`crates/core/src/metrics.rs`):
|
||
- Lock-free `AtomicU64`-backed counters + gauges. No `prometheus` /
|
||
`metrics-rs` dep — they bring a registry, runtime, and complexity
|
||
we don't need for a fixed set of metric families.
|
||
- Counters: DHCP replies (per arch label), DHCP declined, TFTP
|
||
transfers (per status label), TFTP bytes, HTTP requests (per route
|
||
label).
|
||
- Gauges: ISO count, client count, gate count, gate-imaging count,
|
||
NFS active mounts, uptime, build info.
|
||
- Exposed as plain Prometheus text at `/metrics`.
|
||
|
||
**Code cleanup pass**: clippy `--workspace --all-targets` is now
|
||
warning-free. Replaced `format!()`-into-`String` with
|
||
`std::fmt::Write::write!`, switched manual reverse comparators to
|
||
`Reverse`, fixed `map_or(false, …)` → `is_some_and`, and a handful of
|
||
other idiom fixes.
|
||
|
||
**UI overhaul** for the v0.2.0 pre-beta milestone:
|
||
- Light + dark themes via `:root[data-theme=light]` token swap.
|
||
Toggled by a top-right button or the `T` key. Persisted in
|
||
localStorage; pre-paint inline script avoids dark→light flash.
|
||
- New SVG logos: a refined anvil (`logo.svg`) and a SMIL-animated
|
||
`anvil-forge.svg` (rising sparks + pulsing underglow). Pure SVG —
|
||
no GIFs, no CSS keyframes for the sparks.
|
||
- "Forge progress" widget on the Dashboard and Forge Gate: animated
|
||
anvil paired with a `linear-gradient(warn → accent)` progress bar
|
||
with a moving sheen. Goes idle (greyscale, no sheen) at zero
|
||
imaging load.
|
||
- Loader replaced "Loading…" text with the same anvil.
|
||
- Sidebar gains a **Hosts** tab.
|
||
|
||
**Windows boot validation**:
|
||
- New integration test synthesizes an ISO9660 with the `SOURCES\BOOT.WIM`
|
||
sentinel, uploads it, and asserts:
|
||
1. introspection labels it `windows_pe` with `has_boot_wim=true`,
|
||
2. the boot entry is `BootKind::Wimboot` with all five canonical
|
||
files (`bootmgr`, `bootmgr.efi`, `bcd`, `boot.sdi`, `boot.wim`),
|
||
3. the rendered iPXE script chains wimboot with `initrd --name`
|
||
entries for each, and
|
||
4. **no** trust-store strings appear: `bcdedit`, `testsigning`,
|
||
`certutil`, `httpdisk`, `test-signed` are all explicitly
|
||
forbidden in the rendered output.
|
||
- WinPE bootstrap (`startnet.cmd`) now picks up Bootimus v0.1.58
|
||
fixes: explicit `net start Workstation` before `net use`, surfaces
|
||
errors instead of blind retries.
|
||
|
||
**Test count**: 66 → up from 56 in v0.1.0.
|
||
|
||
## What's deferred to Phase 6
|
||
|
||
- Full ISO9660 + Joliet + Rock Ridge parser (current lookup is plain ISO9660 — Debian ISOs with Rock Ridge extensions may miss some paths).
|
||
- Real-hardware Windows boot validation (plumbing tested; no MS ISO pushed through the full pipeline yet).
|
||
- Real-hardware NFS validation (mount manager tested; no real NAS pushed
|
||
through the full pipeline yet).
|
||
- UEFI HTTP Boot protocol (option 60 = `HTTPClient`) untested on real firmware.
|
||
- Raspberry Pi netboot quirks (option 43 vendor-specific, per-MAC prefixes).
|
||
- Multi-replica / daemonset deployment model (single replica for now).
|
||
- Pure-Rust SMB server (replace smbd) — slim image, no Samba.
|
||
- Auto-install / autounattend file library (Bootimus v0.1.58 pattern).
|
||
- Per-client / per-group menus (Bootimus v0.1.16 pattern).
|
||
- Real-hardware integration: at minimum a Linux ISO booted on a real
|
||
BIOS box, a Windows ISO booted via wimboot on a real UEFI box, and a
|
||
Pi 4 booting from an NFS-mounted Raspberry Pi OS ISO.
|
||
- Distro profile manifest (Bootimus v0.1.27 pattern) — currently
|
||
introspection logic is hard-coded; could become data-driven so an
|
||
operator can add a new distro profile from the UI without rebuilding.
|
||
- Wake-on-LAN trigger (Bootimus v0.1.16 pattern) — power-on a host then
|
||
imaging starts unattended via a per-MAC binding.
|
||
- Syslog receiver (smee feature) — capture client-side install syslog
|
||
for diagnostic visibility.
|
||
- IPv6 PXE / DHCPv6 — currently IPv4 only.
|