271 lines
14 KiB
Markdown
271 lines
14 KiB
Markdown
# PXEForge architecture
|
||
|
||
## Protocol stack
|
||
|
||
```
|
||
Client firmware PXE ROM
|
||
│
|
||
│ DHCPDISCOVER (UDP/67 broadcast, option 60 "PXEClient", option 93 arch)
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ PXEForge │
|
||
│ │
|
||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
|
||
│ │ DHCP proxy │ │ TFTP server │ │ HTTP server (axum) │ │
|
||
│ │ :67, :4011 │ │ :69 │ │ :80 │ │
|
||
│ │ (dhcproto) │ │ (custom) │ │ │ │
|
||
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────────────┘ │
|
||
│ │ │ │ │
|
||
│ └────────────────────┼─────────────────────┘ │
|
||
│ │ │
|
||
│ ┌──────────▼────────────┐ │
|
||
│ │ IsoStore (on disk) │ │
|
||
│ │ + ClientRegistry │ │
|
||
│ └───────────────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
│
|
||
▼ reply with option 60, option 66 tftp-server, option 67 bootfile
|
||
Client firmware PXE ROM
|
||
│
|
||
│ TFTP RRQ: snponly.efi (or undionly.kpxe for BIOS)
|
||
▼
|
||
Client runs iPXE
|
||
│
|
||
│ DHCPDISCOVER with option 77 "iPXE"
|
||
▼
|
||
PXEForge sees user-class "iPXE" → replies with HTTP URL: /boot.ipxe
|
||
│
|
||
│ HTTP GET /boot.ipxe (iPXE menu, auto-generated from IsoStore)
|
||
▼
|
||
User picks entry; iPXE chains /boot/<id>.ipxe
|
||
│
|
||
│ HTTP GET kernel + initrd (or wimboot + WIM files)
|
||
▼
|
||
Kernel boots with distro-specific args pointing back at /iso/<id>.iso
|
||
```
|
||
|
||
## Crate layout
|
||
|
||
| Crate | Responsibility |
|
||
|---------------------|-------------------------------------------------------------------|
|
||
| `pxeforge-core` | Shared types: `Config`, `ClientArch`, `FirmwareClass`, `ClientRegistry` |
|
||
| `pxeforge-ipxe-assets` | Embeds bundled iPXE binaries via `rust-embed` |
|
||
| `pxeforge-iso-store` | On-disk ISO store, introspection, boot-entry generation |
|
||
| `pxeforge-dhcp-proxy` | UDP listener + `dhcproto` reply builder; pure `decide()` unit-testable |
|
||
| `pxeforge-tftp` | RFC 1350 + OACK (blksize / tsize / windowsize). Serves only embedded assets — no filesystem |
|
||
| `pxeforge-http-api` | `axum` router: web UI, API, iPXE script generation, ISO streaming |
|
||
| `pxeforge-webui` | Single `index.html` served as static string |
|
||
| `pxeforge` (bin) | Wires everything together, runs the three servers concurrently |
|
||
|
||
## Key decisions and why
|
||
|
||
### DHCP proxy only, not a full DHCP server
|
||
|
||
Proxy mode (RFC 4578) replies with boot parameters (`siaddr`, option 66, 67)
|
||
but never sets `yiaddr` — it does not lease IPs. The client merges proxy
|
||
replies with its normal DHCP lease from the network's existing DHCP server.
|
||
|
||
This sidesteps needing `CAP_NET_RAW` or AF_PACKET. A full DHCP server has
|
||
to craft Ethernet frames to a client that doesn't yet have an IP — that
|
||
requires raw sockets and elevated privileges. A proxy replies on UDP to a
|
||
client that already has an IP (or will shortly, from the other DHCP server),
|
||
so plain `SOCK_DGRAM` is enough.
|
||
|
||
### Two-stage iPXE chain
|
||
|
||
1. Firmware PXE ROM sends DHCPDISCOVER with option 60 = `PXEClient`,
|
||
option 93 = arch.
|
||
2. PXEForge replies with TFTP server + arch-specific iPXE binary
|
||
(`undionly.kpxe` for Legacy BIOS, `snponly.efi` for x86_64 UEFI, etc.).
|
||
3. Client TFTPs the iPXE binary and runs it.
|
||
4. iPXE does its own DHCP, setting option 77 (user-class) to `iPXE`.
|
||
5. PXEForge detects the user-class and this time replies with an HTTP URL
|
||
in option 67 pointing at `/boot.ipxe`.
|
||
6. iPXE fetches and executes that script, which chains the selected OS.
|
||
|
||
The split is essential because firmware PXE ROMs only speak TFTP; they
|
||
don't do HTTP. iPXE adds HTTP (and a lot more), and is small enough to fit
|
||
in the TFTP hop.
|
||
|
||
### TFTP option negotiation is mandatory
|
||
|
||
Default 512-byte blocks means an ~1 MiB iPXE binary is ~2000 packets.
|
||
On anything less than pristine wired Ethernet this is unusably slow or
|
||
outright fails. We negotiate:
|
||
|
||
- `blksize` (RFC 2348): up to ~1468 bytes for Ethernet MTU
|
||
- `tsize` (RFC 2349): file size; some PXE ROMs require it present
|
||
- `windowsize` (RFC 7440): 8–16 gives order-of-magnitude throughput gains
|
||
|
||
### Do not touch the client OS trust store
|
||
|
||
Whatever we do for Windows, we never:
|
||
|
||
- ship drivers signed with test/development certificates
|
||
- instruct users to enable `bcdedit /set testsigning on`
|
||
- install any certificate into the target's root/trust store
|
||
|
||
iVentoy's `httpdisk.sys` approach broke this rule. PXEForge doesn't.
|
||
|
||
### Linux ISO boot uses kernel+initrd extraction, not sanboot
|
||
|
||
Loading a full ISO into RAM via `memdisk` or `sanboot` fails:
|
||
- Above ~1–2 GiB, RAM emulation is too slow or too large.
|
||
- Most modern distros can't find the emulated CD device from the initramfs.
|
||
|
||
We instead extract `vmlinuz` + `initrd` at upload time (well, look up their
|
||
locations — we serve them via ISO9660 byte-range lookup, no extraction on
|
||
disk) and pass distro-specific kernel args that point the installer back at
|
||
the HTTP-served ISO.
|
||
|
||
### No iPXE UX
|
||
|
||
iPXE is an implementation detail. The web UI only accepts `.iso` uploads
|
||
and shows distros, not `.ipxe` scripts or boot targets. If someone who
|
||
knows iPXE wants to peek, they can `curl /boot.ipxe` — that's fine. But
|
||
the UI never surfaces it.
|
||
|
||
## Unit tests
|
||
|
||
Run with `cargo test --workspace --lib`. Current coverage:
|
||
|
||
- `ClientArch` alias handling (0x0009 → x86_64 UEFI)
|
||
- `FirmwareClass` classification (iPXE wins over PXEClient echo)
|
||
- Raw option-93 extraction from the wire
|
||
- TFTP RRQ parsing with options
|
||
- HTTP Range header parsing (full, open-ended, suffix, explicit)
|
||
- Distro family detection from volume label
|
||
- ISO id slugification
|
||
|
||
## Phase 3 hardening
|
||
|
||
A code-review pass after Phase 2 turned up 16 issues across the protocol
|
||
stack and container surface. Everything P0/P1/P2 is addressed in this
|
||
release:
|
||
|
||
**Protocol reliability** (review P1 #10/#12):
|
||
- TFTP retransmit loop rewritten with explicit window tracking so UEFI SNP
|
||
clients no longer hang on files that end mid-window. Covered by 4 new
|
||
unit tests (`plan_window_*` in `crates/tftp`).
|
||
- DHCP proxy now honors the broadcast-flag bit (RFC 2131 §4.1) when
|
||
choosing unicast vs broadcast reply destination.
|
||
|
||
**Container posture** (review P1 #2/#13/#14/#15):
|
||
- Multi-arch container (linux/amd64 + linux/arm64) via `buildx`.
|
||
- New entrypoint (`deploy/docker/entrypoint.sh`) chowns data dirs as root,
|
||
then drops to uid 10001 via `gosu` — fixes the "bind-mount comes up
|
||
root-owned" problem that breaks ISO upload on standard Docker hosts.
|
||
- `/healthz` and `/readyz` split from `/api/status` — readyz fails if no
|
||
iPXE binaries are bundled, giving K8s probes a real signal.
|
||
- Startup aborts with a clear error if `PXEFORGE_PUBLIC_IP` can't be
|
||
auto-detected (no more silent `127.0.0.1` advertisement).
|
||
- `scripts/fetch-ipxe.sh` fails non-zero if zero binaries download; the
|
||
Dockerfile uses arch-scoped paths (`x86_64-efi/snponly.efi` etc.).
|
||
|
||
**Windows boot plumbing** (new):
|
||
- `pxeforge-iso-store::smb::SmbManager` supervises `smbd` on the Windows
|
||
toggle: `start` → spawn + write `smb.conf`; `reconcile` → SIGHUP on
|
||
share changes; `stop` → SIGTERM. `SmbState` surfaced to the UI for
|
||
visibility.
|
||
- `extract_windows_iso` shells out to `7z` (or `bsdtar` fallback) to
|
||
unpack the ISO tree into `smb_dir/<slug>/` as the SMB share root.
|
||
- `WimPatcher` (from Phase 2) is still in place for `boot.wim` injection.
|
||
- Guardrail: flipping the Windows toggle in `/api/settings` is rejected
|
||
with a 400 if `wimboot` isn't bundled.
|
||
|
||
**iVentoy/Bootimus parity** (P2):
|
||
- ISO sizes in menu labels (`[ 4376 MB]`), iVentoy format.
|
||
- `Reboot Computer` + `Exit and continue BIOS boot` in Tools menu.
|
||
- Number-key hotkeys (1..9) on boot entries, letter hotkeys on tools.
|
||
- Clients tab cross-joins the gate queue so an operator sees "at gate #2"
|
||
or "assigned: ubuntu-linux" status inline.
|
||
|
||
**Developer ergonomics** (new):
|
||
- `pxeforge seed --from <path>` CLI to import ISOs from a directory.
|
||
Same pipeline as web upload (slug, sha256, introspection, boot-entry).
|
||
- `docker-compose.yml` with `pxeforge` (host network, real PXE) and
|
||
`pxeforge-dev` (published ports, DHCP disabled, for API testing).
|
||
|
||
**API cleanliness**:
|
||
- All timestamps now serialized as RFC 3339 strings (the `time` crate's
|
||
default 9-tuple broke browser `Date` parsing).
|
||
- Gate poll retains assignment until the operator releases it; if the
|
||
client's chain fails, it reuses the assignment instead of falling back
|
||
to the menu.
|
||
|
||
## Phase 4 — UI restructure + remote storage
|
||
|
||
The web UI was rebuilt around six tabs (Dashboard / Network / Forge Gate /
|
||
Storage / Terminal / About) inspired by the iVentoy layout the user
|
||
attached and Netbox Labs's compact-card pattern. The old hierarchical
|
||
"Monitoring / Content / Configuration" sidebar grouping is gone — every
|
||
tab is one click from the brand bar.
|
||
|
||
**NFS share manager** (`crates/iso-store/src/nfs.rs`):
|
||
- Operators add a remote share via Storage → NFS shares; PXEForge mounts
|
||
it under `<work_dir>/nfs/<id>/` and walks it for `*.iso` files.
|
||
- Each ISO found is registered with `IsoStore::register_external` using
|
||
a new `IsoSource::Nfs { mount_id, relative_path }` variant. The store
|
||
resolves these to disk lazily via `set_nfs_root`, so adding NFS
|
||
required exactly one new field on `IsoMeta` (with a serde default for
|
||
forward compatibility with old `meta.json` files).
|
||
- Versions: `vers=3` and `vers=4.1` only. v3 also gets `nolock` since
|
||
appliances commonly disable `lockd`. All mounts use `soft,timeo=100`
|
||
so a dead server surfaces as a UI error rather than wedging iPXE.
|
||
- State persists to `<work_dir>/nfs.json`. On startup the manager
|
||
re-attempts every spec; failures are logged per-mount and surfaced in
|
||
the UI rather than blocking startup.
|
||
- Container requirement: `mount.nfs` binary plus `CAP_SYS_ADMIN`. The
|
||
default Dockerfile bundles `nfs-common`. OpenShift operators must
|
||
swap to a more permissive SCC or use a CSI driver.
|
||
|
||
**Live log + operator terminal** (`crates/core/src/log_bus.rs`,
|
||
`crates/http-api/src/{log_stream,terminal}.rs`):
|
||
- A `LogBusLayer` feeds every tracing event into a 500-line ring buffer
|
||
plus a `tokio::sync::broadcast` channel.
|
||
- `/api/log/stream` is an SSE endpoint that emits the recent buffer
|
||
followed by live updates. Slow clients see a `lagged` event rather
|
||
than dropping the stream.
|
||
- `/api/terminal` accepts a single command line and dispatches to a
|
||
whitelist (`status`, `isos`, `clients`, `gate {list,assign,release}`,
|
||
`nfs {list,mount,unmount,scan}`, `smb {status,start,stop,reload}`,
|
||
`log {clear,tail}`). Output is mirrored onto the LogBus so reading the
|
||
live tail tells the same story as scrolling the terminal pane.
|
||
- The whitelist exists deliberately — exposing a raw shell to the web
|
||
would be an RCE endpoint.
|
||
|
||
**Network tab**:
|
||
- `/api/network` exposes auto-detected `nic_name`, `subnet_mask`, and
|
||
`gateway` (parsed from `ip route` / `ip addr` at startup). These are
|
||
read-only by design — silently changing the public IP on a hot UI
|
||
would break PXE for every client mid-boot.
|
||
- The only writable network field is `dns_server`, an optional
|
||
informational hint stored in `Settings`. PXEForge does not run a DNS
|
||
server; the field exists so operators don't have to dig out the
|
||
upstream DNS at 3 AM.
|
||
|
||
**Bootimus parity nicks** (Bootimus v0.1.55 → v0.1.62):
|
||
- Storage table tints amber for ISOs that won't boot with current
|
||
settings (Windows ISO when Windows is disabled, Linux ISO with no
|
||
detected kernel + over the sanboot size threshold). Each row carries
|
||
an inline reason — same affordance as Bootimus's "Image Properties"
|
||
warning.
|
||
- Dashboard surfaces a "Images that won't boot" panel reusing the same
|
||
predicate, so the operator sees the problem before they pick the ISO
|
||
in the gate.
|
||
- Streaming uploads are already in place via axum multipart; the v0.1.62
|
||
fix to "502 on big upload" doesn't apply.
|
||
|
||
## What's deferred to Phase 5
|
||
|
||
- Full ISO9660 + Joliet + Rock Ridge parser (current lookup is plain ISO9660 — Debian ISOs with Rock Ridge extensions may miss some paths).
|
||
- Real-hardware Windows boot validation (plumbing tested; no MS ISO pushed through the full pipeline yet).
|
||
- Real-hardware NFS validation (mount manager tested; no real NAS pushed
|
||
through the full pipeline yet).
|
||
- UEFI HTTP Boot protocol (option 60 = `HTTPClient`) untested on real firmware.
|
||
- Raspberry Pi netboot quirks (option 43 vendor-specific, per-MAC prefixes).
|
||
- Multi-replica / daemonset deployment model (single replica for now).
|
||
- Pure-Rust SMB server (replace smbd) — slim image, no Samba.
|
||
- Auto-install / autounattend file library (Bootimus v0.1.58 pattern).
|
||
- Per-client / per-group menus (Bootimus v0.1.16 pattern).
|