Initial commit: PXEForge Phases 1-4
Container-native PXE boot server in Rust, designed as a clean-room alternative to iVentoy that never touches the client OS trust store. This is the first commit of the project; it lands the full output of Phases 1, 2, 3, and 4 in one shot. ## Phase 1 — protocol stack - 8-crate workspace (core, dhcp-proxy, tftp, http-api, iso-store, ipxe-assets, webui, pxeforge bin). - DHCP proxy (RFC 4578): replies with boot info only, never leases — sidesteps CAP_NET_RAW. Architecture-aware bootfile selection from option 93 (BIOS, IA32, x64-UEFI alias 0x0007/0x0009, ARM64). - TFTP server with full OACK negotiation: blksize, tsize, windowsize. Without it a 1 MiB iPXE binary takes 2000 packets and unusably long. - Two-stage iPXE chain: firmware PXE -> TFTP iPXE binary -> iPXE re-DHCPs with user-class iPXE -> HTTP /boot.ipxe -> kernel+initrd. - HTTP server (axum) with byte-Range ISO streaming and an in-place ISO9660 lookup so kernel/initrd are served from inside the ISO without ever extracting it to disk. - Linux ISOs boot via kernel+initrd extraction (memdisk/sanboot fail for >1-2 GiB modern distros). Distro-family detection drives the cmdline (Debian/Ubuntu, RHEL/Fedora, openSUSE, Arch, Alpine). ## Phase 2 — UX + Windows - Hierarchical PXE menu (Default / Installers / Tools / Gated Deployment) generated from settings — no hand-written .ipxe paths surface in the UI. Number-key + letter hotkeys, BIOS+UEFI variants for some RHEL ISOs. - Gated Deployment "horse-race" queue: clients join, operator picks one ISO, every gate launches simultaneously via tokio::sync::Notify. - Bootimus-pattern Windows: WimPatcher injects a CRLF startnet.cmd into boot.wim so vanilla WinPE net-uses an SMB share and runs setup.exe. All Microsoft-signed; no test certs, no testsigning, no httpdisk.sys. SmbManager supervises smbd start/stop/SIGHUP. - Netbox-style dark UI, fully offline (no CDN, no external fonts). ## Phase 3 — MVP hardening - TFTP retransmit rewrite with explicit window tracking — UEFI SNP clients no longer hang on files that end mid-window. 4 new tests. - DHCP broadcast-flag honored per RFC 2131 §4.1. - Multi-arch container (linux/amd64 + linux/arm64). Entrypoint chowns bind-mounts as root then drops to uid 10001 via gosu. - /healthz + /readyz split from /api/status — readyz fails if no iPXE binaries are bundled. - pxeforge seed --from <path> CLI: same pipeline as web upload (slug, sha256, introspection, boot-entry). - All timestamps RFC 3339 (browser Date couldn't parse the 9-tuple). - Gate poll retains assignment until operator releases — clients that retry on transient network errors reuse the assignment instead of falling back to the menu. - Custom OpenShift SCC: hostNetwork + NET_BIND_SERVICE only, no NET_RAW. ## Phase 4 — UI restructure + remote storage - Web UI rebuilt around six tabs inspired by the iVentoy layout: Dashboard / Network / Forge Gate / Storage / Terminal / About. Old "Monitoring/Content/Configuration" sidebar groups are gone. - NFS share manager (crates/iso-store/src/nfs.rs): mount NFSv3 or NFSv4.1 shares as ISO sources instead of uploading every file into the PVC. New IsoSource enum on IsoMeta lets the store resolve Local vs NFS lazily. Persisted to <work_dir>/nfs.json; failed mounts surface in the UI rather than blocking startup. - Dockerfile gains nfs-common + iproute2; mounting NFS in-container also requires CAP_SYS_ADMIN. Documented in docs/architecture.md. - LogBus + tracing layer in core: 500-line ring buffer + broadcast channel feed an SSE endpoint at /api/log/stream. - Operator terminal at /api/terminal: whitelisted commands (status, isos, clients, gate, nfs, smb, log) — deliberately not a shell. Output mirrored onto the LogBus so the live tail and the terminal pane share one timeline. - Network tab: read-only nic_name / subnet_mask / gateway probed from `ip` at startup; only DNS server is editable. Editing IP/mask on a hot UI would silently break PXE for every client mid-boot. - Bootimus parity (releases v0.1.55 -> v0.1.62): amber row tint on un-bootable ISOs with inline reasons, dashboard "won't boot" panel. ## Tests 56 tests passing across the workspace: - 16 core (LogBus, gate, settings, arch, client) - 1 dhcp-proxy (raw option-93 extraction) - 8 http-api unit (range parsing, terminal split/format) - 13 http-api integration (gated deployment, range, settings, NFS, terminal, log SSE, network endpoint, ui assets, no-external-urls) - 12 iso-store (introspect, slugify, smb, windows wim, NFS options) - 6 tftp (RRQ parsing, plan_window edges) cargo build --workspace and cargo clippy --workspace --all-targets both finish clean (warnings only, no errors).
This commit is contained in:
Miles Ward
@@ -0,0 +1,270 @@
|
||||
# PXEForge architecture
|
||||
|
||||
## Protocol stack
|
||||
|
||||
```
|
||||
Client firmware PXE ROM
|
||||
│
|
||||
│ DHCPDISCOVER (UDP/67 broadcast, option 60 "PXEClient", option 93 arch)
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ PXEForge │
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │
|
||||
│ │ DHCP proxy │ │ TFTP server │ │ HTTP server (axum) │ │
|
||||
│ │ :67, :4011 │ │ :69 │ │ :80 │ │
|
||||
│ │ (dhcproto) │ │ (custom) │ │ │ │
|
||||
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────────────┘ │
|
||||
│ │ │ │ │
|
||||
│ └────────────────────┼─────────────────────┘ │
|
||||
│ │ │
|
||||
│ ┌──────────▼────────────┐ │
|
||||
│ │ IsoStore (on disk) │ │
|
||||
│ │ + ClientRegistry │ │
|
||||
│ └───────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼ reply with option 60, option 66 tftp-server, option 67 bootfile
|
||||
Client firmware PXE ROM
|
||||
│
|
||||
│ TFTP RRQ: snponly.efi (or undionly.kpxe for BIOS)
|
||||
▼
|
||||
Client runs iPXE
|
||||
│
|
||||
│ DHCPDISCOVER with option 77 "iPXE"
|
||||
▼
|
||||
PXEForge sees user-class "iPXE" → replies with HTTP URL: /boot.ipxe
|
||||
│
|
||||
│ HTTP GET /boot.ipxe (iPXE menu, auto-generated from IsoStore)
|
||||
▼
|
||||
User picks entry; iPXE chains /boot/<id>.ipxe
|
||||
│
|
||||
│ HTTP GET kernel + initrd (or wimboot + WIM files)
|
||||
▼
|
||||
Kernel boots with distro-specific args pointing back at /iso/<id>.iso
|
||||
```
|
||||
|
||||
## Crate layout
|
||||
|
||||
| Crate | Responsibility |
|
||||
|---------------------|-------------------------------------------------------------------|
|
||||
| `pxeforge-core` | Shared types: `Config`, `ClientArch`, `FirmwareClass`, `ClientRegistry` |
|
||||
| `pxeforge-ipxe-assets` | Embeds bundled iPXE binaries via `rust-embed` |
|
||||
| `pxeforge-iso-store` | On-disk ISO store, introspection, boot-entry generation |
|
||||
| `pxeforge-dhcp-proxy` | UDP listener + `dhcproto` reply builder; pure `decide()` unit-testable |
|
||||
| `pxeforge-tftp` | RFC 1350 + OACK (blksize / tsize / windowsize). Serves only embedded assets — no filesystem |
|
||||
| `pxeforge-http-api` | `axum` router: web UI, API, iPXE script generation, ISO streaming |
|
||||
| `pxeforge-webui` | Single `index.html` served as static string |
|
||||
| `pxeforge` (bin) | Wires everything together, runs the three servers concurrently |
|
||||
|
||||
## Key decisions and why
|
||||
|
||||
### DHCP proxy only, not a full DHCP server
|
||||
|
||||
Proxy mode (RFC 4578) replies with boot parameters (`siaddr`, option 66, 67)
|
||||
but never sets `yiaddr` — it does not lease IPs. The client merges proxy
|
||||
replies with its normal DHCP lease from the network's existing DHCP server.
|
||||
|
||||
This sidesteps needing `CAP_NET_RAW` or AF_PACKET. A full DHCP server has
|
||||
to craft Ethernet frames to a client that doesn't yet have an IP — that
|
||||
requires raw sockets and elevated privileges. A proxy replies on UDP to a
|
||||
client that already has an IP (or will shortly, from the other DHCP server),
|
||||
so plain `SOCK_DGRAM` is enough.
|
||||
|
||||
### Two-stage iPXE chain
|
||||
|
||||
1. Firmware PXE ROM sends DHCPDISCOVER with option 60 = `PXEClient`,
|
||||
option 93 = arch.
|
||||
2. PXEForge replies with TFTP server + arch-specific iPXE binary
|
||||
(`undionly.kpxe` for Legacy BIOS, `snponly.efi` for x86_64 UEFI, etc.).
|
||||
3. Client TFTPs the iPXE binary and runs it.
|
||||
4. iPXE does its own DHCP, setting option 77 (user-class) to `iPXE`.
|
||||
5. PXEForge detects the user-class and this time replies with an HTTP URL
|
||||
in option 67 pointing at `/boot.ipxe`.
|
||||
6. iPXE fetches and executes that script, which chains the selected OS.
|
||||
|
||||
The split is essential because firmware PXE ROMs only speak TFTP; they
|
||||
don't do HTTP. iPXE adds HTTP (and a lot more), and is small enough to fit
|
||||
in the TFTP hop.
|
||||
|
||||
### TFTP option negotiation is mandatory
|
||||
|
||||
Default 512-byte blocks means an ~1 MiB iPXE binary is ~2000 packets.
|
||||
On anything less than pristine wired Ethernet this is unusably slow or
|
||||
outright fails. We negotiate:
|
||||
|
||||
- `blksize` (RFC 2348): up to ~1468 bytes for Ethernet MTU
|
||||
- `tsize` (RFC 2349): file size; some PXE ROMs require it present
|
||||
- `windowsize` (RFC 7440): 8–16 gives order-of-magnitude throughput gains
|
||||
|
||||
### Do not touch the client OS trust store
|
||||
|
||||
Whatever we do for Windows, we never:
|
||||
|
||||
- ship drivers signed with test/development certificates
|
||||
- instruct users to enable `bcdedit /set testsigning on`
|
||||
- install any certificate into the target's root/trust store
|
||||
|
||||
iVentoy's `httpdisk.sys` approach broke this rule. PXEForge doesn't.
|
||||
|
||||
### Linux ISO boot uses kernel+initrd extraction, not sanboot
|
||||
|
||||
Loading a full ISO into RAM via `memdisk` or `sanboot` fails:
|
||||
- Above ~1–2 GiB, RAM emulation is too slow or too large.
|
||||
- Most modern distros can't find the emulated CD device from the initramfs.
|
||||
|
||||
We instead extract `vmlinuz` + `initrd` at upload time (well, look up their
|
||||
locations — we serve them via ISO9660 byte-range lookup, no extraction on
|
||||
disk) and pass distro-specific kernel args that point the installer back at
|
||||
the HTTP-served ISO.
|
||||
|
||||
### No iPXE UX
|
||||
|
||||
iPXE is an implementation detail. The web UI only accepts `.iso` uploads
|
||||
and shows distros, not `.ipxe` scripts or boot targets. If someone who
|
||||
knows iPXE wants to peek, they can `curl /boot.ipxe` — that's fine. But
|
||||
the UI never surfaces it.
|
||||
|
||||
## Unit tests
|
||||
|
||||
Run with `cargo test --workspace --lib`. Current coverage:
|
||||
|
||||
- `ClientArch` alias handling (0x0009 → x86_64 UEFI)
|
||||
- `FirmwareClass` classification (iPXE wins over PXEClient echo)
|
||||
- Raw option-93 extraction from the wire
|
||||
- TFTP RRQ parsing with options
|
||||
- HTTP Range header parsing (full, open-ended, suffix, explicit)
|
||||
- Distro family detection from volume label
|
||||
- ISO id slugification
|
||||
|
||||
## Phase 3 hardening
|
||||
|
||||
A code-review pass after Phase 2 turned up 16 issues across the protocol
|
||||
stack and container surface. Everything P0/P1/P2 is addressed in this
|
||||
release:
|
||||
|
||||
**Protocol reliability** (review P1 #10/#12):
|
||||
- TFTP retransmit loop rewritten with explicit window tracking so UEFI SNP
|
||||
clients no longer hang on files that end mid-window. Covered by 4 new
|
||||
unit tests (`plan_window_*` in `crates/tftp`).
|
||||
- DHCP proxy now honors the broadcast-flag bit (RFC 2131 §4.1) when
|
||||
choosing unicast vs broadcast reply destination.
|
||||
|
||||
**Container posture** (review P1 #2/#13/#14/#15):
|
||||
- Multi-arch container (linux/amd64 + linux/arm64) via `buildx`.
|
||||
- New entrypoint (`deploy/docker/entrypoint.sh`) chowns data dirs as root,
|
||||
then drops to uid 10001 via `gosu` — fixes the "bind-mount comes up
|
||||
root-owned" problem that breaks ISO upload on standard Docker hosts.
|
||||
- `/healthz` and `/readyz` split from `/api/status` — readyz fails if no
|
||||
iPXE binaries are bundled, giving K8s probes a real signal.
|
||||
- Startup aborts with a clear error if `PXEFORGE_PUBLIC_IP` can't be
|
||||
auto-detected (no more silent `127.0.0.1` advertisement).
|
||||
- `scripts/fetch-ipxe.sh` fails non-zero if zero binaries download; the
|
||||
Dockerfile uses arch-scoped paths (`x86_64-efi/snponly.efi` etc.).
|
||||
|
||||
**Windows boot plumbing** (new):
|
||||
- `pxeforge-iso-store::smb::SmbManager` supervises `smbd` on the Windows
|
||||
toggle: `start` → spawn + write `smb.conf`; `reconcile` → SIGHUP on
|
||||
share changes; `stop` → SIGTERM. `SmbState` surfaced to the UI for
|
||||
visibility.
|
||||
- `extract_windows_iso` shells out to `7z` (or `bsdtar` fallback) to
|
||||
unpack the ISO tree into `smb_dir/<slug>/` as the SMB share root.
|
||||
- `WimPatcher` (from Phase 2) is still in place for `boot.wim` injection.
|
||||
- Guardrail: flipping the Windows toggle in `/api/settings` is rejected
|
||||
with a 400 if `wimboot` isn't bundled.
|
||||
|
||||
**iVentoy/Bootimus parity** (P2):
|
||||
- ISO sizes in menu labels (`[ 4376 MB]`), iVentoy format.
|
||||
- `Reboot Computer` + `Exit and continue BIOS boot` in Tools menu.
|
||||
- Number-key hotkeys (1..9) on boot entries, letter hotkeys on tools.
|
||||
- Clients tab cross-joins the gate queue so an operator sees "at gate #2"
|
||||
or "assigned: ubuntu-linux" status inline.
|
||||
|
||||
**Developer ergonomics** (new):
|
||||
- `pxeforge seed --from <path>` CLI to import ISOs from a directory.
|
||||
Same pipeline as web upload (slug, sha256, introspection, boot-entry).
|
||||
- `docker-compose.yml` with `pxeforge` (host network, real PXE) and
|
||||
`pxeforge-dev` (published ports, DHCP disabled, for API testing).
|
||||
|
||||
**API cleanliness**:
|
||||
- All timestamps now serialized as RFC 3339 strings (the `time` crate's
|
||||
default 9-tuple broke browser `Date` parsing).
|
||||
- Gate poll retains assignment until the operator releases it; if the
|
||||
client's chain fails, it reuses the assignment instead of falling back
|
||||
to the menu.
|
||||
|
||||
## Phase 4 — UI restructure + remote storage
|
||||
|
||||
The web UI was rebuilt around six tabs (Dashboard / Network / Forge Gate /
|
||||
Storage / Terminal / About) inspired by the iVentoy layout the user
|
||||
attached and Netbox Labs's compact-card pattern. The old hierarchical
|
||||
"Monitoring / Content / Configuration" sidebar grouping is gone — every
|
||||
tab is one click from the brand bar.
|
||||
|
||||
**NFS share manager** (`crates/iso-store/src/nfs.rs`):
|
||||
- Operators add a remote share via Storage → NFS shares; PXEForge mounts
|
||||
it under `<work_dir>/nfs/<id>/` and walks it for `*.iso` files.
|
||||
- Each ISO found is registered with `IsoStore::register_external` using
|
||||
a new `IsoSource::Nfs { mount_id, relative_path }` variant. The store
|
||||
resolves these to disk lazily via `set_nfs_root`, so adding NFS
|
||||
required exactly one new field on `IsoMeta` (with a serde default for
|
||||
forward compatibility with old `meta.json` files).
|
||||
- Versions: `vers=3` and `vers=4.1` only. v3 also gets `nolock` since
|
||||
appliances commonly disable `lockd`. All mounts use `soft,timeo=100`
|
||||
so a dead server surfaces as a UI error rather than wedging iPXE.
|
||||
- State persists to `<work_dir>/nfs.json`. On startup the manager
|
||||
re-attempts every spec; failures are logged per-mount and surfaced in
|
||||
the UI rather than blocking startup.
|
||||
- Container requirement: `mount.nfs` binary plus `CAP_SYS_ADMIN`. The
|
||||
default Dockerfile bundles `nfs-common`. OpenShift operators must
|
||||
swap to a more permissive SCC or use a CSI driver.
|
||||
|
||||
**Live log + operator terminal** (`crates/core/src/log_bus.rs`,
|
||||
`crates/http-api/src/{log_stream,terminal}.rs`):
|
||||
- A `LogBusLayer` feeds every tracing event into a 500-line ring buffer
|
||||
plus a `tokio::sync::broadcast` channel.
|
||||
- `/api/log/stream` is an SSE endpoint that emits the recent buffer
|
||||
followed by live updates. Slow clients see a `lagged` event rather
|
||||
than dropping the stream.
|
||||
- `/api/terminal` accepts a single command line and dispatches to a
|
||||
whitelist (`status`, `isos`, `clients`, `gate {list,assign,release}`,
|
||||
`nfs {list,mount,unmount,scan}`, `smb {status,start,stop,reload}`,
|
||||
`log {clear,tail}`). Output is mirrored onto the LogBus so reading the
|
||||
live tail tells the same story as scrolling the terminal pane.
|
||||
- The whitelist exists deliberately — exposing a raw shell to the web
|
||||
would be an RCE endpoint.
|
||||
|
||||
**Network tab**:
|
||||
- `/api/network` exposes auto-detected `nic_name`, `subnet_mask`, and
|
||||
`gateway` (parsed from `ip route` / `ip addr` at startup). These are
|
||||
read-only by design — silently changing the public IP on a hot UI
|
||||
would break PXE for every client mid-boot.
|
||||
- The only writable network field is `dns_server`, an optional
|
||||
informational hint stored in `Settings`. PXEForge does not run a DNS
|
||||
server; the field exists so operators don't have to dig out the
|
||||
upstream DNS at 3 AM.
|
||||
|
||||
**Bootimus parity nicks** (Bootimus v0.1.55 → v0.1.62):
|
||||
- Storage table tints amber for ISOs that won't boot with current
|
||||
settings (Windows ISO when Windows is disabled, Linux ISO with no
|
||||
detected kernel + over the sanboot size threshold). Each row carries
|
||||
an inline reason — same affordance as Bootimus's "Image Properties"
|
||||
warning.
|
||||
- Dashboard surfaces a "Images that won't boot" panel reusing the same
|
||||
predicate, so the operator sees the problem before they pick the ISO
|
||||
in the gate.
|
||||
- Streaming uploads are already in place via axum multipart; the v0.1.62
|
||||
fix to "502 on big upload" doesn't apply.
|
||||
|
||||
## What's deferred to Phase 5
|
||||
|
||||
- Full ISO9660 + Joliet + Rock Ridge parser (current lookup is plain ISO9660 — Debian ISOs with Rock Ridge extensions may miss some paths).
|
||||
- Real-hardware Windows boot validation (plumbing tested; no MS ISO pushed through the full pipeline yet).
|
||||
- Real-hardware NFS validation (mount manager tested; no real NAS pushed
|
||||
through the full pipeline yet).
|
||||
- UEFI HTTP Boot protocol (option 60 = `HTTPClient`) untested on real firmware.
|
||||
- Raspberry Pi netboot quirks (option 43 vendor-specific, per-MAC prefixes).
|
||||
- Multi-replica / daemonset deployment model (single replica for now).
|
||||
- Pure-Rust SMB server (replace smbd) — slim image, no Samba.
|
||||
- Auto-install / autounattend file library (Bootimus v0.1.58 pattern).
|
||||
- Per-client / per-group menus (Bootimus v0.1.16 pattern).
|
||||
Reference in New Issue
Block a user