docs: add Linux network-boot runbook

2026-04-30 11:35:47 -04:00
parent 40064906de
commit e1b7154b51
1 changed files with 403 additions and 0 deletions
@@ -0,0 +1,403 @@
+# Runbook: Boot a Linux machine from an ISO over the network
+
+End-to-end walkthrough: spin up PXEForge, load an Ubuntu (or any
+Linux) ISO into it, target a specific bare-metal or VM client by its
+MAC address, and have that machine PXE-boot the installer over the
+LAN — no USB stick, no console babysitting.
+
+This runbook assumes:
+
+- You have **one Linux host** to run the PXEForge container (any
+  distro with Docker / Podman; 2 GB RAM, ~50 GB disk for the ISO
+  library).
+- That host sits on the **same broadcast domain / VLAN** as the
+  client you want to boot. PXE is L2-broadcast — routed/VLAN’d
+  networks need a DHCP relay and are out of scope here.
+- An **existing DHCP server** is already handing out IP leases on
+  that VLAN (your home router, OPNsense, Windows Server, etc.).
+  PXEForge runs as a *DHCP proxy* — it never leases IPs, it only
+  layers the boot information on top of the existing DHCP exchange.
+- The target client is configured to **PXE-boot** in BIOS/UEFI
+  firmware (usually `F12` boot menu → Network, or set as first boot
+  device).
+
+If those don’t hold, stop and read [troubleshooting.md](troubleshooting.md)
+or [docs/architecture.md](../docs/architecture.md) first.
+
+---
+
+## 0. Pick your host’s LAN IP
+
+You need the IPv4 address PXEForge will advertise to clients. From
+the host:
+
+```bash
+ip -4 -o addr show | awk '{print $2, $4}'
+```
+
+Pick the address on the interface that faces the PXE VLAN — for
+example `10.0.0.5/24` on `eno1`. From here on we call it
+`PXE_HOST_IP`.
+
+> **Why this matters.** Every URL handed to clients (TFTP server,
+> iPXE chain URL, ISO URL) is built from this IP. If PXEForge
+> auto-detects the wrong interface or loopback, clients will fetch
+> from an unreachable address and silently fail. The startup will
+> *fail loudly* if it can only auto-detect a loopback address.
+
+---
+
+## 1. Run PXEForge
+
+The MVP path is a single `docker run` against the published image,
+with `--network host` so the container can see DHCP broadcasts on
+the LAN.
+
+```bash
+mkdir -p ~/pxeforge/isos ~/pxeforge/work
+
+docker run -d --name pxeforge \
+  --restart unless-stopped \
+  --network host \
+  -e PXEFORGE_PUBLIC_IP=10.0.0.5 \
+  -e PXEFORGE_DHCP_MODE=proxy \
+  -v ~/pxeforge/isos:/var/lib/pxeforge/isos \
+  -v ~/pxeforge/work:/var/lib/pxeforge/work \
+  ghcr.io/YOUR-ORG/pxeforge:0.2.0
+```
+
+Substitute your `PXEFORGE_PUBLIC_IP`, of course. If you’re building
+from this repo instead of pulling, see the
+[README quick start](../README.md#quick-start--mvp-container-recommended).
+
+### Verify it’s alive
+
+```bash
+curl -fsS http://10.0.0.5/healthz   # → 200 ok
+curl -fsS http://10.0.0.5/readyz    # → 200 ready  (iPXE binaries present)
+curl -fsS http://10.0.0.5/api/status | jq .
+```
+
+If `/readyz` is **not** 200, your container is missing iPXE binaries.
+Fix that before going further — clients have nothing to boot
+otherwise. See [README — Container health probes](../README.md#container-health-probes).
+
+### Check the listening ports
+
+PXEForge holds three privileged UDP/TCP ports. From another shell on
+the host:
+
+```bash
+sudo ss -lnup | grep -E ':(67|69|4011)\b'   # DHCP proxy + TFTP
+sudo ss -lntp | grep ':80\b'                # HTTP UI / boot scripts
+```
+
+All four should be present. If port 67 is taken by `dnsmasq` or the
+host’s own DHCP, stop that service or run PXEForge on a separate box —
+two listeners on `:67` will fight.
+
+---
+
+## 2. Load the ISO
+
+Two options. Pick one.
+
+### 2a. Web UI upload (recommended for one-offs)
+
+1. Open `http://10.0.0.5/` in a browser.
+2. Sidebar → **Storage**.
+3. Click **Upload ISO**, pick e.g. `ubuntu-24.04.1-live-server-amd64.iso`.
+4. Wait for upload + introspection. The row turns into a card showing:
+   - Distro family (`debian_ubuntu`)
+   - Volume label
+   - Detected kernel/initrd paths (`/casper/vmlinuz`, `/casper/initrd`)
+   - File size and SHA-256
+
+Big ISOs stream — there is no 2 GB limit, but expect upload to be
+gated by your browser ↔ host link. The UI shows a progress bar; the
+animated anvil on the Dashboard tab fires up while imaging is in
+flight.
+
+### 2b. Bulk seed from a directory (recommended for fresh deploys / CI)
+
+If you already have a folder of ISOs on the host, skip the browser:
+
+```bash
+# Dry run first — see what would be imported, no writes:
+docker exec pxeforge pxeforge seed \
+  --from /seed \
+  --dry-run
+
+# For real, mount the source dir read-only into the container:
+docker run --rm \
+  -v /my/iso-library:/seed:ro \
+  -v ~/pxeforge/isos:/var/lib/pxeforge/isos \
+  -v ~/pxeforge/work:/var/lib/pxeforge/work \
+  -e PXEFORGE_PUBLIC_IP=10.0.0.5 \
+  ghcr.io/YOUR-ORG/pxeforge:0.2.0 seed --from /seed
+```
+
+Each `*.iso` in `/seed` runs through the same upload pipeline as the
+web UI: copy → introspection → boot-entry generation → metadata
+sidecar. Re-running is idempotent.
+
+### Confirm the ISO is registered
+
+```bash
+curl -fsS http://10.0.0.5/api/isos | jq '.[] | {id, name, family, size}'
+```
+
+You should see something like:
+
+```json
+{
+  "id": "ubuntu-24-04-1-live-server-amd64",
+  "name": "ubuntu-24.04.1-live-server-amd64.iso",
+  "family": "debian_ubuntu",
+  "size": 2748000000
+}
+```
+
+The `id` is the **slug**. Remember it — you’ll bind a MAC to it in
+the next step.
+
+---
+
+## 3. Find the target machine’s MAC address
+
+You need the MAC of the **NIC that will PXE**, not the OS’s
+loopback or wifi.
+
+### 3a. From the target itself (if it’s already running an OS)
+
+```bash
+ip -o link | awk '/ether/ {print $2, $17}'    # Linux
+```
+
+Pick the line for the wired NIC plugged into the PXE VLAN.
+
+### 3b. From the firmware (if it’s a fresh box)
+
+Most BIOS/UEFI screens display the NIC MAC during the network-boot
+attempt — usually as `MAC: AA-BB-CC-DD-EE-FF` flashing on the splash
+right before "PXE-E53: No boot filename received". Write it down.
+
+### 3c. By letting it boot once and watching PXEForge
+
+Easiest if the box is in front of you:
+
+1. Power on, hit `F12`, pick **Network boot**.
+2. Without any binding configured, the client will land on the
+   PXEForge menu (Default / Installers / Tools / Gated Deployment).
+3. Don’t pick anything. On your laptop:
+   ```bash
+   curl -fsS http://10.0.0.5/api/clients | jq .
+   ```
+4. The most-recent entry is your target. Copy its `mac`.
+
+From here on we call this MAC `TARGET_MAC` (e.g. `aa:bb:cc:dd:ee:ff`).
+Hyphens vs colons, upper vs lower case — PXEForge normalizes both.
+
+---
+
+## 4. Pin that machine to the Ubuntu ISO
+
+This is the **per-MAC host binding**. With it set, the client won’t
+see the menu at all — it goes straight to the bound boot entry,
+Tinkerbell-style.
+
+### 4a. Via the web UI
+
+1. Sidebar → **Hosts**.
+2. **Add binding**:
+   - **MAC**: `aa:bb:cc:dd:ee:ff`
+   - **Target**: pick `ubuntu-24-04-1-live-server-amd64` from the dropdown.
+   - **Label**: free-form, e.g. `lab-rack3-node07`.
+3. Save.
+
+### 4b. Via the API
+
+```bash
+curl -fsS -X POST http://10.0.0.5/api/hosts \
+  -H 'content-type: application/json' \
+  -d '{
+    "mac":    "aa:bb:cc:dd:ee:ff",
+    "target": "ubuntu-24-04-1-live-server-amd64",
+    "label":  "lab-rack3-node07"
+  }' | jq .
+```
+
+The binding is persisted to `~/pxeforge/work/hosts.json` and survives
+container restart.
+
+### Confirm
+
+```bash
+curl -fsS http://10.0.0.5/api/hosts | jq '.[] | select(.mac=="aa:bb:cc:dd:ee:ff")'
+```
+
+You should see your entry with `created_at` and `updated_at`
+timestamps.
+
+---
+
+## 5. Trigger the network boot on the target
+
+Now actually boot the machine.
+
+### 5a. Boot order
+
+In firmware setup, set the wired NIC as the **first** boot device
+(or hold `F12` / `F9` / `Esc` — vendor-specific — to pick "Network
+Boot" interactively).
+
+### 5b. What you should see on the target screen
+
+In order, with timing:
+
+| Stage | Approximate duration | What appears |
+|-------|---------------------:|--------------|
+| Firmware DHCPDISCOVER | ~1 s   | `Start PXE over IPv4` / `Station IP address …` |
+| TFTP iPXE binary fetch | ~1 s  | `TFTP… snponly.efi` (or `undionly.kpxe` for legacy BIOS) |
+| iPXE banner            | ~1 s  | The blue iPXE splash, version string |
+| iPXE second-stage DHCP | ~1 s  | `Configuring (net0 …)` then `ok` |
+| HTTP boot script fetch | <1 s  | `http://10.0.0.5/boot.ipxe?mac=…` |
+| Per-MAC chain          | <1 s  | `PXEForge: per-MAC binding -> ubuntu-24-04-1-…` |
+| Kernel + initrd HTTP   | 5–30 s | Two 200-OK fetches against `/iso/<id>/casper/vmlinuz` and `…/initrd` |
+| Kernel boot            | 5–10 s | Kernel banner, then the Ubuntu/cloud-init splash |
+| Installer comes up     | 30–60 s | The distro’s normal Live/installer environment |
+
+If everything works, you’re looking at the Ubuntu Server installer
+welcome screen end-to-end **without ever touching a USB stick**.
+
+### 5c. Watch it from the server
+
+In a third shell, tail the live log:
+
+```bash
+curl -N http://10.0.0.5/api/log/stream
+```
+
+You’ll see each protocol step as it happens:
+
+```
+INFO pxeforge::dhcp: reply mac=aa:bb:cc:dd:ee:ff arch=X8664Uefi target=tftp/snponly.efi
+INFO pxeforge::tftp: RRQ snponly.efi blksize=1468 windowsize=8 → 982 KiB in 412 ms
+INFO pxeforge::dhcp: reply mac=aa:bb:cc:dd:ee:ff (iPXE) target=http/boot.ipxe
+INFO pxeforge::http: GET /boot.ipxe?mac=aa:bb:cc:dd:ee:ff → host binding hit
+INFO pxeforge::http: GET /iso/ubuntu-…/casper/vmlinuz Range=bytes=0- 200 OK 14 MiB
+INFO pxeforge::http: GET /iso/ubuntu-…/casper/initrd  Range=bytes=0- 200 OK 75 MiB
+```
+
+The **Terminal** tab in the web UI shows the same thing live, plus a
+short whitelisted command palette (`status`, `clients`, `gate`,
+`hosts`, `log`).
+
+### 5d. Internet-side ISO sources
+
+The runbook title says “via the internet” — the **client** itself
+boots from your LAN, but the underlying ISO can come from anywhere
+your *host* can reach:
+
+- **Direct upload** from a remote workstation via the web UI (HTTPS
+  reverse-proxied if you put PXEForge behind nginx/Caddy).
+- **NFS mount** of a remote share — Sidebar → **Storage** → **NFS** →
+  `nfs://files.lab.example.com/exports/isos`. Mounted ISOs show up in
+  the same list and are PXE-bootable directly without copying.
+- **Pre-seed** from a CI job that `curl`s a vendor mirror and runs
+  `pxeforge seed --from`.
+
+PXEForge itself never reaches out to the internet at boot time — all
+client traffic stays on the LAN, served from the host.
+
+---
+
+## 6. After the install
+
+Once Ubuntu has finished installing to the target’s disk, you want
+the next reboot to come up off the new local disk, **not** PXE
+again. Two ways:
+
+### 6a. One-shot — release the binding
+
+```bash
+curl -fsS -X DELETE http://10.0.0.5/api/hosts/aa:bb:cc:dd:ee:ff
+```
+
+Without a binding, the client either gets the menu (BIOS still set
+to PXE first) or boots local disk normally.
+
+### 6b. Permanent — pin to local disk
+
+Re-bind to the reserved local-boot target:
+
+```bash
+curl -fsS -X POST http://10.0.0.5/api/hosts \
+  -H 'content-type: application/json' \
+  -d '{ "mac": "aa:bb:cc:dd:ee:ff", "target": "_local", "label": "lab-rack3-node07 (installed)" }'
+```
+
+Now if anyone hits `F12 → Network` by accident, PXEForge replies
+with a script that says *"chain back to local HDD"* and the box
+boots its real OS instead of re-imaging itself. This is the safest
+default for production hardware.
+
+---
+
+## 7. Re-imaging — the “Gated Deployment” flow
+
+Different scenario: you have **a rack of 30 servers** to image
+identically, all at once. Don’t bind 30 MACs by hand. Use the gate.
+
+1. **Don’t** create host bindings.
+2. PXE-boot every machine. They land on the menu.
+3. On each: select **Gated Deployment**. They get position #1, #2,
+   …, #30 and start long-polling.
+4. In the UI: **Forge Gate** tab shows all 30 lined up. Pick the
+   ISO, click **Assign to all waiting**.
+5. Every client’s open long-poll wakes up at the same instant and
+   chains the same boot script. They all start imaging
+   simultaneously — the “horse race gate” opens.
+
+The animated anvil widget on the Dashboard runs while any client is
+still in the kernel-fetch phase.
+
+---
+
+## Cheat sheet
+
+| Goal | Command |
+|------|---------|
+| Health check | `curl http://$IP/healthz` |
+| List ISOs | `curl http://$IP/api/isos \| jq .` |
+| List clients seen | `curl http://$IP/api/clients \| jq .` |
+| Bind MAC → ISO | `POST /api/hosts` with `{mac,target,label}` |
+| Bind MAC → local disk | same with `target=_local` |
+| Release binding | `DELETE /api/hosts/<mac>` |
+| Live log | `curl -N http://$IP/api/log/stream` |
+| Prometheus metrics | `curl http://$IP/metrics` |
+| Bulk import folder | `pxeforge seed --from /path` |
+
+---
+
+## Where to look when things break
+
+- **Client gets `PXE-E53: No boot filename received`** — DHCP proxy
+  isn’t replying. Check `:67` is bound (`ss -lnup`), check
+  `--network host`, check the host firewall on UDP 67/69/4011.
+- **iPXE shows `No more network devices`** — firmware NIC isn’t in
+  PXE mode, or VLAN tagging is wrong.
+- **iPXE prints `Connection timed out (http://…)`** — `PXEFORGE_PUBLIC_IP`
+  is wrong. Clients can’t reach that IP. Check `/api/status` →
+  `public_base_url` and `ping` it from the client subnet.
+- **Kernel panics during initrd load** — corrupt ISO upload. Check
+  `/api/isos`, compare the SHA-256 to the vendor’s, re-upload.
+- **Boot menu shows but the bound entry doesn’t fire** — the binding
+  target slug doesn’t match any ISO `id`. Recheck
+  `GET /api/hosts` against `GET /api/isos`. The binding falls back
+  to the menu on miss (by design — never lock a client out).
+- **General confusion** — Terminal tab → `status`, then `log`. That
+  tells you what protocol stages have run and which haven’t.
+
+For deeper protocol-level debugging, see
+[docs/architecture.md](../docs/architecture.md).