docs: add Linux network-boot runbook

2026-04-30 11:35:47 -04:00
parent 40064906de
commit e1b7154b51
1 changed files with 403 additions and 0 deletions
@@ -0,0 +1,403 @@
 # Runbook: Boot a Linux machine from an ISO over the network
 End-to-end walkthrough: spin up PXEForge, load an Ubuntu (or any
 Linux) ISO into it, target a specific bare-metal or VM client by its
 MAC address, and have that machine PXE-boot the installer over the
 LAN — no USB stick, no console babysitting.
 This runbook assumes:
 - You have **one Linux host** to run the PXEForge container (any
  distro with Docker / Podman; 2 GB RAM, ~50 GB disk for the ISO
  library).
 - That host sits on the **same broadcast domain / VLAN** as the
  client you want to boot. PXE is L2-broadcast — routed/VLAN’d
  networks need a DHCP relay and are out of scope here.
 - An **existing DHCP server** is already handing out IP leases on
  that VLAN (your home router, OPNsense, Windows Server, etc.).
  PXEForge runs as a *DHCP proxy* — it never leases IPs, it only
  layers the boot information on top of the existing DHCP exchange.
 - The target client is configured to **PXE-boot** in BIOS/UEFI
  firmware (usually `F12` boot menu → Network, or set as first boot
  device).
 If those don’t hold, stop and read [troubleshooting.md](troubleshooting.md)
 or [docs/architecture.md](../docs/architecture.md) first.
 ---
 ## 0. Pick your host’s LAN IP
 You need the IPv4 address PXEForge will advertise to clients. From
 the host:
 ```bash
 ip -4 -o addr show | awk '{print $2, $4}'
 ```
 Pick the address on the interface that faces the PXE VLAN — for
 example `10.0.0.5/24` on `eno1`. From here on we call it
 `PXE_HOST_IP`.
 > **Why this matters.** Every URL handed to clients (TFTP server,
 > iPXE chain URL, ISO URL) is built from this IP. If PXEForge
 > auto-detects the wrong interface or loopback, clients will fetch
 > from an unreachable address and silently fail. The startup will
 > *fail loudly* if it can only auto-detect a loopback address.
 ---
 ## 1. Run PXEForge
 The MVP path is a single `docker run` against the published image,
 with `--network host` so the container can see DHCP broadcasts on
 the LAN.
 ```bash
 mkdir -p ~/pxeforge/isos ~/pxeforge/work
 docker run -d --name pxeforge \
  --restart unless-stopped \
  --network host \
  -e PXEFORGE_PUBLIC_IP=10.0.0.5 \
  -e PXEFORGE_DHCP_MODE=proxy \
  -v ~/pxeforge/isos:/var/lib/pxeforge/isos \
  -v ~/pxeforge/work:/var/lib/pxeforge/work \
  ghcr.io/YOUR-ORG/pxeforge:0.2.0
 ```
 Substitute your `PXEFORGE_PUBLIC_IP`, of course. If you’re building
 from this repo instead of pulling, see the
 [README quick start](../README.md#quick-start--mvp-container-recommended).
 ### Verify it’s alive
 ```bash
 curl -fsS http://10.0.0.5/healthz   # → 200 ok
 curl -fsS http://10.0.0.5/readyz    # → 200 ready  (iPXE binaries present)
 curl -fsS http://10.0.0.5/api/status | jq .
 ```
 If `/readyz` is **not** 200, your container is missing iPXE binaries.
 Fix that before going further — clients have nothing to boot
 otherwise. See [README — Container health probes](../README.md#container-health-probes).
 ### Check the listening ports
 PXEForge holds three privileged UDP/TCP ports. From another shell on
 the host:
 ```bash
 sudo ss -lnup | grep -E ':(67|69|4011)\b'   # DHCP proxy + TFTP
 sudo ss -lntp | grep ':80\b'                # HTTP UI / boot scripts
 ```
 All four should be present. If port 67 is taken by `dnsmasq` or the
 host’s own DHCP, stop that service or run PXEForge on a separate box —
 two listeners on `:67` will fight.
 ---
 ## 2. Load the ISO
 Two options. Pick one.
 ### 2a. Web UI upload (recommended for one-offs)
 1. Open `http://10.0.0.5/` in a browser.
 2. Sidebar → **Storage**.
 3. Click **Upload ISO**, pick e.g. `ubuntu-24.04.1-live-server-amd64.iso`.
 4. Wait for upload + introspection. The row turns into a card showing:
   - Distro family (`debian_ubuntu`)
   - Volume label
   - Detected kernel/initrd paths (`/casper/vmlinuz`, `/casper/initrd`)
   - File size and SHA-256
 Big ISOs stream — there is no 2 GB limit, but expect upload to be
 gated by your browser ↔ host link. The UI shows a progress bar; the
 animated anvil on the Dashboard tab fires up while imaging is in
 flight.
 ### 2b. Bulk seed from a directory (recommended for fresh deploys / CI)
 If you already have a folder of ISOs on the host, skip the browser:
 ```bash
 # Dry run first — see what would be imported, no writes:
 docker exec pxeforge pxeforge seed \
  --from /seed \
  --dry-run
 # For real, mount the source dir read-only into the container:
 docker run --rm \
  -v /my/iso-library:/seed:ro \
  -v ~/pxeforge/isos:/var/lib/pxeforge/isos \
  -v ~/pxeforge/work:/var/lib/pxeforge/work \
  -e PXEFORGE_PUBLIC_IP=10.0.0.5 \
  ghcr.io/YOUR-ORG/pxeforge:0.2.0 seed --from /seed
 ```
 Each `*.iso` in `/seed` runs through the same upload pipeline as the
 web UI: copy → introspection → boot-entry generation → metadata
 sidecar. Re-running is idempotent.
 ### Confirm the ISO is registered
 ```bash
 curl -fsS http://10.0.0.5/api/isos | jq '.[] | {id, name, family, size}'
 ```
 You should see something like:
 ```json
 {
  "id": "ubuntu-24-04-1-live-server-amd64",
  "name": "ubuntu-24.04.1-live-server-amd64.iso",
  "family": "debian_ubuntu",
  "size": 2748000000
 }
 ```
 The `id` is the **slug**. Remember it — you’ll bind a MAC to it in
 the next step.
 ---
 ## 3. Find the target machine’s MAC address
 You need the MAC of the **NIC that will PXE**, not the OS’s
 loopback or wifi.
 ### 3a. From the target itself (if it’s already running an OS)
 ```bash
 ip -o link | awk '/ether/ {print $2, $17}'    # Linux
 ```
 Pick the line for the wired NIC plugged into the PXE VLAN.
 ### 3b. From the firmware (if it’s a fresh box)
 Most BIOS/UEFI screens display the NIC MAC during the network-boot
 attempt — usually as `MAC: AA-BB-CC-DD-EE-FF` flashing on the splash
 right before "PXE-E53: No boot filename received". Write it down.
 ### 3c. By letting it boot once and watching PXEForge
 Easiest if the box is in front of you:
 1. Power on, hit `F12`, pick **Network boot**.
 2. Without any binding configured, the client will land on the
   PXEForge menu (Default / Installers / Tools / Gated Deployment).
 3. Don’t pick anything. On your laptop:
   ```bash
   curl -fsS http://10.0.0.5/api/clients | jq .
   ```
 4. The most-recent entry is your target. Copy its `mac`.
 From here on we call this MAC `TARGET_MAC` (e.g. `aa:bb:cc:dd:ee:ff`).
 Hyphens vs colons, upper vs lower case — PXEForge normalizes both.
 ---
 ## 4. Pin that machine to the Ubuntu ISO
 This is the **per-MAC host binding**. With it set, the client won’t
 see the menu at all — it goes straight to the bound boot entry,
 Tinkerbell-style.
 ### 4a. Via the web UI
 1. Sidebar → **Hosts**.
 2. **Add binding**:
   - **MAC**: `aa:bb:cc:dd:ee:ff`
   - **Target**: pick `ubuntu-24-04-1-live-server-amd64` from the dropdown.
   - **Label**: free-form, e.g. `lab-rack3-node07`.
 3. Save.
 ### 4b. Via the API
 ```bash
 curl -fsS -X POST http://10.0.0.5/api/hosts \
  -H 'content-type: application/json' \
  -d '{
    "mac":    "aa:bb:cc:dd:ee:ff",
    "target": "ubuntu-24-04-1-live-server-amd64",
    "label":  "lab-rack3-node07"
  }' | jq .
 ```
 The binding is persisted to `~/pxeforge/work/hosts.json` and survives
 container restart.
 ### Confirm
 ```bash
 curl -fsS http://10.0.0.5/api/hosts | jq '.[] | select(.mac=="aa:bb:cc:dd:ee:ff")'
 ```
 You should see your entry with `created_at` and `updated_at`
 timestamps.
 ---
 ## 5. Trigger the network boot on the target
 Now actually boot the machine.
 ### 5a. Boot order
 In firmware setup, set the wired NIC as the **first** boot device
 (or hold `F12` / `F9` / `Esc` — vendor-specific — to pick "Network
 Boot" interactively).
 ### 5b. What you should see on the target screen
 In order, with timing:
 | Stage | Approximate duration | What appears |
 |-------|---------------------:|--------------|
 | Firmware DHCPDISCOVER | ~1 s   | `Start PXE over IPv4` / `Station IP address …` |
 | TFTP iPXE binary fetch | ~1 s  | `TFTP… snponly.efi` (or `undionly.kpxe` for legacy BIOS) |
 | iPXE banner            | ~1 s  | The blue iPXE splash, version string |
 | iPXE second-stage DHCP | ~1 s  | `Configuring (net0 …)` then `ok` |
 | HTTP boot script fetch | <1 s  | `http://10.0.0.5/boot.ipxe?mac=…` |
 | Per-MAC chain          | <1 s  | `PXEForge: per-MAC binding -> ubuntu-24-04-1-…` |
 | Kernel + initrd HTTP   | 5–30 s | Two 200-OK fetches against `/iso/<id>/casper/vmlinuz` and `…/initrd` |
 | Kernel boot            | 5–10 s | Kernel banner, then the Ubuntu/cloud-init splash |
 | Installer comes up     | 30–60 s | The distro’s normal Live/installer environment |
 If everything works, you’re looking at the Ubuntu Server installer
 welcome screen end-to-end **without ever touching a USB stick**.
 ### 5c. Watch it from the server
 In a third shell, tail the live log:
 ```bash
 curl -N http://10.0.0.5/api/log/stream
 ```
 You’ll see each protocol step as it happens:
 ```
 INFO pxeforge::dhcp: reply mac=aa:bb:cc:dd:ee:ff arch=X8664Uefi target=tftp/snponly.efi
 INFO pxeforge::tftp: RRQ snponly.efi blksize=1468 windowsize=8 → 982 KiB in 412 ms
 INFO pxeforge::dhcp: reply mac=aa:bb:cc:dd:ee:ff (iPXE) target=http/boot.ipxe
 INFO pxeforge::http: GET /boot.ipxe?mac=aa:bb:cc:dd:ee:ff → host binding hit
 INFO pxeforge::http: GET /iso/ubuntu-…/casper/vmlinuz Range=bytes=0- 200 OK 14 MiB
 INFO pxeforge::http: GET /iso/ubuntu-…/casper/initrd  Range=bytes=0- 200 OK 75 MiB
 ```
 The **Terminal** tab in the web UI shows the same thing live, plus a
 short whitelisted command palette (`status`, `clients`, `gate`,
 `hosts`, `log`).
 ### 5d. Internet-side ISO sources
 The runbook title says “via the internet” — the **client** itself
 boots from your LAN, but the underlying ISO can come from anywhere
 your *host* can reach:
 - **Direct upload** from a remote workstation via the web UI (HTTPS
  reverse-proxied if you put PXEForge behind nginx/Caddy).
 - **NFS mount** of a remote share — Sidebar → **Storage** → **NFS** →
  `nfs://files.lab.example.com/exports/isos`. Mounted ISOs show up in
  the same list and are PXE-bootable directly without copying.
 - **Pre-seed** from a CI job that `curl`s a vendor mirror and runs
  `pxeforge seed --from`.
 PXEForge itself never reaches out to the internet at boot time — all
 client traffic stays on the LAN, served from the host.
 ---
 ## 6. After the install
 Once Ubuntu has finished installing to the target’s disk, you want
 the next reboot to come up off the new local disk, **not** PXE
 again. Two ways:
 ### 6a. One-shot — release the binding
 ```bash
 curl -fsS -X DELETE http://10.0.0.5/api/hosts/aa:bb:cc:dd:ee:ff
 ```
 Without a binding, the client either gets the menu (BIOS still set
 to PXE first) or boots local disk normally.
 ### 6b. Permanent — pin to local disk
 Re-bind to the reserved local-boot target:
 ```bash
 curl -fsS -X POST http://10.0.0.5/api/hosts \
  -H 'content-type: application/json' \
  -d '{ "mac": "aa:bb:cc:dd:ee:ff", "target": "_local", "label": "lab-rack3-node07 (installed)" }'
 ```
 Now if anyone hits `F12 → Network` by accident, PXEForge replies
 with a script that says *"chain back to local HDD"* and the box
 boots its real OS instead of re-imaging itself. This is the safest
 default for production hardware.
 ---
 ## 7. Re-imaging — the “Gated Deployment” flow
 Different scenario: you have **a rack of 30 servers** to image
 identically, all at once. Don’t bind 30 MACs by hand. Use the gate.
 1. **Don’t** create host bindings.
 2. PXE-boot every machine. They land on the menu.
 3. On each: select **Gated Deployment**. They get position #1, #2,
   …, #30 and start long-polling.
 4. In the UI: **Forge Gate** tab shows all 30 lined up. Pick the
   ISO, click **Assign to all waiting**.
 5. Every client’s open long-poll wakes up at the same instant and
   chains the same boot script. They all start imaging
   simultaneously — the “horse race gate” opens.
 The animated anvil widget on the Dashboard runs while any client is
 still in the kernel-fetch phase.
 ---
 ## Cheat sheet
 | Goal | Command |
 |------|---------|
 | Health check | `curl http://$IP/healthz` |
 | List ISOs | `curl http://$IP/api/isos \| jq .` |
 | List clients seen | `curl http://$IP/api/clients \| jq .` |
 | Bind MAC → ISO | `POST /api/hosts` with `{mac,target,label}` |
 | Bind MAC → local disk | same with `target=_local` |
 | Release binding | `DELETE /api/hosts/<mac>` |
 | Live log | `curl -N http://$IP/api/log/stream` |
 | Prometheus metrics | `curl http://$IP/metrics` |
 | Bulk import folder | `pxeforge seed --from /path` |
 ---
 ## Where to look when things break
 - **Client gets `PXE-E53: No boot filename received`** — DHCP proxy
  isn’t replying. Check `:67` is bound (`ss -lnup`), check
  `--network host`, check the host firewall on UDP 67/69/4011.
 - **iPXE shows `No more network devices`** — firmware NIC isn’t in
  PXE mode, or VLAN tagging is wrong.
 - **iPXE prints `Connection timed out (http://…)`** — `PXEFORGE_PUBLIC_IP`
  is wrong. Clients can’t reach that IP. Check `/api/status` →
  `public_base_url` and `ping` it from the client subnet.
 - **Kernel panics during initrd load** — corrupt ISO upload. Check
  `/api/isos`, compare the SHA-256 to the vendor’s, re-upload.
 - **Boot menu shows but the bound entry doesn’t fire** — the binding
  target slug doesn’t match any ISO `id`. Recheck
  `GET /api/hosts` against `GET /api/isos`. The binding falls back
  to the menu on miss (by design — never lock a client out).
 - **General confusion** — Terminal tab → `status`, then `log`. That
  tells you what protocol stages have run and which haven’t.
 For deeper protocol-level debugging, see
 [docs/architecture.md](../docs/architecture.md).