From e1b7154b5110dc8c2290c921e191b96161486a21 Mon Sep 17 00:00:00 2001 From: 503432756 Date: Thu, 30 Apr 2026 11:35:47 -0400 Subject: [PATCH] docs: add Linux network-boot runbook --- runbooks/linux-network-boot.md | 403 +++++++++++++++++++++++++++++++++ 1 file changed, 403 insertions(+) create mode 100644 runbooks/linux-network-boot.md diff --git a/runbooks/linux-network-boot.md b/runbooks/linux-network-boot.md new file mode 100644 index 0000000..b2b0510 --- /dev/null +++ b/runbooks/linux-network-boot.md @@ -0,0 +1,403 @@ +# Runbook: Boot a Linux machine from an ISO over the network + +End-to-end walkthrough: spin up PXEForge, load an Ubuntu (or any +Linux) ISO into it, target a specific bare-metal or VM client by its +MAC address, and have that machine PXE-boot the installer over the +LAN — no USB stick, no console babysitting. + +This runbook assumes: + +- You have **one Linux host** to run the PXEForge container (any + distro with Docker / Podman; 2 GB RAM, ~50 GB disk for the ISO + library). +- That host sits on the **same broadcast domain / VLAN** as the + client you want to boot. PXE is L2-broadcast — routed/VLAN’d + networks need a DHCP relay and are out of scope here. +- An **existing DHCP server** is already handing out IP leases on + that VLAN (your home router, OPNsense, Windows Server, etc.). + PXEForge runs as a *DHCP proxy* — it never leases IPs, it only + layers the boot information on top of the existing DHCP exchange. +- The target client is configured to **PXE-boot** in BIOS/UEFI + firmware (usually `F12` boot menu → Network, or set as first boot + device). + +If those don’t hold, stop and read [troubleshooting.md](troubleshooting.md) +or [docs/architecture.md](../docs/architecture.md) first. + +--- + +## 0. Pick your host’s LAN IP + +You need the IPv4 address PXEForge will advertise to clients. From +the host: + +```bash +ip -4 -o addr show | awk '{print $2, $4}' +``` + +Pick the address on the interface that faces the PXE VLAN — for +example `10.0.0.5/24` on `eno1`. From here on we call it +`PXE_HOST_IP`. + +> **Why this matters.** Every URL handed to clients (TFTP server, +> iPXE chain URL, ISO URL) is built from this IP. If PXEForge +> auto-detects the wrong interface or loopback, clients will fetch +> from an unreachable address and silently fail. The startup will +> *fail loudly* if it can only auto-detect a loopback address. + +--- + +## 1. Run PXEForge + +The MVP path is a single `docker run` against the published image, +with `--network host` so the container can see DHCP broadcasts on +the LAN. + +```bash +mkdir -p ~/pxeforge/isos ~/pxeforge/work + +docker run -d --name pxeforge \ + --restart unless-stopped \ + --network host \ + -e PXEFORGE_PUBLIC_IP=10.0.0.5 \ + -e PXEFORGE_DHCP_MODE=proxy \ + -v ~/pxeforge/isos:/var/lib/pxeforge/isos \ + -v ~/pxeforge/work:/var/lib/pxeforge/work \ + ghcr.io/YOUR-ORG/pxeforge:0.2.0 +``` + +Substitute your `PXEFORGE_PUBLIC_IP`, of course. If you’re building +from this repo instead of pulling, see the +[README quick start](../README.md#quick-start--mvp-container-recommended). + +### Verify it’s alive + +```bash +curl -fsS http://10.0.0.5/healthz # → 200 ok +curl -fsS http://10.0.0.5/readyz # → 200 ready (iPXE binaries present) +curl -fsS http://10.0.0.5/api/status | jq . +``` + +If `/readyz` is **not** 200, your container is missing iPXE binaries. +Fix that before going further — clients have nothing to boot +otherwise. See [README — Container health probes](../README.md#container-health-probes). + +### Check the listening ports + +PXEForge holds three privileged UDP/TCP ports. From another shell on +the host: + +```bash +sudo ss -lnup | grep -E ':(67|69|4011)\b' # DHCP proxy + TFTP +sudo ss -lntp | grep ':80\b' # HTTP UI / boot scripts +``` + +All four should be present. If port 67 is taken by `dnsmasq` or the +host’s own DHCP, stop that service or run PXEForge on a separate box — +two listeners on `:67` will fight. + +--- + +## 2. Load the ISO + +Two options. Pick one. + +### 2a. Web UI upload (recommended for one-offs) + +1. Open `http://10.0.0.5/` in a browser. +2. Sidebar → **Storage**. +3. Click **Upload ISO**, pick e.g. `ubuntu-24.04.1-live-server-amd64.iso`. +4. Wait for upload + introspection. The row turns into a card showing: + - Distro family (`debian_ubuntu`) + - Volume label + - Detected kernel/initrd paths (`/casper/vmlinuz`, `/casper/initrd`) + - File size and SHA-256 + +Big ISOs stream — there is no 2 GB limit, but expect upload to be +gated by your browser ↔ host link. The UI shows a progress bar; the +animated anvil on the Dashboard tab fires up while imaging is in +flight. + +### 2b. Bulk seed from a directory (recommended for fresh deploys / CI) + +If you already have a folder of ISOs on the host, skip the browser: + +```bash +# Dry run first — see what would be imported, no writes: +docker exec pxeforge pxeforge seed \ + --from /seed \ + --dry-run + +# For real, mount the source dir read-only into the container: +docker run --rm \ + -v /my/iso-library:/seed:ro \ + -v ~/pxeforge/isos:/var/lib/pxeforge/isos \ + -v ~/pxeforge/work:/var/lib/pxeforge/work \ + -e PXEFORGE_PUBLIC_IP=10.0.0.5 \ + ghcr.io/YOUR-ORG/pxeforge:0.2.0 seed --from /seed +``` + +Each `*.iso` in `/seed` runs through the same upload pipeline as the +web UI: copy → introspection → boot-entry generation → metadata +sidecar. Re-running is idempotent. + +### Confirm the ISO is registered + +```bash +curl -fsS http://10.0.0.5/api/isos | jq '.[] | {id, name, family, size}' +``` + +You should see something like: + +```json +{ + "id": "ubuntu-24-04-1-live-server-amd64", + "name": "ubuntu-24.04.1-live-server-amd64.iso", + "family": "debian_ubuntu", + "size": 2748000000 +} +``` + +The `id` is the **slug**. Remember it — you’ll bind a MAC to it in +the next step. + +--- + +## 3. Find the target machine’s MAC address + +You need the MAC of the **NIC that will PXE**, not the OS’s +loopback or wifi. + +### 3a. From the target itself (if it’s already running an OS) + +```bash +ip -o link | awk '/ether/ {print $2, $17}' # Linux +``` + +Pick the line for the wired NIC plugged into the PXE VLAN. + +### 3b. From the firmware (if it’s a fresh box) + +Most BIOS/UEFI screens display the NIC MAC during the network-boot +attempt — usually as `MAC: AA-BB-CC-DD-EE-FF` flashing on the splash +right before "PXE-E53: No boot filename received". Write it down. + +### 3c. By letting it boot once and watching PXEForge + +Easiest if the box is in front of you: + +1. Power on, hit `F12`, pick **Network boot**. +2. Without any binding configured, the client will land on the + PXEForge menu (Default / Installers / Tools / Gated Deployment). +3. Don’t pick anything. On your laptop: + ```bash + curl -fsS http://10.0.0.5/api/clients | jq . + ``` +4. The most-recent entry is your target. Copy its `mac`. + +From here on we call this MAC `TARGET_MAC` (e.g. `aa:bb:cc:dd:ee:ff`). +Hyphens vs colons, upper vs lower case — PXEForge normalizes both. + +--- + +## 4. Pin that machine to the Ubuntu ISO + +This is the **per-MAC host binding**. With it set, the client won’t +see the menu at all — it goes straight to the bound boot entry, +Tinkerbell-style. + +### 4a. Via the web UI + +1. Sidebar → **Hosts**. +2. **Add binding**: + - **MAC**: `aa:bb:cc:dd:ee:ff` + - **Target**: pick `ubuntu-24-04-1-live-server-amd64` from the dropdown. + - **Label**: free-form, e.g. `lab-rack3-node07`. +3. Save. + +### 4b. Via the API + +```bash +curl -fsS -X POST http://10.0.0.5/api/hosts \ + -H 'content-type: application/json' \ + -d '{ + "mac": "aa:bb:cc:dd:ee:ff", + "target": "ubuntu-24-04-1-live-server-amd64", + "label": "lab-rack3-node07" + }' | jq . +``` + +The binding is persisted to `~/pxeforge/work/hosts.json` and survives +container restart. + +### Confirm + +```bash +curl -fsS http://10.0.0.5/api/hosts | jq '.[] | select(.mac=="aa:bb:cc:dd:ee:ff")' +``` + +You should see your entry with `created_at` and `updated_at` +timestamps. + +--- + +## 5. Trigger the network boot on the target + +Now actually boot the machine. + +### 5a. Boot order + +In firmware setup, set the wired NIC as the **first** boot device +(or hold `F12` / `F9` / `Esc` — vendor-specific — to pick "Network +Boot" interactively). + +### 5b. What you should see on the target screen + +In order, with timing: + +| Stage | Approximate duration | What appears | +|-------|---------------------:|--------------| +| Firmware DHCPDISCOVER | ~1 s | `Start PXE over IPv4` / `Station IP address …` | +| TFTP iPXE binary fetch | ~1 s | `TFTP… snponly.efi` (or `undionly.kpxe` for legacy BIOS) | +| iPXE banner | ~1 s | The blue iPXE splash, version string | +| iPXE second-stage DHCP | ~1 s | `Configuring (net0 …)` then `ok` | +| HTTP boot script fetch | <1 s | `http://10.0.0.5/boot.ipxe?mac=…` | +| Per-MAC chain | <1 s | `PXEForge: per-MAC binding -> ubuntu-24-04-1-…` | +| Kernel + initrd HTTP | 5–30 s | Two 200-OK fetches against `/iso//casper/vmlinuz` and `…/initrd` | +| Kernel boot | 5–10 s | Kernel banner, then the Ubuntu/cloud-init splash | +| Installer comes up | 30–60 s | The distro’s normal Live/installer environment | + +If everything works, you’re looking at the Ubuntu Server installer +welcome screen end-to-end **without ever touching a USB stick**. + +### 5c. Watch it from the server + +In a third shell, tail the live log: + +```bash +curl -N http://10.0.0.5/api/log/stream +``` + +You’ll see each protocol step as it happens: + +``` +INFO pxeforge::dhcp: reply mac=aa:bb:cc:dd:ee:ff arch=X8664Uefi target=tftp/snponly.efi +INFO pxeforge::tftp: RRQ snponly.efi blksize=1468 windowsize=8 → 982 KiB in 412 ms +INFO pxeforge::dhcp: reply mac=aa:bb:cc:dd:ee:ff (iPXE) target=http/boot.ipxe +INFO pxeforge::http: GET /boot.ipxe?mac=aa:bb:cc:dd:ee:ff → host binding hit +INFO pxeforge::http: GET /iso/ubuntu-…/casper/vmlinuz Range=bytes=0- 200 OK 14 MiB +INFO pxeforge::http: GET /iso/ubuntu-…/casper/initrd Range=bytes=0- 200 OK 75 MiB +``` + +The **Terminal** tab in the web UI shows the same thing live, plus a +short whitelisted command palette (`status`, `clients`, `gate`, +`hosts`, `log`). + +### 5d. Internet-side ISO sources + +The runbook title says “via the internet” — the **client** itself +boots from your LAN, but the underlying ISO can come from anywhere +your *host* can reach: + +- **Direct upload** from a remote workstation via the web UI (HTTPS + reverse-proxied if you put PXEForge behind nginx/Caddy). +- **NFS mount** of a remote share — Sidebar → **Storage** → **NFS** → + `nfs://files.lab.example.com/exports/isos`. Mounted ISOs show up in + the same list and are PXE-bootable directly without copying. +- **Pre-seed** from a CI job that `curl`s a vendor mirror and runs + `pxeforge seed --from`. + +PXEForge itself never reaches out to the internet at boot time — all +client traffic stays on the LAN, served from the host. + +--- + +## 6. After the install + +Once Ubuntu has finished installing to the target’s disk, you want +the next reboot to come up off the new local disk, **not** PXE +again. Two ways: + +### 6a. One-shot — release the binding + +```bash +curl -fsS -X DELETE http://10.0.0.5/api/hosts/aa:bb:cc:dd:ee:ff +``` + +Without a binding, the client either gets the menu (BIOS still set +to PXE first) or boots local disk normally. + +### 6b. Permanent — pin to local disk + +Re-bind to the reserved local-boot target: + +```bash +curl -fsS -X POST http://10.0.0.5/api/hosts \ + -H 'content-type: application/json' \ + -d '{ "mac": "aa:bb:cc:dd:ee:ff", "target": "_local", "label": "lab-rack3-node07 (installed)" }' +``` + +Now if anyone hits `F12 → Network` by accident, PXEForge replies +with a script that says *"chain back to local HDD"* and the box +boots its real OS instead of re-imaging itself. This is the safest +default for production hardware. + +--- + +## 7. Re-imaging — the “Gated Deployment” flow + +Different scenario: you have **a rack of 30 servers** to image +identically, all at once. Don’t bind 30 MACs by hand. Use the gate. + +1. **Don’t** create host bindings. +2. PXE-boot every machine. They land on the menu. +3. On each: select **Gated Deployment**. They get position #1, #2, + …, #30 and start long-polling. +4. In the UI: **Forge Gate** tab shows all 30 lined up. Pick the + ISO, click **Assign to all waiting**. +5. Every client’s open long-poll wakes up at the same instant and + chains the same boot script. They all start imaging + simultaneously — the “horse race gate” opens. + +The animated anvil widget on the Dashboard runs while any client is +still in the kernel-fetch phase. + +--- + +## Cheat sheet + +| Goal | Command | +|------|---------| +| Health check | `curl http://$IP/healthz` | +| List ISOs | `curl http://$IP/api/isos \| jq .` | +| List clients seen | `curl http://$IP/api/clients \| jq .` | +| Bind MAC → ISO | `POST /api/hosts` with `{mac,target,label}` | +| Bind MAC → local disk | same with `target=_local` | +| Release binding | `DELETE /api/hosts/` | +| Live log | `curl -N http://$IP/api/log/stream` | +| Prometheus metrics | `curl http://$IP/metrics` | +| Bulk import folder | `pxeforge seed --from /path` | + +--- + +## Where to look when things break + +- **Client gets `PXE-E53: No boot filename received`** — DHCP proxy + isn’t replying. Check `:67` is bound (`ss -lnup`), check + `--network host`, check the host firewall on UDP 67/69/4011. +- **iPXE shows `No more network devices`** — firmware NIC isn’t in + PXE mode, or VLAN tagging is wrong. +- **iPXE prints `Connection timed out (http://…)`** — `PXEFORGE_PUBLIC_IP` + is wrong. Clients can’t reach that IP. Check `/api/status` → + `public_base_url` and `ping` it from the client subnet. +- **Kernel panics during initrd load** — corrupt ISO upload. Check + `/api/isos`, compare the SHA-256 to the vendor’s, re-upload. +- **Boot menu shows but the bound entry doesn’t fire** — the binding + target slug doesn’t match any ISO `id`. Recheck + `GET /api/hosts` against `GET /api/isos`. The binding falls back + to the menu on miss (by design — never lock a client out). +- **General confusion** — Terminal tab → `status`, then `log`. That + tells you what protocol stages have run and which haven’t. + +For deeper protocol-level debugging, see +[docs/architecture.md](../docs/architecture.md).