Files

T

503432756 e1b7154b51 docs: add Linux network-boot runbook

2026-04-30 11:35:47 -04:00

13 KiB

Raw Blame History

Runbook: Boot a Linux machine from an ISO over the network

End-to-end walkthrough: spin up PXEForge, load an Ubuntu (or any Linux) ISO into it, target a specific bare-metal or VM client by its MAC address, and have that machine PXE-boot the installer over the LAN — no USB stick, no console babysitting.

This runbook assumes:

You have one Linux host to run the PXEForge container (any distro with Docker / Podman; 2 GB RAM, ~50 GB disk for the ISO library).
That host sits on the same broadcast domain / VLAN as the client you want to boot. PXE is L2-broadcast — routed/VLAN’d networks need a DHCP relay and are out of scope here.
An existing DHCP server is already handing out IP leases on that VLAN (your home router, OPNsense, Windows Server, etc.). PXEForge runs as a DHCP proxy — it never leases IPs, it only layers the boot information on top of the existing DHCP exchange.
The target client is configured to PXE-boot in BIOS/UEFI firmware (usually F12 boot menu → Network, or set as first boot device).

If those don’t hold, stop and read troubleshooting.md or docs/architecture.md first.

0. Pick your host’s LAN IP

You need the IPv4 address PXEForge will advertise to clients. From the host:

ip -4 -o addr show | awk '{print $2, $4}'

Pick the address on the interface that faces the PXE VLAN — for example 10.0.0.5/24 on eno1. From here on we call it PXE_HOST_IP.

Why this matters. Every URL handed to clients (TFTP server, iPXE chain URL, ISO URL) is built from this IP. If PXEForge auto-detects the wrong interface or loopback, clients will fetch from an unreachable address and silently fail. The startup will fail loudly if it can only auto-detect a loopback address.

1. Run PXEForge

The MVP path is a single docker run against the published image, with --network host so the container can see DHCP broadcasts on the LAN.

mkdir -p ~/pxeforge/isos ~/pxeforge/work

docker run -d --name pxeforge \
  --restart unless-stopped \
  --network host \
  -e PXEFORGE_PUBLIC_IP=10.0.0.5 \
  -e PXEFORGE_DHCP_MODE=proxy \
  -v ~/pxeforge/isos:/var/lib/pxeforge/isos \
  -v ~/pxeforge/work:/var/lib/pxeforge/work \
  ghcr.io/YOUR-ORG/pxeforge:0.2.0

Substitute your PXEFORGE_PUBLIC_IP, of course. If you’re building from this repo instead of pulling, see the README quick start.

Verify it’s alive

curl -fsS http://10.0.0.5/healthz   # → 200 ok
curl -fsS http://10.0.0.5/readyz    # → 200 ready  (iPXE binaries present)
curl -fsS http://10.0.0.5/api/status | jq .

If /readyz is not 200, your container is missing iPXE binaries. Fix that before going further — clients have nothing to boot otherwise. See README — Container health probes.

Check the listening ports

PXEForge holds three privileged UDP/TCP ports. From another shell on the host:

sudo ss -lnup | grep -E ':(67|69|4011)\b'   # DHCP proxy + TFTP
sudo ss -lntp | grep ':80\b'                # HTTP UI / boot scripts

All four should be present. If port 67 is taken by dnsmasq or the host’s own DHCP, stop that service or run PXEForge on a separate box — two listeners on :67 will fight.

2. Load the ISO

Two options. Pick one.

2a. Web UI upload (recommended for one-offs)

Open http://10.0.0.5/ in a browser.
Sidebar → Storage.
Click Upload ISO, pick e.g. ubuntu-24.04.1-live-server-amd64.iso.
Wait for upload + introspection. The row turns into a card showing:
- Distro family (debian_ubuntu)
- Volume label
- Detected kernel/initrd paths (/casper/vmlinuz, /casper/initrd)
- File size and SHA-256

Big ISOs stream — there is no 2 GB limit, but expect upload to be gated by your browser ↔ host link. The UI shows a progress bar; the animated anvil on the Dashboard tab fires up while imaging is in flight.

2b. Bulk seed from a directory (recommended for fresh deploys / CI)

If you already have a folder of ISOs on the host, skip the browser:

# Dry run first — see what would be imported, no writes:
docker exec pxeforge pxeforge seed \
  --from /seed \
  --dry-run

# For real, mount the source dir read-only into the container:
docker run --rm \
  -v /my/iso-library:/seed:ro \
  -v ~/pxeforge/isos:/var/lib/pxeforge/isos \
  -v ~/pxeforge/work:/var/lib/pxeforge/work \
  -e PXEFORGE_PUBLIC_IP=10.0.0.5 \
  ghcr.io/YOUR-ORG/pxeforge:0.2.0 seed --from /seed

Each *.iso in /seed runs through the same upload pipeline as the web UI: copy → introspection → boot-entry generation → metadata sidecar. Re-running is idempotent.

Confirm the ISO is registered

curl -fsS http://10.0.0.5/api/isos | jq '.[] | {id, name, family, size}'

You should see something like:

{
  "id": "ubuntu-24-04-1-live-server-amd64",
  "name": "ubuntu-24.04.1-live-server-amd64.iso",
  "family": "debian_ubuntu",
  "size": 2748000000
}

The id is the slug. Remember it — you’ll bind a MAC to it in the next step.

3. Find the target machine’s MAC address

You need the MAC of the NIC that will PXE, not the OS’s loopback or wifi.

3a. From the target itself (if it’s already running an OS)

ip -o link | awk '/ether/ {print $2, $17}'    # Linux

Pick the line for the wired NIC plugged into the PXE VLAN.

3b. From the firmware (if it’s a fresh box)

Most BIOS/UEFI screens display the NIC MAC during the network-boot attempt — usually as MAC: AA-BB-CC-DD-EE-FF flashing on the splash right before "PXE-E53: No boot filename received". Write it down.

3c. By letting it boot once and watching PXEForge

Easiest if the box is in front of you:

Power on, hit F12, pick Network boot.
Without any binding configured, the client will land on the PXEForge menu (Default / Installers / Tools / Gated Deployment).

Don’t pick anything. On your laptop:

curl -fsS http://10.0.0.5/api/clients | jq .

The most-recent entry is your target. Copy its mac.

From here on we call this MAC TARGET_MAC (e.g. aa:bb:cc:dd:ee:ff). Hyphens vs colons, upper vs lower case — PXEForge normalizes both.

4. Pin that machine to the Ubuntu ISO

This is the per-MAC host binding. With it set, the client won’t see the menu at all — it goes straight to the bound boot entry, Tinkerbell-style.

4a. Via the web UI

Sidebar → Hosts.
Add binding:
- MAC: aa:bb:cc:dd:ee:ff
- Target: pick ubuntu-24-04-1-live-server-amd64 from the dropdown.
- Label: free-form, e.g. lab-rack3-node07.
Save.

4b. Via the API

curl -fsS -X POST http://10.0.0.5/api/hosts \
  -H 'content-type: application/json' \
  -d '{
    "mac":    "aa:bb:cc:dd:ee:ff",
    "target": "ubuntu-24-04-1-live-server-amd64",
    "label":  "lab-rack3-node07"
  }' | jq .

The binding is persisted to ~/pxeforge/work/hosts.json and survives container restart.

Confirm

curl -fsS http://10.0.0.5/api/hosts | jq '.[] | select(.mac=="aa:bb:cc:dd:ee:ff")'

You should see your entry with created_at and updated_at timestamps.

5. Trigger the network boot on the target

Now actually boot the machine.

5a. Boot order

In firmware setup, set the wired NIC as the first boot device (or hold F12 / F9 / Esc — vendor-specific — to pick "Network Boot" interactively).

5b. What you should see on the target screen

In order, with timing:

Stage	Approximate duration	What appears
Firmware DHCPDISCOVER	~1 s	`Start PXE over IPv4` / `Station IP address …`
TFTP iPXE binary fetch	~1 s	`TFTP… snponly.efi` (or `undionly.kpxe` for legacy BIOS)
iPXE banner	~1 s	The blue iPXE splash, version string
iPXE second-stage DHCP	~1 s	`Configuring (net0 …)` then `ok`
HTTP boot script fetch	<1 s	`http://10.0.0.5/boot.ipxe?mac=…`
Per-MAC chain	<1 s	`PXEForge: per-MAC binding -> ubuntu-24-04-1-…`
Kernel + initrd HTTP	5–30 s	Two 200-OK fetches against `/iso/<id>/casper/vmlinuz` and `…/initrd`
Kernel boot	5–10 s	Kernel banner, then the Ubuntu/cloud-init splash
Installer comes up	30–60 s	The distro’s normal Live/installer environment

If everything works, you’re looking at the Ubuntu Server installer welcome screen end-to-end without ever touching a USB stick.

5c. Watch it from the server

In a third shell, tail the live log:

curl -N http://10.0.0.5/api/log/stream

You’ll see each protocol step as it happens:

INFO pxeforge::dhcp: reply mac=aa:bb:cc:dd:ee:ff arch=X8664Uefi target=tftp/snponly.efi
INFO pxeforge::tftp: RRQ snponly.efi blksize=1468 windowsize=8 → 982 KiB in 412 ms
INFO pxeforge::dhcp: reply mac=aa:bb:cc:dd:ee:ff (iPXE) target=http/boot.ipxe
INFO pxeforge::http: GET /boot.ipxe?mac=aa:bb:cc:dd:ee:ff → host binding hit
INFO pxeforge::http: GET /iso/ubuntu-…/casper/vmlinuz Range=bytes=0- 200 OK 14 MiB
INFO pxeforge::http: GET /iso/ubuntu-…/casper/initrd  Range=bytes=0- 200 OK 75 MiB

The Terminal tab in the web UI shows the same thing live, plus a short whitelisted command palette (status, clients, gate, hosts, log).

5d. Internet-side ISO sources

The runbook title says “via the internet” — the client itself boots from your LAN, but the underlying ISO can come from anywhere your host can reach:

Direct upload from a remote workstation via the web UI (HTTPS reverse-proxied if you put PXEForge behind nginx/Caddy).
NFS mount of a remote share — Sidebar → Storage → NFS → nfs://files.lab.example.com/exports/isos. Mounted ISOs show up in the same list and are PXE-bootable directly without copying.
Pre-seed from a CI job that curls a vendor mirror and runs pxeforge seed --from.

PXEForge itself never reaches out to the internet at boot time — all client traffic stays on the LAN, served from the host.

6. After the install

Once Ubuntu has finished installing to the target’s disk, you want the next reboot to come up off the new local disk, not PXE again. Two ways:

6a. One-shot — release the binding

curl -fsS -X DELETE http://10.0.0.5/api/hosts/aa:bb:cc:dd:ee:ff

Without a binding, the client either gets the menu (BIOS still set to PXE first) or boots local disk normally.

6b. Permanent — pin to local disk

Re-bind to the reserved local-boot target:

curl -fsS -X POST http://10.0.0.5/api/hosts \
  -H 'content-type: application/json' \
  -d '{ "mac": "aa:bb:cc:dd:ee:ff", "target": "_local", "label": "lab-rack3-node07 (installed)" }'

Now if anyone hits F12 → Network by accident, PXEForge replies with a script that says "chain back to local HDD" and the box boots its real OS instead of re-imaging itself. This is the safest default for production hardware.

7. Re-imaging — the “Gated Deployment” flow

Different scenario: you have a rack of 30 servers to image identically, all at once. Don’t bind 30 MACs by hand. Use the gate.

Don’t create host bindings.
PXE-boot every machine. They land on the menu.
On each: select Gated Deployment. They get position #1, #2, …, #30 and start long-polling.
In the UI: Forge Gate tab shows all 30 lined up. Pick the ISO, click Assign to all waiting.
Every client’s open long-poll wakes up at the same instant and chains the same boot script. They all start imaging simultaneously — the “horse race gate” opens.

The animated anvil widget on the Dashboard runs while any client is still in the kernel-fetch phase.

Cheat sheet

Goal	Command
Health check	`curl http://$IP/healthz`
List ISOs	`curl http://$IP/api/isos \| jq .`
List clients seen	`curl http://$IP/api/clients \| jq .`
Bind MAC → ISO	`POST /api/hosts` with `{mac,target,label}`
Bind MAC → local disk	same with `target=_local`
Release binding	`DELETE /api/hosts/<mac>`
Live log	`curl -N http://$IP/api/log/stream`
Prometheus metrics	`curl http://$IP/metrics`
Bulk import folder	`pxeforge seed --from /path`

Where to look when things break

Client gets PXE-E53: No boot filename received — DHCP proxy isn’t replying. Check :67 is bound (ss -lnup), check --network host, check the host firewall on UDP 67/69/4011.
iPXE shows No more network devices — firmware NIC isn’t in PXE mode, or VLAN tagging is wrong.
iPXE prints Connection timed out (http://…) — PXEFORGE_PUBLIC_IP is wrong. Clients can’t reach that IP. Check /api/status → public_base_url and ping it from the client subnet.
Kernel panics during initrd load — corrupt ISO upload. Check /api/isos, compare the SHA-256 to the vendor’s, re-upload.
Boot menu shows but the bound entry doesn’t fire — the binding target slug doesn’t match any ISO id. Recheck GET /api/hosts against GET /api/isos. The binding falls back to the menu on miss (by design — never lock a client out).
General confusion — Terminal tab → status, then log. That tells you what protocol stages have run and which haven’t.

For deeper protocol-level debugging, see docs/architecture.md.

13 KiB Raw Blame History Unescape Escape