OpenPXE/runbooks/linux-network-boot.md

# Runbook: Boot a Linux machine from an ISO over the network

End-to-end walkthrough: spin up PXEForge, load an Ubuntu (or any
Linux) ISO into it, target a specific bare-metal or VM client by its
MAC address, and have that machine PXE-boot the installer over the
LAN — no USB stick, no console babysitting.

This runbook assumes:

- You have **one Linux host** to run the PXEForge container (any
  distro with Docker / Podman; 2 GB RAM, ~50 GB disk for the ISO
  library).
- That host sits on the **same broadcast domain / VLAN** as the
  client you want to boot. PXE is L2-broadcast — routed/VLAN’d
  networks need a DHCP relay and are out of scope here.
- An **existing DHCP server** is already handing out IP leases on
  that VLAN (your home router, OPNsense, Windows Server, etc.).
  PXEForge runs as a *DHCP proxy* — it never leases IPs, it only
  layers the boot information on top of the existing DHCP exchange.
- The target client is configured to **PXE-boot** in BIOS/UEFI
  firmware (usually `F12` boot menu → Network, or set as first boot
  device).

If those don’t hold, stop and read [troubleshooting.md](troubleshooting.md)
or [docs/architecture.md](../docs/architecture.md) first.

---

## 0. Pick your host’s LAN IP

You need the IPv4 address PXEForge will advertise to clients. From
the host:

```bash
ip -4 -o addr show | awk '{print $2, $4}'
```

Pick the address on the interface that faces the PXE VLAN — for
example `10.0.0.5/24` on `eno1`. From here on we call it
`PXE_HOST_IP`.

> **Why this matters.** Every URL handed to clients (TFTP server,
> iPXE chain URL, ISO URL) is built from this IP. If PXEForge
> auto-detects the wrong interface or loopback, clients will fetch
> from an unreachable address and silently fail. The startup will
> *fail loudly* if it can only auto-detect a loopback address.

---

## 1. Run PXEForge

The MVP path is a single `docker run` against the published image,
with `--network host` so the container can see DHCP broadcasts on
the LAN.

```bash
mkdir -p ~/pxeforge/isos ~/pxeforge/work

docker run -d --name pxeforge \
  --restart unless-stopped \
  --network host \
  -e PXEFORGE_PUBLIC_IP=10.0.0.5 \
  -e PXEFORGE_DHCP_MODE=proxy \
  -v ~/pxeforge/isos:/var/lib/pxeforge/isos \
  -v ~/pxeforge/work:/var/lib/pxeforge/work \
  ghcr.io/YOUR-ORG/pxeforge:0.2.0
```

Substitute your `PXEFORGE_PUBLIC_IP`, of course. If you’re building
from this repo instead of pulling, see the
[README quick start](../README.md#quick-start--mvp-container-recommended).

### Verify it’s alive

```bash
curl -fsS http://10.0.0.5/healthz   # → 200 ok
curl -fsS http://10.0.0.5/readyz    # → 200 ready  (iPXE binaries present)
curl -fsS http://10.0.0.5/api/status | jq .
```

If `/readyz` is **not** 200, your container is missing iPXE binaries.
Fix that before going further — clients have nothing to boot
otherwise. See [README — Container health probes](../README.md#container-health-probes).

### Check the listening ports

PXEForge holds three privileged UDP/TCP ports. From another shell on
the host:

```bash
sudo ss -lnup | grep -E ':(67|69|4011)\b'   # DHCP proxy + TFTP
sudo ss -lntp | grep ':80\b'                # HTTP UI / boot scripts
```

All four should be present. If port 67 is taken by `dnsmasq` or the
host’s own DHCP, stop that service or run PXEForge on a separate box —
two listeners on `:67` will fight.

---

## 2. Load the ISO

Two options. Pick one.

### 2a. Web UI upload (recommended for one-offs)

1. Open `http://10.0.0.5/` in a browser.
2. Sidebar → **Storage**.
3. Click **Upload ISO**, pick e.g. `ubuntu-24.04.1-live-server-amd64.iso`.
4. Wait for upload + introspection. The row turns into a card showing:
   - Distro family (`debian_ubuntu`)
   - Volume label
   - Detected kernel/initrd paths (`/casper/vmlinuz`, `/casper/initrd`)
   - File size and SHA-256

Big ISOs stream — there is no 2 GB limit, but expect upload to be
gated by your browser ↔ host link. The UI shows a progress bar; the
animated anvil on the Dashboard tab fires up while imaging is in
flight.

### 2b. Bulk seed from a directory (recommended for fresh deploys / CI)

If you already have a folder of ISOs on the host, skip the browser:

```bash
# Dry run first — see what would be imported, no writes:
docker exec pxeforge pxeforge seed \
  --from /seed \
  --dry-run

# For real, mount the source dir read-only into the container:
docker run --rm \
  -v /my/iso-library:/seed:ro \
  -v ~/pxeforge/isos:/var/lib/pxeforge/isos \
  -v ~/pxeforge/work:/var/lib/pxeforge/work \
  -e PXEFORGE_PUBLIC_IP=10.0.0.5 \
  ghcr.io/YOUR-ORG/pxeforge:0.2.0 seed --from /seed
```

Each `*.iso` in `/seed` runs through the same upload pipeline as the
web UI: copy → introspection → boot-entry generation → metadata
sidecar. Re-running is idempotent.

### Confirm the ISO is registered

```bash
curl -fsS http://10.0.0.5/api/isos | jq '.[] | {id, name, family, size}'
```

You should see something like:

```json
{
  "id": "ubuntu-24-04-1-live-server-amd64",
  "name": "ubuntu-24.04.1-live-server-amd64.iso",
  "family": "debian_ubuntu",
  "size": 2748000000
}
```

The `id` is the **slug**. Remember it — you’ll bind a MAC to it in
the next step.

---

## 3. Find the target machine’s MAC address

You need the MAC of the **NIC that will PXE**, not the OS’s
loopback or wifi.

### 3a. From the target itself (if it’s already running an OS)

```bash
ip -o link | awk '/ether/ {print $2, $17}'    # Linux
```

Pick the line for the wired NIC plugged into the PXE VLAN.

### 3b. From the firmware (if it’s a fresh box)

Most BIOS/UEFI screens display the NIC MAC during the network-boot
attempt — usually as `MAC: AA-BB-CC-DD-EE-FF` flashing on the splash
right before "PXE-E53: No boot filename received". Write it down.

### 3c. By letting it boot once and watching PXEForge

Easiest if the box is in front of you:

1. Power on, hit `F12`, pick **Network boot**.
2. Without any binding configured, the client will land on the
   PXEForge menu (Default / Installers / Tools / Gated Deployment).
3. Don’t pick anything. On your laptop:
   ```bash
   curl -fsS http://10.0.0.5/api/clients | jq .
   ```
4. The most-recent entry is your target. Copy its `mac`.

From here on we call this MAC `TARGET_MAC` (e.g. `aa:bb:cc:dd:ee:ff`).
Hyphens vs colons, upper vs lower case — PXEForge normalizes both.

---

## 4. Pin that machine to the Ubuntu ISO

This is the **per-MAC host binding**. With it set, the client won’t
see the menu at all — it goes straight to the bound boot entry,
Tinkerbell-style.

### 4a. Via the web UI

1. Sidebar → **Hosts**.
2. **Add binding**:
   - **MAC**: `aa:bb:cc:dd:ee:ff`
   - **Target**: pick `ubuntu-24-04-1-live-server-amd64` from the dropdown.
   - **Label**: free-form, e.g. `lab-rack3-node07`.
3. Save.

### 4b. Via the API

```bash
curl -fsS -X POST http://10.0.0.5/api/hosts \
  -H 'content-type: application/json' \
  -d '{
    "mac":    "aa:bb:cc:dd:ee:ff",
    "target": "ubuntu-24-04-1-live-server-amd64",
    "label":  "lab-rack3-node07"
  }' | jq .
```

The binding is persisted to `~/pxeforge/work/hosts.json` and survives
container restart.

### Confirm

```bash
curl -fsS http://10.0.0.5/api/hosts | jq '.[] | select(.mac=="aa:bb:cc:dd:ee:ff")'
```

You should see your entry with `created_at` and `updated_at`
timestamps.

---

## 5. Trigger the network boot on the target

Now actually boot the machine.

### 5a. Boot order

In firmware setup, set the wired NIC as the **first** boot device
(or hold `F12` / `F9` / `Esc` — vendor-specific — to pick "Network
Boot" interactively).

### 5b. What you should see on the target screen

In order, with timing:

| Stage | Approximate duration | What appears |
|-------|---------------------:|--------------|
| Firmware DHCPDISCOVER | ~1 s   | `Start PXE over IPv4` / `Station IP address …` |
| TFTP iPXE binary fetch | ~1 s  | `TFTP… snponly.efi` (or `undionly.kpxe` for legacy BIOS) |
| iPXE banner            | ~1 s  | The blue iPXE splash, version string |
| iPXE second-stage DHCP | ~1 s  | `Configuring (net0 …)` then `ok` |
| HTTP boot script fetch | <1 s  | `http://10.0.0.5/boot.ipxe?mac=…` |
| Per-MAC chain          | <1 s  | `PXEForge: per-MAC binding -> ubuntu-24-04-1-…` |
| Kernel + initrd HTTP   | 5–30 s | Two 200-OK fetches against `/iso/<id>/casper/vmlinuz` and `…/initrd` |
| Kernel boot            | 5–10 s | Kernel banner, then the Ubuntu/cloud-init splash |
| Installer comes up     | 30–60 s | The distro’s normal Live/installer environment |

If everything works, you’re looking at the Ubuntu Server installer
welcome screen end-to-end **without ever touching a USB stick**.

### 5c. Watch it from the server

In a third shell, tail the live log:

```bash
curl -N http://10.0.0.5/api/log/stream
```

You’ll see each protocol step as it happens:

```
INFO pxeforge::dhcp: reply mac=aa:bb:cc:dd:ee:ff arch=X8664Uefi target=tftp/snponly.efi
INFO pxeforge::tftp: RRQ snponly.efi blksize=1468 windowsize=8 → 982 KiB in 412 ms
INFO pxeforge::dhcp: reply mac=aa:bb:cc:dd:ee:ff (iPXE) target=http/boot.ipxe
INFO pxeforge::http: GET /boot.ipxe?mac=aa:bb:cc:dd:ee:ff → host binding hit
INFO pxeforge::http: GET /iso/ubuntu-…/casper/vmlinuz Range=bytes=0- 200 OK 14 MiB
INFO pxeforge::http: GET /iso/ubuntu-…/casper/initrd  Range=bytes=0- 200 OK 75 MiB
```

The **Terminal** tab in the web UI shows the same thing live, plus a
short whitelisted command palette (`status`, `clients`, `gate`,
`hosts`, `log`).

### 5d. Internet-side ISO sources

The runbook title says “via the internet” — the **client** itself
boots from your LAN, but the underlying ISO can come from anywhere
your *host* can reach:

- **Direct upload** from a remote workstation via the web UI (HTTPS
  reverse-proxied if you put PXEForge behind nginx/Caddy).
- **NFS mount** of a remote share — Sidebar → **Storage** → **NFS** →
  `nfs://files.lab.example.com/exports/isos`. Mounted ISOs show up in
  the same list and are PXE-bootable directly without copying.
- **Pre-seed** from a CI job that `curl`s a vendor mirror and runs
  `pxeforge seed --from`.

PXEForge itself never reaches out to the internet at boot time — all
client traffic stays on the LAN, served from the host.

---

## 6. After the install

Once Ubuntu has finished installing to the target’s disk, you want
the next reboot to come up off the new local disk, **not** PXE
again. Two ways:

### 6a. One-shot — release the binding

```bash
curl -fsS -X DELETE http://10.0.0.5/api/hosts/aa:bb:cc:dd:ee:ff
```

Without a binding, the client either gets the menu (BIOS still set
to PXE first) or boots local disk normally.

### 6b. Permanent — pin to local disk

Re-bind to the reserved local-boot target:

```bash
curl -fsS -X POST http://10.0.0.5/api/hosts \
  -H 'content-type: application/json' \
  -d '{ "mac": "aa:bb:cc:dd:ee:ff", "target": "_local", "label": "lab-rack3-node07 (installed)" }'
```

Now if anyone hits `F12 → Network` by accident, PXEForge replies
with a script that says *"chain back to local HDD"* and the box
boots its real OS instead of re-imaging itself. This is the safest
default for production hardware.

---

## 7. Re-imaging — the “Gated Deployment” flow

Different scenario: you have **a rack of 30 servers** to image
identically, all at once. Don’t bind 30 MACs by hand. Use the gate.

1. **Don’t** create host bindings.
2. PXE-boot every machine. They land on the menu.
3. On each: select **Gated Deployment**. They get position #1, #2,
   …, #30 and start long-polling.
4. In the UI: **Forge Gate** tab shows all 30 lined up. Pick the
   ISO, click **Assign to all waiting**.
5. Every client’s open long-poll wakes up at the same instant and
   chains the same boot script. They all start imaging
   simultaneously — the “horse race gate” opens.

The animated anvil widget on the Dashboard runs while any client is
still in the kernel-fetch phase.

---

## Cheat sheet

| Goal | Command |
|------|---------|
| Health check | `curl http://$IP/healthz` |
| List ISOs | `curl http://$IP/api/isos \| jq .` |
| List clients seen | `curl http://$IP/api/clients \| jq .` |
| Bind MAC → ISO | `POST /api/hosts` with `{mac,target,label}` |
| Bind MAC → local disk | same with `target=_local` |
| Release binding | `DELETE /api/hosts/<mac>` |
| Live log | `curl -N http://$IP/api/log/stream` |
| Prometheus metrics | `curl http://$IP/metrics` |
| Bulk import folder | `pxeforge seed --from /path` |

---

## Where to look when things break

- **Client gets `PXE-E53: No boot filename received`** — DHCP proxy
  isn’t replying. Check `:67` is bound (`ss -lnup`), check
  `--network host`, check the host firewall on UDP 67/69/4011.
- **iPXE shows `No more network devices`** — firmware NIC isn’t in
  PXE mode, or VLAN tagging is wrong.
- **iPXE prints `Connection timed out (http://…)`** — `PXEFORGE_PUBLIC_IP`
  is wrong. Clients can’t reach that IP. Check `/api/status` →
  `public_base_url` and `ping` it from the client subnet.
- **Kernel panics during initrd load** — corrupt ISO upload. Check
  `/api/isos`, compare the SHA-256 to the vendor’s, re-upload.
- **Boot menu shows but the bound entry doesn’t fire** — the binding
  target slug doesn’t match any ISO `id`. Recheck
  `GET /api/hosts` against `GET /api/isos`. The binding falls back
  to the menu on miss (by design — never lock a client out).
- **General confusion** — Terminal tab → `status`, then `log`. That
  tells you what protocol stages have run and which haven’t.

For deeper protocol-level debugging, see
[docs/architecture.md](../docs/architecture.md).