OpenPXE/runbooks/linux-network-boot.md

# Runbook: Boot a Linux machine from an ISO over the network

End-to-end walkthrough: spin up OpenPXE, load an Ubuntu (or any
Linux) ISO into it, target a specific bare-metal or VM client by its
MAC address, and have that machine PXE-boot the installer over the
LAN — no USB stick, no console babysitting.

This runbook assumes:

- You have **one Linux host** to run the OpenPXE container (any
  distro with Docker / Podman; 2 GB RAM, ~50 GB disk for the ISO
  library).
- That host sits on the **same broadcast domain / VLAN** as the
  client you want to boot. PXE is L2-broadcast — routed/VLAN’d
  networks need a DHCP relay and are out of scope here.
- An **existing DHCP server** is already handing out IP leases on
  that VLAN (your home router, OPNsense, Windows Server, etc.).
  OpenPXE runs as a *DHCP proxy* — it never leases IPs, it only
  layers the boot information on top of the existing DHCP exchange.
- The target client is configured to **PXE-boot** in BIOS/UEFI
  firmware (usually `F12` boot menu → Network, or set as first boot
  device).

If those don’t hold, stop and read [troubleshooting.md](troubleshooting.md)
or [docs/architecture.md](../docs/architecture.md) first.

---

## 0. Pick your host’s LAN IP

You need the IPv4 address OpenPXE will advertise to clients. From
the host:

```bash
ip -4 -o addr show | awk '{print $2, $4}'
```

Pick the address on the interface that faces the PXE VLAN — for
example `10.0.0.5/24` on `eno1`. From here on we call it
`PXE_HOST_IP`.

> **Why this matters.** Every URL handed to clients (TFTP server,
> iPXE chain URL, ISO URL) is built from this IP. If OpenPXE
> auto-detects the wrong interface or loopback, clients will fetch
> from an unreachable address and silently fail. The startup will
> *fail loudly* if it can only auto-detect a loopback address.

---

## 1. Run OpenPXE

The MVP path is a single `docker run` against the published image,
with `--network host` so the container can see DHCP broadcasts on
the LAN.

```bash
mkdir -p ~/openpxe/isos ~/openpxe/work

docker run -d --name openpxe \
  --restart unless-stopped \
  --network host \
  -e OPENPXE_PUBLIC_IP=10.0.0.5 \
  -e OPENPXE_DHCP_MODE=proxy \
  -v ~/openpxe/isos:/var/lib/openpxe/isos \
  -v ~/openpxe/work:/var/lib/openpxe/work \
  ghcr.io/YOUR-ORG/openpxe:0.2.0
```

Substitute your `OPENPXE_PUBLIC_IP`, of course. If you’re building
from this repo instead of pulling, see the
[README quick start](../README.md#quick-start--mvp-container-recommended).

### Verify it’s alive

```bash
curl -fsS http://10.0.0.5/healthz   # → 200 ok
curl -fsS http://10.0.0.5/readyz    # → 200 ready  (iPXE binaries present)
curl -fsS http://10.0.0.5/api/status | jq .
```

If `/readyz` is **not** 200, your container is missing iPXE binaries.
Fix that before going further — clients have nothing to boot
otherwise. See [README — Container health probes](../README.md#container-health-probes).

### Check the listening ports

OpenPXE holds three privileged UDP/TCP ports. From another shell on
the host:

```bash
sudo ss -lnup | grep -E ':(67|69|4011)\b'   # DHCP proxy + TFTP
sudo ss -lntp | grep ':80\b'                # HTTP UI / boot scripts
```

All four should be present. If port 67 is taken by `dnsmasq` or the
host’s own DHCP, stop that service or run OpenPXE on a separate box —
two listeners on `:67` will fight.

---

## 2. Load the ISO

Two options. Pick one.

### 2a. Web UI upload (recommended for one-offs)

1. Open `http://10.0.0.5/` in a browser.
2. Sidebar → **Storage**.
3. Click **Upload ISO**, pick e.g. `ubuntu-24.04.1-live-server-amd64.iso`.
4. Wait for upload + introspection. The row turns into a card showing:
   - Distro family (`debian_ubuntu`)
   - Volume label
   - Detected kernel/initrd paths (`/casper/vmlinuz`, `/casper/initrd`)
   - File size and SHA-256

Big ISOs stream — there is no 2 GB limit, but expect upload to be
throttled by your browser ↔ host link. The UI shows a progress bar; the
animated OpenPXE mark on the Dashboard tab fires up while imaging is in
flight.

### 2b. Bulk seed from a directory (recommended for fresh deploys / CI)

If you already have a folder of ISOs on the host, skip the browser:

```bash
# Dry run first — see what would be imported, no writes:
docker exec openpxe openpxe seed \
  --from /seed \
  --dry-run

# For real, mount the source dir read-only into the container:
docker run --rm \
  -v /my/iso-library:/seed:ro \
  -v ~/openpxe/isos:/var/lib/openpxe/isos \
  -v ~/openpxe/work:/var/lib/openpxe/work \
  -e OPENPXE_PUBLIC_IP=10.0.0.5 \
  ghcr.io/YOUR-ORG/openpxe:0.2.0 seed --from /seed
```

Each `*.iso` in `/seed` runs through the same upload pipeline as the
web UI: copy → introspection → boot-entry generation → metadata
sidecar. Re-running is idempotent.

### Confirm the ISO is registered

```bash
curl -fsS http://10.0.0.5/api/isos | jq '.[] | {id, name, family, size}'
```

You should see something like:

```json
{
  "id": "ubuntu-24-04-1-live-server-amd64",
  "name": "ubuntu-24.04.1-live-server-amd64.iso",
  "family": "debian_ubuntu",
  "size": 2748000000
}
```

The `id` is the **slug**. Remember it — you’ll bind a MAC to it in
the next step.

---

## 3. Find the target machine’s MAC address

You need the MAC of the **NIC that will PXE**, not the OS’s
loopback or wifi.

### 3a. From the target itself (if it’s already running an OS)

```bash
ip -o link | awk '/ether/ {print $2, $17}'    # Linux
```

Pick the line for the wired NIC plugged into the PXE VLAN.

### 3b. From the firmware (if it’s a fresh box)

Most BIOS/UEFI screens display the NIC MAC during the network-boot
attempt — usually as `MAC: AA-BB-CC-DD-EE-FF` flashing on the splash
right before "PXE-E53: No boot filename received". Write it down.

### 3c. By letting it boot once and watching OpenPXE

Easiest if the box is in front of you:

1. Power on, hit `F12`, pick **Network boot**.
2. Without any binding configured, the client will land on the
   OpenPXE menu (Default / Installers / Tools / Queued Deployment).
3. Don’t pick anything. On your laptop:
   ```bash
   curl -fsS http://10.0.0.5/api/clients | jq .
   ```
4. The most-recent entry is your target. Copy its `mac`.

From here on we call this MAC `TARGET_MAC` (e.g. `aa:bb:cc:dd:ee:ff`).
Hyphens vs colons, upper vs lower case — OpenPXE normalizes both.

---

## 4. Pin that machine to the Ubuntu ISO

This is the **per-MAC host binding**. With it set, the client won’t
see the menu at all — it goes straight to the bound boot entry,
Tinkerbell-style.

### 4a. Via the web UI

1. Sidebar → **Hosts**.
2. **Add binding**:
   - **MAC**: `aa:bb:cc:dd:ee:ff`
   - **Target**: pick `ubuntu-24-04-1-live-server-amd64` from the dropdown.
   - **Label**: free-form, e.g. `lab-rack3-node07`.
3. Save.

### 4b. Via the API

```bash
curl -fsS -X POST http://10.0.0.5/api/hosts \
  -H 'content-type: application/json' \
  -d '{
    "mac":    "aa:bb:cc:dd:ee:ff",
    "target": "ubuntu-24-04-1-live-server-amd64",
    "label":  "lab-rack3-node07"
  }' | jq .
```

The binding is persisted to `~/openpxe/work/hosts.json` and survives
container restart.

### Confirm

```bash
curl -fsS http://10.0.0.5/api/hosts | jq '.[] | select(.mac=="aa:bb:cc:dd:ee:ff")'
```

You should see your entry with `created_at` and `updated_at`
timestamps.

---

## 5. Trigger the network boot on the target

Now actually boot the machine.

### 5a. Boot order

In firmware setup, set the wired NIC as the **first** boot device
(or hold `F12` / `F9` / `Esc` — vendor-specific — to pick "Network
Boot" interactively).

### 5b. What you should see on the target screen

In order, with timing:

| Stage | Approximate duration | What appears |
|-------|---------------------:|--------------|
| Firmware DHCPDISCOVER | ~1 s   | `Start PXE over IPv4` / `Station IP address …` |
| TFTP iPXE binary fetch | ~1 s  | `TFTP… snponly.efi` (or `undionly.kpxe` for legacy BIOS) |
| iPXE banner            | ~1 s  | The blue iPXE splash, version string |
| iPXE second-stage DHCP | ~1 s  | `Configuring (net0 …)` then `ok` |
| HTTP boot script fetch | <1 s  | `http://10.0.0.5/boot.ipxe?mac=…` |
| Per-MAC chain          | <1 s  | `OpenPXE: per-MAC binding -> ubuntu-24-04-1-…` |
| Kernel + initrd HTTP   | 5–30 s | Two 200-OK fetches against `/iso/<id>/casper/vmlinuz` and `…/initrd` |
| Kernel boot            | 5–10 s | Kernel banner, then the Ubuntu/cloud-init splash |
| Installer comes up     | 30–60 s | The distro’s normal Live/installer environment |

If everything works, you’re looking at the Ubuntu Server installer
welcome screen end-to-end **without ever touching a USB stick**.

### 5c. Watch it from the server

In a third shell, tail the live log:

```bash
curl -N http://10.0.0.5/api/log/stream
```

You’ll see each protocol step as it happens:

```
INFO openpxe::dhcp: reply mac=aa:bb:cc:dd:ee:ff arch=X8664Uefi target=tftp/snponly.efi
INFO openpxe::tftp: RRQ snponly.efi blksize=1468 windowsize=8 → 982 KiB in 412 ms
INFO openpxe::dhcp: reply mac=aa:bb:cc:dd:ee:ff (iPXE) target=http/boot.ipxe
INFO openpxe::http: GET /boot.ipxe?mac=aa:bb:cc:dd:ee:ff → host binding hit
INFO openpxe::http: GET /iso/ubuntu-…/casper/vmlinuz Range=bytes=0- 200 OK 14 MiB
INFO openpxe::http: GET /iso/ubuntu-…/casper/initrd  Range=bytes=0- 200 OK 75 MiB
```

The **Terminal** tab in the web UI shows the same thing live, plus a
short whitelisted command palette (`status`, `clients`, `queue`,
`hosts`, `log`).

### 5d. Internet-side ISO sources

The runbook title says “via the internet” — the **client** itself
boots from your LAN, but the underlying ISO can come from anywhere
your *host* can reach:

- **Direct upload** from a remote workstation via the web UI (HTTPS
  reverse-proxied if you put OpenPXE behind nginx/Caddy).
- **NFS mount** of a remote share — Sidebar → **Storage** → **NFS** →
  `nfs://files.lab.example.com/exports/isos`. Mounted ISOs show up in
  the same list and are PXE-bootable directly without copying.
- **Pre-seed** from a CI job that `curl`s a vendor mirror and runs
  `openpxe seed --from`.

OpenPXE itself never reaches out to the internet at boot time — all
client traffic stays on the LAN, served from the host.

---

## 6. After the install

Once Ubuntu has finished installing to the target’s disk, you want
the next reboot to come up off the new local disk, **not** PXE
again. Two ways:

### 6a. One-shot — release the binding

```bash
curl -fsS -X DELETE http://10.0.0.5/api/hosts/aa:bb:cc:dd:ee:ff
```

Without a binding, the client either gets the menu (BIOS still set
to PXE first) or boots local disk normally.

### 6b. Permanent — pin to local disk

Re-bind to the reserved local-boot target:

```bash
curl -fsS -X POST http://10.0.0.5/api/hosts \
  -H 'content-type: application/json' \
  -d '{ "mac": "aa:bb:cc:dd:ee:ff", "target": "_local", "label": "lab-rack3-node07 (installed)" }'
```

Now if anyone hits `F12 → Network` by accident, OpenPXE replies
with a script that says *"chain back to local HDD"* and the box
boots its real OS instead of re-imaging itself. This is the safest
default for production hardware.

---

## 7. Re-imaging — the “Queued Deployment” flow

Different scenario: you have **a rack of 30 servers** to image
identically, all at once. Don’t bind 30 MACs by hand. Use the queue.

1. **Don’t** create host bindings.
2. PXE-boot every machine. They land on the menu.
3. On each: select **Queued Deployment**. They get position #1, #2,
   …, #30 and start long-polling.
4. In the UI: **Queue** tab shows all 30 lined up. Pick the
   ISO, click **Assign to all waiting**.
5. Every client’s open long-poll wakes up at the same instant and
   chains the same boot script. They all start imaging
   simultaneously.

The animated OpenPXE progress widget on the Dashboard runs while any client is
still in the kernel-fetch phase.

---

## Cheat sheet

| Goal | Command |
|------|---------|
| Health check | `curl http://$IP/healthz` |
| List ISOs | `curl http://$IP/api/isos \| jq .` |
| List clients seen | `curl http://$IP/api/clients \| jq .` |
| Bind MAC → ISO | `POST /api/hosts` with `{mac,target,label}` |
| Bind MAC → local disk | same with `target=_local` |
| Release binding | `DELETE /api/hosts/<mac>` |
| Live log | `curl -N http://$IP/api/log/stream` |
| Prometheus metrics | `curl http://$IP/metrics` |
| Bulk import folder | `openpxe seed --from /path` |

---

## Where to look when things break

- **Client gets `PXE-E53: No boot filename received`** — DHCP proxy
  isn’t replying. Check `:67` is bound (`ss -lnup`), check
  `--network host`, check the host firewall on UDP 67/69/4011.
- **iPXE shows `No more network devices`** — firmware NIC isn’t in
  PXE mode, or VLAN tagging is wrong.
- **iPXE prints `Connection timed out (http://…)`** — `OPENPXE_PUBLIC_IP`
  is wrong. Clients can’t reach that IP. Check `/api/status` →
  `public_base_url` and `ping` it from the client subnet.
- **Kernel panics during initrd load** — corrupt ISO upload. Check
  `/api/isos`, compare the SHA-256 to the vendor’s, re-upload.
- **Boot menu shows but the bound entry doesn’t fire** — the binding
  target slug doesn’t match any ISO `id`. Recheck
  `GET /api/hosts` against `GET /api/isos`. The binding falls back
  to the menu on miss (by design — never lock a client out).
- **General confusion** — Terminal tab → `status`, then `log`. That
  tells you what protocol stages have run and which haven’t.

For deeper protocol-level debugging, see
[docs/architecture.md](../docs/architecture.md).