v0.7.4: probe-based introspection — remote shares classify, gparted bug fixed, Storage pagination
Introspection (the headline): detection is now probe-based. Instead of
grepping raw sectors for filename strings, we walk the ISO9660
directory tree and check whether the well-known boot files actually
exist — and the same probes run over NFS READ3 / SFTP seek-reads, so
share-hosted ISOs finally classify instead of registering as Unknown.
iso-store:
- New iso_fs module: the read-only ISO9660 walker (generalized from
http-api) over an IsoReadAt trait — local files, NFS, SFTP, and the
in-memory test images all share it. Iterative walk, 4 MiB directory
cap, strict-mastering trailing-dot normalization (VMLINUZ.;1 now
matches /vmlinuz), CachingReadAt collapses repeated directory reads
during the probe pass (~60 → ~6 round-trips per remote ISO).
- introspect.rs rewritten (INTROSPECT_REV 2): PVD label → El Torito →
/sources/boot.wim probe → verified Linux kernel+initrd probe table →
local-only 16 MiB UDF-Windows scan → filename-token fallback.
* Fixes the false-Windows bug: any Linux ISO shipping GRUB/syslinux
chainload modules contains the literal "bootmgr", so gparted-live
classified as WindowsPe. Linux probes now run first; the byte scan
only sees ISOs nothing else claimed. Local ISOs re-probe once on
startup via the rev bump — no re-upload.
* Kernel entries are emitted only when kernel+initrd verifiably
exist (no more guessed paths that 404 at boot). Debian-live /
d-i netinst / CoreOS shapes classify for the UI but keep their
working sanboot entries (their boot protocols need args we don't
render yet; CoreOS additionally needs its embedded ignition).
* Label + filename vocab extended: rhcos/coreos/openshift/okd,
gparted/clonezilla/kali/tails, almalinux/rocky, sles, manjaro.
- NFS + SFTP managers: per-ISO IsoReadAt readers (READ3-at-offset with
short-read looping / seek+read_exact), background introspection pass
after each scan — entries register instantly with a provisional
filename-based report (rev 0, optimistic sanboot preserved) and
upgrade in place as probes land (30s/ISO timeout, failures keep the
provisional). locate_in_iso() exposes the walker to the HTTP layer.
- remote_cache: introspection results persisted per protocol keyed
share/path@size and gated on INTROSPECT_REV — container restarts
re-probe only new/replaced ISOs; upgrades re-probe exactly once.
- SMB: smbclient can't seek, so SMB ISOs get the filename-token family
(rev stays 0 → sanboot entry + "awaiting introspection" label).
- IsoStore::update_external_introspection swaps in completed reports
and regenerates boot entries, preserving category/password.
http-api:
- /iso/{id}/{*path} now serves files from inside NFS/SFTP-hosted ISOs
(remote ISO9660 lookup + ranged share stream) — verified kernel
entries on remote Linux ISOs are actually bootable, end to end.
- iso_fs.rs deleted in favor of the shared iso-store module.
- full_flow fixtures build real directory trees via the shared
test-image builder (new iso-store feature) — a label-only blob no
longer earns a kernel entry, by design.
webui:
- Available images: paged 5 per page with a quiet footer pager
(Showing X–Y of N · Prev/Next), filter-then-paginate, page resets on
search input. Fifty images is five clean pages, not a scroll wall.
- Hosts/Queue profile: "Unattended file (in Storage → Advanced)" so
the picker says where the files live.
- Row badge keys on introspect_rev: probed remote ISOs read like local
ones; un-probed say "awaiting introspection".
Validation: clippy pedantic clean, fmt clean, 316 workspace tests
green (+17: walker, probe shapes incl. gparted regression + CoreOS,
filename table, cache round-trips), webui syntax-checked.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
Miles Ward
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
934cfbab46
commit
6524aa4118
@@ -49,6 +49,10 @@ base64.workspace = true
|
||||
tokio = { workspace = true, features = ["macros", "rt", "rt-multi-thread", "time"] }
|
||||
tower = { workspace = true }
|
||||
tempfile = "3.12"
|
||||
# v0.7.4: probe-based introspection verifies kernel paths against the
|
||||
# real ISO9660 tree, so the full-flow tests synthesize images with the
|
||||
# shared test builder instead of label-only blobs.
|
||||
openpxe-iso-store = { workspace = true, features = ["test-image"] }
|
||||
serde_json = { workspace = true }
|
||||
time = { workspace = true }
|
||||
# v0.4.61: integration tests need to generate real PNG bytes for the
|
||||
|
||||
+94
-17
@@ -19,7 +19,6 @@ use crate::ipxe_script::{
|
||||
render_entry, render_family_menu, render_local_hdd, render_menu, render_nic_info,
|
||||
render_queue_entry, render_shell, render_tools_menu, render_util,
|
||||
};
|
||||
use crate::iso_fs;
|
||||
use crate::log_stream;
|
||||
use crate::state::AppState;
|
||||
use crate::terminal;
|
||||
@@ -36,6 +35,7 @@ use openpxe_core::{
|
||||
LogoSlot, NotifyConfig, Settings, SsoConfig, ALLOWED_LOGO_MIMES, MAX_LOGO_BYTES,
|
||||
};
|
||||
use openpxe_ipxe_assets::asset_slice;
|
||||
use openpxe_iso_store::iso_fs;
|
||||
use openpxe_iso_store::{
|
||||
render_template, IsoCategory, IsoMeta, IsoSource, NfsAddRequest, SftpAddRequest, SmbAddRequest,
|
||||
SmbState, UnattendedKind, UnattendedMeta,
|
||||
@@ -1087,28 +1087,105 @@ async fn iso_file(
|
||||
State(state): State<AppState>,
|
||||
AxumPath((id, path)): AxumPath<(String, String)>,
|
||||
) -> Response {
|
||||
// In-ISO file extraction is only supported for local ISOs — it
|
||||
// needs random-access reads into the ISO9660 directory tree, which
|
||||
// smbclient's whole-file streaming can't do efficiently. SMB-
|
||||
// sourced ISOs use the raw streaming endpoint above instead.
|
||||
let Some(iso_path) = state.iso_store.iso_path_for(&id) else {
|
||||
let Some(meta) = state.iso_store.get(&id) else {
|
||||
return (StatusCode::NOT_FOUND, "no such iso").into_response();
|
||||
};
|
||||
let p = iso_path.clone();
|
||||
let in_path = format!("/{path}");
|
||||
let loc = tokio::task::spawn_blocking(move || iso_fs::lookup(&p, &in_path))
|
||||
.await
|
||||
.ok()
|
||||
.flatten();
|
||||
let Some(loc) = loc else {
|
||||
return (StatusCode::NOT_FOUND, "not found inside iso").into_response();
|
||||
};
|
||||
match stream_byte_range(&iso_path, loc.offset, loc.length).await {
|
||||
Ok(r) => r,
|
||||
Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, format!("{e}")).into_response(),
|
||||
match &meta.source {
|
||||
IsoSource::Local => {
|
||||
let Some(iso_path) = state.iso_store.local_path(&meta) else {
|
||||
return (StatusCode::NOT_FOUND, "no such iso").into_response();
|
||||
};
|
||||
let p = iso_path.clone();
|
||||
let loc = tokio::task::spawn_blocking(move || iso_fs::lookup_local(&p, &in_path))
|
||||
.await
|
||||
.ok()
|
||||
.flatten();
|
||||
let Some(loc) = loc else {
|
||||
return (StatusCode::NOT_FOUND, "not found inside iso").into_response();
|
||||
};
|
||||
match stream_byte_range(&iso_path, loc.offset, loc.length).await {
|
||||
Ok(r) => r,
|
||||
Err(e) => (StatusCode::INTERNAL_SERVER_ERROR, format!("{e}")).into_response(),
|
||||
}
|
||||
}
|
||||
// v0.7.4: remote ISOs serve in-ISO files too — the same ISO9660
|
||||
// walk runs over NFS READ3 / SFTP seek-reads, then the located
|
||||
// byte range streams through the share manager. This is what
|
||||
// makes the verified kernel/initrd boot entries on share-hosted
|
||||
// Linux ISOs actually bootable.
|
||||
IsoSource::Nfs {
|
||||
share_id,
|
||||
relative_path,
|
||||
} => {
|
||||
match state
|
||||
.nfs_shares
|
||||
.locate_in_iso(share_id, relative_path, &in_path)
|
||||
.await
|
||||
{
|
||||
Ok(Some(loc)) => {
|
||||
match state
|
||||
.nfs_shares
|
||||
.stream_iso(share_id, relative_path, loc.offset, Some(loc.length))
|
||||
.await
|
||||
{
|
||||
Ok(stream) => in_iso_stream_response(Body::from_stream(stream), loc.length),
|
||||
Err(e) => {
|
||||
(StatusCode::BAD_GATEWAY, format!("nfs stream: {e}")).into_response()
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(None) => (StatusCode::NOT_FOUND, "not found inside iso").into_response(),
|
||||
Err(e) => (StatusCode::BAD_GATEWAY, format!("nfs lookup: {e}")).into_response(),
|
||||
}
|
||||
}
|
||||
IsoSource::Sftp {
|
||||
share_id,
|
||||
relative_path,
|
||||
} => {
|
||||
match state
|
||||
.sftp_shares
|
||||
.locate_in_iso(share_id, relative_path, &in_path)
|
||||
.await
|
||||
{
|
||||
Ok(Some(loc)) => {
|
||||
match state
|
||||
.sftp_shares
|
||||
.stream_iso(share_id, relative_path, loc.offset, Some(loc.length))
|
||||
.await
|
||||
{
|
||||
Ok(stream) => in_iso_stream_response(Body::from_stream(stream), loc.length),
|
||||
Err(e) => {
|
||||
(StatusCode::BAD_GATEWAY, format!("sftp stream: {e}")).into_response()
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(None) => (StatusCode::NOT_FOUND, "not found inside iso").into_response(),
|
||||
Err(e) => (StatusCode::BAD_GATEWAY, format!("sftp lookup: {e}")).into_response(),
|
||||
}
|
||||
}
|
||||
// smbclient streams sequentially — no seeks, no ISO9660 walk.
|
||||
// SMB ISOs never emit kernel entries, so nothing requests this.
|
||||
IsoSource::Smb { .. } => (
|
||||
StatusCode::NOT_FOUND,
|
||||
"in-ISO files are not available for SMB-sourced ISOs",
|
||||
)
|
||||
.into_response(),
|
||||
}
|
||||
}
|
||||
|
||||
/// 200 response wrapping an in-ISO byte-range stream from a share
|
||||
/// manager. Content-Length is the located file's length — the stream is
|
||||
/// already bounded to exactly that range.
|
||||
fn in_iso_stream_response(body: Body, length: u64) -> Response {
|
||||
Response::builder()
|
||||
.status(StatusCode::OK)
|
||||
.header(header::CONTENT_TYPE, "application/octet-stream")
|
||||
.header(header::CONTENT_LENGTH, length)
|
||||
.body(body)
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
async fn stream_file_range(
|
||||
path: &std::path::Path,
|
||||
range: Option<&HeaderValue>,
|
||||
|
||||
@@ -1,135 +0,0 @@
|
||||
//! Minimal read-only ISO9660 lookup. Given an uploaded ISO file and an
|
||||
//! in-ISO path (e.g. `/casper/vmlinuz`), locate the file and return a
|
||||
//! `(start_byte, length_bytes)` pair so the HTTP handler can stream just
|
||||
//! that range from the on-disk ISO without full extraction.
|
||||
//!
|
||||
//! We only implement what we need: the Primary Volume Descriptor and Rock
|
||||
//! Ridge / Joliet extensions are ignored. Paths are matched case-insensitive
|
||||
//! against plain ISO9660 filenames (uppercase, `;1` version suffix stripped).
|
||||
//! This is sufficient for the kernel/initrd and wimboot files we serve;
|
||||
//! if a requested path isn't found, the handler returns 404 and the user
|
||||
//! can still download the whole ISO via `/iso/<id>.iso`.
|
||||
|
||||
use std::io::{Read, Seek, SeekFrom};
|
||||
use std::path::Path;
|
||||
|
||||
const SECTOR: u64 = 2048;
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct FileLocation {
|
||||
pub offset: u64,
|
||||
pub length: u64,
|
||||
}
|
||||
|
||||
/// Look up `in_iso_path` (leading slash optional, case-insensitive) in the
|
||||
/// ISO at `iso_path`. Returns None on any parsing or IO failure.
|
||||
pub fn lookup(iso_path: &Path, in_iso_path: &str) -> Option<FileLocation> {
|
||||
let mut f = std::fs::File::open(iso_path).ok()?;
|
||||
let root = read_root_directory(&mut f)?;
|
||||
let components: Vec<&str> = in_iso_path
|
||||
.trim_start_matches('/')
|
||||
.split('/')
|
||||
.filter(|c| !c.is_empty())
|
||||
.collect();
|
||||
if components.is_empty() {
|
||||
return None;
|
||||
}
|
||||
walk(&mut f, root.offset, root.length, &components)
|
||||
}
|
||||
|
||||
fn read_root_directory(f: &mut std::fs::File) -> Option<FileLocation> {
|
||||
// Primary Volume Descriptor at LBA 16.
|
||||
let mut pvd = [0u8; 2048];
|
||||
f.seek(SeekFrom::Start(16 * SECTOR)).ok()?;
|
||||
f.read_exact(&mut pvd).ok()?;
|
||||
if pvd[0] != 0x01 || &pvd[1..6] != b"CD001" {
|
||||
return None;
|
||||
}
|
||||
// Root directory record is at offset 156, length 34.
|
||||
let rec = &pvd[156..156 + 34];
|
||||
let (offset, length) = parse_dir_record_ext(rec)?;
|
||||
Some(FileLocation {
|
||||
offset: offset * SECTOR,
|
||||
length,
|
||||
})
|
||||
}
|
||||
|
||||
/// Walk components down the directory tree starting at `dir_offset`.
|
||||
fn walk(
|
||||
f: &mut std::fs::File,
|
||||
dir_offset: u64,
|
||||
dir_len: u64,
|
||||
components: &[&str],
|
||||
) -> Option<FileLocation> {
|
||||
let mut dir = vec![0u8; dir_len as usize];
|
||||
f.seek(SeekFrom::Start(dir_offset)).ok()?;
|
||||
f.read_exact(&mut dir).ok()?;
|
||||
|
||||
let target = components[0];
|
||||
let rest = &components[1..];
|
||||
let mut i = 0;
|
||||
while i < dir.len() {
|
||||
let len = dir[i] as usize;
|
||||
if len == 0 {
|
||||
// Padding to sector boundary.
|
||||
let next = (i / SECTOR as usize + 1) * SECTOR as usize;
|
||||
if next <= i {
|
||||
break;
|
||||
}
|
||||
i = next;
|
||||
continue;
|
||||
}
|
||||
if i + len > dir.len() {
|
||||
break;
|
||||
}
|
||||
let rec = &dir[i..i + len];
|
||||
let name = dir_record_name(rec);
|
||||
let is_dir = (rec.get(25).copied().unwrap_or(0) & 0x02) != 0;
|
||||
// Skip "." (0x00) and ".." (0x01) pseudo-entries.
|
||||
let is_pseudo = matches!(rec.get(32).copied(), Some(1))
|
||||
&& rec.get(33).copied() == Some(0x00)
|
||||
|| matches!(rec.get(32).copied(), Some(1)) && rec.get(33).copied() == Some(0x01);
|
||||
if !is_pseudo && name.eq_ignore_ascii_case(target) {
|
||||
let (child_off, child_len) = parse_dir_record_ext(rec)?;
|
||||
if rest.is_empty() && !is_dir {
|
||||
return Some(FileLocation {
|
||||
offset: child_off * SECTOR,
|
||||
length: child_len,
|
||||
});
|
||||
} else if !rest.is_empty() && is_dir {
|
||||
return walk(f, child_off * SECTOR, child_len, rest);
|
||||
}
|
||||
}
|
||||
i += len;
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Extract (extent LBA, data length in bytes) from a directory record.
|
||||
/// Layout per ISO9660: bytes 2..10 extent LBA (LE+BE duplicate), 10..18
|
||||
/// data length (LE+BE duplicate). We trust the little-endian copy.
|
||||
fn parse_dir_record_ext(rec: &[u8]) -> Option<(u64, u64)> {
|
||||
if rec.len() < 34 {
|
||||
return None;
|
||||
}
|
||||
let lba = u32::from_le_bytes(rec[2..6].try_into().ok()?) as u64;
|
||||
let len = u32::from_le_bytes(rec[10..14].try_into().ok()?) as u64;
|
||||
Some((lba, len))
|
||||
}
|
||||
|
||||
/// Extract the identifier from a directory record, stripping ISO9660's
|
||||
/// `;1` version suffix.
|
||||
fn dir_record_name(rec: &[u8]) -> String {
|
||||
let name_len = *rec.get(32).unwrap_or(&0) as usize;
|
||||
if name_len == 0 || rec.len() < 33 + name_len {
|
||||
return String::new();
|
||||
}
|
||||
let raw = &rec[33..33 + name_len];
|
||||
let s = String::from_utf8_lossy(raw).to_string();
|
||||
// Strip `;N` version suffix.
|
||||
if let Some(i) = s.rfind(';') {
|
||||
s[..i].to_string()
|
||||
} else {
|
||||
s
|
||||
}
|
||||
}
|
||||
@@ -9,8 +9,9 @@
|
||||
//! and Linux kernel/initrd, without having to
|
||||
//! re-extract on every request)
|
||||
//!
|
||||
//! The `<id>/<path>` handler uses a read-only ISO9660 shim (see `iso_fs`)
|
||||
//! that lseeks into the ISO on disk — so we never keep extracted copies.
|
||||
//! The `<id>/<path>` handler uses the read-only ISO9660 walker from
|
||||
//! `openpxe_iso_store::iso_fs` — seeking into the image wherever it
|
||||
//! lives (local disk, NFS, SFTP), so we never keep extracted copies.
|
||||
#![forbid(unsafe_code)]
|
||||
|
||||
pub mod app;
|
||||
@@ -18,7 +19,6 @@ pub mod auth;
|
||||
pub mod error;
|
||||
pub mod grub_script;
|
||||
pub mod ipxe_script;
|
||||
pub mod iso_fs;
|
||||
pub mod log_stream;
|
||||
pub mod notify;
|
||||
pub mod saml_routes;
|
||||
|
||||
@@ -18,23 +18,16 @@ use openpxe_iso_store::{IsoStore, NfsShareManager, SftpShareManager, SmbShareMan
|
||||
use tempfile::tempdir;
|
||||
use tower::ServiceExt;
|
||||
|
||||
/// Build a tiny valid ISO9660 blob with volume label "ALPINE-TEST" so
|
||||
/// introspection identifies it as Alpine.
|
||||
/// Build a tiny Alpine-shaped ISO9660 image: volume label "ALPINE-TEST"
|
||||
/// plus the real `/boot/vmlinuz-lts` + `/boot/initramfs-lts` tree.
|
||||
/// v0.7.4's probe-based introspection verifies those paths exist before
|
||||
/// emitting a kernel boot entry — a label-only blob no longer counts.
|
||||
fn fake_alpine_iso() -> Vec<u8> {
|
||||
let mut buf = vec![0u8; 32 * 2048];
|
||||
let off = 16 * 2048;
|
||||
buf[off] = 0x01;
|
||||
buf[off + 1..off + 6].copy_from_slice(b"CD001");
|
||||
buf[off + 6] = 0x01;
|
||||
let label = b"ALPINE-TEST".to_vec();
|
||||
let mut padded = label.clone();
|
||||
padded.resize(32, b' ');
|
||||
buf[off + 40..off + 40 + 32].copy_from_slice(&padded);
|
||||
let term = 17 * 2048;
|
||||
buf[term] = 0xFF;
|
||||
buf[term + 1..term + 6].copy_from_slice(b"CD001");
|
||||
buf[term + 6] = 0x01;
|
||||
buf
|
||||
openpxe_iso_store::iso_fs::testiso::TestIsoBuilder::new("ALPINE-TEST")
|
||||
.el_torito(true)
|
||||
.file("/boot/vmlinuz-lts", b"fake-kernel-bytes")
|
||||
.file("/boot/initramfs-lts", b"fake-initramfs-bytes")
|
||||
.build()
|
||||
}
|
||||
|
||||
fn multipart_iso_body(filename: &str, bytes: &[u8]) -> (String, Vec<u8>) {
|
||||
@@ -1248,9 +1241,10 @@ async fn chunked_upload_writes_progressively_and_finishes_iso() {
|
||||
.method("POST")
|
||||
.uri("/api/uploads")
|
||||
.header("content-type", "application/json")
|
||||
.body(Body::from(
|
||||
r#"{"filename":"chunked-alpine.iso","size_bytes":65536}"#,
|
||||
))
|
||||
.body(Body::from(format!(
|
||||
r#"{{"filename":"chunked-alpine.iso","size_bytes":{}}}"#,
|
||||
iso.len()
|
||||
)))
|
||||
.unwrap(),
|
||||
)
|
||||
.await
|
||||
|
||||
Reference in New Issue
Block a user