diff --git a/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.i18n.yaml b/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.i18n.yaml index 84bd998a51..217af9cbb3 100644 --- a/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.md -2026-09-16-experimental-voice-input.md: 60a871cc6ba7e604896ce4e339bcfbe45d174f9b -2026-09-16-experimental-voice-input.zh.md: 76d14511d1b014bb6d40cc4bf3e9e6ed19efc3f6 +2026-09-16-experimental-voice-input.md: 06ecb9ff9e83b82147ca2d3ae1386d7ec9c6d382 +2026-09-16-experimental-voice-input.zh.md: f1ad2770f768316f533a081b9dd06ace33e959ec diff --git a/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.md b/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.md index 60a871cc6b..06ecb9ff9e 100644 --- a/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.md +++ b/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.md @@ -14,9 +14,9 @@ Five public packages under `packages/experimental/` compose one optional, defaul Provider ids select exact registrations. Resolution captures an instance, and deregistration closes admission, cancels and joins accepted work. There is no provider fallback: a local failure cannot authorize uploading speech to a cloud service. A new cloud provider supplies its own credentials and explicit processing location through the same registry. -Recordings and transcripts remain transient until the user submits ordinary text. Recognition does not start an agent turn or create an audio Session event. The input facade accepts asynchronous plain text against a captured draft revision, preserving reference chips and undo history. Changed drafts retain the recognized text for explicit insertion instead of replacing later edits. Session changes and plugin disposal invalidate late results and release recording resources. +Recordings and transcripts remain transient until the user submits ordinary text. Recognition does not start an agent turn or create an audio Session event. The input facade accepts asynchronous plain text against a captured draft revision, preserving reference chips and undo history. Changed drafts retain the recognized text for explicit insertion instead of replacing later edits. Session changes and plugin disposal invalidate late results and release recording resources. Capture failures notify the current recording activity immediately, independently of resource-close latency. Cancellation, failure and plugin withdrawal join one release promise, so ownership lasts until the AudioContext closes. -SenseVoice ONNX and Silero VAD run in one managed CPU subprocess. The native sherpa-onnx package includes ONNX Runtime; explicit preparation only downloads and verifies models; subsequent calls reuse the warm model until the configured idle deadline. Every activation verifies cached assets and restores complete caches to standby without starting the worker. Readiness therefore follows disk contents rather than the lifetime of a Provider instance. Inspection uses the same owned task as preparation, so cancellation and disposal join it. Disabling the provider joins its work and process exit while retaining reusable disk assets. Native browser recording and resampling avoid an additional audio conversion executable. The authenticated private loopback protocol carries complete PCM recordings, allowing cancellation to terminate the worker without sharing a long-running inference with another request. +SenseVoice ONNX and Silero VAD run in one managed CPU subprocess. The native sherpa-onnx package includes ONNX Runtime; explicit preparation only downloads and verifies models; subsequent calls reuse the warm model until the configured idle deadline. Every activation verifies cached assets and restores complete caches to standby without starting the worker. Readiness therefore follows disk contents rather than the lifetime of a Provider instance. Recordings wait in the bounded queue during wake-up. The worker checks readiness at execution, so queued work cannot implicitly download resources invalidated by an earlier cache inspection. Inspection uses the same owned task as preparation, so cancellation and disposal join it. Disabling the provider joins its work and process exit while retaining reusable disk assets. Native browser recording and resampling avoid an additional audio conversion executable. The authenticated private loopback protocol carries complete PCM recordings, allowing cancellation to terminate the worker without sharing a long-running inference with another request. The Host owns preparation independently of the view that started it. An ordered step list survives view remounts and records completion only as execution advances. Recognition weights, VAD and final verification have separate operations; downloads use pinned byte counts, while other work reports elapsed time. Preparation status and progress stay in bundle details through `plugins.bundle.config`, keyed by package name. The generic `plugins.bundle.activation` slot carries explicit enablement intent and detail navigation; the voice occupant waits for Host cache inspection and offers installation guidance only for unprepared local models. Enabling an already prepared provider does not prompt or download. A single composer activity Slot expands the voice controls while preserving the draft editor and submit action. The context meter precedes the idle microphone; recording and transcription hide the meter along with ordinary toolbar controls until the activity closes. Shared primitives provide buttons, disclosure and status markers; only measured waveform drawing belongs to the voice UI. Recordings start on click and transcribe on Stop, with feedback inside the activity. Persistent preferences remain in plugin settings. diff --git a/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.zh.md b/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.zh.md index 76d14511d1..f1ad2770f7 100644 --- a/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.zh.md +++ b/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.zh.md @@ -14,9 +14,9 @@ Status: implemented Provider id 选择确切的注册。解析捕获实例,注销先拒绝新任务,再取消并等待已接收的任务。不存在 Provider 回退:本地失败不能授权将语音上传至云端服务。新增云端 Provider 通过同一注册表提供自己的凭据配置与显式处理位置。 -录音与转写在用户提交普通文字前都是临时数据。识别不启动智能体轮次,也不创建音频 Session 事件。输入门面按照捕获的草稿版本接收异步纯文本,保留引用芯片与撤销历史。草稿变化时保留识别文字,等待显式插入,而非替换之后的编辑。切换 Session 和释放插件会使迟到结果失效并释放录音资源。 +录音与转写在用户提交普通文字前都是临时数据。识别不启动智能体轮次,也不创建音频 Session 事件。输入门面按照捕获的草稿版本接收异步纯文本,保留引用芯片与撤销历史。草稿变化时保留识别文字,等待显式插入,而非替换之后的编辑。切换 Session 和释放插件会使迟到结果失效并释放录音资源。采集失败立即通知当前录音活动,不受资源关闭耗时影响。取消、失败和插件撤销共同等待同一个释放 Promise,所有权持续到 AudioContext 关闭。 -SenseVoice ONNX 与 Silero VAD 在一个托管 CPU 子进程中运行。原生 sherpa-onnx 包包含 ONNX Runtime;显式准备只下载并校验模型;后续调用复用预热模型,直到配置的空闲截止时间。每次启用都会校验缓存资源,完整缓存恢复为待唤醒状态,不启动工作进程。就绪状态因此取决于磁盘内容,而非 Provider 实例的存活时间。检查复用准备任务的生命周期,取消与释放都会等待检查结束。禁用 Provider 等待其任务和进程退出,同时保留可复用的磁盘资源。浏览器原生录音与重采样避免增加音频转换可执行文件。带认证的私有环回协议传输完整 PCM 录音,取消时可以终止工作进程,而不与另一请求共享长时间推理。 +SenseVoice ONNX 与 Silero VAD 在一个托管 CPU 子进程中运行。原生 sherpa-onnx 包包含 ONNX Runtime;显式准备只下载并校验模型;后续调用复用预热模型,直到配置的空闲截止时间。每次启用都会校验缓存资源,完整缓存恢复为待唤醒状态,不启动工作进程。就绪状态因此取决于磁盘内容,而非 Provider 实例的存活时间。录音在唤醒期间进入有界队列等待。工作进程在执行时检查就绪状态,因此较早的缓存检查使资源失效后,排队任务也不能隐式下载资源。检查复用准备任务的生命周期,取消与释放都会等待检查结束。禁用 Provider 等待其任务和进程退出,同时保留可复用的磁盘资源。浏览器原生录音与重采样避免增加音频转换可执行文件。带认证的私有环回协议传输完整 PCM 录音,取消时可以终止工作进程,而不与另一请求共享长时间推理。 Host 独立于发起视图拥有准备任务。有序步骤列表在视图重新挂载后保留,仅随着执行推进记录完成。识别模型、VAD 与最终校验分别准备;下载使用固定字节数,其他工作报告等待时间。准备状态和进度通过以包名为 key 的 `plugins.bundle.config` 显示在详情页。通用 `plugins.bundle.activation` Slot 传递显式启用意图和详情导航;语音贡献等待 Host 缓存检查,只为尚未准备的本地模型提供安装引导。启用已准备的 Provider 不弹窗也不下载。单个输入框活动 Slot 展开语音控件,同时保留草稿编辑器和提交按钮。空闲时上下文用量按钮位于麦克风之前;录音和转写期间,用量按钮与普通工具栏控件一起隐藏,关闭活动后恢复。按钮、折叠与状态标记复用共享原语,语音 UI 只负责实测波形绘制。点击开始录音,停止后转写,反馈留在活动栏内。持久化偏好仍在插件设置中。 diff --git a/apps/web/tests/voice-input.e2e.ts b/apps/web/tests/voice-input.e2e.ts index 64f45f6943..4dbd0a4405 100644 --- a/apps/web/tests/voice-input.e2e.ts +++ b/apps/web/tests/voice-input.e2e.ts @@ -15,6 +15,7 @@ import { connectFreshWorkspace, newEnglishPage } from './support.ts' const fixture = fileURLToPath(new URL('../../../snapshots/web/fresh-round-trip/session.v3.jsonl', import.meta.url)) const expected = fileURLToPath(new URL('../../../snapshots/web/voice-input/ui.expected.md', import.meta.url)) +const interruptedExpected = fileURLToPath(new URL('../../../snapshots/web/voice-input/interrupted.expected.md', import.meta.url)) const recordingExpected = fileURLToPath(new URL('../../../snapshots/web/voice-input/recording.expected.md', import.meta.url)) const bundle = fileURLToPath(new URL('../../../packages/experimental/voice-input-bundle', import.meta.url)) @@ -57,6 +58,13 @@ it.skipIf(webSnapshotMode() === 'record')('records from cached standby and submi const browser = await chromium.launch({ args: ['--use-fake-ui-for-media-stream', '--use-fake-device-for-media-stream'] }) resources.browser = browser const page = await newEnglishPage(browser), tripwire = watchConsole(page) + await page.addInitScript(() => { + const holder = window as Window & { voiceTestRecorder?: MediaRecorder } + const NativeRecorder = window.MediaRecorder + window.MediaRecorder = class extends NativeRecorder { + constructor(stream: MediaStream, options?: MediaRecorderOptions) { super(stream, options); holder.voiceTestRecorder = this } + } + }) await page.goto(scaffold.authenticatedUrl) await connectFreshWorkspace(page, scaffold.workspaceCwd) const input = page.locator('[data-composer-input]'), mic = page.getByRole('button', { name: 'Start recording', exact: true }) @@ -117,6 +125,21 @@ it.skipIf(webSnapshotMode() === 'record')('records from cached standby and submi await meter.click() await page.getByRole('dialog', { name: 'of context used', exact: true }).waitFor() await page.keyboard.press('Escape') + await mic.click() + await stop.waitFor() + await page.evaluate(() => { + const recorder = (window as Window & { voiceTestRecorder?: MediaRecorder }).voiceTestRecorder + if (!recorder) throw new Error('No active test microphone') + recorder.dispatchEvent(new Event('error')) + }) + await page.getByRole('button', { name: 'Record again', exact: true }).waitFor() + expect(await stop.count()).toBe(0) + await compareOrRefreshGolden(interruptedExpected, + await captureStableAria(page, '[data-composer-card]', scaffold.workspaceCwd), webSnapshotMode()) + expect(recognize).toHaveBeenCalledOnce() + await page.getByRole('button', { name: 'Record again', exact: true }).click() + await stop.waitFor() + await page.keyboard.press('Escape') expect(messages).toBe(1) expect(tripwire.pageErrors).toEqual([]) }, 120_000) diff --git a/packages/experimental/client-ui-voice-input/README.i18n.yaml b/packages/experimental/client-ui-voice-input/README.i18n.yaml index 31bed298b1..a65ffca160 100644 --- a/packages/experimental/client-ui-voice-input/README.i18n.yaml +++ b/packages/experimental/client-ui-voice-input/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/experimental/client-ui-voice-input/README.md -README.md: c126a2cea4f39e5bea1328687a288322a4e3cce2 -README.zh.md: afe709e9119abd1fea30c9d73849c8518cbd9a2d +README.md: b37a39c2885063c6597b0b300e0840873ea7fe7d +README.zh.md: f052c2338c04eb0eef3a78925684f14cf522de62 diff --git a/packages/experimental/client-ui-voice-input/README.md b/packages/experimental/client-ui-voice-input/README.md index c126a2cea4..b37a39c288 100644 --- a/packages/experimental/client-ui-voice-input/README.md +++ b/packages/experimental/client-ui-voice-input/README.md @@ -35,7 +35,7 @@ Enable the [voice input bundle](../voice-input-bundle/README.md) from Plugins. A
Maintainer details — click to expand -Native MediaRecorder captures audio and Web Audio converts it to the Host PCM format after flushing the final recorder chunks. The original editor selection carries a draft revision: changed drafts retain the transcript for explicit insertion or discard beside the microphone. Session changes, hiding the page during capture and plugin disposal invalidate late results and release tracks. One plugin-owned readiness subscription serves the composer, installation prompt and details. These views use the existing Slot lifecycle and own no preparation tasks. No runtime invariant companion is published because readiness comes from one Host subscription and recording state belongs to one capture operation. +Native MediaRecorder captures audio and Web Audio converts it to the Host PCM format after flushing the final recorder chunks. Capture failures immediately end the recording UI and offer retry inside the toolbar. Cancellation, failure and plugin disposal share one resource-release promise; the plugin retains ownership until AudioContext closure settles. The original editor selection carries a draft revision: changed drafts retain the transcript for explicit insertion or discard beside the microphone. Session changes, hiding the page during capture and plugin disposal invalidate late results and release tracks. One plugin-owned readiness subscription serves the composer, installation prompt and details. These views use the existing Slot lifecycle and own no preparation tasks. No runtime invariant companion is published because readiness comes from one Host subscription and recording state belongs to one capture operation.
diff --git a/packages/experimental/client-ui-voice-input/README.zh.md b/packages/experimental/client-ui-voice-input/README.zh.md index afe709e911..f052c2338c 100644 --- a/packages/experimental/client-ui-voice-input/README.zh.md +++ b/packages/experimental/client-ui-voice-input/README.zh.md @@ -35,7 +35,7 @@ kind: "package-reference"
维护者信息 — 点击展开 -原生 MediaRecorder 捕获音频,等待最后一块数据后由 Web Audio 转换为 Host 的 PCM 格式。原编辑器选区携带草稿版本:草稿变化时,转写文字保留在麦克风旁,供显式插入或丢弃。切换 Session、录音期间隐藏页面和释放插件会使迟到结果失效并释放轨道。插件拥有的一份准备状态订阅服务于输入框、安装引导和详情。这些视图使用既有 Slot 生命周期,不拥有准备任务。不发布运行时不变量伴随模块,因为准备状态来自一份 Host 订阅,录音状态属于一个捕获操作。 +原生 MediaRecorder 捕获音频,等待最后一块数据后由 Web Audio 转换为 Host 的 PCM 格式。采集失败立即结束录音界面,并在活动栏内提供重试。取消、失败和插件释放共享同一个资源释放 Promise;插件保留所有权,直到 AudioContext 关闭任务结束。原编辑器选区携带草稿版本:草稿变化时,转写文字保留在麦克风旁,供显式插入或丢弃。切换 Session、录音期间隐藏页面和释放插件会使迟到结果失效并释放轨道。插件拥有的一份准备状态订阅服务于输入框、安装引导和详情。这些视图使用既有 Slot 生命周期,不拥有准备任务。不发布运行时不变量伴随模块,因为准备状态来自一份 Host 订阅,录音状态属于一个捕获操作。
diff --git a/packages/experimental/client-ui-voice-input/src/client/VoiceInput.tsx b/packages/experimental/client-ui-voice-input/src/client/VoiceInput.tsx index 977810e3ad..b3be6ed7c9 100644 --- a/packages/experimental/client-ui-voice-input/src/client/VoiceInput.tsx +++ b/packages/experimental/client-ui-voice-input/src/client/VoiceInput.tsx @@ -54,7 +54,8 @@ export function VoiceInput({ sessionId, inputActions, locked, onActiveChange, createRecording, transcribe, useSpeechReadiness, t }: VoiceInputProps) { const readiness = useSpeechReadiness(value => value), catalog = readiness.catalog const provider = catalog?.providers.find(item => item.id === catalog.selection.providerId) - const usable = readiness.connected && (provider?.preparation.phase === 'ready' || provider?.preparation.phase === 'standby') + const usable = readiness.connected && (provider?.preparation.phase === 'ready' || provider?.preparation.phase === 'standby' + || provider?.preparation.phase === 'waking') const [phase, setPhase] = useState('idle'), [message, setMessage] = useState(''), [pending, setPending] = useState('') const current = useRef(), generation = useRef(0) const expanded = phase !== 'idle' @@ -119,8 +120,13 @@ export function VoiceInput({ sessionId, inputActions, locked, onActiveChange, current.current = active setMessage(''); setPending(''); setPhase('requesting') try { - await active.capture.start() - if (run !== generation.current) return + await active.capture.start((failure) => { + if (run !== generation.current || current.current !== active || active.finishing) return + current.current = undefined + clearTimeout(active.timer); active.abort.abort() + feedback(failureText(failure)) + }) + if (run !== generation.current || current.current !== active) return setPhase('recording') active.timer = setTimeout(() => { void finish() }, active.maxDurationSeconds * 1000) } catch (failure) { diff --git a/packages/experimental/client-ui-voice-input/src/client/audio.ts b/packages/experimental/client-ui-voice-input/src/client/audio.ts index bc63b0e910..b5147e4cde 100644 --- a/packages/experimental/client-ui-voice-input/src/client/audio.ts +++ b/packages/experimental/client-ui-voice-input/src/client/audio.ts @@ -2,7 +2,7 @@ /** Capture failure whose message is localized by the caller. */ export class RecordingError extends Error { - constructor(readonly kind: 'unavailable' | 'permission' | 'empty' | 'cancelled') { super(kind); this.name = 'RecordingError' } + constructor(readonly kind: 'unavailable' | 'permission' | 'empty' | 'cancelled' | 'interrupted') { super(kind); this.name = 'RecordingError' } } /** @@ -46,14 +46,16 @@ export class Recording { private samples = new Float32Array(256) private chunks: Blob[] = [] private readonly lifetime = new AbortController() + private disposal: Promise | undefined constructor(private readonly onDispose: () => void) {} /** * Acquire the microphone for this recording. + * @param onError - receives failures during capture, before asynchronous resource release finishes. * @returns after capture starts; a cancelled permission grant immediately releases its tracks. */ - async start(): Promise { + async start(onError?: (error: RecordingError) => void): Promise { const devices = (navigator as Partial).mediaDevices if (!devices || typeof MediaRecorder === 'undefined') throw new RecordingError('unavailable') let stream: MediaStream @@ -71,8 +73,14 @@ export class Recording { this.analyser.fftSize = this.samples.length this.context.createMediaStreamSource(stream).connect(this.analyser) this.recorder = new MediaRecorder(stream) - this.recorder.ondataavailable = (event) => { if (event.data.size > 0) this.chunks.push(event.data) } - this.recorder.onerror = () => { void this.dispose().catch(() => undefined) } + this.recorder.ondataavailable = (event) => { if (!this.lifetime.signal.aborted && event.data.size > 0) this.chunks.push(event.data) } + this.recorder.onerror = () => { + if (this.lifetime.signal.aborted) return + void this.dispose().catch(() => undefined) + try { onError?.(new RecordingError('interrupted')) } catch (error) { + console.error('Speech recording error handler failed', error) + } + } this.recorder.start() } catch (error) { await this.dispose(); throw error } } @@ -121,8 +129,20 @@ export class Recording { } finally { await this.dispose() } } - /** Release this recording. @returns after capture stops and the AudioContext closes; a pending permission prompt is invalidated. */ - async dispose(): Promise { + /** + * Release this recording and invalidate pending permission grants. + * @returns the shared release promise, including any AudioContext close failure. + */ + dispose(): Promise { + if (!this.disposal) { + const closing = Promise.withResolvers() + this.disposal = closing.promise + void this.release().then(closing.resolve, closing.reject) + } + return this.disposal + } + + private async release(): Promise { this.lifetime.abort(new RecordingError('cancelled')) if (this.recorder?.state === 'recording') this.recorder.stop() this.stream?.getTracks().forEach((track) => { track.stop() }) diff --git a/packages/experimental/client-ui-voice-input/src/client/locales.ts b/packages/experimental/client-ui-voice-input/src/client/locales.ts index 9b9c33cb99..09a7a91dd7 100644 --- a/packages/experimental/client-ui-voice-input/src/client/locales.ts +++ b/packages/experimental/client-ui-voice-input/src/client/locales.ts @@ -60,6 +60,7 @@ export const zh = { auto: '自动识别', zh: '中文', en: '英语', yue: '粤语', ja: '日语', ko: '韩语', start: '开始录音', stop: '停止并识别', cancel: '取消', insert: '插入文字', loading: '正在读取识别服务…', requesting: '请允许使用麦克风…', recording: '正在录音…', + interrupted: '录音中断,请重试。', transcribing: '正在识别语音…', local: '音频在运行 DSH 的机器上识别。', cloud: '音频将发送至所选云端服务。', idle: '停止录音后,识别文字会插入草稿,由你确认发送。', @@ -131,6 +132,7 @@ export const en: Record = { auto: 'Detect automatically', zh: 'Chinese', en: 'English', yue: 'Cantonese', ja: 'Japanese', ko: 'Korean', start: 'Start recording', stop: 'Stop and transcribe', cancel: 'Cancel', insert: 'Insert text', loading: 'Loading recognition services…', requesting: 'Allow microphone access to continue…', recording: 'Recording…', + interrupted: 'Recording was interrupted. Please try again.', transcribing: 'Transcribing speech…', local: 'Audio is recognized on the machine running DSH.', cloud: 'Audio will be sent to the selected cloud service.', idle: 'Stop recording to insert the transcript into your draft. Review it before sending.', diff --git a/packages/experimental/client-ui-voice-input/tests/audio-fixture.client.ts b/packages/experimental/client-ui-voice-input/tests/audio-fixture.client.ts new file mode 100644 index 0000000000..6e8b2fad64 --- /dev/null +++ b/packages/experimental/client-ui-voice-input/tests/audio-fixture.client.ts @@ -0,0 +1,48 @@ +/** Controlled browser audio devices with real Recording ownership and conversion. */ +import { vi } from 'vitest' +import { Recording } from '../src/client/audio.ts' + +/** + * Install per-test audio devices; callers restore globals and release acquired recordings. + * @param options - device failures or an empty final recording. + * @returns the recording, device controls, and resource observations. + */ +export function captureFixture(options: { empty?: boolean; recorderError?: boolean; constructError?: boolean } = {}) { + const trackStop = vi.fn(), close = vi.fn(async () => {}), disposed = vi.fn() + const decoding = vi.fn(async (_data: ArrayBuffer) => ({ duration: 2 })) + const rendering = vi.fn(async () => ({ getChannelData: () => new Float32Array([0.5, -0.5]) })) + let failRecorder: () => void + class Recorder { + state = 'inactive' + mimeType = 'audio/webm' + ondataavailable?: (event: { data: Blob }) => void + onstop?: () => void + onerror?: () => void + constructor() { + if (options.constructError) throw new Error('recorder unavailable') + failRecorder = () => { this.onerror!() } + } + start() { this.state = 'recording' } + stop() { + this.state = 'inactive' + queueMicrotask(() => { + this.ondataavailable?.({ data: new Blob() }) + if (!options.empty) this.ondataavailable?.({ data: new Blob(['final audio']) }) + if (options.recorderError) this.onerror?.() + else this.onstop?.() + }) + } + } + const offline = vi.fn(function Offline(_channels: number, _frames: number, _rate: number) { + return { destination: {}, createBufferSource: () => ({ buffer: null, connect() {}, start() {} }), startRendering: rendering } + }) + vi.stubGlobal('navigator', { mediaDevices: { getUserMedia: async () => ({ getTracks: () => [{ stop: trackStop }] }) } }) + vi.stubGlobal('MediaRecorder', Recorder) + vi.stubGlobal('AudioContext', function Audio() { return { state: 'running', close, decodeAudioData: decoding, + createMediaStreamSource: () => ({ connect: vi.fn() }), + createAnalyser: () => ({ fftSize: 256, getFloatTimeDomainData: (buffer: Float32Array) => { buffer.fill(0.25) } }), + } }) + vi.stubGlobal('OfflineAudioContext', offline) + return { recording: new Recording(disposed), trackStop, close, disposed, decoding, rendering, offline, + failRecorder: () => { failRecorder() } } +} diff --git a/packages/experimental/client-ui-voice-input/tests/audio.client.spec.ts b/packages/experimental/client-ui-voice-input/tests/audio.client.spec.ts index 9aefa4263f..8362d4c10e 100644 --- a/packages/experimental/client-ui-voice-input/tests/audio.client.spec.ts +++ b/packages/experimental/client-ui-voice-input/tests/audio.client.spec.ts @@ -1,6 +1,7 @@ /** Browser recording bytes conform to the Host's fixed PCM format. */ import { afterEach, expect, it, vi } from 'vitest' import { audioBase64, encodeWave, Recording, RecordingError } from '../src/client/audio.ts' +import { captureFixture } from './audio-fixture.client.ts' afterEach(() => { vi.unstubAllGlobals() }) @@ -41,45 +42,6 @@ it('releases a microphone granted after cancellation', async () => { expect(dispose).toHaveBeenCalledOnce() }) -function captureFixture(options: { empty?: boolean; recorderError?: boolean; constructError?: boolean } = {}) { - const trackStop = vi.fn(), close = vi.fn(async () => {}), disposed = vi.fn() - const decoding = vi.fn(async (_data: ArrayBuffer) => ({ duration: 2 })) - const rendering = vi.fn(async () => ({ getChannelData: () => new Float32Array([0.5, -0.5]) })) - let failRecorder: () => void - class Recorder { - state = 'inactive' - mimeType = 'audio/webm' - ondataavailable?: (event: { data: Blob }) => void - onstop?: () => void - onerror?: () => void - constructor() { - if (options.constructError) throw new Error('recorder unavailable') - failRecorder = () => { this.onerror!() } - } - start() { this.state = 'recording' } - stop() { - this.state = 'inactive' - queueMicrotask(() => { - this.ondataavailable?.({ data: new Blob() }) - if (!options.empty) this.ondataavailable?.({ data: new Blob(['final audio']) }) - if (options.recorderError) this.onerror?.() - else this.onstop?.() - }) - } - } - const offline = vi.fn(function Offline(_channels: number, _frames: number, _rate: number) { - return { destination: {}, createBufferSource: () => ({ buffer: null, connect() {}, start() {} }), startRendering: rendering } - }) - vi.stubGlobal('navigator', { mediaDevices: { getUserMedia: async () => ({ getTracks: () => [{ stop: trackStop }] }) } }) - vi.stubGlobal('MediaRecorder', Recorder) - vi.stubGlobal('AudioContext', function Audio() { return { state: 'running', close, decodeAudioData: decoding, - createMediaStreamSource: () => ({ connect: vi.fn() }), - createAnalyser: () => ({ fftSize: 256, getFloatTimeDomainData: (buffer: Float32Array) => { buffer.fill(0.25) } }), - } }) - vi.stubGlobal('OfflineAudioContext', offline) - return { recording: new Recording(disposed), trackStop, close, disposed, decoding, rendering, offline, - failRecorder: () => { failRecorder() } } -} it('flushes final recording bytes, bounds timer overshoot and closes audio resources', async () => { const b = captureFixture() @@ -124,5 +86,51 @@ it('releases active capture on a spontaneous recorder error even when AudioConte b.failRecorder() await vi.waitFor(() => { expect(b.disposed).toHaveBeenCalledOnce() }) expect(b.trackStop).toHaveBeenCalled() - await expect(b.recording.stop(120)).rejects.toMatchObject({ kind: 'empty' }) + await expect(b.recording.dispose()).rejects.toThrow('audio device disconnected') +}) + +it.each([false, true])('shares pending AudioContext closure across repeated release (failure=%s)', async (failed) => { + const b = captureFixture(), closing = Promise.withResolvers() + b.close.mockReturnValueOnce(closing.promise) + await b.recording.start() + const first = b.recording.dispose(), second = b.recording.dispose() + const results = Promise.allSettled([first, second]) + try { + expect(first).toBe(second) + expect(b.trackStop).toHaveBeenCalledOnce() + expect(b.close).toHaveBeenCalledOnce() + expect(b.disposed).not.toHaveBeenCalled() + } finally { + if (failed) closing.reject(new Error('audio close failed')) + else closing.resolve(undefined) + await results + } + expect(b.disposed).toHaveBeenCalledOnce() + expect(b.recording.dispose()).toBe(first) + expect((await results).map(result => result.status)).toEqual(failed ? ['rejected', 'rejected'] : ['fulfilled', 'fulfilled']) +}) + +it('reports a device interruption once while resource closure is still pending', async () => { + const b = captureFixture(), closing = Promise.withResolvers(), failure = vi.fn() + b.close.mockReturnValueOnce(closing.promise) + await b.recording.start(failure) + try { + b.failRecorder(); b.failRecorder() + expect(failure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ kind: 'interrupted' })) + expect(b.trackStop).toHaveBeenCalledOnce() + expect(b.disposed).not.toHaveBeenCalled() + } finally { closing.resolve(undefined); await b.recording.dispose() } + expect(b.disposed).toHaveBeenCalledOnce() +}) + +it('contains a failing interruption callback and still releases the microphone', async () => { + const b = captureFixture(), error = new Error('callback failed'), report = vi.spyOn(console, 'error').mockImplementation(() => {}) + try { + await b.recording.start(() => { throw error }) + b.failRecorder() + await b.recording.dispose() + expect(b.trackStop).toHaveBeenCalledOnce() + expect(b.disposed).toHaveBeenCalledOnce() + expect(report).toHaveBeenCalledWith('Speech recording error handler failed', error) + } finally { report.mockRestore(); await b.recording.dispose() } }) diff --git a/packages/experimental/client-ui-voice-input/tests/mount.client.spec.ts b/packages/experimental/client-ui-voice-input/tests/mount.client.spec.ts index e660132aa1..088d526d23 100644 --- a/packages/experimental/client-ui-voice-input/tests/mount.client.spec.ts +++ b/packages/experimental/client-ui-voice-input/tests/mount.client.spec.ts @@ -9,6 +9,7 @@ import { createSnapshotStore } from '@deepseek-ai/dsh-client-store' import { mountVoiceInput, inject } from '../src/client/mount.ts' import { apply as hostApply } from '../src/index.ts' import { VoiceInput, type VoiceInputInjected } from '../src/client/VoiceInput.tsx' +import { captureFixture } from './audio-fixture.client.ts' const REMOTE: TypertRemoteContribution = { package: '@deepseek-ai/dsh-experimental-api-speech-to-text', @@ -94,3 +95,37 @@ it('rolls back the Remote contribution when the slot registration fails', async expect(b.unmount).toHaveBeenCalledOnce() } finally { await b.ctx.fiber.dispose() } }) + +it('joins the same audio closure when cancellation overlaps Client plugin withdrawal', async () => { + const b = await fixture(), audio = captureFixture(), closing = Promise.withResolvers() + audio.close.mockReturnValueOnce(closing.promise) + let cancelled: Promise | undefined, removed: Promise | undefined + try { + const fiber = b.ctx.plugin({ inject: [...inject], apply: ctx => mountVoiceInput(ctx, REMOTE) }) + await fiber + const entry = b.ctx.slots.entries('conversation.input.activity').find(item => item.component === VoiceInput)! + const actions = (entry.inject as unknown as () => VoiceInputInjected)() + const recording = actions.createRecording() + await recording.start() + cancelled = recording.dispose() + const originalDispose = recording.dispose.bind(recording), joined = Promise.withResolvers<{ pending: Promise }>() + vi.spyOn(recording, 'dispose').mockImplementationOnce(() => { + const pending = originalDispose() + joined.resolve({ pending }) + return pending + }) + removed = fiber.dispose() + const { pending } = await joined.promise + expect(pending).toBe(cancelled) + expect(b.unmount).not.toHaveBeenCalled() + closing.resolve(undefined) + await cancelled; await removed + expect(audio.close).toHaveBeenCalledOnce() + expect(b.unmount).toHaveBeenCalledOnce() + } finally { + closing.resolve(undefined) + try { await cancelled; await removed } finally { + try { await b.ctx.fiber.dispose() } finally { vi.unstubAllGlobals() } + } + } +}) diff --git a/packages/experimental/client-ui-voice-input/tests/voice-input.client.spec.tsx b/packages/experimental/client-ui-voice-input/tests/voice-input.client.spec.tsx index 9bb28bcc89..d9252d766d 100644 --- a/packages/experimental/client-ui-voice-input/tests/voice-input.client.spec.tsx +++ b/packages/experimental/client-ui-voice-input/tests/voice-input.client.spec.tsx @@ -12,13 +12,14 @@ import { VoiceInput, type VoiceInputProps } from '../src/client/VoiceInput.tsx' import { RecordingError, type Recording } from '../src/client/audio.ts' import type { SpeechReadiness } from '../src/client/readiness.ts' import { zh } from '../src/client/locales.ts' +import { captureFixture } from './audio-fixture.client.ts' beforeEach(() => { vi.stubGlobal('requestAnimationFrame', vi.fn(() => 1)); vi.stubGlobal('cancelAnimationFrame', vi.fn()) }) afterEach(() => { cleanup(); vi.useRealTimers(); vi.unstubAllGlobals() }) const id = 'sensevoice-local' as SpeechProviderId const transcript: Transcript = { text: '检查 TypeScript 类型', audioSeconds: 2, inferenceSeconds: 0.4 } -function fixture() { - const capture = { start: vi.fn(async () => {}), stop: vi.fn(async () => new Uint8Array(48)), +function fixture(recording?: Recording) { + const capture = { start: vi.fn(async () => {}), stop: vi.fn(async () => new Uint8Array(48)), amplitude: () => 0, dispose: vi.fn(async () => {}) } const inputActions = { notify: vi.fn(), captureInsertion: vi.fn(() => ({ start: 3, end: 3, draftRev: 1 })), insertText: vi.fn(() => true), setDraft: vi.fn(), addAttachments: vi.fn(() => true), removeAttachment: vi.fn(), pruneAttachments: vi.fn(), submit: vi.fn() } @@ -30,7 +31,7 @@ function fixture() { async () => ({ ok: true, value: transcript })) const props: VoiceInputProps = { sessionId: 'one' as SessionId, inputActions, transcribe, locked: false, onActiveChange: vi.fn(), prepare: vi.fn(async () => {}), cancelPreparation: vi.fn(async () => {}), configure: vi.fn(async () => {}), - useSpeechReadiness: bindSnapshotSelector(readiness), createRecording: () => capture as unknown as Recording, + useSpeechReadiness: bindSnapshotSelector(readiness), createRecording: () => recording ?? capture as unknown as Recording, t: makeTranslate(zh, commonZh) } const view = render() return { props, capture, inputActions, readiness, transcribe, view } @@ -201,3 +202,57 @@ it('coalesces repeated capture and stop clicks before the toolbar rerenders', as expect(b.capture.start).toHaveBeenCalledOnce() expect(b.capture.stop).toHaveBeenCalledOnce() }) + +it('ends the recording UI immediately on a device failure and clears automatic transcription', async () => { + const devices = captureFixture(), b = fixture(devices.recording), closing = Promise.withResolvers() + devices.close.mockReturnValueOnce(closing.promise) + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }) + try { + fireEvent.click(screen.getByRole('button', { name: zh.start })) + await act(async () => {}) + expect(screen.getByRole('button', { name: zh.stop })).toBeTruthy() + act(() => { devices.failRecorder() }) + expect(screen.queryByRole('button', { name: zh.stop })).toBeNull() + expect(screen.queryByRole('img', { name: zh.recording })).toBeNull() + expect(screen.getByRole('status').textContent).toBe(zh.interrupted) + expect(devices.disposed).not.toHaveBeenCalled() + await act(async () => { await vi.advanceTimersByTimeAsync(120000) }) + expect(b.transcribe).not.toHaveBeenCalled() + expect(b.inputActions.notify).not.toHaveBeenCalled() + } finally { closing.resolve(undefined); await devices.recording.dispose() } +}) + +it.each(['cancel', 'session', 'unmount', 'finishing', 'finished'])('ignores interruption notifications after %s', async (action) => { + const b = fixture() + await start() + const failed = b.capture.start.mock.calls[0]![0]! + if (action === 'cancel') fireEvent.click(screen.getByRole('button', { name: zh.cancel })) + if (action === 'session') b.view.rerender() + if (action === 'unmount') b.view.unmount() + const pending = Promise.withResolvers>() + if (action === 'finishing') b.transcribe.mockReturnValueOnce(pending.promise) + if (action === 'finishing' || action === 'finished') { stop(); await act(async () => {}) } + try { + act(() => { failed(new RecordingError('interrupted')) }) + expect(screen.queryByText(zh.interrupted)).toBeNull() + expect(b.inputActions.notify).not.toHaveBeenCalled() + } finally { pending.resolve({ ok: true, value: transcript }); await act(async () => {}) } +}) + +it('does not restore recording when capture reports failure before start settles', async () => { + const b = fixture() + b.capture.start.mockImplementationOnce(async (failed) => { failed!(new RecordingError('interrupted')) }) + fireEvent.click(screen.getByRole('button', { name: zh.start })) + await act(async () => {}) + expect(screen.getByRole('status').textContent).toBe(zh.interrupted) + expect(screen.queryByRole('button', { name: zh.stop })).toBeNull() + expect(b.transcribe).not.toHaveBeenCalled() +}) + +it('allows recording while verified local resources are waking', async () => { + const b = fixture(), state = b.readiness.getSnapshot() + act(() => { b.readiness.set({ ...state, catalog: { ...state.catalog!, providers: [{ ...state.catalog!.providers[0]!, preparation: { phase: 'waking' } }] } }) }) + await start() + stop() + await waitFor(() => { expect(b.inputActions.insertText).toHaveBeenCalledWith(transcript.text, expect.anything()) }) +}) diff --git a/packages/experimental/speech-to-text-sensevoice/README.i18n.yaml b/packages/experimental/speech-to-text-sensevoice/README.i18n.yaml index 740410587c..751ef66b7c 100644 --- a/packages/experimental/speech-to-text-sensevoice/README.i18n.yaml +++ b/packages/experimental/speech-to-text-sensevoice/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/experimental/speech-to-text-sensevoice/README.md -README.md: 62912955502c16504838a91e2dc866d31d7170ff -README.zh.md: 9251ac78107d7003d7aa47a8af5616685592d684 +README.md: d8d26118ee5ae6632db0c59e67e3f405d89e85e1 +README.zh.md: c6d992c2828d624d00cefc2a62cc0043edf7cada diff --git a/packages/experimental/speech-to-text-sensevoice/README.md b/packages/experimental/speech-to-text-sensevoice/README.md index 6291295550..d8d26118ee 100644 --- a/packages/experimental/speech-to-text-sensevoice/README.md +++ b/packages/experimental/speech-to-text-sensevoice/README.md @@ -37,7 +37,7 @@ Every activation checks the selected model, tokens and VAD on disk. Complete ver
Maintainer details — click to expand -One managed Node subprocess serializes inference. `threads`, `segmentSeconds`, `vadThreshold`, `minSpeechSeconds` and `minSilenceSeconds` tune CPU inference and segmentation; `maxPending` bounds accepted work. Preparation and inference have separate deadlines. Cancellation terminates an active worker and joins its process range; cancelled queued recordings never run. `idleTimeoutMs` releases the worker after inactivity, with zero retaining it until disposal. Models remain cached. Authenticated loopback requests keep audio in memory. Electron workers run in Node mode and copy VAD buffers to satisfy the V8 memory cage. No runtime invariant companion is published because `dsh-subprocess` owns process-range observations and this provider has no independent projection to reconcile. +One managed Node subprocess serializes inference. Recordings submitted during wake-up join the bounded queue. At execution, the worker requires ready or standby resources. Transcription never starts preparation downloads. `threads`, `segmentSeconds`, `vadThreshold`, `minSpeechSeconds` and `minSilenceSeconds` tune CPU inference and segmentation; `maxPending` bounds accepted work. Preparation and inference have separate deadlines. Cancellation terminates an active worker and joins its process range; cancelled queued recordings never run. `idleTimeoutMs` releases the worker after inactivity, with zero retaining it until disposal. Models remain cached. Authenticated loopback requests keep audio in memory. Electron workers run in Node mode and copy VAD buffers to satisfy the V8 memory cage. No runtime invariant companion is published because `dsh-subprocess` owns process-range observations and this provider has no independent projection to reconcile. The Host owns preparation across page and Session changes: check resources, download recognition weights and tokens, download VAD, verify files, and load the worker. Explicit model paths omit their download steps. Downloads report bytes; other steps report elapsed time. An idle worker retains prepared paths and needs only model loading on wake-up. diff --git a/packages/experimental/speech-to-text-sensevoice/README.zh.md b/packages/experimental/speech-to-text-sensevoice/README.zh.md index 9251ac7810..c6d992c282 100644 --- a/packages/experimental/speech-to-text-sensevoice/README.zh.md +++ b/packages/experimental/speech-to-text-sensevoice/README.zh.md @@ -37,7 +37,7 @@ kind: "package-reference"
维护者信息 — 点击展开 -一个托管 Node 子进程串行执行推理。`threads`、`segmentSeconds`、`vadThreshold`、`minSpeechSeconds` 和 `minSilenceSeconds` 调节 CPU 推理与分段;`maxPending` 限制已接收任务数。准备和推理分别有截止时间。取消会终止运行中的工作进程并等待其进程范围退出;已取消的排队录音不会执行。`idleTimeoutMs` 在空闲后释放进程,零表示保留至插件释放。模型继续保留在缓存中。带认证的环回请求仅在内存中保存音频。Electron 工作进程使用 Node 模式,并复制 VAD 缓冲区以兼容 V8 内存笼。`dsh-subprocess` 拥有进程范围观测;本包没有需要协调的独立投影,因此不发布不变量伴随模块。 +一个托管 Node 子进程串行执行推理。唤醒期间提交的录音进入有界队列。工作进程在执行时要求资源处于就绪或待唤醒状态。转写不会启动准备下载。`threads`、`segmentSeconds`、`vadThreshold`、`minSpeechSeconds` 和 `minSilenceSeconds` 调节 CPU 推理与分段;`maxPending` 限制已接收任务数。准备和推理分别有截止时间。取消会终止运行中的工作进程并等待其进程范围退出;已取消的排队录音不会执行。`idleTimeoutMs` 在空闲后释放进程,零表示保留至插件释放。模型继续保留在缓存中。带认证的环回请求仅在内存中保存音频。Electron 工作进程使用 Node 模式,并复制 VAD 缓冲区以兼容 V8 内存笼。`dsh-subprocess` 拥有进程范围观测;本包没有需要协调的独立投影,因此不发布不变量伴随模块。 Host 在页面和 Session 变化期间拥有准备任务:检查资源、下载识别权重与词表、下载 VAD、校验文件、加载工作进程。显式模型路径会省略对应下载步骤。下载报告字节,其他步骤报告等待时间。空闲进程保留已准备路径,唤醒时只需加载模型。 diff --git a/packages/experimental/speech-to-text-sensevoice/src/index.ts b/packages/experimental/speech-to-text-sensevoice/src/index.ts index 5fd467f9f1..c0363d9def 100644 --- a/packages/experimental/speech-to-text-sensevoice/src/index.ts +++ b/packages/experimental/speech-to-text-sensevoice/src/index.ts @@ -28,11 +28,7 @@ export function apply(ctx: Context, config: Config): void { setupEstimate: { recommendedDiskBytes: config.precision === 'int8' ? 1_000_000_000 : 2_000_000_000, expectedMemoryBytes: config.precision === 'int8' ? 1_000_000_000 : 2_000_000_000, minimumMinutes: 1, maximumMinutes: 10 } }, preparation: worker, - transcribe: async (input, signal) => { - const phase = worker.snapshot().phase - if (phase !== 'ready' && phase !== 'standby') throw new Error('Prepare the local speech provider before recording') - return await worker.transcribe(input, signal) - }, + transcribe: async (input, signal) => await worker.transcribe(input, signal), }) worker.inspect() return async () => { diff --git a/packages/experimental/speech-to-text-sensevoice/src/recognizer.ts b/packages/experimental/speech-to-text-sensevoice/src/recognizer.ts index 797c86ea0e..db4d3c759a 100644 --- a/packages/experimental/speech-to-text-sensevoice/src/recognizer.ts +++ b/packages/experimental/speech-to-text-sensevoice/src/recognizer.ts @@ -194,6 +194,7 @@ export class SenseVoiceWorker { /** * Queue one bounded recording; cancellation never leaves inference running after settlement. + * Verified resources accept recordings while the worker wakes; other preparation states reject without downloading. * @param input - complete WAV and language hint. * @param signal - caller cancellation. * @returns recognized text; cancelled waiting jobs never acquire the worker. @@ -260,6 +261,8 @@ export class SenseVoiceWorker { } private async execute(input: SpeechInput, signal: AbortSignal): Promise { + const phase = this.state.phase + if (phase !== 'ready' && phase !== 'standby') throw new Error('Prepare the local speech provider before recording') try { const worker = await this.start(signal) using call = deadline(signal, this.config.inferenceTimeoutMs, 'SPEECH_INFERENCE_TIMEOUT') @@ -272,7 +275,7 @@ export class SenseVoiceWorker { return result } catch (error) { await this.stop() - if (this.runtime) this.publish({ phase: 'standby' }) + this.publish({ phase: 'standby' }) throw error } } diff --git a/packages/experimental/speech-to-text-sensevoice/tests/local.e2e.ts b/packages/experimental/speech-to-text-sensevoice/tests/local.e2e.ts index 5d37d74c19..44e10ab9a9 100644 --- a/packages/experimental/speech-to-text-sensevoice/tests/local.e2e.ts +++ b/packages/experimental/speech-to-text-sensevoice/tests/local.e2e.ts @@ -2,7 +2,7 @@ import { readFile } from 'node:fs/promises' import { Context } from '@deepseek-ai/cordis' import LocalSubprocess from '@deepseek-ai/dsh-subprocess-local' -import { expect, it } from 'vitest' +import { expect, it, vi } from 'vitest' import { Config } from '../src/config.ts' import { SenseVoiceWorker } from '../src/recognizer.ts' @@ -17,6 +17,8 @@ it.skipIf(!dataRoot || !audioPath)('prepares a real local recognizer, transcribe precision: process.env.DSH_SPEECH_E2E_PRECISION === 'fp32' ? 'fp32' : 'int8', })) try { + worker.prepare() + await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('ready') }, { timeout: 3_600_000 }) const input = { audio: await readFile(audioPath!), language: 'zh' } const cold = await worker.transcribe(input, new AbortController().signal) const warm = await worker.transcribe(input, new AbortController().signal) diff --git a/packages/experimental/speech-to-text-sensevoice/tests/provider-queue.spec.ts b/packages/experimental/speech-to-text-sensevoice/tests/provider-queue.spec.ts new file mode 100644 index 0000000000..06dd34c12a --- /dev/null +++ b/packages/experimental/speech-to-text-sensevoice/tests/provider-queue.spec.ts @@ -0,0 +1,70 @@ +/** Verified models accept bounded concurrent recordings while the managed worker starts. */ +import { createServer } from 'node:http' +import { mkdtemp, writeFile, rm, readFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { Context } from '@deepseek-ai/cordis' +import LocalSubprocess from '@deepseek-ai/dsh-subprocess-local' +import SpeechToText from '@deepseek-ai/dsh-experimental-speech-to-text' +import { afterEach, expect, it, vi } from 'vitest' +import * as Provider from '../src/index.ts' +import { SenseVoiceWorker } from '../src/recognizer.ts' +import { inspectRuntime, prepareRuntime } from '../src/runtime.ts' +vi.mock('../src/runtime.ts', () => ({ inspectRuntime: vi.fn(), prepareRuntime: vi.fn() })) +afterEach(() => { vi.restoreAllMocks() }) + +it.each([false, true])('queues recordings during startup and honors waiting cancellation (%s)', async (cancelled) => { + const root = await mkdtemp(join(tmpdir(), 'dsh-speech-queue-')), ctx = new Context() + const startupEntered = Promise.withResolvers(), releaseStartup = Promise.withResolvers() + const gate = createServer((_request, response) => { + startupEntered.resolve(undefined) + void releaseStartup.promise.then(() => { response.end('ready') }) + }) + try { + await new Promise((resolve, reject) => { gate.once('error', reject); gate.listen(0, '127.0.0.1', resolve) }) + const address = gate.address() + if (!address || typeof address === 'string') throw Error('Missing gate address') + const worker = join(root, 'worker.mjs') + const fixtureUrl = new URL('./worker.fixture.mjs', import.meta.url).href + await writeFile(worker, `await fetch('http://127.0.0.1:${address.port}/');\nawait import(${JSON.stringify(fixtureUrl)});\n`) + vi.mocked(inspectRuntime).mockResolvedValue({ tokens: root, model: root, vad: root, worker }) + await ctx.plugin(LocalSubprocess) + await ctx.plugin(SpeechToText) + const spawn = vi.spyOn(ctx.subprocess, 'spawn') + const enqueue = vi.spyOn(SenseVoiceWorker.prototype, 'transcribe') + await ctx.plugin(Provider, { dataRoot: root, idleTimeoutMs: 0, maxPending: 2 }) + const speech = ctx.get('speechToText')! + await vi.waitFor(() => { expect(speech.snapshot().providers[0]?.preparation.phase).toBe('standby') }) + const firstSpec = speech.resolve({ audio: new Uint8Array([1, 2]), language: 'zh' }) + const secondSpec = speech.resolve({ audio: new Uint8Array([3, 4]), language: 'en' }) + const first = speech.transcribe(firstSpec, new AbortController().signal) + const firstResult = first.then(value => ({ value }), (error: unknown) => ({ error })) + try { + await startupEntered.promise + expect(speech.snapshot().providers[0]?.preparation.phase).toBe('waking') + const waiting = new AbortController() + const second = speech.transcribe(secondSpec, waiting.signal) + const secondResult = second.then(value => ({ value }), (error: unknown) => ({ error })) + await Promise.resolve() + expect(enqueue).toHaveBeenCalledTimes(2) + await expect(speech.transcribe(secondSpec, new AbortController().signal)).rejects.toThrow('queue is full') + if (cancelled) waiting.abort(new Error('waiting recording cancelled')) + releaseStartup.resolve(undefined) + expect(await firstResult).toMatchObject({ value: { text: 'zh' } }) + if (cancelled) expect(await secondResult).toMatchObject({ error: { message: 'waiting recording cancelled' } }) + else expect(await secondResult).toMatchObject({ value: { text: 'en' } }) + expect(await readFile(join(root, 'requests'), 'utf8')).toBe(cancelled ? 'zh\n' : 'zh\nen\n') + expect(spawn).toHaveBeenCalledOnce() + expect(prepareRuntime).not.toHaveBeenCalled() + } finally { releaseStartup.resolve(undefined); await firstResult } + } finally { + releaseStartup.resolve(undefined) + try { await ctx.fiber.dispose() } finally { + try { + if (gate.listening) await new Promise((resolve, reject) => { + gate.close((error) => { if (error) reject(error); else resolve() }) + }) + } finally { await rm(root, { recursive: true, force: true }) } + } + } +}) diff --git a/packages/experimental/speech-to-text-sensevoice/tests/provider.spec.ts b/packages/experimental/speech-to-text-sensevoice/tests/provider.spec.ts index 64b217059a..a650d3d122 100644 --- a/packages/experimental/speech-to-text-sensevoice/tests/provider.spec.ts +++ b/packages/experimental/speech-to-text-sensevoice/tests/provider.spec.ts @@ -14,14 +14,13 @@ it.each(['int8', 'fp32'] as const)('registers %s lazily and joins inference befo await ctx.plugin(LocalSubprocess) await ctx.plugin(SpeechToText) const transcribe = vi.spyOn(SenseVoiceWorker.prototype, 'transcribe') - .mockResolvedValue({ text: 'hello', audioSeconds: 1, inferenceSeconds: 0.1 }) const fiber = ctx.plugin(Provider, { dataRoot: process.cwd(), precision }) await fiber expect(transcribe).not.toHaveBeenCalled() const speech = ctx.get('speechToText')! expect(speech.listProviders()).toMatchObject([{ id: 'sensevoice-local', location: 'host-local' }]) await expect(speech.transcribe(speech.resolve({ audio: new Uint8Array() }), new AbortController().signal)).rejects.toThrow('Prepare') - vi.spyOn(SenseVoiceWorker.prototype, 'snapshot').mockReturnValue({ phase: 'ready' }) + transcribe.mockResolvedValue({ text: 'hello', audioSeconds: 1, inferenceSeconds: 0.1 }) expect((await speech.transcribe(speech.resolve({ audio: new Uint8Array() }), new AbortController().signal)).text).toBe('hello') await fiber.dispose() expect(speech.listProviders()).toEqual([]) diff --git a/packages/experimental/speech-to-text-sensevoice/tests/worker.spec.ts b/packages/experimental/speech-to-text-sensevoice/tests/worker.spec.ts index 09af51a3c7..3cc12d1198 100644 --- a/packages/experimental/speech-to-text-sensevoice/tests/worker.spec.ts +++ b/packages/experimental/speech-to-text-sensevoice/tests/worker.spec.ts @@ -30,6 +30,16 @@ async function fixture(overrides: Partial = {}) { return { root, worker, spawn, ctx } } +async function prepare(worker: SenseVoiceWorker): Promise { + const ready = Promise.withResolvers() + const unsubscribe = worker.subscribe(() => { + const state = worker.snapshot() + if (state.phase === 'ready') ready.resolve(undefined) + if (state.phase === 'failed') ready.reject(new Error(state.message)) + }) + try { worker.prepare(); await ready.promise } finally { unsubscribe() } +} + const audio = new Uint8Array([1, 2]) const signal = (): AbortSignal => new AbortController().signal async function requests(root: string): Promise { @@ -42,6 +52,7 @@ async function requests(root: string): Promise { it('keeps one worker warm across recordings and joins its exit on disposal', async () => { const { worker, spawn } = await fixture() expect(spawn).not.toHaveBeenCalled() + await prepare(worker) expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh') expect((await worker.transcribe({ audio, language: 'en' }, signal())).text).toBe('en') expect(spawn).toHaveBeenCalledOnce() @@ -110,6 +121,7 @@ it('cancels cache inspection and joins it without late notifications when the pr it('cancels active inference before the next recording can acquire a fresh worker', async () => { const { worker, root, spawn } = await fixture() + await prepare(worker) const cancel = new AbortController() const first = worker.transcribe({ audio, language: 'hold' }, cancel.signal) const rejected = expect(first).rejects.toThrow() @@ -124,6 +136,7 @@ it('cancels active inference before the next recording can acquire a fresh worke it('bounds the queue and never executes a cancelled waiting recording', async () => { const { worker, root } = await fixture({ maxPending: 2 }) + await prepare(worker) const active = new AbortController(), waiting = new AbortController() const first = worker.transcribe({ audio, language: 'hold' }, active.signal) const firstFailure = expect(first).rejects.toThrow() @@ -138,6 +151,7 @@ it('bounds the queue and never executes a cancelled waiting recording', async () it('joins running and waiting requests when its provider is disposed', async () => { const { worker, root } = await fixture() + await prepare(worker) const first = worker.transcribe({ audio, language: 'hold' }, signal()) const firstFailure = expect(first).rejects.toThrow() await vi.waitFor(async () => { expect(await requests(root)).toContain('hold') }, { timeout: 10000 }) @@ -149,6 +163,7 @@ it('joins running and waiting requests when its provider is disposed', async () it('stops an idle worker and reloads it on demand', async () => { const { worker, spawn } = await fixture({ idleTimeoutMs: 25 }) + await prepare(worker) await worker.transcribe({ audio, language: 'en' }, signal()) const first = spawn.mock.results[0]!.value as SubprocessHandle await first.done @@ -196,6 +211,7 @@ it('cancels Host preparation, retains retry admission, and exposes preparation f it('recovers after a provider error or unexpected worker exit', async () => { const { worker } = await fixture() + await prepare(worker) await expect(worker.transcribe({ audio, language: 'error' }, signal())).rejects.toThrow('provider failed') await expect(worker.transcribe({ audio, language: 'crash' }, signal())).rejects.toThrow() expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh') @@ -203,10 +219,15 @@ it('recovers after a provider error or unexpected worker exit', async () => { it('applies inference deadlines and reports runtime preparation failure', async () => { const { worker } = await fixture({ inferenceTimeoutMs: 100 }) + await prepare(worker) await expect(worker.transcribe({ audio, language: 'hold' }, signal())).rejects.toThrow() const fresh = await fixture() vi.mocked(prepareRuntime).mockRejectedValueOnce(new Error('runtime missing')) - await expect(fresh.worker.transcribe({ audio, language: 'zh' }, signal())).rejects.toThrow('runtime missing') + fresh.worker.prepare() + await vi.waitFor(() => { expect(fresh.worker.snapshot()).toMatchObject({ phase: 'failed', message: 'runtime missing' }) }) + await fresh.worker.cancel() + expect(fresh.worker.snapshot().phase).toBe('failed') + await expect(fresh.worker.transcribe({ audio, language: 'zh' }, signal())).rejects.toThrow('Prepare') }) it('retains complete download snapshots while throttling only intermediate notifications', async () => { @@ -248,6 +269,7 @@ it('joins a worker that exits before readiness and exposes retryable preparation it('reclaims the process range after an unexpected idle exit before replacing the worker', async () => { const { worker, spawn } = await fixture() + await prepare(worker) await worker.transcribe({ audio, language: 'en' }, signal()) const handle = spawn.mock.results[0]!.value as SubprocessHandle handle.terminate() @@ -259,6 +281,7 @@ it('reclaims the process range after an unexpected idle exit before replacing th it('retains a worker whose idle cleanup cannot observe exit until that range is joined', async () => { const { worker, spawn, ctx } = await fixture({ idleTimeoutMs: 100 }) + await prepare(worker) const warn = vi.spyOn(ctx.logger, 'warn').mockImplementation(() => {}) await worker.transcribe({ audio, language: 'en' }, signal()) const handle = spawn.mock.results[0]!.value as SubprocessHandle @@ -305,6 +328,7 @@ it('omits preparation steps supplied by explicit deployment paths', async () => it('returns prepared resources to standby when the user cancels an idle wake-up', async () => { const { worker } = await fixture({ idleTimeoutMs: 100 }) + await prepare(worker) await worker.transcribe({ audio, language: 'en' }, signal()) await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('standby') }, { timeout: 10000 }) const cancel = new AbortController() @@ -321,6 +345,29 @@ it('returns prepared resources to standby when the user cancels an idle wake-up' it('launches source workers through the repository ESM loader', async () => { const { worker, spawn, root } = await fixture() vi.mocked(prepareRuntime).mockResolvedValueOnce({ tokens: root, worker: fileURLToPath(new URL('./worker-source.fixture.ts', import.meta.url)), model: root, vad: root }) + await prepare(worker) expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh') expect(spawn.mock.calls[0]?.[0].argv).toContain('--import') }) + +it('requires explicit preparation before direct transcription without starting downloads', async () => { + const { worker, spawn } = await fixture() + await expect(worker.transcribe({ audio, language: 'zh' }, signal())).rejects.toThrow('Prepare') + expect(worker.snapshot().phase).toBe('unprepared') + expect(prepareRuntime).not.toHaveBeenCalled() + expect(spawn).not.toHaveBeenCalled() +}) + +it('rechecks readiness after a queued cache inspection without implicitly downloading missing resources', async () => { + const { worker, root, spawn } = await fixture() + vi.mocked(inspectRuntime).mockResolvedValueOnce({ tokens: root, model: root, vad: root, + worker: fileURLToPath(new URL('./worker.fixture.mjs', import.meta.url)) }) + worker.inspect() + await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('standby') }) + vi.mocked(inspectRuntime).mockResolvedValueOnce(undefined) + worker.inspect() + await expect(worker.transcribe({ audio, language: 'zh' }, signal())).rejects.toThrow('Prepare') + expect(worker.snapshot().phase).toBe('unprepared') + expect(prepareRuntime).not.toHaveBeenCalled() + expect(spawn).not.toHaveBeenCalled() +}) diff --git a/snapshots/web/voice-input/interrupted.expected.md b/snapshots/web/voice-input/interrupted.expected.md new file mode 100644 index 0000000000..2e1eed1683 --- /dev/null +++ b/snapshots/web/voice-input/interrupted.expected.md @@ -0,0 +1,6 @@ +- textbox "Message or run a task, / commands, @ files or sessions" +- button "Cancel": + - img +- status "Recording was interrupted. Please try again." +- button "Record again" +- button "Send message" [disabled]