diff --git a/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.i18n.yaml b/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.i18n.yaml new file mode 100644 index 0000000000..ec0c24bf9c --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.md +2026-09-16-experimental-voice-input.md: f385f4dc2d872f467aace6a07b09a11acdf8ab9f +2026-09-16-experimental-voice-input.zh.md: 15e55efe33905487f660f1373bc9c425378d9e46 diff --git a/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.md b/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.md new file mode 100644 index 0000000000..f385f4dc2d --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.md @@ -0,0 +1,35 @@ +# Agent Note: Experimental voice input with explicit recognition providers + +Status: implemented + +English | [中文](2026-09-16-experimental-voice-input.zh.md) + +## Problem + +Dictation must preserve user control over when a coding task starts. Local recognition brings model files, a native runtime and CPU work; merely installing or enabling a UI contribution must not allocate these resources. Later cloud providers need a selection mechanism that does not silently change where recordings are processed. + +## Decision + +Five public packages under `packages/experimental/` compose one optional, default-disabled voice input bundle. Service Definition, SenseVoice Provider and Remote Consumer are separate because provider deployment and browser transport evolve independently. The browser UI dynamically mounts its generated Remote contribution. Stable API Remotes do not acquire an experimental dependency. + +Provider ids select exact registrations. Resolution captures an instance, and deregistration closes admission, cancels and joins accepted work. There is no provider fallback: a local failure cannot authorize uploading speech to a cloud service. A new cloud provider supplies its own credentials and explicit processing location through the same registry. + +Recordings and transcripts remain transient until the user submits ordinary text. Recognition does not start an agent turn or create an audio Session event. The input facade accepts asynchronous plain text against a captured draft revision, preserving reference chips and undo history. Changed drafts retain the recognized text for explicit insertion instead of replacing later edits. Session changes and plugin disposal invalidate late results and release recording resources. + +SenseVoice ONNX and Silero VAD run in one managed CPU subprocess. The native sherpa-onnx package includes ONNX Runtime; explicit preparation only downloads and verifies models; subsequent calls reuse the warm model until the configured idle deadline. Disabling the provider joins its work and process exit while retaining reusable disk assets. Native browser recording and resampling avoid an additional audio conversion executable. The authenticated private loopback protocol carries complete PCM recordings, allowing cancellation to terminate the worker without sharing a long-running inference with another request. + +The Host owns preparation independently of the view that started it. An ordered step list survives view remounts and records completion only as execution advances. Recognition weights, VAD and final verification have separate operations; downloads use pinned byte counts, while other work reports elapsed time. Package status uses `plugins.bundle.status` and details use the upstream `plugins.bundle.config`, both keyed by package name. A single composer activity Slot expands the voice controls without replacing the draft editor. Shared primitives provide buttons, disclosure and status markers; only measured waveform drawing belongs to the voice UI. Recordings start on click and transcribe on Stop, with feedback inside the activity. Persistent preferences remain in plugin settings. + +## Alternatives considered + +Always loading the model on plugin activation would make a UI toggle incur substantial startup time and memory even without dictation. Python/FunASR/PyTorch adds interpreter provisioning, dependency installation and disk cost; completed ONNX artifacts and platform-native packages serve CPU dictation. End users never export models. Browser-native speech recognition does not provide a uniform local-processing guarantee. Streaming voice interaction would require partial-result revision, turn arbitration and bidirectional audio semantics that complete-recording dictation does not need. + +## Consequences + +First use needs download time and disk space. Local means the Host machine, including a remote Host. A cancelled active inference loses its warm worker and the next request reloads it. Optional preparation paths support existing offline environments without silently changing defaults. + +Verification covers exact-provider routing, registration teardown, bounded audio parsing, microphone release, stale-draft and Session-switch handling, worker queue/deadline recovery, and verified download publication. Real local inference uses the same managed worker as the product. Browser composition verifies optional enablement and draft insertion; ordinary Session replay verifies that only the submitted text reaches the model. + +Real Node 24 and Electron 44 children complete FP32 and INT8 inference. Electron copies VAD output to avoid external-buffer restrictions. Three runs of the 4.2-second public Chinese sample agree; this short sample is not a complete accuracy evaluation. Desktop microphone requests are restricted to audio from the primary application window, with macOS authorization handled by the operating system. Windows, Linux and signed installers still require platform acceptance. + +INT8 is the default to reduce download and storage cost; `precision: fp32` selects the reference weights. Public English and concatenated bilingual samples show word differences between precisions, so quantization is not treated as accuracy-equivalent. A roughly 101-second repeated-speech sample completes in both runtimes; its observed post-request RSS is not a peak-memory bound. diff --git a/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.zh.md b/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.zh.md new file mode 100644 index 0000000000..15e55efe33 --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.zh.md @@ -0,0 +1,35 @@ +# Agent Note：使用显式识别 Provider 的实验性语音输入 + +Status: implemented + +[English](2026-09-16-experimental-voice-input.md) | 中文 + +## 问题 + +听写必须保留用户对编码任务启动时机的控制。本地识别带来模型文件、原生运行时和 CPU 工作；仅安装或启用 UI 贡献不应分配这些资源。之后的云端 Provider 需要明确的选择机制，不能悄悄改变录音的处理位置。 + +## 决策 + +`packages/experimental/` 下的五个公开包组合成一个默认禁用的可选语音输入 Bundle。服务定义、SenseVoice Provider 和 Remote 消费者分开，因为 Provider 部署与浏览器传输独立演化。浏览器 UI 动态挂载其生成的 Remote 贡献。稳定 API Remotes 不引入实验性依赖。 + +Provider id 选择确切的注册。解析捕获实例，注销先拒绝新任务，再取消并等待已接收的任务。不存在 Provider 回退：本地失败不能授权将语音上传至云端服务。新增云端 Provider 通过同一注册表提供自己的凭据配置与显式处理位置。 + +录音与转写在用户提交普通文字前都是临时数据。识别不启动智能体轮次，也不创建音频 Session 事件。输入门面按照捕获的草稿版本接收异步纯文本，保留引用芯片与撤销历史。草稿变化时保留识别文字，等待显式插入，而非替换之后的编辑。切换 Session 和释放插件会使迟到结果失效并释放录音资源。 + +SenseVoice ONNX 与 Silero VAD 在一个托管 CPU 子进程中运行。原生 sherpa-onnx 包包含 ONNX Runtime；显式准备只下载并校验模型；后续调用复用预热模型，直到配置的空闲截止时间。禁用 Provider 等待其任务和进程退出，同时保留可复用的磁盘资源。浏览器原生录音与重采样避免增加音频转换可执行文件。带认证的私有环回协议传输完整 PCM 录音，取消时可以终止工作进程，而不与另一请求共享长时间推理。 + +Host 独立于发起视图拥有准备任务。有序步骤列表在视图重新挂载后保留，仅随着执行推进记录完成。识别模型、VAD 与最终校验分别准备；下载使用固定字节数，其他工作报告等待时间。包状态使用 `plugins.bundle.status`，详情使用上游 `plugins.bundle.config`，均以包名为 key。单个输入框活动 Slot 展开语音控件，同时保留草稿编辑器。按钮、折叠与状态标记复用共享原语，语音 UI 只负责实测波形绘制。点击开始录音，停止后转写，反馈留在活动栏内。持久化偏好仍在插件设置中。 + +## 考虑的替代方案 + +在插件启用时加载模型，会让一个 UI 开关在尚未听写时就产生大量启动时间和内存占用。Python/FunASR/PyTorch 增加解释器、依赖安装与磁盘成本；ONNX 成品和平台原生包满足 CPU 听写。用户端不执行模型导出。浏览器原生语音识别不能提供一致的本地处理保证。流式语音交互需要部分结果修订、轮次仲裁与双向音频语义，而完整录音听写无需这些机制。 + +## 影响与验证 + +首次使用需要下载时间和磁盘空间。本地指 Host 所在机器，包括远程 Host。取消运行中的推理会丢失预热进程，下次请求重新加载。可选准备路径支持已有离线环境，不会悄悄改变默认行为。 + +验证覆盖精确 Provider 路由、注册释放、有界音频解析、麦克风释放、过期草稿和 Session 切换处理、工作进程队列与截止时间恢复，以及经过校验的下载发布。真实本地推理使用与产品相同的托管工作进程。浏览器组合验证可选启用与草稿插入；普通 Session 回放验证只有提交的文字进入模型。 + +Node 24 与 Electron 44 的真实子进程均完成 FP32 和 INT8 识别。Electron 的 VAD 输出使用复制缓冲区，避免 external buffer 限制。4.2 秒公开中文样本的三次识别结果一致；该短样本不代表完整准确率评测。桌面麦克风请求限制为主应用窗口的音频请求，macOS 权限由系统处理。Windows、Linux 真实机器及签名后的安装包仍需平台验收。 + +INT8 以降低下载和存储成本为默认值，`precision: fp32` 选择参考权重。公开英文与拼接的中英双语样本显示两种精度存在用词差异，因此不将量化视为准确率等效。约 101 秒的重复语音样本在两种运行时均完成识别；请求后的 RSS 观测不构成峰值内存上限。 diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index d4838dfe32..3b1f408f0f 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -109,6 +109,7 @@ External packages installed for runtime use or distributed inside the prebuilt b | [`resolve.exports`](https://github.com/lukeed/resolve.exports) | MIT | | [`semver`](https://github.com/npm/node-semver) | ISC | | [`sharp`](https://github.com/lovell/sharp) | Apache-2.0 | +| [`sherpa-onnx-node`](https://github.com/csukuangfj/sherpa-onnx) | Apache-2.0 | | [`shiki`](https://github.com/shikijs/shiki) | MIT | | [`simple-icons`](https://github.com/simple-icons/simple-icons) | CC0-1.0 | | [`supports-color`](https://github.com/chalk/supports-color) | MIT | diff --git a/apps/cli/package.json b/apps/cli/package.json index 5204f55a29..1945ba8424 100644 --- a/apps/cli/package.json +++ b/apps/cli/package.json @@ -105,7 +105,8 @@ "@deepseek-ai/dsh-http-proxy": "workspace:^", "@deepseek-ai/dsh-mcp-resources": "workspace:^", "@deepseek-ai/dsh-plugin-manager": "workspace:^", - "@deepseek-ai/dsh-atomic-write": "workspace:^" + "@deepseek-ai/dsh-atomic-write": "workspace:^", + "@deepseek-ai/dsh-experimental-voice-input-bundle": "workspace:^" }, "devDependencies": { "@agentclientprotocol/sdk": "1.4.0", diff --git a/apps/desktop/README.i18n.yaml b/apps/desktop/README.i18n.yaml index de2415f74b..6755dce11c 100644 --- a/apps/desktop/README.i18n.yaml +++ b/apps/desktop/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write apps/desktop/README.md -README.md: 393f41f82ed20fbc71b1659cd2417a8d7f9b5013 -README.zh.md: e5bdb032b47795dc28848561f08f0464f83e49d3 +README.md: 179992c17a3d93dd0bcb65567dd4bb6748306460 +README.zh.md: 0357908160f88566bff9d760905b4522d9a73d05 diff --git a/apps/desktop/README.md b/apps/desktop/README.md index 393f41f82e..179992c17a 100644 --- a/apps/desktop/README.md +++ b/apps/desktop/README.md @@ -8,6 +8,8 @@ Desktop’s local native directory flow opens an Electron folder dialog attached Creator and the Web Plugin Manager use Desktop’s bundled pnpm under Electron Node mode without requiring pnpm on PATH. The private Node launcher environment applies only to package operations. +Desktop microphone access is restricted to audio requests from the primary `dsh-app://app` frame. macOS uses system microphone authorization and a packaged usage description. Native speech libraries remain outside ASAR and use the existing runtime signing path; recognition runs in a managed Node-mode child. + ## Key technical decisions The original artwork lives in `resources/icon.png` and `resources/icon.svg`; platform adaptations retain the whale and gradients in `resources/icon-windows.*` and `resources/icon-macos.*`. Export each platform SVG as a transparent 1024×1024 PNG. Electron-builder generates the multi-size ICO for the Windows application, installer, and uninstaller ([Windows icon requirements](https://learn.microsoft.com/en-us/windows/apps/design/iconography/app-icon-construction)). The installation pages use matching artwork in both themes; the uninstaller's welcome and finish pages share `installer/assets/uninstaller-sidebar.png`, converted to a 164×314 BMP during preparation. diff --git a/apps/desktop/README.zh.md b/apps/desktop/README.zh.md index e5bdb032b4..0357908160 100644 --- a/apps/desktop/README.zh.md +++ b/apps/desktop/README.zh.md @@ -8,6 +8,8 @@ Desktop 的本地原生目录流程打开绑定应用窗口的 Electron 文件 Creator 和 Web Plugin Manager 在 Electron Node 模式下使用 Desktop 内置 pnpm，无需 PATH 中存在 pnpm。私有 Node 启动器环境仅应用于包操作。 +桌面麦克风访问仅允许主 `dsh-app://app` 页面发起的音频请求。macOS 使用系统麦克风授权与随包用途说明。原生语音库保留在 ASAR 外并使用既有运行时签名路径；识别在托管的 Node 模式子进程中执行。 + ## 关键技术决策 设计师原稿位于 `resources/icon.png` 和 `resources/icon.svg`；平台适配保留鲸鱼与渐变，分别位于 `resources/icon-windows.*` 和 `resources/icon-macos.*`。将各平台 SVG 导出为透明的 1024×1024 PNG。electron-builder 为 Windows 应用、安装程序和卸载程序生成多尺寸 ICO（[Windows 图标要求](https://learn.microsoft.com/en-us/windows/apps/design/iconography/app-icon-construction)）。安装页面在两种主题下使用匹配的图案；卸载程序的欢迎和完成页共用 `installer/assets/uninstaller-sidebar.png`，准备阶段将其转换为 164×314 BMP。 diff --git a/apps/desktop/electron-builder.config.d.mts b/apps/desktop/electron-builder.config.d.mts index 3aefaec12f..01dcbd0e99 100644 --- a/apps/desktop/electron-builder.config.d.mts +++ b/apps/desktop/electron-builder.config.d.mts @@ -15,6 +15,7 @@ export interface DesktopElectronBuilderConfig { readonly asarUnpack: readonly string[] readonly extraResources: readonly [{ readonly from: string, readonly to: 'runtime' }] readonly mac: { + readonly extendInfo: { readonly NSMicrophoneUsageDescription: string } readonly identity: string | undefined readonly forceCodeSigning: boolean readonly notarize: boolean diff --git a/apps/desktop/electron-builder.config.mjs b/apps/desktop/electron-builder.config.mjs index ca846818b6..98d41b64a9 100644 --- a/apps/desktop/electron-builder.config.mjs +++ b/apps/desktop/electron-builder.config.mjs @@ -102,6 +102,7 @@ export function createElectronBuilderConfig( identity: macOSSigning?.signingIdentity, forceCodeSigning: true, hardenedRuntime: true, + extendInfo: { NSMicrophoneUsageDescription: 'DeepSeek Harness uses your microphone to transcribe speech into message drafts.' }, // ASAR-unpacked native runtime files are pre-signed; PAK resources are sealed by their enclosing bundle. signIgnore: ['/Contents/Resources/app\\.asar\\.unpacked/dsh(?:/|$)', '/Contents/Resources/runtime/primary-runtime(?:/|$)', '\\.pak$'], notarize: true, diff --git a/apps/desktop/src/main.ts b/apps/desktop/src/main.ts index 4eb65c0521..e82289da82 100644 --- a/apps/desktop/src/main.ts +++ b/apps/desktop/src/main.ts @@ -20,6 +20,7 @@ import { resolveDesktopPaths } from './paths.ts' import { DesktopProjectManager, type DesktopProjectHooks } from './project-manager.ts' import { DesktopHostProcess } from './host-process.ts' import { installDesktopDirectoryPicker } from './directory-picker.ts' +import { installMicrophonePermissions } from './microphone-permissions.ts' import { DesktopBackendController } from './backend-controller.ts' import { DESKTOP_IPC, SCHEME, assertDesktopSender, type DesktopUpdateState } from './ipc.ts' import { formatDesktopMessage, resolveDesktopLocale } from './locale.ts' @@ -289,6 +290,7 @@ async function main(): Promise { }) installDesktopDirectoryPicker(() => mainWindow) + installMicrophonePermissions(session.defaultSession, () => mainWindow?.webContents) ipcMain.handle(DESKTOP_IPC.boot, async (event) => { assertDesktopSender(event, ['app']) diff --git a/apps/desktop/src/microphone-permissions.ts b/apps/desktop/src/microphone-permissions.ts new file mode 100644 index 0000000000..ba644ac80f --- /dev/null +++ b/apps/desktop/src/microphone-permissions.ts @@ -0,0 +1,31 @@ +/** Microphone access belongs to the primary application frame and the operating system. */ +import { systemPreferences, type Session, type WebContents } from 'electron' + +function applicationFrame(url: string): boolean { + try { + const parsed = new URL(url) + return parsed.protocol === 'dsh-app:' && parsed.hostname === 'app' + } catch (_error) { return false /* Invalid frame URLs cannot request microphone access. */ } +} + +/** + * Handle microphone requests from the owned application window; other permissions retain Electron's defaults. + * @param session - application's browser session. + * @param primary - current primary window contents, absent while no window is open. + */ +export function installMicrophonePermissions(session: Session, primary: () => WebContents | undefined): void { + session.setPermissionCheckHandler((contents, permission, origin, details) => { + if (permission !== 'media') return true + return contents != null && contents === primary() && details.isMainFrame + && applicationFrame(origin) && details.mediaType === 'audio' + && (process.platform !== 'darwin' || systemPreferences.getMediaAccessStatus('microphone') === 'granted') + }) + session.setPermissionRequestHandler((contents, permission, callback, details) => { + if (permission !== 'media') { callback(true); return } + const allowed = contents === primary() && details.isMainFrame && applicationFrame(details.requestingUrl) + && 'mediaTypes' in details && details.mediaTypes.length === 1 && details.mediaTypes[0] === 'audio' + if (!allowed) { callback(false); return } + if (process.platform !== 'darwin') { callback(true); return } + void systemPreferences.askForMediaAccess('microphone').then(callback, () => { callback(false) }) + }) +} diff --git a/apps/desktop/tests/macos-signature.spec.ts b/apps/desktop/tests/macos-signature.spec.ts index 3286f23e89..514e150ab5 100644 --- a/apps/desktop/tests/macos-signature.spec.ts +++ b/apps/desktop/tests/macos-signature.spec.ts @@ -40,6 +40,7 @@ describe('desktop macOS release signature', () => { const { createElectronBuilderConfig } = await import('../electron-builder.config.mjs') const config = createElectronBuilderConfig(RELEASE_ENVIRONMENT, 'darwin', 'arm64') expect(portablePath(config.directories.output)).toContain('/.desktop-build/targets/mac-arm64/artifacts') + expect(config.mac.extendInfo.NSMicrophoneUsageDescription).toContain('microphone') expect(config.extraResources).toHaveLength(1) expect(config.extraResources[0]?.to).toBe('runtime') expect(portablePath(config.extraResources[0]?.from ?? '')).toContain('/.desktop-build/targets/mac-arm64/runtime') diff --git a/apps/desktop/tests/main-startup.spec.ts b/apps/desktop/tests/main-startup.spec.ts index 4856b0a5cb..96e1e1d5fb 100644 --- a/apps/desktop/tests/main-startup.spec.ts +++ b/apps/desktop/tests/main-startup.spec.ts @@ -135,7 +135,8 @@ vi.mock('electron', () => ({ handle: (channel: string, handler: (event: { senderFrame: { url: string } }) => unknown) => { if (harness.handlers.has(channel)) throw new Error(`duplicate IPC handler ${channel}`); harness.handlers.set(channel, handler) }, }, Menu: harness.menu, - session: { defaultSession: { webRequest: { onBeforeSendHeaders: harness.socketHeaders } } }, + session: { defaultSession: { setPermissionCheckHandler: vi.fn(), setPermissionRequestHandler: vi.fn(), + webRequest: { onBeforeSendHeaders: harness.socketHeaders } } }, protocol: { registerSchemesAsPrivileged: vi.fn(), handle: vi.fn() }, })) vi.mock('../src/paths.ts', () => ({ resolveDesktopPaths: () => ({ profile: 'desktop-test-profile' }) })) diff --git a/apps/desktop/tests/microphone-permissions.spec.ts b/apps/desktop/tests/microphone-permissions.spec.ts new file mode 100644 index 0000000000..9cdd416481 --- /dev/null +++ b/apps/desktop/tests/microphone-permissions.spec.ts @@ -0,0 +1,51 @@ +/** Desktop media permissions exclude external pages, subframes and camera requests. */ +import type { Session, WebContents } from 'electron' +import { afterEach, expect, it, vi } from 'vitest' +import { installMicrophonePermissions } from '../src/microphone-permissions.ts' + +const access = vi.hoisted(() => ({ getMediaAccessStatus: vi.fn(() => 'granted'), askForMediaAccess: vi.fn(async () => true) })) +vi.mock('electron', () => ({ systemPreferences: access })) +afterEach(() => { vi.unstubAllGlobals(); vi.clearAllMocks() }) +function fixture(platform: NodeJS.Platform = 'darwin') { + vi.stubGlobal('process', { ...process, platform }) + const setPermissionCheckHandler = vi.fn() + const setPermissionRequestHandler = vi.fn() + const primary = {} as WebContents + installMicrophonePermissions({ setPermissionCheckHandler, setPermissionRequestHandler } as unknown as Session, () => primary) + return { primary, check: setPermissionCheckHandler.mock.calls[0]![0]!, request: setPermissionRequestHandler.mock.calls[0]![0]! } +} +it('allows only audio checks from the owned primary application frame', () => { + const f = fixture(), details = { isMainFrame: true, mediaType: 'audio' as const } + expect(f.check(f.primary, 'media', 'dsh-app://app', details)).toBe(true) + expect(f.check(null, 'media', 'dsh-app://app', details)).toBe(false) + expect(f.check({} as WebContents, 'media', 'dsh-app://app', details)).toBe(false) + expect(f.check(f.primary, 'media', 'https://example.com', details)).toBe(false) + expect(f.check(f.primary, 'media', 'invalid', details)).toBe(false) + expect(f.check(f.primary, 'media', 'dsh-app://app', { ...details, isMainFrame: false })).toBe(false) + expect(f.check(f.primary, 'media', 'dsh-app://app', { ...details, mediaType: 'video' })).toBe(false) + access.getMediaAccessStatus.mockReturnValueOnce('denied') + expect(f.check(f.primary, 'media', 'dsh-app://app', details)).toBe(false) + expect(f.check(f.primary, 'clipboard-sanitized-write', 'dsh-app://app', details)).toBe(true) +}) +it('requests macOS microphone access and reports operating-system rejection', async () => { + const f = fixture(), done = vi.fn(), details = { isMainFrame: true, requestingUrl: 'dsh-app://app/', mediaTypes: ['audio'] } + f.request(f.primary, 'media', done, details) + await vi.waitFor(() => { expect(done).toHaveBeenLastCalledWith(true) }) + expect(access.askForMediaAccess).toHaveBeenCalledWith('microphone') + access.askForMediaAccess.mockRejectedValueOnce(new Error('permission denied')) + f.request(f.primary, 'media', done, details) + await vi.waitFor(() => { expect(done).toHaveBeenLastCalledWith(false) }) + for (const denied of [{ ...details, mediaTypes: ['audio', 'video'] }, { ...details, mediaTypes: [] }, + { ...details, isMainFrame: false }, { isMainFrame: true, requestingUrl: 'dsh-app://app/' }]) { + f.request(f.primary, 'media', done, denied); expect(done).toHaveBeenLastCalledWith(false) + } + f.request(f.primary, 'clipboard-sanitized-write', done, details) + expect(done).toHaveBeenLastCalledWith(true) +}) +it.each(['win32', 'linux'] as const)('leaves system authorization to Chromium on %s', (platform) => { + const f = fixture(platform), done = vi.fn() + expect(f.check(f.primary, 'media', 'dsh-app://app', { isMainFrame: true, mediaType: 'audio' })).toBe(true) + f.request(f.primary, 'media', done, { isMainFrame: true, requestingUrl: 'dsh-app://app/', mediaTypes: ['audio'] }) + expect(done).toHaveBeenCalledWith(true) + expect(access.askForMediaAccess).not.toHaveBeenCalled() +}) diff --git a/apps/web/package.json b/apps/web/package.json index acdb74244a..a4858c3199 100644 --- a/apps/web/package.json +++ b/apps/web/package.json @@ -60,6 +60,8 @@ "@deepseek-ai/dsh-launch-environment": "workspace:^", "@deepseek-ai/dsh-workspace-changes": "workspace:^", "@deepseek-ai/dsh-subprocess-local": "workspace:^", - "@deepseek-ai/dsh-experimental-auto-review": "workspace:^" + "@deepseek-ai/dsh-experimental-auto-review": "workspace:^", + "@deepseek-ai/dsh-experimental-speech-to-text": "workspace:^", + "@deepseek-ai/dsh-experimental-voice-input-bundle": "workspace:^" } } diff --git a/apps/web/tests/scaffold.ts b/apps/web/tests/scaffold.ts index f807eb83c6..8a83895643 100644 --- a/apps/web/tests/scaffold.ts +++ b/apps/web/tests/scaffold.ts @@ -305,8 +305,8 @@ export interface LaunchOptions { profileResolutionMode?: Extract /** Enable the real Open In rows with deterministic launch-environment facts. */ openInAppEnvironment?: LaunchEnvironmentSnapshot - /** Compare the replayed root session with `replayFixture`; defaults on for a manifest-owned canonical recording. */ - compareReplaySession?: boolean + /** Compare the replayed root Session; `read-only` also forbids refresh writes to a borrowed fixture. */ + compareReplaySession?: boolean | 'read-only' /** * Optional product overlay applied after the shipped Web surface and before * the scaffold's hermetic test patches, matching the launcher's `--patch` @@ -920,7 +920,7 @@ export async function launchWebScaffold(options: LaunchOptions = {}): Promise { + const scratch = await mkdtemp(join(tmpdir(), 'dsh-voice-browser-')) + const resources: { scaffold?: WebScaffold; browser?: Browser } = {} + onTestFinished(async () => { + try { await resources.browser?.close() } finally { + try { await resources.scaffold?.close() } finally { await rm(scratch, { recursive: true, force: true }) } + } + }) + const overlay = join(scratch, 'voice.patch.yml') + await writeFile(overlay, '- id: speech-to-text-sensevoice\n disabled: true\n') + const prompt = fixtureUserPrompts(await readFile(fixture, 'utf8'))[0]! + const prefix = 'Use the bash tool to ' + const scaffold = await launchWebScaffold({ profile: { packages: [{ dir: bundle, enabled: true }] }, + extraOverlayPath: overlay, replayFixture: fixture, compareReplaySession: 'read-only', + }) + resources.scaffold = scaffold + const recognize = vi.fn(async (input: SpeechInput) => { + expect(Buffer.from(input.audio).subarray(0, 4).toString()).toBe('RIFF') + expect(input.audio.byteLength).toBeGreaterThan(44) + return { text: prompt.slice(prefix.length), audioSeconds: 1, inferenceSeconds: 0.1 } + }) + await scaffold.ctx.plugin({ inject: ['speechToText'], apply(ctx) { + ctx.effect(() => ctx.speechToText.register({ + info: { id: 'sensevoice-local' as SpeechProviderId, name: 'Recorded recognizer', location: 'host-local' }, + transcribe: recognize, + })) + } }) + let messages = 0 + scaffold.ctx.on('session/event', (_session, event) => { + if (event.type === 'user/message' && event.data.source.kind === 'user') messages++ + }) + const browser = await chromium.launch({ args: ['--use-fake-ui-for-media-stream', '--use-fake-device-for-media-stream'] }) + resources.browser = browser + const page = await newEnglishPage(browser), tripwire = watchConsole(page) + await page.goto(scaffold.authenticatedUrl) + await connectFreshWorkspace(page, scaffold.workspaceCwd) + const input = page.locator('[data-composer-input]'), mic = page.getByRole('button', { name: 'Start recording', exact: true }) + await mic.waitFor() + const micBox = await mic.boundingBox(), modelBox = await page.getByRole('button', { name: /^Select model, current/ }).boundingBox() + const sendBox = await page.getByRole('button', { name: 'Send message', exact: true }).boundingBox() + expect(micBox!.x).toBeGreaterThan(modelBox!.x) + expect(micBox!.x).toBeLessThan(sendBox!.x) + await input.fill(prefix) + await mic.click() + await page.getByRole('button', { name: 'Stop and transcribe', exact: true }).waitFor() + await compareOrRefreshGolden(recordingExpected, + await captureStableAria(page, '[data-composer-card]', scaffold.workspaceCwd), webSnapshotMode()) + expect(await page.getByRole('dialog').count()).toBe(0) + // Controlled fake microphone audio occupies this recording interval; it is not a readiness wait. + await page.waitForTimeout(300) + await page.getByRole('button', { name: 'Stop and transcribe', exact: true }).click() + await expect.poll(() => input.innerText()).toBe(prompt) + expect(recognize).toHaveBeenCalledOnce() + expect(messages).toBe(0) + await compareOrRefreshGolden(expected, await captureStableAria(page, '[class*="centerCol"]', scaffold.workspaceCwd), webSnapshotMode()) + const settled = scaffold.whenTurnSettled() + await input.press('Enter') + await settled + await page.getByText('DONE', { exact: true }).waitFor() + expect(messages).toBe(1) + expect(tripwire.pageErrors).toEqual([]) +}, 120_000) diff --git a/apps/web/tsconfig.json b/apps/web/tsconfig.json index 83b574f9bf..880d27e739 100644 --- a/apps/web/tsconfig.json +++ b/apps/web/tsconfig.json @@ -25,6 +25,7 @@ "tests/default-product-isolation.e2e.ts", "tests/diff-context.e2e.ts", "tests/scaffold.ts", + "tests/voice-input.e2e.ts", "tests/auto-review-fixture.ts", "tests/scaffold-generation.spec.ts", "tests/scaffold-hermetic.e2e.ts", diff --git a/docs/capability-seams.i18n.yaml b/docs/capability-seams.i18n.yaml index 7157fab563..f59bbdc570 100644 --- a/docs/capability-seams.i18n.yaml +++ b/docs/capability-seams.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/capability-seams.md -capability-seams.md: df39022715cf1000daca9fdeaf668e9cedf88bb1 -capability-seams.zh.md: fc9c8fd46375d77fb42fcc6cf0aef160527d100e +capability-seams.md: c97c2d97b9555f74bbb0151a46116da177eb5a8b +capability-seams.zh.md: c825f666b0466d52c82caab5212e88e5af1d2f9a diff --git a/docs/capability-seams.md b/docs/capability-seams.md index df39022715..c97c2d97b9 100644 --- a/docs/capability-seams.md +++ b/docs/capability-seams.md @@ -57,6 +57,8 @@ flowchart LR pkg_subagent_in_process_driver["subagent-in-process-driver"] pkg_invariants["invariants"] pkg_message_feedback["message-feedback"] + pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"] + svc_speechController["ctx.speechController
Experimental transcription Remote"] svc_sessionController["ctx.sessionController
Host Session Remote controller"] svc_sessionFileReferences["ctx.sessionFileReferences
Session-addressed file-reference Remote adapter"] svc_sessionSkillCatalog["ctx.sessionSkillCatalog
Session-addressed skill Remote adapter"] @@ -207,6 +209,9 @@ flowchart LR pkg_subagent_dsh_sdk["subagent-dsh-sdk"] pkg_tool_subagent_control["tool-subagent-control"] pkg_tool_ralph["tool-ralph"] + pkg_experimental_speech_to_text["experimental-speech-to-text"] + svc_speechToText["ctx.speechToText
Experimental speech recognition providers"] + pkg_experimental_speech_to_text_sensevoice["experimental-speech-to-text-sensevoice"] pkg_experimental_agent_team["experimental-agent-team"] svc_agentTeams["ctx.agentTeams
Agent Teams coordination domain"] pkg_experimental_tool_agent_team["experimental-tool-agent-team"] @@ -284,12 +289,15 @@ flowchart LR pkg_credentials_local --> svc_credentials pkg_deepseek_llm_api_extensions --> svc_deepseekLlmApiExtensions pkg_experimental_agent_team --> svc_agentTeams + pkg_experimental_api_speech_to_text --> svc_speechController pkg_experimental_browser_use_chrome_devtools_mcp --> svc_browserUse pkg_experimental_browser_use_playwright_mcp --> svc_browserUse pkg_experimental_browser_use_stagehand_native --> svc_browserUse pkg_experimental_computer_use_cua_driver_mcp --> svc_computerUse pkg_experimental_computer_use_cua_driver_native --> svc_computerUse pkg_experimental_ptc_runtime_python --> svc_ptcRuntime + pkg_experimental_speech_to_text --> svc_speechToText + pkg_experimental_speech_to_text_sensevoice --> svc_speechToText pkg_file_reference --> svc_fileReferences pkg_file_reference_local --> svc_fileReferences pkg_fs --> svc_fs @@ -475,6 +483,7 @@ flowchart LR svc_shellEnv --> pkg_tool_bash svc_shellEnv --> pkg_tool_pwsh svc_skills --> pkg_tool_skill + svc_speechToText --> pkg_experimental_api_speech_to_text svc_spillStore --> pkg_spill_policy svc_ssh --> pkg_fs_ssh svc_ssh --> pkg_sandbox_ssh @@ -540,6 +549,7 @@ flowchart LR | `ctx.tokenMeter` | `core` | [`token-meter`](../packages/llm/token-meter) | - | [`compaction-basic`](../packages/compaction/compaction-basic) | - | Owns isolated per-session replay folds; pressure consumers share immutable revisioned measurements. | | `ctx.toolResultPruner` | `core` | [`compaction-tool-result-pruner`](../packages/compaction/compaction-tool-result-pruner) | - | [`compaction-basic`](../packages/compaction/compaction-basic) | - | Rewrites oversized current tool results through replayable single-node surface replacements before summary compaction. | | `ctx.sessions` | `core` | [`session`](../packages/core/session) | - | [`agent-loop`](../packages/core/agent-loop), [`agent`](../packages/core/agent), [`session-persistence`](../packages/session/session-persistence), [`session-query`](../packages/session-query/session-query), [`session-query-sqlite`](../packages/session-query/session-query-sqlite), [`subagent-in-process-driver`](../packages/subagent/subagent-in-process-driver), [`invariants`](../packages/runtime-diagnostics/invariants), [`message-feedback`](../packages/feedback/message-feedback) | - | Owns append-only Session instances and emits the durable session event feed. | +| `ctx.speechController` | `core` | [`experimental-api-speech-to-text`](../packages/experimental/api-speech-to-text) | - | - | - | Validates bounded browser audio before provider dispatch. | | `ctx.sessionController` | `core` | [`api-session-controller`](../packages/api/session-controller) | - | - | - | Owns Session commands, cold reads, durable-event following, live control state, model catalogs, workspace opening, and Agent activation policy. | | `ctx.sessionFileReferences` | `core` | [`api-session-controller`](../packages/api/session-controller) | - | - | - | Delegates file-reference discovery through the Session Controller's established Agent lookup policy. | | `ctx.sessionSkillCatalog` | `core` | [`api-session-controller`](../packages/api/session-controller) | - | - | - | Lists the Session composition's user-invocable skills without activating a cold Agent. | @@ -594,6 +604,7 @@ flowchart LR | `ctx.fs` | `seam` | [`fs`](../packages/fs/fs) | [`fs-local`](../packages/fs/fs-local), [`fs-sandbox`](../packages/fs/fs-sandbox), [`fs-ssh`](../packages/ssh/fs-ssh) | [`tool-fs`](../packages/fs/tool-fs) | [`fs-observation-policy`](../packages/fs/fs-observation-policy) | tool-fs executes read/write/edit through ctx.fs; fs-sandbox fences mutations by the shared sandbox mode; fs-observation-policy contributes observed-state checks through the fs/* event gate. | | `ctx.compaction` | `seam` | [`compaction`](../packages/compaction/compaction) | [`compaction-basic`](../packages/compaction/compaction-basic) | [`compaction-basic`](../packages/compaction/compaction-basic) | - | The basic backend consumes post-step pressure and request-error recovery events; there is no model-facing compact tool. | | `ctx.subagents` | `seam` | [`subagent`](../packages/subagent/subagent) | [`subagent-spawn-in-process`](../packages/subagent/subagent-spawn-in-process), [`subagent-fork-in-process`](../packages/subagent/subagent-fork-in-process), [`subagent-acp`](../packages/subagent/subagent-acp), [`subagent-codex`](../packages/subagent/subagent-codex), [`subagent-claude-code`](../packages/subagent/subagent-claude-code), [`subagent-dsh-sdk`](../packages/subagent/subagent-dsh-sdk) | [`tool-subagent`](../packages/subagent/tool-subagent), [`tool-subagent-control`](../packages/subagent/tool-subagent-control), [`tool-ralph`](../packages/workflow/tool-ralph) | - | Providers implement transports; the service also owns optional Activation-based continuation orchestration, tool-subagent selects one-shot or continuable delegation, tool-subagent-control delivers follow-ups, and tool-ralph requires one fresh structured-output route. | +| `ctx.speechToText` | `seam` | [`experimental-speech-to-text`](../packages/experimental/speech-to-text) | [`experimental-speech-to-text-sensevoice`](../packages/experimental/speech-to-text-sensevoice) | [`experimental-api-speech-to-text`](../packages/experimental/api-speech-to-text) | - | Routes explicit recognizers; the browser uses the authenticated Remote and keeps transcripts in the draft until submission. | | `ctx.agentTeams` | `core` | [`experimental-agent-team`](../packages/experimental/agent-team) | - | [`experimental-tool-agent-team`](../packages/experimental/tool-agent-team), [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | - | Owns the implicit-root roster, durable peer mailbox, shared task DAG, continuable-child lifecycle, and generated Team Remote methods; tool-agent-team contributes model controls and client-ui-agent-team mounts the browser contribution. | | `ctx.inspector` | `core` | `inspector` | - | - | - | Owns the Worker-hosted CDP target and the transport-independent Host and Client observation and Cordis-tree query API. | | `ctx.jobs` | `seam` | [`jobs`](../packages/jobs/jobs) | [`jobs-local`](../packages/jobs/jobs-local) | [`tool-bash`](../packages/shell/tool-bash), [`tool-terminal`](../packages/terminal/tool-terminal), [`tool-subagent`](../packages/subagent/tool-subagent), [`tool-jobs`](../packages/jobs/tool-jobs) | - | Producers (background bash, PTY sends, and subagent delegations) register running work; tool-jobs is the model-facing controller that reads, lists, and kills it; jobs-local is the process-local registry. | diff --git a/docs/capability-seams.zh.md b/docs/capability-seams.zh.md index fc9c8fd463..c825f666b0 100644 --- a/docs/capability-seams.zh.md +++ b/docs/capability-seams.zh.md @@ -59,6 +59,8 @@ flowchart LR pkg_subagent_in_process_driver["subagent-in-process-driver"] pkg_invariants["invariants"] pkg_message_feedback["message-feedback"] + pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"] + svc_speechController["ctx.speechController
Experimental transcription Remote"] svc_sessionController["ctx.sessionController
Host Session Remote controller"] svc_sessionFileReferences["ctx.sessionFileReferences
Session-addressed file-reference Remote adapter"] svc_sessionSkillCatalog["ctx.sessionSkillCatalog
Session-addressed skill Remote adapter"] @@ -209,6 +211,9 @@ flowchart LR pkg_subagent_dsh_sdk["subagent-dsh-sdk"] pkg_tool_subagent_control["tool-subagent-control"] pkg_tool_ralph["tool-ralph"] + pkg_experimental_speech_to_text["experimental-speech-to-text"] + svc_speechToText["ctx.speechToText
Experimental speech recognition providers"] + pkg_experimental_speech_to_text_sensevoice["experimental-speech-to-text-sensevoice"] pkg_experimental_agent_team["experimental-agent-team"] svc_agentTeams["ctx.agentTeams
Agent Teams coordination domain"] pkg_experimental_tool_agent_team["experimental-tool-agent-team"] @@ -286,12 +291,15 @@ flowchart LR pkg_credentials_local --> svc_credentials pkg_deepseek_llm_api_extensions --> svc_deepseekLlmApiExtensions pkg_experimental_agent_team --> svc_agentTeams + pkg_experimental_api_speech_to_text --> svc_speechController pkg_experimental_browser_use_chrome_devtools_mcp --> svc_browserUse pkg_experimental_browser_use_playwright_mcp --> svc_browserUse pkg_experimental_browser_use_stagehand_native --> svc_browserUse pkg_experimental_computer_use_cua_driver_mcp --> svc_computerUse pkg_experimental_computer_use_cua_driver_native --> svc_computerUse pkg_experimental_ptc_runtime_python --> svc_ptcRuntime + pkg_experimental_speech_to_text --> svc_speechToText + pkg_experimental_speech_to_text_sensevoice --> svc_speechToText pkg_file_reference --> svc_fileReferences pkg_file_reference_local --> svc_fileReferences pkg_fs --> svc_fs @@ -477,6 +485,7 @@ flowchart LR svc_shellEnv --> pkg_tool_bash svc_shellEnv --> pkg_tool_pwsh svc_skills --> pkg_tool_skill + svc_speechToText --> pkg_experimental_api_speech_to_text svc_spillStore --> pkg_spill_policy svc_ssh --> pkg_fs_ssh svc_ssh --> pkg_sandbox_ssh @@ -542,6 +551,7 @@ flowchart LR | `ctx.tokenMeter` | `core` | [`token-meter`](../packages/llm/token-meter) | - | [`compaction-basic`](../packages/compaction/compaction-basic) | - | 拥有按会话隔离的回放折叠区；压力消费方共享不可变且带修订版本的测量结果。 | | `ctx.toolResultPruner` | `core` | [`compaction-tool-result-pruner`](../packages/compaction/compaction-tool-result-pruner) | - | [`compaction-basic`](../packages/compaction/compaction-basic) | - | 在摘要压缩前，通过可回放的单节点表层替换来改写过大的当前工具结果。 | | `ctx.sessions` | `core` | [`session`](../packages/core/session) | - | [`agent-loop`](../packages/core/agent-loop), [`agent`](../packages/core/agent), [`session-persistence`](../packages/session/session-persistence), [`session-query`](../packages/session-query/session-query), [`session-query-sqlite`](../packages/session-query/session-query-sqlite), [`subagent-in-process-driver`](../packages/subagent/subagent-in-process-driver), [`invariants`](../packages/runtime-diagnostics/invariants), [`message-feedback`](../packages/feedback/message-feedback) | - | 拥有仅追加的 Session 实例，并发出持久的会话事件流。 | +| `ctx.speechController` | `core` | [`experimental-api-speech-to-text`](../packages/experimental/api-speech-to-text) | - | - | - | 在 Provider 调用前校验受限的浏览器音频。 | | `ctx.sessionController` | `core` | [`api-session-controller`](../packages/api/session-controller) | - | - | - | 负责 Session 命令、冷读取、持久事件跟随、实时控制状态、模型目录、workspace 打开与 Agent 激活策略。 | | `ctx.sessionFileReferences` | `core` | [`api-session-controller`](../packages/api/session-controller) | - | - | - | 通过 Session Controller 的既有 Agent lookup 策略委托文件引用发现。 | | `ctx.sessionSkillCatalog` | `core` | [`api-session-controller`](../packages/api/session-controller) | - | - | - | 在不激活冷 Agent 的前提下列出 Session 组合中允许用户调用的 skill。 | @@ -596,6 +606,7 @@ flowchart LR | `ctx.fs` | `seam` | [`fs`](../packages/fs/fs) | [`fs-local`](../packages/fs/fs-local), [`fs-sandbox`](../packages/fs/fs-sandbox), [`fs-ssh`](../packages/ssh/fs-ssh) | [`tool-fs`](../packages/fs/tool-fs) | [`fs-observation-policy`](../packages/fs/fs-observation-policy) | tool-fs 通过 ctx.fs 执行读取／写入／编辑；fs-sandbox 按共享沙箱模式限制变更；fs-observation-policy 通过 fs/* 事件门禁贡献基于观测状态的检查。 | | `ctx.compaction` | `seam` | [`compaction`](../packages/compaction/compaction) | [`compaction-basic`](../packages/compaction/compaction-basic) | [`compaction-basic`](../packages/compaction/compaction-basic) | - | 基础后端消费步骤后的压力事件和请求错误恢复事件；不存在面向模型的压缩工具。 | | `ctx.subagents` | `seam` | [`subagent`](../packages/subagent/subagent) | [`subagent-spawn-in-process`](../packages/subagent/subagent-spawn-in-process), [`subagent-fork-in-process`](../packages/subagent/subagent-fork-in-process), [`subagent-acp`](../packages/subagent/subagent-acp), [`subagent-codex`](../packages/subagent/subagent-codex), [`subagent-claude-code`](../packages/subagent/subagent-claude-code), [`subagent-dsh-sdk`](../packages/subagent/subagent-dsh-sdk) | [`tool-subagent`](../packages/subagent/tool-subagent), [`tool-subagent-control`](../packages/subagent/tool-subagent-control), [`tool-ralph`](../packages/workflow/tool-ralph) | - | 提供方实现传输；该服务还负责可选的、基于 Activation 的延续编排，tool-subagent 选择一次性或可延续委派，tool-subagent-control 传递后续消息，而 tool-ralph 要求一条全新的结构化输出路由。 | +| `ctx.speechToText` | `seam` | [`experimental-speech-to-text`](../packages/experimental/speech-to-text) | [`experimental-speech-to-text-sensevoice`](../packages/experimental/speech-to-text-sensevoice) | [`experimental-api-speech-to-text`](../packages/experimental/api-speech-to-text) | - | 路由显式选择的识别器；浏览器使用带认证的 Remote，并在提交前将转写保留在草稿中。 | | `ctx.agentTeams` | `core` | [`experimental-agent-team`](../packages/experimental/agent-team) | - | [`experimental-tool-agent-team`](../packages/experimental/tool-agent-team), [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | - | 负责隐式 Root roster、持久 peer mailbox、共享任务 DAG、continuable child 生命周期与生成式 Team Remote method；tool-agent-team 提供模型控制工具，client-ui-agent-team 挂载浏览器 contribution。 | | `ctx.inspector` | `core` | `inspector` | - | - | - | 负责 Worker 托管的 CDP target，以及独立于传输的 Host 和 Client observation 与 Cordis tree query API。 | | `ctx.jobs` | `seam` | [`jobs`](../packages/jobs/jobs) | [`jobs-local`](../packages/jobs/jobs-local) | [`tool-bash`](../packages/shell/tool-bash), [`tool-terminal`](../packages/terminal/tool-terminal), [`tool-subagent`](../packages/subagent/tool-subagent), [`tool-jobs`](../packages/jobs/tool-jobs) | - | 生产方（后台 bash、PTY 发送和 subagent 委派）登记正在运行的工作；tool-jobs 是面向模型的控制器，用于读取、列出和终止这些工作；jobs-local 是进程本地注册表。 | diff --git a/docs/config-catalog.i18n.yaml b/docs/config-catalog.i18n.yaml index c049eab040..f576e2aea7 100644 --- a/docs/config-catalog.i18n.yaml +++ b/docs/config-catalog.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/config-catalog.md -config-catalog.md: c3e84ba47ad687915ae1c641e8600bd43a2102b6 -config-catalog.zh.md: b8b86030d96ebd8ea49cc6e545007a0c319720f3 +config-catalog.md: fe01cf82f37e94bb6d2709e8991652e252ebc196 +config-catalog.zh.md: 02d732455384e90c24a00869f2cde6d5d3a066ee diff --git a/docs/config-catalog.md b/docs/config-catalog.md index c3e84ba47a..fe01cf82f3 100644 --- a/docs/config-catalog.md +++ b/docs/config-catalog.md @@ -572,6 +572,24 @@ export interface Config { Source: [`packages/experimental/agent-team/src/types.ts:130`](../packages/experimental/agent-team/src/types.ts) + + +## `@deepseek-ai/dsh-experimental-api-speech-to-text` + +Requires: `speechToText` · `typert` + +```ts config-catalog +/** Limits applied before decoding or calling a provider. */ +export interface Config { + /** Maximum decoded WAV bytes per request. */ + maxAudioBytes: number + /** Maximum PCM recording duration in seconds. */ + maxDurationSeconds: number +} +``` + +Source: [`packages/experimental/api-speech-to-text/src/index.ts:20`](../packages/experimental/api-speech-to-text/src/index.ts) + ## `@deepseek-ai/dsh-experimental-browser-use-chrome-devtools-mcp` @@ -802,6 +820,76 @@ export interface Config { Source: [`packages/experimental/ptc-runtime-python/src/index.ts:42`](../packages/experimental/ptc-runtime-python/src/index.ts) + + +## `@deepseek-ai/dsh-experimental-speech-to-text` + +```ts config-catalog +/** Composition defaults resolved before a transcription starts. */ +export interface Config { + /** Registered provider selected when the caller omits an id. */ + defaultProvider: string + /** Provider language hint selected when the caller omits one. */ + language: string +} +``` + +Source: [`packages/experimental/speech-to-text/src/index.ts:18`](../packages/experimental/speech-to-text/src/index.ts) + + + +## `@deepseek-ai/dsh-experimental-speech-to-text-sensevoice` + +Requires: `speechToText` · `subprocess` + +```ts config-catalog +/** Local runtime, inference, and retention settings. */ +export interface Config { + /** Unique registration id; consumers select this exact id. */ + providerId: string + /** Absolute directory for verified ONNX models. */ + dataRoot: string + /** Existing directory containing the selected ONNX model and tokens.txt; omission downloads verified files. */ + modelDirectory?: string | undefined + /** Existing Silero VAD ONNX file; omission downloads the verified model. */ + vadModelPath?: string | undefined + /** Weight precision; INT8 minimizes first-use download and model storage. */ + precision: 'int8' | 'fp32' + /** Hugging Face-compatible origin for pinned model URLs, including private mirrors. */ + modelOrigin: string + /** CPU intra-operation thread count. */ + threads: number + /** Maximum speech segment length passed to the recognizer. */ + segmentSeconds: number + /** Silero speech probability threshold. */ + vadThreshold: number + /** Minimum speech duration retained by VAD. */ + minSpeechSeconds: number + /** Silence separating two speech segments. */ + minSilenceSeconds: number + /** Maximum decoded WAV bytes accepted by the private worker. */ + maxAudioBytes: number + /** Deadline for runtime preparation and cold model loading. */ + prepareTimeoutMs: number + /** Deadline for one inference after the worker is ready. */ + inferenceTimeoutMs: number + /** Idle period before stopping the worker; zero keeps it warm. */ + idleTimeoutMs: number + /** Maximum accepted running and waiting transcriptions. */ + maxPending: number + /** Managed process termination grace period. */ + graceMs: number + /** Maximum retained worker diagnostic bytes. */ + maxLogBytes: number + /** Maximum transcript response bytes. */ + maxResponseBytes: number + /** Minimum interval between intermediate download progress notifications. */ + progressIntervalMs: number +} +``` + +Source: [`packages/experimental/speech-to-text-sensevoice/src/config.ts:6`](../packages/experimental/speech-to-text-sensevoice/src/config.ts) + ## `@deepseek-ai/dsh-experimental-tool-agent-team` @@ -3761,6 +3849,7 @@ These load from a `cordis.yml` entry with no `config:` block; they declare no co - `@deepseek-ai/dsh-deepseek-llm-api-extensions` ([`packages/llm/deepseek-llm-api-extensions/src/index.ts`](../packages/llm/deepseek-llm-api-extensions/src/index.ts)) - `@deepseek-ai/dsh-experimental-auto-review` — requires `llm` · `permissionPresets` · `sessions` · `tools` ([`packages/experimental/auto-review/src/index.ts`](../packages/experimental/auto-review/src/index.ts)) - `@deepseek-ai/dsh-experimental-client-ui-agent-team` ([`packages/experimental/client-ui-agent-team/src/index.ts`](../packages/experimental/client-ui-agent-team/src/index.ts)) +- `@deepseek-ai/dsh-experimental-client-ui-voice-input` ([`packages/experimental/client-ui-voice-input/src/index.ts`](../packages/experimental/client-ui-voice-input/src/index.ts)) - `@deepseek-ai/dsh-experimental-computer-use-cua-driver-native` — requires `computerUse` · `tools` · `systemPrompt` ([`packages/experimental/computer-use-cua-driver-native/src/index.ts`](../packages/experimental/computer-use-cua-driver-native/src/index.ts)) - `@deepseek-ai/dsh-fs-observation-policy` ([`packages/fs/fs-observation-policy/src/index.ts`](../packages/fs/fs-observation-policy/src/index.ts)) - `@deepseek-ai/dsh-fs-ssh` — requires `ssh` · `sandboxPolicy` ([`packages/ssh/fs-ssh/src/index.ts`](../packages/ssh/fs-ssh/src/index.ts)) @@ -3834,6 +3923,7 @@ Imported as libraries by other packages; a `cordis.yml` cannot load them. - `@deepseek-ai/dsh-experimental-agent-team-profile` ([`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts)) - `@deepseek-ai/dsh-experimental-agent-team-web-profile` ([`packages/experimental/agent-team-web-profile/src/index.ts`](../packages/experimental/agent-team-web-profile/src/index.ts)) - `@deepseek-ai/dsh-experimental-browser-use-runtime` ([`packages/experimental/browser-use-runtime/src/index.ts`](../packages/experimental/browser-use-runtime/src/index.ts)) +- `@deepseek-ai/dsh-experimental-voice-input-bundle` ([`packages/experimental/voice-input-bundle/src/index.ts`](../packages/experimental/voice-input-bundle/src/index.ts)) - `@deepseek-ai/dsh-experimental-webworker-packer` ([`packages/experimental/webworker-packer/src/index.ts`](../packages/experimental/webworker-packer/src/index.ts)) - `@deepseek-ai/dsh-experimental-webworker-runtime` ([`packages/experimental/webworker-runtime/src/index.ts`](../packages/experimental/webworker-runtime/src/index.ts)) - `@deepseek-ai/dsh-home-paths` ([`packages/util/home-paths/src/index.ts`](../packages/util/home-paths/src/index.ts)) diff --git a/docs/config-catalog.zh.md b/docs/config-catalog.zh.md index b8b86030d9..02d7324553 100644 --- a/docs/config-catalog.zh.md +++ b/docs/config-catalog.zh.md @@ -574,6 +574,24 @@ export interface Config { 来源：[`packages/experimental/agent-team/src/types.ts:130`](../packages/experimental/agent-team/src/types.ts) + + +## `@deepseek-ai/dsh-experimental-api-speech-to-text` + +需要： `speechToText` · `typert` + +```ts config-catalog +/** Limits applied before decoding or calling a provider. */ +export interface Config { + /** Maximum decoded WAV bytes per request. */ + maxAudioBytes: number + /** Maximum PCM recording duration in seconds. */ + maxDurationSeconds: number +} +``` + +来源： [`packages/experimental/api-speech-to-text/src/index.ts:20`](../packages/experimental/api-speech-to-text/src/index.ts) + ## `@deepseek-ai/dsh-experimental-browser-use-chrome-devtools-mcp` @@ -804,6 +822,76 @@ export interface Config { 来源：[`packages/experimental/ptc-runtime-python/src/index.ts:42`](../packages/experimental/ptc-runtime-python/src/index.ts) + + +## `@deepseek-ai/dsh-experimental-speech-to-text` + +```ts config-catalog +/** Composition defaults resolved before a transcription starts. */ +export interface Config { + /** Registered provider selected when the caller omits an id. */ + defaultProvider: string + /** Provider language hint selected when the caller omits one. */ + language: string +} +``` + +来源： [`packages/experimental/speech-to-text/src/index.ts:18`](../packages/experimental/speech-to-text/src/index.ts) + + + +## `@deepseek-ai/dsh-experimental-speech-to-text-sensevoice` + +需要： `speechToText` · `subprocess` + +```ts config-catalog +/** Local runtime, inference, and retention settings. */ +export interface Config { + /** Unique registration id; consumers select this exact id. */ + providerId: string + /** Absolute directory for verified ONNX models. */ + dataRoot: string + /** Existing directory containing the selected ONNX model and tokens.txt; omission downloads verified files. */ + modelDirectory?: string | undefined + /** Existing Silero VAD ONNX file; omission downloads the verified model. */ + vadModelPath?: string | undefined + /** Weight precision; INT8 minimizes first-use download and model storage. */ + precision: 'int8' | 'fp32' + /** Hugging Face-compatible origin for pinned model URLs, including private mirrors. */ + modelOrigin: string + /** CPU intra-operation thread count. */ + threads: number + /** Maximum speech segment length passed to the recognizer. */ + segmentSeconds: number + /** Silero speech probability threshold. */ + vadThreshold: number + /** Minimum speech duration retained by VAD. */ + minSpeechSeconds: number + /** Silence separating two speech segments. */ + minSilenceSeconds: number + /** Maximum decoded WAV bytes accepted by the private worker. */ + maxAudioBytes: number + /** Deadline for runtime preparation and cold model loading. */ + prepareTimeoutMs: number + /** Deadline for one inference after the worker is ready. */ + inferenceTimeoutMs: number + /** Idle period before stopping the worker; zero keeps it warm. */ + idleTimeoutMs: number + /** Maximum accepted running and waiting transcriptions. */ + maxPending: number + /** Managed process termination grace period. */ + graceMs: number + /** Maximum retained worker diagnostic bytes. */ + maxLogBytes: number + /** Maximum transcript response bytes. */ + maxResponseBytes: number + /** Minimum interval between intermediate download progress notifications. */ + progressIntervalMs: number +} +``` + +来源： [`packages/experimental/speech-to-text-sensevoice/src/config.ts:6`](../packages/experimental/speech-to-text-sensevoice/src/config.ts) + ## `@deepseek-ai/dsh-experimental-tool-agent-team` @@ -3763,6 +3851,7 @@ export interface Config { - `@deepseek-ai/dsh-deepseek-llm-api-extensions`（[`packages/llm/deepseek-llm-api-extensions/src/index.ts`](../packages/llm/deepseek-llm-api-extensions/src/index.ts)） - `@deepseek-ai/dsh-experimental-auto-review` — 需要 `llm` · `permissionPresets` · `sessions` · `tools`（[`packages/experimental/auto-review/src/index.ts`](../packages/experimental/auto-review/src/index.ts)） - `@deepseek-ai/dsh-experimental-client-ui-agent-team`（[`packages/experimental/client-ui-agent-team/src/index.ts`](../packages/experimental/client-ui-agent-team/src/index.ts)） +- `@deepseek-ai/dsh-experimental-client-ui-voice-input` ([`packages/experimental/client-ui-voice-input/src/index.ts`](../packages/experimental/client-ui-voice-input/src/index.ts)) - `@deepseek-ai/dsh-experimental-computer-use-cua-driver-native` — requires `computerUse` · `tools` · `systemPrompt` ([`packages/experimental/computer-use-cua-driver-native/src/index.ts`](../packages/experimental/computer-use-cua-driver-native/src/index.ts)) - `@deepseek-ai/dsh-fs-observation-policy`（[`packages/fs/fs-observation-policy/src/index.ts`](../packages/fs/fs-observation-policy/src/index.ts)） - `@deepseek-ai/dsh-fs-ssh` — 需要 `ssh` · `sandboxPolicy`（[`packages/ssh/fs-ssh/src/index.ts`](../packages/ssh/fs-ssh/src/index.ts)） @@ -3835,6 +3924,7 @@ export interface Config { - `@deepseek-ai/dsh-experimental-agent-team-profile`（[`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts)） - `@deepseek-ai/dsh-experimental-agent-team-web-profile`（[`packages/experimental/agent-team-web-profile/src/index.ts`](../packages/experimental/agent-team-web-profile/src/index.ts)） - `@deepseek-ai/dsh-experimental-browser-use-runtime` ([`packages/experimental/browser-use-runtime/src/index.ts`](../packages/experimental/browser-use-runtime/src/index.ts)) +- `@deepseek-ai/dsh-experimental-voice-input-bundle`（[`packages/experimental/voice-input-bundle/src/index.ts`](../packages/experimental/voice-input-bundle/src/index.ts)） - `@deepseek-ai/dsh-experimental-webworker-packer`（[`packages/experimental/webworker-packer/src/index.ts`](../packages/experimental/webworker-packer/src/index.ts)） - `@deepseek-ai/dsh-experimental-webworker-runtime`（[`packages/experimental/webworker-runtime/src/index.ts`](../packages/experimental/webworker-runtime/src/index.ts)） - `@deepseek-ai/dsh-home-paths`（[`packages/util/home-paths/src/index.ts`](../packages/util/home-paths/src/index.ts)） diff --git a/docs/subsystems/README.i18n.yaml b/docs/subsystems/README.i18n.yaml index 92e23f39f1..212654acc1 100644 --- a/docs/subsystems/README.i18n.yaml +++ b/docs/subsystems/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/subsystems/README.md -README.md: b4d6aa226979525ef658f9cf1d33d5ca3e59dcf6 -README.zh.md: 9134fcf37789911aec05dcdc9e9c3dc036a33873 +README.md: 34522c050a224fa5dae6ef78a324125adef59f72 +README.zh.md: d5a132f203cb16810a55c3b699a7a2fb2f37d217 diff --git a/docs/subsystems/README.md b/docs/subsystems/README.md index b4d6aa2269..34522c050a 100644 --- a/docs/subsystems/README.md +++ b/docs/subsystems/README.md @@ -45,6 +45,7 @@ One page per subsystem of the DeepSeek Harness: what it is, the data structures | [skills.md](skills.md) | the skill service: discovery priority, `SkillSummary`/`SkillDefinition`, session-prefix catalog, model-facing `skill` loading | | [compaction.md](compaction.md) | the compaction seam: the `compaction/*` session events, `CompactionResult`, the `CompactionEngine` interface | | [subagent.md](subagent.md) | the subagent seam: the named-provider registry, `SubagentStartRequest`/`Result`/`Run`, the start-time-vs-runtime capability split | +| [voice-input.md](voice-input.md) | experimental named recognizers, transient audio and revision-guarded draft insertion | | [agent-team.md](agent-team.md) | Agent Teams: implicit Lead identity, named continuable teammates, durable peer mailbox, and shared task DAG | | [web.md](web.md) | the web access seam: `WebSearchRequest`/`Result`, `WebFetchRequest`/`Result`, `WebFetchBody`, provider availability, `WebError` | | [spill.md](spill.md) | the spill storage seam: `SaveTextSpill`, `SpillOwner`/`SpillSource`, `SpillRef`, the branded `SpillLocator` | diff --git a/docs/subsystems/README.zh.md b/docs/subsystems/README.zh.md index 9134fcf377..d5a132f203 100644 --- a/docs/subsystems/README.zh.md +++ b/docs/subsystems/README.zh.md @@ -45,6 +45,7 @@ | [skills.md](skills.zh.md) | skill（技能）服务：发现优先级、`SkillSummary`/`SkillDefinition`、会话前缀目录、面向模型的 `skill` 加载 | | [compaction.md](compaction.zh.md) | 压缩（compaction）seam：`compaction/*` 会话事件、`CompactionResult`、`CompactionEngine` 接口 | | [subagent.md](subagent.zh.md) | subagent seam：命名提供方注册表、`SubagentStartRequest`/`Result`/`Run`、启动时与运行时能力拆分 | +| [voice-input.md](voice-input.zh.md) | 实验性具名语音识别、临时音频和受版本保护的草稿插入 | | [agent-team.md](agent-team.zh.md) | Agent Teams：隐式 Lead 身份、具名 continuable teammate、持久 peer mailbox 与共享任务 DAG | | [web.md](web.zh.md) | Web 访问 seam：`WebSearchRequest`/`Result`、`WebFetchRequest`/`Result`、`WebFetchBody`、提供方可用性、`WebError` | | [spill.md](spill.zh.md) | spill 存储 seam：`SaveTextSpill`、`SpillOwner`/`SpillSource`、`SpillRef`、品牌类型 `SpillLocator` | diff --git a/docs/subsystems/slots.i18n.yaml b/docs/subsystems/slots.i18n.yaml index d25c088405..30df105998 100644 --- a/docs/subsystems/slots.i18n.yaml +++ b/docs/subsystems/slots.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/subsystems/slots.md -slots.md: 177bbc5b8f56b7276f271a2ddb7ac71ac8888363 -slots.zh.md: c746ac88704c2e22e5c50ef82a893fde9f091668 +slots.md: ff1d69c08f1ff5093135d4a979eb86ae352147a4 +slots.zh.md: fd9ad5bc8c49febe9e39ba6ab19abd31a4a7d697 diff --git a/docs/subsystems/slots.md b/docs/subsystems/slots.md index 177bbc5b8f..ff1d69c08f 100644 --- a/docs/subsystems/slots.md +++ b/docs/subsystems/slots.md @@ -6,6 +6,8 @@ Slots are the Web Client's typed React composition system. [`dsh-client-ui-slots This page documents slot ownership, component inputs, extension APIs, and the shipped hierarchy. The surrounding boot, Remote, Client model, and Conversation paths are in [Web Client architecture](web-client.md). +`plugins.bundle.status` is keyed by npm package name and renders short status before the bundle switch; `plugins.bundle.config` supplies bundle detail configuration. `conversation.input.activity` supplies one action between the model selector and Send, with toolbar expansion released on unmount. + ## Declaration and lifecycle `SlotMap` is the compile-time registry. A package declaration-merges the key, cardinality, scope, owner props, keyed props, and optional slot-level inject face. The runtime declaration is the matching `children` entry on the component that owns the render location. diff --git a/docs/subsystems/slots.zh.md b/docs/subsystems/slots.zh.md index c746ac8870..fd9ad5bc8c 100644 --- a/docs/subsystems/slots.zh.md +++ b/docs/subsystems/slots.zh.md @@ -6,6 +6,8 @@ Slots 是 Web Client 的类型化 React 组合系统。[`dsh-client-ui-slots`](. 本文记录 slot 的所有权、组件输入、扩展 API 与当前层级。外围的启动、Remote、Client model 与 Conversation 数据通路见 [Web Client 架构](web-client.zh.md)。 +`plugins.bundle.status` 以 npm 包名为 key，在 Bundle 卡片开关前显示简短状态；`plugins.bundle.config` 提供 Bundle 详情配置。`conversation.input.activity` 提供模型选择器与发送按钮之间的单个活动控件，可展开工具栏并在卸载时释放展开状态。 + ## 声明与生命周期 `SlotMap` 是编译期注册表。包通过声明合并写入 key、cardinality（基数）、scope、owner props、keyed props 与可选的 slot 级 inject face。运行时声明则是拥有该渲染位置的组件在 `children` 中给出的对应条目。 diff --git a/docs/subsystems/voice-input.i18n.yaml b/docs/subsystems/voice-input.i18n.yaml new file mode 100644 index 0000000000..bde639522c --- /dev/null +++ b/docs/subsystems/voice-input.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write docs/subsystems/voice-input.md +voice-input.md: d72bf01a372413fbc33a93a8a3a77f5152a3807b +voice-input.zh.md: c45275516166820f9dfac2d7d96c3a43b70d72d8 diff --git a/docs/subsystems/voice-input.md b/docs/subsystems/voice-input.md new file mode 100644 index 0000000000..d72bf01a37 --- /dev/null +++ b/docs/subsystems/voice-input.md @@ -0,0 +1,158 @@ +# Voice input + +English | [中文](voice-input.zh.md) + +Experimental speech recognition has three roles: the [Service Definition](../../packages/experimental/speech-to-text/README.md) routes named providers, the [SenseVoice provider](../../packages/experimental/speech-to-text-sensevoice/README.md) owns local inference, and the [Remote consumer](../../packages/experimental/api-speech-to-text/README.md) serves the browser. The [optional bundle](../../packages/experimental/voice-input-bundle/README.md) composes them with the microphone UI. + +## Provider selection + +`SpeechProviderId` brands the registration identity. `SpeechProviderInfo` carries a display name and `host-local` or `cloud` processing location. `SpeechRequest` contains WAV bytes and optional provider/language selection; `resolve()` produces `SpeechSpec` with one captured provider instance. A missing provider fails; replacement or withdrawal invalidates resolved work. Audio never falls back to a different provider. + +`SpeechProvider.transcribe()` accepts one complete `SpeechInput` and an `AbortSignal`. `Transcript` returns plain `text`, `audioSeconds` and `inferenceSeconds`. Providers honor cancellation and settle their work before deregistration completes. The local provider serializes calls, bounds its queue and manages its worker lifetime independently of Sessions. + +## Browser and Host ownership + +The browser owns microphone tracks and the unsent draft. `TranscriptionRequest` sends canonical base64 PCM16 WAV through the authenticated Remote. `SpeechCatalog` advertises available providers, the default selection and recording byte/duration limits. The API validates this process input before recognition. + +The input facade captures a revision-bearing selection before recording. `InputActions.insertText()` inserts one undoable plain-text edit only while the selection revision is current and submission permits editing. A rejected insertion leaves the transcript available for explicit insertion. Switching Sessions or disposing the plugin invalidates late results. Recognition itself writes no Session event; ordinary user submission owns the final model-visible text. + +## Preparation and settings + +The Host provider owns one preparation task across page and Session changes. The Client shares one `follow()` subscription across the composer, plugin status, details. Optional `SpeechSetupEstimate` metadata supplies provider-specific planning hints, separate from measured progress. `SpeechPreparationStepKind` identifies ordered resource operations; `SpeechPreparationStep` records each status and start time. The complete `SpeechPreparationState` retains these steps after cancellation or failure. The collapsed UI shows the current operation; expanding lists all steps, with byte progress or elapsed time only for the running step. Closing an observer never cancels preparation. Verified files survive retries and idle worker reclamation. + +The microphone occupies `conversation.input.activity`, between the model selector and Send. Clicking starts capture and expands the toolbar; Stop transcribes and inserts into the draft. The activity preserves the editor and submit action, owns local feedback, and releases expansion on unmount. Cancel, Escape or hiding the page discards capture. Waveform history displays measured microphone amplitude. Bundle details store recognition preferences, and `plugins.bundle.status` displays short readiness beside the enable switch. + +## Design rationale + +The [voice input decision](../../.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.md) explains transient audio, explicit provider selection and lazy local preparation. + + + + + +## Cordis API + +Generated from source by `scripts/gen-cordis-catalog.ts` (verified fresh by `pnpm run verify-cordis-catalog` in doc-sync; regenerate with `pnpm run gen-cordis-catalog`) — the language sides differ only in locale-specific paired document paths. Signature blocks use a `ts cordis-catalog` fence and keep the original source JSDoc; dispatch modes are defined in the [primer](../cordis-primer.md#dispatch-modes), and the framework-inherited `ctx` API lives in [cordis-api/inherited.md](../cordis-api/inherited.md). + + + +### `ctx.speechController` — `SpeechController` + +Speech calls never activate or submit to an Agent. + +```ts cordis-catalog +/** + * Read provider choices without preparing a recognizer. + * @returns available providers, resolved default, and recording limits. + */ +@Remote catalog(): SpeechCatalog + +/** + * Follow provider readiness independently of Session and preparation lifetimes. + * @param signal - Client observation lifetime. + * @returns initial and subsequent complete readiness snapshots. + */ +@Remote({ mode: 'stream' }) async *follow(signal: AbortSignal): AsyncIterable + +/** + * Persist the user's recognition preferences. + * @param patch - changed preference fields. + * @returns after preferences are saved. + */ +@Remote configure(patch: SpeechSelectionPatch): Promise + +/** + * Start or join one Host-owned preparation task. + * @param providerId - selected recognizer. + */ +@Remote prepare(providerId: SpeechProviderId): void + +/** + * Explicitly cancel resource preparation. + * @param providerId - selected recognizer. + * @returns after the preparation task settles. + */ +@Remote cancelPreparation(providerId: SpeechProviderId): Promise + +/** + * Validate and transcribe one recording through the explicit provider selection. + * @param request - canonical WAV encoded as base64, provider id and language hint. + * @param signal - Client cancellation or Remote contribution disposal. + * @returns final transcript without adding a Session event. + */ +@Remote async transcribe(request: TranscriptionRequest, signal: AbortSignal): Promise +``` + +Source: [`packages/experimental/api-speech-to-text/src/index.ts`](../../packages/experimental/api-speech-to-text/src/index.ts) + + + +### `ctx.speechToText` — `SpeechToText` + +Registry shared by all transcription consumers in one Host composition. + +```ts cordis-catalog +/** + * Register one recognizer; duplicate ids fail without replacing the original. + * @param provider - recognizer owned by the contributing fiber. + * @returns idempotent disposer which rejects admission, cancels, and joins accepted work. + */ +register(provider: SpeechProvider): () => Promise + +/** + * Read the current recognizer roster. + * @returns available provider facts in registration order. + */ +listProviders(): readonly SpeechProviderInfo[] + +/** + * Observe complete readiness snapshots; a slow reader coalesces intermediate progress. + * @param caller - observer lifetime, independent of any preparation task. + * @returns an initial snapshot followed by the latest provider states. + */ +async *follow(caller: AbortSignal): AsyncIterable + +/** + * Read provider readiness and current user preferences together. + * @returns one detached complete observation. + */ +snapshot(): SpeechSnapshot + +/** + * Persist changed preference fields in the ordinary user-settings document. + * @param patch - explicit provider or language changes. + * @returns after persistence and the resolved preference update. + */ +async configure(patch: SpeechSelectionPatch): Promise + +/** + * Start or join provider-owned preparation. + * @param id - exact registered provider identity. + */ +prepare(id: SpeechProviderId): void + +/** + * Explicitly cancel provider preparation without tying it to a browser connection. + * @param id - exact registered provider identity. + * @returns after the preparation task settles. + */ +async cancelPreparation(id: SpeechProviderId): Promise + +/** + * Apply composition defaults and capture the selected provider. Missing providers fail explicitly. + * @param request - complete recording and optional selection. + * @returns provider-pinned input for transcribe(). + */ +resolve(request: SpeechRequest): SpeechSpec + +/** + * Execute exactly the resolved provider; no fallback sends audio elsewhere. + * @param spec - resolved input; a withdrawn or replaced registration is rejected. + * @param signal - caller cancellation. + * @returns final transcript after provider settlement. + */ +async transcribe(spec: SpeechSpec, signal: AbortSignal): Promise +``` + +Source: [`packages/experimental/speech-to-text/src/index.ts`](../../packages/experimental/speech-to-text/src/index.ts) + diff --git a/docs/subsystems/voice-input.zh.md b/docs/subsystems/voice-input.zh.md new file mode 100644 index 0000000000..c452755161 --- /dev/null +++ b/docs/subsystems/voice-input.zh.md @@ -0,0 +1,158 @@ +# 语音输入 + +[English](voice-input.md) | 中文 + +实验性语音识别包含三个角色：[服务定义](../../packages/experimental/speech-to-text/README.zh.md)路由具名 Provider，[SenseVoice Provider](../../packages/experimental/speech-to-text-sensevoice/README.zh.md)拥有本地推理，[Remote 消费者](../../packages/experimental/api-speech-to-text/README.zh.md)服务浏览器。[可选 Bundle](../../packages/experimental/voice-input-bundle/README.zh.md)将它们与麦克风 UI 组合。 + +## Provider 选择 + +`SpeechProviderId` 为注册标识添加类型品牌。`SpeechProviderInfo` 携带显示名称及 `host-local` 或 `cloud` 处理位置。`SpeechRequest` 包含 WAV 字节和可选的 Provider、语言选择；`resolve()` 生成捕获一个 Provider 实例的 `SpeechSpec`。缺失的 Provider 会报错；替换或撤销会使已解析的任务失效。音频不会回退到其他 Provider。 + +`SpeechProvider.transcribe()` 接收完整的 `SpeechInput` 与 `AbortSignal`。`Transcript` 返回纯文本 `text`、`audioSeconds` 和 `inferenceSeconds`。Provider 响应取消，并在注销完成前结束其任务。本地 Provider 串行执行调用、限制队列，并独立于 Session 管理工作进程生命周期。 + +## 浏览器与 Host 所有权 + +浏览器拥有麦克风轨道与未发送的草稿。`TranscriptionRequest` 通过带认证的 Remote 发送规范 base64 PCM16 WAV。`SpeechCatalog` 公布可用 Provider、默认选择及录音字节和时长限制。API 在识别前校验这些进程输入。 + +输入门面在录音前捕获带版本的选区。`InputActions.insertText()` 仅在选区版本仍有效且提交状态允许编辑时，插入一次可撤销的纯文本编辑。插入被拒绝时，转写文字保留以供显式插入。切换 Session 或释放插件使迟到结果失效。识别本身不写入 Session 事件；普通用户提交拥有最终的模型可见文字。 + +## 准备与设置 + +Host Provider 在页面和 Session 变化期间拥有同一个准备任务。Client 在输入框、插件状态、详情之间共享一份 `follow()` 订阅。可选 `SpeechSetupEstimate` 元数据提供各 Provider 的资源预期，与实测进度分开。`SpeechPreparationStepKind` 标识有序资源操作；`SpeechPreparationStep` 记录各步骤状态与开始时间。完整 `SpeechPreparationState` 在取消或失败后保留这些步骤。折叠 UI 显示当前操作，展开后列出所有步骤，仅进行中步骤显示字节进度或等待时间。关闭观察者不会取消准备。已校验文件在重试及空闲进程回收后继续复用。 + +麦克风占用模型选择器与发送按钮之间的 `conversation.input.activity`。点击开始录音并展开工具栏；停止后转写并插入草稿。活动栏保留编辑器与提交按钮，拥有局部反馈，并在卸载时释放展开状态。取消、Escape 或隐藏页面会丢弃录音。波形历史展示实测麦克风音量。Bundle 详情保存识别偏好，`plugins.bundle.status` 在启用开关旁显示简短准备状态。 + +## 设计依据 + +[语音输入决策](../../.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.zh.md)解释临时音频、显式 Provider 选择和延迟准备本地运行时。 + + + + + +## Cordis API + +Generated from source by `scripts/gen-cordis-catalog.ts` (verified fresh by `pnpm run verify-cordis-catalog` in doc-sync; regenerate with `pnpm run gen-cordis-catalog`) — the language sides differ only in locale-specific paired document paths. Signature blocks use a `ts cordis-catalog` fence and keep the original source JSDoc; dispatch modes are defined in the [primer](../cordis-primer.zh.md#dispatch-modes), and the framework-inherited `ctx` API lives in [cordis-api/inherited.md](../cordis-api/inherited.md). + + + +### `ctx.speechController` — `SpeechController` + +Speech calls never activate or submit to an Agent. + +```ts cordis-catalog +/** + * Read provider choices without preparing a recognizer. + * @returns available providers, resolved default, and recording limits. + */ +@Remote catalog(): SpeechCatalog + +/** + * Follow provider readiness independently of Session and preparation lifetimes. + * @param signal - Client observation lifetime. + * @returns initial and subsequent complete readiness snapshots. + */ +@Remote({ mode: 'stream' }) async *follow(signal: AbortSignal): AsyncIterable + +/** + * Persist the user's recognition preferences. + * @param patch - changed preference fields. + * @returns after preferences are saved. + */ +@Remote configure(patch: SpeechSelectionPatch): Promise + +/** + * Start or join one Host-owned preparation task. + * @param providerId - selected recognizer. + */ +@Remote prepare(providerId: SpeechProviderId): void + +/** + * Explicitly cancel resource preparation. + * @param providerId - selected recognizer. + * @returns after the preparation task settles. + */ +@Remote cancelPreparation(providerId: SpeechProviderId): Promise + +/** + * Validate and transcribe one recording through the explicit provider selection. + * @param request - canonical WAV encoded as base64, provider id and language hint. + * @param signal - Client cancellation or Remote contribution disposal. + * @returns final transcript without adding a Session event. + */ +@Remote async transcribe(request: TranscriptionRequest, signal: AbortSignal): Promise +``` + +Source: [`packages/experimental/api-speech-to-text/src/index.ts`](../../packages/experimental/api-speech-to-text/src/index.ts) + + + +### `ctx.speechToText` — `SpeechToText` + +Registry shared by all transcription consumers in one Host composition. + +```ts cordis-catalog +/** + * Register one recognizer; duplicate ids fail without replacing the original. + * @param provider - recognizer owned by the contributing fiber. + * @returns idempotent disposer which rejects admission, cancels, and joins accepted work. + */ +register(provider: SpeechProvider): () => Promise + +/** + * Read the current recognizer roster. + * @returns available provider facts in registration order. + */ +listProviders(): readonly SpeechProviderInfo[] + +/** + * Observe complete readiness snapshots; a slow reader coalesces intermediate progress. + * @param caller - observer lifetime, independent of any preparation task. + * @returns an initial snapshot followed by the latest provider states. + */ +async *follow(caller: AbortSignal): AsyncIterable + +/** + * Read provider readiness and current user preferences together. + * @returns one detached complete observation. + */ +snapshot(): SpeechSnapshot + +/** + * Persist changed preference fields in the ordinary user-settings document. + * @param patch - explicit provider or language changes. + * @returns after persistence and the resolved preference update. + */ +async configure(patch: SpeechSelectionPatch): Promise + +/** + * Start or join provider-owned preparation. + * @param id - exact registered provider identity. + */ +prepare(id: SpeechProviderId): void + +/** + * Explicitly cancel provider preparation without tying it to a browser connection. + * @param id - exact registered provider identity. + * @returns after the preparation task settles. + */ +async cancelPreparation(id: SpeechProviderId): Promise + +/** + * Apply composition defaults and capture the selected provider. Missing providers fail explicitly. + * @param request - complete recording and optional selection. + * @returns provider-pinned input for transcribe(). + */ +resolve(request: SpeechRequest): SpeechSpec + +/** + * Execute exactly the resolved provider; no fallback sends audio elsewhere. + * @param spec - resolved input; a withdrawn or replaced registration is rejected. + * @param signal - caller cancellation. + * @returns final transcript after provider settlement. + */ +async transcribe(spec: SpeechSpec, signal: AbortSignal): Promise +``` + +Source: [`packages/experimental/speech-to-text/src/index.ts`](../../packages/experimental/speech-to-text/src/index.ts) + diff --git a/packages/boot/app-boot/src/profile.ts b/packages/boot/app-boot/src/profile.ts index d187f3c10a..71a8da853c 100644 --- a/packages/boot/app-boot/src/profile.ts +++ b/packages/boot/app-boot/src/profile.ts @@ -165,6 +165,7 @@ export const DEFAULT_PROFILE_BUNDLES: readonly string[] = ['@deepseek-ai/dsh-bas * manager ([rationale](../../../../.agents/notes/implemented/process/2026-09-15-shipped-optional-bundles.md)). */ export const OPTIONAL_BUNDLES: readonly string[] = [ + '@deepseek-ai/dsh-experimental-voice-input-bundle', '@deepseek-ai/dsh-experimental-agent-team-profile', '@deepseek-ai/dsh-experimental-agent-team-web-profile', ] diff --git a/packages/client/ui-attachment/tests/message-image.client.spec.tsx b/packages/client/ui-attachment/tests/message-image.client.spec.tsx index 3d3711a77a..e195dd6533 100644 --- a/packages/client/ui-attachment/tests/message-image.client.spec.tsx +++ b/packages/client/ui-attachment/tests/message-image.client.spec.tsx @@ -282,6 +282,8 @@ describe('ImageGallery', () => { useTrajectory, useInput, inputActions: { + captureInsertion: () => ({ start: 0, end: 0, draftRev: 0 }), + insertText: () => false, setDraft: vi.fn(), addAttachments: vi.fn(() => true), removeAttachment: vi.fn(), diff --git a/packages/client/ui-chat/tests/chat-view.client.spec.tsx b/packages/client/ui-chat/tests/chat-view.client.spec.tsx index 8a63203668..546c13c7ff 100644 --- a/packages/client/ui-chat/tests/chat-view.client.spec.tsx +++ b/packages/client/ui-chat/tests/chat-view.client.spec.tsx @@ -392,6 +392,8 @@ function makeHarness( }, useInput: (() => { throw new Error('unused') }), inputActions: { + captureInsertion: () => ({ start: 0, end: 0, draftRev: 0 }), + insertText: () => false, setDraft: () => {}, addAttachments: () => true, removeAttachment: () => {}, diff --git a/packages/client/ui-conversation/README.i18n.yaml b/packages/client/ui-conversation/README.i18n.yaml index 6f234d0b92..1217ad6dd9 100644 --- a/packages/client/ui-conversation/README.i18n.yaml +++ b/packages/client/ui-conversation/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/client/ui-conversation/README.md -README.md: 749c2d0d99ce7e5afe2f9a03b1451d62a17206e7 -README.zh.md: 98d7d9672922ca68ea4df5499f8789e8c2fde162 +README.md: 83fcfbb527e631e25bddb532fa60d4def0ef33f0 +README.zh.md: 80f7df87e3277729e37f573371cf3d2b21dff721 diff --git a/packages/client/ui-conversation/README.md b/packages/client/ui-conversation/README.md index 749c2d0d99..83fcfbb527 100644 --- a/packages/client/ui-conversation/README.md +++ b/packages/client/ui-conversation/README.md @@ -114,6 +114,10 @@ try { The selector must be a pure function of the owner currency. Its non-null return is delivered to the component as `matched`; `PropsRuntime<'conversation.composer'>` supplies the standard Session and global props. Chain order remains ascending `priority`, then registration order, and the first non-null selector wins. The shell keeps the default composer mounted beneath a takeover. Request state, listeners, response encoding, and any request-specific child slots belong to the business package; they are not carried by `SessionSnapshot` or declared by this core package. +`InputActions.captureInsertion()` captures the draft selection and revision; `insertText(text, span)` inserts one undoable plain-text edit only while that revision is current and the editor permits editing. Asynchronous consumers retain rejected results for user action. + +`conversation.input.activity` hosts one control between the model selector and Send. Its `onActiveChange` callback expands that control across the toolbar and hides ordinary accessory controls while preserving the editor and submit action. The occupant releases expansion on unmount and owns any activity-specific feedback. + ## Model Experience diff --git a/packages/client/ui-conversation/README.zh.md b/packages/client/ui-conversation/README.zh.md index 98d7d96729..80f7df87e3 100644 --- a/packages/client/ui-conversation/README.zh.md +++ b/packages/client/ui-conversation/README.zh.md @@ -114,6 +114,10 @@ try { selector 必须是 owner currency 的纯函数。非 null 返回值作为 `matched` 传给组件；`PropsRuntime<'conversation.composer'>` 提供标准 Session 与 global props。Chain 顺序仍按 `priority` 升序，再按注册顺序；首个返回非 null 的 selector 获选。Shell 会在 takeover 下保持默认 composer 挂载。Request 状态、listener、response encoding 和任何 request-specific child slot 都属于业务 package，不进入 `SessionSnapshot`，也不由 core 包声明。 +`InputActions.captureInsertion()` 捕获草稿选区与版本；`insertText(text, span)` 仅在版本未变且编辑器允许编辑时，插入一次可撤销的纯文本编辑。异步消费者在插入被拒绝后负责保留结果，等待用户操作。 + +`conversation.input.activity` 在模型选择器与发送按钮之间承载一个控件。其 `onActiveChange` 回调将控件展开至整条工具栏并隐藏普通辅助控件，同时保留编辑器和提交按钮。占用者在卸载时释放展开状态，并拥有活动专属反馈。 + ## 模型体验 diff --git a/packages/client/ui-conversation/src/client/apply.ts b/packages/client/ui-conversation/src/client/apply.ts index 1402639dac..27e96749fd 100644 --- a/packages/client/ui-conversation/src/client/apply.ts +++ b/packages/client/ui-conversation/src/client/apply.ts @@ -323,6 +323,7 @@ export function apply(ctx: Context, config: Config = Config({})): void { 'conversation.input.plan': { kind: 'single', scope: 'session' }, 'conversation.input.right': { kind: 'list', scope: 'session' }, 'conversation.input.model': { kind: 'single', scope: 'session' }, + 'conversation.input.activity': { kind: 'single', scope: 'session' }, 'conversation.composer.dock': { kind: 'list', scope: 'session' }, }, inject: (sessionId: SessionId | undefined): ComposerBarInjected => { diff --git a/packages/client/ui-conversation/src/client/contract/input.ts b/packages/client/ui-conversation/src/client/contract/input.ts index b4cd6b9bdf..4036736374 100644 --- a/packages/client/ui-conversation/src/client/contract/input.ts +++ b/packages/client/ui-conversation/src/client/contract/input.ts @@ -220,6 +220,15 @@ export interface SessionInputResolver { * paste/…) stay InputBar-private and never ride this face. */ export interface InputActions { + /** @returns a revision-guarded copy of the current editor selection. */ + captureInsertion(): TokenSpan + /** + * Insert asynchronous text without replacing subsequent edits or reference chips. + * @param text - plain text to insert in one undo step. + * @param span - captured selection and draft revision. + * @returns false when the draft changed or submission locked the editor. + */ + insertText(text: string, span: TokenSpan): boolean /** Replace the whole draft (persisted-draft seed and programmatic writes). */ setDraft(text: string): void /** Append ordered browser-owned attachment ids; busy admission phases refuse. */ diff --git a/packages/client/ui-conversation/src/client/contract/slots.ts b/packages/client/ui-conversation/src/client/contract/slots.ts index cccd6a0fe6..b1ef334cb5 100644 --- a/packages/client/ui-conversation/src/client/contract/slots.ts +++ b/packages/client/ui-conversation/src/client/contract/slots.ts @@ -183,6 +183,8 @@ declare module '@deepseek-ai/dsh-client-ui-slots' { 'conversation.input.left': { kind: 'list'; scope: 'session' } /** Compact controls before the composer submit action. */ 'conversation.input.right': { kind: 'list'; scope: 'session' } + /** Compact action after the model selector; it can expand across the toolbar while retaining the editor and submit action. */ + 'conversation.input.activity': { kind: 'single'; scope: 'session'; owner: InputActivityOwnerProps } /** Resident composer body, including the no-Session inert state. */ 'conversation.composer.bar': { kind: 'single'; scope: 'session-maybe'; owner: ComposerBarOwnerProps } /** Optional draft-attachment rail and drop target. */ @@ -351,6 +353,12 @@ export interface InputControlOwnerProps { locked: boolean } +/** A toolbar activity hides ordinary accessory controls while expanded; its occupant must release expansion on unmount. */ +export interface InputActivityOwnerProps extends InputControlOwnerProps { + /** @param active - whether the occupant needs the toolbar width before the submit action. */ + onActiveChange: (active: boolean) => void +} + /** Full props of the resident composer bar. */ export type ComposerBarProps = PropsRuntime<'conversation.composer.bar'> @@ -358,7 +366,7 @@ export type ComposerBarProps = | 'conversation.input.attachments' | 'conversation.input.overlay' | 'conversation.input.permission' | 'conversation.input.left' | 'conversation.input.plan' - | 'conversation.input.right' | 'conversation.input.model' + | 'conversation.input.right' | 'conversation.input.model' | 'conversation.input.activity' | 'conversation.composer.dock' > & InjectFace diff --git a/packages/client/ui-conversation/src/client/input/editor/runtime.ts b/packages/client/ui-conversation/src/client/input/editor/runtime.ts index 2054624342..ca6047c4ec 100644 --- a/packages/client/ui-conversation/src/client/input/editor/runtime.ts +++ b/packages/client/ui-conversation/src/client/input/editor/runtime.ts @@ -210,6 +210,19 @@ export class DraftEditorRuntime { return applied } + /** + * Insert an asynchronous text result as one independent undo operation. + * @param span - owner-validated insertion range. + * @param text - text sanitized with the same rules as paste. + * @returns whether the range mapped and the edit applied. + */ + insertAsyncText(span: DetectSpan, text: string): boolean { + let applied = false + const clean = text.replace(REFERENCE_PLACEHOLDER_RE, '') + this.applyEdit(() => { applied = $replaceDetectSpanWithText(span, clean) }, PASTE_TAG) + return applied + } + /** * Insert a reference chip with the existing trailing-space rule. * @param span - detect-coordinate range. diff --git a/packages/client/ui-conversation/src/client/input/facade.ts b/packages/client/ui-conversation/src/client/input/facade.ts index 4f4d0aa300..8931d1c4d0 100644 --- a/packages/client/ui-conversation/src/client/input/facade.ts +++ b/packages/client/ui-conversation/src/client/input/facade.ts @@ -117,6 +117,12 @@ export class SessionInputShell implements SessionInput { } /** The public provide-channel action face (one stable identity per session). */ readonly actions: InputActions = { + captureInsertion: () => ({ ...this.caretSpan(), draftRev: this.rev }), + insertText: (text, span) => { + if (this.snapshot.phase === 'adjudicating' || this.snapshot.phase === 'submitting' || this.disposed) return false + if (span.draftRev !== this.rev) return false + return this.draftEditor.insertAsyncText(span, text) + }, setDraft: (text) => { this.setDraft(text) }, addAttachments: ids => this.addAttachments(ids), removeAttachment: (id) => { this.removeAttachment(id) }, diff --git a/packages/client/ui-conversation/src/client/skeleton/InputBar.module.css b/packages/client/ui-conversation/src/client/skeleton/InputBar.module.css index 6369e5728d..4e6764a14a 100644 --- a/packages/client/ui-conversation/src/client/skeleton/InputBar.module.css +++ b/packages/client/ui-conversation/src/client/skeleton/InputBar.module.css @@ -392,3 +392,10 @@ font-size: 12px; cursor: pointer; } + +.standardControls { display: flex; align-items: center; gap: 12px; min-width: 0; } +.tools[hidden], .standardControls[hidden] { display: none; } +.trailingActive { flex: 1; margin-left: 0; } +.activity { display: flex; align-items: center; } +.activity:empty { display: none; } +.activityExpanded { flex: 1; min-width: 0; } diff --git a/packages/client/ui-conversation/src/client/skeleton/InputBar.tsx b/packages/client/ui-conversation/src/client/skeleton/InputBar.tsx index 283f50c456..9349a58b48 100644 --- a/packages/client/ui-conversation/src/client/skeleton/InputBar.tsx +++ b/packages/client/ui-conversation/src/client/skeleton/InputBar.tsx @@ -55,6 +55,8 @@ export const InputBar = memo(function InputBar({ const busyEnter = useBusyEnter(s => s) void useLexicon // hook seat stays bound by the inject compartment; text-ref decoration rides the shell's editor transforms const commandMenuOpen = useMenuLauncher(source => source === 'command') + const [activity, setActivity] = useState(false) + useEffect(() => { setActivity(false) }, [sessionId]) const promptError = useSession(s => s.promptError) ?? null const running = useSession(s => s.running) ?? false const subagent = useSession(s => s.subagent) ?? null @@ -406,7 +408,7 @@ export const InputBar = memo(function InputBar({ showPlaceholder={draft === '' && attachments.length === 0 && !claimActive} />
-
+