Merge pull request #4412 from deepseek-harness/worktree/experimental-voice-input
feat(voice-input): add optional local SenseVoice dictation
This commit is contained in:
commit
2cbf8df484
133 files changed
+7024
-27
No files matched your search
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.md
|
||||
2026-09-16-experimental-voice-input.md: 30c81ab233f27242dcf1fe2ef89bbb86542f9c04
|
||||
2026-09-16-experimental-voice-input.zh.md: 76aa8d6fc4bea32f194b20d05db4bf49af0be313
|
||||
@@ -0,0 +1,35 @@
|
||||
# Agent Note: Experimental voice input with explicit recognition providers
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-09-16-experimental-voice-input.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
Dictation must preserve user control over when a coding task starts. Local recognition brings model files, a native runtime and CPU work; merely installing or enabling a UI contribution must not allocate these resources. Later cloud providers need a selection mechanism that does not silently change where recordings are processed.
|
||||
|
||||
## Decision
|
||||
|
||||
Five public packages under `packages/experimental/` compose one optional, default-disabled voice input bundle. Service Definition, SenseVoice Provider and Remote Consumer are separate because provider deployment and browser transport evolve independently. The browser UI dynamically mounts its generated Remote contribution. Stable API Remotes do not acquire an experimental dependency.
|
||||
|
||||
Provider ids select exact registrations. Resolution captures an instance, and deregistration closes admission, cancels and joins accepted work. There is no provider fallback: a local failure cannot authorize uploading speech to a cloud service. A new cloud provider supplies its own credentials and explicit processing location through the same registry.
|
||||
|
||||
Recordings and transcripts remain transient until the user submits ordinary text. Recognition does not start an agent turn or create an audio Session event. The input facade accepts asynchronous plain text against a captured draft revision, preserving reference chips and undo history. Changed drafts retain the recognized text for explicit insertion instead of replacing later edits. Session changes and plugin disposal invalidate late results and release recording resources. Capture failures notify the current recording activity immediately, independently of resource-close latency. Cancellation, failure and plugin withdrawal join one release promise, so ownership lasts until the AudioContext closes.
|
||||
|
||||
SenseVoice ONNX and Silero VAD run in one managed CPU subprocess. The native sherpa-onnx package includes ONNX Runtime; explicit preparation only downloads and verifies models; subsequent calls reuse the warm model until the configured idle deadline. Every activation verifies cached assets and restores complete caches to standby without starting the worker. Readiness therefore follows disk contents rather than the lifetime of a Provider instance. Recordings wait in the bounded queue during wake-up. The worker checks readiness at execution, so queued work cannot implicitly download resources invalidated by an earlier cache inspection. Inspection uses the same owned task as preparation, so cancellation and disposal join it. Disabling the provider joins its work and process exit while retaining reusable disk assets. Native browser recording and resampling avoid an additional audio conversion executable. The authenticated private loopback protocol carries complete PCM recordings, allowing cancellation to terminate the worker without sharing a long-running inference with another request.
|
||||
|
||||
The Host owns preparation independently of the view that started it. An ordered step list survives view remounts and records completion only as execution advances. Recognition weights, VAD and final verification have separate operations; downloads use pinned byte counts, while other work reports elapsed time. Preparation status and progress stay in bundle details through `plugins.bundle.config`, keyed by package name. The generic `plugins.bundle.activation` slot carries explicit enablement intent and detail navigation; the voice occupant waits for Host cache inspection and offers installation guidance only for unprepared local models. Enabling an already prepared provider does not prompt or download. A single composer activity Slot expands the voice controls while preserving the draft editor and submit action. The context meter sits below the composer; recording and transcription hide the meter along with ordinary toolbar controls until the activity closes. Shared primitives provide buttons, disclosure and status markers; only measured waveform drawing belongs to the voice UI. Recordings start on click and transcribe on Stop, with feedback inside the activity. Persistent preferences remain in plugin settings.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
Always loading the model on plugin activation would make a UI toggle incur substantial startup time and memory even without dictation. Python/FunASR/PyTorch adds interpreter provisioning, dependency installation and disk cost; completed ONNX artifacts and platform-native packages serve CPU dictation. End users never export models. Browser-native speech recognition does not provide a uniform local-processing guarantee. Streaming voice interaction would require partial-result revision, turn arbitration and bidirectional audio semantics that complete-recording dictation does not need.
|
||||
|
||||
## Consequences
|
||||
|
||||
First use needs download time and disk space. Local means the Host machine, including a remote Host. A cancelled active inference loses its warm worker and the next request reloads it. Optional preparation paths support existing offline environments without silently changing defaults.
|
||||
|
||||
Verification covers exact-provider routing, registration teardown, bounded audio parsing, microphone release, stale-draft and Session-switch handling, worker queue/deadline recovery, and verified download publication. Real local inference uses the same managed worker as the product. Browser composition verifies optional enablement and draft insertion; ordinary Session replay verifies that only the submitted text reaches the model.
|
||||
|
||||
Real Node 24 and Electron 44 children complete FP32 and INT8 inference. Electron copies VAD output to avoid external-buffer restrictions. Three runs of the 4.2-second public Chinese sample agree; this short sample is not a complete accuracy evaluation. Desktop microphone requests are restricted to audio from the primary application window, with macOS authorization handled by the operating system. Windows, Linux and signed installers still require platform acceptance.
|
||||
|
||||
INT8 is the default to reduce download and storage cost; `precision: fp32` selects the reference weights. Public English and concatenated bilingual samples show word differences between precisions, so quantization is not treated as accuracy-equivalent. A roughly 101-second repeated-speech sample completes in both runtimes; its observed post-request RSS is not a peak-memory bound.
|
||||
@@ -0,0 +1,35 @@
|
||||
# Agent Note:使用显式识别 Provider 的实验性语音输入
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-09-16-experimental-voice-input.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
听写必须保留用户对编码任务启动时机的控制。本地识别带来模型文件、原生运行时和 CPU 工作;仅安装或启用 UI 贡献不应分配这些资源。之后的云端 Provider 需要明确的选择机制,不能悄悄改变录音的处理位置。
|
||||
|
||||
## 决策
|
||||
|
||||
`packages/experimental/` 下的五个公开包组合成一个默认禁用的可选语音输入 Bundle。服务定义、SenseVoice Provider 和 Remote 消费者分开,因为 Provider 部署与浏览器传输独立演化。浏览器 UI 动态挂载其生成的 Remote 贡献。稳定 API Remotes 不引入实验性依赖。
|
||||
|
||||
Provider id 选择确切的注册。解析捕获实例,注销先拒绝新任务,再取消并等待已接收的任务。不存在 Provider 回退:本地失败不能授权将语音上传至云端服务。新增云端 Provider 通过同一注册表提供自己的凭据配置与显式处理位置。
|
||||
|
||||
录音与转写在用户提交普通文字前都是临时数据。识别不启动智能体轮次,也不创建音频 Session 事件。输入门面按照捕获的草稿版本接收异步纯文本,保留引用芯片与撤销历史。草稿变化时保留识别文字,等待显式插入,而非替换之后的编辑。切换 Session 和释放插件会使迟到结果失效并释放录音资源。采集失败立即通知当前录音活动,不受资源关闭耗时影响。取消、失败和插件撤销共同等待同一个释放 Promise,所有权持续到 AudioContext 关闭。
|
||||
|
||||
SenseVoice ONNX 与 Silero VAD 在一个托管 CPU 子进程中运行。原生 sherpa-onnx 包包含 ONNX Runtime;显式准备只下载并校验模型;后续调用复用预热模型,直到配置的空闲截止时间。每次启用都会校验缓存资源,完整缓存恢复为待唤醒状态,不启动工作进程。就绪状态因此取决于磁盘内容,而非 Provider 实例的存活时间。录音在唤醒期间进入有界队列等待。工作进程在执行时检查就绪状态,因此较早的缓存检查使资源失效后,排队任务也不能隐式下载资源。检查复用准备任务的生命周期,取消与释放都会等待检查结束。禁用 Provider 等待其任务和进程退出,同时保留可复用的磁盘资源。浏览器原生录音与重采样避免增加音频转换可执行文件。带认证的私有环回协议传输完整 PCM 录音,取消时可以终止工作进程,而不与另一请求共享长时间推理。
|
||||
|
||||
Host 独立于发起视图拥有准备任务。有序步骤列表在视图重新挂载后保留,仅随着执行推进记录完成。识别模型、VAD 与最终校验分别准备;下载使用固定字节数,其他工作报告等待时间。准备状态和进度通过以包名为 key 的 `plugins.bundle.config` 显示在详情页。通用 `plugins.bundle.activation` Slot 传递显式启用意图和详情导航;语音贡献等待 Host 缓存检查,只为尚未准备的本地模型提供安装引导。启用已准备的 Provider 不弹窗也不下载。单个输入框活动 Slot 展开语音控件,同时保留草稿编辑器和提交按钮。空闲时上下文用量按钮位于输入框下方;录音和转写期间,用量按钮与普通工具栏控件一起隐藏,关闭活动后恢复。按钮、折叠与状态标记复用共享原语,语音 UI 只负责实测波形绘制。点击开始录音,停止后转写,反馈留在活动栏内。持久化偏好仍在插件设置中。
|
||||
|
||||
## 考虑的替代方案
|
||||
|
||||
在插件启用时加载模型,会让一个 UI 开关在尚未听写时就产生大量启动时间和内存占用。Python/FunASR/PyTorch 增加解释器、依赖安装与磁盘成本;ONNX 成品和平台原生包满足 CPU 听写。用户端不执行模型导出。浏览器原生语音识别不能提供一致的本地处理保证。流式语音交互需要部分结果修订、轮次仲裁与双向音频语义,而完整录音听写无需这些机制。
|
||||
|
||||
## 影响与验证
|
||||
|
||||
首次使用需要下载时间和磁盘空间。本地指 Host 所在机器,包括远程 Host。取消运行中的推理会丢失预热进程,下次请求重新加载。可选准备路径支持已有离线环境,不会悄悄改变默认行为。
|
||||
|
||||
验证覆盖精确 Provider 路由、注册释放、有界音频解析、麦克风释放、过期草稿和 Session 切换处理、工作进程队列与截止时间恢复,以及经过校验的下载发布。真实本地推理使用与产品相同的托管工作进程。浏览器组合验证可选启用与草稿插入;普通 Session 回放验证只有提交的文字进入模型。
|
||||
|
||||
Node 24 与 Electron 44 的真实子进程均完成 FP32 和 INT8 识别。Electron 的 VAD 输出使用复制缓冲区,避免 external buffer 限制。4.2 秒公开中文样本的三次识别结果一致;该短样本不代表完整准确率评测。桌面麦克风请求限制为主应用窗口的音频请求,macOS 权限由系统处理。Windows、Linux 真实机器及签名后的安装包仍需平台验收。
|
||||
|
||||
INT8 以降低下载和存储成本为默认值,`precision: fp32` 选择参考权重。公开英文与拼接的中英双语样本显示两种精度存在用词差异,因此不将量化视为准确率等效。约 101 秒的重复语音样本在两种运行时均完成识别;请求后的 RSS 观测不构成峰值内存上限。
|
||||
@@ -111,6 +111,7 @@ External packages installed for runtime use or distributed inside the prebuilt b
|
||||
| [`resolve.exports`](https://github.com/lukeed/resolve.exports) | MIT |
|
||||
| [`semver`](https://github.com/npm/node-semver) | ISC |
|
||||
| [`sharp`](https://github.com/lovell/sharp) | Apache-2.0 |
|
||||
| [`sherpa-onnx-node`](https://github.com/csukuangfj/sherpa-onnx) | Apache-2.0 |
|
||||
| [`shiki`](https://github.com/shikijs/shiki) | MIT |
|
||||
| [`simple-icons`](https://github.com/simple-icons/simple-icons) | CC0-1.0 |
|
||||
| [`supports-color`](https://github.com/chalk/supports-color) | MIT |
|
||||
|
||||
@@ -106,7 +106,8 @@
|
||||
"@deepseek-ai/dsh-http-proxy": "workspace:^",
|
||||
"@deepseek-ai/dsh-mcp-resources": "workspace:^",
|
||||
"@deepseek-ai/dsh-plugin-manager": "workspace:^",
|
||||
"@deepseek-ai/dsh-atomic-write": "workspace:^"
|
||||
"@deepseek-ai/dsh-atomic-write": "workspace:^",
|
||||
"@deepseek-ai/dsh-experimental-voice-input-bundle": "workspace:^"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@agentclientprotocol/sdk": "1.4.0",
|
||||
|
||||
@@ -61,6 +61,8 @@
|
||||
"@deepseek-ai/dsh-launch-environment": "workspace:^",
|
||||
"@deepseek-ai/dsh-workspace-changes": "workspace:^",
|
||||
"@deepseek-ai/dsh-subprocess-local": "workspace:^",
|
||||
"@deepseek-ai/dsh-experimental-auto-review": "workspace:^"
|
||||
"@deepseek-ai/dsh-experimental-auto-review": "workspace:^",
|
||||
"@deepseek-ai/dsh-experimental-speech-to-text": "workspace:^",
|
||||
"@deepseek-ai/dsh-experimental-voice-input-bundle": "workspace:^"
|
||||
}
|
||||
}
|
||||
@@ -3,12 +3,16 @@
|
||||
- button "刷新"
|
||||
- button "添加插件"
|
||||
- heading "官方" [level=3]
|
||||
- text: "5"
|
||||
- text: "6"
|
||||
- list:
|
||||
- listitem:
|
||||
- button "查看 智能体团队": 智能体团队
|
||||
- text: Beta 启用团队协作、团队工具、成员列表和共享任务看板。
|
||||
- switch "启用 智能体团队"
|
||||
- listitem:
|
||||
- button "查看 语音输入": 语音输入
|
||||
- text: Beta 在本机使用 SenseVoice 转写录音;首次使用需安装依赖
|
||||
- switch "启用 语音输入"
|
||||
- listitem:
|
||||
- button "查看 终端": 终端
|
||||
- text: 限制每条命令最多能跑多久、最多输出多少内容。
|
||||
|
||||
@@ -3,12 +3,16 @@
|
||||
- button "刷新"
|
||||
- button "添加插件"
|
||||
- heading "官方" [level=3]
|
||||
- text: "5"
|
||||
- text: "6"
|
||||
- list:
|
||||
- listitem:
|
||||
- button "查看 智能体团队": 智能体团队
|
||||
- text: Beta 启用团队协作、团队工具、成员列表和共享任务看板。
|
||||
- switch "启用 智能体团队"
|
||||
- listitem:
|
||||
- button "查看 语音输入": 语音输入
|
||||
- text: Beta 在本机使用 SenseVoice 转写录音;首次使用需安装依赖
|
||||
- switch "启用 语音输入"
|
||||
- listitem:
|
||||
- button "查看 终端": 终端
|
||||
- text: 限制每条命令最多能跑多久、最多输出多少内容。
|
||||
|
||||
@@ -3,12 +3,16 @@
|
||||
- button "刷新"
|
||||
- button "添加插件"
|
||||
- heading "官方" [level=3]
|
||||
- text: "5"
|
||||
- text: "6"
|
||||
- list:
|
||||
- listitem:
|
||||
- button "查看 智能体团队": 智能体团队
|
||||
- text: Beta 启用团队协作、团队工具、成员列表和共享任务看板。
|
||||
- switch "启用 智能体团队"
|
||||
- listitem:
|
||||
- button "查看 语音输入": 语音输入
|
||||
- text: Beta 在本机使用 SenseVoice 转写录音;首次使用需安装依赖
|
||||
- switch "启用 语音输入"
|
||||
- listitem:
|
||||
- button "查看 终端": 终端
|
||||
- text: 限制每条命令最多能跑多久、最多输出多少内容。
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
- dialog "Set up voice input before recording":
|
||||
- heading "Set up voice input before recording" [level=2]
|
||||
- button "Cancel"
|
||||
- paragraph: Voice input is enabled. First use requires downloading and preparing local recognition models. Open the plugin details to review disk space, memory and time estimates before starting setup.
|
||||
- button "Later"
|
||||
- button "Go to setup"
|
||||
@@ -87,10 +87,10 @@ describe('web e2e: plugin configuration pages', () => {
|
||||
|
||||
// Every page the shipped web composition exposes: the shell executor, the
|
||||
// agent loop, subagent selection, and the DeepSeek search provider, after
|
||||
// the official Team bundle the installation ships switched off.
|
||||
// the official bundles the installation ships switched off.
|
||||
await panel.getByRole('button', { name: '查看 网页搜索', exact: true }).waitFor({ timeout: 20_000 })
|
||||
const official = panel.locator('[data-plugin-group="official"]')
|
||||
expect(await official.locator('[data-plugin-package]').count()).toBe(1)
|
||||
expect(await official.locator('[data-plugin-package]').count()).toBe(2)
|
||||
expect(await official.locator('[data-plugin-item]').count()).toBe(4)
|
||||
for (const title of ['终端', 'Agent 循环', 'Subagent', '网页搜索']) {
|
||||
expect(await official.getByRole('button', { name: `查看 ${title}`, exact: true }).count()).toBe(1)
|
||||
|
||||
@@ -124,9 +124,10 @@ describe('web e2e: plugin manager', () => {
|
||||
// group, followed by the official plugins that registered their configuration, and its other bundles
|
||||
// stay off the page.
|
||||
expect(await panel.locator('[data-plugin-group="bundles"] [data-plugin-package]').count()).toBe(1)
|
||||
expect(await panel.locator('[data-plugin-group="official"] [data-plugin-package]').count()).toBe(1)
|
||||
expect(await panel.locator('[data-plugin-group="official"] [data-plugin-package]').count()).toBe(2)
|
||||
expect(await panel.locator('[data-plugin-group="official"] [data-plugin-item]').count()).toBe(4)
|
||||
expect(await panel.getByText('Beta', { exact: true }).count()).toBe(1)
|
||||
expect(await panel.getByText('Beta', { exact: true }).count()).toBe(2)
|
||||
expect(await panel.getByRole('switch', { name: '启用 语音输入', exact: true }).getAttribute('aria-checked')).toBe('false')
|
||||
// A bundle that is off still shows the rows its patch declares, without switches.
|
||||
await panel.getByRole('button', { name: '查看 @fixture/bundle' }).click()
|
||||
await panel.locator('[data-plugin-row]', { hasText: 'fixture-row' }).waitFor({ timeout: 10_000 })
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
/** Browser dictation submits only reviewed text through the ordinary recorded Session flow. */
|
||||
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { chromium, type Browser } from 'playwright'
|
||||
import { expect, it, onTestFinished, vi } from 'vitest'
|
||||
import type { SpeechInput, SpeechProviderId } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import type {} from '@deepseek-ai/dsh-experimental-speech-to-text'
|
||||
import {
|
||||
captureStableAria, compareOrRefreshGolden, fixtureUserPrompts, launchWebScaffold, webSnapshotMode, watchConsole,
|
||||
type WebScaffold,
|
||||
} from './scaffold.ts'
|
||||
import { connectFreshWorkspace, newEnglishPage } from './support.ts'
|
||||
|
||||
const fixture = fileURLToPath(new URL('../../../snapshots/web/fresh-round-trip/session.v3.jsonl', import.meta.url))
|
||||
const expected = fileURLToPath(new URL('../../../snapshots/web/voice-input/ui.expected.md', import.meta.url))
|
||||
const interruptedExpected = fileURLToPath(new URL('../../../snapshots/web/voice-input/interrupted.expected.md', import.meta.url))
|
||||
const recordingExpected = fileURLToPath(new URL('../../../snapshots/web/voice-input/recording.expected.md', import.meta.url))
|
||||
const bundle = fileURLToPath(new URL('../../../packages/experimental/voice-input-bundle', import.meta.url))
|
||||
|
||||
it.skipIf(webSnapshotMode() === 'record')('records from cached standby and submits only the reviewed transcript through Session replay', async () => {
|
||||
const scratch = await mkdtemp(join(tmpdir(), 'dsh-voice-browser-'))
|
||||
const resources: { scaffold?: WebScaffold; browser?: Browser } = {}
|
||||
onTestFinished(async () => {
|
||||
try { await resources.browser?.close() } finally {
|
||||
try { await resources.scaffold?.close() } finally { await rm(scratch, { recursive: true, force: true }) }
|
||||
}
|
||||
})
|
||||
const overlay = join(scratch, 'voice.patch.yml')
|
||||
await writeFile(overlay, '- id: speech-to-text-sensevoice\n disabled: true\n')
|
||||
const prompt = fixtureUserPrompts(await readFile(fixture, 'utf8'))[0]!
|
||||
const prefix = 'Use the bash tool to '
|
||||
const scaffold = await launchWebScaffold({ profile: { packages: [{ dir: bundle, enabled: true }] },
|
||||
extraOverlayPath: overlay, replayFixture: fixture, compareReplaySession: 'read-only',
|
||||
})
|
||||
resources.scaffold = scaffold
|
||||
const recognize = vi.fn(async (input: SpeechInput) => {
|
||||
expect(Buffer.from(input.audio).subarray(0, 4).toString()).toBe('RIFF')
|
||||
expect(input.audio.byteLength).toBeGreaterThan(44)
|
||||
return { text: prompt.slice(prefix.length), audioSeconds: 1, inferenceSeconds: 0.1 }
|
||||
})
|
||||
await scaffold.ctx.plugin({ inject: ['speechToText'], apply(ctx) {
|
||||
ctx.effect(() => ctx.speechToText.register({
|
||||
info: { id: 'sensevoice-local' as SpeechProviderId, name: 'Recorded recognizer', location: 'host-local', languages: ['auto'] },
|
||||
preparation: {
|
||||
snapshot: () => ({ phase: 'standby' }), subscribe: () => () => {},
|
||||
prepare: () => { throw new Error('Cached resources must allow recording without preparation') },
|
||||
cancel: async () => {},
|
||||
},
|
||||
transcribe: recognize,
|
||||
}))
|
||||
} })
|
||||
let messages = 0
|
||||
scaffold.ctx.on('session/event', (_session, event) => {
|
||||
if (event.type === 'user/message' && event.data.source.kind === 'user') messages++
|
||||
})
|
||||
const browser = await chromium.launch({ args: ['--use-fake-ui-for-media-stream', '--use-fake-device-for-media-stream'] })
|
||||
resources.browser = browser
|
||||
const page = await newEnglishPage(browser), tripwire = watchConsole(page)
|
||||
await page.addInitScript(() => {
|
||||
const holder = window as Window & { voiceTestRecorder?: MediaRecorder }
|
||||
const NativeRecorder = window.MediaRecorder
|
||||
window.MediaRecorder = class extends NativeRecorder {
|
||||
constructor(stream: MediaStream, options?: MediaRecorderOptions) { super(stream, options); holder.voiceTestRecorder = this }
|
||||
}
|
||||
})
|
||||
await page.goto(scaffold.authenticatedUrl)
|
||||
await connectFreshWorkspace(page, scaffold.workspaceCwd)
|
||||
const input = page.locator('[data-composer-input]'), mic = page.getByRole('button', { name: 'Start recording', exact: true })
|
||||
await mic.waitFor()
|
||||
const micBox = await mic.boundingBox(), modelBox = await page.getByRole('button', { name: /^Select model, current/ }).boundingBox()
|
||||
const sendBox = await page.getByRole('button', { name: 'Send message', exact: true }).boundingBox()
|
||||
expect(micBox!.x).toBeGreaterThan(modelBox!.x)
|
||||
expect(micBox!.x).toBeLessThan(sendBox!.x)
|
||||
await input.fill(prefix)
|
||||
await mic.click()
|
||||
await page.getByRole('button', { name: 'Stop and transcribe', exact: true }).waitFor()
|
||||
await compareOrRefreshGolden(recordingExpected,
|
||||
await captureStableAria(page, '[data-composer-card]', scaffold.workspaceCwd), webSnapshotMode())
|
||||
expect(await page.getByRole('dialog').count()).toBe(0)
|
||||
// Controlled fake microphone audio occupies this recording interval; it is not a readiness wait.
|
||||
await page.waitForTimeout(300)
|
||||
await page.getByRole('button', { name: 'Stop and transcribe', exact: true }).click()
|
||||
await expect.poll(() => input.innerText()).toBe(prompt)
|
||||
expect(recognize).toHaveBeenCalledOnce()
|
||||
expect(messages).toBe(0)
|
||||
await compareOrRefreshGolden(expected, await captureStableAria(page, '[class*="centerCol"]', scaffold.workspaceCwd), webSnapshotMode())
|
||||
const settled = scaffold.whenTurnSettled()
|
||||
await input.press('Enter')
|
||||
await settled
|
||||
await page.getByText('DONE', { exact: true }).waitFor()
|
||||
const meter = page.getByRole('button', { name: /% of context used/ })
|
||||
await meter.waitFor()
|
||||
for (const width of [1280, 420]) {
|
||||
await page.setViewportSize({ width, height: 900 })
|
||||
await expect.poll(async () => {
|
||||
const micRect = await mic.boundingBox(), meterRect = await meter.boundingBox()
|
||||
return micRect !== null && meterRect !== null && meterRect.y >= micRect.y + micRect.height
|
||||
}).toBe(true)
|
||||
await meter.click()
|
||||
await page.getByRole('dialog', { name: 'of context used', exact: true }).waitFor()
|
||||
await page.keyboard.press('Escape')
|
||||
}
|
||||
await mic.hover()
|
||||
await page.getByRole('tooltip', { name: 'Dictate', exact: true }).waitFor()
|
||||
const controlBackground = await page.getByRole('button', { name: 'Add files or run commands', exact: true })
|
||||
.evaluate(element => getComputedStyle(element).backgroundColor)
|
||||
await mic.click()
|
||||
const stop = page.getByRole('button', { name: 'Stop and transcribe', exact: true })
|
||||
await stop.waitFor()
|
||||
for (const width of [1280, 420]) {
|
||||
await page.setViewportSize({ width, height: 900 })
|
||||
const waveformBox = await page.getByRole('img', { name: 'Recording…', exact: true }).boundingBox()
|
||||
const stopBox = await stop.boundingBox(), cancelBox = await page.getByRole('button', { name: 'Cancel', exact: true }).boundingBox()
|
||||
expect(waveformBox!.height).toBeLessThanOrEqual(Math.min(stopBox!.height, cancelBox!.height))
|
||||
}
|
||||
await input.hover()
|
||||
expect(await meter.isVisible()).toBe(false)
|
||||
expect(await stop.evaluate(element => getComputedStyle(element).backgroundColor)).toBe(controlBackground)
|
||||
expect(await page.getByRole('button', { name: 'Cancel', exact: true })
|
||||
.evaluate(element => getComputedStyle(element).backgroundColor)).toBe(controlBackground)
|
||||
await page.keyboard.press('Escape')
|
||||
await mic.waitFor()
|
||||
await meter.waitFor()
|
||||
await meter.click()
|
||||
await page.getByRole('dialog', { name: 'of context used', exact: true }).waitFor()
|
||||
await page.keyboard.press('Escape')
|
||||
await mic.click()
|
||||
await stop.waitFor()
|
||||
await page.evaluate(() => {
|
||||
const recorder = (window as Window & { voiceTestRecorder?: MediaRecorder }).voiceTestRecorder
|
||||
if (!recorder) throw new Error('No active test microphone')
|
||||
recorder.dispatchEvent(new Event('error'))
|
||||
})
|
||||
await page.getByRole('button', { name: 'Record again', exact: true }).waitFor()
|
||||
expect(await stop.count()).toBe(0)
|
||||
await compareOrRefreshGolden(interruptedExpected,
|
||||
await captureStableAria(page, '[data-composer-card]', scaffold.workspaceCwd), webSnapshotMode())
|
||||
expect(recognize).toHaveBeenCalledOnce()
|
||||
await page.getByRole('button', { name: 'Record again', exact: true }).click()
|
||||
await stop.waitFor()
|
||||
await page.keyboard.press('Escape')
|
||||
expect(messages).toBe(1)
|
||||
expect(tripwire.pageErrors).toEqual([])
|
||||
}, 120_000)
|
||||
@@ -0,0 +1,44 @@
|
||||
/** Empty local caches guide explicit enablement to plugin setup without starting installation. */
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { chromium, type Browser } from 'playwright'
|
||||
import { expect, it, onTestFinished } from 'vitest'
|
||||
import { captureStableAria, compareOrRefreshGolden, launchWebScaffold, watchConsole, webSnapshotMode, type WebScaffold } from './scaffold.ts'
|
||||
import { newEnglishPage } from './support.ts'
|
||||
|
||||
const bundle = fileURLToPath(new URL('../../../packages/experimental/voice-input-bundle', import.meta.url))
|
||||
const expected = fileURLToPath(new URL('./expected/voice-setup.expected.md', import.meta.url))
|
||||
|
||||
it('guides a newly enabled voice plugin to installation and lets the user postpone it', async () => {
|
||||
const resources: { scaffold?: WebScaffold; browser?: Browser } = {}
|
||||
onTestFinished(async () => {
|
||||
try { await resources.browser?.close() } finally { await resources.scaffold?.close() }
|
||||
})
|
||||
const scaffold = await launchWebScaffold({ profile: { packages: [{ dir: bundle, enabled: false }] } })
|
||||
resources.scaffold = scaffold
|
||||
const browser = await chromium.launch()
|
||||
resources.browser = browser
|
||||
const page = await newEnglishPage(browser), tripwire = watchConsole(page)
|
||||
await page.goto(scaffold.authenticatedUrl)
|
||||
await page.getByRole('button', { name: 'Plugins', exact: true }).click()
|
||||
const toggle = page.getByRole('switch', { name: 'Enable Voice input', exact: true })
|
||||
await toggle.waitFor()
|
||||
expect(await toggle.getAttribute('aria-checked')).toBe('false')
|
||||
const dialog = page.getByRole('dialog', { name: 'Set up voice input before recording', exact: true })
|
||||
expect(await dialog.count()).toBe(0)
|
||||
await toggle.click()
|
||||
await dialog.waitFor()
|
||||
await compareOrRefreshGolden(expected, await captureStableAria(page, '[role="dialog"]', scaffold.workspaceCwd), webSnapshotMode())
|
||||
await dialog.getByRole('button', { name: 'Later', exact: true }).click()
|
||||
await dialog.waitFor({ state: 'hidden' })
|
||||
expect(await toggle.getAttribute('aria-checked')).toBe('true')
|
||||
const card = page.locator('[data-plugin-package="@deepseek-ai/dsh-experimental-voice-input-bundle"]')
|
||||
expect(await card.getByRole('status').count()).toBe(0)
|
||||
await toggle.click()
|
||||
await expect.poll(() => toggle.getAttribute('aria-checked')).toBe('false')
|
||||
await toggle.click()
|
||||
await dialog.getByRole('button', { name: 'Go to setup', exact: true }).click()
|
||||
await page.getByRole('button', { name: 'Download and prepare', exact: true }).waitFor()
|
||||
expect(await dialog.count()).toBe(0)
|
||||
expect(await page.getByText('Local models will be downloaded to the machine running DSH. No Python or compiler is required.', { exact: true }).count()).toBe(1)
|
||||
expect(tripwire.pageErrors).toEqual([])
|
||||
})
|
||||
@@ -25,6 +25,8 @@
|
||||
"tests/default-product-isolation.e2e.ts",
|
||||
"tests/diff-context.e2e.ts",
|
||||
"tests/scaffold.ts",
|
||||
"tests/voice-input.e2e.ts",
|
||||
"tests/voice-setup.e2e.ts",
|
||||
"tests/prefix-proxy.ts",
|
||||
"tests/auto-review-fixture.ts",
|
||||
"tests/scaffold-generation.spec.ts",
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/capability-seams.md
|
||||
capability-seams.md: eea821874c1932d7daa5f0a2aec3f18653757142
|
||||
capability-seams.zh.md: c7794eae7bfd6456b7878a5b98a27fdec9f2adab
|
||||
capability-seams.md: bbd44bce011404ee476131b9f498618a8ef5c708
|
||||
capability-seams.zh.md: 4e9f826bc27420ba372ac6feb688cc23bc58e141
|
||||
@@ -64,6 +64,8 @@ flowchart LR
|
||||
pkg_subagent_in_process_driver["subagent-in-process-driver"]
|
||||
pkg_invariants["invariants"]
|
||||
pkg_message_feedback["message-feedback"]
|
||||
pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"]
|
||||
svc_speechController["ctx.speechController<br/>Experimental transcription Remote"]
|
||||
svc_sessionController["ctx.sessionController<br/>Host Session Remote controller"]
|
||||
svc_sessionFileReferences["ctx.sessionFileReferences<br/>Session-addressed file-reference Remote adapter"]
|
||||
svc_sessionSkillCatalog["ctx.sessionSkillCatalog<br/>Session-addressed skill Remote adapter"]
|
||||
@@ -215,6 +217,9 @@ flowchart LR
|
||||
pkg_subagent_dsh_sdk["subagent-dsh-sdk"]
|
||||
pkg_tool_subagent_control["tool-subagent-control"]
|
||||
pkg_tool_ralph["tool-ralph"]
|
||||
pkg_experimental_speech_to_text["experimental-speech-to-text"]
|
||||
svc_speechToText["ctx.speechToText<br/>Experimental speech recognition providers"]
|
||||
pkg_experimental_speech_to_text_sensevoice["experimental-speech-to-text-sensevoice"]
|
||||
pkg_experimental_agent_team["experimental-agent-team"]
|
||||
svc_agentTeams["ctx.agentTeams<br/>Agent Teams coordination domain"]
|
||||
pkg_experimental_tool_agent_team["experimental-tool-agent-team"]
|
||||
@@ -293,12 +298,15 @@ flowchart LR
|
||||
pkg_credentials_local --> svc_credentials
|
||||
pkg_deepseek_llm_api_extensions --> svc_deepseekLlmApiExtensions
|
||||
pkg_experimental_agent_team --> svc_agentTeams
|
||||
pkg_experimental_api_speech_to_text --> svc_speechController
|
||||
pkg_experimental_browser_use_chrome_devtools_mcp --> svc_browserUse
|
||||
pkg_experimental_browser_use_playwright_mcp --> svc_browserUse
|
||||
pkg_experimental_browser_use_stagehand_native --> svc_browserUse
|
||||
pkg_experimental_computer_use_cua_driver_mcp --> svc_computerUse
|
||||
pkg_experimental_computer_use_cua_driver_native --> svc_computerUse
|
||||
pkg_experimental_ptc_runtime_python --> svc_ptcRuntime
|
||||
pkg_experimental_speech_to_text --> svc_speechToText
|
||||
pkg_experimental_speech_to_text_sensevoice --> svc_speechToText
|
||||
pkg_file_reference --> svc_fileReferences
|
||||
pkg_file_reference_local --> svc_fileReferences
|
||||
pkg_fs --> svc_fs
|
||||
@@ -490,6 +498,7 @@ flowchart LR
|
||||
svc_shellEnv --> pkg_tool_bash
|
||||
svc_shellEnv --> pkg_tool_pwsh
|
||||
svc_skills --> pkg_tool_skill
|
||||
svc_speechToText --> pkg_experimental_api_speech_to_text
|
||||
svc_spillStore --> pkg_spill_policy
|
||||
svc_ssh --> pkg_fs_ssh
|
||||
svc_ssh --> pkg_sandbox_ssh
|
||||
@@ -557,6 +566,7 @@ flowchart LR
|
||||
| `ctx.tokenMeter` | `core` | [`token-meter`](../packages/llm/token-meter) | - | [`compaction-basic`](../packages/compaction/compaction-basic) | - | Owns isolated per-session replay folds; pressure consumers share immutable revisioned measurements. |
|
||||
| `ctx.toolResultPruner` | `core` | [`compaction-tool-result-pruner`](../packages/compaction/compaction-tool-result-pruner) | - | [`compaction-basic`](../packages/compaction/compaction-basic) | - | Rewrites oversized current tool results through replayable single-node surface replacements before summary compaction. |
|
||||
| `ctx.sessions` | `core` | [`session`](../packages/core/session) | - | [`agent-loop`](../packages/core/agent-loop), [`agent`](../packages/core/agent), [`session-persistence`](../packages/session/session-persistence), [`session-query`](../packages/session-query/session-query), [`session-query-sqlite`](../packages/session-query/session-query-sqlite), [`subagent-in-process-driver`](../packages/subagent/subagent-in-process-driver), [`invariants`](../packages/runtime-diagnostics/invariants), [`message-feedback`](../packages/feedback/message-feedback) | - | Owns append-only Session instances and emits the durable session event feed. |
|
||||
| `ctx.speechController` | `core` | [`experimental-api-speech-to-text`](../packages/experimental/api-speech-to-text) | - | - | - | Validates bounded browser audio before provider dispatch. |
|
||||
| `ctx.sessionController` | `core` | [`api-session-controller`](../packages/api/session-controller) | - | - | - | Owns Session commands, cold reads, durable-event following, live control state, model catalogs, workspace opening, and Agent activation policy. |
|
||||
| `ctx.sessionFileReferences` | `core` | [`api-session-controller`](../packages/api/session-controller) | - | - | - | Delegates file-reference discovery through the Session Controller's established Agent lookup policy. |
|
||||
| `ctx.sessionSkillCatalog` | `core` | [`api-session-controller`](../packages/api/session-controller) | - | - | - | Lists the Session composition's user-invocable skills without activating a cold Agent. |
|
||||
@@ -612,6 +622,7 @@ flowchart LR
|
||||
| `ctx.fs` | `seam` | [`fs`](../packages/fs/fs) | [`fs-local`](../packages/fs/fs-local), [`fs-sandbox`](../packages/fs/fs-sandbox), [`fs-ssh`](../packages/ssh/fs-ssh) | [`tool-fs`](../packages/fs/tool-fs) | [`fs-observation-policy`](../packages/fs/fs-observation-policy) | tool-fs executes read/write/edit through ctx.fs; fs-sandbox fences mutations by the shared sandbox mode; fs-observation-policy contributes observed-state checks through the fs/* event gate. |
|
||||
| `ctx.compaction` | `seam` | [`compaction`](../packages/compaction/compaction) | [`compaction-basic`](../packages/compaction/compaction-basic) | [`compaction-basic`](../packages/compaction/compaction-basic) | - | The basic backend consumes post-step pressure and request-error recovery events; there is no model-facing compact tool. |
|
||||
| `ctx.subagents` | `seam` | [`subagent`](../packages/subagent/subagent) | [`subagent-spawn-in-process`](../packages/subagent/subagent-spawn-in-process), [`subagent-fork-in-process`](../packages/subagent/subagent-fork-in-process), [`subagent-acp`](../packages/subagent/subagent-acp), [`subagent-codex`](../packages/subagent/subagent-codex), [`subagent-claude-code`](../packages/subagent/subagent-claude-code), [`subagent-dsh-sdk`](../packages/subagent/subagent-dsh-sdk) | [`tool-subagent`](../packages/subagent/tool-subagent), [`tool-subagent-control`](../packages/subagent/tool-subagent-control), [`tool-ralph`](../packages/workflow/tool-ralph) | - | Providers implement transports; the service also owns optional Activation-based continuation orchestration, tool-subagent selects one-shot or continuable delegation, tool-subagent-control delivers follow-ups, and tool-ralph requires one fresh structured-output route. |
|
||||
| `ctx.speechToText` | `seam` | [`experimental-speech-to-text`](../packages/experimental/speech-to-text) | [`experimental-speech-to-text-sensevoice`](../packages/experimental/speech-to-text-sensevoice) | [`experimental-api-speech-to-text`](../packages/experimental/api-speech-to-text) | - | Routes explicit recognizers; the browser uses the authenticated Remote and keeps transcripts in the draft until submission. |
|
||||
| `ctx.agentTeams` | `core` | [`experimental-agent-team`](../packages/experimental/agent-team) | - | [`experimental-tool-agent-team`](../packages/experimental/tool-agent-team), [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | - | Owns the implicit-root roster, durable peer mailbox, shared task DAG, continuable-child lifecycle, and generated Team Remote methods; tool-agent-team contributes model controls and client-ui-agent-team mounts the browser contribution. |
|
||||
| `ctx.inspector` | `core` | `inspector` | - | - | - | Owns the Worker-hosted CDP target and the transport-independent Host and Client observation and Cordis-tree query API. |
|
||||
| `ctx.jobs` | `seam` | [`jobs`](../packages/jobs/jobs) | [`jobs-local`](../packages/jobs/jobs-local) | [`tool-bash`](../packages/shell/tool-bash), [`tool-pwsh`](../packages/shell/tool-pwsh), [`tool-terminal`](../packages/terminal/tool-terminal), [`tool-subagent`](../packages/subagent/tool-subagent), [`tool-jobs`](../packages/jobs/tool-jobs), [`api-job-controller`](../packages/api/job-controller) | - | Producers (background bash/pwsh, PTY sends, and subagent delegations) register running work; record-declaring jobs additionally stream raw output for non-consuming observers; tool-jobs is the model-facing controller that reads, lists, and kills it; jobs-local is the process-local registry. |
|
||||
|
||||
@@ -66,6 +66,8 @@ flowchart LR
|
||||
pkg_subagent_in_process_driver["subagent-in-process-driver"]
|
||||
pkg_invariants["invariants"]
|
||||
pkg_message_feedback["message-feedback"]
|
||||
pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"]
|
||||
svc_speechController["ctx.speechController<br/>Experimental transcription Remote"]
|
||||
svc_sessionController["ctx.sessionController<br/>Host Session Remote controller"]
|
||||
svc_sessionFileReferences["ctx.sessionFileReferences<br/>Session-addressed file-reference Remote adapter"]
|
||||
svc_sessionSkillCatalog["ctx.sessionSkillCatalog<br/>Session-addressed skill Remote adapter"]
|
||||
@@ -217,6 +219,9 @@ flowchart LR
|
||||
pkg_subagent_dsh_sdk["subagent-dsh-sdk"]
|
||||
pkg_tool_subagent_control["tool-subagent-control"]
|
||||
pkg_tool_ralph["tool-ralph"]
|
||||
pkg_experimental_speech_to_text["experimental-speech-to-text"]
|
||||
svc_speechToText["ctx.speechToText<br/>Experimental speech recognition providers"]
|
||||
pkg_experimental_speech_to_text_sensevoice["experimental-speech-to-text-sensevoice"]
|
||||
pkg_experimental_agent_team["experimental-agent-team"]
|
||||
svc_agentTeams["ctx.agentTeams<br/>Agent Teams coordination domain"]
|
||||
pkg_experimental_tool_agent_team["experimental-tool-agent-team"]
|
||||
@@ -295,12 +300,15 @@ flowchart LR
|
||||
pkg_credentials_local --> svc_credentials
|
||||
pkg_deepseek_llm_api_extensions --> svc_deepseekLlmApiExtensions
|
||||
pkg_experimental_agent_team --> svc_agentTeams
|
||||
pkg_experimental_api_speech_to_text --> svc_speechController
|
||||
pkg_experimental_browser_use_chrome_devtools_mcp --> svc_browserUse
|
||||
pkg_experimental_browser_use_playwright_mcp --> svc_browserUse
|
||||
pkg_experimental_browser_use_stagehand_native --> svc_browserUse
|
||||
pkg_experimental_computer_use_cua_driver_mcp --> svc_computerUse
|
||||
pkg_experimental_computer_use_cua_driver_native --> svc_computerUse
|
||||
pkg_experimental_ptc_runtime_python --> svc_ptcRuntime
|
||||
pkg_experimental_speech_to_text --> svc_speechToText
|
||||
pkg_experimental_speech_to_text_sensevoice --> svc_speechToText
|
||||
pkg_file_reference --> svc_fileReferences
|
||||
pkg_file_reference_local --> svc_fileReferences
|
||||
pkg_fs --> svc_fs
|
||||
@@ -492,6 +500,7 @@ flowchart LR
|
||||
svc_shellEnv --> pkg_tool_bash
|
||||
svc_shellEnv --> pkg_tool_pwsh
|
||||
svc_skills --> pkg_tool_skill
|
||||
svc_speechToText --> pkg_experimental_api_speech_to_text
|
||||
svc_spillStore --> pkg_spill_policy
|
||||
svc_ssh --> pkg_fs_ssh
|
||||
svc_ssh --> pkg_sandbox_ssh
|
||||
@@ -559,6 +568,7 @@ flowchart LR
|
||||
| `ctx.tokenMeter` | `core` | [`token-meter`](../packages/llm/token-meter) | - | [`compaction-basic`](../packages/compaction/compaction-basic) | - | 拥有按会话隔离的回放折叠区;压力消费方共享不可变且带修订版本的测量结果。 |
|
||||
| `ctx.toolResultPruner` | `core` | [`compaction-tool-result-pruner`](../packages/compaction/compaction-tool-result-pruner) | - | [`compaction-basic`](../packages/compaction/compaction-basic) | - | 在摘要压缩前,通过可回放的单节点表层替换来改写过大的当前工具结果。 |
|
||||
| `ctx.sessions` | `core` | [`session`](../packages/core/session) | - | [`agent-loop`](../packages/core/agent-loop), [`agent`](../packages/core/agent), [`session-persistence`](../packages/session/session-persistence), [`session-query`](../packages/session-query/session-query), [`session-query-sqlite`](../packages/session-query/session-query-sqlite), [`subagent-in-process-driver`](../packages/subagent/subagent-in-process-driver), [`invariants`](../packages/runtime-diagnostics/invariants), [`message-feedback`](../packages/feedback/message-feedback) | - | 拥有仅追加的 Session 实例,并发出持久的会话事件流。 |
|
||||
| `ctx.speechController` | `core` | [`experimental-api-speech-to-text`](../packages/experimental/api-speech-to-text) | - | - | - | 在 Provider 调用前校验受限的浏览器音频。 |
|
||||
| `ctx.sessionController` | `core` | [`api-session-controller`](../packages/api/session-controller) | - | - | - | 负责 Session 命令、冷读取、持久事件跟随、实时控制状态、模型目录、workspace 打开与 Agent 激活策略。 |
|
||||
| `ctx.sessionFileReferences` | `core` | [`api-session-controller`](../packages/api/session-controller) | - | - | - | 通过 Session Controller 的既有 Agent lookup 策略委托文件引用发现。 |
|
||||
| `ctx.sessionSkillCatalog` | `core` | [`api-session-controller`](../packages/api/session-controller) | - | - | - | 在不激活冷 Agent 的前提下列出 Session 组合中允许用户调用的 skill。 |
|
||||
@@ -614,6 +624,7 @@ flowchart LR
|
||||
| `ctx.fs` | `seam` | [`fs`](../packages/fs/fs) | [`fs-local`](../packages/fs/fs-local), [`fs-sandbox`](../packages/fs/fs-sandbox), [`fs-ssh`](../packages/ssh/fs-ssh) | [`tool-fs`](../packages/fs/tool-fs) | [`fs-observation-policy`](../packages/fs/fs-observation-policy) | tool-fs 通过 ctx.fs 执行读取/写入/编辑;fs-sandbox 按共享沙箱模式限制变更;fs-observation-policy 通过 fs/* 事件门禁贡献基于观测状态的检查。 |
|
||||
| `ctx.compaction` | `seam` | [`compaction`](../packages/compaction/compaction) | [`compaction-basic`](../packages/compaction/compaction-basic) | [`compaction-basic`](../packages/compaction/compaction-basic) | - | 基础后端消费步骤后的压力事件和请求错误恢复事件;不存在面向模型的压缩工具。 |
|
||||
| `ctx.subagents` | `seam` | [`subagent`](../packages/subagent/subagent) | [`subagent-spawn-in-process`](../packages/subagent/subagent-spawn-in-process), [`subagent-fork-in-process`](../packages/subagent/subagent-fork-in-process), [`subagent-acp`](../packages/subagent/subagent-acp), [`subagent-codex`](../packages/subagent/subagent-codex), [`subagent-claude-code`](../packages/subagent/subagent-claude-code), [`subagent-dsh-sdk`](../packages/subagent/subagent-dsh-sdk) | [`tool-subagent`](../packages/subagent/tool-subagent), [`tool-subagent-control`](../packages/subagent/tool-subagent-control), [`tool-ralph`](../packages/workflow/tool-ralph) | - | 提供方实现传输;该服务还负责可选的、基于 Activation 的延续编排,tool-subagent 选择一次性或可延续委派,tool-subagent-control 传递后续消息,而 tool-ralph 要求一条全新的结构化输出路由。 |
|
||||
| `ctx.speechToText` | `seam` | [`experimental-speech-to-text`](../packages/experimental/speech-to-text) | [`experimental-speech-to-text-sensevoice`](../packages/experimental/speech-to-text-sensevoice) | [`experimental-api-speech-to-text`](../packages/experimental/api-speech-to-text) | - | 路由显式选择的识别器;浏览器使用带认证的 Remote,并在提交前将转写保留在草稿中。 |
|
||||
| `ctx.agentTeams` | `core` | [`experimental-agent-team`](../packages/experimental/agent-team) | - | [`experimental-tool-agent-team`](../packages/experimental/tool-agent-team), [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | - | 负责隐式 Root roster、持久 peer mailbox、共享任务 DAG、continuable child 生命周期与生成式 Team Remote method;tool-agent-team 提供模型控制工具,client-ui-agent-team 挂载浏览器 contribution。 |
|
||||
| `ctx.inspector` | `core` | `inspector` | - | - | - | 负责 Worker 托管的 CDP target,以及独立于传输的 Host 和 Client observation 与 Cordis tree query API。 |
|
||||
| `ctx.jobs` | `seam` | [`jobs`](../packages/jobs/jobs) | [`jobs-local`](../packages/jobs/jobs-local) | [`tool-bash`](../packages/shell/tool-bash), [`tool-pwsh`](../packages/shell/tool-pwsh), [`tool-terminal`](../packages/terminal/tool-terminal), [`tool-subagent`](../packages/subagent/tool-subagent), [`tool-jobs`](../packages/jobs/tool-jobs), [`api-job-controller`](../packages/api/job-controller) | - | 生产方(后台 bash/pwsh、PTY 发送和 subagent 委派)登记正在运行的工作;声明 record 的 job 还为非消费观察者流式提供原始输出;tool-jobs 是面向模型的控制器,用于读取、列出和终止这些工作;jobs-local 是进程本地注册表。 |
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/config-catalog.md
|
||||
config-catalog.md: 59b15e7bbb518abdee55308c191b109547b5185e
|
||||
config-catalog.zh.md: 4e89ea92944f6e1f2ae8ae0b57f0bec9e9f835d9
|
||||
config-catalog.md: 65c8de43fcb36dcab56c04350ef11d2364905796
|
||||
config-catalog.zh.md: 19f62adeece3983ac67895b124c370b2aa2eb7dc
|
||||
@@ -635,6 +635,24 @@ export interface Config {
|
||||
|
||||
Source: [`packages/experimental/agent-team/src/types.ts:130`](../packages/experimental/agent-team/src/types.ts)
|
||||
|
||||
<a id="deepseek-aidsh-experimental-api-speech-to-text"></a>
|
||||
|
||||
## `@deepseek-ai/dsh-experimental-api-speech-to-text`
|
||||
|
||||
Requires: `speechToText` · `typert`
|
||||
|
||||
```ts config-catalog
|
||||
/** Limits applied before decoding or calling a provider. */
|
||||
export interface Config {
|
||||
/** Maximum decoded WAV bytes per request. */
|
||||
maxAudioBytes: number
|
||||
/** Maximum PCM recording duration in seconds. */
|
||||
maxDurationSeconds: number
|
||||
}
|
||||
```
|
||||
|
||||
Source: [`packages/experimental/api-speech-to-text/src/index.ts:20`](../packages/experimental/api-speech-to-text/src/index.ts)
|
||||
|
||||
<a id="deepseek-aidsh-experimental-browser-use-chrome-devtools-mcp"></a>
|
||||
|
||||
## `@deepseek-ai/dsh-experimental-browser-use-chrome-devtools-mcp`
|
||||
@@ -865,6 +883,76 @@ export interface Config {
|
||||
|
||||
Source: [`packages/experimental/ptc-runtime-python/src/index.ts:42`](../packages/experimental/ptc-runtime-python/src/index.ts)
|
||||
|
||||
<a id="deepseek-aidsh-experimental-speech-to-text"></a>
|
||||
|
||||
## `@deepseek-ai/dsh-experimental-speech-to-text`
|
||||
|
||||
```ts config-catalog
|
||||
/** Composition defaults resolved before a transcription starts. */
|
||||
export interface Config {
|
||||
/** Registered provider selected when the caller omits an id. */
|
||||
defaultProvider: string
|
||||
/** Provider language hint selected when the caller omits one. */
|
||||
language: string
|
||||
}
|
||||
```
|
||||
|
||||
Source: [`packages/experimental/speech-to-text/src/index.ts:18`](../packages/experimental/speech-to-text/src/index.ts)
|
||||
|
||||
<a id="deepseek-aidsh-experimental-speech-to-text-sensevoice"></a>
|
||||
|
||||
## `@deepseek-ai/dsh-experimental-speech-to-text-sensevoice`
|
||||
|
||||
Requires: `speechToText` · `subprocess`
|
||||
|
||||
```ts config-catalog
|
||||
/** Local runtime, inference, and retention settings. */
|
||||
export interface Config {
|
||||
/** Unique registration id; consumers select this exact id. */
|
||||
providerId: string
|
||||
/** Absolute directory for verified ONNX models. */
|
||||
dataRoot: string
|
||||
/** Existing directory containing the selected ONNX model and tokens.txt; omission downloads verified files. */
|
||||
modelDirectory?: string | undefined
|
||||
/** Existing Silero VAD ONNX file; omission downloads the verified model. */
|
||||
vadModelPath?: string | undefined
|
||||
/** Weight precision; INT8 minimizes first-use download and model storage. */
|
||||
precision: 'int8' | 'fp32'
|
||||
/** Hugging Face-compatible origin for pinned model URLs, including private mirrors. */
|
||||
modelOrigin: string
|
||||
/** CPU intra-operation thread count. */
|
||||
threads: number
|
||||
/** Maximum speech segment length passed to the recognizer. */
|
||||
segmentSeconds: number
|
||||
/** Silero speech probability threshold. */
|
||||
vadThreshold: number
|
||||
/** Minimum speech duration retained by VAD. */
|
||||
minSpeechSeconds: number
|
||||
/** Silence separating two speech segments. */
|
||||
minSilenceSeconds: number
|
||||
/** Maximum decoded WAV bytes accepted by the private worker. */
|
||||
maxAudioBytes: number
|
||||
/** Deadline for runtime preparation and cold model loading. */
|
||||
prepareTimeoutMs: number
|
||||
/** Deadline for one inference after the worker is ready. */
|
||||
inferenceTimeoutMs: number
|
||||
/** Idle period before stopping the worker; zero keeps it warm. */
|
||||
idleTimeoutMs: number
|
||||
/** Maximum accepted running and waiting transcriptions. */
|
||||
maxPending: number
|
||||
/** Managed process termination grace period. */
|
||||
graceMs: number
|
||||
/** Maximum retained worker diagnostic bytes. */
|
||||
maxLogBytes: number
|
||||
/** Maximum transcript response bytes. */
|
||||
maxResponseBytes: number
|
||||
/** Minimum interval between intermediate download progress notifications. */
|
||||
progressIntervalMs: number
|
||||
}
|
||||
```
|
||||
|
||||
Source: [`packages/experimental/speech-to-text-sensevoice/src/config.ts:6`](../packages/experimental/speech-to-text-sensevoice/src/config.ts)
|
||||
|
||||
<a id="deepseek-aidsh-experimental-tool-agent-team"></a>
|
||||
|
||||
## `@deepseek-ai/dsh-experimental-tool-agent-team`
|
||||
@@ -3907,6 +3995,7 @@ These load from a `cordis.yml` entry with no `config:` block; they declare no co
|
||||
- `@deepseek-ai/dsh-deepseek-llm-api-extensions` ([`packages/llm/deepseek-llm-api-extensions/src/index.ts`](../packages/llm/deepseek-llm-api-extensions/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-auto-review` — requires `llm` · `permissionPresets` · `sessions` · `tools` ([`packages/experimental/auto-review/src/index.ts`](../packages/experimental/auto-review/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-client-ui-agent-team` ([`packages/experimental/client-ui-agent-team/src/index.ts`](../packages/experimental/client-ui-agent-team/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-client-ui-voice-input` ([`packages/experimental/client-ui-voice-input/src/index.ts`](../packages/experimental/client-ui-voice-input/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-computer-use-cua-driver-native` — requires `computerUse` · `tools` · `systemPrompt` ([`packages/experimental/computer-use-cua-driver-native/src/index.ts`](../packages/experimental/computer-use-cua-driver-native/src/index.ts))
|
||||
- `@deepseek-ai/dsh-fs-observation-policy` ([`packages/fs/fs-observation-policy/src/index.ts`](../packages/fs/fs-observation-policy/src/index.ts))
|
||||
- `@deepseek-ai/dsh-fs-ssh` — requires `ssh` · `sandboxPolicy` ([`packages/ssh/fs-ssh/src/index.ts`](../packages/ssh/fs-ssh/src/index.ts))
|
||||
@@ -3979,6 +4068,7 @@ Imported as libraries by other packages; a `cordis.yml` cannot load them.
|
||||
- `@deepseek-ai/dsh-deque` ([`packages/util/deque/src/index.ts`](../packages/util/deque/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-agent-team-profile` ([`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-browser-use-runtime` ([`packages/experimental/browser-use-runtime/src/index.ts`](../packages/experimental/browser-use-runtime/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-voice-input-bundle` ([`packages/experimental/voice-input-bundle/src/index.ts`](../packages/experimental/voice-input-bundle/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-webworker-packer` ([`packages/experimental/webworker-packer/src/index.ts`](../packages/experimental/webworker-packer/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-webworker-runtime` ([`packages/experimental/webworker-runtime/src/index.ts`](../packages/experimental/webworker-runtime/src/index.ts))
|
||||
- `@deepseek-ai/dsh-home-paths` ([`packages/util/home-paths/src/index.ts`](../packages/util/home-paths/src/index.ts))
|
||||
|
||||
@@ -639,6 +639,24 @@ export interface Config {
|
||||
|
||||
来源:[`packages/experimental/agent-team/src/types.ts:130`](../packages/experimental/agent-team/src/types.ts)
|
||||
|
||||
<a id="deepseek-aidsh-experimental-api-speech-to-text"></a>
|
||||
|
||||
## `@deepseek-ai/dsh-experimental-api-speech-to-text`
|
||||
|
||||
需要: `speechToText` · `typert`
|
||||
|
||||
```ts config-catalog
|
||||
/** Limits applied before decoding or calling a provider. */
|
||||
export interface Config {
|
||||
/** Maximum decoded WAV bytes per request. */
|
||||
maxAudioBytes: number
|
||||
/** Maximum PCM recording duration in seconds. */
|
||||
maxDurationSeconds: number
|
||||
}
|
||||
```
|
||||
|
||||
来源: [`packages/experimental/api-speech-to-text/src/index.ts:20`](../packages/experimental/api-speech-to-text/src/index.ts)
|
||||
|
||||
<a id="deepseek-aidsh-experimental-browser-use-chrome-devtools-mcp"></a>
|
||||
|
||||
## `@deepseek-ai/dsh-experimental-browser-use-chrome-devtools-mcp`
|
||||
@@ -869,6 +887,76 @@ export interface Config {
|
||||
|
||||
来源:[`packages/experimental/ptc-runtime-python/src/index.ts:42`](../packages/experimental/ptc-runtime-python/src/index.ts)
|
||||
|
||||
<a id="deepseek-aidsh-experimental-speech-to-text"></a>
|
||||
|
||||
## `@deepseek-ai/dsh-experimental-speech-to-text`
|
||||
|
||||
```ts config-catalog
|
||||
/** Composition defaults resolved before a transcription starts. */
|
||||
export interface Config {
|
||||
/** Registered provider selected when the caller omits an id. */
|
||||
defaultProvider: string
|
||||
/** Provider language hint selected when the caller omits one. */
|
||||
language: string
|
||||
}
|
||||
```
|
||||
|
||||
来源: [`packages/experimental/speech-to-text/src/index.ts:18`](../packages/experimental/speech-to-text/src/index.ts)
|
||||
|
||||
<a id="deepseek-aidsh-experimental-speech-to-text-sensevoice"></a>
|
||||
|
||||
## `@deepseek-ai/dsh-experimental-speech-to-text-sensevoice`
|
||||
|
||||
需要: `speechToText` · `subprocess`
|
||||
|
||||
```ts config-catalog
|
||||
/** Local runtime, inference, and retention settings. */
|
||||
export interface Config {
|
||||
/** Unique registration id; consumers select this exact id. */
|
||||
providerId: string
|
||||
/** Absolute directory for verified ONNX models. */
|
||||
dataRoot: string
|
||||
/** Existing directory containing the selected ONNX model and tokens.txt; omission downloads verified files. */
|
||||
modelDirectory?: string | undefined
|
||||
/** Existing Silero VAD ONNX file; omission downloads the verified model. */
|
||||
vadModelPath?: string | undefined
|
||||
/** Weight precision; INT8 minimizes first-use download and model storage. */
|
||||
precision: 'int8' | 'fp32'
|
||||
/** Hugging Face-compatible origin for pinned model URLs, including private mirrors. */
|
||||
modelOrigin: string
|
||||
/** CPU intra-operation thread count. */
|
||||
threads: number
|
||||
/** Maximum speech segment length passed to the recognizer. */
|
||||
segmentSeconds: number
|
||||
/** Silero speech probability threshold. */
|
||||
vadThreshold: number
|
||||
/** Minimum speech duration retained by VAD. */
|
||||
minSpeechSeconds: number
|
||||
/** Silence separating two speech segments. */
|
||||
minSilenceSeconds: number
|
||||
/** Maximum decoded WAV bytes accepted by the private worker. */
|
||||
maxAudioBytes: number
|
||||
/** Deadline for runtime preparation and cold model loading. */
|
||||
prepareTimeoutMs: number
|
||||
/** Deadline for one inference after the worker is ready. */
|
||||
inferenceTimeoutMs: number
|
||||
/** Idle period before stopping the worker; zero keeps it warm. */
|
||||
idleTimeoutMs: number
|
||||
/** Maximum accepted running and waiting transcriptions. */
|
||||
maxPending: number
|
||||
/** Managed process termination grace period. */
|
||||
graceMs: number
|
||||
/** Maximum retained worker diagnostic bytes. */
|
||||
maxLogBytes: number
|
||||
/** Maximum transcript response bytes. */
|
||||
maxResponseBytes: number
|
||||
/** Minimum interval between intermediate download progress notifications. */
|
||||
progressIntervalMs: number
|
||||
}
|
||||
```
|
||||
|
||||
来源: [`packages/experimental/speech-to-text-sensevoice/src/config.ts:6`](../packages/experimental/speech-to-text-sensevoice/src/config.ts)
|
||||
|
||||
<a id="deepseek-aidsh-experimental-tool-agent-team"></a>
|
||||
|
||||
## `@deepseek-ai/dsh-experimental-tool-agent-team`
|
||||
@@ -3911,6 +3999,7 @@ export interface Config {
|
||||
- `@deepseek-ai/dsh-deepseek-llm-api-extensions`([`packages/llm/deepseek-llm-api-extensions/src/index.ts`](../packages/llm/deepseek-llm-api-extensions/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-auto-review` — 需要 `llm` · `permissionPresets` · `sessions` · `tools`([`packages/experimental/auto-review/src/index.ts`](../packages/experimental/auto-review/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-client-ui-agent-team`([`packages/experimental/client-ui-agent-team/src/index.ts`](../packages/experimental/client-ui-agent-team/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-client-ui-voice-input` ([`packages/experimental/client-ui-voice-input/src/index.ts`](../packages/experimental/client-ui-voice-input/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-computer-use-cua-driver-native` — requires `computerUse` · `tools` · `systemPrompt` ([`packages/experimental/computer-use-cua-driver-native/src/index.ts`](../packages/experimental/computer-use-cua-driver-native/src/index.ts))
|
||||
- `@deepseek-ai/dsh-fs-observation-policy`([`packages/fs/fs-observation-policy/src/index.ts`](../packages/fs/fs-observation-policy/src/index.ts))
|
||||
- `@deepseek-ai/dsh-fs-ssh` — 需要 `ssh` · `sandboxPolicy`([`packages/ssh/fs-ssh/src/index.ts`](../packages/ssh/fs-ssh/src/index.ts))
|
||||
@@ -3982,6 +4071,7 @@ export interface Config {
|
||||
- `@deepseek-ai/dsh-deque`([`packages/util/deque/src/index.ts`](../packages/util/deque/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-agent-team-profile`([`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-browser-use-runtime` ([`packages/experimental/browser-use-runtime/src/index.ts`](../packages/experimental/browser-use-runtime/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-voice-input-bundle`([`packages/experimental/voice-input-bundle/src/index.ts`](../packages/experimental/voice-input-bundle/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-webworker-packer`([`packages/experimental/webworker-packer/src/index.ts`](../packages/experimental/webworker-packer/src/index.ts))
|
||||
- `@deepseek-ai/dsh-experimental-webworker-runtime`([`packages/experimental/webworker-runtime/src/index.ts`](../packages/experimental/webworker-runtime/src/index.ts))
|
||||
- `@deepseek-ai/dsh-home-paths`([`packages/util/home-paths/src/index.ts`](../packages/util/home-paths/src/index.ts))
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/module-graph.md
|
||||
module-graph.md: af297ec0b0d143ad8917eac00e2faf44f3477b5d
|
||||
module-graph.zh.md: f56bdc422d9e74d922cb015c9d35e020cbd69b89
|
||||
module-graph.md: 8511f674c84bcfa11fb9e2b137b12025ec540c3e
|
||||
module-graph.zh.md: 67ba8d7f6b2582200b8a2ecc2215f7b317880081
|
||||
@@ -232,17 +232,22 @@ flowchart TD
|
||||
subgraph group_experimental["packages/experimental"]
|
||||
pkg_experimental_agent_team["experimental-agent-team"]
|
||||
pkg_experimental_agent_team_profile["experimental-agent-team-profile"]
|
||||
pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"]
|
||||
pkg_experimental_auto_review["experimental-auto-review"]
|
||||
pkg_experimental_browser_use_chrome_devtools_mcp["experimental-browser-use-chrome-devtools-mcp"]
|
||||
pkg_experimental_browser_use_playwright_mcp["experimental-browser-use-playwright-mcp"]
|
||||
pkg_experimental_browser_use_runtime["experimental-browser-use-runtime"]
|
||||
pkg_experimental_browser_use_stagehand_native["experimental-browser-use-stagehand-native"]
|
||||
pkg_experimental_client_ui_agent_team["experimental-client-ui-agent-team"]
|
||||
pkg_experimental_client_ui_voice_input["experimental-client-ui-voice-input"]
|
||||
pkg_experimental_computer_use_cua_driver_mcp["experimental-computer-use-cua-driver-mcp"]
|
||||
pkg_experimental_computer_use_cua_driver_native["experimental-computer-use-cua-driver-native"]
|
||||
pkg_experimental_inspector["experimental-inspector"]
|
||||
pkg_experimental_ptc_runtime_python["experimental-ptc-runtime-python"]
|
||||
pkg_experimental_speech_to_text["experimental-speech-to-text"]
|
||||
pkg_experimental_speech_to_text_sensevoice["experimental-speech-to-text-sensevoice"]
|
||||
pkg_experimental_tool_agent_team["experimental-tool-agent-team"]
|
||||
pkg_experimental_voice_input_bundle["experimental-voice-input-bundle"]
|
||||
pkg_experimental_webworker_packer["experimental-webworker-packer"]
|
||||
pkg_experimental_webworker_runtime["experimental-webworker-runtime"]
|
||||
end
|
||||
@@ -508,6 +513,7 @@ flowchart TD
|
||||
pkg_session_log_export --> pkg_session_persistence
|
||||
pkg_hmr --> pkg_app_boot
|
||||
pkg_hmr --> pkg_cmdline
|
||||
pkg_experimental_speech_to_text --> pkg_settings
|
||||
pkg_ptc_runtime --> pkg_sandbox
|
||||
pkg_sandbox_local --> pkg_llm
|
||||
pkg_sandbox_local --> pkg_sandbox
|
||||
@@ -603,9 +609,13 @@ flowchart TD
|
||||
pkg_tmux_context --> pkg_session
|
||||
pkg_tmux_context --> pkg_session_projection
|
||||
pkg_tmux_context --> pkg_shell
|
||||
pkg_experimental_api_speech_to_text --> pkg_experimental_speech_to_text
|
||||
pkg_experimental_api_speech_to_text --> pkg_typert_protocol
|
||||
pkg_experimental_ptc_runtime_python --> pkg_ptc_runtime
|
||||
pkg_experimental_ptc_runtime_python --> pkg_timeout
|
||||
pkg_experimental_ptc_runtime_python --> pkg_util_values
|
||||
pkg_experimental_speech_to_text_sensevoice --> pkg_experimental_speech_to_text
|
||||
pkg_experimental_speech_to_text_sensevoice --> pkg_subprocess
|
||||
pkg_commands --> pkg_agent
|
||||
pkg_commands --> pkg_attachment
|
||||
pkg_commands --> pkg_brand
|
||||
@@ -1261,6 +1271,21 @@ flowchart TD
|
||||
pkg_experimental_client_ui_agent_team --> pkg_experimental_agent_team
|
||||
pkg_experimental_client_ui_agent_team --> pkg_session
|
||||
pkg_experimental_client_ui_agent_team --> pkg_typert_protocol
|
||||
pkg_experimental_client_ui_voice_input --> pkg_api_gateway
|
||||
pkg_experimental_client_ui_voice_input --> pkg_api_remotes
|
||||
pkg_experimental_client_ui_voice_input --> pkg_api_session_controller
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_locale
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_store
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_ui_conversation
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_ui_plugin_manager
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_ui_primitives
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_ui_renderer
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_ui_session
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_ui_slots
|
||||
pkg_experimental_client_ui_voice_input --> pkg_experimental_api_speech_to_text
|
||||
pkg_experimental_client_ui_voice_input --> pkg_experimental_speech_to_text
|
||||
pkg_experimental_client_ui_voice_input --> pkg_session
|
||||
pkg_experimental_client_ui_voice_input --> pkg_typert_protocol
|
||||
pkg_experimental_tool_agent_team --> pkg_agent
|
||||
pkg_experimental_tool_agent_team --> pkg_experimental_agent_team
|
||||
pkg_experimental_tool_agent_team --> pkg_session
|
||||
@@ -1385,6 +1410,7 @@ flowchart TD
|
||||
| [`client-web`](../packages/client/web) | `client` | — |
|
||||
| [`office-to-pdf`](../packages/document/office-to-pdf) | `document` | — |
|
||||
| [`experimental-agent-team-profile`](../packages/experimental/agent-team-profile) | `experimental` | — |
|
||||
| [`experimental-voice-input-bundle`](../packages/experimental/voice-input-bundle) | `experimental` | — |
|
||||
| [`experimental-webworker-packer`](../packages/experimental/webworker-packer) | `experimental` | — |
|
||||
| [`client-ui-cordis`](../packages/extensions/ui-cordis) | `extensions` | — |
|
||||
| [`cordis-client-runner`](../packages/extensions/cordis-client-runner) | `extensions` | — |
|
||||
@@ -1457,6 +1483,7 @@ flowchart TD
|
||||
| [`spill-local`](../packages/spill/spill-local) | `spill` | [`spill`](../packages/spill/spill) |
|
||||
| [`session-log-export`](../packages/session-query/session-log-export) | `session-query` | [`session`](../packages/core/session), [`session-persistence`](../packages/session/session-persistence) |
|
||||
| [`hmr`](../packages/boot/hmr) | `boot` | [`app-boot`](../packages/boot/app-boot), [`cmdline`](../packages/boot/cmdline) |
|
||||
| [`experimental-speech-to-text`](../packages/experimental/speech-to-text) | `experimental` | [`settings`](../packages/settings/settings) |
|
||||
| [`ptc-runtime`](../packages/ptc-runtime/ptc-runtime) | `ptc-runtime` | [`sandbox`](../packages/sandbox/sandbox) |
|
||||
| [`sandbox-local`](../packages/sandbox/sandbox-local) | `sandbox` | [`llm`](../packages/llm/llm), [`sandbox`](../packages/sandbox/sandbox), [`session`](../packages/core/session) |
|
||||
| [`session-persistence-jsonl`](../packages/session/session-persistence-jsonl) | `session` | [`session`](../packages/core/session), [`session-persistence`](../packages/session/session-persistence) |
|
||||
@@ -1480,7 +1507,9 @@ flowchart TD
|
||||
| [`file-reference`](../packages/context/file-reference) | `context` | [`agent`](../packages/core/agent) |
|
||||
| [`time-context`](../packages/context/time-context) | `context` | [`agent`](../packages/core/agent), [`invariants`](../packages/runtime-diagnostics/invariants), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`session-projection`](../packages/session/session-projection) |
|
||||
| [`tmux-context`](../packages/context/tmux-context) | `context` | [`agent`](../packages/core/agent), [`session`](../packages/core/session), [`session-projection`](../packages/session/session-projection), [`shell`](../packages/shell/shell) |
|
||||
| [`experimental-api-speech-to-text`](../packages/experimental/api-speech-to-text) | `experimental` | [`experimental-speech-to-text`](../packages/experimental/speech-to-text), [`typert-protocol`](../packages/typert/protocol) |
|
||||
| [`experimental-ptc-runtime-python`](../packages/experimental/ptc-runtime-python) | `experimental` | [`ptc-runtime`](../packages/ptc-runtime/ptc-runtime), [`timeout`](../packages/util/timeout), [`util-values`](../packages/util/values) |
|
||||
| [`experimental-speech-to-text-sensevoice`](../packages/experimental/speech-to-text-sensevoice) | `experimental` | [`experimental-speech-to-text`](../packages/experimental/speech-to-text), [`subprocess`](../packages/subprocess/subprocess) |
|
||||
| [`commands`](../packages/interaction/commands) | `interaction` | [`agent`](../packages/core/agent), [`attachment`](../packages/attachment/attachment), [`brand`](../packages/util/brand), [`invariants`](../packages/runtime-diagnostics/invariants), [`llm`](../packages/llm/llm), [`scope`](../packages/core/scope), [`session`](../packages/core/session), [`typert-protocol`](../packages/typert/protocol) |
|
||||
| [`user-approval`](../packages/interaction/user-approval) | `interaction` | [`agent`](../packages/core/agent), [`brand`](../packages/util/brand), [`invariants`](../packages/runtime-diagnostics/invariants), [`llm`](../packages/llm/llm), [`scope`](../packages/core/scope), [`session`](../packages/core/session), [`system-prompt`](../packages/core/system-prompt) |
|
||||
| [`user-questions`](../packages/interaction/user-questions) | `interaction` | [`agent`](../packages/core/agent), [`llm`](../packages/llm/llm), [`scope`](../packages/core/scope) |
|
||||
@@ -1593,6 +1622,7 @@ flowchart TD
|
||||
| [`subagent-fork-in-process`](../packages/subagent/subagent-fork-in-process) | `subagent` | [`agent`](../packages/core/agent), [`session`](../packages/core/session), [`subagent`](../packages/subagent/subagent), [`subagent-in-process-driver`](../packages/subagent/subagent-in-process-driver) |
|
||||
| [`subagent-spawn-in-process`](../packages/subagent/subagent-spawn-in-process) | `subagent` | [`subagent`](../packages/subagent/subagent), [`subagent-in-process-driver`](../packages/subagent/subagent-in-process-driver) |
|
||||
| [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | `experimental` | [`api-remotes`](../packages/api/remotes), [`api-session-controller`](../packages/api/session-controller), [`client-locale`](../packages/client/locale), [`client-ui-conversation`](../packages/client/ui-conversation), [`client-ui-primitives`](../packages/client/ui-primitives), [`client-ui-renderer`](../packages/client/ui-renderer), [`client-ui-session`](../packages/client/ui-session), [`client-ui-slots`](../packages/client/ui-slots), [`client-ui-workspace`](../packages/client/ui-workspace), [`experimental-agent-team`](../packages/experimental/agent-team), [`session`](../packages/core/session), [`typert-protocol`](../packages/typert/protocol) |
|
||||
| [`experimental-client-ui-voice-input`](../packages/experimental/client-ui-voice-input) | `experimental` | [`api-gateway`](../packages/api/gateway), [`api-remotes`](../packages/api/remotes), [`api-session-controller`](../packages/api/session-controller), [`client-locale`](../packages/client/locale), [`client-store`](../packages/client/store), [`client-ui-conversation`](../packages/client/ui-conversation), [`client-ui-plugin-manager`](../packages/client/ui-plugin-manager), [`client-ui-primitives`](../packages/client/ui-primitives), [`client-ui-renderer`](../packages/client/ui-renderer), [`client-ui-session`](../packages/client/ui-session), [`client-ui-slots`](../packages/client/ui-slots), [`experimental-api-speech-to-text`](../packages/experimental/api-speech-to-text), [`experimental-speech-to-text`](../packages/experimental/speech-to-text), [`session`](../packages/core/session), [`typert-protocol`](../packages/typert/protocol) |
|
||||
| [`experimental-tool-agent-team`](../packages/experimental/tool-agent-team) | `experimental` | [`agent`](../packages/core/agent), [`experimental-agent-team`](../packages/experimental/agent-team), [`session`](../packages/core/session), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) |
|
||||
| [`sdk-client`](../packages/sdk/client) | `sdk` | [`llm`](../packages/llm/llm), [`sdk-protocol`](../packages/sdk/protocol), [`session`](../packages/core/session) |
|
||||
| [`sdk-jsonrpc-server`](../packages/sdk/server) | `sdk` | [`agent`](../packages/core/agent), [`attachment`](../packages/attachment/attachment), [`llm`](../packages/llm/llm), [`llm-deepseek`](../packages/llm/llm-deepseek), [`scope`](../packages/core/scope), [`sdk-protocol`](../packages/sdk/protocol), [`session`](../packages/core/session), [`subagent`](../packages/subagent/subagent) |
|
||||
|
||||
@@ -234,17 +234,22 @@ flowchart TD
|
||||
subgraph group_experimental["packages/experimental"]
|
||||
pkg_experimental_agent_team["experimental-agent-team"]
|
||||
pkg_experimental_agent_team_profile["experimental-agent-team-profile"]
|
||||
pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"]
|
||||
pkg_experimental_auto_review["experimental-auto-review"]
|
||||
pkg_experimental_browser_use_chrome_devtools_mcp["experimental-browser-use-chrome-devtools-mcp"]
|
||||
pkg_experimental_browser_use_playwright_mcp["experimental-browser-use-playwright-mcp"]
|
||||
pkg_experimental_browser_use_runtime["experimental-browser-use-runtime"]
|
||||
pkg_experimental_browser_use_stagehand_native["experimental-browser-use-stagehand-native"]
|
||||
pkg_experimental_client_ui_agent_team["experimental-client-ui-agent-team"]
|
||||
pkg_experimental_client_ui_voice_input["experimental-client-ui-voice-input"]
|
||||
pkg_experimental_computer_use_cua_driver_mcp["experimental-computer-use-cua-driver-mcp"]
|
||||
pkg_experimental_computer_use_cua_driver_native["experimental-computer-use-cua-driver-native"]
|
||||
pkg_experimental_inspector["experimental-inspector"]
|
||||
pkg_experimental_ptc_runtime_python["experimental-ptc-runtime-python"]
|
||||
pkg_experimental_speech_to_text["experimental-speech-to-text"]
|
||||
pkg_experimental_speech_to_text_sensevoice["experimental-speech-to-text-sensevoice"]
|
||||
pkg_experimental_tool_agent_team["experimental-tool-agent-team"]
|
||||
pkg_experimental_voice_input_bundle["experimental-voice-input-bundle"]
|
||||
pkg_experimental_webworker_packer["experimental-webworker-packer"]
|
||||
pkg_experimental_webworker_runtime["experimental-webworker-runtime"]
|
||||
end
|
||||
@@ -510,6 +515,7 @@ flowchart TD
|
||||
pkg_session_log_export --> pkg_session_persistence
|
||||
pkg_hmr --> pkg_app_boot
|
||||
pkg_hmr --> pkg_cmdline
|
||||
pkg_experimental_speech_to_text --> pkg_settings
|
||||
pkg_ptc_runtime --> pkg_sandbox
|
||||
pkg_sandbox_local --> pkg_llm
|
||||
pkg_sandbox_local --> pkg_sandbox
|
||||
@@ -605,9 +611,13 @@ flowchart TD
|
||||
pkg_tmux_context --> pkg_session
|
||||
pkg_tmux_context --> pkg_session_projection
|
||||
pkg_tmux_context --> pkg_shell
|
||||
pkg_experimental_api_speech_to_text --> pkg_experimental_speech_to_text
|
||||
pkg_experimental_api_speech_to_text --> pkg_typert_protocol
|
||||
pkg_experimental_ptc_runtime_python --> pkg_ptc_runtime
|
||||
pkg_experimental_ptc_runtime_python --> pkg_timeout
|
||||
pkg_experimental_ptc_runtime_python --> pkg_util_values
|
||||
pkg_experimental_speech_to_text_sensevoice --> pkg_experimental_speech_to_text
|
||||
pkg_experimental_speech_to_text_sensevoice --> pkg_subprocess
|
||||
pkg_commands --> pkg_agent
|
||||
pkg_commands --> pkg_attachment
|
||||
pkg_commands --> pkg_brand
|
||||
@@ -1263,6 +1273,21 @@ flowchart TD
|
||||
pkg_experimental_client_ui_agent_team --> pkg_experimental_agent_team
|
||||
pkg_experimental_client_ui_agent_team --> pkg_session
|
||||
pkg_experimental_client_ui_agent_team --> pkg_typert_protocol
|
||||
pkg_experimental_client_ui_voice_input --> pkg_api_gateway
|
||||
pkg_experimental_client_ui_voice_input --> pkg_api_remotes
|
||||
pkg_experimental_client_ui_voice_input --> pkg_api_session_controller
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_locale
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_store
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_ui_conversation
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_ui_plugin_manager
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_ui_primitives
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_ui_renderer
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_ui_session
|
||||
pkg_experimental_client_ui_voice_input --> pkg_client_ui_slots
|
||||
pkg_experimental_client_ui_voice_input --> pkg_experimental_api_speech_to_text
|
||||
pkg_experimental_client_ui_voice_input --> pkg_experimental_speech_to_text
|
||||
pkg_experimental_client_ui_voice_input --> pkg_session
|
||||
pkg_experimental_client_ui_voice_input --> pkg_typert_protocol
|
||||
pkg_experimental_tool_agent_team --> pkg_agent
|
||||
pkg_experimental_tool_agent_team --> pkg_experimental_agent_team
|
||||
pkg_experimental_tool_agent_team --> pkg_session
|
||||
@@ -1387,6 +1412,7 @@ flowchart TD
|
||||
| [`client-web`](../packages/client/web) | `client` | — |
|
||||
| [`office-to-pdf`](../packages/document/office-to-pdf) | `document` | — |
|
||||
| [`experimental-agent-team-profile`](../packages/experimental/agent-team-profile) | `experimental` | — |
|
||||
| [`experimental-voice-input-bundle`](../packages/experimental/voice-input-bundle) | `experimental` | — |
|
||||
| [`experimental-webworker-packer`](../packages/experimental/webworker-packer) | `experimental` | — |
|
||||
| [`client-ui-cordis`](../packages/extensions/ui-cordis) | `extensions` | — |
|
||||
| [`cordis-client-runner`](../packages/extensions/cordis-client-runner) | `extensions` | — |
|
||||
@@ -1459,6 +1485,7 @@ flowchart TD
|
||||
| [`spill-local`](../packages/spill/spill-local) | `spill` | [`spill`](../packages/spill/spill) |
|
||||
| [`session-log-export`](../packages/session-query/session-log-export) | `session-query` | [`session`](../packages/core/session), [`session-persistence`](../packages/session/session-persistence) |
|
||||
| [`hmr`](../packages/boot/hmr) | `boot` | [`app-boot`](../packages/boot/app-boot), [`cmdline`](../packages/boot/cmdline) |
|
||||
| [`experimental-speech-to-text`](../packages/experimental/speech-to-text) | `experimental` | [`settings`](../packages/settings/settings) |
|
||||
| [`ptc-runtime`](../packages/ptc-runtime/ptc-runtime) | `ptc-runtime` | [`sandbox`](../packages/sandbox/sandbox) |
|
||||
| [`sandbox-local`](../packages/sandbox/sandbox-local) | `sandbox` | [`llm`](../packages/llm/llm), [`sandbox`](../packages/sandbox/sandbox), [`session`](../packages/core/session) |
|
||||
| [`session-persistence-jsonl`](../packages/session/session-persistence-jsonl) | `session` | [`session`](../packages/core/session), [`session-persistence`](../packages/session/session-persistence) |
|
||||
@@ -1482,7 +1509,9 @@ flowchart TD
|
||||
| [`file-reference`](../packages/context/file-reference) | `context` | [`agent`](../packages/core/agent) |
|
||||
| [`time-context`](../packages/context/time-context) | `context` | [`agent`](../packages/core/agent), [`invariants`](../packages/runtime-diagnostics/invariants), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`session-projection`](../packages/session/session-projection) |
|
||||
| [`tmux-context`](../packages/context/tmux-context) | `context` | [`agent`](../packages/core/agent), [`session`](../packages/core/session), [`session-projection`](../packages/session/session-projection), [`shell`](../packages/shell/shell) |
|
||||
| [`experimental-api-speech-to-text`](../packages/experimental/api-speech-to-text) | `experimental` | [`experimental-speech-to-text`](../packages/experimental/speech-to-text), [`typert-protocol`](../packages/typert/protocol) |
|
||||
| [`experimental-ptc-runtime-python`](../packages/experimental/ptc-runtime-python) | `experimental` | [`ptc-runtime`](../packages/ptc-runtime/ptc-runtime), [`timeout`](../packages/util/timeout), [`util-values`](../packages/util/values) |
|
||||
| [`experimental-speech-to-text-sensevoice`](../packages/experimental/speech-to-text-sensevoice) | `experimental` | [`experimental-speech-to-text`](../packages/experimental/speech-to-text), [`subprocess`](../packages/subprocess/subprocess) |
|
||||
| [`commands`](../packages/interaction/commands) | `interaction` | [`agent`](../packages/core/agent), [`attachment`](../packages/attachment/attachment), [`brand`](../packages/util/brand), [`invariants`](../packages/runtime-diagnostics/invariants), [`llm`](../packages/llm/llm), [`scope`](../packages/core/scope), [`session`](../packages/core/session), [`typert-protocol`](../packages/typert/protocol) |
|
||||
| [`user-approval`](../packages/interaction/user-approval) | `interaction` | [`agent`](../packages/core/agent), [`brand`](../packages/util/brand), [`invariants`](../packages/runtime-diagnostics/invariants), [`llm`](../packages/llm/llm), [`scope`](../packages/core/scope), [`session`](../packages/core/session), [`system-prompt`](../packages/core/system-prompt) |
|
||||
| [`user-questions`](../packages/interaction/user-questions) | `interaction` | [`agent`](../packages/core/agent), [`llm`](../packages/llm/llm), [`scope`](../packages/core/scope) |
|
||||
@@ -1595,6 +1624,7 @@ flowchart TD
|
||||
| [`subagent-fork-in-process`](../packages/subagent/subagent-fork-in-process) | `subagent` | [`agent`](../packages/core/agent), [`session`](../packages/core/session), [`subagent`](../packages/subagent/subagent), [`subagent-in-process-driver`](../packages/subagent/subagent-in-process-driver) |
|
||||
| [`subagent-spawn-in-process`](../packages/subagent/subagent-spawn-in-process) | `subagent` | [`subagent`](../packages/subagent/subagent), [`subagent-in-process-driver`](../packages/subagent/subagent-in-process-driver) |
|
||||
| [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | `experimental` | [`api-remotes`](../packages/api/remotes), [`api-session-controller`](../packages/api/session-controller), [`client-locale`](../packages/client/locale), [`client-ui-conversation`](../packages/client/ui-conversation), [`client-ui-primitives`](../packages/client/ui-primitives), [`client-ui-renderer`](../packages/client/ui-renderer), [`client-ui-session`](../packages/client/ui-session), [`client-ui-slots`](../packages/client/ui-slots), [`client-ui-workspace`](../packages/client/ui-workspace), [`experimental-agent-team`](../packages/experimental/agent-team), [`session`](../packages/core/session), [`typert-protocol`](../packages/typert/protocol) |
|
||||
| [`experimental-client-ui-voice-input`](../packages/experimental/client-ui-voice-input) | `experimental` | [`api-gateway`](../packages/api/gateway), [`api-remotes`](../packages/api/remotes), [`api-session-controller`](../packages/api/session-controller), [`client-locale`](../packages/client/locale), [`client-store`](../packages/client/store), [`client-ui-conversation`](../packages/client/ui-conversation), [`client-ui-plugin-manager`](../packages/client/ui-plugin-manager), [`client-ui-primitives`](../packages/client/ui-primitives), [`client-ui-renderer`](../packages/client/ui-renderer), [`client-ui-session`](../packages/client/ui-session), [`client-ui-slots`](../packages/client/ui-slots), [`experimental-api-speech-to-text`](../packages/experimental/api-speech-to-text), [`experimental-speech-to-text`](../packages/experimental/speech-to-text), [`session`](../packages/core/session), [`typert-protocol`](../packages/typert/protocol) |
|
||||
| [`experimental-tool-agent-team`](../packages/experimental/tool-agent-team) | `experimental` | [`agent`](../packages/core/agent), [`experimental-agent-team`](../packages/experimental/agent-team), [`session`](../packages/core/session), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) |
|
||||
| [`sdk-client`](../packages/sdk/client) | `sdk` | [`llm`](../packages/llm/llm), [`sdk-protocol`](../packages/sdk/protocol), [`session`](../packages/core/session) |
|
||||
| [`sdk-jsonrpc-server`](../packages/sdk/server) | `sdk` | [`agent`](../packages/core/agent), [`attachment`](../packages/attachment/attachment), [`llm`](../packages/llm/llm), [`llm-deepseek`](../packages/llm/llm-deepseek), [`scope`](../packages/core/scope), [`sdk-protocol`](../packages/sdk/protocol), [`session`](../packages/core/session), [`subagent`](../packages/subagent/subagent) |
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/subsystems/README.md
|
||||
README.md: d54bd4f45c23100cd1d53080f1ca807a9c177d3d
|
||||
README.zh.md: 81ef25b69547672e016a2b72e2293daa8bb42eb1
|
||||
README.md: db21e0fe6bea366c4d7efc1d96d247dd255f54da
|
||||
README.zh.md: 4b2b28dbf9a39da0c0ce3a062c4de3261501f7d9
|
||||
@@ -46,6 +46,7 @@ One page per subsystem of the DeepSeek Harness: what it is, the data structures
|
||||
| [skills.md](skills.md) | the skill service: discovery priority, `SkillSummary`/`SkillDefinition`, session-prefix catalog, model-facing `skill` loading |
|
||||
| [compaction.md](compaction.md) | the compaction seam: the `compaction/*` session events, `CompactionResult`, the `CompactionEngine` interface |
|
||||
| [subagent.md](subagent.md) | the subagent seam: the named-provider registry, `SubagentStartRequest`/`Result`/`Run`, the start-time-vs-runtime capability split |
|
||||
| [voice-input.md](voice-input.md) | experimental named recognizers, transient audio and revision-guarded draft insertion |
|
||||
| [agent-team.md](agent-team.md) | Agent Teams: implicit Lead identity, named continuable teammates, durable peer mailbox, and shared task DAG |
|
||||
| [web.md](web.md) | the web access seam: `WebSearchRequest`/`Result`, `WebFetchRequest`/`Result`, `WebFetchBody`, provider availability, `WebError` |
|
||||
| [spill.md](spill.md) | the spill storage seam: `SaveTextSpill`, `SpillOwner`/`SpillSource`, `SpillRef`, the branded `SpillLocator` |
|
||||
|
||||
@@ -46,6 +46,7 @@
|
||||
| [skills.md](skills.zh.md) | skill(技能)服务:发现优先级、`SkillSummary`/`SkillDefinition`、会话前缀目录、面向模型的 `skill` 加载 |
|
||||
| [compaction.md](compaction.zh.md) | 压缩(compaction)seam:`compaction/*` 会话事件、`CompactionResult`、`CompactionEngine` 接口 |
|
||||
| [subagent.md](subagent.zh.md) | subagent seam:命名提供方注册表、`SubagentStartRequest`/`Result`/`Run`、启动时与运行时能力拆分 |
|
||||
| [voice-input.md](voice-input.zh.md) | 实验性具名语音识别、临时音频和受版本保护的草稿插入 |
|
||||
| [agent-team.md](agent-team.zh.md) | Agent Teams:隐式 Lead 身份、具名 continuable teammate、持久 peer mailbox 与共享任务 DAG |
|
||||
| [web.md](web.zh.md) | Web 访问 seam:`WebSearchRequest`/`Result`、`WebFetchRequest`/`Result`、`WebFetchBody`、提供方可用性、`WebError` |
|
||||
| [spill.md](spill.zh.md) | spill 存储 seam:`SaveTextSpill`、`SpillOwner`/`SpillSource`、`SpillRef`、品牌类型 `SpillLocator` |
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/subsystems/voice-input.md
|
||||
voice-input.md: f1d3f36d067f756235e976fa6eafe70f94b19ccc
|
||||
voice-input.zh.md: a32b539fc1237bb70d68668de939084d5f634a82
|
||||
@@ -0,0 +1,158 @@
|
||||
# Voice input
|
||||
|
||||
English | [中文](voice-input.zh.md)
|
||||
|
||||
Experimental speech recognition has three roles: the [Service Definition](../../packages/experimental/speech-to-text/README.md) routes named providers, the [SenseVoice provider](../../packages/experimental/speech-to-text-sensevoice/README.md) owns local inference, and the [Remote consumer](../../packages/experimental/api-speech-to-text/README.md) serves the browser. The [optional bundle](../../packages/experimental/voice-input-bundle/README.md) composes them with the microphone UI.
|
||||
|
||||
## Provider selection
|
||||
|
||||
`SpeechProviderId` brands the registration identity. `SpeechProviderInfo` carries a display name, accepted `languages` hints and `host-local` or `cloud` processing location. `SpeechRequest` contains WAV bytes and optional provider/language selection; `resolve()` produces `SpeechSpec` with one captured provider instance. A missing provider or unsupported language fails; replacement or withdrawal invalidates resolved work. Audio never falls back to a different provider.
|
||||
|
||||
`SpeechProvider.transcribe()` accepts one complete `SpeechInput` and an `AbortSignal`. `Transcript` returns plain `text`, `audioSeconds` and `inferenceSeconds`. Providers honor cancellation and settle their work before deregistration completes. The local provider serializes calls, bounds its queue and manages its worker lifetime independently of Sessions.
|
||||
|
||||
## Browser and Host ownership
|
||||
|
||||
The browser owns microphone tracks and the unsent draft. `TranscriptionRequest` sends canonical base64 PCM16 WAV through the authenticated Remote. `SpeechCatalog` advertises available providers, the default selection and recording byte/duration limits. The API validates this process input before recognition.
|
||||
|
||||
The input facade captures a revision-bearing selection before recording. `InputActions.insertText()` inserts one undoable plain-text edit only while the selection revision is current and submission permits editing. A rejected insertion leaves the transcript available for explicit insertion. Switching Sessions or disposing the plugin invalidates late results. Recognition itself writes no Session event; ordinary user submission owns the final model-visible text.
|
||||
|
||||
## Preparation and settings
|
||||
|
||||
The Host provider owns one preparation task across page and Session changes. The Client shares one `follow()` subscription across the composer, setup prompt and bundle details. Optional `SpeechSetupEstimate` metadata supplies provider-specific planning hints, separate from measured progress. `SpeechPreparationStepKind` identifies ordered resource operations; `SpeechPreparationStep` records each status and start time. The complete `SpeechPreparationState` retains these steps after cancellation or failure. The collapsed UI shows the current operation; expanding lists all steps, with byte progress or elapsed time only for the running step. Closing an observer never cancels preparation. Verified files survive retries and idle worker reclamation.
|
||||
|
||||
The microphone occupies `conversation.input.activity`, between the model selector and Send. Clicking starts capture and expands the toolbar; Stop transcribes and inserts into the draft. The activity preserves the editor and submit action, owns local feedback, and releases expansion on unmount. Cancel, Escape or hiding the page discards capture. Waveform history displays measured microphone amplitude. Bundle details contain recognition preferences, preparation state and progress. Explicit enablement uses `plugins.bundle.activation` to guide users with missing models to setup; the list shows only the bundle description and switch.
|
||||
|
||||
## Design rationale
|
||||
|
||||
The [voice input decision](../../.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.md) explains transient audio, explicit provider selection and lazy local preparation.
|
||||
|
||||
<!-- BEGIN GENERATED cordis-surface (gen-cordis-catalog.ts) — do not edit between markers -->
|
||||
|
||||
<a id="cordis-surface"></a>
|
||||
|
||||
## Cordis API
|
||||
|
||||
Generated from source by `scripts/gen-cordis-catalog.ts` (verified fresh by `pnpm run verify-cordis-catalog` in doc-sync; regenerate with `pnpm run gen-cordis-catalog`) — the language sides differ only in locale-specific paired document paths. Signature blocks use a `ts cordis-catalog` fence and keep the original source JSDoc; dispatch modes are defined in the [primer](../cordis-primer.md#dispatch-modes), and the framework-inherited `ctx` API lives in [cordis-api/inherited.md](../cordis-api/inherited.md).
|
||||
|
||||
<a id="ctxspeechcontroller--speechcontroller"></a>
|
||||
|
||||
### `ctx.speechController` — `SpeechController`
|
||||
|
||||
Speech calls never activate or submit to an Agent.
|
||||
|
||||
```ts cordis-catalog
|
||||
/**
|
||||
* Read provider choices without preparing a recognizer.
|
||||
* @returns available providers, resolved default, and recording limits.
|
||||
*/
|
||||
@Remote catalog(): SpeechCatalog
|
||||
|
||||
/**
|
||||
* Follow provider readiness independently of Session and preparation lifetimes.
|
||||
* @param signal - Client observation lifetime.
|
||||
* @returns initial and subsequent complete readiness snapshots.
|
||||
*/
|
||||
@Remote({ mode: 'stream' }) async *follow(signal: AbortSignal): AsyncIterable<SpeechCatalog>
|
||||
|
||||
/**
|
||||
* Persist the user's recognition preferences.
|
||||
* @param patch - changed preference fields.
|
||||
* @returns after preferences are saved.
|
||||
*/
|
||||
@Remote configure(patch: SpeechSelectionPatch): Promise<void>
|
||||
|
||||
/**
|
||||
* Start or join one Host-owned preparation task.
|
||||
* @param providerId - selected recognizer.
|
||||
*/
|
||||
@Remote prepare(providerId: SpeechProviderId): void
|
||||
|
||||
/**
|
||||
* Explicitly cancel resource preparation.
|
||||
* @param providerId - selected recognizer.
|
||||
* @returns after the preparation task settles.
|
||||
*/
|
||||
@Remote cancelPreparation(providerId: SpeechProviderId): Promise<void>
|
||||
|
||||
/**
|
||||
* Validate and transcribe one recording through the explicit provider selection.
|
||||
* @param request - canonical WAV encoded as base64, provider id and language hint.
|
||||
* @param signal - Client cancellation or Remote contribution disposal.
|
||||
* @returns final transcript without adding a Session event.
|
||||
*/
|
||||
@Remote async transcribe(request: TranscriptionRequest, signal: AbortSignal): Promise<Transcript>
|
||||
```
|
||||
|
||||
Source: [`packages/experimental/api-speech-to-text/src/index.ts`](../../packages/experimental/api-speech-to-text/src/index.ts)
|
||||
|
||||
<a id="ctxspeechtotext--speechtotext"></a>
|
||||
|
||||
### `ctx.speechToText` — `SpeechToText`
|
||||
|
||||
Registry shared by all transcription consumers in one Host composition.
|
||||
|
||||
```ts cordis-catalog
|
||||
/**
|
||||
* Register one recognizer; duplicate ids fail without replacing the original.
|
||||
* @param provider - recognizer owned by the contributing fiber.
|
||||
* @returns idempotent disposer which rejects admission, cancels, and joins accepted work.
|
||||
*/
|
||||
register(provider: SpeechProvider): () => Promise<void>
|
||||
|
||||
/**
|
||||
* Read the current recognizer roster.
|
||||
* @returns available provider facts in registration order.
|
||||
*/
|
||||
listProviders(): readonly SpeechProviderInfo[]
|
||||
|
||||
/**
|
||||
* Observe complete readiness snapshots; a slow reader coalesces intermediate progress.
|
||||
* @param caller - observer lifetime, independent of any preparation task.
|
||||
* @returns an initial snapshot followed by the latest provider states.
|
||||
*/
|
||||
async *follow(caller: AbortSignal): AsyncIterable<SpeechSnapshot>
|
||||
|
||||
/**
|
||||
* Read provider readiness and current user preferences together.
|
||||
* @returns one detached complete observation.
|
||||
*/
|
||||
snapshot(): SpeechSnapshot
|
||||
|
||||
/**
|
||||
* Persist changed preference fields; the resulting language must be accepted by the selected provider.
|
||||
* @param patch - explicit provider or language changes.
|
||||
* @returns after persistence and the resolved preference update.
|
||||
*/
|
||||
async configure(patch: SpeechSelectionPatch): Promise<void>
|
||||
|
||||
/**
|
||||
* Start or join provider-owned preparation.
|
||||
* @param id - exact registered provider identity.
|
||||
*/
|
||||
prepare(id: SpeechProviderId): void
|
||||
|
||||
/**
|
||||
* Explicitly cancel provider preparation without tying it to a browser connection.
|
||||
* @param id - exact registered provider identity.
|
||||
* @returns after the preparation task settles.
|
||||
*/
|
||||
async cancelPreparation(id: SpeechProviderId): Promise<void>
|
||||
|
||||
/**
|
||||
* Apply composition defaults and capture the selected provider. Missing providers and unsupported languages fail explicitly.
|
||||
* @param request - complete recording and optional selection.
|
||||
* @returns provider-pinned input for transcribe().
|
||||
*/
|
||||
resolve(request: SpeechRequest): SpeechSpec
|
||||
|
||||
/**
|
||||
* Execute exactly the resolved provider; no fallback sends audio elsewhere.
|
||||
* @param spec - resolved input; a withdrawn or replaced registration is rejected.
|
||||
* @param signal - caller cancellation.
|
||||
* @returns final transcript after provider settlement.
|
||||
*/
|
||||
async transcribe(spec: SpeechSpec, signal: AbortSignal): Promise<Transcript>
|
||||
```
|
||||
|
||||
Source: [`packages/experimental/speech-to-text/src/index.ts`](../../packages/experimental/speech-to-text/src/index.ts)
|
||||
<!-- END GENERATED cordis-surface -->
|
||||
@@ -0,0 +1,158 @@
|
||||
# 语音输入
|
||||
|
||||
[English](voice-input.md) | 中文
|
||||
|
||||
实验性语音识别包含三个角色:[服务定义](../../packages/experimental/speech-to-text/README.zh.md)路由具名 Provider,[SenseVoice Provider](../../packages/experimental/speech-to-text-sensevoice/README.zh.md)拥有本地推理,[Remote 消费者](../../packages/experimental/api-speech-to-text/README.zh.md)服务浏览器。[可选 Bundle](../../packages/experimental/voice-input-bundle/README.zh.md)将它们与麦克风 UI 组合。
|
||||
|
||||
## Provider 选择
|
||||
|
||||
`SpeechProviderId` 为注册标识添加类型品牌。`SpeechProviderInfo` 携带显示名称、支持的 `languages` 语言提示及 `host-local` 或 `cloud` 处理位置。`SpeechRequest` 包含 WAV 字节和可选的 Provider、语言选择;`resolve()` 生成捕获一个 Provider 实例的 `SpeechSpec`。缺失的 Provider 或不支持的语言会报错;替换或撤销会使已解析的任务失效。音频不会回退到其他 Provider。
|
||||
|
||||
`SpeechProvider.transcribe()` 接收完整的 `SpeechInput` 与 `AbortSignal`。`Transcript` 返回纯文本 `text`、`audioSeconds` 和 `inferenceSeconds`。Provider 响应取消,并在注销完成前结束其任务。本地 Provider 串行执行调用、限制队列,并独立于 Session 管理工作进程生命周期。
|
||||
|
||||
## 浏览器与 Host 所有权
|
||||
|
||||
浏览器拥有麦克风轨道与未发送的草稿。`TranscriptionRequest` 通过带认证的 Remote 发送规范 base64 PCM16 WAV。`SpeechCatalog` 公布可用 Provider、默认选择及录音字节和时长限制。API 在识别前校验这些进程输入。
|
||||
|
||||
输入门面在录音前捕获带版本的选区。`InputActions.insertText()` 仅在选区版本仍有效且提交状态允许编辑时,插入一次可撤销的纯文本编辑。插入被拒绝时,转写文字保留以供显式插入。切换 Session 或释放插件使迟到结果失效。识别本身不写入 Session 事件;普通用户提交拥有最终的模型可见文字。
|
||||
|
||||
## 准备与设置
|
||||
|
||||
Host Provider 在页面和 Session 变化期间拥有同一个准备任务。Client 在输入框、安装引导弹窗、Bundle 详情之间共享一份 `follow()` 订阅。可选 `SpeechSetupEstimate` 元数据提供各 Provider 的资源预期,与实测进度分开。`SpeechPreparationStepKind` 标识有序资源操作;`SpeechPreparationStep` 记录各步骤状态与开始时间。完整 `SpeechPreparationState` 在取消或失败后保留这些步骤。折叠 UI 显示当前操作,展开后列出所有步骤,仅进行中步骤显示字节进度或等待时间。关闭观察者不会取消准备。已校验文件在重试及空闲进程回收后继续复用。
|
||||
|
||||
麦克风占用模型选择器与发送按钮之间的 `conversation.input.activity`。点击开始录音并展开工具栏;停止后转写并插入草稿。活动栏保留编辑器与提交按钮,拥有局部反馈,并在卸载时释放展开状态。取消、Escape 或隐藏页面会丢弃录音。波形历史展示实测麦克风音量。Bundle 详情包含识别偏好、准备状态和进度。显式启用时通过 `plugins.bundle.activation` 引导缺少模型的用户前往安装;列表只显示 Bundle 描述和开关。
|
||||
|
||||
## 设计依据
|
||||
|
||||
[语音输入决策](../../.agents/notes/implemented/architecture/2026-09-16-experimental-voice-input.zh.md)解释临时音频、显式 Provider 选择和延迟准备本地运行时。
|
||||
|
||||
<!-- BEGIN GENERATED cordis-surface (gen-cordis-catalog.ts) — do not edit between markers -->
|
||||
|
||||
<a id="cordis-surface"></a>
|
||||
|
||||
## Cordis API
|
||||
|
||||
Generated from source by `scripts/gen-cordis-catalog.ts` (verified fresh by `pnpm run verify-cordis-catalog` in doc-sync; regenerate with `pnpm run gen-cordis-catalog`) — the language sides differ only in locale-specific paired document paths. Signature blocks use a `ts cordis-catalog` fence and keep the original source JSDoc; dispatch modes are defined in the [primer](../cordis-primer.zh.md#dispatch-modes), and the framework-inherited `ctx` API lives in [cordis-api/inherited.md](../cordis-api/inherited.md).
|
||||
|
||||
<a id="ctxspeechcontroller--speechcontroller"></a>
|
||||
|
||||
### `ctx.speechController` — `SpeechController`
|
||||
|
||||
Speech calls never activate or submit to an Agent.
|
||||
|
||||
```ts cordis-catalog
|
||||
/**
|
||||
* Read provider choices without preparing a recognizer.
|
||||
* @returns available providers, resolved default, and recording limits.
|
||||
*/
|
||||
@Remote catalog(): SpeechCatalog
|
||||
|
||||
/**
|
||||
* Follow provider readiness independently of Session and preparation lifetimes.
|
||||
* @param signal - Client observation lifetime.
|
||||
* @returns initial and subsequent complete readiness snapshots.
|
||||
*/
|
||||
@Remote({ mode: 'stream' }) async *follow(signal: AbortSignal): AsyncIterable<SpeechCatalog>
|
||||
|
||||
/**
|
||||
* Persist the user's recognition preferences.
|
||||
* @param patch - changed preference fields.
|
||||
* @returns after preferences are saved.
|
||||
*/
|
||||
@Remote configure(patch: SpeechSelectionPatch): Promise<void>
|
||||
|
||||
/**
|
||||
* Start or join one Host-owned preparation task.
|
||||
* @param providerId - selected recognizer.
|
||||
*/
|
||||
@Remote prepare(providerId: SpeechProviderId): void
|
||||
|
||||
/**
|
||||
* Explicitly cancel resource preparation.
|
||||
* @param providerId - selected recognizer.
|
||||
* @returns after the preparation task settles.
|
||||
*/
|
||||
@Remote cancelPreparation(providerId: SpeechProviderId): Promise<void>
|
||||
|
||||
/**
|
||||
* Validate and transcribe one recording through the explicit provider selection.
|
||||
* @param request - canonical WAV encoded as base64, provider id and language hint.
|
||||
* @param signal - Client cancellation or Remote contribution disposal.
|
||||
* @returns final transcript without adding a Session event.
|
||||
*/
|
||||
@Remote async transcribe(request: TranscriptionRequest, signal: AbortSignal): Promise<Transcript>
|
||||
```
|
||||
|
||||
Source: [`packages/experimental/api-speech-to-text/src/index.ts`](../../packages/experimental/api-speech-to-text/src/index.ts)
|
||||
|
||||
<a id="ctxspeechtotext--speechtotext"></a>
|
||||
|
||||
### `ctx.speechToText` — `SpeechToText`
|
||||
|
||||
Registry shared by all transcription consumers in one Host composition.
|
||||
|
||||
```ts cordis-catalog
|
||||
/**
|
||||
* Register one recognizer; duplicate ids fail without replacing the original.
|
||||
* @param provider - recognizer owned by the contributing fiber.
|
||||
* @returns idempotent disposer which rejects admission, cancels, and joins accepted work.
|
||||
*/
|
||||
register(provider: SpeechProvider): () => Promise<void>
|
||||
|
||||
/**
|
||||
* Read the current recognizer roster.
|
||||
* @returns available provider facts in registration order.
|
||||
*/
|
||||
listProviders(): readonly SpeechProviderInfo[]
|
||||
|
||||
/**
|
||||
* Observe complete readiness snapshots; a slow reader coalesces intermediate progress.
|
||||
* @param caller - observer lifetime, independent of any preparation task.
|
||||
* @returns an initial snapshot followed by the latest provider states.
|
||||
*/
|
||||
async *follow(caller: AbortSignal): AsyncIterable<SpeechSnapshot>
|
||||
|
||||
/**
|
||||
* Read provider readiness and current user preferences together.
|
||||
* @returns one detached complete observation.
|
||||
*/
|
||||
snapshot(): SpeechSnapshot
|
||||
|
||||
/**
|
||||
* Persist changed preference fields; the resulting language must be accepted by the selected provider.
|
||||
* @param patch - explicit provider or language changes.
|
||||
* @returns after persistence and the resolved preference update.
|
||||
*/
|
||||
async configure(patch: SpeechSelectionPatch): Promise<void>
|
||||
|
||||
/**
|
||||
* Start or join provider-owned preparation.
|
||||
* @param id - exact registered provider identity.
|
||||
*/
|
||||
prepare(id: SpeechProviderId): void
|
||||
|
||||
/**
|
||||
* Explicitly cancel provider preparation without tying it to a browser connection.
|
||||
* @param id - exact registered provider identity.
|
||||
* @returns after the preparation task settles.
|
||||
*/
|
||||
async cancelPreparation(id: SpeechProviderId): Promise<void>
|
||||
|
||||
/**
|
||||
* Apply composition defaults and capture the selected provider. Missing providers and unsupported languages fail explicitly.
|
||||
* @param request - complete recording and optional selection.
|
||||
* @returns provider-pinned input for transcribe().
|
||||
*/
|
||||
resolve(request: SpeechRequest): SpeechSpec
|
||||
|
||||
/**
|
||||
* Execute exactly the resolved provider; no fallback sends audio elsewhere.
|
||||
* @param spec - resolved input; a withdrawn or replaced registration is rejected.
|
||||
* @param signal - caller cancellation.
|
||||
* @returns final transcript after provider settlement.
|
||||
*/
|
||||
async transcribe(spec: SpeechSpec, signal: AbortSignal): Promise<Transcript>
|
||||
```
|
||||
|
||||
Source: [`packages/experimental/speech-to-text/src/index.ts`](../../packages/experimental/speech-to-text/src/index.ts)
|
||||
<!-- END GENERATED cordis-surface -->
|
||||
@@ -186,6 +186,7 @@ export const DEFAULT_PROFILE_BUNDLES: readonly string[] = ['@deepseek-ai/dsh-bas
|
||||
* manager ([rationale](../../../../.agents/notes/implemented/process/2026-09-15-shipped-optional-bundles.md)).
|
||||
*/
|
||||
export const OPTIONAL_BUNDLES: readonly string[] = [
|
||||
'@deepseek-ai/dsh-experimental-voice-input-bundle',
|
||||
'@deepseek-ai/dsh-experimental-agent-team-profile',
|
||||
]
|
||||
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/experimental/README.md
|
||||
README.md: f9fa7be744ff84993373b0cb15451e1a5d5641a9
|
||||
README.zh.md: 58e7d1614f8ab200e0dad8ebb53122b371b5430d
|
||||
README.md: a3fc9ee815fcee1b918c81a464db250755684381
|
||||
README.zh.md: d038735fc488d7ef53f8514159ed6eec63c6f55d
|
||||
@@ -24,6 +24,11 @@ The experimental group contains prototype capabilities whose contracts can chang
|
||||
|
||||
| Package | Role | ctx key |
|
||||
|---|---|---|
|
||||
| [`speech-to-text`](speech-to-text/README.md) | Named speech recognition providers | `ctx.speechToText` |
|
||||
| [`speech-to-text-sensevoice`](speech-to-text-sensevoice/README.md) | Managed local SenseVoice inference | — |
|
||||
| [`api-speech-to-text`](api-speech-to-text/README.md) | Authenticated transient transcription Remote | `ctx.speechController` |
|
||||
| [`client-ui-voice-input`](client-ui-voice-input/README.md) | Microphone capture and guarded draft insertion | — |
|
||||
| [`voice-input-bundle`](voice-input-bundle/README.md) | Default-disabled optional voice input composition | — |
|
||||
| [`agent-team-profile`](agent-team-profile/README.md) | Agent Teams collaboration, tools, and Web UI bundle | — |
|
||||
| [`agent-team`](agent-team/README.md) | Named teammates with durable messages and a shared task board | `ctx.agentTeams` |
|
||||
| [`client-ui-agent-team`](client-ui-agent-team/README.md) | Team roster, task board, and teammate navigation for Web | — |
|
||||
|
||||
@@ -24,6 +24,11 @@ kind: "package-group"
|
||||
|
||||
| 包 | 职责 | ctx 键 |
|
||||
|---|---|---|
|
||||
| [`speech-to-text`](speech-to-text/README.zh.md) | 具名语音识别 Provider | `ctx.speechToText` |
|
||||
| [`speech-to-text-sensevoice`](speech-to-text-sensevoice/README.zh.md) | 托管本地 SenseVoice 推理 | — |
|
||||
| [`api-speech-to-text`](api-speech-to-text/README.zh.md) | 带认证的临时转写 Remote | `ctx.speechController` |
|
||||
| [`client-ui-voice-input`](client-ui-voice-input/README.zh.md) | 麦克风录音与版本检查后的草稿插入 | — |
|
||||
| [`voice-input-bundle`](voice-input-bundle/README.zh.md) | 默认禁用的可选语音输入组合 | — |
|
||||
| [`agent-team-profile`](agent-team-profile/README.zh.md) | Agent Teams 协作、工具与 Web UI 组合包 | — |
|
||||
| [`agent-team`](agent-team/README.zh.md) | 具名 teammate,成员之间持久消息与共享任务板 | `ctx.agentTeams` |
|
||||
| [`client-ui-agent-team`](client-ui-agent-team/README.zh.md) | Web Team roster、任务板与 teammate 导航 | — |
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/experimental/api-speech-to-text/README.md
|
||||
README.md: ecc83c15b4217323c074733ae77259261598803c
|
||||
README.zh.md: 46ac6a16c902c32f2effb23a98141b1afbdccd37
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
description: "Expose bounded transient transcription through the authenticated Web Remote."
|
||||
kind: "package-reference"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-experimental-api-speech-to-text
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
## Summary
|
||||
|
||||
The `speech` Remote connects browser recordings to `ctx.speechToText`. It exposes provider discovery and one complete-recording transcription call.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Use this package](#use-this-package)
|
||||
- [Understand the implementation](#understand-the-implementation)
|
||||
- [Further Exploration](#further-exploration)
|
||||
- [Model Experience](#model-experience)
|
||||
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
||||
- [Dev Note](#dev-note)
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## Use this package
|
||||
|
||||
Compose with the speech Service Definition and Typert. `maxAudioBytes` and `maxDurationSeconds` limit accepted recordings. The browser UI mounts this package’s generated `/remote` contribution when enabled.
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## Understand the implementation
|
||||
|
||||
<details>
|
||||
<summary>Maintainer details — click to expand</summary>
|
||||
|
||||
`catalog()` exposes provider identities, the default selection and recording limits. `transcribe()` validates canonical base64 and 16 kHz mono PCM16 WAV before resolving the selected provider. The existing gateway owns authentication and cancellation transport. Audio is transient, never a Session event or attachment; only the user’s later ordinary submission records recognized text. No runtime invariant companion is published because validation is stateless and preparation belongs to the provider.
|
||||
|
||||
`follow()` streams complete catalogs, including preparation states and current preferences. `prepare()` starts or joins a Host task; `cancelPreparation()` explicitly cancels it. `configure()` persists the supplied preference fields. A disconnected observer does not cancel preparation.
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## Further Exploration
|
||||
|
||||
[Voice input subsystem](../../../docs/subsystems/voice-input.md)
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## Model Experience
|
||||
|
||||
None, as recordings and preparation remain outside model requests; ordinary user submission owns any later text.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
No direct effect; ordinary submission owns the message content.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- No file upload, persistent transcript history or streaming protocol is exposed. Base64 adds transport overhead; recordings remain bounded by the advertised limits.
|
||||
|
||||
-----
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### Dev Note
|
||||
|
||||
<details>
|
||||
<summary>Maintainer details — click to expand</summary>
|
||||
|
||||
None.
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
description: "通过带认证的 Web Remote 提供有界临时转写。"
|
||||
kind: "package-reference"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-experimental-api-speech-to-text
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
## 概述
|
||||
|
||||
`speech` Remote 将浏览器录音连接至 `ctx.speechToText`,提供 Provider 发现和完整录音转写调用。
|
||||
|
||||
## 目录
|
||||
|
||||
- [使用此包](#use-this-package)
|
||||
- [理解实现](#understand-the-implementation)
|
||||
- [进一步探索](#further-exploration)
|
||||
- [模型体验](#model-experience)
|
||||
- [已知限制与后续工作](#known-limitations-and-deferred-work)
|
||||
- [开发备注](#dev-note)
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## 使用此包
|
||||
|
||||
与语音服务定义和 Typert 组合。`maxAudioBytes` 与 `maxDurationSeconds` 限制接收的录音。浏览器 UI 启用时挂载本包生成的 `/remote` 贡献。
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## 理解实现
|
||||
|
||||
<details>
|
||||
<summary>维护者信息 — 点击展开</summary>
|
||||
|
||||
`catalog()` 暴露 Provider 标识、默认选择与录音限制。`transcribe()` 在解析指定 Provider 前校验规范 base64 与 16 kHz 单声道 PCM16 WAV。现有网关负责认证和取消传输。音频是临时数据,不成为 Session 事件或附件;只有用户之后的普通提交才记录识别文字。不发布运行时不变量伴随模块,因为校验无状态,准备属于 Provider。
|
||||
|
||||
`follow()` 推送包含准备状态和当前偏好的完整目录。`prepare()` 启动或加入 Host 任务;`cancelPreparation()` 显式取消任务。`configure()` 持久化传入的偏好字段。观察连接断开不会取消准备。
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## 进一步探索
|
||||
|
||||
[语音输入子系统](../../../docs/subsystems/voice-input.zh.md)
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## 模型体验
|
||||
|
||||
无,因为录音与准备不进入模型请求;之后的文字由普通用户提交拥有。
|
||||
|
||||
#### KV 缓存影响
|
||||
|
||||
没有直接影响;普通提交拥有消息内容。
|
||||
|
||||
## 已知限制与后续工作
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- 不提供文件上传、持久化转写历史或流式协议。Base64 会增加传输开销;录音受公布的限制约束。
|
||||
|
||||
-----
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### 开发备注
|
||||
|
||||
<details>
|
||||
<summary>维护者信息 — 点击展开</summary>
|
||||
|
||||
无。
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,60 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-experimental-api-speech-to-text",
|
||||
"description": "Authenticated experimental speech transcription for browser clients",
|
||||
"version": "0.1.6-alpha.2",
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
|
||||
"directory": "packages/experimental/api-speech-to-text"
|
||||
},
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json",
|
||||
"./types": {
|
||||
"types": "./lib/types/types.d.ts",
|
||||
"default": "./lib/types/types.js"
|
||||
},
|
||||
"./typert": {
|
||||
"types": "./lib/typert.host.d.ts",
|
||||
"default": "./lib/typert.host.js"
|
||||
},
|
||||
"./remote": {
|
||||
"types": "./lib/typert.remote-client.d.ts",
|
||||
"default": "./lib/typert.remote-client.js"
|
||||
}
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/types/**/*.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/typert.host.js",
|
||||
"lib/typert.host.d.ts",
|
||||
"lib/typert.remote-client.js",
|
||||
"lib/typert.remote-client.d.ts"
|
||||
],
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@deepseek-ai/schemastery": "workspace:^",
|
||||
"zod": "^4.4.3"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:^",
|
||||
"@deepseek-ai/dsh-experimental-speech-to-text": "workspace:^",
|
||||
"@deepseek-ai/dsh-typert-protocol": "workspace:^"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:^",
|
||||
"@deepseek-ai/dsh-experimental-speech-to-text": "workspace:^",
|
||||
"@deepseek-ai/dsh-typert-protocol": "workspace:^"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
/** Authenticated, cancellation-aware Client access to the speech capability. */
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import z from '@deepseek-ai/schemastery'
|
||||
import { Remote, RemoteError, TypertRemoteService } from '@deepseek-ai/dsh-typert-protocol'
|
||||
import type {} from '@deepseek-ai/dsh-experimental-speech-to-text'
|
||||
import type { SpeechProviderId, SpeechSelectionPatch, Transcript } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import type { SpeechCatalog, TranscriptionRequest } from './types.ts'
|
||||
import { validateWave } from '@deepseek-ai/dsh-experimental-speech-to-text/wave'
|
||||
|
||||
export type * from './types.ts'
|
||||
|
||||
declare module '@deepseek-ai/cordis' {
|
||||
interface Context {
|
||||
/** Experimental speech Remote controller. */
|
||||
speechController: SpeechController
|
||||
}
|
||||
}
|
||||
|
||||
/** Limits applied before decoding or calling a provider. */
|
||||
export interface Config {
|
||||
/** Maximum decoded WAV bytes per request. */
|
||||
maxAudioBytes: number
|
||||
/** Maximum PCM recording duration in seconds. */
|
||||
maxDurationSeconds: number
|
||||
}
|
||||
|
||||
/** Speech calls never activate or submit to an Agent. */
|
||||
export default class SpeechController extends TypertRemoteService {
|
||||
static inject = ['speechToText', 'typert']
|
||||
static Config: z<Config> = z.object({
|
||||
maxAudioBytes: z.natural().min(46).default(4 * 1024 * 1024),
|
||||
maxDurationSeconds: z.number().min(1).default(120),
|
||||
})
|
||||
|
||||
constructor(ctx: Context, private readonly config: Config) {
|
||||
super(ctx, 'speechController', { namespace: 'speech' })
|
||||
}
|
||||
|
||||
/**
|
||||
* Read provider choices without preparing a recognizer.
|
||||
* @returns available providers, resolved default, and recording limits.
|
||||
*/
|
||||
@Remote
|
||||
catalog(): SpeechCatalog {
|
||||
return { ...this.ctx.speechToText.snapshot(), ...this.config }
|
||||
}
|
||||
|
||||
/**
|
||||
* Follow provider readiness independently of Session and preparation lifetimes.
|
||||
* @param signal - Client observation lifetime.
|
||||
* @returns initial and subsequent complete readiness snapshots.
|
||||
*/
|
||||
@Remote({ mode: 'stream' })
|
||||
async *follow(signal: AbortSignal): AsyncIterable<SpeechCatalog> {
|
||||
for await (const snapshot of this.ctx.speechToText.follow(signal)) yield { ...snapshot, ...this.config }
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist the user's recognition preferences.
|
||||
* @param patch - changed preference fields.
|
||||
* @returns after preferences are saved.
|
||||
*/
|
||||
@Remote
|
||||
configure(patch: SpeechSelectionPatch): Promise<void> { return this.ctx.speechToText.configure(patch) }
|
||||
|
||||
/**
|
||||
* Start or join one Host-owned preparation task.
|
||||
* @param providerId - selected recognizer.
|
||||
*/
|
||||
@Remote
|
||||
prepare(providerId: SpeechProviderId): void { this.ctx.speechToText.prepare(providerId) }
|
||||
|
||||
/**
|
||||
* Explicitly cancel resource preparation.
|
||||
* @param providerId - selected recognizer.
|
||||
* @returns after the preparation task settles.
|
||||
*/
|
||||
@Remote
|
||||
cancelPreparation(providerId: SpeechProviderId): Promise<void> { return this.ctx.speechToText.cancelPreparation(providerId) }
|
||||
|
||||
/**
|
||||
* Validate and transcribe one recording through the explicit provider selection.
|
||||
* @param request - canonical WAV encoded as base64, provider id and language hint.
|
||||
* @param signal - Client cancellation or Remote contribution disposal.
|
||||
* @returns final transcript without adding a Session event.
|
||||
*/
|
||||
@Remote
|
||||
async transcribe(request: TranscriptionRequest, signal: AbortSignal): Promise<Transcript> {
|
||||
signal.throwIfAborted()
|
||||
const encoded = request.audioBase64
|
||||
if (encoded.length > Math.ceil(this.config.maxAudioBytes / 3) * 4) {
|
||||
throw new RemoteError('speech/invalid-audio', 'Audio is invalid or exceeds the configured byte limit', { reason: 'encoding-or-size' })
|
||||
}
|
||||
const audio = Buffer.from(encoded, 'base64')
|
||||
try {
|
||||
if (audio.toString('base64') !== encoded) throw new Error('Audio must use canonical base64 encoding')
|
||||
if (audio.length > this.config.maxAudioBytes) throw new Error('Audio exceeds the configured byte limit')
|
||||
validateWave(audio, this.config.maxDurationSeconds)
|
||||
const spec = this.ctx.speechToText.resolve({ audio,
|
||||
...request.providerId === undefined ? {} : { providerId: request.providerId },
|
||||
...request.language === undefined ? {} : { language: request.language },
|
||||
})
|
||||
return await this.ctx.speechToText.transcribe(spec, signal)
|
||||
} catch (error) {
|
||||
signal.throwIfAborted()
|
||||
const reason = error instanceof Error ? error.message : String(error)
|
||||
throw new RemoteError('speech/transcription-failed', reason, { reason })
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
/** JSON-safe inputs and results of the experimental speech Remote namespace. */
|
||||
import type { SpeechProviderId, SpeechSnapshot } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import type {} from '@deepseek-ai/dsh-typert-protocol'
|
||||
|
||||
declare module '@deepseek-ai/dsh-typert-protocol' {
|
||||
interface RemoteErrorDetailsMap {
|
||||
/** Audio encoding or intake limits prevented transcription. */
|
||||
'speech/invalid-audio': { readonly reason: string }
|
||||
/** The selected provider rejected transcription. */
|
||||
'speech/transcription-failed': { readonly reason: string }
|
||||
}
|
||||
}
|
||||
|
||||
/** Complete recording submitted for transcription, separate from Session admission. */
|
||||
export interface TranscriptionRequest {
|
||||
readonly audioBase64: string
|
||||
readonly providerId?: SpeechProviderId
|
||||
readonly language?: string
|
||||
}
|
||||
|
||||
/** Current provider choices and audio intake limits. */
|
||||
export interface SpeechCatalog extends SpeechSnapshot {
|
||||
readonly maxAudioBytes: number
|
||||
readonly maxDurationSeconds: number
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
/** Authenticated speech domain methods validate audio before selecting a recognizer. */
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import { afterEach, expect, it, vi } from 'vitest'
|
||||
import SpeechToText from '@deepseek-ai/dsh-experimental-speech-to-text'
|
||||
import type { SpeechProviderId } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import SpeechController from '../src/index.ts'
|
||||
|
||||
const roots: Context[] = []
|
||||
afterEach(async () => { await Promise.all(roots.splice(0).map(root => root.fiber.dispose())) })
|
||||
const id = 'sensevoice-local' as SpeechProviderId
|
||||
function fixture(maxAudioBytes = 32044, maxDurationSeconds = 1) {
|
||||
const ctx = new Context(); roots.push(ctx)
|
||||
const speech = new SpeechToText(ctx, { defaultProvider: id, language: 'auto' })
|
||||
const recognize = vi.fn(async () => ({ text: '你好', audioSeconds: 1, inferenceSeconds: 0.1 }))
|
||||
speech.register({ info: { id, name: 'test', location: 'host-local', languages: ['auto', 'zh', 'en', 'ja'] }, transcribe: recognize })
|
||||
return { api: new SpeechController(ctx, { maxAudioBytes, maxDurationSeconds }), recognize }
|
||||
}
|
||||
function recording(): string {
|
||||
const b = Buffer.alloc(32044)
|
||||
b.write('RIFF'); b.writeUInt32LE(b.length - 8, 4); b.write('WAVEfmt ', 8)
|
||||
b.writeUInt32LE(16, 16); b.writeUInt16LE(1, 20); b.writeUInt16LE(1, 22)
|
||||
b.writeUInt32LE(16000, 24); b.writeUInt32LE(32000, 28); b.writeUInt16LE(2, 32)
|
||||
b.writeUInt16LE(16, 34); b.write('data', 36); b.writeUInt32LE(32000, 40)
|
||||
return b.toString('base64')
|
||||
}
|
||||
|
||||
it('lists limits without inference and forwards explicit provider and language', async () => {
|
||||
const { api, recognize } = fixture()
|
||||
expect(api.catalog()).toMatchObject({ selection: { providerId: id }, maxAudioBytes: 32044, providers: [{ id }] })
|
||||
expect(recognize).not.toHaveBeenCalled()
|
||||
api.prepare(id)
|
||||
await api.cancelPreparation(id)
|
||||
const lifetime = new AbortController(), observation = api.follow(lifetime.signal)[Symbol.asyncIterator]()
|
||||
expect((await observation.next()).value).toMatchObject({ providers: [{ id, preparation: { phase: 'ready' } }] })
|
||||
lifetime.abort(); await observation.next()
|
||||
expect(await api.transcribe({ audioBase64: recording(), providerId: id, language: 'zh' }, new AbortController().signal)).toMatchObject({ text: '你好' })
|
||||
expect(recognize).toHaveBeenCalledWith(expect.objectContaining({ language: 'zh' }), expect.any(AbortSignal))
|
||||
await api.transcribe({ audioBase64: recording() }, new AbortController().signal)
|
||||
expect(recognize).toHaveBeenLastCalledWith(expect.objectContaining({ language: 'auto' }), expect.any(AbortSignal))
|
||||
})
|
||||
|
||||
it('rejects oversized, noncanonical, malformed and overlong audio without inference', async () => {
|
||||
const { api, recognize } = fixture()
|
||||
for (const audioBase64 of ['x'.repeat(50000), 'AAAA?', '', 'QQ==', recording() + '\n']) {
|
||||
await expect(api.transcribe({ audioBase64 }, new AbortController().signal)).rejects.toThrow()
|
||||
}
|
||||
await expect(fixture(32043).api.transcribe({ audioBase64: recording() }, new AbortController().signal)).rejects.toThrow('byte limit')
|
||||
const oversized = Buffer.concat([Buffer.from(recording(), 'base64'), Buffer.alloc(2)]).toString('base64')
|
||||
await expect(api.transcribe({ audioBase64: oversized }, new AbortController().signal)).rejects.toThrow('byte limit')
|
||||
await expect(fixture(32044, 0.5).api.transcribe({ audioBase64: recording() }, new AbortController().signal)).rejects.toThrow('exceeds')
|
||||
expect(recognize).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('preserves cancellation and reports recognizer failures', async () => {
|
||||
const { api, recognize } = fixture()
|
||||
await expect(api.configure({ language: 'zh' })).rejects.toThrow('user-settings')
|
||||
await expect(api.transcribe({ audioBase64: recording() }, AbortSignal.abort(new Error('cancel')))).rejects.toThrow('cancel')
|
||||
recognize.mockRejectedValueOnce(new Error('offline'))
|
||||
await expect(api.transcribe({ audioBase64: recording() }, new AbortController().signal)).rejects.toMatchObject({ code: 'speech/transcription-failed', message: 'offline' })
|
||||
recognize.mockRejectedValueOnce('failed')
|
||||
await expect(api.transcribe({ audioBase64: recording() }, new AbortController().signal)).rejects.toMatchObject({ message: 'failed' })
|
||||
})
|
||||
@@ -0,0 +1,28 @@
|
||||
/** Wire audio validation rejects malformed headers and inconsistent duration. */
|
||||
import { expect, it } from 'vitest'
|
||||
import { validateWave } from '@deepseek-ai/dsh-experimental-speech-to-text/wave'
|
||||
|
||||
function wave(): Buffer {
|
||||
const bytes = Buffer.alloc(44 + 32000)
|
||||
bytes.write('RIFF'); bytes.writeUInt32LE(bytes.length - 8, 4); bytes.write('WAVEfmt ', 8)
|
||||
bytes.writeUInt32LE(16, 16); bytes.writeUInt16LE(1, 20); bytes.writeUInt16LE(1, 22)
|
||||
bytes.writeUInt32LE(16000, 24); bytes.writeUInt32LE(32000, 28); bytes.writeUInt16LE(2, 32)
|
||||
bytes.writeUInt16LE(16, 34); bytes.write('data', 36); bytes.writeUInt32LE(32000, 40)
|
||||
return bytes
|
||||
}
|
||||
|
||||
it('accepts complete PCM audio and rejects duration beyond the caller limit', () => {
|
||||
expect(validateWave(wave(), 1)).toBe(1)
|
||||
expect(() => validateWave(wave(), 0.9)).toThrow('exceeds')
|
||||
})
|
||||
|
||||
it('rejects each inconsistent RIFF field and truncated recordings', () => {
|
||||
expect(() => validateWave(new Uint8Array(), 2)).toThrow('canonical')
|
||||
for (const offset of [0, 8, 12, 16, 20, 22, 24, 28, 32, 34, 36, 4, 40]) {
|
||||
const bytes = wave(); bytes[offset] = bytes[offset]! ^ 1
|
||||
expect(() => validateWave(bytes, 2)).toThrow('canonical')
|
||||
}
|
||||
const odd = Buffer.concat([wave(), Buffer.alloc(1)])
|
||||
odd.writeUInt32LE(odd.length - 8, 4); odd.writeUInt32LE(odd.length - 44, 40)
|
||||
expect(() => validateWave(odd, 2)).toThrow('canonical')
|
||||
})
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../speech-to-text"
|
||||
},
|
||||
{
|
||||
"path": "../../typert/protocol"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
import { defineConfig } from 'tsdown'
|
||||
|
||||
export default defineConfig({ entry: ['lib/types/index.js'], outDir: 'lib', format: ['esm'], platform: 'node', target: 'es2024', fixedExtension: false, dts: false, clean: false })
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/experimental/client-ui-voice-input/README.md
|
||||
README.md: b9966a6bbfc8dd9ec3209f93c0355fbe5ceff383
|
||||
README.zh.md: 83e7e1f23267ec838dbf7f44f5add4a78da06048
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
description: "Record speech and insert reviewable transcripts into the conversation draft."
|
||||
kind: "package-reference"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-experimental-client-ui-voice-input
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
## Summary
|
||||
|
||||
This optional browser plugin adds a microphone between the model selector and Send. Clicking it opens a recording toolbar with measured audio levels, Cancel and Stop. The waveform stays shorter than the recording buttons. Stopping transcribes into the draft. Recognition preferences and model preparation live in plugin settings; language choices come from the selected provider.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Use this package](#use-this-package)
|
||||
- [Understand the implementation](#understand-the-implementation)
|
||||
- [Further Exploration](#further-exploration)
|
||||
- [Model Experience](#model-experience)
|
||||
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
||||
- [Dev Note](#dev-note)
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## Use this package
|
||||
|
||||
Enable the [voice input bundle](../voice-input-bundle/README.md) from Plugins. After cache inspection, missing models trigger a setup prompt with Go to setup and Later actions. Go to setup opens bundle details; Download and prepare starts installation there. Intact cached models need no prompt or repeated installation. Before preparation, local providers can show their estimated disk, memory and setup-time requirements. The preparation summary starts collapsed; expand it to inspect completed, current and pending steps. Once ready, click the microphone, allow access and click Stop to transcribe. Cancel or Escape discards capture. Recognition feedback stays inside the toolbar. Provider and language preferences are saved from bundle details through the Settings service. Browser microphone access requires HTTPS or loopback and operating-system permission.
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## Understand the implementation
|
||||
|
||||
<details>
|
||||
<summary>Maintainer details — click to expand</summary>
|
||||
|
||||
Native MediaRecorder captures audio and Web Audio converts it to the Host PCM format after flushing the final recorder chunks. Window blur during the microphone permission request does not cancel capture; blur during recording does. Capture failures immediately end the recording UI and offer retry inside the toolbar. Cancellation, failure and plugin disposal share one resource-release promise; the plugin retains ownership until AudioContext closure settles. The original editor selection carries a draft revision: changed drafts retain the transcript for explicit insertion or discard beside the microphone. Session changes, hiding the page during capture and plugin disposal invalidate late results and release tracks. One plugin-owned readiness subscription serves the composer, installation prompt and details. These views use the existing Slot lifecycle and own no preparation tasks. No runtime invariant companion is published because readiness comes from one Host subscription and recording state belongs to one capture operation.
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## Further Exploration
|
||||
|
||||
[Voice input subsystem](../../../docs/subsystems/voice-input.md)
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## Model Experience
|
||||
|
||||
None, as recordings and preparation remain outside model requests; ordinary user submission owns any later text.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
No direct effect; ordinary submission owns the message content.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- No automatic send, always-on microphone, wake word, streaming captions or speech synthesis. “Local” means the Host machine, which can differ from the browser’s machine.
|
||||
|
||||
-----
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### Dev Note
|
||||
|
||||
<details>
|
||||
<summary>Maintainer details — click to expand</summary>
|
||||
|
||||
None.
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
description: "录制语音并将可确认的转写文字插入会话草稿。"
|
||||
kind: "package-reference"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-experimental-client-ui-voice-input
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
## 概述
|
||||
|
||||
此可选浏览器插件在模型选择器与发送按钮之间添加麦克风。点击后展开录音栏,显示实际音量以及取消、停止按钮。声波高度低于录音按钮高度。停止后转写到草稿。识别偏好和模型准备放在插件设置中;语言选项来自所选 Provider。
|
||||
|
||||
## 目录
|
||||
|
||||
- [使用此包](#use-this-package)
|
||||
- [理解实现](#understand-the-implementation)
|
||||
- [进一步探索](#further-exploration)
|
||||
- [模型体验](#model-experience)
|
||||
- [已知限制与后续工作](#known-limitations-and-deferred-work)
|
||||
- [开发备注](#dev-note)
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## 使用此包
|
||||
|
||||
在插件管理页启用[语音输入 Bundle](../voice-input-bundle/README.zh.md)。缓存检查后,缺少模型时弹出安装引导,可选择“前往安装”或“稍后”。“前往安装”打开 Bundle 详情,再点击“下载并准备”才开始安装。缓存完整时不弹引导,也无需重复安装。准备前,本地 Provider 可展示磁盘、内存与安装时间的估算。准备摘要默认折叠;展开后可查看已完成、当前和未开始的步骤。就绪后点击麦克风,允许访问,再点击停止以转写。取消或 Escape 丢弃录音。识别反馈留在工具栏内部。Provider 和语言偏好在 Bundle 详情中通过 Settings 服务保存。浏览器麦克风访问需要 HTTPS 或环回地址,以及操作系统权限。
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## 理解实现
|
||||
|
||||
<details>
|
||||
<summary>维护者信息 — 点击展开</summary>
|
||||
|
||||
原生 MediaRecorder 捕获音频,等待最后一块数据后由 Web Audio 转换为 Host 的 PCM 格式。等待麦克风授权时窗口失焦不会取消采集;录音期间失焦会取消。采集失败立即结束录音界面,并在活动栏内提供重试。取消、失败和插件释放共享同一个资源释放 Promise;插件保留所有权,直到 AudioContext 关闭任务结束。原编辑器选区携带草稿版本:草稿变化时,转写文字保留在麦克风旁,供显式插入或丢弃。切换 Session、录音期间隐藏页面和释放插件会使迟到结果失效并释放轨道。插件拥有的一份准备状态订阅服务于输入框、安装引导和详情。这些视图使用既有 Slot 生命周期,不拥有准备任务。不发布运行时不变量伴随模块,因为准备状态来自一份 Host 订阅,录音状态属于一个捕获操作。
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## 进一步探索
|
||||
|
||||
[语音输入子系统](../../../docs/subsystems/voice-input.zh.md)
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## 模型体验
|
||||
|
||||
无,因为录音与准备不进入模型请求;之后的文字由普通用户提交拥有。
|
||||
|
||||
#### KV 缓存影响
|
||||
|
||||
没有直接影响;普通提交拥有消息内容。
|
||||
|
||||
## 已知限制与后续工作
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- 不提供自动发送、常开麦克风、唤醒词、流式字幕或语音合成。“本地”指 Host 所在机器,可能与浏览器所在机器不同。
|
||||
|
||||
-----
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### 开发备注
|
||||
|
||||
<details>
|
||||
<summary>维护者信息 — 点击展开</summary>
|
||||
|
||||
无。
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,97 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-experimental-client-ui-voice-input",
|
||||
"description": "Record speech and insert editable text into the conversation draft",
|
||||
"version": "0.1.6-alpha.2",
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
|
||||
"directory": "packages/experimental/client-ui-voice-input"
|
||||
},
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json",
|
||||
"./client": {
|
||||
"types": "./lib/types/client/index.d.ts",
|
||||
"default": "./lib/client.js"
|
||||
}
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/client.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "MIT",
|
||||
"dependencies": {},
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:^",
|
||||
"@deepseek-ai/dsh-api-remotes": "workspace:^",
|
||||
"@deepseek-ai/dsh-api-session-controller": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-locale": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-conversation": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-primitives": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-renderer": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-slots": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-typert-protocol": "workspace:^",
|
||||
"@deepseek-ai/dsh-experimental-api-speech-to-text": "workspace:^",
|
||||
"@deepseek-ai/dsh-experimental-speech-to-text": "workspace:^",
|
||||
"@deepseek-ai/dsh-api-gateway": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-store": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-plugin-manager": "workspace:^"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:^",
|
||||
"@deepseek-ai/dsh-api-remotes": "workspace:^",
|
||||
"@deepseek-ai/dsh-api-session-controller": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-locale": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-conversation": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-primitives": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-renderer": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-slots": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-typert-protocol": "workspace:^",
|
||||
"@deepseek-ai/dsh-experimental-api-speech-to-text": "workspace:^",
|
||||
"@deepseek-ai/dsh-experimental-speech-to-text": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-test-runtime": "workspace:^",
|
||||
"@types/react": "~18.3.1",
|
||||
"react": "^18.2.0",
|
||||
"react-dom": "^18.2.0",
|
||||
"@testing-library/react": "^16.1.0",
|
||||
"@deepseek-ai/dsh-api-gateway": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-store": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-plugin-manager": "workspace:^"
|
||||
},
|
||||
"dsh": {
|
||||
"client": {
|
||||
"inject": [
|
||||
"@deepseek-ai/dsh-api-remotes",
|
||||
"@deepseek-ai/dsh-api-session-controller",
|
||||
"@deepseek-ai/dsh-client-locale",
|
||||
"@deepseek-ai/dsh-client-ui-conversation",
|
||||
"@deepseek-ai/dsh-client-ui-primitives",
|
||||
"@deepseek-ai/dsh-api-gateway",
|
||||
"@deepseek-ai/dsh-client-ui-plugin-manager"
|
||||
],
|
||||
"platform": "web",
|
||||
"external": [
|
||||
"@deepseek-ai/dsh-api-gateway/client"
|
||||
]
|
||||
}
|
||||
},
|
||||
"scripts": {
|
||||
"bundle": "tsdown",
|
||||
"watch": "tsdown --watch"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
/** Provider-owned preparation steps and persisted recognition preferences. */
|
||||
import { useEffect, useState } from 'react'
|
||||
import { Button, DisclosureRow, IconChevronDownOutlineRegular, StateDot, type StateDotState } from '@deepseek-ai/dsh-client-ui-primitives'
|
||||
import type { InjectFace, PropsLocale } from '@deepseek-ai/dsh-client-ui-slots'
|
||||
import type { SpeechPreparationState, SpeechProviderId, SpeechProviderView, SpeechSelectionPatch } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import type { VoiceInputInjected } from './VoiceInput.tsx'
|
||||
import { NS } from './locales.ts'
|
||||
import css from './VoiceInput.module.css'
|
||||
|
||||
/** One recognizer's resource readiness and explicit preparation controls. */
|
||||
export type PreparationCardProps = Pick<InjectFace<VoiceInputInjected>, 'prepare' | 'cancelPreparation'>
|
||||
& PropsLocale<typeof NS> & { readonly provider: SpeechProviderView; readonly connected: boolean }
|
||||
|
||||
function preparationTone(state: SpeechPreparationState): StateDotState {
|
||||
if (state.phase === 'ready' || state.phase === 'standby') return 'done'
|
||||
if (state.phase === 'failed') return 'error'
|
||||
if (state.phase === 'unprepared' || state.phase === 'cancelled') return 'idle'
|
||||
return 'ongoing'
|
||||
}
|
||||
|
||||
function byteText(state: Extract<SpeechPreparationState, { phase: 'downloading' }>, t: PropsLocale<typeof NS>['t']): string {
|
||||
return state.totalBytes === undefined ? t('downloadUnknown', { completed: (state.completedBytes / 1_000_000).toFixed(1) })
|
||||
: t('downloadBytes', { completed: (state.completedBytes / 1_000_000).toFixed(1),
|
||||
total: (state.totalBytes / 1_000_000).toFixed(1), percent: String(Math.floor(state.completedBytes / state.totalBytes * 100)) })
|
||||
}
|
||||
|
||||
function DownloadProgress({ state, t }: { state: SpeechPreparationState } & PropsLocale<typeof NS>) {
|
||||
return state.phase === 'downloading' && state.totalBytes !== undefined
|
||||
? <progress className={css.progress} aria-label={t('downloadProgress')} value={state.completedBytes} max={state.totalBytes} /> : null
|
||||
}
|
||||
|
||||
/** Render a collapsed current-step summary or all Host-owned preparation steps. */
|
||||
export function PreparationCard({ provider, connected, prepare, cancelPreparation, t }: PreparationCardProps) {
|
||||
const state = provider.preparation
|
||||
const [expanded, setExpanded] = useState(false), [now, setNow] = useState(Date.now), [error, setError] = useState('')
|
||||
const current = state.steps?.find(step => step.status === 'running' || step.status === 'failed' || step.status === 'cancelled')
|
||||
const preparing = preparationTone(state) === 'ongoing'
|
||||
const startedAt = current?.startedAt ?? ('startedAt' in state ? state.startedAt : undefined)
|
||||
useEffect(() => {
|
||||
if (!preparing || startedAt === undefined) return
|
||||
const timer = setInterval(() => { setNow(Date.now()) }, 1000)
|
||||
return () => { clearInterval(timer) }
|
||||
}, [preparing, startedAt])
|
||||
const run = async (action: () => Promise<unknown>): Promise<void> => {
|
||||
setError('')
|
||||
try { await action() } catch (failure) { setError(failure instanceof Error ? failure.message : String(failure)) }
|
||||
}
|
||||
const metric = state.phase === 'downloading' ? byteText(state, t)
|
||||
: preparing && startedAt !== undefined ? t('elapsed', { seconds: String(Math.max(0, Math.floor((now - startedAt) / 1000))) }) : ''
|
||||
const summary = !connected ? t('reconnecting') : preparing && current ? t(`step.${current.kind}`, { name: provider.name })
|
||||
: state.phase === 'downloading' ? t('short.downloading')
|
||||
: state.phase === 'failed' ? t('short.failed')
|
||||
: state.phase === 'ready' && provider.location === 'cloud' ? t('cloudReady')
|
||||
: t(`preparation.${state.phase}`, { name: provider.name })
|
||||
return <section className={css.preparation} data-speech-provider={provider.id}>
|
||||
<strong>{provider.name}</strong>
|
||||
{state.phase === 'unprepared' && provider.location === 'host-local' && provider.setupEstimate && <div className={css.setupEstimate}>
|
||||
<p>{t('setup.local')}</p>
|
||||
<dl>
|
||||
<div><dt>{t('setup.disk')}</dt><dd>{t('setup.diskValue', {
|
||||
gb: String(provider.setupEstimate.recommendedDiskBytes / 1_000_000_000),
|
||||
})}</dd></div>
|
||||
<div><dt>{t('setup.memory')}</dt><dd>{t('setup.memoryValue', {
|
||||
gb: String(provider.setupEstimate.expectedMemoryBytes / 1_000_000_000),
|
||||
})}</dd></div>
|
||||
<div><dt>{t('setup.time')}</dt><dd>{t('setup.timeValue', {
|
||||
min: String(provider.setupEstimate.minimumMinutes), max: String(provider.setupEstimate.maximumMinutes),
|
||||
})}</dd></div>
|
||||
</dl>
|
||||
<small>{t('setup.estimateNote')}</small>
|
||||
</div>}
|
||||
<DisclosureRow icon={<StateDot state={preparationTone(state)} size={16} appearance="step" />}
|
||||
title={expanded ? t('preparationSteps') : summary} open={expanded} expandable={(state.steps?.length ?? 0) > 0} expandOnRowClick previewChevron={false}
|
||||
onToggle={() => { setExpanded(value => !value) }} rowClassName={css.summaryRow} titleClassName={css.summaryTitle}
|
||||
collapsedContent={<><span className={css.metric} role="status">{metric}</span>{state.steps && <IconChevronDownOutlineRegular />}</>}>
|
||||
<ol className={css.steps}>
|
||||
{state.steps?.map(step => <li key={step.kind} data-step={step.kind} data-step-state={step.status}
|
||||
aria-label={`${t(`step.${step.kind}`, { name: provider.name })} · ${t(`stepStatus.${step.status}`)}`}>
|
||||
<StateDot appearance="step" size={16} state={step.status === 'complete' ? 'done' : step.status === 'running' ? 'ongoing'
|
||||
: step.status === 'failed' ? 'error' : 'idle'} />
|
||||
<div className={css.stepBody}><span>{t(`step.${step.kind}`, { name: provider.name })}</span>
|
||||
{step.status === 'running' && <div className={css.stepProgress}>
|
||||
<span className={css.metric} role="status">{metric}</span><DownloadProgress state={state} t={t} />
|
||||
{state.phase === 'downloading' && <small>{state.resource}</small>}
|
||||
</div>}
|
||||
</div>
|
||||
</li>)}
|
||||
</ol>
|
||||
</DisclosureRow>
|
||||
{!expanded && <DownloadProgress state={state} t={t} />}
|
||||
{state.phase === 'failed' && <p className={css.preparationError} role="alert">{t('preparationFailed', { message: state.message })}</p>}
|
||||
<div className={css.preparationActions}>
|
||||
{['unprepared', 'cancelled', 'failed'].includes(state.phase) && <Button variant="outline" size="sm" disabled={!connected}
|
||||
onClick={() => { void run(() => prepare(provider.id)) }}>{t(state.phase === 'unprepared' ? 'prepare' : 'retryPrepare')}</Button>}
|
||||
{preparing && state.phase !== 'waking' && <Button variant="ghost" size="sm" disabled={!connected || state.phase === 'cancelling'}
|
||||
onClick={() => { void run(() => cancelPreparation(provider.id)) }}>{t('cancelPrepare')}</Button>}
|
||||
</div>
|
||||
{error && <p className={css.preparationError} role="alert">{t('failed', { message: error })}</p>}
|
||||
</section>
|
||||
}
|
||||
|
||||
/** Recognition preferences and preparation cards shared by plugin details and Settings. */
|
||||
export function VoicePreparation({ useSpeechReadiness, ...props }: InjectFace<VoiceInputInjected> & PropsLocale<typeof NS>) {
|
||||
const readiness = useSpeechReadiness(value => value), catalog = readiness.catalog
|
||||
const [saving, setSaving] = useState(false), [error, setError] = useState('')
|
||||
const configure = async (patch: SpeechSelectionPatch): Promise<void> => {
|
||||
setSaving(true); setError('')
|
||||
try { await props.configure(patch) } catch (failure) { setError(failure instanceof Error ? failure.message : String(failure)) }
|
||||
finally { setSaving(false) }
|
||||
}
|
||||
const selected = catalog?.providers.find(provider => provider.id === catalog.selection.providerId)
|
||||
const languageNames: Readonly<Record<string, string>> = {
|
||||
auto: props.t('auto'), zh: props.t('zh'), en: props.t('en'), yue: props.t('yue'), ja: props.t('ja'), ko: props.t('ko'),
|
||||
}
|
||||
return <div>
|
||||
{catalog && <div className={css.preferences}>
|
||||
<label>{props.t('provider')}<select value={catalog.selection.providerId} disabled={!readiness.connected || saving}
|
||||
onChange={(event) => { void configure({ providerId: event.target.value as SpeechProviderId }) }}>
|
||||
{catalog.providers.map(provider => <option key={provider.id} value={provider.id}>{provider.name}</option>)}
|
||||
</select></label>
|
||||
<label>{props.t('language')}<select value={catalog.selection.language} disabled={!readiness.connected || saving}
|
||||
onChange={(event) => { void configure({ language: event.target.value }) }}>
|
||||
{selected?.languages.map(language =>
|
||||
<option key={language} value={language}>{languageNames[language] ?? language}</option>)}
|
||||
</select></label>
|
||||
<p>{props.t(selected?.location === 'cloud' ? 'cloud' : 'local')}</p>
|
||||
{error && <p role="alert">{props.t('failed', { message: error })}</p>}
|
||||
</div>}
|
||||
{catalog?.providers.map(provider =>
|
||||
<PreparationCard key={provider.id} provider={provider} connected={readiness.connected} {...props} />)}
|
||||
{!catalog && <p role="status">{props.t('loading')}</p>}
|
||||
{readiness.error && <p role="alert">{props.t('failed', { message: readiness.error })}</p>}
|
||||
</div>
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
.trigger { width: 28px; padding: 0; flex: none; }
|
||||
.triggerAnchor { display: inline-flex; flex: none; }
|
||||
.captureRow { display: flex; align-items: center; gap: 12px; width: 100%; min-width: 0; min-height: 34px; }
|
||||
.roundButton { width: 32px; height: 32px; padding: 0; border-radius: 50%; corner-shape: round; flex: none; background: var(--dsw-specific-selector); }
|
||||
.roundButton:hover:not(:disabled) { background: var(--dsw-alias-interactive-bg-hover-solid); }
|
||||
.waveform { display: block; flex: 1; width: 0; height: 24px; min-width: 24px; color: var(--dsw-alias-label-secondary); }
|
||||
.activityMessage { display: flex; align-items: center; gap: 8px; flex: 1; min-width: 0; font-size: 12px; color: var(--dsw-alias-label-secondary); overflow: hidden; white-space: nowrap; text-overflow: ellipsis; }
|
||||
.inlineAction { flex: none; }
|
||||
|
||||
.preparation { display: flex; flex-direction: column; gap: 10px; padding: 14px; margin: 12px 0; border: 0.5px solid var(--dsw-alias-border-l1); border-radius: 10px; font-size: 13px; }
|
||||
.preparation p { margin: 0; }
|
||||
.preparationActions { display: flex; gap: 8px; }
|
||||
.preparationActions:empty { display: none; }
|
||||
.preparationError { color: var(--dsw-alias-state-error-primary); overflow-wrap: anywhere; font-size: 12px; }
|
||||
.summaryRow { min-height: 28px; height: auto; flex-wrap: wrap; gap: 4px; }
|
||||
.summaryTitle { flex: 1; min-width: 0; }
|
||||
.metric { font-size: 12px; color: var(--dsw-alias-label-tertiary); }
|
||||
.progress { width: 100%; height: 6px; accent-color: var(--dsw-alias-state-business-primary); }
|
||||
.steps { list-style: none; display: flex; flex-direction: column; gap: 14px; padding: 16px 0 2px; margin: 0; }
|
||||
.steps li { display: flex; align-items: flex-start; gap: 10px; line-height: 18px; }
|
||||
.steps li > :first-child { margin-top: 1px; }
|
||||
.steps li[data-step-state='pending'] { color: var(--dsw-alias-label-tertiary); }
|
||||
.stepBody { flex: 1; min-width: 0; }
|
||||
.stepProgress { display: flex; flex-direction: column; gap: 6px; margin-top: 6px; }
|
||||
.stepProgress small { font-size: 11px; color: var(--dsw-alias-label-tertiary); overflow-wrap: anywhere; }
|
||||
.preferences { display: grid; gap: 12px; font-size: 13px; margin: 18px 0; }
|
||||
.preferences label { display: grid; grid-template-columns: 100px minmax(120px, 280px); align-items: center; gap: 12px; }
|
||||
.preferences select { border: 0.5px solid var(--dsw-alias-border-l1); border-radius: 6px; padding: 6px 10px; background: transparent; color: inherit; font: inherit; }
|
||||
.preferences p { margin: 0; }
|
||||
.settingsCard { list-style: none; padding: 16px; border: 0.5px solid var(--dsw-alias-border-l1); border-radius: 10px; }
|
||||
.settingsCard > p { font-size: 13px; color: var(--dsw-alias-label-secondary); margin: 8px 0 0; }
|
||||
.setupEstimate { display: grid; gap: 10px; padding: 12px; border-radius: 8px; background: var(--dsw-alias-interactive-bg-hover); color: var(--dsw-alias-label-secondary); line-height: 1.6; }
|
||||
.setupEstimate dl { display: grid; gap: 6px; margin: 0; }
|
||||
.setupEstimate dl > div { display: grid; grid-template-columns: 72px minmax(0, 1fr); gap: 10px; }
|
||||
.setupEstimate dt { color: var(--dsw-alias-label-tertiary); }
|
||||
.setupEstimate dd { margin: 0; }
|
||||
.setupEstimate small { color: var(--dsw-alias-label-tertiary); font-size: 12px; }
|
||||
@@ -0,0 +1,162 @@
|
||||
/** Click-to-record toolbar activity; transcripts remain in the original Session draft. */
|
||||
import { useEffect, useLayoutEffect, useRef, useState } from 'react'
|
||||
import type { HostObservable, InjectFace, PropsLocale, PropsRuntime } from '@deepseek-ai/dsh-client-ui-slots'
|
||||
import type { TokenSpan } from '@deepseek-ai/dsh-client-ui-conversation/client'
|
||||
import type { TranscriptionRequest } from '@deepseek-ai/dsh-experimental-api-speech-to-text/types'
|
||||
import type { SpeechProviderId, SpeechSelection, SpeechSelectionPatch, Transcript } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import type { RemoteResult } from '@deepseek-ai/dsh-typert-protocol'
|
||||
import { RecordingError, audioBase64, type Recording } from './audio.ts'
|
||||
import type { SpeechReadiness } from './readiness.ts'
|
||||
import { Waveform } from './Waveform.tsx'
|
||||
import { NS } from './locales.ts'
|
||||
import { Button, IconCloseOutlineRegular, IconStopFillRegular, IconMicrophoneOutlineRegular, StateDot, Tooltip } from '@deepseek-ai/dsh-client-ui-primitives'
|
||||
import css from './VoiceInput.module.css'
|
||||
|
||||
/** Host calls injected without exposing a Cordis Context to React. */
|
||||
export interface VoiceInputActions {
|
||||
/** @returns one microphone operation owned by the plugin lifecycle. */
|
||||
createRecording: () => Recording
|
||||
transcribe: (request: TranscriptionRequest, signal: AbortSignal) => Promise<RemoteResult<Transcript>>
|
||||
prepare: (providerId: SpeechProviderId) => Promise<void>
|
||||
cancelPreparation: (providerId: SpeechProviderId) => Promise<void>
|
||||
configure: (patch: SpeechSelectionPatch) => Promise<void>
|
||||
}
|
||||
|
||||
/** Entry-injected Host readiness and microphone operations. */
|
||||
export interface VoiceInputInjected extends VoiceInputActions {
|
||||
hooks: { speechReadiness: HostObservable<SpeechReadiness> }
|
||||
}
|
||||
|
||||
/** Composer-owned expansion and draft actions; preferences stay in plugin settings. */
|
||||
export type VoiceInputProps = Pick<PropsRuntime<'conversation.input.activity'>, 'sessionId' | 'inputActions' | 'locked' | 'onActiveChange'>
|
||||
& PropsLocale<typeof NS> & InjectFace<VoiceInputInjected>
|
||||
|
||||
type Phase = 'idle' | 'requesting' | 'recording' | 'transcribing' | 'feedback'
|
||||
interface ActiveRecording {
|
||||
readonly capture: Recording
|
||||
readonly abort: AbortController
|
||||
readonly span: TokenSpan
|
||||
readonly selection: SpeechSelection
|
||||
readonly maxDurationSeconds: number
|
||||
readonly maxAudioBytes: number
|
||||
phase: 'requesting' | 'recording' | 'transcribing'
|
||||
timer?: ReturnType<typeof setTimeout> | undefined
|
||||
}
|
||||
|
||||
async function disposeRecording(capture: Recording): Promise<void> {
|
||||
try { await capture.dispose() } catch (_error) {
|
||||
// Tracks stop before AudioContext.close(); a close failure must not hide the recording result.
|
||||
}
|
||||
}
|
||||
|
||||
/** Render a compact microphone or an expanded capture, transcription, or retry row. */
|
||||
export function VoiceInput({ sessionId, inputActions, locked, onActiveChange,
|
||||
createRecording, transcribe, useSpeechReadiness, t }: VoiceInputProps) {
|
||||
const readiness = useSpeechReadiness(value => value), catalog = readiness.catalog
|
||||
const provider = catalog?.providers.find(item => item.id === catalog.selection.providerId)
|
||||
const usable = readiness.connected && (provider?.preparation.phase === 'ready' || provider?.preparation.phase === 'standby'
|
||||
|| provider?.preparation.phase === 'waking')
|
||||
const [phase, setPhase] = useState<Phase>('idle'), [message, setMessage] = useState(''), [pending, setPending] = useState('')
|
||||
const current = useRef<ActiveRecording>(), generation = useRef(0)
|
||||
const expanded = phase !== 'idle'
|
||||
useLayoutEffect(() => { onActiveChange(expanded); return () => { onActiveChange(false) } }, [expanded, onActiveChange])
|
||||
|
||||
const cancel = (): void => {
|
||||
generation.current++
|
||||
const active = current.current
|
||||
current.current = undefined
|
||||
if (active) { clearTimeout(active.timer); active.abort.abort(); void disposeRecording(active.capture) }
|
||||
setPending(''); setMessage(''); setPhase('idle')
|
||||
}
|
||||
useEffect(() => {
|
||||
setPending(''); setMessage(''); setPhase('idle')
|
||||
const blur = (): void => { if (current.current?.phase === 'recording') cancel() }
|
||||
const visibility = (): void => {
|
||||
if (document.hidden && current.current && current.current.phase !== 'transcribing') cancel()
|
||||
}
|
||||
const escape = (event: KeyboardEvent): void => {
|
||||
if (event.key === 'Escape' && current.current) { event.preventDefault(); cancel() }
|
||||
}
|
||||
window.addEventListener('blur', blur); document.addEventListener('visibilitychange', visibility)
|
||||
document.addEventListener('keydown', escape)
|
||||
return () => {
|
||||
window.removeEventListener('blur', blur); document.removeEventListener('visibilitychange', visibility)
|
||||
document.removeEventListener('keydown', escape)
|
||||
generation.current++
|
||||
const active = current.current
|
||||
current.current = undefined
|
||||
if (active) { clearTimeout(active.timer); active.abort.abort(); void disposeRecording(active.capture) }
|
||||
}
|
||||
}, [sessionId])
|
||||
|
||||
const feedback = (text: string): void => { setMessage(text); setPhase('feedback') }
|
||||
const failureText = (failure: unknown): string => failure instanceof RecordingError ? t(failure.kind)
|
||||
: t('failed', { message: failure instanceof Error ? failure.message : String(failure) })
|
||||
const finish = async (): Promise<void> => {
|
||||
const active = current.current
|
||||
if (!active || active.phase !== 'recording') return
|
||||
active.phase = 'transcribing'
|
||||
const run = generation.current
|
||||
clearTimeout(active.timer); setPhase('transcribing')
|
||||
try {
|
||||
const audio = await active.capture.stop(active.maxDurationSeconds)
|
||||
if (run !== generation.current) return
|
||||
if (audio.byteLength > active.maxAudioBytes) { feedback(t('tooLarge')); return }
|
||||
const result = await transcribe({ audioBase64: audioBase64(audio), ...active.selection }, active.abort.signal)
|
||||
if (run !== generation.current) return
|
||||
if (!result.ok) { feedback(t('failed', { message: result.error.message })); return }
|
||||
if (result.value.text === '') { feedback(t('empty')); return }
|
||||
if (!inputActions.insertText(result.value.text, active.span)) { setPending(result.value.text); feedback(t('conflict')); return }
|
||||
setPhase('idle')
|
||||
} catch (failure) {
|
||||
await disposeRecording(active.capture)
|
||||
if (run === generation.current) feedback(failureText(failure))
|
||||
} finally { if (run === generation.current) current.current = undefined }
|
||||
}
|
||||
const start = async (): Promise<void> => {
|
||||
if (!catalog || !usable || locked || current.current) return
|
||||
const run = ++generation.current
|
||||
const active: ActiveRecording = { capture: createRecording(), abort: new AbortController(),
|
||||
span: inputActions.captureInsertion(), selection: catalog.selection,
|
||||
maxDurationSeconds: catalog.maxDurationSeconds, maxAudioBytes: catalog.maxAudioBytes, phase: 'requesting' }
|
||||
current.current = active
|
||||
setMessage(''); setPending(''); setPhase('requesting')
|
||||
try {
|
||||
await active.capture.start((failure) => {
|
||||
if (run !== generation.current || current.current !== active || active.phase === 'transcribing') return
|
||||
current.current = undefined
|
||||
clearTimeout(active.timer); active.abort.abort()
|
||||
feedback(failureText(failure))
|
||||
})
|
||||
if (run !== generation.current || current.current !== active) return
|
||||
active.phase = 'recording'
|
||||
setPhase('recording')
|
||||
active.timer = setTimeout(() => { void finish() }, active.maxDurationSeconds * 1000)
|
||||
} catch (failure) {
|
||||
await disposeRecording(active.capture)
|
||||
if (run === generation.current) { current.current = undefined; feedback(failureText(failure)) }
|
||||
}
|
||||
}
|
||||
if (!expanded) return <Tooltip label={t(usable ? 'dictate' : 'prepareRequired')} side="top" portal>
|
||||
<span className={css.triggerAnchor}><Button className={css.trigger} size="sm" disabled={!usable || locked}
|
||||
aria-label={t('start')} onMouseDown={(event) => { event.preventDefault() }}
|
||||
onClick={() => { void start() }}><IconMicrophoneOutlineRegular size={18} /></Button></span>
|
||||
</Tooltip>
|
||||
return <div className={css.captureRow} data-voice-activity={phase}>
|
||||
<Button type="button" className={css.roundButton} size="sm" aria-label={t(pending ? 'discard' : 'cancel')}
|
||||
onClick={cancel}><IconCloseOutlineRegular size={14} /></Button>
|
||||
{phase === 'recording' ? <Waveform recording={current.current?.capture} label={t('recording')} />
|
||||
: <span className={css.activityMessage} role="status" title={pending || message}>
|
||||
{(phase === 'requesting' || phase === 'transcribing') && <StateDot state="ongoing" />}
|
||||
{phase === 'feedback' ? message : t(phase === 'requesting' ? 'requesting'
|
||||
: provider?.preparation.phase === 'waking' ? 'wakingShort' : 'transcribingShort')}</span>}
|
||||
{phase === 'recording' && <Button type="button" className={css.roundButton} size="sm" aria-label={t('stop')}
|
||||
onClick={() => { void finish() }}><IconStopFillRegular size={14} /></Button>}
|
||||
{phase === 'feedback' && (pending
|
||||
? <Button className={css.inlineAction} size="sm" type="button" onClick={() => {
|
||||
if (inputActions.insertText(pending, inputActions.captureInsertion())) { setPending(''); setPhase('idle') }
|
||||
}}>{t('insert')}</Button>
|
||||
: <Button className={css.roundButton} size="sm" type="button" aria-label={t('retryRecording')} disabled={!usable || locked}
|
||||
onClick={() => { void start() }}><IconMicrophoneOutlineRegular size={18} /></Button>)}
|
||||
</div>
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
/** First-use guidance after explicit enablement, once the Host finishes inspecting its cache. */
|
||||
import { useEffect } from 'react'
|
||||
import { Button, Modal } from '@deepseek-ai/dsh-client-ui-primitives'
|
||||
import type { InjectFace, PropsLocale, PropsRuntime } from '@deepseek-ai/dsh-client-ui-slots'
|
||||
import type {} from '@deepseek-ai/dsh-client-ui-plugin-manager/client'
|
||||
import type { VoiceInputInjected } from './VoiceInput.tsx'
|
||||
import type { NS } from './locales.ts'
|
||||
|
||||
/** Registration shares for the bundle's activation guidance. */
|
||||
export type VoiceSetupPromptProps = PropsRuntime<'plugins.bundle.activation'> & PropsLocale<typeof NS>
|
||||
& Pick<InjectFace<VoiceInputInjected>, 'useSpeechReadiness'>
|
||||
|
||||
/**
|
||||
* Offer navigation to installation without starting a download.
|
||||
* @param props - activation navigation and the shared Host readiness observer.
|
||||
* @returns the shared modal only when the selected local provider needs preparation.
|
||||
*/
|
||||
export function VoiceSetupPrompt({ useSpeechReadiness, onDismiss, onOpenDetails, t }: VoiceSetupPromptProps) {
|
||||
const readiness = useSpeechReadiness(value => value)
|
||||
const provider = readiness.catalog?.providers.find(item => item.id === readiness.catalog?.selection.providerId)
|
||||
const phase = readiness.connected ? provider?.preparation.phase : undefined
|
||||
const needsSetup = phase === 'unprepared' && provider?.location === 'host-local'
|
||||
useEffect(() => {
|
||||
if (phase !== undefined && phase !== 'checking' && !needsSetup) onDismiss()
|
||||
}, [phase, needsSetup, onDismiss])
|
||||
return <Modal open={needsSetup} title={t('setupPrompt.title')} closeLabel={t('cancel')} onClose={onDismiss}
|
||||
footer={<>
|
||||
<Button variant="ghost" onClick={onDismiss}>{t('setupPrompt.later')}</Button>
|
||||
<Button variant="primary" onClick={onOpenDetails}>{t('setupPrompt.open')}</Button>
|
||||
</>}>
|
||||
<p>{t('setupPrompt.body')}</p>
|
||||
</Modal>
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
/** Live microphone amplitude history; animation updates SVG geometry without React state churn. */
|
||||
import { useEffect, useRef } from 'react'
|
||||
import type { Recording } from './audio.ts'
|
||||
import css from './VoiceInput.module.css'
|
||||
|
||||
/** Render recent measured microphone levels; silent audio remains a dotted baseline. */
|
||||
export function Waveform({ recording, label }: { readonly recording: Recording | undefined; readonly label: string }) {
|
||||
const svg = useRef<SVGSVGElement>(null)
|
||||
useEffect(() => {
|
||||
const bars = Array.from((svg.current as SVGSVGElement).querySelectorAll('line')).reverse()
|
||||
.map(element => ({ element, level: 0 }))
|
||||
let frame: number, previous = -Infinity
|
||||
const draw = (now: number): void => {
|
||||
if (now - previous >= 50) {
|
||||
previous = now
|
||||
let next = recording?.amplitude() ?? 0
|
||||
for (const bar of bars) {
|
||||
const previousLevel = bar.level
|
||||
bar.level = next; next = previousLevel
|
||||
const height = 1 + Math.min(1, bar.level * 5) * 17
|
||||
bar.element.setAttribute('y1', String(20 - height)); bar.element.setAttribute('y2', String(20 + height))
|
||||
}
|
||||
}
|
||||
frame = requestAnimationFrame(draw)
|
||||
}
|
||||
frame = requestAnimationFrame(draw)
|
||||
return () => { cancelAnimationFrame(frame) }
|
||||
}, [recording])
|
||||
return <svg ref={svg} className={css.waveform} viewBox="0 0 640 40" preserveAspectRatio="none" role="img" aria-label={label}>
|
||||
{Array.from({ length: 80 }, (_, index) => <line key={index} x1={index * 8 + 4} x2={index * 8 + 4}
|
||||
y1="19" y2="21" stroke="currentColor" strokeWidth="3" strokeLinecap="round" opacity={0.25 + index / 120} />)}
|
||||
</svg>
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
/** Browser-owned microphone capture and native Web Audio resampling. */
|
||||
|
||||
/** Capture failure whose message is localized by the caller. */
|
||||
export class RecordingError extends Error {
|
||||
constructor(readonly kind: 'unavailable' | 'permission' | 'empty' | 'cancelled' | 'interrupted') { super(kind); this.name = 'RecordingError' }
|
||||
}
|
||||
|
||||
/**
|
||||
* Encode mono floating-point samples as the canonical PCM16 WAV accepted by the Host.
|
||||
* @param samples - native-resampled 16 kHz mono samples.
|
||||
* @returns complete little-endian WAV bytes.
|
||||
*/
|
||||
export function encodeWave(samples: Float32Array): Uint8Array<ArrayBuffer> {
|
||||
const bytes = new Uint8Array(44 + samples.length * 2)
|
||||
const view = new DataView(bytes.buffer)
|
||||
const text = (at: number, value: string): void => { for (let i = 0; i < value.length; i++) bytes[at + i] = value.charCodeAt(i) }
|
||||
text(0, 'RIFF'); view.setUint32(4, bytes.length - 8, true); text(8, 'WAVE'); text(12, 'fmt ')
|
||||
view.setUint32(16, 16, true); view.setUint16(20, 1, true); view.setUint16(22, 1, true)
|
||||
view.setUint32(24, 16000, true); view.setUint32(28, 32000, true)
|
||||
view.setUint16(32, 2, true); view.setUint16(34, 16, true); text(36, 'data')
|
||||
view.setUint32(40, samples.length * 2, true)
|
||||
for (const [i, sample] of samples.entries()) {
|
||||
const value = Math.max(-1, Math.min(1, sample))
|
||||
view.setInt16(44 + i * 2, Math.round(value * (value < 0 ? 32768 : 32767)), true)
|
||||
}
|
||||
return bytes
|
||||
}
|
||||
|
||||
/**
|
||||
* Encode the binary recording for the existing JSON Remote carrier.
|
||||
* @param bytes - complete recording.
|
||||
* @returns base64 with no data URL prefix.
|
||||
*/
|
||||
export function audioBase64(bytes: Uint8Array): string {
|
||||
let text = ''
|
||||
for (let i = 0; i < bytes.length; i += 8192) text += String.fromCharCode(...bytes.subarray(i, i + 8192))
|
||||
return btoa(text)
|
||||
}
|
||||
|
||||
/** One microphone acquisition, including a permission prompt that may settle after cancellation. */
|
||||
export class Recording {
|
||||
private stream: MediaStream | undefined
|
||||
private recorder: MediaRecorder | undefined
|
||||
private context: AudioContext | undefined
|
||||
private analyser: AnalyserNode | undefined
|
||||
private samples = new Float32Array(256)
|
||||
private chunks: Blob[] = []
|
||||
private readonly lifetime = new AbortController()
|
||||
private disposal: Promise<void> | undefined
|
||||
|
||||
constructor(private readonly onDispose: () => void) {}
|
||||
|
||||
/**
|
||||
* Acquire the microphone for this recording.
|
||||
* @param onError - receives failures during capture, before asynchronous resource release finishes.
|
||||
* @returns after capture starts; a cancelled permission grant immediately releases its tracks.
|
||||
*/
|
||||
async start(onError?: (error: RecordingError) => void): Promise<void> {
|
||||
const devices = (navigator as Partial<Navigator>).mediaDevices
|
||||
if (!devices || typeof MediaRecorder === 'undefined') throw new RecordingError('unavailable')
|
||||
let stream: MediaStream
|
||||
try {
|
||||
stream = await devices.getUserMedia({ audio: { echoCancellation: true, noiseSuppression: true }, video: false })
|
||||
} catch (error) {
|
||||
if (error instanceof DOMException && error.name === 'NotAllowedError') throw new RecordingError('permission')
|
||||
throw error
|
||||
}
|
||||
if (this.lifetime.signal.aborted) { stream.getTracks().forEach((track) => { track.stop() }); throw new RecordingError('cancelled') }
|
||||
this.stream = stream
|
||||
try {
|
||||
this.context = new AudioContext()
|
||||
this.analyser = this.context.createAnalyser()
|
||||
this.analyser.fftSize = this.samples.length
|
||||
this.context.createMediaStreamSource(stream).connect(this.analyser)
|
||||
this.recorder = new MediaRecorder(stream)
|
||||
this.recorder.ondataavailable = (event) => { if (!this.lifetime.signal.aborted && event.data.size > 0) this.chunks.push(event.data) }
|
||||
this.recorder.onerror = () => {
|
||||
if (this.lifetime.signal.aborted) return
|
||||
void this.dispose().catch(() => undefined)
|
||||
try { onError?.(new RecordingError('interrupted')) } catch (error) {
|
||||
console.error('Speech recording error handler failed', error)
|
||||
}
|
||||
}
|
||||
this.recorder.start()
|
||||
} catch (error) { await this.dispose(); throw error }
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the live microphone signal.
|
||||
* @returns the measured RMS level, or zero outside capture.
|
||||
*/
|
||||
amplitude(): number {
|
||||
if (!this.analyser) return 0
|
||||
this.analyser.getFloatTimeDomainData(this.samples)
|
||||
let sum = 0
|
||||
for (const sample of this.samples) sum += sample * sample
|
||||
return Math.sqrt(sum / this.samples.length)
|
||||
}
|
||||
|
||||
/**
|
||||
* Finish capture and resample the recording.
|
||||
* @param maxDurationSeconds - truncate timer overshoot to the Host limit.
|
||||
* @returns one recording after the final MediaRecorder chunk arrives.
|
||||
*/
|
||||
async stop(maxDurationSeconds: number): Promise<Uint8Array<ArrayBuffer>> {
|
||||
const recorder = this.recorder
|
||||
const context = this.context
|
||||
if (!recorder || !context || recorder.state !== 'recording') { await this.dispose(); throw new RecordingError('empty') }
|
||||
try {
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
recorder.onstop = () => { resolve() }
|
||||
recorder.onerror = () => { reject(new RecordingError('empty')) }
|
||||
recorder.stop()
|
||||
})
|
||||
this.stream?.getTracks().forEach((track) => { track.stop() })
|
||||
this.lifetime.signal.throwIfAborted()
|
||||
const blob = new Blob(this.chunks, { type: recorder.mimeType })
|
||||
if (blob.size === 0) throw new RecordingError('empty')
|
||||
const decoded = await context.decodeAudioData(await blob.arrayBuffer())
|
||||
this.lifetime.signal.throwIfAborted()
|
||||
const offline = new OfflineAudioContext(1, Math.max(1, Math.floor(Math.min(decoded.duration, maxDurationSeconds) * 16000)), 16000)
|
||||
const source = offline.createBufferSource()
|
||||
source.buffer = decoded
|
||||
source.connect(offline.destination)
|
||||
source.start()
|
||||
const resampled = await offline.startRendering()
|
||||
this.lifetime.signal.throwIfAborted()
|
||||
return encodeWave(resampled.getChannelData(0))
|
||||
} finally { await this.dispose() }
|
||||
}
|
||||
|
||||
/**
|
||||
* Release this recording and invalidate pending permission grants.
|
||||
* @returns the shared release promise, including any AudioContext close failure.
|
||||
*/
|
||||
dispose(): Promise<void> {
|
||||
if (!this.disposal) {
|
||||
const closing = Promise.withResolvers<void>()
|
||||
this.disposal = closing.promise
|
||||
void this.release().then(closing.resolve, closing.reject)
|
||||
}
|
||||
return this.disposal
|
||||
}
|
||||
|
||||
private async release(): Promise<void> {
|
||||
this.lifetime.abort(new RecordingError('cancelled'))
|
||||
if (this.recorder?.state === 'recording') this.recorder.stop()
|
||||
this.stream?.getTracks().forEach((track) => { track.stop() })
|
||||
this.stream = undefined
|
||||
const context = this.context
|
||||
this.context = undefined
|
||||
this.analyser = undefined
|
||||
this.chunks = []
|
||||
try { if (context && context.state !== 'closed') await context.close() }
|
||||
finally { this.onDispose() }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
/** Browser entry for the optional speech Remote contribution and composer control. */
|
||||
import type { Context } from '@deepseek-ai/cordis'
|
||||
import speechRemote from '@deepseek-ai/dsh-experimental-api-speech-to-text/remote'
|
||||
import { mountVoiceInput } from './mount.ts'
|
||||
|
||||
export { inject } from './mount.ts'
|
||||
|
||||
/**
|
||||
* Activate the experimental microphone contribution.
|
||||
* @param ctx - Client runtime.
|
||||
* @returns complete UI and Remote disposer.
|
||||
*/
|
||||
export async function apply(ctx: Context): Promise<() => Promise<void>> {
|
||||
return await mountVoiceInput(ctx, speechRemote)
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
/** Locale-owned copy for experimental voice input. */
|
||||
import type {} from '@deepseek-ai/dsh-client-ui-slots'
|
||||
/** Dictionary namespace for every voice control. */
|
||||
export const NS = 'voice-input'
|
||||
|
||||
/** Chinese dictionary and key source. */
|
||||
export const zh = {
|
||||
dictate: '听写',
|
||||
'setupPrompt.title': '使用语音输入前需要安装',
|
||||
'setupPrompt.body': '语音输入插件已开启。首次使用需要在本机下载并准备识别模型,请前往插件详情页查看空间、内存和时间说明,再开始安装。',
|
||||
'setupPrompt.later': '稍后',
|
||||
'setupPrompt.open': '前往安装',
|
||||
'setup.local': '将在运行 DSH 的机器上下载本地模型,无需安装 Python 或编译工具。',
|
||||
'setup.disk': '硬盘空间',
|
||||
'setup.diskValue': '建议预留约 {gb} GB,包含模型、运行时和下载缓存',
|
||||
'setup.memory': '运行内存',
|
||||
'setup.memoryValue': '模型加载后约 {gb} GB,识别时可能更高',
|
||||
'setup.time': '首次准备',
|
||||
'setup.timeValue': '参考 {min}–{max} 分钟,取决于网络和机器性能',
|
||||
'setup.estimateNote': '以上为估算;网络较慢时可能更久。已下载的资源会复用,默认在闲置后释放模型内存。',
|
||||
'step.check': '检查本地资源',
|
||||
'step.model': '准备语音识别模型',
|
||||
'step.vad': '准备语音检测模型',
|
||||
'step.verify': '校验模型文件',
|
||||
'step.load': '加载 {name}',
|
||||
'stepStatus.pending': '未开始',
|
||||
'stepStatus.running': '进行中',
|
||||
'stepStatus.complete': '已完成',
|
||||
'stepStatus.failed': '失败',
|
||||
'stepStatus.cancelled': '已取消',
|
||||
'short.downloading': '下载中',
|
||||
'short.failed': '准备失败',
|
||||
retryRecording: '重新录音',
|
||||
preparationSteps: '准备步骤',
|
||||
'prepare': '下载并准备',
|
||||
'retryPrepare': '重试准备',
|
||||
'cancelPrepare': '取消准备',
|
||||
'downloadProgress': '下载进度',
|
||||
'downloadBytes': '下载:{completed} / {total} MB,{percent}%',
|
||||
'downloadUnknown': '已下载 {completed} MB',
|
||||
'elapsed': '已等待 {seconds} 秒',
|
||||
'preparationFailed': '准备失败:{message}',
|
||||
'reconnecting': '正在连接语音服务…',
|
||||
'preparation.unprepared': '首次使用需要下载本地识别模型。',
|
||||
'preparation.checking': '正在检查本地资源…',
|
||||
'preparation.loading': '正在加载 {name}…',
|
||||
'preparation.waking': '正在唤醒本地语音…',
|
||||
'preparation.ready': '本地语音已就绪。',
|
||||
'preparation.standby': '本地资源已准备,录音时自动唤醒。',
|
||||
'preparation.cancelled': '准备已取消,已完成的下载会保留。',
|
||||
'preparation.cancelling': '正在取消准备…',
|
||||
|
||||
cloudReady: '云端语音识别已就绪。',
|
||||
prepareRequired: '请先在语音插件设置中准备识别服务',
|
||||
discard: '丢弃识别文字',
|
||||
transcribingShort: '识别中…', wakingShort: '唤醒中…',
|
||||
provider: '识别服务', language: '识别语言',
|
||||
auto: '自动识别', zh: '中文', en: '英语', yue: '粤语', ja: '日语', ko: '韩语',
|
||||
start: '开始录音', stop: '停止并识别', cancel: '取消', insert: '插入文字',
|
||||
loading: '正在读取识别服务…', requesting: '请允许使用麦克风…', recording: '正在录音…',
|
||||
interrupted: '录音中断,请重试。',
|
||||
local: '音频在运行 DSH 的机器上识别。', cloud: '音频将发送至所选云端服务。',
|
||||
empty: '未识别到语音', cancelled: '已取消语音输入。',
|
||||
conflict: '草稿已被修改。识别文字已保留,可在当前光标位置插入。',
|
||||
failed: '语音识别失败:{message}',
|
||||
unavailable: '当前浏览器不支持录音,请使用支持麦克风的浏览器。',
|
||||
permission: '麦克风权限未开启,请在浏览器和系统设置中允许访问。',
|
||||
tooLarge: '录音超过服务限制,请缩短录音后重试。',
|
||||
} as const
|
||||
|
||||
/** Keys accepted by the voice input translator. */
|
||||
export type VoiceKey = keyof typeof zh
|
||||
|
||||
/** English dictionary with the same complete key set. */
|
||||
export const en: Record<VoiceKey, string> = {
|
||||
dictate: 'Dictate',
|
||||
'setupPrompt.title': 'Set up voice input before recording',
|
||||
'setupPrompt.body': 'Voice input is enabled. First use requires downloading and preparing local recognition models. Open the plugin details to review disk space, memory and time estimates before starting setup.',
|
||||
'setupPrompt.later': 'Later',
|
||||
'setupPrompt.open': 'Go to setup',
|
||||
'setup.local': 'Local models will be downloaded to the machine running DSH. No Python or compiler is required.',
|
||||
'setup.disk': 'Disk space',
|
||||
'setup.diskValue': 'Allow about {gb} GB for models, runtime, and download caches',
|
||||
'setup.memory': 'Memory',
|
||||
'setup.memoryValue': 'About {gb} GB with the model loaded; recognition may use more',
|
||||
'setup.time': 'First setup',
|
||||
'setup.timeValue': 'Allow roughly {min}–{max} minutes, depending on the network and machine',
|
||||
'setup.estimateNote': 'These are estimates; slow networks may take longer. Downloads are reused; idle models release memory by default.',
|
||||
'step.check': 'Check local resources',
|
||||
'step.model': 'Prepare recognition model',
|
||||
'step.vad': 'Prepare voice detection model',
|
||||
'step.verify': 'Verify model files',
|
||||
'step.load': 'Load {name}',
|
||||
'stepStatus.pending': 'Not started',
|
||||
'stepStatus.running': 'In progress',
|
||||
'stepStatus.complete': 'Completed',
|
||||
'stepStatus.failed': 'Failed',
|
||||
'stepStatus.cancelled': 'Cancelled',
|
||||
'short.downloading': 'Downloading',
|
||||
'short.failed': 'Setup failed',
|
||||
retryRecording: 'Record again',
|
||||
preparationSteps: 'Preparation steps',
|
||||
'prepare': 'Download and prepare',
|
||||
'retryPrepare': 'Retry preparation',
|
||||
'cancelPrepare': 'Cancel preparation',
|
||||
'downloadProgress': 'Download progress',
|
||||
'downloadBytes': 'Download: {completed} / {total} MB, {percent}%',
|
||||
'downloadUnknown': 'Downloaded {completed} MB',
|
||||
'elapsed': 'Waiting {seconds} seconds',
|
||||
'preparationFailed': 'Preparation failed: {message}',
|
||||
'reconnecting': 'Connecting to speech recognition…',
|
||||
'preparation.unprepared': 'First use downloads the local recognition models.',
|
||||
'preparation.checking': 'Checking local resources…',
|
||||
'preparation.loading': 'Loading {name}…',
|
||||
'preparation.waking': 'Waking local speech recognition…',
|
||||
'preparation.ready': 'Local speech recognition is ready.',
|
||||
'preparation.standby': 'Local resources are prepared. Recording wakes the recognizer.',
|
||||
'preparation.cancelled': 'Preparation cancelled. Completed downloads are retained.',
|
||||
'preparation.cancelling': 'Cancelling preparation…',
|
||||
|
||||
cloudReady: 'Cloud speech recognition is ready.',
|
||||
prepareRequired: 'Prepare recognition in the voice plugin settings first',
|
||||
discard: 'Discard transcript',
|
||||
transcribingShort: 'Transcribing…', wakingShort: 'Waking…',
|
||||
provider: 'Recognition service', language: 'Recognition language',
|
||||
auto: 'Detect automatically', zh: 'Chinese', en: 'English', yue: 'Cantonese', ja: 'Japanese', ko: 'Korean',
|
||||
start: 'Start recording', stop: 'Stop and transcribe', cancel: 'Cancel', insert: 'Insert text',
|
||||
loading: 'Loading recognition services…', requesting: 'Allow microphone access to continue…', recording: 'Recording…',
|
||||
interrupted: 'Recording was interrupted. Please try again.',
|
||||
local: 'Audio is recognized on the machine running DSH.', cloud: 'Audio will be sent to the selected cloud service.',
|
||||
empty: 'No speech recognized', cancelled: 'Voice input cancelled.',
|
||||
conflict: 'Your draft changed. The transcript is preserved and can be inserted at the current cursor.',
|
||||
failed: 'Speech recognition failed: {message}',
|
||||
unavailable: 'This browser cannot record audio. Use a browser with microphone support.',
|
||||
permission: 'Microphone access is disabled. Allow it in browser and system settings.',
|
||||
tooLarge: 'The recording exceeds the service limit. Try a shorter recording.',
|
||||
}
|
||||
|
||||
declare module '@deepseek-ai/dsh-client-ui-slots' {
|
||||
interface LocaleNamespaceMap {
|
||||
/** Experimental microphone and transcription controls. */
|
||||
'voice-input': VoiceKey
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
/** Source-safe lifecycle for the optional speech Remote and browser UI. */
|
||||
import type { Context } from '@deepseek-ai/cordis'
|
||||
import type {} from '@deepseek-ai/dsh-api-remotes/client'
|
||||
import type {} from '@deepseek-ai/dsh-experimental-api-speech-to-text/remote'
|
||||
import type {} from '@deepseek-ai/dsh-client-locale/client'
|
||||
import type {} from '@deepseek-ai/dsh-client-ui-renderer/client'
|
||||
import type {} from '@deepseek-ai/dsh-client-ui-conversation/client'
|
||||
import type { TypertRemoteContribution } from '@deepseek-ai/dsh-typert-protocol'
|
||||
import { VoiceInput, type VoiceInputInjected } from './VoiceInput.tsx'
|
||||
import { Recording } from './audio.ts'
|
||||
import { en, NS, zh } from './locales.ts'
|
||||
import type {} from '@deepseek-ai/dsh-client-ui-plugin-manager/client'
|
||||
import { observeReadiness } from './readiness.ts'
|
||||
import { VoicePreparation } from './PreparationCard.tsx'
|
||||
import { VoiceSetupPrompt } from './VoiceSetupPrompt.tsx'
|
||||
|
||||
export const inject = ['remote', 'slots', 'locale']
|
||||
|
||||
function registerUi(ctx: Context): void {
|
||||
ctx.effect(() => ctx.locale.register(NS, { zh, en }))
|
||||
const recordings = new Set<Recording>()
|
||||
const readiness = observeReadiness(ctx)
|
||||
ctx.effect(() => readiness.dispose)
|
||||
ctx.effect(() => async () => { await Promise.all([...recordings].map(recording => recording.dispose())) })
|
||||
const actions: VoiceInputInjected = {
|
||||
hooks: { speechReadiness: readiness.state },
|
||||
createRecording: () => {
|
||||
const recording = new Recording(() => { recordings.delete(recording) })
|
||||
recordings.add(recording)
|
||||
return recording
|
||||
},
|
||||
transcribe: async (request, signal) => await ctx.remote.speech.transcribe(request, signal),
|
||||
configure: async (patch) => { const result = await ctx.remote.speech.configure(patch); if (!result.ok) throw result.error },
|
||||
prepare: async (providerId) => { const result = await ctx.remote.speech.prepare(providerId); if (!result.ok) throw result.error },
|
||||
cancelPreparation: async (providerId) => {
|
||||
const result = await ctx.remote.speech.cancelPreparation(providerId); if (!result.ok) throw result.error
|
||||
},
|
||||
}
|
||||
ctx.slots.inject('conversation.input.activity', () => ctx.slots.register({
|
||||
name: 'conversation.input.activity', locale: NS, inject: () => actions,
|
||||
}, VoiceInput))
|
||||
ctx.slots.inject('plugins.bundle.config', () => ctx.slots.register({ name: 'plugins.bundle.config',
|
||||
key: '@deepseek-ai/dsh-experimental-voice-input-bundle', locale: NS, inject: () => actions,
|
||||
}, VoicePreparation))
|
||||
ctx.slots.inject('plugins.bundle.activation', () => ctx.slots.register({ name: 'plugins.bundle.activation',
|
||||
key: '@deepseek-ai/dsh-experimental-voice-input-bundle', locale: NS, inject: () => actions,
|
||||
}, VoiceSetupPrompt))
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
* Mount this experimental namespace without adding it to stable API Remotes.
|
||||
* @param ctx - Client runtime owning the Remote, dictionaries and slots.
|
||||
* @param contribution - generated speech Remote definitions.
|
||||
* @returns disposer joining UI and Remote withdrawal.
|
||||
*/
|
||||
export async function mountVoiceInput(ctx: Context, contribution: TypertRemoteContribution): Promise<() => Promise<void>> {
|
||||
const disposeRemote = await ctx.remote.$mount(contribution)
|
||||
const ui = ctx.inject(['remote.speech', 'slots', 'locale'], registerUi)
|
||||
try { await ui } catch (error) { await ui.dispose(); await disposeRemote(); throw error }
|
||||
return async () => { await ui.dispose(); await disposeRemote() }
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
/** One reconnecting Host readiness mirror shared by every voice UI occurrence. */
|
||||
import type { Context } from '@deepseek-ai/cordis'
|
||||
import { RemoteStreamCarrierError } from '@deepseek-ai/dsh-api-gateway/client'
|
||||
import { createSnapshotStore, type SnapshotStore } from '@deepseek-ai/dsh-client-store'
|
||||
import type { SpeechCatalog } from '@deepseek-ai/dsh-experimental-api-speech-to-text/types'
|
||||
import type {} from '@deepseek-ai/dsh-experimental-api-speech-to-text/remote'
|
||||
|
||||
/** Complete provider state, with a separate transport failure that never rewrites Host readiness. */
|
||||
export interface SpeechReadiness {
|
||||
readonly catalog: SpeechCatalog | null
|
||||
readonly connected: boolean
|
||||
readonly error: string | null
|
||||
}
|
||||
|
||||
/**
|
||||
* Subscribe once per Client plugin, independent of rendered pages and Sessions.
|
||||
* @param ctx - mounted speech Remote owner.
|
||||
* @returns shared readiness snapshot and joined observation cleanup.
|
||||
*/
|
||||
export function observeReadiness(ctx: Context): { state: SnapshotStore<SpeechReadiness>; dispose: () => Promise<void> } {
|
||||
const state = createSnapshotStore<SpeechReadiness>({ catalog: null, connected: false, error: null })
|
||||
let disposed = false
|
||||
const isDisposed = (): boolean => disposed
|
||||
const fail = (error: unknown): void => {
|
||||
state.set({ ...state.getSnapshot(), connected: false, error: error instanceof Error ? error.message : String(error) })
|
||||
}
|
||||
const stream = ctx.remote.$stream<SpeechCatalog>({
|
||||
name: 'Speech readiness', open: signal => ctx.remote.speech.follow(signal),
|
||||
ended: () => new RemoteStreamCarrierError('Speech readiness stream ended'), carrierFailed: fail,
|
||||
})
|
||||
const observing = (async () => {
|
||||
try {
|
||||
for await (const item of stream) {
|
||||
state.set({ catalog: item.value, connected: true, error: null })
|
||||
item.accept()
|
||||
}
|
||||
} catch (error) { if (!isDisposed()) fail(error) }
|
||||
})()
|
||||
return { state, dispose: async () => { disposed = true; await stream.dispose(); await observing } }
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
/** CSS module names emitted by the browser bundle. */
|
||||
declare module '*.module.css' {
|
||||
const classes: Record<string, string>
|
||||
export default classes
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
/** Host marker for the optional browser voice input contribution. */
|
||||
/** Browser behavior is provided by the client export. */
|
||||
export function apply(): void {}
|
||||
@@ -0,0 +1,48 @@
|
||||
/** Controlled browser audio devices with real Recording ownership and conversion. */
|
||||
import { vi } from 'vitest'
|
||||
import { Recording } from '../src/client/audio.ts'
|
||||
|
||||
/**
|
||||
* Install per-test audio devices; callers restore globals and release acquired recordings.
|
||||
* @param options - device failures or an empty final recording.
|
||||
* @returns the recording, device controls, and resource observations.
|
||||
*/
|
||||
export function captureFixture(options: { empty?: boolean; recorderError?: boolean; constructError?: boolean } = {}) {
|
||||
const trackStop = vi.fn(), close = vi.fn(async () => {}), disposed = vi.fn()
|
||||
const decoding = vi.fn(async (_data: ArrayBuffer) => ({ duration: 2 }))
|
||||
const rendering = vi.fn(async () => ({ getChannelData: () => new Float32Array([0.5, -0.5]) }))
|
||||
let failRecorder: () => void
|
||||
class Recorder {
|
||||
state = 'inactive'
|
||||
mimeType = 'audio/webm'
|
||||
ondataavailable?: (event: { data: Blob }) => void
|
||||
onstop?: () => void
|
||||
onerror?: () => void
|
||||
constructor() {
|
||||
if (options.constructError) throw new Error('recorder unavailable')
|
||||
failRecorder = () => { this.onerror!() }
|
||||
}
|
||||
start() { this.state = 'recording' }
|
||||
stop() {
|
||||
this.state = 'inactive'
|
||||
queueMicrotask(() => {
|
||||
this.ondataavailable?.({ data: new Blob() })
|
||||
if (!options.empty) this.ondataavailable?.({ data: new Blob(['final audio']) })
|
||||
if (options.recorderError) this.onerror?.()
|
||||
else this.onstop?.()
|
||||
})
|
||||
}
|
||||
}
|
||||
const offline = vi.fn(function Offline(_channels: number, _frames: number, _rate: number) {
|
||||
return { destination: {}, createBufferSource: () => ({ buffer: null, connect() {}, start() {} }), startRendering: rendering }
|
||||
})
|
||||
vi.stubGlobal('navigator', { mediaDevices: { getUserMedia: async () => ({ getTracks: () => [{ stop: trackStop }] }) } })
|
||||
vi.stubGlobal('MediaRecorder', Recorder)
|
||||
vi.stubGlobal('AudioContext', function Audio() { return { state: 'running', close, decodeAudioData: decoding,
|
||||
createMediaStreamSource: () => ({ connect: vi.fn() }),
|
||||
createAnalyser: () => ({ fftSize: 256, getFloatTimeDomainData: (buffer: Float32Array) => { buffer.fill(0.25) } }),
|
||||
} })
|
||||
vi.stubGlobal('OfflineAudioContext', offline)
|
||||
return { recording: new Recording(disposed), trackStop, close, disposed, decoding, rendering, offline,
|
||||
failRecorder: () => { failRecorder() } }
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
/** Browser recording bytes conform to the Host's fixed PCM format. */
|
||||
import { afterEach, expect, it, vi } from 'vitest'
|
||||
import { audioBase64, encodeWave, Recording, RecordingError } from '../src/client/audio.ts'
|
||||
import { captureFixture } from './audio-fixture.client.ts'
|
||||
|
||||
afterEach(() => { vi.unstubAllGlobals() })
|
||||
|
||||
it('encodes bounded PCM samples and base64 across chunk boundaries', () => {
|
||||
const bytes = encodeWave(new Float32Array([-2, -0.5, 0, 0.5, 2]))
|
||||
const view = new DataView(bytes.buffer)
|
||||
expect(new TextDecoder().decode(bytes.slice(0, 4))).toBe('RIFF')
|
||||
expect(view.getUint32(24, true)).toBe(16000)
|
||||
expect(view.getUint32(40, true)).toBe(10)
|
||||
expect([0, 1, 2, 3, 4].map(i => view.getInt16(44 + i * 2, true))).toEqual([-32768, -16384, 0, 16384, 32767])
|
||||
const large = encodeWave(new Float32Array(17000))
|
||||
expect(Buffer.from(audioBase64(large), 'base64')).toEqual(Buffer.from(large))
|
||||
})
|
||||
|
||||
it('reports unavailable recording and denied permission', async () => {
|
||||
vi.stubGlobal('navigator', {})
|
||||
const recording = new Recording(() => {})
|
||||
await expect(recording.start()).rejects.toMatchObject({ kind: 'unavailable' })
|
||||
vi.stubGlobal('MediaRecorder', function RecorderStub() {})
|
||||
vi.stubGlobal('navigator', { mediaDevices: { getUserMedia: async () => { throw new DOMException('denied', 'NotAllowedError') } } })
|
||||
await expect(recording.start()).rejects.toMatchObject({ kind: 'permission' })
|
||||
vi.stubGlobal('navigator', { mediaDevices: { getUserMedia: async () => { throw new Error('device lost') } } })
|
||||
await expect(recording.start()).rejects.toThrow('device lost')
|
||||
expect(new RecordingError('empty').message).toBe('empty')
|
||||
})
|
||||
|
||||
it('releases a microphone granted after cancellation', async () => {
|
||||
const permission = Promise.withResolvers<{ getTracks: () => { stop: () => void }[] }>()
|
||||
const stop = vi.fn(), dispose = vi.fn()
|
||||
vi.stubGlobal('MediaRecorder', function RecorderStub() {})
|
||||
vi.stubGlobal('navigator', { mediaDevices: { getUserMedia: () => permission.promise } })
|
||||
const recording = new Recording(dispose)
|
||||
const acquiring = recording.start()
|
||||
await recording.dispose()
|
||||
permission.resolve({ getTracks: () => [{ stop }] })
|
||||
await expect(acquiring).rejects.toMatchObject({ kind: 'cancelled' })
|
||||
expect(stop).toHaveBeenCalledOnce()
|
||||
expect(dispose).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
|
||||
it('flushes final recording bytes, bounds timer overshoot and closes audio resources', async () => {
|
||||
const b = captureFixture()
|
||||
expect(b.recording.amplitude()).toBe(0)
|
||||
await b.recording.start()
|
||||
expect(b.recording.amplitude()).toBe(0.25)
|
||||
const result = await b.recording.stop(1)
|
||||
expect(new TextDecoder().decode(b.decoding.mock.calls[0]![0])).toBe('final audio')
|
||||
expect(b.offline).toHaveBeenCalledWith(1, 16000, 16000)
|
||||
expect(result).toEqual(encodeWave(new Float32Array([0.5, -0.5])))
|
||||
expect(b.trackStop).toHaveBeenCalled()
|
||||
expect(b.close).toHaveBeenCalledOnce()
|
||||
await b.recording.dispose()
|
||||
expect(b.close).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it.each([{ empty: true }, { recorderError: true }, { constructError: true }])('releases failed capture resources: %j', async (options) => {
|
||||
const b = captureFixture(options)
|
||||
if (options.constructError) await expect(b.recording.start()).rejects.toThrow('recorder unavailable')
|
||||
else { await b.recording.start(); await expect(b.recording.stop(120)).rejects.toMatchObject({ kind: 'empty' }) }
|
||||
expect(b.trackStop).toHaveBeenCalled()
|
||||
expect(b.close).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('discards decoding results that finish after cancellation', async () => {
|
||||
const b = captureFixture(), decoded = Promise.withResolvers<{ duration: number }>()
|
||||
b.decoding.mockReturnValueOnce(decoded.promise)
|
||||
await b.recording.start()
|
||||
const stopping = b.recording.stop(120), rejected = expect(stopping).rejects.toMatchObject({ kind: 'cancelled' })
|
||||
await vi.waitFor(() => { expect(b.decoding).toHaveBeenCalledOnce() })
|
||||
await b.recording.dispose()
|
||||
decoded.resolve({ duration: 2 })
|
||||
await rejected
|
||||
expect(b.rendering).not.toHaveBeenCalled()
|
||||
await expect(b.recording.stop(120)).rejects.toMatchObject({ kind: 'empty' })
|
||||
})
|
||||
|
||||
it('releases active capture on a spontaneous recorder error even when AudioContext.close rejects', async () => {
|
||||
const b = captureFixture()
|
||||
await b.recording.start()
|
||||
b.close.mockRejectedValueOnce(new Error('audio device disconnected'))
|
||||
b.failRecorder()
|
||||
await vi.waitFor(() => { expect(b.disposed).toHaveBeenCalledOnce() })
|
||||
expect(b.trackStop).toHaveBeenCalled()
|
||||
await expect(b.recording.dispose()).rejects.toThrow('audio device disconnected')
|
||||
})
|
||||
|
||||
it.each([false, true])('shares pending AudioContext closure across repeated release (failure=%s)', async (failed) => {
|
||||
const b = captureFixture(), closing = Promise.withResolvers<undefined>()
|
||||
b.close.mockReturnValueOnce(closing.promise)
|
||||
await b.recording.start()
|
||||
const first = b.recording.dispose(), second = b.recording.dispose()
|
||||
const results = Promise.allSettled([first, second])
|
||||
try {
|
||||
expect(first).toBe(second)
|
||||
expect(b.trackStop).toHaveBeenCalledOnce()
|
||||
expect(b.close).toHaveBeenCalledOnce()
|
||||
expect(b.disposed).not.toHaveBeenCalled()
|
||||
} finally {
|
||||
if (failed) closing.reject(new Error('audio close failed'))
|
||||
else closing.resolve(undefined)
|
||||
await results
|
||||
}
|
||||
expect(b.disposed).toHaveBeenCalledOnce()
|
||||
expect(b.recording.dispose()).toBe(first)
|
||||
expect((await results).map(result => result.status)).toEqual(failed ? ['rejected', 'rejected'] : ['fulfilled', 'fulfilled'])
|
||||
})
|
||||
|
||||
it('reports a device interruption once while resource closure is still pending', async () => {
|
||||
const b = captureFixture(), closing = Promise.withResolvers<undefined>(), failure = vi.fn()
|
||||
b.close.mockReturnValueOnce(closing.promise)
|
||||
await b.recording.start(failure)
|
||||
try {
|
||||
b.failRecorder(); b.failRecorder()
|
||||
expect(failure).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ kind: 'interrupted' }))
|
||||
expect(b.trackStop).toHaveBeenCalledOnce()
|
||||
expect(b.disposed).not.toHaveBeenCalled()
|
||||
} finally { closing.resolve(undefined); await b.recording.dispose() }
|
||||
expect(b.disposed).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('contains a failing interruption callback and still releases the microphone', async () => {
|
||||
const b = captureFixture(), error = new Error('callback failed'), report = vi.spyOn(console, 'error').mockImplementation(() => {})
|
||||
try {
|
||||
await b.recording.start(() => { throw error })
|
||||
b.failRecorder()
|
||||
await b.recording.dispose()
|
||||
expect(b.trackStop).toHaveBeenCalledOnce()
|
||||
expect(b.disposed).toHaveBeenCalledOnce()
|
||||
expect(report).toHaveBeenCalledWith('Speech recording error handler failed', error)
|
||||
} finally { report.mockRestore(); await b.recording.dispose() }
|
||||
})
|
||||
@@ -0,0 +1,148 @@
|
||||
/** The optional namespace and microphone ownership follow Client plugin disposal. */
|
||||
import assert from 'node:assert/strict'
|
||||
import { Recording } from '../src/client/audio.ts'
|
||||
import { Context, Service } from '@deepseek-ai/cordis'
|
||||
import { LocaleRuntime } from '@deepseek-ai/dsh-client-locale/client'
|
||||
import { SlotRegistry } from '@deepseek-ai/dsh-client-ui-renderer/client'
|
||||
import type { SpeechProviderId } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import { RemoteError, type TypertRemoteContribution } from '@deepseek-ai/dsh-typert-protocol'
|
||||
import { expect, it, vi } from 'vitest'
|
||||
import { createSnapshotStore } from '@deepseek-ai/dsh-client-store'
|
||||
import { mountVoiceInput, inject } from '../src/client/mount.ts'
|
||||
import { apply as hostApply } from '../src/index.ts'
|
||||
import { VoiceInput, type VoiceInputInjected } from '../src/client/VoiceInput.tsx'
|
||||
import { captureFixture } from './audio-fixture.client.ts'
|
||||
|
||||
const REMOTE: TypertRemoteContribution = {
|
||||
package: '@deepseek-ai/dsh-experimental-api-speech-to-text',
|
||||
descriptors: [],
|
||||
}
|
||||
|
||||
vi.mock('../src/client/readiness.ts', () => ({ observeReadiness: () => ({
|
||||
state: createSnapshotStore({ catalog: null, connected: true, error: null }), dispose: async () => {},
|
||||
}) }))
|
||||
|
||||
/** Narrow the erased registry payload before exercising its registered actions. */
|
||||
function assertVoiceActions(value: Record<string, unknown>): asserts value is Record<string, unknown> & VoiceInputInjected {
|
||||
assert(typeof value.createRecording === 'function')
|
||||
assert(typeof value.configure === 'function')
|
||||
assert(typeof value.prepare === 'function')
|
||||
assert(typeof value.cancelPreparation === 'function')
|
||||
assert(typeof value.transcribe === 'function')
|
||||
assert(typeof value.hooks === 'object' && value.hooks !== null)
|
||||
}
|
||||
|
||||
async function fixture(fail = false) {
|
||||
const ctx = new Context(), unmount = vi.fn(async () => {})
|
||||
class Remote extends Service {
|
||||
constructor() { super(ctx, 'remote') }
|
||||
async $mount(contribution: TypertRemoteContribution) {
|
||||
expect(contribution).toBe(REMOTE)
|
||||
return unmount
|
||||
}
|
||||
}
|
||||
new Remote()
|
||||
const configure = vi.fn(async () => ({ ok: true, value: {} }))
|
||||
const prepare = vi.fn(async () => ({ ok: true, value: {} }))
|
||||
const cancelPreparation = vi.fn(async () => ({ ok: true, value: {} }))
|
||||
const transcribe = vi.fn(async () => ({ ok: true, value: {} }))
|
||||
ctx.provide('remote.speech', { configure, prepare, cancelPreparation, transcribe })
|
||||
ctx.provide('locale', new LocaleRuntime(ctx))
|
||||
await ctx.plugin(SlotRegistry)
|
||||
ctx.slots.register({ name: 'root', children: {
|
||||
'conversation.input.activity': { kind: 'single', scope: 'session' },
|
||||
'plugins.bundle.activation': { kind: 'keyed', scope: 'root' },
|
||||
'plugins.bundle.config': { kind: 'keyed', scope: 'root' },
|
||||
} } as never,
|
||||
() => null)
|
||||
if (fail) vi.spyOn(ctx.slots, 'inject').mockImplementationOnce(() => { throw new Error('slot failed') })
|
||||
return { ctx, unmount, configure, prepare, cancelPreparation, transcribe }
|
||||
}
|
||||
|
||||
it('withdraws its Remote, localized slot and microphone captures on disposal', async () => {
|
||||
hostApply()
|
||||
const b = await fixture()
|
||||
try {
|
||||
const fiber = b.ctx.plugin({ inject: [...inject], apply: ctx => mountVoiceInput(ctx, REMOTE) })
|
||||
await fiber
|
||||
const entry = b.ctx.slots.entries('conversation.input.activity').find(item => item.component === VoiceInput)
|
||||
expect(entry).toMatchObject({ locale: 'voice-input' })
|
||||
const actions = entry!.inject!()
|
||||
assertVoiceActions(actions)
|
||||
const finished = actions.createRecording()
|
||||
assert(finished instanceof Recording)
|
||||
await finished.dispose()
|
||||
const pending = actions.createRecording()
|
||||
assert(pending instanceof Recording)
|
||||
const dispose = vi.spyOn(pending, 'dispose')
|
||||
await actions.configure({ language: 'zh' })
|
||||
await actions.prepare('local' as SpeechProviderId)
|
||||
await actions.cancelPreparation('local' as SpeechProviderId)
|
||||
for (const slot of ['plugins.bundle.config', 'plugins.bundle.activation'] as const) {
|
||||
const item = b.ctx.slots.entries(slot)[0]!
|
||||
expect(item.locale).toBe('voice-input')
|
||||
const injected = item.inject!()
|
||||
expect(injected.hooks).toBe(actions.hooks)
|
||||
}
|
||||
const failure = { ok: false, error: new RemoteError('gateway/internal', 'offline', {}) }
|
||||
b.configure.mockResolvedValueOnce(failure as never)
|
||||
b.prepare.mockResolvedValueOnce(failure as never)
|
||||
b.cancelPreparation.mockResolvedValueOnce(failure as never)
|
||||
await expect(actions.configure({ language: 'en' })).rejects.toThrow('offline')
|
||||
await expect(actions.prepare('local' as SpeechProviderId)).rejects.toThrow('offline')
|
||||
await expect(actions.cancelPreparation('local' as SpeechProviderId)).rejects.toThrow('offline')
|
||||
const request = { providerId: 'local' as SpeechProviderId, audioBase64: '' }, signal = new AbortController().signal
|
||||
await actions.transcribe(request, signal)
|
||||
expect(b.configure).toHaveBeenCalledTimes(2)
|
||||
expect(b.transcribe).toHaveBeenCalledWith(request, signal)
|
||||
await fiber.dispose()
|
||||
expect(dispose).toHaveBeenCalledOnce()
|
||||
expect(b.ctx.slots.entries('conversation.input.activity')).toHaveLength(0)
|
||||
expect(b.ctx.slots.entries('plugins.bundle.activation')).toHaveLength(0)
|
||||
expect(b.unmount).toHaveBeenCalledOnce()
|
||||
} finally { await b.ctx.fiber.dispose() }
|
||||
})
|
||||
|
||||
it('rolls back the Remote contribution when the slot registration fails', async () => {
|
||||
const b = await fixture(true)
|
||||
try {
|
||||
await expect(mountVoiceInput(b.ctx, REMOTE)).rejects.toThrow('slot failed')
|
||||
expect(b.unmount).toHaveBeenCalledOnce()
|
||||
} finally { await b.ctx.fiber.dispose() }
|
||||
})
|
||||
|
||||
it('joins the same audio closure when cancellation overlaps Client plugin withdrawal', async () => {
|
||||
const b = await fixture(), audio = captureFixture(), closing = Promise.withResolvers<undefined>()
|
||||
audio.close.mockReturnValueOnce(closing.promise)
|
||||
let cancelled: Promise<void> | undefined, removed: Promise<void> | undefined
|
||||
try {
|
||||
const fiber = b.ctx.plugin({ inject: [...inject], apply: ctx => mountVoiceInput(ctx, REMOTE) })
|
||||
await fiber
|
||||
const entry = b.ctx.slots.entries('conversation.input.activity').find(item => item.component === VoiceInput)!
|
||||
const actions = entry.inject!()
|
||||
assertVoiceActions(actions)
|
||||
const recording = actions.createRecording()
|
||||
assert(recording instanceof Recording)
|
||||
await recording.start()
|
||||
cancelled = recording.dispose()
|
||||
const originalDispose = recording.dispose.bind(recording), joined = Promise.withResolvers<{ pending: Promise<void> }>()
|
||||
vi.spyOn(recording, 'dispose').mockImplementationOnce(() => {
|
||||
const pending = originalDispose()
|
||||
joined.resolve({ pending })
|
||||
return pending
|
||||
})
|
||||
removed = fiber.dispose()
|
||||
const { pending } = await joined.promise
|
||||
expect(pending).toBe(cancelled)
|
||||
expect(b.unmount).not.toHaveBeenCalled()
|
||||
closing.resolve(undefined)
|
||||
await cancelled; await removed
|
||||
expect(audio.close).toHaveBeenCalledOnce()
|
||||
expect(b.unmount).toHaveBeenCalledOnce()
|
||||
} finally {
|
||||
closing.resolve(undefined)
|
||||
try { await cancelled; await removed } finally {
|
||||
try { await b.ctx.fiber.dispose() } finally { vi.unstubAllGlobals() }
|
||||
}
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,133 @@
|
||||
// @vitest-environment jsdom
|
||||
/** Collapsed progress summaries and Host-owned steps survive view remounts. */
|
||||
import { act, cleanup, fireEvent, render, screen, waitFor, within } from '@testing-library/react'
|
||||
import { bindSnapshotSelector, makeTranslate } from '@deepseek-ai/dsh-client-test-runtime'
|
||||
import { createSnapshotStore } from '@deepseek-ai/dsh-client-store'
|
||||
import { zh as commonZh } from '@deepseek-ai/dsh-client-locale/src/locales/zh.ts'
|
||||
import type { SpeechPreparationState, SpeechPreparationStep, SpeechProviderId } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import { afterEach, expect, it, vi } from 'vitest'
|
||||
import { PreparationCard, VoicePreparation } from '../src/client/PreparationCard.tsx'
|
||||
import type { VoiceInputProps } from '../src/client/VoiceInput.tsx'
|
||||
import type { SpeechReadiness } from '../src/client/readiness.ts'
|
||||
import { zh } from '../src/client/locales.ts'
|
||||
|
||||
afterEach(() => { cleanup(); vi.useRealTimers() })
|
||||
const id = 'local' as SpeechProviderId, t = makeTranslate(zh, commonZh)
|
||||
const steps: SpeechPreparationStep[] = [
|
||||
{ kind: 'check', status: 'complete' }, { kind: 'model', status: 'complete' },
|
||||
{ kind: 'vad', status: 'complete' }, { kind: 'verify', status: 'running', startedAt: 0 }, { kind: 'load', status: 'pending' },
|
||||
]
|
||||
function fixture(preparation: SpeechPreparationState, connected = true) {
|
||||
const prepare = vi.fn(async (_id: SpeechProviderId) => {}), cancelPreparation = vi.fn(async (_id: SpeechProviderId) => {})
|
||||
const props = { provider: { id, name: 'SenseVoiceSmall', location: 'host-local' as const, languages: ['auto', 'zh', 'en', 'ja'], preparation },
|
||||
connected, prepare, cancelPreparation, t }
|
||||
return { ...render(<PreparationCard {...props} />), prepare, cancelPreparation, props }
|
||||
}
|
||||
it('collapses by default, expands all steps, and shows details only for the active step', async () => {
|
||||
vi.useFakeTimers(); vi.setSystemTime(5000)
|
||||
const b = fixture({ phase: 'checking', step: 'verify', startedAt: 0, steps })
|
||||
expect(screen.queryByRole('list')).toBeNull()
|
||||
expect(screen.getByText('已等待 5 秒')).toBeTruthy()
|
||||
expect(screen.queryByRole('progressbar')).toBeNull()
|
||||
fireEvent.click(screen.getByRole('button', { name: /校验模型文件/ }))
|
||||
expect(screen.getAllByRole('listitem')).toHaveLength(5)
|
||||
expect(b.container.querySelectorAll('[data-step-state="complete"] [data-state="done"]')).toHaveLength(3)
|
||||
expect(b.container.querySelectorAll('[data-step-state="pending"] [data-state="idle"]')).toHaveLength(1)
|
||||
expect(screen.getAllByText('已等待 5 秒')).toHaveLength(1)
|
||||
await act(async () => { await vi.advanceTimersByTimeAsync(1000) })
|
||||
expect(screen.getByText('已等待 6 秒')).toBeTruthy()
|
||||
b.unmount()
|
||||
fixture({ phase: 'checking', step: 'verify', startedAt: 0, steps })
|
||||
expect(screen.getByText('已等待 6 秒')).toBeTruthy()
|
||||
expect(screen.queryByRole('list')).toBeNull()
|
||||
})
|
||||
it('uses measured byte progress in both the collapsed summary and expanded active step', () => {
|
||||
const state: SpeechPreparationState = { phase: 'downloading', step: 'model', resource: 'model.pt',
|
||||
completedBytes: 638_000_000, totalBytes: 936_000_000,
|
||||
steps: steps.map(step => ({ ...step, status: step.kind === 'model' ? 'running' : step.kind === 'verify' ? 'complete' : step.status })),
|
||||
}
|
||||
const b = fixture(state)
|
||||
expect(screen.getByRole('status').textContent).toContain('638.0 / 936.0 MB,68%')
|
||||
expect(screen.getByRole('progressbar').getAttribute('value')).toBe('638000000')
|
||||
fireEvent.click(screen.getByRole('button', { name: /准备语音识别模型/ }))
|
||||
expect(screen.getByRole('progressbar').getAttribute('max')).toBe('936000000')
|
||||
expect(screen.getByText('model.pt')).toBeTruthy()
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.cancelPrepare }))
|
||||
expect(b.cancelPreparation).toHaveBeenCalledWith(id)
|
||||
})
|
||||
it('keeps unknown download totals indeterminate and disables commands while disconnected', () => {
|
||||
const b = fixture({ phase: 'downloading', resource: 'model.pt', completedBytes: 638_000_000 })
|
||||
expect(screen.queryByRole('progressbar')).toBeNull()
|
||||
expect(screen.getByRole('status').textContent).toBe('已下载 638.0 MB')
|
||||
b.rerender(<PreparationCard {...b.props} connected={false} />)
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.cancelPrepare }))
|
||||
expect(b.cancelPreparation).not.toHaveBeenCalled()
|
||||
expect(screen.getByText(zh.reconnecting)).toBeTruthy()
|
||||
})
|
||||
it('starts preparation and offers retry without showing progress for completed or pending steps', async () => {
|
||||
const b = fixture({ phase: 'unprepared', steps: steps.map(step => ({ kind: step.kind, status: 'pending' })) })
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.prepare }))
|
||||
expect(b.prepare).toHaveBeenCalledWith(id)
|
||||
b.rerender(<PreparationCard {...b.props} provider={{ ...b.props.provider,
|
||||
preparation: { phase: 'failed', message: 'network unavailable', steps: [{ kind: 'model', status: 'failed' }] } }} />)
|
||||
b.prepare.mockRejectedValueOnce(new Error('offline'))
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.retryPrepare }))
|
||||
await screen.findByText('语音识别失败:offline')
|
||||
b.prepare.mockRejectedValueOnce('network closed')
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.retryPrepare }))
|
||||
await screen.findByText('语音识别失败:network closed')
|
||||
fireEvent.click(screen.getByRole('button', { name: zh['short.failed'] }))
|
||||
expect(screen.queryByRole('progressbar')).toBeNull()
|
||||
})
|
||||
it('shows waking without an explicit preparation cancellation control', () => {
|
||||
fixture({ phase: 'waking', startedAt: Date.now() - 5000 })
|
||||
expect(screen.getByText('已等待 5 秒')).toBeTruthy()
|
||||
expect(screen.queryByRole('button', { name: zh.cancelPrepare })).toBeNull()
|
||||
})
|
||||
it('persists settings and reports disconnection in the detail card', async () => {
|
||||
const store = createSnapshotStore<SpeechReadiness>({ connected: true, error: null, catalog: {
|
||||
selection: { providerId: id, language: 'auto' }, maxAudioBytes: 100, maxDurationSeconds: 120,
|
||||
providers: [{ id, name: 'SenseVoiceSmall', location: 'host-local', languages: ['auto', 'zh', 'en', 'ja'], preparation: { phase: 'ready' } },
|
||||
{ id: 'cloud' as SpeechProviderId, name: 'Cloud', location: 'cloud', languages: ['auto', 'en', 'zh', 'ja', 'fr'], preparation: { phase: 'ready' } }],
|
||||
} })
|
||||
const configure = vi.fn<VoiceInputProps['configure']>(async () => {})
|
||||
const props = { useSpeechReadiness: bindSnapshotSelector(store), configure, t,
|
||||
prepare: vi.fn(async () => {}), cancelPreparation: vi.fn(async () => {}), transcribe: vi.fn(), createRecording: vi.fn() }
|
||||
const view = render(<VoicePreparation {...props} />)
|
||||
fireEvent.change(screen.getByLabelText(zh.provider), { target: { value: 'cloud' } })
|
||||
await waitFor(() => { expect(configure).toHaveBeenCalledWith({ providerId: 'cloud' }) })
|
||||
const current = store.getSnapshot()
|
||||
act(() => { store.set({ ...current, catalog: { ...current.catalog!, selection: { providerId: 'cloud' as SpeechProviderId, language: 'en' } } }) })
|
||||
expect(screen.getByText(zh.cloud)).toBeTruthy()
|
||||
expect(within(screen.getByLabelText(zh.language)).getAllByRole<HTMLOptionElement>('option').map(item => item.value))
|
||||
.toEqual(['auto', 'en', 'zh', 'ja', 'fr'])
|
||||
expect(within(screen.getByLabelText(zh.language)).getByRole('option', { name: 'fr' })).toBeTruthy()
|
||||
configure.mockRejectedValueOnce(new Error('settings are read-only'))
|
||||
fireEvent.change(screen.getByLabelText(zh.language), { target: { value: 'zh' } })
|
||||
await screen.findByText('语音识别失败:settings are read-only')
|
||||
configure.mockRejectedValueOnce('settings are locked')
|
||||
fireEvent.change(screen.getByLabelText(zh.language), { target: { value: 'ja' } })
|
||||
await screen.findByText('语音识别失败:settings are locked')
|
||||
act(() => { store.set({ catalog: null, connected: false, error: 'disconnected' }) })
|
||||
view.rerender(<VoicePreparation {...props} />)
|
||||
expect(screen.getByText(zh.loading)).toBeTruthy()
|
||||
expect(screen.getByRole('alert').textContent).toContain('disconnected')
|
||||
})
|
||||
it.each(['cancelled', 'standby', 'cancelling'] as const)('displays Host state %s', (phase) => {
|
||||
fixture(phase === 'cancelling' ? { phase, startedAt: Date.now() } : { phase })
|
||||
expect(screen.getByText(zh[`preparation.${phase}`])).toBeTruthy()
|
||||
})
|
||||
|
||||
it('shows provider-specific installation estimates before preparation and removes them while running', () => {
|
||||
const b = fixture({ phase: 'unprepared' })
|
||||
const provider = { ...b.props.provider, setupEstimate: {
|
||||
recommendedDiskBytes: 5_000_000_000, expectedMemoryBytes: 2_000_000_000, minimumMinutes: 5, maximumMinutes: 30,
|
||||
} }
|
||||
b.rerender(<PreparationCard {...b.props} provider={provider} />)
|
||||
expect(screen.getByText(zh['setup.local'])).toBeTruthy()
|
||||
expect(screen.getByText(/建议预留约 5 GB/)).toBeTruthy()
|
||||
expect(screen.getByText(/模型加载后约 2 GB/)).toBeTruthy()
|
||||
expect(screen.getByText(/参考 5–30 分钟/)).toBeTruthy()
|
||||
b.rerender(<PreparationCard {...b.props} provider={{ ...provider, preparation: { phase: 'checking', startedAt: Date.now() } }} />)
|
||||
expect(screen.queryByText(zh['setup.local'])).toBeNull()
|
||||
})
|
||||
@@ -0,0 +1,61 @@
|
||||
/** All voice views share one readiness subscription, including reconnect and terminal failure. */
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import type { RemoteStreamOptions } from '@deepseek-ai/dsh-api-gateway/client'
|
||||
import { RemoteStreamCarrierError } from '@deepseek-ai/dsh-api-gateway/client'
|
||||
import type { SpeechProviderId } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import type { SpeechCatalog } from '@deepseek-ai/dsh-experimental-api-speech-to-text/types'
|
||||
import { expect, it, vi, onTestFinished } from 'vitest'
|
||||
import { observeReadiness } from '../src/client/readiness.ts'
|
||||
|
||||
const catalog: SpeechCatalog = { providers: [], selection: { providerId: 'local' as SpeechProviderId, language: 'auto' },
|
||||
maxAudioBytes: 100, maxDurationSeconds: 120 }
|
||||
|
||||
function fixture(failure?: unknown) {
|
||||
const abort = new AbortController(), release = Promise.withResolvers<undefined>(), accept = vi.fn()
|
||||
let options!: RemoteStreamOptions<SpeechCatalog>
|
||||
const follow = vi.fn()
|
||||
const stream = {
|
||||
signal: abort.signal,
|
||||
async *[Symbol.asyncIterator]() {
|
||||
yield { value: catalog, accept }
|
||||
await release.promise
|
||||
if (failure !== undefined) { abort.abort(); throw failure }
|
||||
},
|
||||
dispose: async () => { abort.abort(); release.resolve(undefined) },
|
||||
}
|
||||
// Only the Gateway-owned transport is replaced; the readiness publication remains real.
|
||||
const ctx = new Context()
|
||||
onTestFinished(async () => { await ctx.fiber.dispose() })
|
||||
ctx.provide('remote', { speech: { follow },
|
||||
$stream: (value: typeof options) => { options = value; return stream },
|
||||
})
|
||||
const mirror = observeReadiness(ctx)
|
||||
return { mirror, options, follow, accept, release }
|
||||
}
|
||||
|
||||
it('accepts a Host snapshot and keeps preparation ownership outside observation disposal', async () => {
|
||||
const b = fixture()
|
||||
await vi.waitFor(() => { expect(b.mirror.state.getSnapshot()).toEqual({ catalog, connected: true, error: null }) })
|
||||
expect(b.accept).toHaveBeenCalledOnce()
|
||||
const signal = new AbortController().signal
|
||||
b.options.open(signal)
|
||||
expect(b.follow).toHaveBeenCalledWith(signal)
|
||||
expect(b.options.ended(true)).toBeInstanceOf(RemoteStreamCarrierError)
|
||||
b.options.carrierFailed?.(new RemoteStreamCarrierError('connection lost'))
|
||||
expect(b.mirror.state.getSnapshot()).toMatchObject({ connected: false, error: 'connection lost' })
|
||||
await b.mirror.dispose()
|
||||
})
|
||||
|
||||
it.each([new Error('invalid snapshot'), 'invalid snapshot'])('reports a terminal stream failure after its transport aborts', async (failure) => {
|
||||
const b = fixture(failure)
|
||||
b.release.resolve(undefined)
|
||||
await vi.waitFor(() => { expect(b.mirror.state.getSnapshot()).toMatchObject({ connected: false, error: 'invalid snapshot' }) })
|
||||
await b.mirror.dispose()
|
||||
})
|
||||
|
||||
it('does not publish a disposal rejection as a Host failure', async () => {
|
||||
const b = fixture(new Error('disposed'))
|
||||
await vi.waitFor(() => { expect(b.mirror.state.getSnapshot().connected).toBe(true) })
|
||||
await b.mirror.dispose()
|
||||
expect(b.mirror.state.getSnapshot().error).toBeNull()
|
||||
})
|
||||
@@ -0,0 +1,61 @@
|
||||
// @vitest-environment jsdom
|
||||
/** Activation guidance waits for cache inspection and leaves installation to the detail page. */
|
||||
import { act, cleanup, fireEvent, render, screen } from '@testing-library/react'
|
||||
import { bindSnapshotSelector, makeTranslate } from '@deepseek-ai/dsh-client-test-runtime'
|
||||
import { createSnapshotStore } from '@deepseek-ai/dsh-client-store'
|
||||
import type { SpeechPreparationState, SpeechProviderId } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import { afterEach, expect, it, vi } from 'vitest'
|
||||
import { VoiceSetupPrompt, type VoiceSetupPromptProps } from '../src/client/VoiceSetupPrompt.tsx'
|
||||
import type { SpeechReadiness } from '../src/client/readiness.ts'
|
||||
import { zh } from '../src/client/locales.ts'
|
||||
|
||||
afterEach(cleanup)
|
||||
|
||||
function fixture() {
|
||||
const state = createSnapshotStore<SpeechReadiness>({ catalog: null, connected: false, error: null })
|
||||
const onDismiss = vi.fn(), onOpenDetails = vi.fn(), providerId = 'local' as SpeechProviderId
|
||||
const props = { packageName: 'voice-bundle', t: makeTranslate(zh), onDismiss, onOpenDetails,
|
||||
useSpeechReadiness: bindSnapshotSelector(state) } as VoiceSetupPromptProps
|
||||
render(<VoiceSetupPrompt {...props} />)
|
||||
return { onDismiss, onOpenDetails, publish(preparation: SpeechPreparationState, location: 'host-local' | 'cloud' = 'host-local') {
|
||||
act(() => { state.set({ connected: true, error: null, catalog: {
|
||||
selection: { providerId, language: 'auto' }, maxAudioBytes: 1000, maxDurationSeconds: 120,
|
||||
providers: [{ id: providerId, name: 'SenseVoice', location, languages: ['auto'], preparation }],
|
||||
} }) })
|
||||
} }
|
||||
}
|
||||
|
||||
it('waits for inspection, then offers the installation page without starting preparation', () => {
|
||||
const b = fixture()
|
||||
expect(screen.queryByText(zh['setupPrompt.title'])).toBeNull()
|
||||
b.publish({ phase: 'checking', step: 'check', startedAt: 0 })
|
||||
expect(screen.queryByText(zh['setupPrompt.title'])).toBeNull()
|
||||
expect(b.onDismiss).not.toHaveBeenCalled()
|
||||
b.publish({ phase: 'unprepared' })
|
||||
expect(screen.getByText(zh['setupPrompt.body'])).toBeTruthy()
|
||||
fireEvent.click(screen.getByRole('button', { name: zh['setupPrompt.open'] }))
|
||||
expect(b.onOpenDetails).toHaveBeenCalledOnce()
|
||||
expect(b.onDismiss).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('allows postponing installation while leaving the plugin enabled', () => {
|
||||
const b = fixture()
|
||||
b.publish({ phase: 'unprepared' })
|
||||
fireEvent.click(screen.getByRole('button', { name: zh['setupPrompt.later'] }))
|
||||
expect(b.onDismiss).toHaveBeenCalledOnce()
|
||||
expect(b.onOpenDetails).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it.each(['standby', 'ready', 'failed'] as const)('does not offer installation for %s resources', (phase) => {
|
||||
const b = fixture()
|
||||
b.publish(phase === 'failed' ? { phase, message: 'cache unreadable' } : { phase })
|
||||
expect(screen.queryByText(zh['setupPrompt.title'])).toBeNull()
|
||||
expect(b.onDismiss).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('does not describe cloud-provider setup as a local model installation', () => {
|
||||
const b = fixture()
|
||||
b.publish({ phase: 'unprepared' }, 'cloud')
|
||||
expect(screen.queryByText(zh['setupPrompt.title'])).toBeNull()
|
||||
expect(b.onDismiss).toHaveBeenCalledOnce()
|
||||
})
|
||||
@@ -0,0 +1,303 @@
|
||||
// @vitest-environment jsdom
|
||||
/** Click capture preserves draft edits and Session ownership without composer banners. */
|
||||
import { afterEach, beforeEach, expect, it, vi } from 'vitest'
|
||||
import { act, cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react'
|
||||
import { bindSnapshotSelector, makeTranslate, RemoteError } from '@deepseek-ai/dsh-client-test-runtime'
|
||||
import { createSnapshotStore } from '@deepseek-ai/dsh-client-store'
|
||||
import { zh as commonZh } from '@deepseek-ai/dsh-client-locale/src/locales/zh.ts'
|
||||
import type { SessionId } from '@deepseek-ai/dsh-session/types'
|
||||
import type { SpeechProviderId, Transcript } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import type { RemoteResult } from '@deepseek-ai/dsh-typert-protocol'
|
||||
import { VoiceInput, type VoiceInputProps } from '../src/client/VoiceInput.tsx'
|
||||
import { RecordingError, Recording } from '../src/client/audio.ts'
|
||||
import type { SpeechReadiness } from '../src/client/readiness.ts'
|
||||
import { zh } from '../src/client/locales.ts'
|
||||
import { captureFixture } from './audio-fixture.client.ts'
|
||||
|
||||
beforeEach(() => { vi.stubGlobal('requestAnimationFrame', vi.fn(() => 1)); vi.stubGlobal('cancelAnimationFrame', vi.fn()) })
|
||||
afterEach(() => { cleanup(); vi.useRealTimers(); vi.unstubAllGlobals() })
|
||||
const id = 'sensevoice-local' as SpeechProviderId
|
||||
const transcript: Transcript = { text: '检查 TypeScript 类型', audioSeconds: 2, inferenceSeconds: 0.4 }
|
||||
function fixture(recording?: Recording) {
|
||||
const capture = Object.assign(new Recording(() => {}), { start: vi.fn<Recording['start']>(async () => {}), stop: vi.fn(async () => new Uint8Array(48)),
|
||||
amplitude: () => 0, dispose: vi.fn(async () => {}) })
|
||||
const inputActions = { notify: vi.fn(), captureInsertion: vi.fn(() => ({ start: 3, end: 3, draftRev: 1 })), insertText: vi.fn(() => true),
|
||||
setDraft: vi.fn(), addAttachments: vi.fn(() => true), removeAttachment: vi.fn(), pruneAttachments: vi.fn(), submit: vi.fn() }
|
||||
const readiness = createSnapshotStore<SpeechReadiness>({ connected: true, error: null, catalog: {
|
||||
providers: [{ id, name: 'SenseVoiceSmall', location: 'host-local', languages: ['auto', 'zh', 'en', 'ja'], preparation: { phase: 'ready' } }],
|
||||
selection: { providerId: id, language: 'auto' }, maxAudioBytes: 100, maxDurationSeconds: 120,
|
||||
} })
|
||||
const transcribe = vi.fn<(request: unknown, signal: AbortSignal) => Promise<RemoteResult<Transcript>>>(
|
||||
async () => ({ ok: true, value: transcript }))
|
||||
const props: VoiceInputProps = { sessionId: 'one' as SessionId, inputActions, transcribe, locked: false, onActiveChange: vi.fn(),
|
||||
prepare: vi.fn(async () => {}), cancelPreparation: vi.fn(async () => {}), configure: vi.fn(async () => {}),
|
||||
useSpeechReadiness: bindSnapshotSelector(readiness), createRecording: () => recording ?? capture,
|
||||
t: makeTranslate(zh, commonZh) }
|
||||
const view = render(<VoiceInput {...props} />)
|
||||
return { props, capture, inputActions, readiness, transcribe, view }
|
||||
}
|
||||
async function start(): Promise<void> {
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.start }))
|
||||
await screen.findByRole('button', { name: zh.stop })
|
||||
}
|
||||
function stop(): void { fireEvent.click(screen.getByRole('button', { name: zh.stop })) }
|
||||
|
||||
it('clicks to record, shows measured audio, and stops to insert without sending or a popup', async () => {
|
||||
const b = fixture()
|
||||
fireEvent.mouseEnter(screen.getByRole('button', { name: zh.start }).parentElement!)
|
||||
expect(screen.getByRole('tooltip').textContent).toBe('听写')
|
||||
fireEvent.mouseDown(screen.getByRole('button', { name: zh.start }))
|
||||
await start()
|
||||
expect(b.props.onActiveChange).toHaveBeenLastCalledWith(true)
|
||||
expect(screen.getByRole('img', { name: zh.recording })).toBeTruthy()
|
||||
expect(b.transcribe).not.toHaveBeenCalled()
|
||||
stop()
|
||||
await waitFor(() => { expect(b.inputActions.insertText).toHaveBeenCalledWith(transcript.text, { start: 3, end: 3, draftRev: 1 }) })
|
||||
expect(b.inputActions.submit).not.toHaveBeenCalled()
|
||||
expect(b.inputActions.notify).not.toHaveBeenCalled()
|
||||
expect(screen.queryByRole('dialog')).toBeNull()
|
||||
expect(b.props.onActiveChange).toHaveBeenLastCalledWith(false)
|
||||
})
|
||||
|
||||
it.each(['cancel', 'escape', 'blur', 'hidden'])('releases a recording without transcription on %s', async (how) => {
|
||||
const b = fixture(); await start()
|
||||
if (how === 'cancel') fireEvent.click(screen.getByRole('button', { name: zh.cancel }))
|
||||
if (how === 'escape') fireEvent.keyDown(document, { key: 'Escape' })
|
||||
if (how === 'blur') fireEvent.blur(window)
|
||||
if (how === 'hidden') {
|
||||
Object.defineProperty(document, 'hidden', { configurable: true, value: true })
|
||||
try { fireEvent(document, new Event('visibilitychange')) } finally { delete (document as { hidden?: boolean }).hidden }
|
||||
}
|
||||
await screen.findByRole('button', { name: zh.start })
|
||||
expect(b.capture.dispose).toHaveBeenCalled()
|
||||
expect(b.transcribe).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it.each([false, true])('invalidates permission settlement after cancel (failure=%s)', async (failure) => {
|
||||
const b = fixture(), permission = Promise.withResolvers<undefined>()
|
||||
b.capture.start.mockReturnValueOnce(permission.promise)
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.start }))
|
||||
expect(screen.getByRole('status').textContent).toBe(zh.requesting)
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.cancel }))
|
||||
if (failure) permission.reject(new Error('permission closed'))
|
||||
else permission.resolve(undefined)
|
||||
await act(async () => {})
|
||||
expect(b.capture.dispose).toHaveBeenCalled()
|
||||
expect(b.inputActions.notify).not.toHaveBeenCalled()
|
||||
expect(screen.queryByText('permission closed')).toBeNull()
|
||||
})
|
||||
|
||||
it('keeps provider and language fixed and preserves conflicting text for explicit insertion', async () => {
|
||||
const b = fixture(); b.inputActions.insertText.mockReturnValue(false)
|
||||
await start()
|
||||
const state = b.readiness.getSnapshot()
|
||||
act(() => { b.readiness.set({ ...state, catalog: { ...state.catalog!, selection: { providerId: id, language: 'en' } } }) })
|
||||
stop()
|
||||
const insert = await screen.findByRole('button', { name: zh.insert })
|
||||
expect(b.transcribe).toHaveBeenCalledWith(expect.objectContaining({ language: 'auto' }), expect.any(AbortSignal))
|
||||
fireEvent.click(insert)
|
||||
expect(screen.getByRole('button', { name: zh.insert })).toBeTruthy()
|
||||
b.inputActions.insertText.mockReturnValue(true)
|
||||
fireEvent.click(insert)
|
||||
expect(screen.getByRole('button', { name: zh.start })).toBeTruthy()
|
||||
b.inputActions.insertText.mockReturnValue(false)
|
||||
await start(); stop()
|
||||
fireEvent.click(await screen.findByRole('button', { name: zh.discard }))
|
||||
expect(screen.queryByRole('button', { name: zh.insert })).toBeNull()
|
||||
})
|
||||
|
||||
it.each(['transcription', 'conversion', 'failed conversion'])('discards late %s after switching Sessions', async (phase) => {
|
||||
const b = fixture(), result = Promise.withResolvers<RemoteResult<Transcript>>()
|
||||
const conversion = Promise.withResolvers<Uint8Array<ArrayBuffer>>()
|
||||
if (phase === 'transcription') b.transcribe.mockReturnValueOnce(result.promise)
|
||||
else b.capture.stop.mockReturnValueOnce(conversion.promise)
|
||||
await start(); stop()
|
||||
await act(async () => {})
|
||||
b.view.rerender(<VoiceInput {...b.props} sessionId={'two' as SessionId} />)
|
||||
if (phase === 'transcription') result.resolve({ ok: true, value: transcript })
|
||||
else if (phase === 'conversion') conversion.resolve(new Uint8Array(48))
|
||||
else conversion.reject(new Error('cancelled'))
|
||||
await act(async () => {})
|
||||
expect(b.inputActions.insertText).not.toHaveBeenCalled()
|
||||
expect(b.inputActions.notify).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('keeps empty-speech and service errors inside the toolbar with retry and dismissal', async () => {
|
||||
const b = fixture()
|
||||
b.transcribe.mockResolvedValueOnce({ ok: true, value: { ...transcript, text: '' } })
|
||||
await start(); stop()
|
||||
expect((await screen.findByRole('status')).textContent).toBe(zh.empty)
|
||||
expect(b.inputActions.notify).not.toHaveBeenCalled()
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.retryRecording }))
|
||||
await screen.findByRole('button', { name: zh.stop })
|
||||
b.transcribe.mockResolvedValueOnce({ ok: false, error: new RemoteError('gateway/internal', 'offline', {}) })
|
||||
stop()
|
||||
await screen.findByText('语音识别失败:offline')
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.cancel }))
|
||||
expect(screen.getByRole('button', { name: zh.start })).toBeTruthy()
|
||||
})
|
||||
|
||||
it.each([new RecordingError('permission'), new Error('device lost'), 'device lost'])('handles capture failure locally (%s)', async (failure) => {
|
||||
const b = fixture(); b.capture.start.mockRejectedValueOnce(failure)
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.start }))
|
||||
await screen.findByText(failure instanceof RecordingError ? zh.permission : '语音识别失败:device lost')
|
||||
expect(b.inputActions.notify).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it.each([new RecordingError('empty'), new Error('decode failed'), 'decode failed'])('handles conversion failure locally (%s)', async (failure) => {
|
||||
const b = fixture(); b.capture.stop.mockRejectedValueOnce(failure)
|
||||
await start(); stop()
|
||||
await screen.findByText(failure instanceof RecordingError ? zh.empty : '语音识别失败:decode failed')
|
||||
expect(b.capture.dispose).toHaveBeenCalledOnce()
|
||||
expect(b.inputActions.insertText).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('bounds recordings, indicates idle wake-up, and cancels an in-flight transcription', async () => {
|
||||
const b = fixture(), result = Promise.withResolvers<RemoteResult<Transcript>>()
|
||||
b.transcribe.mockReturnValueOnce(result.promise)
|
||||
vi.useFakeTimers()
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.start })); await act(async () => {})
|
||||
await act(async () => { await vi.advanceTimersByTimeAsync(120000) })
|
||||
expect(b.capture.stop).toHaveBeenCalledWith(120)
|
||||
fireEvent.blur(window)
|
||||
const state = b.readiness.getSnapshot()
|
||||
act(() => { b.readiness.set({ ...state, catalog: { ...state.catalog!, providers: [
|
||||
{ ...state.catalog!.providers[0]!, preparation: { phase: 'waking', startedAt: Date.now() } },
|
||||
] } }) })
|
||||
expect(screen.getByRole('status').textContent).toBe(zh.wakingShort)
|
||||
expect(b.transcribe.mock.calls[0]?.[1].aborted).toBe(false)
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.cancel }))
|
||||
expect(b.transcribe.mock.calls[0]?.[1].aborted).toBe(true)
|
||||
result.resolve({ ok: true, value: transcript }); await act(async () => {})
|
||||
expect(b.inputActions.insertText).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects excessive audio and disables unavailable or locked recognition', async () => {
|
||||
const b = fixture(); b.capture.stop.mockResolvedValueOnce(new Uint8Array(101))
|
||||
await start(); stop(); await screen.findByText(zh.tooLarge)
|
||||
expect(b.transcribe).not.toHaveBeenCalled()
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.cancel }))
|
||||
act(() => { b.readiness.set({ catalog: null, connected: false, error: null }) })
|
||||
expect(screen.getByRole<HTMLButtonElement>('button', { name: zh.start }).disabled).toBe(true)
|
||||
b.view.rerender(<VoiceInput {...b.props} locked />)
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.start }))
|
||||
expect(b.capture.start).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('releases expansion and contains cleanup failures on unmount', async () => {
|
||||
const b = fixture()
|
||||
fireEvent.keyDown(document, { key: 'x' }); fireEvent.blur(window); fireEvent(document, new Event('visibilitychange'))
|
||||
await start()
|
||||
b.capture.dispose.mockRejectedValueOnce(new Error('already closed'))
|
||||
b.view.unmount(); await act(async () => {})
|
||||
expect(b.props.onActiveChange).toHaveBeenLastCalledWith(false)
|
||||
expect(b.capture.dispose).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('coalesces repeated capture and stop clicks before the toolbar rerenders', async () => {
|
||||
const b = fixture(), microphone = screen.getByRole<HTMLButtonElement>('button', { name: zh.start })
|
||||
act(() => { microphone.click(); microphone.click() })
|
||||
const stopButton = await screen.findByRole<HTMLButtonElement>('button', { name: zh.stop })
|
||||
act(() => { stopButton.click(); stopButton.click() })
|
||||
await waitFor(() => { expect(b.inputActions.insertText).toHaveBeenCalledOnce() })
|
||||
expect(b.capture.start).toHaveBeenCalledOnce()
|
||||
expect(b.capture.stop).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('ends the recording UI immediately on a device failure and clears automatic transcription', async () => {
|
||||
const devices = captureFixture(), b = fixture(devices.recording), closing = Promise.withResolvers<undefined>()
|
||||
devices.close.mockReturnValueOnce(closing.promise)
|
||||
vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] })
|
||||
try {
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.start }))
|
||||
await act(async () => {})
|
||||
expect(screen.getByRole('button', { name: zh.stop })).toBeTruthy()
|
||||
act(() => { devices.failRecorder() })
|
||||
expect(screen.queryByRole('button', { name: zh.stop })).toBeNull()
|
||||
expect(screen.queryByRole('img', { name: zh.recording })).toBeNull()
|
||||
expect(screen.getByRole('status').textContent).toBe(zh.interrupted)
|
||||
expect(devices.disposed).not.toHaveBeenCalled()
|
||||
await act(async () => { await vi.advanceTimersByTimeAsync(120000) })
|
||||
expect(b.transcribe).not.toHaveBeenCalled()
|
||||
expect(b.inputActions.notify).not.toHaveBeenCalled()
|
||||
} finally { closing.resolve(undefined); await devices.recording.dispose() }
|
||||
})
|
||||
|
||||
it.each(['cancel', 'session', 'unmount', 'finishing', 'finished'])('ignores interruption notifications after %s', async (action) => {
|
||||
const b = fixture()
|
||||
await start()
|
||||
const failed = b.capture.start.mock.calls[0]![0]!
|
||||
if (action === 'cancel') fireEvent.click(screen.getByRole('button', { name: zh.cancel }))
|
||||
if (action === 'session') b.view.rerender(<VoiceInput {...b.props} sessionId={'two' as SessionId} />)
|
||||
if (action === 'unmount') b.view.unmount()
|
||||
const pending = Promise.withResolvers<RemoteResult<Transcript>>()
|
||||
if (action === 'finishing') b.transcribe.mockReturnValueOnce(pending.promise)
|
||||
if (action === 'finishing' || action === 'finished') { stop(); await act(async () => {}) }
|
||||
try {
|
||||
act(() => { failed(new RecordingError('interrupted')) })
|
||||
expect(screen.queryByText(zh.interrupted)).toBeNull()
|
||||
expect(b.inputActions.notify).not.toHaveBeenCalled()
|
||||
} finally { pending.resolve({ ok: true, value: transcript }); await act(async () => {}) }
|
||||
})
|
||||
|
||||
it('does not restore recording when capture reports failure before start settles', async () => {
|
||||
const b = fixture()
|
||||
b.capture.start.mockImplementationOnce(async (failed) => { failed!(new RecordingError('interrupted')) })
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.start }))
|
||||
await act(async () => {})
|
||||
expect(screen.getByRole('status').textContent).toBe(zh.interrupted)
|
||||
expect(screen.queryByRole('button', { name: zh.stop })).toBeNull()
|
||||
expect(b.transcribe).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('allows recording while verified local resources are waking', async () => {
|
||||
const b = fixture(), state = b.readiness.getSnapshot()
|
||||
act(() => { b.readiness.set({ ...state, catalog: { ...state.catalog!, providers: [
|
||||
{ ...state.catalog!.providers[0]!, preparation: { phase: 'waking', startedAt: 0 } },
|
||||
] } }) })
|
||||
await start()
|
||||
stop()
|
||||
await waitFor(() => { expect(b.inputActions.insertText).toHaveBeenCalledWith(transcript.text, expect.anything()) })
|
||||
})
|
||||
|
||||
it('keeps the first recording after the permission prompt causes window blur', async () => {
|
||||
const b = fixture(), permission = Promise.withResolvers<undefined>()
|
||||
b.capture.start.mockReturnValueOnce(permission.promise)
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.start }))
|
||||
fireEvent.blur(window)
|
||||
expect(screen.getByRole('status').textContent).toBe(zh.requesting)
|
||||
expect(b.capture.dispose).not.toHaveBeenCalled()
|
||||
permission.resolve(undefined)
|
||||
await screen.findByRole('button', { name: zh.stop })
|
||||
stop()
|
||||
await waitFor(() => { expect(b.inputActions.insertText).toHaveBeenCalledOnce() })
|
||||
})
|
||||
|
||||
it.each(['escape', 'hidden', 'session'])('discards pending permission after %s', async (action) => {
|
||||
const b = fixture(), permission = Promise.withResolvers<undefined>()
|
||||
b.capture.start.mockReturnValueOnce(permission.promise)
|
||||
fireEvent.click(screen.getByRole('button', { name: zh.start }))
|
||||
if (action === 'escape') fireEvent.keyDown(document, { key: 'Escape' })
|
||||
if (action === 'session') b.view.rerender(<VoiceInput {...b.props} sessionId={'two' as SessionId} />)
|
||||
if (action === 'hidden') {
|
||||
Object.defineProperty(document, 'hidden', { configurable: true, value: true })
|
||||
try { fireEvent(document, new Event('visibilitychange')) } finally { delete (document as { hidden?: boolean }).hidden }
|
||||
}
|
||||
permission.resolve(undefined)
|
||||
await act(async () => {})
|
||||
expect(b.capture.dispose).toHaveBeenCalledOnce()
|
||||
expect(screen.getByRole('button', { name: zh.start })).toBeTruthy()
|
||||
expect(screen.queryByRole('button', { name: zh.stop })).toBeNull()
|
||||
expect(b.transcribe).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('explains preparation on hover even when the microphone is disabled', () => {
|
||||
const b = fixture(), state = b.readiness.getSnapshot()
|
||||
act(() => { b.readiness.set({ ...state, catalog: { ...state.catalog!, providers: [
|
||||
{ ...state.catalog!.providers[0]!, preparation: { phase: 'unprepared' } },
|
||||
] } }) })
|
||||
const mic = screen.getByRole<HTMLButtonElement>('button', { name: zh.start })
|
||||
expect(mic.disabled).toBe(true)
|
||||
fireEvent.mouseEnter(mic.parentElement!)
|
||||
expect(screen.getByRole('tooltip').textContent).toBe(zh.prepareRequired)
|
||||
})
|
||||
@@ -0,0 +1,26 @@
|
||||
// @vitest-environment jsdom
|
||||
/** Measured microphone levels drive geometry; quiet audio never creates a synthetic signal. */
|
||||
import { afterEach, expect, it, vi } from 'vitest'
|
||||
import { act, cleanup, render } from '@testing-library/react'
|
||||
import { Waveform } from '../src/client/Waveform.tsx'
|
||||
import { Recording } from '../src/client/audio.ts'
|
||||
afterEach(() => { cleanup(); vi.unstubAllGlobals() })
|
||||
it('samples real levels, throttles drawing, and cancels animation on unmount', () => {
|
||||
let frame: FrameRequestCallback = () => {}
|
||||
vi.stubGlobal('requestAnimationFrame', vi.fn((callback: FrameRequestCallback) => { frame = callback; return 7 }))
|
||||
const cancel = vi.fn(); vi.stubGlobal('cancelAnimationFrame', cancel)
|
||||
const recording = new Recording(() => {})
|
||||
const amplitude = vi.spyOn(recording, 'amplitude').mockReturnValue(0.1)
|
||||
const view = render(<Waveform recording={recording} label="Recording" />)
|
||||
const lines = view.container.querySelectorAll('line')
|
||||
act(() => { frame(0) })
|
||||
expect(Number(lines[79]!.getAttribute('y2'))).toBeGreaterThan(21)
|
||||
act(() => { frame(20) })
|
||||
expect(amplitude).toHaveBeenCalledOnce()
|
||||
act(() => { frame(50) })
|
||||
expect(amplitude).toHaveBeenCalledTimes(2)
|
||||
view.rerender(<Waveform recording={undefined} label="Recording" />)
|
||||
act(() => { frame(100) })
|
||||
expect(lines[79]!.getAttribute('y2')).toBe('21')
|
||||
view.unmount(); expect(cancel).toHaveBeenCalledWith(7)
|
||||
})
|
||||
@@ -0,0 +1,60 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.client.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../api/remotes/tsconfig.client.json"
|
||||
},
|
||||
{
|
||||
"path": "../../api/session-controller/tsconfig.client.json"
|
||||
},
|
||||
{
|
||||
"path": "../../client/locale"
|
||||
},
|
||||
{
|
||||
"path": "../../client/ui-conversation"
|
||||
},
|
||||
{
|
||||
"path": "../../client/ui-primitives"
|
||||
},
|
||||
{
|
||||
"path": "../../client/ui-renderer"
|
||||
},
|
||||
{
|
||||
"path": "../../client/ui-session"
|
||||
},
|
||||
{
|
||||
"path": "../../client/ui-slots"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../../typert/protocol"
|
||||
},
|
||||
{
|
||||
"path": "../api-speech-to-text"
|
||||
},
|
||||
{
|
||||
"path": "../speech-to-text"
|
||||
},
|
||||
{
|
||||
"path": "../../api/gateway/tsconfig.client.json"
|
||||
},
|
||||
{
|
||||
"path": "../../client/store"
|
||||
},
|
||||
{
|
||||
"path": "../../client/ui-plugin-manager"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
import { clientBundle } from '../../client/tsdown.client.ts'
|
||||
|
||||
export default clientBundle('@deepseek-ai/dsh-experimental-client-ui-voice-input', ['lib/types/index.js'])
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/experimental/speech-to-text-sensevoice/README.md
|
||||
README.md: ea751ca1e080ac4805a4cdff58e47dfb9b1dc75d
|
||||
README.zh.md: fc142d1f8bd0f46d93904a29915d985ed9eb2ba0
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
description: "Prepare and operate a local CPU SenseVoice worker on demand."
|
||||
kind: "package-reference"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-experimental-speech-to-text-sensevoice
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
## Summary
|
||||
|
||||
This provider recognizes speech with SenseVoiceSmall ONNX and Silero VAD on the Host CPU. The platform-specific sherpa-onnx Node package includes ONNX Runtime; users need no Python, compiler or model conversion. Activation checks cached resources without loading models or downloading assets.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Use this package](#use-this-package)
|
||||
- [Understand the implementation](#understand-the-implementation)
|
||||
- [Further Exploration](#further-exploration)
|
||||
- [Model Experience](#model-experience)
|
||||
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
||||
- [Dev Note](#dev-note)
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## Use this package
|
||||
|
||||
The [bundle](../voice-input-bundle/README.md) supplies an absolute `dataRoot` under the DSH home. Preparation downloads revision-pinned model files, verifies their sizes and SHA-256 hashes, then loads them. `precision` defaults to `int8`; `fp32` selects the larger reference weights. `modelDirectory` supplies an existing absolute directory containing the selected ONNX file and `tokens.txt`; `vadModelPath` selects an existing Silero ONNX file. `modelOrigin` selects a Hugging Face-compatible download origin while preserving pinned paths and checksums. Verified completed files remain reusable after cancellation or failure.
|
||||
|
||||
Every activation checks the selected model, tokens and VAD on disk. Complete verified caches restore readiness immediately after inspection; the first recording wakes the worker. Missing files or mismatched managed checksums require explicit preparation, while unreadable paths report an error. Explicit deployment paths are checked for accessibility; their contents remain the deployer's responsibility. Disabling the plugin or restarting the Host does not require downloading intact caches again.
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## Understand the implementation
|
||||
|
||||
<details>
|
||||
<summary>Maintainer details — click to expand</summary>
|
||||
|
||||
One managed Node subprocess serializes inference. Recordings submitted during wake-up join the bounded queue. At execution, the worker requires ready or standby resources. Transcription never starts preparation downloads. `threads`, `segmentSeconds`, `vadThreshold`, `minSpeechSeconds` and `minSilenceSeconds` tune CPU inference and segmentation; `maxPending` bounds accepted work. Preparation and inference have separate deadlines. Invalid language or WAV input is rejected before native inference and retains the loaded worker; inference failures, malformed responses and transport failures reclaim it. Cancellation terminates an active worker and joins its process range; cancelled queued recordings never run. `idleTimeoutMs` releases the worker after inactivity, with zero retaining it until disposal. Models remain cached. Authenticated loopback requests keep audio in memory. Electron workers run in Node mode and copy VAD buffers to satisfy the V8 memory cage. No runtime invariant companion is published because `dsh-subprocess` owns process-range observations and this provider has no independent projection to reconcile.
|
||||
|
||||
The Host owns preparation across page and Session changes: check resources, download recognition weights and tokens, download VAD, verify files, and load the worker. Explicit model paths omit their download steps. Downloads report bytes; other steps report elapsed time. An idle worker retains prepared paths and needs only model loading on wake-up. Cancellation after a preparation result is published waits for cleanup without replacing that result.
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## Further Exploration
|
||||
|
||||
[Voice input subsystem](../../../docs/subsystems/voice-input.md)
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## Model Experience
|
||||
|
||||
None, as recordings and preparation remain outside model requests; ordinary user submission owns any later text.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
No direct effect; ordinary submission owns the message content.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- CPU only. Native packages target macOS arm64/x64, Linux glibc arm64/x64 and Windows x64; Windows ARM64 and Linux musl are not verified deployment targets. INT8 weights occupy about 239 MB, FP32 about 938 MB, plus the runtime and VAD. Model memory exceeds file size and grows with recording length; installation hints are estimates. Language hints are auto, Chinese, English, Cantonese, Japanese and Korean. Disabling the plugin retains model caches. Packaged Desktop releases still require platform signing and microphone-permission acceptance; Linux Desktop distribution is outside this provider.
|
||||
|
||||
-----
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### Dev Note
|
||||
|
||||
<details>
|
||||
<summary>Maintainer details — click to expand</summary>
|
||||
|
||||
None.
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
description: "按需准备和运行本地 CPU SenseVoice 工作进程。"
|
||||
kind: "package-reference"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-experimental-speech-to-text-sensevoice
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
## 概述
|
||||
|
||||
此 Provider 在 Host CPU 上使用 SenseVoiceSmall ONNX 与 Silero VAD 识别语音。各平台的 sherpa-onnx Node 包包含 ONNX Runtime;用户无需 Python、编译器或模型转换。启用时检查缓存资源,不加载模型或下载资源。
|
||||
|
||||
## 目录
|
||||
|
||||
- [使用此包](#use-this-package)
|
||||
- [理解实现](#understand-the-implementation)
|
||||
- [进一步探索](#further-exploration)
|
||||
- [模型体验](#model-experience)
|
||||
- [已知限制与后续工作](#known-limitations-and-deferred-work)
|
||||
- [开发备注](#dev-note)
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## 使用此包
|
||||
|
||||
[Bundle](../voice-input-bundle/README.zh.md) 在 DSH 主目录下提供绝对 `dataRoot`。准备会下载固定修订的模型文件,校验大小与 SHA-256,再加载模型。`precision` 默认为 `int8`,`fp32` 选择较大的参考权重。`modelDirectory` 提供包含所选 ONNX 文件和 `tokens.txt` 的已有绝对目录,`vadModelPath` 选择已有 Silero ONNX 文件。`modelOrigin` 选择兼容 Hugging Face 的下载源,固定路径和校验值保持不变。取消或失败后复用已完成并通过校验的文件。
|
||||
|
||||
每次启用都会检查磁盘上的所选模型、词表与 VAD。缓存完整且校验通过后立即恢复就绪,首次录音再唤醒工作进程。文件缺失或托管文件校验不符时需要显式准备;路径不可读时报告错误。显式部署路径只检查可访问性,文件内容由部署者负责。禁用插件或重启 Host 后,无需重新下载完好的缓存。
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## 理解实现
|
||||
|
||||
<details>
|
||||
<summary>维护者信息 — 点击展开</summary>
|
||||
|
||||
一个托管 Node 子进程串行执行推理。唤醒期间提交的录音进入有界队列。工作进程在执行时要求资源处于就绪或待唤醒状态。转写不会启动准备下载。`threads`、`segmentSeconds`、`vadThreshold`、`minSpeechSeconds` 和 `minSilenceSeconds` 调节 CPU 推理与分段;`maxPending` 限制已接收任务数。准备和推理分别有截止时间。无效语言或 WAV 输入在原生推理前被拒绝,保留已加载的工作进程;推理失败、响应格式错误和传输失败会回收进程。取消会终止运行中的工作进程并等待其进程范围退出;已取消的排队录音不会执行。`idleTimeoutMs` 在空闲后释放进程,零表示保留至插件释放。模型继续保留在缓存中。带认证的环回请求仅在内存中保存音频。Electron 工作进程使用 Node 模式,并复制 VAD 缓冲区以兼容 V8 内存笼。`dsh-subprocess` 拥有进程范围观测;本包没有需要协调的独立投影,因此不发布不变量伴随模块。
|
||||
|
||||
Host 在页面和 Session 变化期间拥有准备任务:检查资源、下载识别权重与词表、下载 VAD、校验文件、加载工作进程。显式模型路径会省略对应下载步骤。下载报告字节,其他步骤报告等待时间。空闲进程保留已准备路径,唤醒时只需加载模型。准备结果发布后再取消,只等待任务收尾,不会覆盖该结果。
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## 进一步探索
|
||||
|
||||
[语音输入子系统](../../../docs/subsystems/voice-input.zh.md)
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## 模型体验
|
||||
|
||||
无,因为录音与准备不进入模型请求;之后的文字由普通用户提交拥有。
|
||||
|
||||
#### KV 缓存影响
|
||||
|
||||
没有直接影响;普通提交拥有消息内容。
|
||||
|
||||
## 已知限制与后续工作
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- 仅使用 CPU。原生包面向 macOS arm64/x64、Linux glibc arm64/x64 和 Windows x64;Windows ARM64 与 Linux musl 尚非已验证部署目标。INT8 权重约 239 MB,FP32 约 938 MB,另需运行时和 VAD。模型内存大于文件大小,并随录音长度增长;安装提示属于估算。语言提示包括自动、中文、英语、粤语、日语和韩语。禁用插件保留模型缓存。桌面发行包仍需各平台签名和麦克风权限验收;Linux 桌面分发不属于此 Provider。
|
||||
|
||||
-----
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### 开发备注
|
||||
|
||||
<details>
|
||||
<summary>维护者信息 — 点击展开</summary>
|
||||
|
||||
无。
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,53 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-experimental-speech-to-text-sensevoice",
|
||||
"description": "Local SenseVoice ONNX transcription with a managed sherpa-onnx process",
|
||||
"version": "0.1.6-alpha.2",
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
|
||||
"directory": "packages/experimental/speech-to-text-sensevoice"
|
||||
},
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json",
|
||||
"./worker": {
|
||||
"types": "./lib/types/worker.d.ts",
|
||||
"default": "./lib/worker.js"
|
||||
}
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/worker.js",
|
||||
"runtime/assets.json",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@deepseek-ai/schemastery": "workspace:^",
|
||||
"@deepseek-ai/dsh-timeout": "workspace:^",
|
||||
"zod": "^4.4.3",
|
||||
"sherpa-onnx-node": "1.13.8"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:^",
|
||||
"@deepseek-ai/dsh-experimental-speech-to-text": "workspace:^",
|
||||
"@deepseek-ai/dsh-subprocess": "workspace:^"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:^",
|
||||
"@deepseek-ai/dsh-experimental-speech-to-text": "workspace:^",
|
||||
"@deepseek-ai/dsh-subprocess": "workspace:^",
|
||||
"@deepseek-ai/dsh-subprocess-local": "workspace:^",
|
||||
"tsx": "^4.22.4"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"models": {
|
||||
"int8": {
|
||||
"name": "model.int8.onnx",
|
||||
"url": "https://huggingface.co/csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17/resolve/2365baeacb507f821a0c8120fcee3d484dba7a07/model.int8.onnx",
|
||||
"bytes": 239233841,
|
||||
"sha256": "c71f0ce00bec95b07744e116345e33d8cbbe08cef896382cf907bf4b51a2cd51"
|
||||
},
|
||||
"fp32": {
|
||||
"name": "model.onnx",
|
||||
"url": "https://huggingface.co/csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17/resolve/2365baeacb507f821a0c8120fcee3d484dba7a07/model.onnx",
|
||||
"bytes": 937617178,
|
||||
"sha256": "977016bd9c79f9eb343430b5cc305e07ab64d5212dff41b0dcfa1694bee9a8cb"
|
||||
}
|
||||
},
|
||||
"tokens": {
|
||||
"name": "tokens.txt",
|
||||
"url": "https://huggingface.co/csukuangfj/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-2024-07-17/resolve/2365baeacb507f821a0c8120fcee3d484dba7a07/tokens.txt",
|
||||
"bytes": 315894,
|
||||
"sha256": "f449eb28dc567533d7fa59be34e2abca8784f771850c78a47fb731a31429a1dc"
|
||||
},
|
||||
"vad": {
|
||||
"name": "silero_vad.onnx",
|
||||
"url": "https://huggingface.co/csukuangfj/vad/resolve/fba88cd2e921609e7675c3aaf51e0b9b295da4bc/silero_vad.onnx",
|
||||
"bytes": 1807522,
|
||||
"sha256": "a35ebf52fd3ce5f1469b2a36158dba761bc47b973ea3382b3186ca15b1f5af28"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
/** Deployment configuration for the managed local SenseVoice recognizer. */
|
||||
import z from '@deepseek-ai/schemastery'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
|
||||
/** Local runtime, inference, and retention settings. */
|
||||
export interface Config {
|
||||
/** Unique registration id; consumers select this exact id. */
|
||||
providerId: string
|
||||
/** Absolute directory for verified ONNX models. */
|
||||
dataRoot: string
|
||||
/** Existing directory containing the selected ONNX model and tokens.txt; omission downloads verified files. */
|
||||
modelDirectory?: string | undefined
|
||||
/** Existing Silero VAD ONNX file; omission downloads the verified model. */
|
||||
vadModelPath?: string | undefined
|
||||
/** Weight precision; INT8 minimizes first-use download and model storage. */
|
||||
precision: 'int8' | 'fp32'
|
||||
/** Hugging Face-compatible origin for pinned model URLs, including private mirrors. */
|
||||
modelOrigin: string
|
||||
/** CPU intra-operation thread count. */
|
||||
threads: number
|
||||
/** Maximum speech segment length passed to the recognizer. */
|
||||
segmentSeconds: number
|
||||
/** Silero speech probability threshold. */
|
||||
vadThreshold: number
|
||||
/** Minimum speech duration retained by VAD. */
|
||||
minSpeechSeconds: number
|
||||
/** Silence separating two speech segments. */
|
||||
minSilenceSeconds: number
|
||||
/** Maximum decoded WAV bytes accepted by the private worker. */
|
||||
maxAudioBytes: number
|
||||
/** Deadline for runtime preparation and cold model loading. */
|
||||
prepareTimeoutMs: number
|
||||
/** Deadline for one inference after the worker is ready. */
|
||||
inferenceTimeoutMs: number
|
||||
/** Idle period before stopping the worker; zero keeps it warm. */
|
||||
idleTimeoutMs: number
|
||||
/** Maximum accepted running and waiting transcriptions. */
|
||||
maxPending: number
|
||||
/** Managed process termination grace period. */
|
||||
graceMs: number
|
||||
/** Maximum retained worker diagnostic bytes. */
|
||||
maxLogBytes: number
|
||||
/** Maximum transcript response bytes. */
|
||||
maxResponseBytes: number
|
||||
/** Minimum interval between intermediate download progress notifications. */
|
||||
progressIntervalMs: number
|
||||
}
|
||||
|
||||
/** Validate deployment-varying runtime choices at plugin activation. */
|
||||
export const Config: z<Partial<Config>, Config> = z.object({
|
||||
providerId: z.string().min(1).default('sensevoice-local'),
|
||||
dataRoot: z.string().min(1).required(),
|
||||
modelDirectory: z.union([z.string().min(1), z.const(undefined)]),
|
||||
vadModelPath: z.union([z.string().min(1), z.const(undefined)]),
|
||||
precision: z.union(['int8', 'fp32']).default('int8'),
|
||||
modelOrigin: z.string().pattern(/^https?:\/\/[^/\s?#@]+\/?$/).default('https://huggingface.co'),
|
||||
threads: z.natural().min(1).default(2),
|
||||
segmentSeconds: z.number().min(1).max(120).default(30),
|
||||
vadThreshold: z.number().min(0).max(1).default(0.5),
|
||||
minSpeechSeconds: z.number().min(0).default(0.25),
|
||||
minSilenceSeconds: z.number().min(0.01).default(0.5),
|
||||
maxAudioBytes: z.natural().min(46).default(4 * 1024 * 1024),
|
||||
prepareTimeoutMs: z.natural().min(1).max(MAX_TIMER_DELAY_MS).default(3_600_000),
|
||||
inferenceTimeoutMs: z.natural().min(1).max(MAX_TIMER_DELAY_MS).default(120_000),
|
||||
idleTimeoutMs: z.natural().max(MAX_TIMER_DELAY_MS).default(300_000),
|
||||
maxPending: z.natural().min(1).default(4),
|
||||
graceMs: z.natural().min(1).max(MAX_TIMER_DELAY_MS).default(1000),
|
||||
maxLogBytes: z.natural().min(1).default(64 * 1024),
|
||||
maxResponseBytes: z.natural().min(1).default(128 * 1024),
|
||||
progressIntervalMs: z.natural().min(1).max(MAX_TIMER_DELAY_MS).default(100),
|
||||
})
|
||||
@@ -0,0 +1,42 @@
|
||||
/** Optional local SenseVoice provider; activation performs no downloads or model loading. */
|
||||
import { isAbsolute } from 'node:path'
|
||||
import type { Context } from '@deepseek-ai/cordis'
|
||||
import type {} from '@deepseek-ai/dsh-experimental-speech-to-text'
|
||||
import type {} from '@deepseek-ai/dsh-subprocess'
|
||||
import type { SpeechProviderId } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import { Config } from './config.ts'
|
||||
import { SenseVoiceWorker } from './recognizer.ts'
|
||||
import { languages } from './input.ts'
|
||||
|
||||
export { Config } from './config.ts'
|
||||
export const name = 'experimental-speech-to-text-sensevoice'
|
||||
export const inject = ['speechToText', 'subprocess']
|
||||
|
||||
/**
|
||||
* Register the local recognizer and inspect disk caches without downloading or loading models.
|
||||
* @param ctx - Host registry and subprocess owner.
|
||||
* @param config - validated runtime configuration.
|
||||
*/
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
for (const path of [config.dataRoot, config.modelDirectory, config.vadModelPath]) {
|
||||
if (path !== undefined && !isAbsolute(path)) throw new Error(`SenseVoice paths must be absolute: ${path}`)
|
||||
}
|
||||
new URL(config.modelOrigin)
|
||||
const worker = new SenseVoiceWorker(ctx, config)
|
||||
const estimatedBytes = config.precision === 'int8' ? 1_000_000_000 : 2_000_000_000
|
||||
ctx.effect(() => {
|
||||
const unregister = ctx.speechToText.register({
|
||||
info: { id: config.providerId as SpeechProviderId, name: `SenseVoiceSmall (${config.precision.toUpperCase()})`, location: 'host-local', languages,
|
||||
setupEstimate: { recommendedDiskBytes: estimatedBytes, expectedMemoryBytes: estimatedBytes,
|
||||
minimumMinutes: 1, maximumMinutes: 10 } },
|
||||
preparation: worker,
|
||||
transcribe: async (input, signal) => await worker.transcribe(input, signal),
|
||||
})
|
||||
worker.inspect()
|
||||
return async () => {
|
||||
const removing = unregister()
|
||||
await worker.dispose()
|
||||
await removing
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
/** CPU SenseVoice inference and Silero segmentation, confined to the recognition process. */
|
||||
import { createRequire } from 'node:module'
|
||||
import type { Transcript } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import type { Config } from './config.ts'
|
||||
import { validateInput } from './input.ts'
|
||||
|
||||
/** Verified files and validated inference settings sent by the Host. */
|
||||
export interface InferenceConfig extends Config {
|
||||
readonly model: string
|
||||
readonly tokens: string
|
||||
readonly vad: string
|
||||
}
|
||||
|
||||
interface Stream { acceptWaveform(audio: { samples: Float32Array; sampleRate: number }): void }
|
||||
interface Recognizer {
|
||||
createStream(): Stream
|
||||
setConfig(config: object): void
|
||||
decode(stream: Stream): void
|
||||
getResult(stream: Stream): { text: string }
|
||||
}
|
||||
interface Detector {
|
||||
acceptWaveform(samples: Float32Array): void
|
||||
isEmpty(): boolean
|
||||
front(externalBuffer: false): { samples: Float32Array }
|
||||
pop(): void
|
||||
reset(): void
|
||||
flush(): void
|
||||
}
|
||||
interface Sherpa {
|
||||
OfflineRecognizer: new (config: object) => Recognizer
|
||||
Vad: new (config: object, bufferSeconds: number) => Detector
|
||||
}
|
||||
|
||||
/**
|
||||
* Load one native model pair; every recording resets VAD and updates its language hint.
|
||||
* @param config - verified ONNX paths and explicit CPU/VAD limits.
|
||||
* @returns synchronous inference confined to its dedicated process.
|
||||
*/
|
||||
export function createTranscriber(config: InferenceConfig): (audio: Uint8Array, language: string) => Transcript {
|
||||
// sherpa-onnx-node publishes a CommonJS Node-API binding with JSDoc but no declarations.
|
||||
const sherpa = createRequire(import.meta.url)('sherpa-onnx-node') as Sherpa
|
||||
const nativeConfig = {
|
||||
featConfig: { sampleRate: 16000, featureDim: 80 },
|
||||
modelConfig: { senseVoice: { model: config.model, language: 'auto', useInverseTextNormalization: 1 },
|
||||
tokens: config.tokens, numThreads: config.threads, provider: 'cpu', debug: 0 },
|
||||
}
|
||||
const recognizer = new sherpa.OfflineRecognizer(nativeConfig)
|
||||
const detector = new sherpa.Vad({
|
||||
sileroVad: { model: config.vad, threshold: config.vadThreshold, minSilenceDuration: config.minSilenceSeconds,
|
||||
minSpeechDuration: config.minSpeechSeconds, maxSpeechDuration: config.segmentSeconds, windowSize: 512 },
|
||||
sampleRate: 16000, numThreads: config.threads, provider: 'cpu', debug: 0,
|
||||
}, config.segmentSeconds + config.minSilenceSeconds + 1)
|
||||
return (audio, language) => {
|
||||
const audioSeconds = validateInput(audio, language, config.maxAudioBytes)
|
||||
const pcm = new DataView(audio.buffer, audio.byteOffset + 44, audio.byteLength - 44)
|
||||
const samples = Float32Array.from({ length: pcm.byteLength / 2 }, (_, i) => pcm.getInt16(i * 2, true) / 32768)
|
||||
nativeConfig.modelConfig.senseVoice.language = language
|
||||
recognizer.setConfig(nativeConfig)
|
||||
detector.reset()
|
||||
const started = performance.now(), texts: string[] = []
|
||||
const drain = (): void => {
|
||||
while (!detector.isEmpty()) {
|
||||
// Electron's V8 memory cage requires copied native buffers, including VAD output.
|
||||
const segment = detector.front(false)
|
||||
const stream = recognizer.createStream()
|
||||
stream.acceptWaveform({ sampleRate: 16000, samples: segment.samples })
|
||||
recognizer.decode(stream)
|
||||
texts.push(recognizer.getResult(stream).text.trim())
|
||||
detector.pop()
|
||||
}
|
||||
}
|
||||
for (let offset = 0; offset < samples.length; offset += 512) {
|
||||
detector.acceptWaveform(samples.subarray(offset, offset + 512))
|
||||
drain()
|
||||
}
|
||||
detector.flush(); drain()
|
||||
return { text: texts.filter(Boolean).join(' ').trim(), audioSeconds, inferenceSeconds: (performance.now() - started) / 1000 }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
/** SenseVoice language support and input failures that leave native inference untouched. */
|
||||
import { validateWave } from '@deepseek-ai/dsh-experimental-speech-to-text/wave'
|
||||
|
||||
/** Language hints accepted by both provider metadata and native inference. */
|
||||
export const languages: readonly string[] = ['auto', 'zh', 'en', 'yue', 'ja', 'ko']
|
||||
|
||||
/** A request rejected before native inference; the loaded worker remains reusable. */
|
||||
export class SpeechInputError extends Error {}
|
||||
|
||||
/**
|
||||
* Validate a recording before touching the native recognizer.
|
||||
* @param audio - untrusted WAV request bytes.
|
||||
* @param language - requested SenseVoice language hint.
|
||||
* @param maxAudioBytes - configured worker byte limit.
|
||||
* @returns validated audio duration in seconds; invalid inputs throw SpeechInputError.
|
||||
*/
|
||||
export function validateInput(audio: Uint8Array, language: string, maxAudioBytes: number): number {
|
||||
if (!languages.includes(language)) throw new SpeechInputError('Unsupported SenseVoice language')
|
||||
if (audio.byteLength > maxAudioBytes) throw new SpeechInputError('Speech audio exceeds the worker byte limit')
|
||||
try { return validateWave(audio, maxAudioBytes / 32000) }
|
||||
catch (error) { throw new SpeechInputError('Invalid speech WAV', { cause: error }) }
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
/** Authenticated loopback transport for one serial native recognizer. */
|
||||
import { createServer, type Server } from 'node:http'
|
||||
import { timingSafeEqual } from 'node:crypto'
|
||||
import type { Transcript } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import { SpeechInputError } from './input.ts'
|
||||
|
||||
/**
|
||||
* Bind an ephemeral loopback listener; model loading completes before readiness is published.
|
||||
* @param token - private per-process authentication secret.
|
||||
* @param maxAudioBytes - maximum retained request bytes.
|
||||
* @param transcribe - synchronous inference owned by this process.
|
||||
* @returns the listening server and its dynamically assigned port.
|
||||
*/
|
||||
export async function startRecognitionServer(token: string, maxAudioBytes: number,
|
||||
transcribe: (audio: Uint8Array, language: string) => Transcript): Promise<{ server: Server; port: number }> {
|
||||
const expected = Buffer.from(`Bearer ${token}`)
|
||||
const server = createServer((request, response) => {
|
||||
const reply = (status: number, value: object): void => {
|
||||
response.writeHead(status, { 'content-type': 'application/json' }).end(JSON.stringify(value))
|
||||
}
|
||||
const authorization = Buffer.from(request.headers.authorization ?? '')
|
||||
if (authorization.length !== expected.length || !timingSafeEqual(authorization, expected)) {
|
||||
request.resume(); reply(401, { error: 'Unauthorized' }); return
|
||||
}
|
||||
const url = new URL(request.url as string, 'http://localhost')
|
||||
if (request.method !== 'POST' || url.pathname !== '/transcribe') {
|
||||
request.resume(); reply(404, { error: 'Unknown endpoint' }); return
|
||||
}
|
||||
const length = Number(request.headers['content-length'])
|
||||
if (!Number.isSafeInteger(length) || length < 46 || length > maxAudioBytes) {
|
||||
request.resume(); reply(413, { error: 'Invalid speech audio size', code: 'invalid-input' }); return
|
||||
}
|
||||
void (async () => {
|
||||
try {
|
||||
const chunks: Buffer[] = []
|
||||
for await (const chunk of request) {
|
||||
const bytes = chunk as Buffer
|
||||
chunks.push(bytes)
|
||||
}
|
||||
reply(200, transcribe(Buffer.concat(chunks), url.searchParams.get('language') ?? 'auto'))
|
||||
} catch (error) {
|
||||
reply(error instanceof SpeechInputError ? 400 : 500, {
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
...error instanceof SpeechInputError ? { code: 'invalid-input' } : {},
|
||||
})
|
||||
}
|
||||
})()
|
||||
})
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
server.once('error', reject)
|
||||
server.listen(0, '127.0.0.1', () => { server.off('error', reject); resolve() })
|
||||
})
|
||||
const address = server.address() as { port: number }
|
||||
return { server, port: address.port }
|
||||
}
|
||||
@@ -0,0 +1,314 @@
|
||||
/** One serial SenseVoice worker with request-owned cancellation and idle reclamation. */
|
||||
import { randomBytes } from 'node:crypto'
|
||||
import { mkdir } from 'node:fs/promises'
|
||||
import type { Context } from '@deepseek-ai/cordis'
|
||||
import type { SubprocessHandle } from '@deepseek-ai/dsh-subprocess'
|
||||
import type { SpeechInput, SpeechPreparationState, SpeechPreparationStep, SpeechPreparationStepKind, Transcript } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import { deadline } from '@deepseek-ai/dsh-timeout'
|
||||
import { z } from 'zod'
|
||||
import type { Config } from './config.ts'
|
||||
import { SpeechInputError } from './input.ts'
|
||||
import { inspectRuntime, prepareRuntime, type RuntimePaths } from './runtime.ts'
|
||||
|
||||
const transcriptSchema = z.object({
|
||||
text: z.string(), audioSeconds: z.number().nonnegative(), inferenceSeconds: z.number().nonnegative(),
|
||||
}).strict()
|
||||
|
||||
interface Worker {
|
||||
readonly handle: SubprocessHandle
|
||||
readonly url: string
|
||||
readonly token: string
|
||||
closed: boolean
|
||||
}
|
||||
|
||||
/** Wait for a cancellable operation without losing ownership of its eventual settlement. */
|
||||
function waitFor<T>(pending: Promise<T>, signal: AbortSignal): Promise<T> {
|
||||
signal.throwIfAborted()
|
||||
return new Promise((resolve, reject) => {
|
||||
const abort = (): void => { reject(signal.reason instanceof Error ? signal.reason : new Error('Speech operation cancelled')) }
|
||||
signal.addEventListener('abort', abort, { once: true })
|
||||
void pending.then(resolve, reject).finally(() => { signal.removeEventListener('abort', abort) })
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Read one bounded worker readiness frame and reject exit before readiness.
|
||||
* @param handle - newly spawned worker with piped stdout.
|
||||
* @param limit - maximum readiness bytes.
|
||||
* @param signal - startup deadline or caller cancellation.
|
||||
* @returns dynamically allocated loopback port.
|
||||
*/
|
||||
export async function readReady(handle: SubprocessHandle, limit: number, signal: AbortSignal): Promise<number> {
|
||||
if (!handle.stdout) throw new Error('Speech worker stdout is unavailable')
|
||||
signal.throwIfAborted()
|
||||
const stdout = handle.stdout
|
||||
let text = ''
|
||||
const pending = Promise.withResolvers<number>()
|
||||
const data = (chunk: Buffer): void => {
|
||||
text += chunk.toString('utf8')
|
||||
if (Buffer.byteLength(text) > limit) { pending.reject(new Error('Speech worker readiness exceeded its byte limit')); return }
|
||||
const end = text.indexOf('\n')
|
||||
if (end < 0) return
|
||||
try {
|
||||
const value = z.object({ port: z.number().int().min(1).max(65535) }).strict().parse(JSON.parse(text.slice(0, end)))
|
||||
pending.resolve(value.port)
|
||||
} catch (error) { pending.reject(new Error('Invalid speech worker readiness', { cause: error })) }
|
||||
}
|
||||
stdout.on('data', data)
|
||||
stdout.on('error', pending.reject)
|
||||
void handle.done.then(() => {
|
||||
pending.reject(new Error(`Speech worker exited before readiness: ${handle.collected.stderr?.readFrom(0).text}`))
|
||||
}, pending.reject)
|
||||
try { return await waitFor(pending.promise, signal) }
|
||||
finally { stdout.off('data', data); stdout.off('error', pending.reject); stdout.resume() }
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode a bounded worker HTTP response; malformed worker output fails the request.
|
||||
* @param response - private authenticated worker response.
|
||||
* @param limit - maximum bytes retained before JSON parsing.
|
||||
* @returns validated final transcript; marked input rejections throw SpeechInputError.
|
||||
*/
|
||||
export async function readTranscript(response: Response, limit: number): Promise<Transcript> {
|
||||
if (!response.body) throw new Error('Speech worker returned no response')
|
||||
const reader = response.body.getReader()
|
||||
const chunks: Uint8Array[] = []
|
||||
let length = 0
|
||||
try {
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read()
|
||||
if (done) break
|
||||
length += value.length
|
||||
if (length > limit) throw new Error('Speech transcript exceeded its byte limit')
|
||||
chunks.push(value)
|
||||
}
|
||||
} finally {
|
||||
await reader.cancel()
|
||||
reader.releaseLock()
|
||||
}
|
||||
const value: unknown = JSON.parse(Buffer.concat(chunks).toString('utf8'))
|
||||
if (!response.ok) {
|
||||
const failure = z.object({ error: z.string(), code: z.literal('invalid-input').optional() }).parse(value)
|
||||
if ((response.status === 400 || response.status === 413) && failure.code === 'invalid-input') {
|
||||
throw new SpeechInputError(failure.error)
|
||||
}
|
||||
throw new Error(failure.error)
|
||||
}
|
||||
return transcriptSchema.parse(value)
|
||||
}
|
||||
|
||||
/** Own one worker across recordings, and join every accepted job on disposal. */
|
||||
export class SenseVoiceWorker {
|
||||
private worker: Worker | undefined
|
||||
private tail: Promise<void> = Promise.resolve()
|
||||
private pending = 0
|
||||
private readonly lifetime = new AbortController()
|
||||
private idle: ReturnType<typeof setTimeout> | undefined
|
||||
private runtime: RuntimePaths | undefined
|
||||
private state: SpeechPreparationState & { readonly steps: readonly SpeechPreparationStep[] }
|
||||
private readonly listeners = new Set<() => void>()
|
||||
private lastProgressAt = 0
|
||||
private preparing: { abort: AbortController; settled: Promise<void>; completed: boolean } | undefined
|
||||
|
||||
constructor(private readonly ctx: Context, private readonly config: Config) {
|
||||
const kinds: SpeechPreparationStepKind[] = ['check']
|
||||
if (config.modelDirectory === undefined) kinds.push('model')
|
||||
if (config.vadModelPath === undefined) kinds.push('vad')
|
||||
kinds.push('verify', 'load')
|
||||
this.state = { phase: 'unprepared', steps: kinds.map(kind => ({ kind, status: 'pending' })) }
|
||||
}
|
||||
|
||||
/**
|
||||
* Read preparation readiness.
|
||||
* @returns the current Host-owned state.
|
||||
*/
|
||||
snapshot(): SpeechPreparationState { return this.state }
|
||||
|
||||
/**
|
||||
* Observe readiness.
|
||||
* @param listener - invalidation callback.
|
||||
* @returns subscription disposer.
|
||||
*/
|
||||
subscribe(listener: () => void): () => void {
|
||||
this.listeners.add(listener)
|
||||
return () => { this.listeners.delete(listener) }
|
||||
}
|
||||
|
||||
private publish(state: SpeechPreparationState): void {
|
||||
if (this.preparing && ['ready', 'standby', 'unprepared', 'failed', 'cancelled'].includes(state.phase)) {
|
||||
this.preparing.completed = true
|
||||
}
|
||||
const previous = this.state
|
||||
const steps = state.steps ?? previous.steps.map((item): SpeechPreparationStep => {
|
||||
if (state.phase === 'ready') return { ...item, status: 'complete' }
|
||||
if (state.step === 'check' && item.kind !== 'check') return { kind: item.kind, status: 'pending' }
|
||||
if (state.step === item.kind) return item.status === 'running' ? item
|
||||
: { kind: item.kind, status: 'running', startedAt: Date.now() }
|
||||
if (item.status !== 'running') return item
|
||||
if (state.phase === 'standby') return { ...item, status: 'cancelled' }
|
||||
if (state.phase === 'failed' || state.phase === 'cancelled') return { ...item, status: state.phase }
|
||||
return state.step === undefined ? item : { ...item, status: 'complete' }
|
||||
})
|
||||
this.state = { ...state, steps }
|
||||
const now = performance.now()
|
||||
if (state.phase === 'downloading' && previous.phase === 'downloading' && state.resource === previous.resource
|
||||
&& state.completedBytes !== state.totalBytes && now - this.lastProgressAt < this.config.progressIntervalMs) return
|
||||
this.lastProgressAt = now
|
||||
for (const listener of this.listeners) listener()
|
||||
}
|
||||
|
||||
/** Inspect disk caches on activation; valid resources enter standby without starting a worker. */
|
||||
inspect(): void {
|
||||
this.runPreparation(async (signal) => {
|
||||
using inspection = deadline(signal, this.config.prepareTimeoutMs, 'SPEECH_PREPARE_TIMEOUT')
|
||||
this.publish({ phase: 'checking', step: 'check', startedAt: Date.now() })
|
||||
const runtime = await inspectRuntime(this.config, inspection.signal)
|
||||
inspection.signal.throwIfAborted()
|
||||
this.runtime = runtime
|
||||
this.publish({ phase: runtime ? 'standby' : 'unprepared', steps: this.state.steps.map(({ kind }) => ({ kind,
|
||||
status: kind === 'check' || runtime && kind !== 'load' ? 'complete' : 'pending',
|
||||
})) })
|
||||
})
|
||||
}
|
||||
|
||||
/** Start one Host-owned preparation task; repeated callers join it. */
|
||||
prepare(): void {
|
||||
this.runPreparation(async (signal) => { await this.start(signal) })
|
||||
}
|
||||
|
||||
private runPreparation(run: (signal: AbortSignal) => Promise<void>): void {
|
||||
if (this.preparing || this.worker) return
|
||||
this.lifetime.signal.throwIfAborted()
|
||||
const abort = new AbortController()
|
||||
const task = { abort, settled: Promise.resolve(), completed: false }
|
||||
this.preparing = task
|
||||
task.settled = this.enqueue(run, abort.signal)
|
||||
.catch((error: unknown) => {
|
||||
if (this.lifetime.signal.aborted) return
|
||||
this.publish(abort.signal.aborted ? { phase: 'cancelled' }
|
||||
: { phase: 'failed', message: error instanceof Error ? error.message : String(error) })
|
||||
}).finally(() => { this.preparing = undefined })
|
||||
}
|
||||
|
||||
/** Cancel unfinished preparation; completed readiness is retained. @returns after its queued or active work settles. */
|
||||
async cancel(): Promise<void> {
|
||||
const task = this.preparing
|
||||
if (!task) return
|
||||
if (!task.completed) {
|
||||
this.publish({ phase: 'cancelling', startedAt: Date.now() })
|
||||
task.abort.abort(new Error('Speech preparation cancelled'))
|
||||
}
|
||||
await task.settled
|
||||
}
|
||||
|
||||
/**
|
||||
* Queue one bounded recording; cancellation never leaves inference running after settlement.
|
||||
* Verified resources accept recordings while the worker wakes; other preparation states reject without downloading.
|
||||
* @param input - complete WAV and language hint.
|
||||
* @param signal - caller cancellation.
|
||||
* @returns recognized text; cancelled waiting jobs never acquire the worker.
|
||||
*/
|
||||
async transcribe(input: SpeechInput, signal: AbortSignal): Promise<Transcript> {
|
||||
return await this.enqueue(async combined => await this.execute(input, combined), signal)
|
||||
}
|
||||
|
||||
private async enqueue<T>(run: (signal: AbortSignal) => Promise<T>, signal: AbortSignal): Promise<T> {
|
||||
const combined = AbortSignal.any([signal, this.lifetime.signal])
|
||||
combined.throwIfAborted()
|
||||
if (this.pending >= this.config.maxPending) throw new Error('Speech transcription queue is full')
|
||||
clearTimeout(this.idle)
|
||||
this.pending++
|
||||
const job = this.tail.then(async () => {
|
||||
combined.throwIfAborted()
|
||||
return await run(combined)
|
||||
})
|
||||
this.tail = job.then(() => undefined, () => undefined).finally(() => {
|
||||
this.pending--
|
||||
if (this.pending === 0 && !this.lifetime.signal.aborted && this.config.idleTimeoutMs > 0) {
|
||||
this.idle = setTimeout(() => {
|
||||
this.tail = this.tail.then(async () => { await this.stop() })
|
||||
void this.tail.catch((error: unknown) => { this.ctx.logger.warn('Speech worker idle cleanup failed', error) })
|
||||
}, this.config.idleTimeoutMs)
|
||||
}
|
||||
})
|
||||
return await job
|
||||
}
|
||||
|
||||
private async start(signal: AbortSignal): Promise<Worker> {
|
||||
if (this.worker?.closed) await this.stop()
|
||||
if (this.worker) return this.worker
|
||||
using setup = deadline(signal, this.config.prepareTimeoutMs, 'SPEECH_PREPARE_TIMEOUT')
|
||||
const cached = this.runtime !== undefined
|
||||
if (!cached) this.publish({ phase: 'checking', step: 'check', startedAt: Date.now() })
|
||||
const runtime = this.runtime ?? await prepareRuntime(this.ctx, this.config, setup.signal, (state) => { this.publish(state) })
|
||||
this.runtime = runtime
|
||||
setup.signal.throwIfAborted()
|
||||
this.publish({ phase: cached ? 'waking' : 'loading', step: 'load', startedAt: Date.now() })
|
||||
await mkdir(this.config.dataRoot, { recursive: true })
|
||||
const token = randomBytes(32).toString('hex')
|
||||
const handle = this.ctx.subprocess.spawn({
|
||||
argv: [process.execPath, ...runtime.worker.endsWith('.ts') ? ['--import', import.meta.resolve('tsx/esm')] : [], runtime.worker, JSON.stringify(Object.assign({}, this.config, runtime))],
|
||||
cwd: this.config.dataRoot, graceMs: this.config.graceMs,
|
||||
env: { DSH_SPEECH_TOKEN: token, ELECTRON_RUN_AS_NODE: '1' },
|
||||
stdio: { stdin: 'ignore', stdout: 'pipe', stderr: { maxBytes: this.config.maxLogBytes } },
|
||||
})
|
||||
try {
|
||||
const port = await readReady(handle, this.config.maxLogBytes, setup.signal)
|
||||
const worker = { handle, url: `http://127.0.0.1:${port}`, token, closed: false }
|
||||
this.worker = worker
|
||||
this.publish({ phase: 'ready' })
|
||||
const exited = (): void => {
|
||||
if (!worker.closed) { worker.closed = true; this.publish({ phase: 'standby' }) }
|
||||
}
|
||||
void handle.done.then(exited, exited)
|
||||
return worker
|
||||
} catch (error) {
|
||||
handle.terminate()
|
||||
await handle.waitForExit()
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
private async execute(input: SpeechInput, signal: AbortSignal): Promise<Transcript> {
|
||||
const phase = this.state.phase
|
||||
if (phase !== 'ready' && phase !== 'standby') throw new Error('Prepare the local speech provider before recording')
|
||||
try {
|
||||
const worker = await this.start(signal)
|
||||
using call = deadline(signal, this.config.inferenceTimeoutMs, 'SPEECH_INFERENCE_TIMEOUT')
|
||||
const response = await fetch(`${worker.url}/transcribe?language=${encodeURIComponent(input.language)}`, {
|
||||
method: 'POST', headers: { authorization: `Bearer ${worker.token}`, 'content-type': 'audio/wav' },
|
||||
body: Buffer.from(input.audio), signal: call.signal,
|
||||
})
|
||||
const result = await readTranscript(response, this.config.maxResponseBytes).catch((error: unknown) => {
|
||||
call.signal.throwIfAborted()
|
||||
throw error
|
||||
})
|
||||
call.signal.throwIfAborted()
|
||||
return result
|
||||
} catch (error) {
|
||||
if (error instanceof SpeechInputError) throw error
|
||||
await this.stop()
|
||||
this.publish({ phase: 'standby' })
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
private async stop(): Promise<void> {
|
||||
const worker = this.worker
|
||||
if (!worker) return
|
||||
worker.closed = true
|
||||
worker.handle.terminate()
|
||||
await worker.handle.waitForExit()
|
||||
this.worker = undefined
|
||||
this.publish({ phase: 'standby' })
|
||||
}
|
||||
|
||||
/** Stop the local recognizer. @returns after admission closes, queued jobs settle, and the managed worker exits. */
|
||||
async dispose(): Promise<void> {
|
||||
this.listeners.clear()
|
||||
this.lifetime.abort(new Error('SenseVoice provider disposed'))
|
||||
clearTimeout(this.idle)
|
||||
try { await this.tail } finally { await this.stop() }
|
||||
await this.preparing?.settled
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
/** Verified runtime assets and managed command execution for local transcription. */
|
||||
import { createHash, randomUUID } from 'node:crypto'
|
||||
import { createReadStream, readFileSync } from 'node:fs'
|
||||
import { access, mkdir, rename, rm, stat } from 'node:fs/promises'
|
||||
import { join } from 'node:path'
|
||||
import { Transform } from 'node:stream'
|
||||
import { pipeline } from 'node:stream/promises'
|
||||
import { createWriteStream } from 'node:fs'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import type { Context } from '@deepseek-ai/cordis'
|
||||
import type {} from '@deepseek-ai/dsh-subprocess'
|
||||
import type { Config } from './config.ts'
|
||||
import type { SpeechPreparationState } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
|
||||
/** Release-pinned downloadable file. */
|
||||
export interface Asset {
|
||||
readonly name: string
|
||||
readonly url: string
|
||||
readonly sha256: string
|
||||
readonly bytes: number
|
||||
}
|
||||
|
||||
interface RuntimeLock {
|
||||
models: Record<Config['precision'], Asset>
|
||||
tokens: Asset
|
||||
vad: Asset
|
||||
}
|
||||
|
||||
/** Prepared model and process paths, private to the local provider. */
|
||||
export interface RuntimePaths {
|
||||
readonly model: string
|
||||
readonly tokens: string
|
||||
readonly vad: string
|
||||
readonly worker: string
|
||||
}
|
||||
|
||||
async function matchesAsset(path: string, asset: Asset, signal: AbortSignal): Promise<boolean> {
|
||||
signal.throwIfAborted()
|
||||
try {
|
||||
const info = await stat(path)
|
||||
if (!info.isFile()) throw new Error(`Speech asset is not a regular file: ${path}`)
|
||||
if (info.size !== asset.bytes) return false
|
||||
const digest = createHash('sha256')
|
||||
for await (const chunk of createReadStream(path, { signal })) digest.update(chunk as Buffer)
|
||||
return digest.digest('hex') === asset.sha256
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
function resolveRuntime(config: Config) {
|
||||
const supported = ['darwin-arm64', 'darwin-x64', 'linux-arm64', 'linux-x64', 'win32-x64']
|
||||
if (!supported.includes(`${process.platform}-${process.arch}`)) throw new Error(`Local speech is unavailable for ${process.platform}-${process.arch}`)
|
||||
const lock = JSON.parse(readFileSync(new URL('../runtime/assets.json', import.meta.url), 'utf8')) as RuntimeLock
|
||||
const modelRoot = config.modelDirectory ?? join(config.dataRoot, 'models', 'sensevoice-onnx')
|
||||
const paths: RuntimePaths = {
|
||||
model: join(modelRoot, lock.models[config.precision].name), tokens: join(modelRoot, lock.tokens.name),
|
||||
vad: config.vadModelPath ?? join(config.dataRoot, 'models', 'silero', lock.vad.name),
|
||||
/* v8 ignore next -- built worker resolution is exercised by real Node and Electron process smokes */
|
||||
worker: fileURLToPath(new URL(import.meta.url.endsWith('.ts') ? './worker.ts' : './worker.js', import.meta.url)),
|
||||
}
|
||||
return { lock, modelRoot, paths }
|
||||
}
|
||||
|
||||
async function verifyRuntime(config: Config, paths: RuntimePaths, lock: RuntimeLock, signal: AbortSignal): Promise<boolean> {
|
||||
signal.throwIfAborted()
|
||||
try { await Promise.all([paths.model, paths.tokens, paths.vad].map(path => access(path))) }
|
||||
catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error
|
||||
return false
|
||||
}
|
||||
const verified = [
|
||||
...config.modelDirectory === undefined ? [[paths.model, lock.models[config.precision]], [paths.tokens, lock.tokens]] as const : [],
|
||||
...config.vadModelPath === undefined ? [[paths.vad, lock.vad]] as const : [],
|
||||
]
|
||||
for (const [path, asset] of verified) if (!await matchesAsset(path, asset, signal)) return false
|
||||
signal.throwIfAborted()
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* Inspect existing models without downloading, writing files or starting the worker.
|
||||
* @param config - model paths and selected precision.
|
||||
* @param signal - provider cancellation or inspection deadline.
|
||||
* @returns cached paths when all files exist and managed assets match their pinned size and hash; otherwise undefined.
|
||||
*/
|
||||
export async function inspectRuntime(config: Config, signal: AbortSignal): Promise<RuntimePaths | undefined> {
|
||||
const { lock, paths } = resolveRuntime(config)
|
||||
return await verifyRuntime(config, paths, lock, signal) ? paths : undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Download into a unique partial file, verify, then publish it atomically.
|
||||
* @param asset - pinned release identity.
|
||||
* @param root - provider-owned cache directory.
|
||||
* @param signal - preparation cancellation.
|
||||
* @param report - Host-owned progress publisher.
|
||||
* @returns verified local file path.
|
||||
*/
|
||||
export async function downloadAsset(asset: Asset, root: string, signal: AbortSignal,
|
||||
report: (state: SpeechPreparationState) => void = () => {}): Promise<string> {
|
||||
signal.throwIfAborted()
|
||||
await mkdir(root, { recursive: true })
|
||||
const destination = join(root, asset.name)
|
||||
if (await matchesAsset(destination, asset, signal)) return destination
|
||||
const partial = `${destination}.${randomUUID()}.part`
|
||||
try {
|
||||
const response = await fetch(asset.url, { signal })
|
||||
if (!response.ok || !response.body) throw new Error(`Speech asset download failed: HTTP ${response.status}`)
|
||||
const digest = createHash('sha256')
|
||||
let completedBytes = 0
|
||||
const publish = (): void => { report({ phase: 'downloading', resource: asset.name, completedBytes, totalBytes: asset.bytes }) }
|
||||
publish()
|
||||
const hashing = new Transform({ transform(chunk: Buffer, _encoding, callback) {
|
||||
completedBytes += chunk.length
|
||||
if (completedBytes > asset.bytes) { callback(new Error(`Speech asset exceeds its pinned size: ${asset.name}`)); return }
|
||||
digest.update(chunk); publish(); callback(null, chunk)
|
||||
} })
|
||||
await pipeline(response.body, hashing, createWriteStream(partial, { flags: 'wx', mode: 0o600 }), { signal })
|
||||
if (completedBytes !== asset.bytes || digest.digest('hex') !== asset.sha256) {
|
||||
throw new Error(`Speech asset checksum or size mismatch: ${asset.name}`)
|
||||
}
|
||||
signal.throwIfAborted()
|
||||
await rename(partial, destination)
|
||||
return destination
|
||||
} finally {
|
||||
await rm(partial, { force: true })
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the bundled native runtime and prepare verified ONNX models on demand.
|
||||
* @param _ctx - Host context owning the preparation task.
|
||||
* @param config - model paths, precision, and download origin.
|
||||
* @param signal - preparation cancellation or deadline.
|
||||
* @param report - Host-owned progress publisher.
|
||||
* @returns verified model and worker paths.
|
||||
*/
|
||||
export async function prepareRuntime(_ctx: Context, config: Config, signal: AbortSignal,
|
||||
report: (state: SpeechPreparationState) => void = () => {}): Promise<RuntimePaths> {
|
||||
const { lock, modelRoot, paths } = resolveRuntime(config)
|
||||
const download = async (asset: Asset, root: string, step: 'model' | 'vad'): Promise<void> => {
|
||||
const url = new URL(asset.url)
|
||||
const pinned = { ...asset, url: `${config.modelOrigin.replace(/\/$/, '')}${url.pathname}` }
|
||||
await downloadAsset(pinned, root, signal, (state) => { report({ ...state, step }) })
|
||||
}
|
||||
if (config.modelDirectory === undefined) {
|
||||
report({ phase: 'checking', step: 'model', startedAt: Date.now() })
|
||||
await download(lock.models[config.precision], modelRoot, 'model')
|
||||
await download(lock.tokens, modelRoot, 'model')
|
||||
}
|
||||
if (config.vadModelPath === undefined) {
|
||||
report({ phase: 'checking', step: 'vad', startedAt: Date.now() })
|
||||
await download(lock.vad, join(config.dataRoot, 'models', 'silero'), 'vad')
|
||||
}
|
||||
report({ phase: 'checking', step: 'verify', startedAt: Date.now() })
|
||||
if (!await verifyRuntime(config, paths, lock, signal)) throw new Error('Speech model verification failed: missing or corrupted model files')
|
||||
return paths
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
/** Private child entry; the Host owns termination, and stdout carries only readiness. */
|
||||
import { z } from 'zod'
|
||||
import { Config } from './config.ts'
|
||||
import { createTranscriber } from './inference.ts'
|
||||
import { startRecognitionServer } from './process-server.ts'
|
||||
|
||||
const raw: unknown = JSON.parse(z.string().parse(process.argv[2]))
|
||||
const paths = z.object({ model: z.string().min(1), tokens: z.string().min(1), vad: z.string().min(1) }).parse(raw)
|
||||
const config = Object.assign(Config(z.record(z.string(), z.unknown()).parse(raw)), paths)
|
||||
const token = z.string().regex(/^[a-f0-9]{64}$/).parse(process.env.DSH_SPEECH_TOKEN)
|
||||
delete process.env.DSH_SPEECH_TOKEN
|
||||
const { port } = await startRecognitionServer(token, config.maxAudioBytes, createTranscriber(config))
|
||||
process.stdout.write(`${JSON.stringify({ port })}\n`)
|
||||
@@ -0,0 +1,55 @@
|
||||
/** Native inference preserves language hints, bounded segments, and Electron-safe buffers. */
|
||||
import { expect, it, vi } from 'vitest'
|
||||
import { Config } from '../src/config.ts'
|
||||
import { createTranscriber } from '../src/inference.ts'
|
||||
|
||||
const native = vi.hoisted(() => ({ speech: true, texts: ['第一句。', '第二句。'], configs: [] as object[],
|
||||
front: vi.fn(), resets: vi.fn(), accepted: [] as number[], windows: [] as number[] }))
|
||||
vi.mock('node:module', () => {
|
||||
return { createRequire: () => () => ({
|
||||
OfflineRecognizer: class {
|
||||
constructor(config: object) { native.configs.push(structuredClone(config)) }
|
||||
setConfig(config: object) { native.configs.push(structuredClone(config)) }
|
||||
createStream() { return { acceptWaveform: ({ samples }: { samples: Float32Array }) => { native.accepted.push(samples.length) } } }
|
||||
decode() {}
|
||||
getResult() { return { text: native.texts.shift() ?? '' } }
|
||||
},
|
||||
Vad: class {
|
||||
ready = false
|
||||
constructor(config: object) { native.configs.push(structuredClone(config)) }
|
||||
acceptWaveform(samples: Float32Array) { native.windows.push(samples.length); this.ready = native.speech }
|
||||
isEmpty() { return !this.ready }
|
||||
front(externalBuffer: false) { native.front(externalBuffer); return { samples: new Float32Array(512) } }
|
||||
pop() { this.ready = false }
|
||||
reset() { native.resets(); this.ready = false }
|
||||
flush() {}
|
||||
},
|
||||
}) }
|
||||
})
|
||||
function wave(samples = 1024): Uint8Array {
|
||||
const b = Buffer.alloc(44 + samples * 2)
|
||||
b.write('RIFF'); b.writeUInt32LE(b.length - 8, 4); b.write('WAVEfmt ', 8); b.writeUInt32LE(16, 16)
|
||||
b.writeUInt16LE(1, 20); b.writeUInt16LE(1, 22); b.writeUInt32LE(16000, 24); b.writeUInt32LE(32000, 28)
|
||||
b.writeUInt16LE(2, 32); b.writeUInt16LE(16, 34); b.write('data', 36); b.writeUInt32LE(samples * 2, 40)
|
||||
return b
|
||||
}
|
||||
it('loads one model pair, carries language changes, and copies VAD buffers for Electron', () => {
|
||||
const transcribe = createTranscriber(Object.assign(Config({ dataRoot: '/cache' }), { model: '/model', tokens: '/tokens', vad: '/vad' }))
|
||||
const result = transcribe(wave(), 'zh')
|
||||
expect(result).toMatchObject({ text: '第一句。 第二句。', audioSeconds: 1024 / 16000 })
|
||||
expect(result.inferenceSeconds).toBeGreaterThanOrEqual(0)
|
||||
expect(native.accepted).toEqual([512, 512])
|
||||
expect(native.front).toHaveBeenCalledWith(false)
|
||||
expect(native.configs.at(-1)).toMatchObject({ modelConfig: { senseVoice: { language: 'zh', useInverseTextNormalization: 1 } } })
|
||||
native.speech = false
|
||||
expect(transcribe(wave(100), 'en').text).toBe('')
|
||||
expect(native.configs.at(-1)).toMatchObject({ modelConfig: { senseVoice: { language: 'en' } } })
|
||||
expect(native.resets).toHaveBeenCalledTimes(2)
|
||||
expect(native.windows.at(-1)).toBe(100)
|
||||
})
|
||||
it('rejects unsupported language, malformed audio and recordings above the worker limit', () => {
|
||||
const transcribe = createTranscriber(Object.assign(Config({ dataRoot: '/cache', maxAudioBytes: 64 }), { model: '/model', tokens: '/tokens', vad: '/vad' }))
|
||||
expect(() => transcribe(wave(1), 'invalid')).toThrow('language')
|
||||
expect(() => transcribe(new Uint8Array(46), 'zh')).toThrow('WAV')
|
||||
expect(() => transcribe(wave(), 'zh')).toThrow('byte limit')
|
||||
})
|
||||
@@ -0,0 +1,145 @@
|
||||
/** ONNX preparation reuses verified files and honors explicit offline deployments. */
|
||||
import { writeFileSync } from 'node:fs'
|
||||
import { createHash } from 'node:crypto'
|
||||
import { access, mkdtemp, mkdir, readFile, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import LocalSubprocess from '@deepseek-ai/dsh-subprocess-local'
|
||||
import SpeechToText from '@deepseek-ai/dsh-experimental-speech-to-text'
|
||||
import { afterEach, expect, it, vi } from 'vitest'
|
||||
import type { SpeechPreparationState } from '@deepseek-ai/dsh-experimental-speech-to-text/types'
|
||||
import { inspectRuntime, prepareRuntime, type Asset } from '../src/runtime.ts'
|
||||
import { Config } from '../src/config.ts'
|
||||
import * as Provider from '../src/index.ts'
|
||||
|
||||
const bytes = Buffer.from('pinned ONNX fixture')
|
||||
const asset = (name: string): Asset => ({ name, url: `https://huggingface.co/owner/model/resolve/revision/${name}`,
|
||||
bytes: bytes.length, sha256: createHash('sha256').update(bytes).digest('hex') })
|
||||
const fake = vi.hoisted(() => ({ lock: {} }))
|
||||
vi.mock('node:fs/promises', async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import('node:fs/promises')>()
|
||||
return { ...actual, access: vi.fn(actual.access) }
|
||||
})
|
||||
vi.mock('node:fs', async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import('node:fs')>()
|
||||
return { ...actual, readFileSync: (...args: Parameters<typeof actual.readFileSync>) => {
|
||||
if (args[0] instanceof URL && args[0].pathname.endsWith('/runtime/assets.json')) return JSON.stringify(fake.lock)
|
||||
return actual.readFileSync(...args)
|
||||
} }
|
||||
})
|
||||
const roots: string[] = []
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks()
|
||||
vi.unstubAllGlobals()
|
||||
for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
it('restores verified cached models on every provider activation without downloads or a worker', async () => {
|
||||
const { root, fetcher } = await fixture(), config = Config({ dataRoot: root })
|
||||
await prepareRuntime(new Context(), config, new AbortController().signal)
|
||||
fetcher.mockClear()
|
||||
const ctx = new Context()
|
||||
try {
|
||||
await ctx.plugin(LocalSubprocess)
|
||||
await ctx.plugin(SpeechToText, { defaultProvider: 'sensevoice-local', language: 'auto' })
|
||||
const spawn = vi.spyOn(ctx.get('subprocess')!, 'spawn'), speech = ctx.get('speechToText')!
|
||||
for (let activation = 0; activation < 2; activation++) {
|
||||
const fiber = ctx.plugin(Provider, config)
|
||||
await fiber
|
||||
await vi.waitFor(() => { expect(speech.snapshot().providers[0]?.preparation.phase).toBe('standby') })
|
||||
expect(speech.snapshot().providers[0]?.preparation.steps?.map(({ kind, status }) => [kind, status])).toEqual([
|
||||
['check', 'complete'], ['model', 'complete'], ['vad', 'complete'], ['verify', 'complete'], ['load', 'pending'],
|
||||
])
|
||||
expect(fetcher).not.toHaveBeenCalled()
|
||||
expect(spawn).not.toHaveBeenCalled()
|
||||
await fiber.dispose()
|
||||
expect(speech.snapshot().providers).toEqual([])
|
||||
}
|
||||
} finally { await ctx.fiber.dispose() }
|
||||
})
|
||||
|
||||
it.each(['model', 'tokens', 'vad'] as const)('requires preparation for a missing or corrupted cached %s', async (file) => {
|
||||
const { root, fetcher } = await fixture(), config = Config({ dataRoot: root }), signal = new AbortController().signal
|
||||
const paths = await prepareRuntime(new Context(), config, signal)
|
||||
fetcher.mockClear()
|
||||
expect(await inspectRuntime(config, signal)).toEqual(paths)
|
||||
await writeFile(paths[file], Buffer.alloc(bytes.length))
|
||||
expect(await inspectRuntime(config, signal)).toBeUndefined()
|
||||
await rm(paths[file])
|
||||
expect(await inspectRuntime(config, signal)).toBeUndefined()
|
||||
expect(fetcher).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('inspects an empty cache without creating it and respects precision and offline paths', async () => {
|
||||
const { root, fetcher } = await fixture(), signal = new AbortController().signal
|
||||
const missingRoot = join(root, 'absent')
|
||||
expect(await inspectRuntime(Config({ dataRoot: missingRoot }), signal)).toBeUndefined()
|
||||
await expect(readFile(missingRoot)).rejects.toThrow()
|
||||
expect(fetcher).not.toHaveBeenCalled()
|
||||
const config = Config({ dataRoot: root }), paths = await prepareRuntime(new Context(), config, signal)
|
||||
expect(await inspectRuntime(Config(Object.assign({}, config, { precision: 'fp32' })), signal)).toBeUndefined()
|
||||
expect(await inspectRuntime(Config(Object.assign({}, config, { modelDirectory: join(root, 'models', 'sensevoice-onnx'), vadModelPath: paths.vad })), signal)).toEqual(paths)
|
||||
})
|
||||
|
||||
it('reports filesystem access errors instead of treating unreadable caches as absent', async () => {
|
||||
const { root, fetcher } = await fixture()
|
||||
const failure = Object.assign(new Error('cache is not readable'), { code: 'EACCES' })
|
||||
vi.mocked(access).mockRejectedValueOnce(failure)
|
||||
await expect(inspectRuntime(Config({ dataRoot: root }), new AbortController().signal)).rejects.toBe(failure)
|
||||
expect(fetcher).not.toHaveBeenCalled()
|
||||
})
|
||||
async function fixture() {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-speech-onnx-')); roots.push(root)
|
||||
fake.lock = { models: { int8: asset('model.int8.onnx'), fp32: asset('model.onnx') }, tokens: asset('tokens.txt'), vad: asset('silero_vad.onnx') }
|
||||
const fetcher = vi.fn(async (_url: string) => new Response(bytes)); vi.stubGlobal('fetch', fetcher)
|
||||
return { root, fetcher }
|
||||
}
|
||||
it.each(['darwin-arm64', 'darwin-x64', 'linux-arm64', 'linux-x64', 'win32-x64'])('prepares only model files on %s', async (target) => {
|
||||
const { root, fetcher } = await fixture(), [platform, arch] = target.split('-')
|
||||
vi.stubGlobal('process', { ...process, platform, arch })
|
||||
const report = vi.fn<(state: SpeechPreparationState) => void>()
|
||||
const config = Config({ dataRoot: root, modelOrigin: 'https://mirror.example' })
|
||||
const runtime = await prepareRuntime(new Context(), config, new AbortController().signal, report)
|
||||
expect(await readFile(runtime.model)).toEqual(bytes)
|
||||
expect(runtime.tokens).toContain('tokens.txt')
|
||||
expect(runtime.vad).toContain('silero_vad.onnx')
|
||||
expect(report.mock.calls.map(([state]) => state.step)).toEqual(expect.arrayContaining(['model', 'vad', 'verify']))
|
||||
expect(report.mock.calls.map(([state]) => state.phase)).not.toContain('installing')
|
||||
expect(fetcher.mock.calls.every(args => args[0].startsWith('https://mirror.example/'))).toBe(true)
|
||||
await prepareRuntime(new Context(), config, new AbortController().signal)
|
||||
expect(fetcher).toHaveBeenCalledTimes(3)
|
||||
})
|
||||
it('selects FP32 explicitly and skips downloads for existing model paths', async () => {
|
||||
const { root, fetcher } = await fixture()
|
||||
const config = Config({ dataRoot: root, precision: 'fp32' })
|
||||
const runtime = await prepareRuntime(new Context(), config, new AbortController().signal)
|
||||
expect(runtime.model).toMatch(/model\.onnx$/)
|
||||
fetcher.mockClear()
|
||||
const offline = await prepareRuntime(new Context(), Config(Object.assign({}, config, { modelDirectory: join(root, 'models', 'sensevoice-onnx'),
|
||||
vadModelPath: runtime.vad })), new AbortController().signal)
|
||||
expect(offline).toEqual(runtime)
|
||||
expect(fetcher).not.toHaveBeenCalled()
|
||||
await rm(runtime.model)
|
||||
const missing = Config(Object.assign({}, config, { modelDirectory: root, vadModelPath: runtime.vad }))
|
||||
await expect(prepareRuntime(new Context(), missing, new AbortController().signal)).rejects.toThrow()
|
||||
})
|
||||
it('rejects unsupported platforms, invalid model paths and cancelled preparation', async () => {
|
||||
const { root } = await fixture()
|
||||
const config = Config({ dataRoot: root })
|
||||
vi.stubGlobal('process', { ...process, platform: 'win32', arch: 'arm64' })
|
||||
await expect(prepareRuntime(new Context(), config, new AbortController().signal)).rejects.toThrow('unavailable')
|
||||
vi.unstubAllGlobals()
|
||||
await mkdir(join(root, 'models', 'sensevoice-onnx', 'model.int8.onnx'), { recursive: true })
|
||||
await expect(prepareRuntime(new Context(), config, new AbortController().signal)).rejects.toThrow()
|
||||
await writeFile(join(root, 'model.int8.onnx'), bytes); await writeFile(join(root, 'tokens.txt'), bytes)
|
||||
await expect(prepareRuntime(new Context(), Config({ dataRoot: root, modelDirectory: root, vadModelPath: join(root, 'tokens.txt') }),
|
||||
AbortSignal.abort(new Error('cancelled')))).rejects.toThrow('cancelled')
|
||||
})
|
||||
|
||||
it('rejects cached files changed before the final verification stage', async () => {
|
||||
const { root } = await fixture()
|
||||
await expect(prepareRuntime(new Context(), Config({ dataRoot: root }), new AbortController().signal, (state) => {
|
||||
if (state.step === 'verify') writeFileSync(join(root, 'models', 'sensevoice-onnx', 'model.int8.onnx'), 'changed')
|
||||
})).rejects.toThrow('verification failed')
|
||||
})
|
||||
@@ -0,0 +1,38 @@
|
||||
/** Explicit opt-in real SenseVoice inference, including the managed runtime when no overrides are supplied. */
|
||||
import { readFile } from 'node:fs/promises'
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import LocalSubprocess from '@deepseek-ai/dsh-subprocess-local'
|
||||
import { expect, it, vi } from 'vitest'
|
||||
import { Config } from '../src/config.ts'
|
||||
import { SenseVoiceWorker } from '../src/recognizer.ts'
|
||||
|
||||
const { DSH_SPEECH_E2E_ROOT: dataRoot, DSH_SPEECH_E2E_AUDIO: audioPath } = process.env
|
||||
|
||||
it.skipIf(!dataRoot || !audioPath)('prepares a real local recognizer, transcribes speech, and releases it', { timeout: 3_660_000, retry: 0 }, async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LocalSubprocess)
|
||||
const spawn = vi.spyOn(ctx.get('subprocess')!, 'spawn')
|
||||
const worker = new SenseVoiceWorker(ctx, Config({ dataRoot: dataRoot!,
|
||||
modelDirectory: process.env.DSH_SPEECH_E2E_MODEL,
|
||||
vadModelPath: process.env.DSH_SPEECH_E2E_VAD,
|
||||
precision: process.env.DSH_SPEECH_E2E_PRECISION === 'fp32' ? 'fp32' : 'int8',
|
||||
}))
|
||||
try {
|
||||
worker.prepare()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('ready') }, { timeout: 3_600_000 })
|
||||
const input = { audio: await readFile(audioPath!), language: 'zh' }
|
||||
const cold = await worker.transcribe(input, new AbortController().signal)
|
||||
const warm = await worker.transcribe(input, new AbortController().signal)
|
||||
expect(cold.text.length).toBeGreaterThan(0)
|
||||
expect(warm.text).toBe(cold.text)
|
||||
expect(warm.audioSeconds).toBeGreaterThan(0)
|
||||
await expect(worker.transcribe({ ...input, language: 'fr' }, new AbortController().signal)).rejects.toThrow('Unsupported SenseVoice language')
|
||||
expect((await worker.transcribe(input, new AbortController().signal)).text).toBe(warm.text)
|
||||
expect(spawn).toHaveBeenCalledOnce()
|
||||
console.info('SenseVoice inference', { cold, warm })
|
||||
} finally {
|
||||
await worker.dispose()
|
||||
await ctx.fiber.dispose()
|
||||
spawn.mockRestore()
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,36 @@
|
||||
/** The private recognizer admits only authenticated bounded audio requests. */
|
||||
import { expect, it, onTestFinished, vi } from 'vitest'
|
||||
import { SpeechInputError } from '../src/input.ts'
|
||||
import { startRecognitionServer } from '../src/process-server.ts'
|
||||
|
||||
it('serves transcripts on an ephemeral loopback port and rejects unauthorized or invalid requests', async () => {
|
||||
const transcribe = vi.fn(() => ({ text: 'hello', audioSeconds: 1, inferenceSeconds: 0.1 }))
|
||||
const { server, port } = await startRecognitionServer('private-token', 100, transcribe)
|
||||
onTestFinished(async () => {
|
||||
server.closeAllConnections()
|
||||
await new Promise<void>((resolve, reject) => { server.close((error) => { if (error) reject(error); else resolve() }) })
|
||||
})
|
||||
const base = `http://127.0.0.1:${port}`
|
||||
const headers = { authorization: 'Bearer private-token' }, body = Buffer.alloc(46)
|
||||
expect((await fetch(base)).status).toBe(401)
|
||||
expect((await fetch(base, { headers: { authorization: 'Bearer another-token' } })).status).toBe(401)
|
||||
expect((await fetch(`${base}/missing`, { headers })).status).toBe(404)
|
||||
expect((await fetch(`${base}/transcribe`, { method: 'POST', headers, body: Buffer.alloc(101) })).status).toBe(413)
|
||||
expect((await fetch(`${base}/transcribe`, { method: 'POST', headers, body: Buffer.alloc(1) })).status).toBe(413)
|
||||
const good = await fetch(`${base}/transcribe?language=zh`, { method: 'POST', headers, body })
|
||||
expect(await good.json()).toEqual({ text: 'hello', audioSeconds: 1, inferenceSeconds: 0.1 })
|
||||
expect(transcribe).toHaveBeenLastCalledWith(body, 'zh')
|
||||
transcribe.mockImplementationOnce(() => { throw new SpeechInputError('bad audio') })
|
||||
const bad = await fetch(`${base}/transcribe`, { method: 'POST', headers, body })
|
||||
expect(bad.status).toBe(400)
|
||||
expect(await bad.json()).toEqual({ error: 'bad audio', code: 'invalid-input' })
|
||||
expect(transcribe).toHaveBeenLastCalledWith(body, 'auto')
|
||||
transcribe.mockImplementationOnce(() => { throw 'native failure' })
|
||||
const failure = await fetch(`${base}/transcribe`, { method: 'POST', headers, body })
|
||||
expect(failure.status).toBe(500)
|
||||
expect(await failure.json()).toEqual({ error: 'native failure' })
|
||||
transcribe.mockImplementationOnce(() => { throw new Error('native decode failed') })
|
||||
const nativeFailure = await fetch(`${base}/transcribe`, { method: 'POST', headers, body })
|
||||
expect(nativeFailure.status).toBe(500)
|
||||
expect(await nativeFailure.json()).toEqual({ error: 'native decode failed' })
|
||||
})
|
||||
@@ -0,0 +1,84 @@
|
||||
/** Private worker readiness and response parsing stop on invalid process/wire output. */
|
||||
import { PassThrough } from 'node:stream'
|
||||
import type { SubprocessHandle } from '@deepseek-ai/dsh-subprocess'
|
||||
import { expect, it } from 'vitest'
|
||||
import { SpeechInputError } from '../src/input.ts'
|
||||
import { readReady, readTranscript } from '../src/recognizer.ts'
|
||||
|
||||
function child() {
|
||||
const stdout = new PassThrough()
|
||||
const done = Promise.withResolvers<{ exitCode: number; signal: null }>()
|
||||
const handle: SubprocessHandle = { stdout, stdin: undefined, stderr: undefined, control: undefined,
|
||||
done: done.promise, collected: {}, terminate: () => {}, waitForExit: async () => true }
|
||||
return { stdout, done, handle }
|
||||
}
|
||||
|
||||
it('reads a fragmented readiness frame then detaches its listener', async () => {
|
||||
const c = child()
|
||||
const pending = readReady(c.handle, 100, new AbortController().signal)
|
||||
c.stdout.write('{"port":'); c.stdout.write('12345}\n')
|
||||
expect(await pending).toBe(12345)
|
||||
expect(c.stdout.listenerCount('data')).toBe(0)
|
||||
c.done.resolve({ exitCode: 0, signal: null }); c.stdout.destroy()
|
||||
})
|
||||
|
||||
it('rejects malformed, oversized and invalid-port readiness frames', async () => {
|
||||
for (const frame of ['no JSON\n', '{"port":0}\n', '{"port":99,"extra":1}\n', 'x'.repeat(101)]) {
|
||||
const c = child()
|
||||
const pending = readReady(c.handle, 100, new AbortController().signal)
|
||||
const rejected = expect(pending).rejects.toThrow()
|
||||
c.stdout.write(frame)
|
||||
await rejected
|
||||
c.done.resolve({ exitCode: 0, signal: null }); c.stdout.destroy()
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects process exit, spawn failure, cancellation and unavailable output', async () => {
|
||||
const exit = child()
|
||||
const pending = readReady(exit.handle, 100, new AbortController().signal)
|
||||
exit.done.resolve({ exitCode: 1, signal: null })
|
||||
await expect(pending).rejects.toThrow('before readiness'); exit.stdout.destroy()
|
||||
const spawn = child()
|
||||
const failed = readReady(spawn.handle, 100, new AbortController().signal)
|
||||
spawn.done.reject(new Error('spawn failed'))
|
||||
await expect(failed).rejects.toThrow('spawn failed'); spawn.stdout.destroy()
|
||||
const c = child(), signal = new AbortController()
|
||||
const cancelled = readReady(c.handle, 100, signal.signal)
|
||||
signal.abort(new Error('cancelled'))
|
||||
await expect(cancelled).rejects.toThrow('cancelled')
|
||||
c.done.resolve({ exitCode: 0, signal: null }); c.stdout.destroy()
|
||||
await expect(readReady({ ...c.handle, stdout: undefined }, 100, signal.signal)).rejects.toThrow('unavailable')
|
||||
const before = child()
|
||||
await expect(readReady(before.handle, 100, AbortSignal.abort(new Error('already cancelled')))).rejects.toThrow('already cancelled')
|
||||
before.done.resolve({ exitCode: 0, signal: null }); before.stdout.destroy()
|
||||
})
|
||||
|
||||
it('validates complete worker transcripts and bounded failures', async () => {
|
||||
const value = { text: '测试', audioSeconds: 1, inferenceSeconds: 0.1 }
|
||||
expect(await readTranscript(Response.json(value), 100)).toEqual(value)
|
||||
await expect(readTranscript(new Response(null), 100)).rejects.toThrow('no response')
|
||||
await expect(readTranscript(Response.json(value), 1)).rejects.toThrow('byte limit')
|
||||
await expect(readTranscript(Response.json({ error: 'unsupported language' }, { status: 400 }), 100)).rejects.toThrow('unsupported language')
|
||||
await expect(readTranscript(Response.json({ ...value, inferenceSeconds: -1 }), 100)).rejects.toThrow()
|
||||
await expect(readTranscript(new Response('broken'), 100)).rejects.toThrow()
|
||||
})
|
||||
|
||||
it('contains stdout failures and primitive cancellation reasons while waiting for readiness', async () => {
|
||||
const c = child(), signal = new AbortController()
|
||||
const failed = readReady(c.handle, 100, signal.signal)
|
||||
c.stdout.emit('error', new Error('pipe closed'))
|
||||
await expect(failed).rejects.toThrow('pipe closed')
|
||||
expect(c.stdout.listenerCount('error')).toBe(0)
|
||||
c.done.resolve({ exitCode: 1, signal: null }); c.stdout.destroy()
|
||||
const cancelled = child(), abort = new AbortController()
|
||||
const pending = readReady(cancelled.handle, 100, abort.signal)
|
||||
abort.abort('closed')
|
||||
await expect(pending).rejects.toThrow('Speech operation cancelled')
|
||||
cancelled.done.resolve({ exitCode: 0, signal: null }); cancelled.stdout.destroy()
|
||||
})
|
||||
|
||||
it.each([400, 413, 500])('classifies marked input failures only on input status %s', async (status) => {
|
||||
const error = await readTranscript(Response.json({ error: 'rejected', code: 'invalid-input' }, { status }), 100).catch((value: unknown) => value)
|
||||
expect(error).toBeInstanceOf(Error)
|
||||
expect(error instanceof SpeechInputError).toBe(status !== 500)
|
||||
})
|
||||
@@ -0,0 +1,70 @@
|
||||
/** Verified models accept bounded concurrent recordings while the managed worker starts. */
|
||||
import { createServer } from 'node:http'
|
||||
import { mkdtemp, writeFile, rm, readFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import LocalSubprocess from '@deepseek-ai/dsh-subprocess-local'
|
||||
import SpeechToText from '@deepseek-ai/dsh-experimental-speech-to-text'
|
||||
import { afterEach, expect, it, vi } from 'vitest'
|
||||
import * as Provider from '../src/index.ts'
|
||||
import { SenseVoiceWorker } from '../src/recognizer.ts'
|
||||
import { inspectRuntime, prepareRuntime } from '../src/runtime.ts'
|
||||
vi.mock('../src/runtime.ts', () => ({ inspectRuntime: vi.fn(), prepareRuntime: vi.fn() }))
|
||||
afterEach(() => { vi.restoreAllMocks() })
|
||||
|
||||
it.each([false, true])('queues recordings during startup and honors waiting cancellation (%s)', async (cancelled) => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-speech-queue-')), ctx = new Context()
|
||||
const startupEntered = Promise.withResolvers<undefined>(), releaseStartup = Promise.withResolvers<undefined>()
|
||||
const gate = createServer((_request, response) => {
|
||||
startupEntered.resolve(undefined)
|
||||
void releaseStartup.promise.then(() => { response.end('ready') })
|
||||
})
|
||||
try {
|
||||
await new Promise<void>((resolve, reject) => { gate.once('error', reject); gate.listen(0, '127.0.0.1', resolve) })
|
||||
const address = gate.address()
|
||||
if (!address || typeof address === 'string') throw Error('Missing gate address')
|
||||
const worker = join(root, 'worker.mjs')
|
||||
const fixtureUrl = new URL('./worker.fixture.mjs', import.meta.url).href
|
||||
await writeFile(worker, `await fetch('http://127.0.0.1:${address.port}/');\nawait import(${JSON.stringify(fixtureUrl)});\n`)
|
||||
vi.mocked(inspectRuntime).mockResolvedValue({ tokens: root, model: root, vad: root, worker })
|
||||
await ctx.plugin(LocalSubprocess)
|
||||
await ctx.plugin(SpeechToText, { defaultProvider: 'sensevoice-local', language: 'auto' })
|
||||
const spawn = vi.spyOn(ctx.subprocess, 'spawn')
|
||||
const enqueue = vi.spyOn(SenseVoiceWorker.prototype, 'transcribe')
|
||||
await ctx.plugin(Provider, { dataRoot: root, idleTimeoutMs: 0, maxPending: 2 })
|
||||
const speech = ctx.get('speechToText')!
|
||||
await vi.waitFor(() => { expect(speech.snapshot().providers[0]?.preparation.phase).toBe('standby') })
|
||||
const firstSpec = speech.resolve({ audio: new Uint8Array([1, 2]), language: 'zh' })
|
||||
const secondSpec = speech.resolve({ audio: new Uint8Array([3, 4]), language: 'en' })
|
||||
const first = speech.transcribe(firstSpec, new AbortController().signal)
|
||||
const firstResult = first.then(value => ({ value }), (error: unknown) => ({ error }))
|
||||
try {
|
||||
await startupEntered.promise
|
||||
expect(speech.snapshot().providers[0]?.preparation.phase).toBe('waking')
|
||||
const waiting = new AbortController()
|
||||
const second = speech.transcribe(secondSpec, waiting.signal)
|
||||
const secondResult = second.then(value => ({ value }), (error: unknown) => ({ error }))
|
||||
await Promise.resolve()
|
||||
expect(enqueue).toHaveBeenCalledTimes(2)
|
||||
await expect(speech.transcribe(secondSpec, new AbortController().signal)).rejects.toThrow('queue is full')
|
||||
if (cancelled) waiting.abort(new Error('waiting recording cancelled'))
|
||||
releaseStartup.resolve(undefined)
|
||||
expect(await firstResult).toMatchObject({ value: { text: 'zh' } })
|
||||
if (cancelled) expect(await secondResult).toMatchObject({ error: { message: 'waiting recording cancelled' } })
|
||||
else expect(await secondResult).toMatchObject({ value: { text: 'en' } })
|
||||
expect(await readFile(join(root, 'requests'), 'utf8')).toBe(cancelled ? 'zh\n' : 'zh\nen\n')
|
||||
expect(spawn).toHaveBeenCalledOnce()
|
||||
expect(prepareRuntime).not.toHaveBeenCalled()
|
||||
} finally { releaseStartup.resolve(undefined); await firstResult }
|
||||
} finally {
|
||||
releaseStartup.resolve(undefined)
|
||||
try { await ctx.fiber.dispose() } finally {
|
||||
try {
|
||||
if (gate.listening) await new Promise<void>((resolve, reject) => {
|
||||
gate.close((error) => { if (error) reject(error); else resolve() })
|
||||
})
|
||||
} finally { await rm(root, { recursive: true, force: true }) }
|
||||
}
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,40 @@
|
||||
/** Provider activation remains lazy and rejects relative deployment paths. */
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import LocalSubprocess from '@deepseek-ai/dsh-subprocess-local'
|
||||
import SpeechToText from '@deepseek-ai/dsh-experimental-speech-to-text'
|
||||
import { afterEach, expect, it, vi } from 'vitest'
|
||||
import * as Provider from '../src/index.ts'
|
||||
import { SenseVoiceWorker } from '../src/recognizer.ts'
|
||||
|
||||
afterEach(() => { vi.restoreAllMocks() })
|
||||
|
||||
it.each(['int8', 'fp32'] as const)('registers %s lazily and joins inference before withdrawal completes', async (precision) => {
|
||||
const ctx = new Context()
|
||||
try {
|
||||
await ctx.plugin(LocalSubprocess)
|
||||
await ctx.plugin(SpeechToText, { defaultProvider: 'sensevoice-local', language: 'auto' })
|
||||
const transcribe = vi.spyOn(SenseVoiceWorker.prototype, 'transcribe')
|
||||
const fiber = ctx.plugin(Provider, { dataRoot: process.cwd(), precision })
|
||||
await fiber
|
||||
expect(transcribe).not.toHaveBeenCalled()
|
||||
const speech = ctx.get('speechToText')!
|
||||
expect(speech.listProviders()).toMatchObject([{ id: 'sensevoice-local', location: 'host-local', languages: ['auto', 'zh', 'en', 'yue', 'ja', 'ko'] }])
|
||||
expect(speech.listProviders()[0]?.setupEstimate).toEqual({
|
||||
recommendedDiskBytes: precision === 'int8' ? 1e9 : 2e9, expectedMemoryBytes: precision === 'int8' ? 1e9 : 2e9,
|
||||
minimumMinutes: 1, maximumMinutes: 10,
|
||||
})
|
||||
await expect(speech.transcribe(speech.resolve({ audio: new Uint8Array() }), new AbortController().signal)).rejects.toThrow('Prepare')
|
||||
transcribe.mockResolvedValue({ text: 'hello', audioSeconds: 1, inferenceSeconds: 0.1 })
|
||||
expect((await speech.transcribe(speech.resolve({ audio: new Uint8Array() }), new AbortController().signal)).text).toBe('hello')
|
||||
await fiber.dispose()
|
||||
expect(speech.listProviders()).toEqual([])
|
||||
} finally { await ctx.fiber.dispose() }
|
||||
})
|
||||
|
||||
it('refuses relative runtime, model and cache paths before registration', () => {
|
||||
for (const field of ['dataRoot', 'modelDirectory', 'vadModelPath']) {
|
||||
expect(() => { Provider.apply(new Context(), Provider.Config({ dataRoot: process.cwd(), [field]: 'relative' })) })
|
||||
.toThrow('absolute')
|
||||
}
|
||||
expect(() => Provider.Config({ dataRoot: process.cwd(), threads: 0 })).toThrow()
|
||||
})
|
||||
@@ -0,0 +1,62 @@
|
||||
/** Preparation publishes verified assets and joins failed or cancelled installer processes. */
|
||||
import { createHash } from 'node:crypto'
|
||||
import { mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { afterEach, expect, it, vi } from 'vitest'
|
||||
import { downloadAsset } from '../src/runtime.ts'
|
||||
|
||||
const cleanup: Array<() => Promise<void>> = []
|
||||
afterEach(async () => { vi.unstubAllGlobals(); for (const close of cleanup.splice(0).reverse()) await close() })
|
||||
async function fixture() {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-speech-runtime-'))
|
||||
cleanup.push(async () => { await rm(root, { recursive: true, force: true }) })
|
||||
return root
|
||||
}
|
||||
const bytes = Buffer.from('verified model')
|
||||
const asset = { name: 'model.bin', url: 'https://example.invalid/model', sha256: createHash('sha256').update(bytes).digest('hex'), bytes: bytes.length }
|
||||
const signal = (): AbortSignal => new AbortController().signal
|
||||
|
||||
it('publishes verified files, reuses them offline, and replaces corrupted cached files', async () => {
|
||||
const root = await fixture(), fetcher = vi.fn(async () => new Response(bytes))
|
||||
vi.stubGlobal('fetch', fetcher)
|
||||
const path = await downloadAsset(asset, root, signal())
|
||||
expect(await readFile(path)).toEqual(bytes)
|
||||
await downloadAsset(asset, root, signal())
|
||||
expect(fetcher).toHaveBeenCalledOnce()
|
||||
await writeFile(path, 'corrupt')
|
||||
await downloadAsset(asset, root, signal())
|
||||
expect(fetcher).toHaveBeenCalledTimes(2)
|
||||
expect(await readdir(root)).toEqual(['model.bin'])
|
||||
})
|
||||
|
||||
it('rejects altered downloads and HTTP failures without leaving partial files', async () => {
|
||||
const root = await fixture()
|
||||
vi.stubGlobal('fetch', async () => new Response('corrupt'))
|
||||
await expect(downloadAsset(asset, root, signal())).rejects.toThrow('checksum')
|
||||
vi.stubGlobal('fetch', async () => new Response(bytes))
|
||||
await expect(downloadAsset({ ...asset, sha256: 'bad-digest' }, root, signal())).rejects.toThrow('checksum')
|
||||
await expect(downloadAsset({ ...asset, bytes: 1 }, root, signal())).rejects.toThrow('pinned size')
|
||||
vi.stubGlobal('fetch', async () => new Response(null, { status: 503 }))
|
||||
await expect(downloadAsset(asset, root, signal())).rejects.toThrow('503')
|
||||
vi.stubGlobal('fetch', async () => new Response(null, { status: 200 }))
|
||||
await expect(downloadAsset(asset, root, signal())).rejects.toThrow('200')
|
||||
expect(await readdir(root)).toEqual([])
|
||||
})
|
||||
|
||||
it('cancels streaming downloads and refuses invalid cache locations', async () => {
|
||||
const root = await fixture(), cancel = new AbortController(), reading = Promise.withResolvers<undefined>()
|
||||
vi.stubGlobal('fetch', async () => new Response(new ReadableStream({
|
||||
start(controller) { cancel.signal.addEventListener('abort', () => { controller.error(cancel.signal.reason) }, { once: true }) },
|
||||
pull() { reading.resolve(undefined) },
|
||||
})))
|
||||
const pending = downloadAsset(asset, root, cancel.signal)
|
||||
const rejected = expect(pending).rejects.toThrow()
|
||||
await reading.promise; cancel.abort()
|
||||
await rejected
|
||||
expect(await readdir(root)).toEqual([])
|
||||
await expect(downloadAsset(asset, root, cancel.signal)).rejects.toThrow()
|
||||
const child = join(root, 'invalid')
|
||||
await writeFile(child, 'file')
|
||||
await expect(downloadAsset(asset, child, signal())).rejects.toThrow()
|
||||
})
|
||||
@@ -0,0 +1,2 @@
|
||||
/** Source-launch fixture shares the protocol peer used by built-launch tests. */
|
||||
import './worker.fixture.mjs'
|
||||
@@ -0,0 +1,23 @@
|
||||
/** Protocol fixture replacing the native recognizer, with real process and HTTP lifetimes. */
|
||||
import { createServer } from 'node:http'
|
||||
import { appendFileSync } from 'node:fs'
|
||||
import { join } from 'node:path'
|
||||
|
||||
const root = JSON.parse(process.argv[2]).model
|
||||
const server = createServer((request, response) => {
|
||||
if (request.headers.authorization !== `Bearer ${process.env.DSH_SPEECH_TOKEN}`) {
|
||||
response.writeHead(401).end(); return
|
||||
}
|
||||
const language = new URL(request.url, 'http://localhost').searchParams.get('language')
|
||||
appendFileSync(join(root, 'requests'), `${language}\n`)
|
||||
request.resume()
|
||||
request.on('end', () => {
|
||||
if (language === 'hold') return
|
||||
if (language === 'crash') { process.exit(1); return }
|
||||
if (language === 'invalid-input') { response.writeHead(400).end(JSON.stringify({ error: 'invalid input', code: 'invalid-input' })); return }
|
||||
if (language === 'error') { response.writeHead(400).end(JSON.stringify({ error: 'provider failed' })); return }
|
||||
response.setHeader('content-type', 'application/json')
|
||||
response.end(JSON.stringify({ text: language, audioSeconds: 1, inferenceSeconds: 0.1 }))
|
||||
})
|
||||
})
|
||||
server.listen(0, '127.0.0.1', () => { process.stdout.write(`${JSON.stringify({ port: server.address().port })}\n`) })
|
||||
@@ -0,0 +1,425 @@
|
||||
/** Local provider queueing, cancellation and process cleanup with real managed subprocesses. */
|
||||
import { mkdtemp, readFile, rm } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import type { SubprocessHandle } from '@deepseek-ai/dsh-subprocess'
|
||||
import LocalSubprocess from '@deepseek-ai/dsh-subprocess-local'
|
||||
import { afterEach, expect, it, vi } from 'vitest'
|
||||
import { Config } from '../src/config.ts'
|
||||
import { SenseVoiceWorker } from '../src/recognizer.ts'
|
||||
import { inspectRuntime, prepareRuntime } from '../src/runtime.ts'
|
||||
|
||||
vi.mock('../src/runtime.ts', () => ({ inspectRuntime: vi.fn(), prepareRuntime: vi.fn() }))
|
||||
const cleanup: Array<() => Promise<void>> = []
|
||||
afterEach(async () => { for (const close of cleanup.splice(0).reverse()) await close(); vi.clearAllMocks() })
|
||||
|
||||
async function fixture(overrides: Partial<Config> = {}) {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-speech-test-'))
|
||||
cleanup.push(async () => { await rm(root, { recursive: true, force: true }) })
|
||||
const ctx = new Context()
|
||||
cleanup.push(async () => { await ctx.fiber.dispose() })
|
||||
await ctx.plugin(LocalSubprocess)
|
||||
const spawn = vi.spyOn(ctx.get('subprocess')!, 'spawn')
|
||||
const config = Config({ dataRoot: root, idleTimeoutMs: 0, ...overrides })
|
||||
vi.mocked(prepareRuntime).mockImplementation(async () => ({ tokens: root,
|
||||
worker: fileURLToPath(new URL('./worker.fixture.mjs', import.meta.url)), model: root, vad: root }))
|
||||
const worker = new SenseVoiceWorker(ctx, config)
|
||||
cleanup.push(async () => { await worker.dispose() })
|
||||
return { root, worker, spawn, ctx }
|
||||
}
|
||||
|
||||
async function prepare(worker: SenseVoiceWorker): Promise<void> {
|
||||
const ready = Promise.withResolvers<undefined>()
|
||||
const unsubscribe = worker.subscribe(() => {
|
||||
const state = worker.snapshot()
|
||||
if (state.phase === 'ready') ready.resolve(undefined)
|
||||
if (state.phase === 'failed') ready.reject(new Error(state.message))
|
||||
})
|
||||
try { worker.prepare(); await ready.promise } finally { unsubscribe() }
|
||||
}
|
||||
|
||||
const audio = new Uint8Array([1, 2])
|
||||
const signal = (): AbortSignal => new AbortController().signal
|
||||
async function requests(root: string): Promise<string> {
|
||||
try { return await readFile(join(root, 'requests'), 'utf8') } catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error
|
||||
return ''
|
||||
}
|
||||
}
|
||||
|
||||
it('keeps one worker warm across recordings and joins its exit on disposal', async () => {
|
||||
const { worker, spawn } = await fixture()
|
||||
expect(spawn).not.toHaveBeenCalled()
|
||||
await prepare(worker)
|
||||
expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh')
|
||||
expect((await worker.transcribe({ audio, language: 'en' }, signal())).text).toBe('en')
|
||||
expect(spawn).toHaveBeenCalledOnce()
|
||||
const handle = spawn.mock.results[0]!.value as SubprocessHandle
|
||||
await worker.dispose()
|
||||
expect(await handle.waitForExit()).toBe(true)
|
||||
await expect(worker.transcribe({ audio, language: 'en' }, signal())).rejects.toThrow('disposed')
|
||||
})
|
||||
|
||||
it('wakes verified caches on the first recording without preparing them again', async () => {
|
||||
const { worker, spawn, root } = await fixture()
|
||||
const paths = { tokens: root, model: root, vad: root, worker: fileURLToPath(new URL('./worker.fixture.mjs', import.meta.url)) }
|
||||
vi.mocked(inspectRuntime).mockResolvedValueOnce(paths)
|
||||
const phases: string[] = []
|
||||
worker.subscribe(() => { phases.push(worker.snapshot().phase) })
|
||||
worker.inspect(); worker.inspect()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('standby') })
|
||||
expect(inspectRuntime).toHaveBeenCalledOnce()
|
||||
expect(spawn).not.toHaveBeenCalled()
|
||||
expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh')
|
||||
expect(phases).toContain('waking')
|
||||
expect(phases).not.toContain('loading')
|
||||
expect(prepareRuntime).not.toHaveBeenCalled()
|
||||
expect(spawn).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('keeps absent caches unprepared and reports unreadable caches as retryable failures', async () => {
|
||||
const { worker, spawn } = await fixture()
|
||||
vi.mocked(inspectRuntime).mockResolvedValueOnce(undefined)
|
||||
worker.inspect()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().steps?.[0]?.status).toBe('complete') })
|
||||
expect(worker.snapshot().phase).toBe('unprepared')
|
||||
expect(spawn).not.toHaveBeenCalled()
|
||||
expect(prepareRuntime).not.toHaveBeenCalled()
|
||||
vi.mocked(inspectRuntime).mockRejectedValueOnce(new Error('cache unreadable'))
|
||||
worker.inspect()
|
||||
await vi.waitFor(() => { expect(worker.snapshot()).toMatchObject({ phase: 'failed', message: 'cache unreadable' }) })
|
||||
worker.prepare()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('ready') }, { timeout: 10000 })
|
||||
expect(prepareRuntime).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('cancels cache inspection and joins it without late notifications when the provider unloads', async () => {
|
||||
const { worker } = await fixture(), entered = Promise.withResolvers<undefined>(), released = Promise.withResolvers<undefined>()
|
||||
vi.mocked(inspectRuntime).mockImplementationOnce(async (_config, signal) => {
|
||||
entered.resolve(undefined)
|
||||
await released.promise
|
||||
signal.throwIfAborted()
|
||||
return undefined
|
||||
})
|
||||
const changed = vi.fn()
|
||||
worker.subscribe(changed)
|
||||
worker.inspect(); await entered.promise
|
||||
const cancelled = worker.cancel()
|
||||
expect(worker.snapshot().phase).toBe('cancelling')
|
||||
let settled = false
|
||||
const disposed = worker.dispose().then(() => { settled = true })
|
||||
const notifications = changed.mock.calls.length
|
||||
await Promise.resolve()
|
||||
expect(settled).toBe(false)
|
||||
released.resolve(undefined)
|
||||
await cancelled; await disposed
|
||||
expect(changed).toHaveBeenCalledTimes(notifications)
|
||||
expect(prepareRuntime).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('cancels active inference before the next recording can acquire a fresh worker', async () => {
|
||||
const { worker, root, spawn } = await fixture()
|
||||
await prepare(worker)
|
||||
const cancel = new AbortController()
|
||||
const first = worker.transcribe({ audio, language: 'hold' }, cancel.signal)
|
||||
const rejected = expect(first).rejects.toThrow()
|
||||
await vi.waitFor(async () => { expect(await requests(root)).toContain('hold') }, { timeout: 10000 })
|
||||
const next = worker.transcribe({ audio, language: 'en' }, signal())
|
||||
cancel.abort(new Error('cancel'))
|
||||
await rejected
|
||||
expect((await next).text).toBe('en')
|
||||
expect(spawn).toHaveBeenCalledTimes(2)
|
||||
expect(await (spawn.mock.results[0]!.value as SubprocessHandle).waitForExit()).toBe(true)
|
||||
})
|
||||
|
||||
it('bounds the queue and never executes a cancelled waiting recording', async () => {
|
||||
const { worker, root } = await fixture({ maxPending: 2 })
|
||||
await prepare(worker)
|
||||
const active = new AbortController(), waiting = new AbortController()
|
||||
const first = worker.transcribe({ audio, language: 'hold' }, active.signal)
|
||||
const firstFailure = expect(first).rejects.toThrow()
|
||||
await vi.waitFor(async () => { expect(await requests(root)).toContain('hold') }, { timeout: 10000 })
|
||||
const second = worker.transcribe({ audio, language: 'discard' }, waiting.signal)
|
||||
const secondFailure = expect(second).rejects.toThrow('waiting cancelled')
|
||||
await expect(worker.transcribe({ audio, language: 'full' }, signal())).rejects.toThrow('queue is full')
|
||||
waiting.abort(new Error('waiting cancelled')); active.abort()
|
||||
await firstFailure; await secondFailure
|
||||
expect(await requests(root)).toBe('hold\n')
|
||||
})
|
||||
|
||||
it('joins running and waiting requests when its provider is disposed', async () => {
|
||||
const { worker, root } = await fixture()
|
||||
await prepare(worker)
|
||||
const first = worker.transcribe({ audio, language: 'hold' }, signal())
|
||||
const firstFailure = expect(first).rejects.toThrow()
|
||||
await vi.waitFor(async () => { expect(await requests(root)).toContain('hold') }, { timeout: 10000 })
|
||||
const second = worker.transcribe({ audio, language: 'discard' }, signal())
|
||||
const secondFailure = expect(second).rejects.toThrow('disposed')
|
||||
await worker.dispose(); await firstFailure; await secondFailure
|
||||
expect(await requests(root)).toBe('hold\n')
|
||||
})
|
||||
|
||||
it('stops an idle worker and reloads it on demand', async () => {
|
||||
const { worker, spawn } = await fixture({ idleTimeoutMs: 25 })
|
||||
await prepare(worker)
|
||||
await worker.transcribe({ audio, language: 'en' }, signal())
|
||||
const first = spawn.mock.results[0]!.value as SubprocessHandle
|
||||
await first.done
|
||||
expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh')
|
||||
expect(spawn).toHaveBeenCalledTimes(2)
|
||||
expect(prepareRuntime).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('owns one preparation task across repeated requests and reports warm standby', async () => {
|
||||
const { worker, spawn } = await fixture({ idleTimeoutMs: 100 })
|
||||
const phases: string[] = []
|
||||
const stopObserving = worker.subscribe(() => { phases.push(worker.snapshot().phase) })
|
||||
expect(worker.snapshot()).toMatchObject({ phase: 'unprepared' })
|
||||
worker.prepare(); worker.prepare()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('ready') }, { timeout: 10000 })
|
||||
worker.prepare()
|
||||
expect(spawn).toHaveBeenCalledOnce()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('standby') }, { timeout: 10000 })
|
||||
worker.prepare()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('ready') }, { timeout: 10000 })
|
||||
expect(phases).toContain('waking')
|
||||
expect(prepareRuntime).toHaveBeenCalledOnce()
|
||||
stopObserving()
|
||||
})
|
||||
|
||||
it('cancels Host preparation, retains retry admission, and exposes preparation failure', async () => {
|
||||
const { worker } = await fixture()
|
||||
vi.mocked(prepareRuntime).mockImplementationOnce(async (_ctx, _config, signal) => {
|
||||
await new Promise((_resolve, reject) => {
|
||||
signal.addEventListener('abort', () => { reject(new Error('cancelled')) }, { once: true })
|
||||
})
|
||||
throw new Error('unreachable')
|
||||
})
|
||||
worker.prepare()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('checking') })
|
||||
await worker.cancel()
|
||||
expect(worker.snapshot().phase).toBe('cancelled')
|
||||
await worker.cancel()
|
||||
vi.mocked(prepareRuntime).mockRejectedValueOnce(new Error('download unavailable'))
|
||||
worker.prepare()
|
||||
await vi.waitFor(() => { expect(worker.snapshot()).toMatchObject({ phase: 'failed', message: 'download unavailable' }) })
|
||||
worker.prepare()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('ready') }, { timeout: 10000 })
|
||||
})
|
||||
|
||||
it('recovers after a provider error or unexpected worker exit', async () => {
|
||||
const { worker } = await fixture()
|
||||
await prepare(worker)
|
||||
await expect(worker.transcribe({ audio, language: 'error' }, signal())).rejects.toThrow('provider failed')
|
||||
await expect(worker.transcribe({ audio, language: 'crash' }, signal())).rejects.toThrow()
|
||||
expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh')
|
||||
})
|
||||
|
||||
it('applies inference deadlines and reports runtime preparation failure', async () => {
|
||||
const { worker } = await fixture({ inferenceTimeoutMs: 100 })
|
||||
await prepare(worker)
|
||||
await expect(worker.transcribe({ audio, language: 'hold' }, signal())).rejects.toThrow()
|
||||
const fresh = await fixture()
|
||||
vi.mocked(prepareRuntime).mockRejectedValueOnce(new Error('runtime missing'))
|
||||
fresh.worker.prepare()
|
||||
await vi.waitFor(() => { expect(fresh.worker.snapshot()).toMatchObject({ phase: 'failed', message: 'runtime missing' }) })
|
||||
await fresh.worker.cancel()
|
||||
expect(fresh.worker.snapshot().phase).toBe('failed')
|
||||
await expect(fresh.worker.transcribe({ audio, language: 'zh' }, signal())).rejects.toThrow('Prepare')
|
||||
})
|
||||
|
||||
it('retains complete download snapshots while throttling only intermediate notifications', async () => {
|
||||
const { worker } = await fixture({ progressIntervalMs: 100 })
|
||||
const now = vi.spyOn(performance, 'now').mockReturnValue(0)
|
||||
const runtime = vi.mocked(prepareRuntime).getMockImplementation()!
|
||||
const updates: unknown[] = []
|
||||
worker.subscribe(() => { updates.push(worker.snapshot()) })
|
||||
vi.mocked(prepareRuntime).mockImplementationOnce(async (ctx, config, signal, report) => {
|
||||
report!({ phase: 'downloading', resource: 'model', completedBytes: 0, totalBytes: 10 })
|
||||
report!({ phase: 'downloading', resource: 'model', completedBytes: 2, totalBytes: 10 })
|
||||
expect(worker.snapshot()).toMatchObject({ phase: 'downloading', resource: 'model', completedBytes: 2, totalBytes: 10 })
|
||||
expect(updates).not.toContainEqual(worker.snapshot())
|
||||
now.mockReturnValue(101)
|
||||
report!({ phase: 'downloading', resource: 'model', completedBytes: 5, totalBytes: 10 })
|
||||
report!({ phase: 'downloading', resource: 'model', completedBytes: 10, totalBytes: 10 })
|
||||
report!({ phase: 'downloading', resource: 'vad', completedBytes: 0, totalBytes: 10 })
|
||||
return await runtime(ctx, config, signal, report)
|
||||
})
|
||||
try {
|
||||
worker.prepare()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('ready') }, { timeout: 10000 })
|
||||
expect(updates).toContainEqual(expect.objectContaining({ phase: 'downloading', resource: 'model', completedBytes: 10, totalBytes: 10 }))
|
||||
expect(updates).toContainEqual(expect.objectContaining({ phase: 'downloading', resource: 'vad', completedBytes: 0, totalBytes: 10 }))
|
||||
} finally { now.mockRestore() }
|
||||
})
|
||||
|
||||
it('joins a worker that exits before readiness and exposes retryable preparation errors', async () => {
|
||||
const { worker, spawn, root } = await fixture()
|
||||
vi.mocked(prepareRuntime).mockResolvedValueOnce({ tokens: root, worker: join(root, 'missing.mjs'), model: root, vad: root })
|
||||
worker.prepare()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('failed') }, { timeout: 10000 })
|
||||
expect(await (spawn.mock.results[0]!.value as SubprocessHandle).waitForExit()).toBe(true)
|
||||
const next = await fixture()
|
||||
vi.mocked(prepareRuntime).mockRejectedValueOnce('download interrupted')
|
||||
next.worker.prepare()
|
||||
await vi.waitFor(() => { expect(next.worker.snapshot()).toMatchObject({ phase: 'failed', message: 'download interrupted' }) })
|
||||
})
|
||||
|
||||
it('reclaims the process range after an unexpected idle exit before replacing the worker', async () => {
|
||||
const { worker, spawn } = await fixture()
|
||||
await prepare(worker)
|
||||
await worker.transcribe({ audio, language: 'en' }, signal())
|
||||
const handle = spawn.mock.results[0]!.value as SubprocessHandle
|
||||
handle.terminate()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('standby') }, { timeout: 10000 })
|
||||
expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh')
|
||||
expect(await handle.waitForExit()).toBe(true)
|
||||
expect(spawn).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('retains a worker whose idle cleanup cannot observe exit until that range is joined', async () => {
|
||||
const { worker, spawn, ctx } = await fixture({ idleTimeoutMs: 100 })
|
||||
await prepare(worker)
|
||||
const warn = vi.spyOn(ctx.logger, 'warn').mockImplementation(() => {})
|
||||
await worker.transcribe({ audio, language: 'en' }, signal())
|
||||
const handle = spawn.mock.results[0]!.value as SubprocessHandle
|
||||
const joined = vi.spyOn(handle, 'waitForExit').mockRejectedValueOnce(new Error('range observation failed'))
|
||||
await vi.waitFor(() => { expect(joined).toHaveBeenCalledOnce() }, { timeout: 10000 })
|
||||
expect(warn).toHaveBeenCalledWith('Speech worker idle cleanup failed', expect.any(Error))
|
||||
await expect(worker.transcribe({ audio, language: 'zh' }, signal())).rejects.toThrow('range observation failed')
|
||||
expect(spawn).toHaveBeenCalledOnce()
|
||||
expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh')
|
||||
expect(await joined.mock.results.at(-1)!.value).toBe(true)
|
||||
expect(spawn).toHaveBeenCalledTimes(2)
|
||||
warn.mockRestore()
|
||||
})
|
||||
|
||||
it('retains completed preparation steps when cancellation settles the active step', async () => {
|
||||
const { worker } = await fixture(), entered = Promise.withResolvers<undefined>()
|
||||
vi.mocked(prepareRuntime).mockImplementationOnce(async (_ctx, _config, signal, report) => {
|
||||
for (const step of ['model', 'vad'] as const) report!({ phase: 'downloading', step, resource: step, completedBytes: 1, totalBytes: 1 })
|
||||
report!({ phase: 'checking', step: 'verify', startedAt: Date.now() })
|
||||
const before = worker.snapshot().steps?.find(step => step.kind === 'verify')?.startedAt
|
||||
report!({ phase: 'checking', step: 'verify', startedAt: Date.now() })
|
||||
expect(worker.snapshot().steps?.find(step => step.kind === 'verify')?.startedAt).toBe(before)
|
||||
entered.resolve(undefined)
|
||||
return await new Promise((_resolve, reject) => {
|
||||
signal.addEventListener('abort', () => { reject(new Error('setup cancelled')) }, { once: true })
|
||||
})
|
||||
})
|
||||
worker.prepare(); await entered.promise
|
||||
expect(worker.snapshot().steps?.map(step => [step.kind, step.status])).toEqual([
|
||||
['check', 'complete'], ['model', 'complete'], ['vad', 'complete'], ['verify', 'running'], ['load', 'pending'],
|
||||
])
|
||||
await worker.cancel()
|
||||
expect(worker.snapshot().steps?.find(step => step.kind === 'verify')?.status).toBe('cancelled')
|
||||
worker.prepare()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('ready') }, { timeout: 10000 })
|
||||
expect(worker.snapshot().steps?.every(step => step.status === 'complete')).toBe(true)
|
||||
})
|
||||
|
||||
it('omits preparation steps supplied by explicit deployment paths', async () => {
|
||||
const { worker } = await fixture({ modelDirectory: process.cwd(), vadModelPath: process.cwd() })
|
||||
expect(worker.snapshot().steps?.map(step => step.kind)).toEqual(['check', 'verify', 'load'])
|
||||
worker.prepare()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('ready') }, { timeout: 10000 })
|
||||
})
|
||||
|
||||
it('returns prepared resources to standby when the user cancels an idle wake-up', async () => {
|
||||
const { worker } = await fixture({ idleTimeoutMs: 100 })
|
||||
await prepare(worker)
|
||||
await worker.transcribe({ audio, language: 'en' }, signal())
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('standby') }, { timeout: 10000 })
|
||||
const cancel = new AbortController()
|
||||
const unsubscribe = worker.subscribe(() => {
|
||||
if (worker.snapshot().phase === 'waking') cancel.abort(new Error('recording cancelled'))
|
||||
})
|
||||
await expect(worker.transcribe({ audio, language: 'zh' }, cancel.signal)).rejects.toThrow('recording cancelled')
|
||||
expect(worker.snapshot().phase).toBe('standby')
|
||||
expect(worker.snapshot().steps?.find(step => step.kind === 'load')?.status).toBe('cancelled')
|
||||
unsubscribe()
|
||||
expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh')
|
||||
})
|
||||
|
||||
it('launches source workers through the repository ESM loader', async () => {
|
||||
const { worker, spawn, root } = await fixture()
|
||||
vi.mocked(prepareRuntime).mockResolvedValueOnce({ tokens: root, worker: fileURLToPath(new URL('./worker-source.fixture.ts', import.meta.url)), model: root, vad: root })
|
||||
await prepare(worker)
|
||||
expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh')
|
||||
expect(spawn.mock.calls[0]?.[0].argv).toContain('--import')
|
||||
})
|
||||
|
||||
it('requires explicit preparation before direct transcription without starting downloads', async () => {
|
||||
const { worker, spawn } = await fixture()
|
||||
await expect(worker.transcribe({ audio, language: 'zh' }, signal())).rejects.toThrow('Prepare')
|
||||
expect(worker.snapshot().phase).toBe('unprepared')
|
||||
expect(prepareRuntime).not.toHaveBeenCalled()
|
||||
expect(spawn).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rechecks readiness after a queued cache inspection without implicitly downloading missing resources', async () => {
|
||||
const { worker, root, spawn } = await fixture()
|
||||
vi.mocked(inspectRuntime).mockResolvedValueOnce({ tokens: root, model: root, vad: root,
|
||||
worker: fileURLToPath(new URL('./worker.fixture.mjs', import.meta.url)) })
|
||||
worker.inspect()
|
||||
await vi.waitFor(() => { expect(worker.snapshot().phase).toBe('standby') })
|
||||
vi.mocked(inspectRuntime).mockResolvedValueOnce(undefined)
|
||||
worker.inspect()
|
||||
await expect(worker.transcribe({ audio, language: 'zh' }, signal())).rejects.toThrow('Prepare')
|
||||
expect(worker.snapshot().phase).toBe('unprepared')
|
||||
expect(prepareRuntime).not.toHaveBeenCalled()
|
||||
expect(spawn).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it.each(['ready', 'standby', 'unprepared', 'failed'] as const)('joins cancellation at committed %s without replacing its result', async (phase) => {
|
||||
const { worker, root, spawn } = await fixture()
|
||||
const cancellation = Promise.withResolvers<Promise<void>>()
|
||||
const unsubscribe = worker.subscribe(() => {
|
||||
if (worker.snapshot().phase === phase) cancellation.resolve(worker.cancel())
|
||||
})
|
||||
try {
|
||||
if (phase === 'ready') worker.prepare()
|
||||
else {
|
||||
if (phase === 'failed') vi.mocked(inspectRuntime).mockRejectedValueOnce(new Error('cache unavailable'))
|
||||
else vi.mocked(inspectRuntime).mockResolvedValueOnce(phase === 'standby'
|
||||
? { tokens: root, model: root, vad: root, worker: fileURLToPath(new URL('./worker.fixture.mjs', import.meta.url)) }
|
||||
: undefined)
|
||||
worker.inspect()
|
||||
}
|
||||
await cancellation.promise
|
||||
expect(worker.snapshot().phase).toBe(phase)
|
||||
if (phase === 'ready' || phase === 'standby') {
|
||||
expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh')
|
||||
expect(spawn).toHaveBeenCalledOnce()
|
||||
} else {
|
||||
await expect(worker.transcribe({ audio, language: 'zh' }, signal())).rejects.toThrow('Prepare')
|
||||
expect(spawn).not.toHaveBeenCalled()
|
||||
await prepare(worker)
|
||||
expect(worker.snapshot().phase).toBe('ready')
|
||||
}
|
||||
} finally { unsubscribe() }
|
||||
})
|
||||
|
||||
it('cancels a queued preparation before any resources are acquired', async () => {
|
||||
const { worker, spawn } = await fixture()
|
||||
worker.prepare()
|
||||
await worker.cancel()
|
||||
expect(worker.snapshot().phase).toBe('cancelled')
|
||||
expect(prepareRuntime).not.toHaveBeenCalled()
|
||||
expect(spawn).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('reuses a loaded worker only after explicitly classified input rejection', async () => {
|
||||
const { worker, spawn } = await fixture()
|
||||
await prepare(worker)
|
||||
await expect(worker.transcribe({ audio, language: 'invalid-input' }, signal())).rejects.toThrow('invalid input')
|
||||
expect(worker.snapshot().phase).toBe('ready')
|
||||
expect((await worker.transcribe({ audio, language: 'en' }, signal())).text).toBe('en')
|
||||
expect(spawn).toHaveBeenCalledOnce()
|
||||
await expect(worker.transcribe({ audio, language: 'error' }, signal())).rejects.toThrow('provider failed')
|
||||
expect(worker.snapshot().phase).toBe('standby')
|
||||
expect((await worker.transcribe({ audio, language: 'zh' }, signal())).text).toBe('zh')
|
||||
expect(spawn).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../util/timeout"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../speech-to-text"
|
||||
},
|
||||
{
|
||||
"path": "../../subprocess/subprocess"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
import { defineConfig } from 'tsdown'
|
||||
|
||||
export default defineConfig(['index', 'worker'].map(name => ({ entry: [`lib/types/${name}.js`], outDir: 'lib', format: ['esm'], platform: 'node', target: 'es2024', fixedExtension: false, dts: false, clean: false })))
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/experimental/speech-to-text/README.md
|
||||
README.md: 6b2481590c2e1ee7c02401ca0838ab62d4a64b86
|
||||
README.zh.md: 46436194dd2350f85fc652b31377fad2f0d79348
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
description: "Named experimental transcription providers and cancellation ownership."
|
||||
kind: "package-reference"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-experimental-speech-to-text
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
## Summary
|
||||
|
||||
This Service Definition selects named speech recognizers through `ctx.speechToText`. Consumers resolve their request before execution; providers register independently.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Use this package](#use-this-package)
|
||||
- [Understand the implementation](#understand-the-implementation)
|
||||
- [Further Exploration](#further-exploration)
|
||||
- [Model Experience](#model-experience)
|
||||
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
|
||||
- [Dev Note](#dev-note)
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## Use this package
|
||||
|
||||
Load through the [voice input bundle](../voice-input-bundle/README.md), or compose the service with a provider and consumer. `defaultProvider` is required and selects an exact registered id; the bundle supplies `sensevoice-local`. `language` supplies the omitted language hint. A missing or duplicate provider fails explicitly.
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## Understand the implementation
|
||||
|
||||
<details>
|
||||
<summary>Maintainer details — click to expand</summary>
|
||||
|
||||
`resolve()` captures the provider instance, recording and language. `transcribe()` rejects a withdrawn or replaced registration. A registration disposer closes admission, aborts accepted requests and joins provider settlement; providers must honor cancellation. No fallback selects a different recognizer or uploads audio. No runtime invariant companion is published because the registry is the sole source of provider and preparation observations.
|
||||
|
||||
When `settings` is present, the `voice-input` namespace persists provider and language choices; composition defaults apply before an override is saved. `configure()` updates only supplied fields and rejects a language unsupported by the selected provider before saving. Providers advertise accepted language hints through `languages`; `resolve()` validates the selected hint before transcription. Provider-owned preparation is observed through complete `SpeechSnapshot` values; closing an observer never cancels preparation.
|
||||
|
||||
The `./wave` helper validates canonical 16 kHz mono PCM16 WAV for the Remote consumer and the native recognition process. Both reject inconsistent headers and lengths before decoding samples.
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## Further Exploration
|
||||
|
||||
[Voice input subsystem](../../../docs/subsystems/voice-input.md)
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## Model Experience
|
||||
|
||||
None, as recordings and preparation remain outside model requests; ordinary user submission owns any later text.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
No direct effect; ordinary submission owns the message content.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- Only complete-recording transcription is supported. Streaming recognition and speech synthesis have no service methods.
|
||||
|
||||
-----
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### Dev Note
|
||||
|
||||
<details>
|
||||
<summary>Maintainer details — click to expand</summary>
|
||||
|
||||
None.
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
description: "具名实验性转写 Provider 及取消操作的所有权。"
|
||||
kind: "package-reference"
|
||||
---
|
||||
|
||||
# @deepseek-ai/dsh-experimental-speech-to-text
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
## 概述
|
||||
|
||||
此服务定义通过 `ctx.speechToText` 选择具名语音识别器。消费者先解析请求再执行,Provider 独立注册。
|
||||
|
||||
## 目录
|
||||
|
||||
- [使用此包](#use-this-package)
|
||||
- [理解实现](#understand-the-implementation)
|
||||
- [进一步探索](#further-exploration)
|
||||
- [模型体验](#model-experience)
|
||||
- [已知限制与后续工作](#known-limitations-and-deferred-work)
|
||||
- [开发备注](#dev-note)
|
||||
|
||||
-----
|
||||
|
||||
<a id="use-this-package"></a>
|
||||
## 使用此包
|
||||
|
||||
通过[语音输入 Bundle](../voice-input-bundle/README.zh.md)加载,或组合此服务、Provider 与消费者。`defaultProvider` 为必填项,选择确切的注册 id;Bundle 提供 `sensevoice-local`。`language` 提供省略的语言提示。缺失或重复的 Provider 会明确报错。
|
||||
|
||||
-----
|
||||
|
||||
<a id="understand-the-implementation"></a>
|
||||
## 理解实现
|
||||
|
||||
<details>
|
||||
<summary>维护者信息 — 点击展开</summary>
|
||||
|
||||
`resolve()` 捕获 Provider 实例、录音和语言。`transcribe()` 拒绝已撤销或替换的注册。注册的清理函数先拒绝新请求,再取消已接收的请求并等待 Provider 完成;Provider 必须响应取消。服务不会回退到其他识别器或上传音频。不发布运行时不变量伴随模块,因为注册表是 Provider 与准备状态观测的唯一来源。
|
||||
|
||||
存在 `settings` 时,`voice-input` 命名空间持久化 Provider 与语言选择;保存覆盖值之前使用组合配置默认值。`configure()` 只更新传入的字段,并在保存前拒绝所选 Provider 不支持的语言。Provider 通过 `languages` 公布支持的语言提示;`resolve()` 在转写前校验所选提示。观察者通过完整 `SpeechSnapshot` 接收 Provider 拥有的准备状态;关闭观察者不会取消准备。
|
||||
|
||||
`./wave` 辅助函数为 Remote 消费者与原生识别进程校验规范的 16 kHz 单声道 PCM16 WAV。两者均在解码采样前拒绝不一致的头部和长度。
|
||||
|
||||
</details>
|
||||
|
||||
-----
|
||||
|
||||
<a id="further-exploration"></a>
|
||||
## 进一步探索
|
||||
|
||||
[语音输入子系统](../../../docs/subsystems/voice-input.zh.md)
|
||||
|
||||
-----
|
||||
|
||||
<a id="model-experience"></a>
|
||||
## 模型体验
|
||||
|
||||
无,因为录音与准备不进入模型请求;之后的文字由普通用户提交拥有。
|
||||
|
||||
#### KV 缓存影响
|
||||
|
||||
没有直接影响;普通提交拥有消息内容。
|
||||
|
||||
## 已知限制与后续工作
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- 仅支持完整录音转写。服务没有流式识别或语音合成方法。
|
||||
|
||||
-----
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### 开发备注
|
||||
|
||||
<details>
|
||||
<summary>维护者信息 — 点击展开</summary>
|
||||
|
||||
无。
|
||||
|
||||
</details>
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-experimental-speech-to-text",
|
||||
"description": "Experimental speech recognition with independently selectable providers",
|
||||
"version": "0.1.6-alpha.2",
|
||||
"publishConfig": {
|
||||
"access": "public"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
|
||||
"directory": "packages/experimental/speech-to-text"
|
||||
},
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json",
|
||||
"./types": {
|
||||
"types": "./lib/types/types.d.ts",
|
||||
"default": "./lib/types/types.js"
|
||||
},
|
||||
"./wave": {
|
||||
"types": "./lib/types/wave.d.ts",
|
||||
"default": "./lib/types/wave.js"
|
||||
}
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/types/**/*.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@deepseek-ai/dsh-brand": "workspace:^",
|
||||
"@deepseek-ai/schemastery": "workspace:^"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:^",
|
||||
"@deepseek-ai/dsh-settings": "workspace:^"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:^",
|
||||
"@deepseek-ai/dsh-settings": "workspace:^"
|
||||
}
|
||||
}
|
||||
Loaded 100 of 133 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user