Yudong
d6042ca0d7
Silero cannot back-date the first segment of a cold detector, so it
reports a start index several frames after speech actually began. Every
recording therefore lost its opening audio: measured across 50
utterances the median loss was 102 ms and the worst 230 ms — a full
syllable, which surfaced as dropped leading words ("麻烦把发票…" →
"把发票…", "用 Python…" → "写一个 Python…").
Only the first segment is affected; later segments aligned within 3 ms
because the detector retains history by then. Tuning cannot fix it:
lowering minSpeechSeconds from 0.25 s to 0.05 s left the median loss at
110 ms, and lowering the threshold to 0.2 still lost up to 174 ms while
risking false triggers.
Restore the dropped onset from the recording itself, bounded by the new
vadOnsetPaddingSeconds (default 0.3 s, zero disables). Measured on the
same corpus with identical DSH settings, this recovers the full no-VAD
accuracy and in the technical domain exceeds it, since segmentation also
trims silence:
语料 条件 无VAD VAD丢起音 VAD+回填 改善
audio clean 12.50% 16.33% 12.50% 23.5%
audio noisy10 17.74% 19.15% 17.94% 6.3%
audio_tech clean 10.34% 15.65% 9.81% 37.3%
audio_tech noisy10 25.73% 30.50% 26.26% 13.9%
Inference cost is unchanged (2.6 s vs 2.6 s on both clean corpora).
6.2 KiB
description, kind
| description | kind |
|---|---|
| Prepare and operate a local CPU SenseVoice worker on demand. | package-reference |
@deepseek-ai/dsh-experimental-speech-to-text-sensevoice
English | 中文
Summary
This provider recognizes speech with SenseVoiceSmall ONNX and Silero VAD on the Host CPU. The platform-specific sherpa-onnx Node package includes ONNX Runtime; users need no Python, compiler or model conversion. Activation checks cached resources without loading models or downloading assets.
Table of Contents
- Use this package
- Understand the implementation
- Further Exploration
- Model Experience
- Known Limitations and Deferred Work
- Dev Note
Use this package
The bundle supplies an absolute dataRoot under the DSH home. Preparation downloads revision-pinned model files, verifies their sizes and SHA-256 hashes, then loads them. precision defaults to int8; fp32 selects the larger reference weights. modelDirectory supplies an existing absolute directory containing the selected ONNX file and tokens.txt; vadModelPath selects an existing Silero ONNX file. Verified completed files remain reusable after cancellation or failure.
Before each missing asset is downloaded, the Host compares the Hugging Face-compatible modelOrigins, which default to https://huggingface.co and HF-Mirror. Concurrent HEAD requests follow the pinned file path and redirects through the Host's fetch proxy; the first 2xx response is tried first. modelProbeTimeoutMs defaults to 3000 ms; when every probe fails, downloads use the configured order. Network, HTTP, certificate and integrity failures try the remaining sources; cancellation, storage failures and unclassified failures stop preparation. Pinned revisions, sizes and SHA-256 hashes apply to every source. Response latency does not measure download throughput; the preparation deadline still bounds the complete download.
An explicit modelOrigin uses only that origin, without probing or public fallback. A single-entry modelOrigins also skips probing. Verified caches and explicit offline paths require no source requests. The voice UI offers the advertised sources before preparation or retry. A manual choice overrides selection for that task only, uses one source without fallback, and cannot replace an active task’s source. The Host rejects choices outside its configured origins; fully offline deployments advertise no download sources.
Download failures identify the asset and source origin, with a classified cause and HTTP status or diagnostic code when available. The public state omits URL credentials, query strings and raw cause messages. Retry reuses verified files; partial downloads restart.
Every activation checks the selected model, tokens and VAD on disk. Complete verified caches restore readiness immediately after inspection; the first recording wakes the worker. Missing files or mismatched managed checksums require explicit preparation, while unreadable paths report an error. Explicit deployment paths are checked for accessibility; their contents remain the deployer's responsibility. Disabling the plugin or restarting the Host does not require downloading intact caches again.
Understand the implementation
Maintainer details — click to expand
One managed Node subprocess serializes inference. Recordings submitted during wake-up join the bounded queue. At execution, the worker requires ready or standby resources. Transcription never starts preparation downloads. threads, segmentSeconds, vadThreshold, minSpeechSeconds, minSilenceSeconds and vadOnsetPaddingSeconds tune CPU inference and segmentation, the last restoring the onset a cold detector reports late; maxPending bounds accepted work. Preparation and inference have separate deadlines. Invalid language or WAV input is rejected before native inference and retains the loaded worker; inference failures, malformed responses and transport failures reclaim it. Cancellation terminates an active worker and joins its process range; cancelled queued recordings never run. idleTimeoutMs releases the worker after inactivity, with zero retaining it until disposal. Models remain cached. Authenticated loopback requests keep audio in memory. Electron workers run in Node mode and copy VAD buffers to satisfy the V8 memory cage.
The Host owns preparation across page and Session changes: check resources, download recognition weights and tokens, download VAD, verify files, and load the worker. Explicit model paths omit their download steps. Downloads report bytes; other steps report elapsed time. An idle worker retains prepared paths and needs only model loading on wake-up. Cancellation after a preparation result is published waits for cleanup without replacing that result.
Further Exploration
Model Experience
None, as recordings and preparation remain outside model requests; ordinary user submission owns any later text.
KV Cache effect
No direct effect; ordinary submission owns the message content.
Known Limitations and Deferred Work
- CPU only. Native packages target macOS arm64/x64, Linux glibc arm64/x64 and Windows x64; Windows ARM64 and Linux musl are not verified deployment targets. INT8 weights occupy about 239 MB, FP32 about 938 MB, plus the runtime and VAD. Model memory exceeds file size and grows with recording length; installation hints are estimates. Language hints are auto, Chinese, English, Cantonese, Japanese and Korean. Disabling the plugin retains model caches. Packaged Desktop releases still require platform signing and microphone-permission acceptance; Linux Desktop distribution is outside this provider.
Dev Note
Maintainer details — click to expand
None.