• ASR Speech Recognition

ASR Speech Recognition

(This page is the recognizer's debugging and benchmark entry point; its place in the chain and turn detection live in VAD and Turn Detection.)

Architecture

The ASR trait lives in service/src/component/asr.rs, while the engine implementation and singleton live in api/src/component/asr/, backed by sherpa-onnx.

pub trait Asr: Send + Sync {
    fn create_stream(&self) -> Box<dyn AsrStream>;
    async fn transcribe(&self, sample_rate: u32, samples: &[f32])
        -> Result<RecognizerResult, AppError>;
}

RecognizerResult { text: String, prob: f32 } is the shared output of both paths. The streaming path is AsrStream: accept_waveform / decode / is_endpoint / get_partial / finish / reset.

AsrManager::init(config).await;          // initialise at startup
let model = AsrManager::global().default(); // Arc<dyn Asr>, shareable across turns
let mut stream = model.create_stream();
stream.accept_waveform(&samples);
stream.decode();

Initialisation happens in the server startup path; there are two consumers: AsrNode (the real-time voice chain) and the Matrix client (whole-utterance transcription).

Models

Models come from https://github.com/k2-fsa/sherpa-onnx and are managed by downloader manifests:

moon run server:downloader -- asr
moon run server:downloader -- asr x_asr

Benchmarks and diagnostics

api/tests/common/asr.rs turns each test's timing into one diagnostic line:

CER=XX.X%(X) WER=XX.X% Acc=XX.X% Perf=XX.X% Total=XX.X% | RTF=X.XXX ASR=X.Xs Audio=X.XXs <verdict>

The old page's "TTS loopback CER 0%, RTF 0.110" were measurements from before the refactor and were not re-measured this round; the commands below are how to find out what they are now.

Tests

api/tests/asr_test.rs has five, all calling AsrManager::create_model directly:

The XAsr tests need data/asr/model/x_asr/default/ downloaded (moon run server:downloader -- asr).

# Void: no model files needed
cargo test --package api --test asr_test -- test_asr_model_void

# XAsr: model required
cargo test --package api --test asr_test -- test_asr_model_x_asr_transcribe --nocapture
cargo test --package api --test asr_test -- test_asr_model_x_asr_streaming --nocapture

# release is faster
cargo test --package api --test asr_test --release -- test_asr_model_x_asr_streaming --nocapture

Chain-level regression lives in api/tests/session/: test_chat_flow_listen_manual, test_chat_flow_listen_auto, test_chat_flow_listen_realtime, test_chat_flow_voice_text_voice_round_trips, test_chat_flow_resample_48k_multi_round.

Adding an ASR model

1. Manifest: apps/server/src/downloader/manifests/asr/<model>.json
2. Implementation: api/src/component/asr/model/<model>/mod.rs, implementing Asr (and AsrStream if streaming)
3. Register the module: add pub mod <model> in api/src/component/asr/model/mod.rs
4. Enum: add a variant to AsrModel in api/src/config/mod.rs
5. Factory: add a match arm in create_model() in api/src/component/asr/mod.rs
6. Tests: add whole-utterance and streaming cases in api/tests/asr_test.rs, at least asserting non-empty output and a partial

Key files