Tail-word retention
The benchmark harness includes a synthetic regression for short push-to-talk clips where the final word is easy to lose. It compares raw Unified, padded Unified, and production v3 before any candidate model is exposed in the app.
Benchmarks
How fast is Presspeech?
Presspeech keeps its latency claims tied to a reproducible benchmark in experiments/swift-bench/. Numbers below are p50 across five trials per backend, first inference excluded.
Setup
Environment
MachineMac mini M4, 10 cores, 16 GB.
OS and toolchainmacOS 26.4.1, Xcode/Swift 6.3.
TrialsFive per backend per clip. First inference after model load excluded from p50.
AudioGenerated 16 kHz mono Float32 WAV clips, all tested against the same backends.
p50 latency
Results
| Clip | Duration | FluidAudio on ANE | Prior GPU path | Speedup |
|---|---|---|---|---|
short-clean |
2.50 s | 1.57× | ||
medium-clean |
3.99 s | 1.83× | ||
disfluent |
5.31 s | 1.97× | ||
longer-technical |
9.49 s | 1.97× |
Machine-readable results: benchmarks/results.json. Reproduction docs: experiments/swift-bench.
Candidate models
Candidate model evaluation
Private corpus
For candidate-model decisions, maintainers can run Unified, the linear-int8 v3 encoder, and current Nemotron candidates over private real-dictation clips. Reports keep transcript text and filenames redacted by default while showing WER, final-word failures, and latency.
Release gate
run-release-asr-checks.sh checks production v3 only when the benchmark and app pin the exact same FluidAudio revision and a validated multi-window corpus is present. The multi-window gate rejects a conservative corpus WER above 10% and any run of more than six consecutively deleted reference words. An explicit candidate-dependency mode can compare unreleased APIs, but cannot produce a production release-pass verdict.
The public-fixture fetcher can also import a deterministic row range from a pinned Google FLEURS language split. This gives encoder candidates reproducible multilingual and cross-script coverage; private human dictation remains a separate product requirement.
Evidence scope: the public comparison below ran on FluidAudio v0.15.5 (19600a4). The app now pins v0.15.6 (4dbf4f9), so the table supports retaining Parakeet v3 over the tested Nemotron candidates; it does not validate the current inference-library revision for release.
| 2026-07-22 v0.15.5 backend | Avg WER | Worst WER | Final-word failures | Avg p50 |
|---|---|---|---|---|
v3 |
1.73% | 14.0% | 1 | 85.0 ms |
nemotron-en |
4.32% | 40.0% | 1 | 679.4 ms |
nemotron-multilingual |
6.22% | 40.0% | 2 | 123.2 ms |
On 25 LibriSpeech dev-clean clips, repaired Nemotron English and Nemotron 3.5 multilingual both stayed behind Parakeet TDT v3. Multilingual is much faster than the older English path, but its accuracy and final-word retention are worse, so v3 remains the production model.
Vocabulary research
Why decoder vocabulary assistance is not in the app
A daily Polish user asked for recognition-time vocabulary assistance because post-transcription dictionary rules require separate entries for inflected names. The need is real, but the current FluidAudio mechanism is a second CTC scoring pass with a precision/recall tradeoff, not a morphological system: aliases can help match variants, but an accepted alias is replaced with one configured canonical form.
Known risk
FluidAudio documents that short or acoustically similar terms can replace ordinary words that were never spoken. Its conservative controls reduce those false replacements but can also lose genuine vocabulary recall.
Presspeech gate
run-vocabulary-bias-regression.sh compares production v3 with four direct-v3 policies over repeated target clips and same-language negative controls. A candidate cannot pass by improving aggregate recall while making even one compared clip worse.
Current status
No vocabulary policy has cleared the product-candidate gate, so Presspeech still uses its deterministic Dictionary & Shortcuts pass. That avoids silently trading occasional proper-name fixes for unrelated word substitutions.
People affected by this gap can help without publishing private speech. Follow the privacy-redacted contribution workflow and attach only the aggregate report to issue #21. The local fixtures, transcripts, vocabulary terms, and paths stay on the contributor's Mac. Technical background: FluidAudio custom-vocabulary documentation at Presspeech's pinned revision.
Reading the numbers
Interpretation
Latency
The ANE path is consistently faster than the prior GPU path on these clips, with p50 results around 92-152 ms.
Accuracy
Both backends produced essentially identical transcripts on the synthetic test audio, including the same TTS artifacts.
Power
The benchmark measures compute latency, not energy. Power measurement remains a future bench improvement.