Presspeech keeps its latency claims tied to a reproducible benchmark in experiments/swift-bench/. Numbers below are p50 across five trials per backend, first inference excluded.
| Clip | Duration | FluidAudio on ANE | Prior GPU path | Speedup |
|---|---|---|---|---|
short-clean |
2.50 s | 1.57× | ||
medium-clean |
3.99 s | 1.83× | ||
disfluent |
5.31 s | 1.97× | ||
longer-technical |
9.49 s | 1.97× |
Machine-readable results: benchmarks/results.json. Reproduction docs: experiments/swift-bench.
The release suite records production v3 on synthetic short-clip cuts with 80 ms and 400 ms capture-grace bounds. This is a report-only diagnostic, not a quality pass or a simulation of live RMS end-of-speech detection. The separate Unified padding gate remains opt-in candidate evidence.
For candidate-model decisions, maintainers can run Unified, the linear-int8 v3 encoder, and current Nemotron candidates over private real-dictation clips. Reports keep transcript text and filenames redacted by default while showing WER, final-word failures, and latency.
run-release-asr-checks.sh requires private real-dictation clips, five hand-audited non-speech controls with no text in any measured trial, the benchmark and app to pin the exact same FluidAudio revision, and validated English and German multi-window corpora. It gates production v3 long-form results with conservative corpus WER and consecutive-deletion limits; private short-dictation WER remains for maintainer review, not an absolute pass threshold. It also reports a same-pin production-v3 versus explicit no-mel comparison as candidate-only evidence. That comparison does not change runtime selection or the release verdict. An explicit candidate-dependency mode can compare unreleased APIs, but cannot produce a production release-pass verdict.
Private dictation fixtures stay on the evaluator's computer; a lightweight run may explicitly omit them but cannot report a production release pass. This matters because public read-speech quality does not establish spontaneous-dictation quality: a multilingual conversational-speech study found substantial performance drops relative to standard read-speech benchmarks (Maheshwari et al., 2024).
The public-fixture fetcher can also import a deterministic row range from a pinned Google FLEURS language split. This gives encoder candidates reproducible multilingual and cross-script coverage; private human dictation is required separately for the release gate.
Language coverage is not a quality guarantee. NVIDIA lists 25 supported languages, but benchmark accuracy varies by language and speech variant. In FluidAudio's published Parakeet TDT v3 CoreML FLEURS results (24 evaluated languages, M4 Pro), WER ranges from 4.0% for Italian to 36.9% for Greek; Danish is 20.2% and Slovenian 27.4%. These are upstream results, not Presspeech measurements, and the read-speech-focused corpus, sample counts, runtime and hardware do not establish accuracy for short, spontaneous dictation. NVIDIA also notes that its European Portuguese training data differs from Brazilian Portuguese used in most evaluation benchmarks. Presspeech's result table above is English-only; use reproducible per-language fixtures and human dictation before making multilingual quality claims. Sources: FluidAudio FLEURS benchmark and the NVIDIA model card and evaluation notes.
A recent upstream report on conversational Latin-American Spanish was corrected by its author: the mixed Spanish/English output was not translation or a CoreML-only defect; the same model behaved worse in a full-precision upstream run. The author attributed it to Parakeet v3's behavior on spontaneous speech and English prior, and the issue was closed as not planned. This is a single upstream report on FluidAudio v0.15.5, not a Presspeech reproduction or evidence about the current v0.15.6 pin. NVIDIA's model card describes automatic language detection; Presspeech's selected language is documented in code as a decoder script-filter hint, not a language-forcing mechanism. To characterize this risk, use the same-input Spanish probe on the production pin, and keep its FLEURS read-speech result distinct from local spontaneous dictation. Both remain evaluation evidence, not a quality claim or release pass.
Evidence scope: the public comparison below ran on FluidAudio v0.15.5 (19600a4). The app pins v0.15.6 (4dbf4f9), so the table supports retaining Parakeet v3 over the tested Nemotron candidates; it does not validate the app's pinned inference-library revision for release.
Broader evidence also argues against extrapolating from this clean read-speech sample: NVIDIA's model card reports 1.93% WER on LibriSpeech test-clean, versus 11.31% on AMI and 11.42% on Earnings-22 (different corpora, not a Presspeech head-to-head). ASR evaluation research likewise finds multi-benchmark averages more informative about noisy real-world performance than a single corpus. Keep multi-window and human-dictation checks in the product gate; these outside results do not establish a Presspeech quality score.
Upstream FluidAudio v0.15.7 changed Parakeet v3's default long-form chunking policy and added final-window re-decode/seam fixes; v0.15.8 added fresh per-window decoder state, blank-window recovery, and text-based seam reconciliation (v0.15.7 notes, v0.15.8 notes). As checked on 2026-09-24, upstream's latest release is v0.17.1 (release notes), a CocoaPods podspec metadata fix with no Swift Package Manager code change from v0.17.0. That code includes an ASR optimization that bulk-resets and copies MLMultiArray storage. The upstream PR reports warm median inference on an 11-second clip falling from 71.1 ms to 48.3 ms with identical transcripts, on an M3 Pro and its Parakeet v3 Orukeet bundle (PR #941 and measurements). This is a candidate signal, not a Presspeech result: the app still pins v0.15.6 and has no same-input v0.17.0 quality or latency measurement. A separate German long-form report on v0.15.5 exposed token-order corruption with mel-context chunking; upstream PR #830 fixed that specific merge-order defect, and the v0.15.6 release notes show it is already in Presspeech's current pin. The PR notes that first-chunk mel-context degradation is separate, so the fix does not establish that all multilingual seams are safe. Presspeech still explicitly uses its reviewed mel-context policy.
A separate upstream Ukrainian right-context repro isolated deterministic single-window word corruption to the shipped v3 CoreML encoder's 6-bit-LUT weights. FluidAudio's opt-in linear-int8 Encoder_v2 decoded all three reported cuts correctly in its controlled swap, but is larger (568 MB vs 425 MB) and still needs broad WER and Apple Neural Engine latency evaluation. Presspeech pins the original encoder today; its benchmark has a candidate-only comparison path, but no Presspeech result establishes a quality or latency gain. Evaluate the same Ukrainian FLEURS fixtures and human-reviewed local dictation, plus cache/memory and ANE latency, before considering promotion. This upstream case motivates testing; it is not a Presspeech reproduction or evidence to change production.
The required Presspeech multi-window release corpus is English LibriSpeech with an English hint; it cannot validate the German failure class or upstream fix. The benchmark guide documents a pinned FLEURS German long-form probe. For an SDK update, save v0.15.6 baselines on both the required English corpus and the same composed German corpus, then compare v0.17.0 using run-release-asr-checks.sh --sdk-upgrade-only --allow-candidate-dependency, moving only the benchmark pin. Compare explicit production-config v3 against those baselines; the candidate SDK-default/no-mel path is also evaluated on German with an explicit de hint, alongside its English long-form and private/public checks. Match fixture digests, trial counts, and hints. Upstream speed figures and fixes motivate evaluation but do not establish a Presspeech quality or latency gain; candidate results are not a production release pass.
| 2026-07-22 v0.15.5 backend | Avg WER | Worst WER | Final-word failures | Avg p50 |
|---|---|---|---|---|
v3 |
1.73% | 14.0% | 1 | 85.0 ms |
nemotron-en |
4.32% | 40.0% | 1 | 679.4 ms |
nemotron-multilingual |
6.22% | 40.0% | 2 | 123.2 ms |
On 25 LibriSpeech dev-clean clips, repaired Nemotron English and Nemotron 3.5 multilingual both stayed behind Parakeet TDT v3. Multilingual is much faster than the older English path, but its accuracy and final-word retention are worse, so v3 remains the production model.
A daily Polish user asked for recognition-time vocabulary assistance because post-transcription dictionary rules require separate entries for inflected names. The need is real, but the current FluidAudio mechanism is a second CTC scoring pass with a precision/recall tradeoff, not a morphological system: aliases can help match variants, but an accepted alias is replaced with one configured canonical form.
FluidAudio documents that short or acoustically similar terms can replace ordinary words that were never spoken. Its conservative controls reduce those false replacements but can also lose genuine vocabulary recall.
run-vocabulary-bias-regression.sh compares production v3 with four direct-v3 policies and four sliding-window paths over repeated target clips and same-language negative controls. A candidate cannot pass by improving aggregate recall while making even one compared clip worse.
No vocabulary policy has cleared the product-candidate gate, so Presspeech still uses its deterministic Dictionary & Shortcuts pass. The corrected full-audio evidence rules out enabling the tested standard policy, not just enabling it by default.
| Question | Baseline | Vocabulary result | Decision |
|---|---|---|---|
| Do five supplied lemmas recover withheld inflected forms? | 14/15 forms; 6.12% WER on 60 negative controls | 0/15 forms; 37.10–37.62% control WER; about 2.54× latency | Failed the held-out-inflection test |
| Do 20 explicitly supplied surface forms help? | 41/67 target hits; 10.50% target WER | 66/67 hits; 8.34% aggregate WER, but seven recordings regressed, unexpected terms appeared, and latency was 2.65–2.67× | Useful recall signal, but failed the per-clip safety gate |
These are bounded natural read-speech experiments, not a universal language claim. They show that this mechanism is neither morphological generalisation nor a safe product candidate under the current evidence. Read the held-out-form report and complete-audio supplied-form replay for cohort limits and measurement details.
People evaluating a materially different policy can still help without publishing private speech. Follow the privacy-redacted contribution workflow and attach only the aggregate report to issue #21. The local fixtures, transcripts, vocabulary terms, and paths stay on the contributor's Mac. Technical background: FluidAudio custom-vocabulary documentation at Presspeech's pinned revision.
The ANE path is consistently faster than the prior GPU path on these clips, with p50 results around 92-152 ms.
Both backends produced essentially identical transcripts on the synthetic test audio, including the same TTS artifacts.
The benchmark measures compute latency, not energy. Power measurement remains a future bench improvement.
Every local engine below was measured on the same audio, on the same machine, scored with the same text normaliser, one engine at a time. Rows are engines, labelled with the apps that offer them, so apps that share an engine share its accuracy. Presspeech's engines are highlighted. Lower is better in every numeric column; the best measured value is in bold. Cloud-only tools are listed but not measured or estimated.
| Engine (apps) | LibriSpeech clean WER | LibriSpeech other WER | FLEURS 5-language WER | Long-form WER | Latency p50 | Latency p90 | Model size | Real-time factor | Peak memory | Offline |
|---|---|---|---|---|---|---|---|---|---|---|
| Parakeet TDT v3 Presspeech (default), MacParakeet, FluidVoice, Handy, VoiceInk, Superwhisper Pro |
1.82% | 4.12% | 6.51% | 3.59% | 54 ms | 91 ms | 483 MB | 0.009 | 83 MB | Yes |
| Parakeet Ultra Presspeech (opt-in), VoiceInk |
1.71% | 3.65% | 5.46% | 3.41% | 57 ms | 122 ms | 632 MB | 0.009 | 726 MB | Yes |
| Whisper large-v3-turbo VoiceInk, Superwhisper, Handy, FluidVoice |
1.89% | 4.12% | 5.39% | 2.63% | 1.10 s | 1.29 s | 1.62 GB | 0.153 | 2.04 GB | Yes |
| Whisper small VoiceInk, Superwhisper, Handy, FluidVoice |
2.77% | 7.21% | 11.37% | 4.67% | 354 ms | 574 ms | 488 MB | 0.053 | 918 MB | Yes |
| Whisper base VoiceInk, Superwhisper, Handy, FluidVoice |
4.33% | 11.44% | 21.99% | 7.12% | 151 ms | 264 ms | 148 MB | 0.023 | 425 MB | Yes |
| Apple DictationTranscriber macOS Speech framework; Apple Speech option in VoiceInk, FluidVoice |
7.47% | 15.69% | 12.70% | 9.33% | 160 ms | 364 ms | macOS asset | 0.028 | n/a (system process) | Yes |
| Apple SpeechTranscriber macOS Speech framework; Apple Speech option in VoiceInk, FluidVoice |
1.80% | 4.75% | no Polish | 2.93% | 117 ms | 288 ms | macOS asset | 0.021 | n/a (system process) | Yes |
| Wispr Flow (cloud) Wispr Flow |
cloud, not measured | — | — | — | — | — | — | — | — | No |
FLEURS 5-language WER is the mean of the German, Spanish, French, Italian and Polish corpus WERs, shown only when an engine supports all five. Model size is the downloaded model on disk (Apple's engines use a macOS system asset whose size is not reported). Peak memory is the benchmark process's footprint; Core ML and the Neural Engine keep part of a model outside the process, so treat it as a lower bound rather than a RAM requirement, and it is not ranked.
| Engine | LibriSpeech test-clean | LibriSpeech test-other | FLEURS German | FLEURS Spanish | FLEURS French | FLEURS Italian | FLEURS Polish | English long-form |
|---|---|---|---|---|---|---|---|---|
| Parakeet TDT v3 Presspeech (default), MacParakeet, FluidVoice, Handy, VoiceInk, Superwhisper Pro |
1.82% | 4.12% | 7.31% | 4.47% | 7.60% | 4.30% | 8.86% | 3.59% |
| Parakeet Ultra Presspeech (opt-in), VoiceInk |
1.71% | 3.65% | 6.19% | 3.45% | 6.76% | 3.54% | 7.37% | 3.41% |
| Whisper large-v3-turbo VoiceInk, Superwhisper, Handy, FluidVoice |
1.89% | 4.12% | 5.52% | 3.80% | 6.69% | 3.98% | 6.96% | 2.63% |
| Whisper small VoiceInk, Superwhisper, Handy, FluidVoice |
2.77% | 7.21% | 11.66% | 5.69% | 14.44% | 8.17% | 16.90% | 4.67% |
| Whisper base VoiceInk, Superwhisper, Handy, FluidVoice |
4.33% | 11.44% | 21.21% | 10.63% | 28.19% | 16.37% | 33.54% | 7.12% |
| Apple DictationTranscriber macOS Speech framework; Apple Speech option in VoiceInk, FluidVoice |
7.47% | 15.69% | 15.25% | 7.88% | 18.33% | 7.40% | 14.64% | 9.33% |
| Apple SpeechTranscriber macOS Speech framework; Apple Speech option in VoiceInk, FluidVoice |
1.80% | 4.75% | 8.12% | 5.06% | 8.62% | 5.75% | not supported | 2.93% |
| Engine | LibriSpeech test-clean | LibriSpeech test-other | FLEURS German | FLEURS Spanish | FLEURS French | FLEURS Italian | FLEURS Polish | English long-form |
|---|---|---|---|---|---|---|---|---|
| Parakeet TDT v3 Presspeech (default), MacParakeet, FluidVoice, Handy, VoiceInk, Superwhisper Pro |
1.82% | 4.12% | 7.31% | 4.47% | 7.33% | 4.30% | 8.86% | 3.59% |
| Parakeet Ultra Presspeech (opt-in), VoiceInk |
1.71% | 3.65% | 6.19% | 3.45% | 6.46% | 3.54% | 7.37% | 3.41% |
| Whisper large-v3-turbo VoiceInk, Superwhisper, Handy, FluidVoice |
1.89% | 4.12% | 5.52% | 3.80% | 6.69% | 3.98% | 7.22% | 2.63% |
| Whisper small VoiceInk, Superwhisper, Handy, FluidVoice |
2.77% | 8.27% | 11.66% | 5.69% | 14.44% | 8.17% | 17.51% | 4.67% |
| Whisper base VoiceInk, Superwhisper, Handy, FluidVoice |
4.33% | 11.60% | 21.21% | 10.63% | 28.19% | 16.37% | 34.20% | 7.12% |
| Apple DictationTranscriber macOS Speech framework; Apple Speech option in VoiceInk, FluidVoice |
not supported | not supported | not supported | not supported | not supported | not supported | not supported | not supported |
| Apple SpeechTranscriber macOS Speech framework; Apple Speech option in VoiceInk, FluidVoice |
not supported | not supported | not supported | not supported | not supported | not supported | not supported | not supported |
Apple's engines have no auto-detect mode and are absent from the auto-detect table. Auto-detect can transcribe in the wrong language; that cost is part of the auto WER. Whisper's auto mode adds a language-detection pass, so its auto-detect latency (in the JSON) is higher than shown here.
| Engine | LibriSpeech test-clean | LibriSpeech test-other | English long-form |
|---|---|---|---|
| Parakeet TDT v3 Presspeech (default), MacParakeet, FluidVoice, Handy, VoiceInk, Superwhisper Pro |
54 ms / 91 ms / 0.009 | 53 ms / 85 ms / 0.009 | 221 ms / 438 ms / 0.005 |
| Parakeet Ultra Presspeech (opt-in), VoiceInk |
57 ms / 122 ms / 0.009 | 54 ms / 82 ms / 0.009 | 206 ms / 326 ms / 0.005 |
| Whisper large-v3-turbo VoiceInk, Superwhisper, Handy, FluidVoice |
1.10 s / 1.29 s / 0.153 | 1.09 s / 1.23 s / 0.171 | 3.04 s / 4.18 s / 0.065 |
| Whisper small VoiceInk, Superwhisper, Handy, FluidVoice |
354 ms / 574 ms / 0.053 | 342 ms / 523 ms / 0.058 | 1.81 s / 2.13 s / 0.036 |
| Whisper base VoiceInk, Superwhisper, Handy, FluidVoice |
151 ms / 264 ms / 0.023 | 137 ms / 236 ms / 0.024 | 840 ms / 1.02 s / 0.017 |
| Apple DictationTranscriber macOS Speech framework; Apple Speech option in VoiceInk, FluidVoice |
160 ms / 364 ms / 0.028 | 179 ms / 512 ms / 0.040 | 1.27 s / 1.81 s / 0.028 |
| Apple SpeechTranscriber macOS Speech framework; Apple Speech option in VoiceInk, FluidVoice |
117 ms / 288 ms / 0.021 | 112 ms / 264 ms / 0.022 | 747 ms / 1.12 s / 0.016 |
| Test set | Clips | Audio | Normaliser |
|---|---|---|---|
| LibriSpeech test-clean | 500 | 61.6 min | whisper EnglishTextNormalizer |
| LibriSpeech test-other | 500 | 54.2 min | whisper EnglishTextNormalizer |
| FLEURS German | 100 | 22.3 min | whisper BasicTextNormalizer |
| FLEURS Spanish | 100 | 20.3 min | whisper BasicTextNormalizer |
| FLEURS French | 100 | 17.1 min | whisper BasicTextNormalizer |
| FLEURS Italian | 100 | 24.7 min | whisper BasicTextNormalizer |
| FLEURS Polish | 100 | 17.3 min | whisper BasicTextNormalizer |
| English long-form | 13 | 10.9 min | whisper EnglishTextNormalizer |
| Engine | Configuration measured | Apps, price and licence |
|---|---|---|
| Parakeet TDT v3 | FluidAudio 0.17.7, CoreML on the Apple Neural Engine (presspeech-bench --backend v3, the app's released chunking config) | Presspeech (default): Free, MIT; MacParakeet (default): Free, GPL-3.0; FluidVoice (option): Free, GPL-3.0; Handy (option): Free, MIT; VoiceInk (option): Free trial, then $25 one-time, GPL-3.0; Superwhisper (Pro option): Pro $8.49/month, Proprietary. |
| Parakeet Ultra | FluidAudio 0.17.7, CoreML on the Apple Neural Engine (presspeech-bench --backend ultra, files checked against the app's pinned manifest) | Presspeech (opt-in): Free, MIT; VoiceInk (option): Free trial, then $25 one-time, GPL-3.0. |
| Whisper large-v3-turbo | whisper.cpp 1.9.5 (Homebrew), Metal GPU, flash attention, beam 5 / best-of 5, 4 threads (whisper-cli defaults); ggml-large-v3-turbo.bin | VoiceInk (option (whisper.cpp)): Free trial, then $25 one-time, GPL-3.0; Superwhisper (Free-tier local Whisper): Free tier, Proprietary; Handy (option): Free, MIT; FluidVoice (option): Free, GPL-3.0. Apps other than VoiceInk do not document which Whisper runtime they use; this row measures whisper.cpp. |
| Whisper small | whisper.cpp 1.9.5 (Homebrew), Metal GPU, flash attention, beam 5 / best-of 5, 4 threads (whisper-cli defaults); ggml-small.bin | VoiceInk (option (whisper.cpp)): Free trial, then $25 one-time, GPL-3.0; Superwhisper (Free-tier local Whisper): Free tier, Proprietary; Handy (option): Free, MIT; FluidVoice (option): Free, GPL-3.0. Apps other than VoiceInk do not document which Whisper runtime they use; this row measures whisper.cpp. |
| Whisper base | whisper.cpp 1.9.5 (Homebrew), Metal GPU, flash attention, beam 5 / best-of 5, 4 threads (whisper-cli defaults); ggml-base.bin | VoiceInk (option (whisper.cpp)): Free trial, then $25 one-time, GPL-3.0; Superwhisper (Free-tier local Whisper): Free tier, Proprietary; Handy (option): Free, MIT; FluidVoice (option): Free, GPL-3.0. Apps other than VoiceInk do not document which Whisper runtime they use; this row measures whisper.cpp. |
| Apple DictationTranscriber | macOS Speech framework: SpeechAnalyzer + DictationTranscriber, on-device, short-form content hint (presspeech-bench --backend apple) | macOS Speech framework (built in): Included with macOS, Proprietary; VoiceInk ("Apple Speech" option): Free trial, then $25 one-time, GPL-3.0; FluidVoice ("Apple Speech" option): Free, GPL-3.0. The public API closest to macOS Dictation, not the Dictation feature itself. VoiceInk and FluidVoice do not say which Speech module their Apple Speech option uses. |
| Apple SpeechTranscriber | macOS Speech framework: SpeechAnalyzer + SpeechTranscriber (.transcription preset), on-device (presspeech-bench --backend apple-speech) | macOS Speech framework (built in): Included with macOS, Proprietary; VoiceInk ("Apple Speech" option): Free trial, then $25 one-time, GPL-3.0; FluidVoice ("Apple Speech" option): Free, GPL-3.0. VoiceInk and FluidVoice do not say which Speech module their Apple Speech option uses. |
| Wispr Flow (cloud) | cloud, not measured | Wispr Flow (only engine): Free 2,000 words/week; Pro $15/month, Proprietary. |
Machine-readable results: benchmarks/engine-leaderboard.json. Reproduce with experiments/swift-bench/run-engine-leaderboard.sh; the committed engine-leaderboard-manifest.json lists every clip ID and PCM checksum.