Mac dictation benchmark methodology for 2026

The primary question is not “Which app is best?” It is: how does each tested configuration perform on defined Mac dictation tasks under a controlled environment? We will report literal recognition error, preservation of critical meaning, cleanup usefulness, end-to-end latency, and insertion reliability as separate outcomes. A product may lead one dimension and trail another. We will not collapse the dimensions into one overall number unless a weighting rule is published before collection and sensitivity analysis shows how other reasonable weights change the ordering.

Claims are limited to the exact version, plan, model, mode, language, hardware, microphone, network, and test window recorded. Results will not justify statements about an untested mobile app, another language, a different local model, or a later release. Marketing adjectives such as “instant,” “perfect,” or “most private” are outside scope. Privacy comparison uses documented and observed processing behavior, not packet inspection presented as a complete legal audit.

Every primary run uses the same Mac model, chip, memory, macOS build, power mode, display arrangement, input device, microphone gain, audio route, and destination-app versions. Automatic operating-system and app updates are disabled during a collection block. Background CPU and memory load are recorded before each block, and runs are paused when the machine is indexing, thermal throttling, downloading updates, or handling another call. Local models receive a warm-up run that is discarded, and model downloads finish before timing begins.

Cloud runs use the same stable network and record connection type, approximate round-trip latency, packet loss, and whether a VPN is active. We do not simulate perfect connectivity while readers work on imperfect networks: a secondary stress block may test a declared impaired profile, but it is reported separately. The microphone is placed at a fixed distance and angle. System noise suppression, voice isolation, automatic gain, and app-specific enhancement settings are either disabled or recorded consistently.

The corpus contains scripted and semi-natural tasks. Scripted audio provides an exact reference for word and character scoring. Semi-natural prompts capture false starts, self-corrections, pauses, filler words, and sentence planning that clean-dictation products are designed to handle. The primary English set contains balanced short messages, multi-sentence prose, commands and punctuation, numbers and dates, proper nouns, technical vocabulary, and noisy-room samples. A Chinese set uses character error rate and language-appropriate tokenization. These two languages define the initial research sample, not the full input-language coverage of any product. Translation is a separate track because its reference and scoring differ from same-language dictation.

At least three speakers with different vocal characteristics record each core category, and no speaker records product-specific variants. Audio files are normalized only for peak safety, not denoised into unrealistic studio speech. Each item receives a stable corpus ID, reference text, speaker code, language, accent self-description, environment tag, task type, critical-token list, and difficulty note. Personally identifying or confidential workplace content is excluded. The eventual release will state which audio can be redistributed and provide hashes for any restricted files.

Each app is installed from its official distribution channel and tested on a publicly available plan. We record app version, account tier, selected recognition model, processing location, language, refinement mode, prompt, hotword list, context setting, insertion method, and every non-default option that may affect output. Products receive equivalent preparation opportunities: if all products support a custom vocabulary, the same declared list is used; a feature available to only one product is tested as that product feature and labeled rather than silently disabled.

Two tracks prevent category confusion. The literal track uses the least transformative documented configuration and measures transcript fidelity. The assisted-writing track uses the vendor's normal recommended cleanup configuration and measures meaning plus editing utility. If an app cannot disable rewriting, it may have no literal-track result. If an app has many local or cloud models, the protocol names each tested configuration and does not treat one result as representative of every model. Paid and free configurations are never mixed under one label.

A reproducible script generates a balanced random order so each product appears throughout the session instead of always being first or last. Product order is blocked by speaker and task category, with a recorded seed. Each product-configuration and corpus item is run at least three times in the primary block. A failed run is never silently replaced: the row remains in the raw data with a failure code, and any retry receives a new attempt number plus a reason such as operator error, app crash, network loss, or insertion failure.

Source audio is played through a fixed virtual or physical loop whose level has been calibrated in advance; live operator speech is not used for comparative timing. Before every run, the target field is cleared, cursor position is confirmed, and clipboard state is reset. Screen and event timestamps capture activation, audio start, audio end, result-ready state, and successful insertion. The operator does not edit output. Raw output and final inserted output are stored separately because a clipboard fallback may succeed even when direct insertion fails.

For English literal transcripts, word error rate is (substitutions + deletions + insertions) divided by reference words after a published normalization pass. We preserve a second score without aggressive normalization so punctuation, capitalization, numbers, and formatting errors remain visible. Chinese literal transcripts use character error rate with the same substitution, deletion, and insertion structure. Critical-token accuracy reports names, numbers, negations, URLs, commands, and domain terms separately because one changed digit may matter more than several harmless articles.

Assisted writing cannot be judged against a literal reference alone. Blind reviewers score meaning preservation, unsupported additions, omission of required facts, readability, and cleanup utility using anchored rubrics. Latency is reported from audio end to result ready and from audio end to text inserted, with median, p90, distribution, and failed-run count—not only an average. Insertion reliability is successful unedited placement divided by valid attempts, with clipboard fallback and wrong-target insertion reported separately. We report uncertainty intervals and per-speaker distributions where the sample supports them.

Human reviewers receive the prompt, reference facts, and anonymized candidate output under a random code. They do not see the product, price, interface, run order, or processing mode. Product names and distinctive boilerplate are removed only when removal does not alter the judged text; otherwise the item is flagged as potentially unblinded. At least two reviewers score every assisted-writing item independently. Disagreements beyond a predefined threshold go to a third reviewer, while the original scores remain in the released data.

The rubric uses examples and anchors rather than a vague one-to-five impression. Meaning preservation asks whether every required proposition and relationship remains intact. Unsupported addition records invented facts even if the prose sounds polished. Cleanup utility measures removal of fillers and false starts without erasing intentional nuance. Reviewers complete calibration items before the study and periodic hidden repeats during it. Agreement statistics and reviewer exclusion rules are published; low agreement blocks a confident comparative claim.

The primary analysis includes all valid attempts under the preregistered rules. A separate intention-to-test view includes operational failures so a product cannot appear strong merely because difficult runs disappeared. Outliers remain unless a recorded equipment or operator fault meets the exclusion rule. We publish the full distribution, sample size, missingness, exclusions, and confidence interval. Multiple comparisons are identified, and exploratory slices are labeled exploratory rather than promoted to primary findings after the fact.

A winner-style claim requires a practically meaningful margin, uncertainty that does not make the ordering fragile, acceptable performance on critical tokens, and no unresolved data-quality failure. A lower median latency is not called faster when p90 behavior or insertion failures reverse the experience. A small word-error advantage is not called more accurate if semantic review shows more altered facts. If results are mixed, the conclusion must be conditional—for example, stronger local literal recognition in the tested configuration but weaker cleanup utility—not forced into a universal ranking.

Privacy is documented as a fact pattern, not reduced to a score that implies a legal conclusion. For each configuration we record whether recognition can complete with networking disabled, which feature stops working offline, what the vendor says is transmitted, stored, retained, synchronized, or used for training, and the date and URL of the supporting policy. We distinguish audio, literal transcript, refined output, metadata, diagnostics, account data, and intentionally synced notes because vendors may treat them differently.

A successful offline run supports a narrow observation about that configuration; it does not prove no data ever leaves the app. A public privacy statement supports what the vendor promises; it does not independently verify implementation. Future results will present these evidence types in separate columns. No packet capture, policy interpretation, or security review will be marketed as a comprehensive privacy certification. Material policy changes after collection trigger a visible note rather than a quiet rewrite of the old test context.

The release package will include the protocol version, collection dates, environment manifest, configuration snapshots, corpus manifest, randomization seed, raw attempt rows, unedited outputs where redistribution is permitted, metric code, normalization rules, reviewer rubric, anonymized ratings, exclusions, and generated summary tables. Each file receives a checksum. Any manual correction is a new field with an audit note; raw output is immutable. Scripts must rebuild published tables from released rows without hand-entered totals.

Results remain pending until the minimum repetitions are complete, timing capture passes validation, blind-review agreement reaches the published threshold, critical-token scoring has been independently checked, and a second person reproduces the tables. If a product changes materially mid-study, we either restart its block under the new version or report the versions separately. The page will show a dated changelog and preserve the prior protocol. Failing the publication gate produces a limitations report, not a partial leaderboard.