Before fine-tuning speech recognition, test the glossary on words that are not there

Define the transcript before changing the context

NVIDIA published Nemotron 3.5 Transcribe on October 28. Its release describes a model for recorded audio whose transcription style can change through a natural-language instruction on each request. The New Stack explains promptable transcription through domain instructions and vocabulary lists. NVIDIA also released a fine-tuning toolkit, so prompting provides a useful baseline before a separate weight-adaptation trial.^[1][2]

NVIDIA reports fewer named-entity errors when supplying an entity list on ContextASR-Bench. MarkTechPost attributes the benchmark figures to NVIDIA. Those results do not establish performance on your recordings or prove that this proposed challenge set has been run. Fix the transcript convention first: verbatim wording, number formatting, punctuation and spelling rules must stay consistent across conditions.^[1][3]

Use vocabulary available before the recording

For this exercise, define a deployment glossary as vocabulary available without inspecting the held-out reference transcript. An existing project-name list or meeting agenda could qualify. Record its origin and when it was available. A list copied from the reference supplies privileged information. Label that condition separately; its result cannot establish what a realistic session glossary will achieve.^[1][2]

Prepare entity-bearing clips and otherwise comparable clips with no glossary entity. Use consented or synthetic recordings if you later execute the trial. Keep the model revision, audio segments and style instruction fixed. Compare no context, coarse domain context, a realistic session glossary, an incomplete glossary and a distractor-only glossary. These are proposed stress conditions, not documented failures of Nemotron.^[1][2]

Diagnose two fictional transcripts

No model produced the following outputs. Suppose the glossary includes the fictional project name Neris. Reference A says, 'Close the Neris ticket.' Its glossary-conditioned output says, 'Close the Neris ticket.' Reference B says, 'Close the support ticket.' Its output says, 'Close the Neris ticket.' Classify both before reading the answer.^[1]

A recovers the spoken entity. B introduces an unspoken entity while replacing an ordinary word. For ordinary word-error alignment, B contains a substitution. In the proposed entity-error sheet, flag it as an unspoken-term introduction. Keep these labels distinct so the entity check does not silently redefine word error rate. Scoring only entity-bearing clips would miss this control failure.^[1]

Keep recovery and introduced errors separate

Define matching rules before inspecting outputs. Specify accepted spelling variants, case, hyphens and spacing. Count recovered entity mentions against all spoken entity mentions. Record wrong replacements separately. On entity-absent clips, count how many clips contain at least one introduced glossary term and report the total control-clip denominator. Also inspect overall word error rate and changes outside entity spans.^[1]

Here is a filled hypothetical row, not a benchmark result: five entity-bearing clips contain one entity each, and five control clips contain none. A session glossary recovers all five spoken entities but introduces a glossary term in one control clip. Record recovery as five of five and affected controls as one of five. Complete recovery alone cannot settle the acceptance decision.^[1]

A reusable sheet needs clip ID, reference location, entity-bearing or absent status, model revision, style instruction, glossary provenance, context condition, spoken entities, recovered entities, wrong replacements, introduced terms, other transcript changes and reviewer notes. Keep per-clip rows so aggregate gains cannot conceal a recurring failure on one kind of recording.^[1][2]

Choose the next trial from the errors

Set task-specific acceptance limits before running the comparison. The sources supply no universal safe threshold. If prompting meets your limits on both clip groups, consider a bounded application trial. If relevant errors remain, decide whether glossary repair, a different prompt or a separately resourced fine-tuning trial addresses them. Reuse the held-out controls afterward. No inference, fine-tuning or employment-outcome test was performed for this article.^[1][2][3]