Automatic speech recognition is very good until somebody mentions the name of a local school, a village, a politician, a festival, or a cricket ground.
Then things get strange.
In an earlier test with audio from Barbados, GPT Transcribe and GPT Audio 1.5 heard the event name “Rise Together” as “Rice Together”, while Qwen3.5-Omni Plus and Flash got it right.
Those are different models from the Qwen3-Omni checkpoint used here, but the result motivated this experiment.
Acoustically, the mistake is understandable.
Culturally, it is wrong.
A person who knows the local context has another signal available: they know that Rise Together is the plausible name.
That led me to a question: can we give an audio-native model a stronger model of Barbados, using text that already contains the names, institutions, places, events and relationships it is likely to hear?
So I took an archive of Barbados newspapers, turned it into 51.6 million tokens, and used it for domain-adaptive pretraining of the Thinker inside Qwen3-Omni.
The result is promising, but not conclusive.
The adapted model produced higher scores on our preliminary Barbados knowledge probe, particularly on people and institutions.
It also got slightly worse on a small set of general-knowledge controls.
And, most importantly, we have not yet shown that it transcribes audio more accurately.
This is a very preliminary result.
It came from our first training run, which we stopped at step 500 of a planned 801 steps.
We were also still extracting the newspaper archive, so the 51.6 million training tokens represent the material available for that run rather than the full corpus we ultimately intend to use.
This post is about what we have actually demonstrated, what broke along the way, and why I think the experiment is still worth pursuing.
The Problem Is Not Just Acoustic A transcription model is doing more than converting sound into letters.
When audio is clean and a word is common, the acoustic evidence can be enough.
But real radio is compressed.
Music leaks into speech.
Presenters talk quickly.
Phone-in guests have different microphones and accents.
Proper nouns may be rare or absent from a model's original training data.
At that point, transcription becomes a contest between plausible sequences: For Barbados, the useful context includes things such as: Kensington Oval Cave Hill Samuel Jackman Prescod Polytechnic Crop Over constituency and parish names local politicians, athletes, performers and organisations the ordinary relationships between all of those things This is related to contextual biasing in automatic speech recognition.
In Deep Context, Pundak and colleagues showed that an end-to-end recogniser could use supplied contextual phrases, including out-of-vocabulary terms, and reported relative word-error-rate improvements of up to 68% on their tasks.
Later work on trie-based deep biasing and shallow fusion reported a 19.5% relative improvement over existing contextual-biasing methods, with particular attention to rare long-tail words.
Those systems explicitly supply a biasing list at inference time.
Our approach is different: we are trying to make local context part of the model's learned language prior.
The papers do not validate our method directly, but they do support the underlying idea that context can help a recogniser resolve rare and ambiguous language.
Why Qwen3-Omni?
I selected Qwen3-Omni-30B-A3B-Instruct because it is natively multimodal.
It accepts text, audio, images and video, and it can produce text or speech.
More importantly for this experiment, its audio understanding and text generation are parts of one end-to-end architecture.
According to the Qwen3-Omni technical report, the model uses a Mixture-of-Experts Thinker-Talker design: The Thinker consumes multimodal representations and generates text.
The Talker is responsible for generating streaming speech tokens.
An Audio Transformer, or AuT, turns audio into representations consumed by the Thinker.
The Thin