Fine-tuning Qwen3-ASR for Sinhala – Part 6: What's Next

In part 5 the first run failed its English check a quarter of the way through, and blending it back towards the original model won most of the English back. This part is the plan from here, the questions I can’t answer yet, and the short version of the series.

Getting English to hold

The run carries on with the doubled replay to the end of the pass. Then I’ll try several blends on the last three checkpoints (steps 908, 1,362 and 1,816) and keep the one with the lowest Sinhala CER whose English is within half a point of the original model’s 5.41%. If no blend manages that with good Sinhala, the fallbacks are, in order: the run with the replay tripled, a lower learning rate, and freezing parts of the model.

What “good enough” means

Before the model goes anywhere, all of these have to hold, measured the way the app runs it:

• It’s picked by Sinhala CER on the dev set, from the blends whose English WER is within half a point of the original model’s.
• On the held-out OpenSLR test speakers, CER is well under part 1’s 30%, with every word in Sinhala or English letters.
• FLEURS English test WER is within about half a point of the base model’s 4.96%.
• Transcripts aren’t cut short more often, and the speed is measured.

That English tolerance assumes Sinhala ships as an opt-in model rather than as the default for everyone. If it’s good enough to be everyone’s model, the bar for English goes up.

Publishing it

Once it’s stable, the model goes on Hugging Face as Nerdstorm/Qwen3-ASR-0.6B-Sinhala-8bit: private first, to check that it downloads and transcribes, then public. The model card will cover the base model, the data and its attribution, the training recipe, the test results against the base model, and the limitations. The licence will be CC BY-SA 4.0, because OpenSLR 52 is share-alike, with Qwen’s Apache 2.0 notice kept. It will need mlx-audio-swift at the corrected revision or later (part 4).

In the app

• The library update the model needs is merged. One thing to watch: English improved on real recordings, but 4 of the app’s 65 synthetic dictation clips now come out wrong where they used to be right (“emoji” became “M O G”, and “comma” became “common”). Four clips is too few to call a trend, but I want to check it against real dictation.
• The clean-up step is English-only. The clean-up prompt for the Qwen3-1.7B model, its self-correction adapter, and the rules for fillers, lists and spoken commands were all written for English. The first Sinhala version will run with clean-up off, so only the simple rules apply.
• Cap the output length. Sinhala loops slip past the speech library’s repetition check (part 5). The fix is to cap each transcript by the length of its audio, for example 64 tokens plus 30 per second, which real speech never reaches. It has to land before the app offers Sinhala.
• Speed. With Qwen3-ASR, the live transcript takes up to 1.6 seconds to show a sentence (the 95th percentile), against my target of 1.5. The fix I have in mind is to start transcribing speculatively in the silence at the end of a sentence, before the app has decided the sentence is over. Sinhala will be slower again because of its extra tokens (part 2), so this matters more for it.

Things I don’t know yet

1. Can 224 hours of read speech teach a 0.6B model a new language well enough for dictation? 7.4% CER halfway through the first pass is encouraging, but read news sentences aren’t dictation.
2. Will it write English words in English letters in real speech? Only 2% of the training sentences have any English. The real test is recordings of my own mixed Sinhala and English. If it doesn’t carry over, the fix is more recordings like that, not more OpenSLR.
3. Is Qwen’s vocabulary good enough for Sinhala? If accuracy or speed disappoints, the next step is adding Sinhala tokens, which means a bigger vocabulary and more training.
4. How much English can I afford to lose? For an opt-in model, maybe half a point. For everyone’s model, close to none.
5. Which languages does the Sinhala model need to keep? The replay set spreads about 100 hours over five languages, and only about 60% of it is English. If the Sinhala model only needs Sinhala and English, the whole replay budget can go to English, which should protect it much better.

Maybe later

• Record a proper test set of my own dictation, mixing Sinhala and English, and add recordings like it to training if the model needs them.
• Report the fractional audio-slot count to mlx-audio-swift (part 4), and the quantiser’s rounding to MLX (part 5). A fix to the first changes the Sinhala model’s inputs, so it would need testing again.
• If an on-device model with strong Sinhala and English appears (a future Qwen release that adds Sinhala, say), look at the base model again.
• Other languages Qwen3-ASR doesn’t cover. The same recipe should carry over: data split by speaker, a replay set labelled by the model itself, and a trainer that copies the app.

The short version

If you only read one part of this series, these are the things I’d take from it:

1. Measure first, on your own audio, and know what your test set has already been trained on (part 1).
2. Choose the base model for what you can control, like telling it the language, and ship it before you fine-tune it (part 2).
3. The transcripts are the spec. The model writes the way its training transcripts do (part 3).
4. Label replay data with the model’s own output, filtered against the human transcript (part 3).
5. Train on exactly what the app feeds the model, quirks included (part 4).
6. Judge by transcripts, not by the loss (part 5).
7. Watch the old languages from the first step, and decide how far they may slip before you start (part 5).
8. Before retraining to fix forgetting, try blending with the original model (part 5).
9. Check what’s actually on disk (part 5).

The results are in part 7.