Fine-tuning Qwen3-ASR for Sinhala – Part 7: The Results

In part 6 the training run was still going, and I’d written down what the model had to do before I’d publish it. This part is how it ended: the blend I picked, the test results, and a hole in my own test set that part 1 had warned me about.

The run

The pass finished at 7:44 in the evening: 1,816 steps of 128 recordings, every Sinhala training recording once and the replay set twice. It took 11 hours 43 minutes of training, plus the 51-minute stop for the blending tests in part 5, and memory peaked at 37 GB. The Sinhala loss went from 1.82 on the first step to about 0.08 on the last, and the replay loss finished about where it started, around 0.02.

English, measured four times on the way, came back as the learning rate fell: 7.19% WER at a quarter and at halfway, 6.72% at three quarters. Then the last check read 62.89%.

One recording, 4,684 words

The model had got stuck on one English recording, 8.9 seconds long. It started well, “The Tibetan Buddhism is based on the teachings of Buddha, but were extended by Mahayana”, and then wrote “, but by the Mahayana” over and over. It stopped 69 seconds later, 4,684 words in, at the speech library’s limit of 8,192 tokens. That one recording in 394 took the English average to 63%. Without it, the model’s English was 6.06%, still 0.65 points behind the original, so it needed blending anyway.

It’s the loop from part 5, in English this time. The speech library stops a runaway output only when its last 24 tokens hold 3 or fewer different tokens, and “, but by the Mahayana” is more than that. It’s the same fix, too: cap each transcript by the length of its audio. For an 8.9-second recording, 64 tokens plus 30 per second is about 330 tokens, not 8,192.

The blend stopped this one as well. With 80% of the training kept, the model wrote “…extended by Mahayana, but by the Mahayana Buddhists.” and finished. That’s still wrong, but it’s one wrong phrase rather than a minute of them.

Setting the dial

As planned in part 6, I exported the last checkpoint at several blends and measured each one on the same dev sets:

NB: 8-bit exports of the last checkpoint. Sinhala on 500 dev recordings, English on FLEURS dev’s 394. The original model scores 5.41% on the same English.

The whole curve had moved since the quarter-way checkpoint. At 80%, the final checkpoint’s Sinhala is better than the quarter-way one’s was with nothing taken away (6.10% against 9.32%), and its English is within 0.3 points of the original.

The rule I’d set picks the lowest Sinhala CER whose English is within half a point of the original’s. That’s 85%, at 0.47 points. I took 80% instead. On 500 recordings, 85%’s 0.09-point lead in Sinhala is noise. And with 394 English recordings, a result 0.47 points out could land either side of the line on the test set. If the model failed there, I couldn’t go back and pick another blend without turning the test set into a second dev set. 80% leaves a margin for the check that can’t be retaken.

The test

The test sets are the ones I’d kept for the end: 8,748 recordings of the 24 OpenSLR speakers held out of training, and FLEURS’s English test. Apart from the early probe in part 5, nothing had been measured on them. I ran them once, on the 80% blend, exported to 8-bit exactly as it will ship and run the way the app runs it: no language given, greedy decoding.

NB: the original model doesn’t support Sinhala. Part 1’s best, Omnilingual with Sinhala-only decoding, got 30% CER and 84% WER, on recordings it had been trained on. The next section splits the Sinhala test in two, and the number to quote is 7.1%, not 6.4%.

Everything part 6 asked for held. English is 0.28 points worse than the original, inside the half point. No transcript was cut short or ran away. A Sinhala recording, 4.3 seconds on average, took a median of 243 ms on my M4 Pro MacBook Pro, 17 times faster than real time. An English one took 226 ms. Sinhala is slower per second of audio because Qwen’s vocabulary splits it into more than twice as many tokens (part 2).

The test set had seen the sentences

Before writing this up I checked my split against part 1’s second lesson, “know what your test set has already been trained on”, and it didn’t pass.

I’d split OpenSLR 52 by speaker, so no voice in the test is in training (part 3). But OpenSLR’s volunteers read from a shared pool of sentences, and many sentences were read by more than one person: the 172,134 training recordings hold only 98,966 different sentences. 6,201 of the 8,748 test recordings (71%) read a sentence that’s also in the training data, in someone else’s voice. The dev set is the same, at 72%.

So most of the test measured a new voice reading a sentence the model had already learnt to write, and that flatters it. I scored the two groups separately:

The gap is smaller than I feared. A sentence the model had already learnt, in a new voice, came out only about a point better than a new one, so it isn’t just reciting what it has seen. But dictation is all new sentences, so 7.1% is the number to expect, and it’s the one on the model card. Everything else held on the new sentences too: every recording was recognised as Sinhala, and 246 of the 305 English words (81%) came out in English letters.

The dev set had the same flaw, so the earlier numbers in part 5 (12.2%, 9.1% and 7.4%) and the blend table above are flattered in the same way. That matters less for choosing between blends, which all had the same advantage, than for the numbers themselves. Part 2’s comparison with Meta’s published 7.2% for its 7B model survives, though: on new sentences the finished model gets 7.1%. The test sets differ, so it’s still a rough comparison.

The fix for the next run is to split by sentence as well as by speaker: keep every recording of a test or dev sentence out of training. That costs 15,850 training recordings, 9.2% of the Sinhala.

English words in English letters

OpenSLR has few chances to show this. 147 test recordings mix Sinhala and English, and 287 are English phrases on their own, most of them search-like (“derana news”, “orlando bloom 2004”). Across all of them, 730 of the 889 English words (82%) came out in English letters:

The last line shows both ways it goes wrong. “email” came out in Sinhala letters, the way OpenSLR spelt it before I rewrote the transcripts (part 3). The training data has it in English letters only 12 times. And ලබාදෙන්න, “give”, came out as two words, ලබා දෙන්න. The training transcripts use both spellings (28 and 49 times). By WER that line has 3 errors in 7 words, although a Sinhala reader would only notice “email”. That’s why CER is the fairer number for Sinhala (part 1).

English words aren’t what makes the WER high, either. Only 138 of the 2,547 new-sentence recordings have one, and the 2,409 that are all Sinhala score 6.85% CER and 30.39% WER. A word there averages about five characters, counting vowel signs, so a 7% CER spread evenly would already put a wrong character in about 30% of words. Most wrong words are near misses, too: 37% are one character off, often the same word with a different ending (කියල and කියලා, පස්සෙ and පස්සේ), and 14% of word errors are only where a space goes.

Whether English words come out right in real dictation, which is full of them, is still the open question from part 6. 434 read sentences can’t answer it.

Publishing it

The model is on Hugging Face as Nerdstorm/Qwen3-ASR-0.6B-Sinhala-8bit, under CC BY-SA 4.0 because OpenSLR 52 is share-alike, with Qwen’s Apache 2.0 notice kept. The model card has the numbers above, including the split by sentence, the training recipe, the data and its attribution, and the limitations. It needs mlx-audio-swift at the corrected revision or later (part 4).

What’s left

• Record real dictation. A proper test set of my own speech, mixing Sinhala and English, is the only way to know whether this works for its real job.
• In the app: cap the output by the audio’s length, which now has two loops as evidence, then offer Sinhala as an opt-in model with clean-up off (part 6).
• The next run: split by sentence as well as speaker, and add mixed Sinhala and English recordings if the dictation test shows English words coming out in Sinhala letters.

What I learnt

1. Split by sentence as well as by speaker. A speaker split keeps voices out of the test, not sentences. Count how many test sentences are in training before trusting a number; here it was 71%.
2. Read the worst transcript, not just the average. One loop in 394 recordings turned 6% into 63%, and the average alone doesn’t say whether that’s one disaster or a general decline.
3. Blending is a dial, and the end of the run is the time to set it. Setting it on the last checkpoint took 20 minutes. A retrain would have taken twelve hours.
4. Leave a margin on the checks you can’t retake. A blend that only just passes on dev can fail on test, and after that the test set can’t pick another.
5. For Sinhala, trust CER over WER. WER counts a spelling variant or a word split as a whole wrong word.

Next: getting the model into the app.