Fine-tuning Qwen3-ASR for Sinhala – Part 5: The First Training Run
In part 4 the trainer passed its checks. This part covers the tests before the real run, and the run itself, which started at 7:09 this morning and should finish this evening.
A test the loss can pass on its own
The last test before a long run is to overfit on purpose: train on a handful of recordings, over and over, until the model knows them by heart. A trainer that can’t manage that is broken. I gave it 32 Sinhala recordings and 100 passes.
The first attempt looked like a pass. The loss, the number training pushes down, fell from 1.8 to 0.014. Then I had the model transcribe eight of the recordings from the audio alone, the way the app does. Four came out right. The other four came out as the sentence of a different recording in the 32.
The loss hid that because of how training scores the model. At each word it’s shown the correct sentence so far (teacher forcing, from part 4) and scored only on the next word. A model that has memorised 32 sentences only needs the first word or two from the audio; after that it can recite the rest. So a wrong start costs a couple of words’ worth of loss out of hundreds. Transcribing gives no such help: the model has to get the start right from the sound alone, and everything after follows from its own choices.
That left two explanations. Either the model hadn’t learnt enough yet, or the trainer was feeding it something slightly different from what the app does, so what it learnt didn’t carry over. The second would ruin a real run, so it mattered which. There was a clue in the log: the loss jumped from 1.8 to 2.8 over the first ten steps before coming down, which means the first updates were too big. The rerun eased in with a warm-up over the first ten steps:
| 32 recordings, 100 passes | Final loss | Transcribed right | Characters wrong |
|---|---|---|---|
| Full learning rate from the start | 0.014 | 4 of 8 | 52% |
| With a warm-up | 0.0000 | 8 of 8 | 0% |
The rerun also used the corrected spectrogram from part 4, so the two changes weren’t tested separately, but the early jump points at the warm-up. Now the test only passes if the loss is under 0.05 and the transcripts have fewer than 5% of their characters wrong.
How fast a 48 GB Mac can train
Three settings decide how long a pass over the data takes: the precision of the arithmetic
(32-bit, or bfloat16, a 16-bit format made for machine learning), how many tokens go through the
model at once, and whether the audio encoder is trained or frozen. The trainer’s bench command
timed real update steps of 128 recordings with each:
| Arithmetic | Tokens at once | Encoder | Seconds per step | One pass | Peak memory |
|---|---|---|---|---|---|
| 32-bit | 8,192 | trained | 123 (swapping) | – | 56.7 GB |
| 32-bit | 4,096 | trained | 21.4 | 9.4 h | 40.2 GB |
| bfloat16 | 4,096 | trained | 18.0 | 7.9 h | 35.2 GB |
| bfloat16 | 8,192 | trained | 69.6 (swapping) | – | 52.0 GB |
| bfloat16 | 8,192 | frozen | 13.4 | 5.9 h | 32.2 GB |
I went with bfloat16, 4,096 tokens and the encoder trained. Anything bigger doesn’t fit in 48 GB: macOS starts swapping to disk and training slows to a crawl. Freezing the encoder would save a quarter of the time, but the encoder is the part that hears, and for a language the model has never heard that seemed the wrong place to save.
Only the arithmetic is bfloat16. The weights the optimiser updates stay 32-bit, because at these learning rates an update is often smaller than bfloat16’s rounding step and would simply be lost. A check compares one batch’s gradients both ways, and they point the same way (cosine similarity 0.995).
The bench ran on a quiet Mac. The real run shares it with everything else I have open: a Linux VM, Live Transcribe with its own model loaded, a browser and an editor. For the first hour, while macOS moved those out to disk, steps took 26 seconds instead of 18. After that they settled at 18 to 19 seconds, and they still slow down whenever I do something heavy. Closing the big apps before a long run is worth it.
Choosing the learning rate
The learning rate is how big a step the optimiser takes at each update. Qwen’s own script uses 2×10⁻⁵. A language the model has never heard often wants more, so I ran two short trials, at that rate and at five times it, on exactly the same batches in the same order. In the first pass every batch is new to the model, so each step’s loss is effectively a test on unseen speech. The trainer logs Sinhala and the replay languages separately:
| Steps | Sinhala, 2×10⁻⁵ | Sinhala, 10⁻⁴ | Replay, 2×10⁻⁵ | Replay, 10⁻⁴ |
|---|---|---|---|---|
| 1–20 | 1.29 | 1.16 | 0.06 | 0.11 |
| 41–60 | 0.29 | 0.33 | 0.07 | 0.38 |
| 81–100 | 0.22 | 0.27 | 0.09 | 0.53 |
| 101–108 | 0.22 | 0.28 | 0.10 | 0.72 |
NB: mean loss per step, lower is better. Before training, the replay loss is about 0.03.
The faster rate learnt Sinhala faster for the first 40 steps and then fell behind, while its grip on the other languages slipped away. By step 108 its replay loss was seven times the slower rate’s, so I stopped it there.
Even the slower rate forgets
After 200 steps at the slower rate (an eighth of a pass), I exported the model to 8-bit the way the app will run it and tested it:
- Sinhala, on 500 recordings from speakers held out of training: 12.2% CER and 43.5% WER. Part 1’s best was 30% and 84%, so after an eighth of a pass it was already far better than anything I could download. It also recognised the language as Sinhala every time, without being told.
- English, on FLEURS test: WER went from 4.96% to 6.36%.
The export wasn’t the cause: the untrained model, exported the same way, scores 4.91%. Of the 647 English transcripts, 370 had changed. Some changes were real mistakes (“45 of the 17 total skiers”). Others were a change of style: numbers and units spelt out (“thirteen”, “50 kilometers” for “50 km”). The Sinhala transcripts write numbers as words, so the model was probably picking up the habit.
So for the real run I kept the slower rate and doubled the replay, from 15% to about 26% of what the model hears, which makes a pass 15% longer. The trainer also measures English four times a pass now, on FLEURS dev (keeping the test set for the end). The rule I set: if English slips by more than about a point at the first check, step 454, stop and change course.
Three bugs no loss curve would show
The real run nearly started from the wrong weights, and it took three fixes.
1. The 32-bit weights weren’t really 32-bit. The trainer converts the app’s 8-bit model to 32-bit at load. MLX’s dequantize does its arithmetic in the type of the scales it’s given, and the model file stores those in bfloat16, so asking for 32-bit output still gave bfloat16-rounded values. Fixed by converting the scales to 32-bit first.
2. Converting the model also converted its copy. The trainer keeps a 32-bit copy of every weight for the optimiser, and runs the model itself in bfloat16. The first attempt at the real run was stopped after a couple of minutes, and the checkpoint it saved was half the size it should have been: the copy had become bfloat16 too. In MLX Swift, asking for a 32-bit version of an array that’s already 32-bit gives back the same array, not a copy, and updating a model’s weights replaces their contents in place. So converting the model converted the “copy” with it. Fixed with a real copy, and a unit test that shows the trap.
3. MLX’s quantiser is slightly off. When exporting back to 8-bit, MLX’s own quantise function left 5.4% of the weights more than half a step from where they should be (up to 1.6 steps), because it rounds to bfloat16 part way through. The export now uses its own quantiser, which keeps every weight within half a step. An untrained round trip scores 4.91% on FLEURS English, against the original’s 4.96%.
None of this showed in the losses. The first two only moved the starting weights slightly, and both learning-rate trials started from the same rounded weights, so their comparison still stands. But that was luck. What caught them was looking at what was actually on disk. The real run restarted from the fixed code at 7:09.
Where it’s at
The first English check came at step 454, a quarter of the way through and about three hours in. Sinhala kept improving: 9.1% CER and 37% WER on the dev set, with the language recognised as Sinhala in every recording. Of the 31 English words in those dev sentences, it wrote 71% in English letters.
English didn’t hold. It went from 5.40% to 7.19% WER on FLEURS dev, nearly two points, where I’d set the line at one. Doubling the replay slowed the forgetting but didn’t stop it. By the rule above that’s a no-go, so I stopped the run at step 458.
Giving English back by blending
The plan was to start again with the replay tripled, about 11.5 more hours. But the forgetting had only fallen in proportion to the replay share, so tripling it would probably still miss. Before spending that, I tried something much cheaper: blending the trained model back towards the original. When the model is exported, each weight keeps a share of what training changed and takes the rest from the original model (the idea is called WiSE-FT). Each blend takes minutes to make and test:
| Share of the training kept | Sinhala CER (dev) | English WER (FLEURS dev) |
|---|---|---|
| 100% | 9.32% | 7.22% |
| 75% | 9.93% | 6.12% |
| 65% | 11.40% | 5.88% |
| 60% | 12.87% | 5.82% |
| 50% | 56.62% | 5.70% |
NB: 8-bit exports of the step-454 checkpoint. The original model scores 5.41% on the same English set.
English comes back much faster than Sinhala goes, down to about 65%, which is within half a point of the original. Below 60%, Sinhala falls apart. At 50%, one 4.3-second recording came out as 4,916 characters: the model got stuck repeating one Sinhala word about 700 times.
That loop is an app problem too. The speech library stops a runaway output when its last 24 tokens hold 3 or fewer different tokens, which catches English loops. A Sinhala word is many tokens (“විභාව” is 10, 8 of them different), so a Sinhala loop never trips it, and that recording took 66 seconds to transcribe. Blends of 60% and up had no loops in 500 recordings, but the app needs to cap the output by the length of the audio before it offers Sinhala.
So instead of starting again, the run carried on from step 458 with the doubled replay, and the blend gets picked at the end. Halfway through, at step 908, Sinhala was down to 7.4% CER and 30% WER on the dev set, and English had held at 7.19%, the same as at step 454, so the forgetting has levelled off. The one number going the wrong way is the mixed-script one: it wrote only about half of the 31 English words in English letters, down from 71%. That’s too few words to read much into, but it’s the one to watch (part 6).
Updated after the test (part 7): most of the dev set’s sentences are also in the training data, read by other speakers, so the Sinhala numbers in this part (12.2%, 9.1%, 7.4% and the blend table) are flattered. On test sentences it had never seen, the finished model gets 7.1%.
What I learnt
- Judge training by transcribing, not by the loss. Teacher forcing hides wrong starts.
- Use a warm-up, even for fine-tuning. A loss that jumps early means the first steps are too big.
- Benchmark on the real data, on the real machine. On a Mac the cliff is swapping, and the other apps you have open count.
- Compare learning rates on identical batches, and watch the old languages as well as the new one. The faster rate looked better on Sinhala for 40 steps.
- Forgetting starts straight away, and it isn’t only errors. A change of style, like spelling out numbers, is forgetting too.
- Set the go/no-go line before the run. Deciding afterwards how much lost English is acceptable is too easy to talk yourself into.
- Check dtypes and file sizes on disk. How a framework copies and converts arrays can quietly change your weights, and no loss curve will tell you.
- Before retraining to fix forgetting, try blending with the original model. It won most of the English back in minutes. Measure where the new language breaks, though: it breaks suddenly, not gradually.
Next: what happens from here, and the questions I can’t answer yet.