Fine-tuning Qwen3-ASR for Sinhala – Part 3: Preparing the Training Data

In part 2 I chose Qwen3-ASR 0.6B as the model to fine-tune. This part is about what it’s trained on. There are two sets: the Sinhala recordings it learns from, and a “replay” set that stops it forgetting everything else.

Sinhala: OpenSLR 52

All of the Sinhala comes from OpenSLR 52: 224 hours of people reading sentences from Sinhala blogs and news sites. It needed three things done before training.

1. Keep the test speakers out of training

I split the recordings by speaker rather than by recording, so nobody heard in training is used to test the model. Otherwise the test partly measures how well the model knows those voices.

Updated after the test (part 7): splitting by speaker wasn’t enough. OpenSLR’s speakers read from a shared pool of sentences, so 71% of the test recordings read a sentence that’s also in training, in another voice. Part 7 scores the new sentences separately, and the data script can now keep test sentences out of training too.

2. English words in English letters

This was the biggest job. OpenSLR’s transcripts spell English words in Sinhala letters: “ඉමිග්‍රේශන් එකට” (immigration එකට), “බ්ලොග් කියවන්නෙක්” (a blog reader). A model trained on them as they are would learn to do the same, which is the opposite of what I want.

So the English words had to be rewritten. The corpus has 64,055 different words, far too many to check by hand, but most are rare: the 29,887 words that appear at least three times cover 94% of the text. Ten AI agents (Claude again) sorted those into three groups:

Only the first group gets rewritten. Loanwords like bus are Sinhala words now, and names mostly carry Sinhala endings (ඉන්දියාවේ is “in India”), so they’d look odd in English letters. The rewrite covers the bare word and the small words written after it: “බ්ලොග් එකට” becomes “blog එකට”, “ලින්ක් කරලා” becomes “link කරලා”, and “මැෂින් එක” becomes “machine එක”.

It changed only 3,364 of the 172,134 training sentences (2.0%). That number is useful by itself. OpenSLR is read blog and news text, so it has very few English words, while real dictation from someone who speaks Sinhala at work is full of them (“meeting එක cancel කරන්න”). Whether the model learns to switch scripts from 2% of its data can only be tested on real dictation, which I’ll have to record myself.

3. Invisible characters

Sinhala uses the zero-width joiner, an invisible character, to build some letter combinations: ශ්‍රී (as in Sri Lanka) has one between ් and ර. Some OpenSLR words also have stray joiners at the start, where they do nothing. The preparation script removes joiners at the edges of words and keeps the ones inside them, so the model isn’t taught to write invisible characters that mean nothing.

Replay: not forgetting everything else

A model fine-tuned on one language gets worse at the others. It’s called catastrophic forgetting, and since Qwen3-ASR is now the app’s model for English, English can’t be allowed to slip. The usual defence is replay: mix recordings in the languages the model already knows into every training batch, so it keeps practising them.

The interesting question was what those recordings should be labelled with. The human transcripts that come with public datasets have their own habits (punctuation, how numbers are written, casing), and training on them would slowly change how the model writes English. So the replay set is labelled by the model itself. I ran the app’s own Qwen3-ASR over every recording and used its transcript as the answer to learn. Training then rewards the model for doing exactly what it already does.

That would also teach the model its own mistakes, so each label was compared with the human transcript and dropped if more than 30% of it was wrong.

Labelling took about two hours, with four copies of the model running at once on the Mac. (These are the numbers from the second labelling, after a fix in part 4 changed the model’s input.)

What the filter caught

Looking at what the 30% rule dropped turned up two problems in the data, not the model.

Silent Spanish. Spanish kept only 82% of its labels, against 98% or more for the others, and at first it looked like the model was cutting Spanish sentences short. It wasn’t. 490 of the Spanish recordings have no speech in them at all: their loudest moment is about 90 dB below full scale, about as quiet as 16-bit audio gets. Told to expect Spanish, the model writes “El.” and stops. Left to work out the language itself, it says there’s no speech, which is the right answer. The rule drops all 490.

Names in brackets. The Chinese transcripts write a translated name with the original in brackets, like 克里斯托弗·加西亚（Christopher Garcia）. The speakers only read the Chinese, so a perfect label lost every bracketed letter in the scoring. Scoring Chinese with and without the brackets, and taking the better of the two, saved 132 good labels. In German and French the bracketed text is read aloud, so the rule is only for Chinese.

The replay set starts as 15% of what the model hears in training. Part 5 explains why that turned out to be too little.

What I learnt

1. Decide how transcripts should be written before training. The model copies whatever the transcripts do. English words in English letters was a product decision, and it meant rewriting the data to match.
2. Split by speaker, and by sentence. Otherwise the test partly measures familiarity with the voices, or with the sentences (part 7).
3. Know what your data doesn’t cover. Only 2% of OpenSLR has English words in it, so mixed-language dictation needs its own recordings.
4. Label replay data with the model’s own output, and filter it against the human transcript. It keeps the model’s behaviour and style without teaching it its own mistakes.
5. Read what your filters drop. Silent recordings and bracketed names were problems in the data that would have looked like problems in the model.
6. Be careful with Unicode. An invisible character decides whether ශ්‍රී is spelt right.

Next: training on the Mac, and why the trainer has to feed the model exactly what the app does.