Fine-tuning Qwen3-ASR for Sinhala – Part 1: Measuring the Models That Already Exist

Back story

Earlier this week I released Live Transcribe, a free and open source dictation app for Apple silicon Macs. You hold fn in any app, speak, and the text is typed where the cursor is. Speech recognition and the clean-up step both run on the Mac with MLX, so audio never leaves the computer.

It works well in English. The first language I wanted to add after that was Sinhala, and there was no speech model I could download that handles it well enough for dictation. So I ended up fine-tuning one. This series is my notes on how that went, written down for anyone else trying to teach a speech model a new language (and for future me):

1. Measuring the models that already exist (this post)
2. Why Qwen3-ASR as the base model
3. Preparing the training data
4. Training on a Mac, the same way the app runs
5. The first training run
6. What’s next
7. The results

Most of the work happened over two days, 25 and 26 September 2026, pairing with Claude Code. It wrote most of the code and ran the measurements, and I made the calls. All the numbers come from those runs, on my M4 Pro MacBook Pro.

What I was after

The goal from the start was a light model that already knows several languages, tuned to learn Sinhala as well. I wasn’t after something that could take production workloads. I wanted something good enough for live transcription on my laptop, useful for writing articles or replying to friends and family in Sinhala, that still fits in a MacBook’s limited memory. On Apple silicon the GPU shares that memory with everything else that’s open, so a small model matters.

The requirements for Sinhala were the same as for English, plus one:

• Light: it has to fit in a MacBook’s memory alongside everything else that’s open. The whole app uses about 3 GB today.
• Offline: everything runs on the Mac.
• Fast: text should appear within about a second of finishing a sentence.
• Permissive licences: the app is MIT licensed, so the model, and whatever it was trained on, have to allow that.
• Mixed script: people who speak Sinhala mix English words in all the time. I want those in English letters, the way people type them: මම meeting එකට යනවා, not මම මීටින් එකට යනවා.

Two numbers: CER and WER

Speech recognition is scored by comparing the model’s transcript with a human one:

• WER (word error rate): the share of words that are wrong, missing or extra.
• CER (character error rate): the same thing, counted in letters.

For reference, the app gets a word or two in every 100 wrong on my English test clips. For Sinhala both numbers matter. Sinhala words are long, built from letters and vowel signs, and one wrong vowel sign makes the whole word wrong. So a transcript can be close letter by letter and still have most of its words wrong. That’s exactly what happened below.

Even the reference transcripts don’t agree on spacing (“හැකි ය” and “හැකිය” both appear), so every model’s Sinhala WER looks worse than its real mistakes. That’s why Meta reports Sinhala by CER, and why CER is the fairer number when comparing models.

Test data

Transcribed Sinhala speech is hard to come by:

• OpenSLR 52 has 224 hours of read speech: 185,293 recordings of 478 speakers, released by Google under CC BY-SA 4.0. It’s the only large set with an open licence.
• FLEURS, the usual multilingual test set, has no Sinhala.
• Common Voice needs an account to download, and its Sinhala set is small.
• YouTube has thousands of hours of Sinhala, but its terms don’t allow downloading and the videos are copyrighted. A model trained on them would be a legal risk for an open source app, so I ruled that out.

I took a random sample of 300 OpenSLR recordings, from 221 speakers. This wasn’t a big or perfect test, but I tried to keep it fair: every model got the same clips, on the same Mac.

Note: OpenSLR 52 is in the training data of Meta’s Omnilingual models below, so they were tested on recordings they had already seen. Their numbers flatter them.

The models

Meta Omnilingual ASR (CTC, 300M and 1B). Released in November 2025 under Apache 2.0, it covers more than 1,600 languages including Sinhala, and it’s very fast. I ran the MLX versions through speech-swift.

Whisper large-v3-turbo. OpenAI’s model lists Sinhala, and the app’s speech library can already run it with the language forced to Sinhala.

Two others claim Sinhala, but I didn’t run them. Meta’s Omnilingual LLM-ASR 7B is the family’s big model, and Meta reports 7.2% CER for Sinhala, but it’s roughly ten times the size of the model I ended up with and runs at about real time, far too slow for live dictation on a laptop. Meta’s MMS models are licensed for non-commercial use only.

The results

NB: scored on the 292 clips whose reference transcripts are in Sinhala letters. The 300M ranges cover its 4-bit and 8-bit versions. Whisper was stopped after 38 clips, and its error rates go over 100% because it kept adding text that isn’t there.

Right sounds, wrong letters

Reading the transcripts, the Omnilingual models mostly heard the speech correctly and then wrote it in the wrong script. About 65% of the 300M model’s transcripts came out in Bengali letters, with some Telugu and Kannada, sometimes switching within a sentence. The 1B model mostly wrote romanised Sinhala (“sangite mehema demateka”).

That comes from how these models work. A CTC model picks a letter (or a blank) for every 20 milliseconds or so of audio, from one alphabet covering all its languages, and there’s no way to tell it which language it’s hearing. Bengali, Telugu and Sinhala letters cover much the same sounds, so it writes whichever script it’s most used to. Meta’s paper says the same: without being told the language, most of its errors are the wrong language or the wrong script.

Forcing Sinhala letters

Since the sounds were right, I tried restricting the output to Sinhala letters. The obvious way, blocking every other letter, failed badly. The blank won almost everywhere and whole words vanished: “රැකියා සොය සොයා යන්නකු” came out as just “ර”.

What worked was lighter. For each slice of audio, keep the model’s own choice between blank, space and letter, and only when it picked a letter, swap in the likeliest Sinhala letter. That halved CER to about 30%, at no cost in speed.

WER stayed at 84%, though. The output sounded right read aloud, but it was misspelt everywhere: vowel signs dropped or added, the hal kirima (්) missing, and words split in the wrong places. Only 32 of the 292 clips came out with CER under 10%.

Allowing English letters as well, for the mixed-script requirement, made it worse (56–61% CER). The model then put Latin letters into most sentences. It can’t choose the script word by word.

There’s one more problem. The model’s alphabet has no zero-width joiner, an invisible character Sinhala needs for letter combinations like ්‍ර (rakaransaya) and ්‍ය (yansaya). 47 of the 300 reference transcripts use them, so no amount of training would get those words right.

Whisper

Whisper was much worse. Forced to Sinhala, it looped on characters (“වෙන්න්න්න්…”), wrote Gujarati or Gurmukhi letters anyway, produced broken Unicode, and took about 3.8 seconds per clip. I stopped it after 38 clips.

What I learnt

1. Measure on your own audio before trusting a table. Meta’s per-language table gives Sinhala 7.2% CER for its 7B LLM-ASR model, and nothing for the small, fast CTC models I could run in the app. 300 clips and an afternoon told me more than the papers.
2. Know what your test set has already been trained on. Good numbers on OpenSLR would have meant little for Omnilingual. Bad numbers on recordings it had already seen meant a lot.
3. A model can hear right and write wrong. Without a way to tell the model the language, the script is a guess. That became the first requirement for the model to fine-tune.
4. Decoding tricks are cheap, but they have a ceiling. Restricting the letters halved the character errors and did almost nothing for the words.
5. Look at both CER and WER. 30% CER sounds almost usable. 84% WER means nearly every word needs fixing, which is useless for dictation.
6. Check the alphabet for every character your language needs. A missing invisible character means some words can never come out right.

Conclusion

None of the models I could download was usable for Sinhala dictation, so the plan became fine-tuning one. The numbers to beat, from the best result above, are 30% CER and 84% WER.

Picking which model to fine-tune came down to a requirement I hadn’t written down yet: one model for every language, not a separate one for Sinhala. That’s the next part.