Fine-tuning Qwen3-ASR for Sinhala – Part 2: Why Qwen3-ASR as the Base Model

In part 1 none of the models I could download handled Sinhala well enough for dictation, so the next question was which model to fine-tune.

What about a model that already knows Sinhala?

That was the first thing I looked at, before Qwen came into it. A model that already knows Sinhala should need less training, and forget less of everything else while it learns. These are the ones that claim it:

Meta’s MMS models were out regardless, being licensed for non-commercial use only.

So the models that know Sinhala either weren’t good at it, or were too big and slow to run live on a laptop. And none of them does English and the other major languages well too, which is what decided it.

One model for every language

The other option was a separate Sinhala model that the app switches to. I decided against it. I wanted one model that already handles the major languages (English, Chinese, Spanish, French and German) with Sinhala added to it. That has some practical advantages:

• People who speak Sinhala mix in English all the time, so a Sinhala model has to handle English words anyway.
• One model is one download (about 1 GB) and one thing in memory. The app doesn’t have to swap models when the language changes.
• Anything that improves the base model improves every language.

The candidates

These are the models the app’s speech library can already run, compared on the things that matter here:

On FLEURS, Qwen3-ASR also beats Parakeet (the app’s model at the time) in all four of the major European languages:

NB: these accuracy and speed figures are from speech-swift’s published benchmarks on an Apple M5 Pro, not my own runs, and Qwen’s LibriSpeech number is on the first 200 test recordings. They are good enough to compare the models, not to quote.

Why Qwen3-ASR

Qwen3-ASR doesn’t know Sinhala at all, so on paper it’s the worst starting point of the three. I chose it anyway, for these reasons.

Reason 1: it can be told the language. Parakeet and Omnilingual are pure speech models: audio in, letters out. Qwen3-ASR is an audio encoder attached to a small Qwen3 language model. The encoder turns the audio into something the language model can read, and the language model writes the transcript as the answer to a prompt. The answer starts with the language:

language English<asr_text>The meeting is at three.

That one detail fixes the biggest problem from part 1. Left alone, the model writes the language itself; the app can also write language Sinhala<asr_text> for it and let the model carry on in Sinhala. There’s no guessing the script.

Reason 2: it writes like a language model. It punctuates, and it can switch languages mid-sentence, which the mixed-script requirement needs.

Reason 3: the alternatives need more surgery. Parakeet’s vocabulary has no Sinhala letters, so adding Sinhala means a new vocabulary and retraining part of the model with NVIDIA’s NeMo tools, on NVIDIA hardware. Omnilingual already knows Sinhala, but with no punctuation, no language input and weaker English than what the app already had (4.3% against 2.4%), it couldn’t be the one model for everything.

Reason 4: it fits what’s already there. Qwen publishes an official fine-tuning script, and the app’s speech library already runs Qwen3-ASR, so there’s nothing new to add to the app.

Which size: 0.6B or 1.7B

Qwen3-ASR comes in two sizes, and the bigger one is more accurate, most of all on Qwen’s less common languages:

NB: the LibriSpeech and speed figures are from the same speech-swift benchmarks. The FLEURS groups are worked out from Qwen’s published averages over 12, 20 and 30 languages, assuming each is a plain average per language, so treat them as approximate.

So “30 languages” holds up well for the big ones, but the 0.6B is weak on the languages Qwen had less data for, and the 1.7B roughly halves the error there. That’s a fair warning for a language it has never heard. I still went with the 0.6B, because a light model was the goal from the start (part 1). It’s more than twice as fast, and speed is the hard limit for live dictation (the live transcript is already a bit slower than I’d like with it, part 6). It’s also small enough to fine-tune fully on a 48 GB Mac, which turned out to matter (part 4).

First, make sure English doesn’t get worse

Changing the app’s model is a product change on its own, before any Sinhala. So I first ran Qwen3-ASR over the app’s 65 English dictation test clips:

About as accurate, and slower, but fast enough. What changed more was the formatting. Qwen writes “uh” as “ah”, leaves numbers as words, and formats email addresses itself. The app’s rules for spoken commands and filler words were written around Parakeet’s output, so they needed fixing for Qwen’s. With those fixed, Qwen3-ASR became the app’s speech model in Live Transcribe 0.2.0 on 25 September, before any Sinhala training started.

The catch: Sinhala costs a lot of tokens

A language model writes text as tokens, pieces of words from a fixed vocabulary. Qwen’s has 151,643 tokens, built from the languages it was trained on. I counted how Sinhala fares:

Sinhala has almost no whole-word tokens: most Sinhala letters end up as one to three byte-sized tokens. That costs twice. The model takes about 2.3 times as many steps per second of speech as it does for English, which I estimated at 250–300 ms for a 5-second sentence (fine for dictation). And it has to learn to spell Sinhala byte by byte, which is harder than learning whole letters.

The standard fix is to add Sinhala tokens to the vocabulary before training. I decided not to, for now. It complicates everything after it, and it’s only worth doing if the fine-tuned model’s accuracy or speed disappoints.

What starting without Sinhala costs

It’s fair to ask whether a base that already knows Sinhala would have been better, if a good one existed. It would have helped in two ways:

• Less forgetting. A base that already had good Sinhala would need smaller changes to its weights, so English would drift less. The forgetting in part 5 is the real cost of starting from zero.
• Cheaper tokens. A vocabulary that covers Sinhala means fewer decoding steps, and fewer chances for the model to get stuck repeating itself.

Qwen didn’t have the forgetting problem with its own 30 languages, because it trained them all together from the start, on far more data, so no language overwrote another. Adding one afterwards is what causes it. For a sense of scale, Whisper was trained on 680,000 hours of audio. One pass of my training is about 417 hours: 218 hours of Sinhala, plus the 100-hour replay set (part 3) twice.

Looking back

Added in the afternoon, once there were numbers: halfway through the first training pass (part 5), the fine-tuned 0.6B was at 7.4% CER on speakers it had never heard. That’s about a quarter of the error of the best Sinhala-knowing model I could run on the Mac (30%), and about the same as Meta’s published 7.2% for its 7B model, which is roughly ten times the size. That last comparison is rough, because the test sets differ, and CER is the fair number to compare for Sinhala (part 1 explains why). If an on-device model with strong Sinhala and English does come along, a future Qwen release say, it’s worth another look.

Updated after the test (part 7): that 7.4% was measured on a dev set that shares most of its sentences with the training data, which flatters it. On test sentences it had never seen, the finished model gets 7.1%, so the comparison with Meta’s figure still roughly holds.

What I learnt

1. Measure the models that already know the language first. Here they either weren’t good at it or were too big to run live, and that’s what made fine-tuning the right call.
2. Choose the base model for what its decoder lets you control, not only its error rate. A model that can be told the language removed the worst problem from part 1 by design.
3. If one model serves every language, the new language can’t cost the old ones. Check the languages you already have on your own clips before switching, and look at formatting as well as WER.
4. Look past a multilingual model’s headline number. The 0.6B is good at its main languages and weak at its last ten, which says something about how it will take to a new one.
5. Count the tokens early. How well the vocabulary covers the new language sets both the decoding speed and how hard the learning is.
6. Ship the base model first. The fine-tune is then a change to something already in use, and differences like “uh” becoming “ah” get found and fixed on their own.
7. Check the licence of everything that goes into the model. Qwen3-ASR is Apache 2.0 and OpenSLR 52 is CC BY-SA 4.0, so the Sinhala model will be CC BY-SA 4.0. The app stays MIT.

Next: the training data, which took more work than I expected.