balbal-v1

A small Turkish speech recognition model (45M parameters, 15M active): Conformer-CTC with a looped refiner and a product-key acoustic memory, decoded with a 6-gram token KenLM. Runs on CPU.

• Small: 15.4M active parameters per audio frame (45.4M stored). Whisper large-v3-turbo has 18× as many parameters, Whisper large-v3 34×.

• Fast on a CPU: 96× faster than real time on 8 threads, 30× on a single thread; a short phone-call turn (≤ 3 s) is transcribed in about 80 ms on one thread.

• Strong on phone calls: 11.93% WER on Mu-Bench Turkish (official harness), ahead of OpenAI GPT-4o-mini-transcribe (12.82), Deepgram Nova-3 (13.37), Microsoft Azure Speech (14.02) and Qwen3-ASR 1.7B (14.76) on the published leaderboard. Google Chirp 3, Gemini and ElevenLabs Scribe v2 are still ahead (full table below).

• Blog post (architecture, training, evaluation): https://erayy.com/balbal-blog

• Try it in the browser: https://huggingface.co/spaces/erayyapagci/balbal-v1-demo

• Inference and training code (Apache 2.0): https://github.com/erayyap/balbal

• Every evaluation transcript behind the tables below (per-clip hypotheses, scores, speed runs): https://huggingface.co/datasets/erayyapagci/balbal-v1-evals

Model

Part	Details
Acoustic model	Conformer-CTC (NeMo EncDecCTCModelBPE), 16 layers, d_model 176, 80 mel, 4x subsampling (40 ms frames)
Looped refiner	2-layer Transformer refiner on top of the encoder output, 3 passes (each pass takes the previous CTC distribution as input)
Acoustic memory	Product-key memory: 500×500 subkeys, 4 heads, top-16, value dim 120, 3-frame context
Vocabulary	SentencePiece 1024 pieces + CTC blank (1025 classes)
Language model	6-gram token KenLM over the SentencePiece pieces
Decoder	flashlight-text lexicon-free beam search: alpha 0.75, beta 1.5, temperature 1.2, beam 64, top-12 tokens, threshold 8
Parameters	45.4M stored / 15.4M active (most of the memory table is not read on any given frame)

Long audio is cut into fixed 30-second chunks (no VAD); each chunk is decoded separately. Output is cased, punctuated Turkish. Numbers, codes and e-mail addresses come out spelled as words ("yüz yirmi üç"); there is no inverse text normalization.

Results

Compared with other open models, all run on the same clips (FLEURS tr test 743 clips, Mu-Bench tr-TR 846 clips):

Model	Params (stored / active)	FLEURS tr WER	Mu-Bench official WER	Mu-Bench official sigWER	CPU speed (RTTF)
balbal-v1	45.4M / 15.4M	6.54	11.93	25.18	96x
Duyu-small	49.3M / 19.3M	6.62	15.70	35.22	27x
MihuAI turkish-stt	66M	8.16	24.59*	49.53*	15x
MihuAI turkish-stt-v2	66M	5.26	10.91*	21.16*	15x‡
omniASR LLM 7B	7B	5.31	21.10	44.68	–
Whisper large-v3-turbo	809M	4.07	24.65†	47.04†	1.6x
Whisper base	74M	25.99	40.88†	68.68†	9.9x

FLEURS is scored with balbal-normalizer (the scoring normalizer in the code repository, normalizer/metrics_v2.py) for every model. The Mu-Bench columns are the official Mu-Bench harness with its GPT-4.1 judge; sigWER (significant WER) is the share of utterances with at least one error that changes the meaning. RTTF (higher is faster) = seconds of audio per second of processing, on the same 100 FLEURS + Mu-Bench clips, batch 1, 8 threads of one CPU (AMD Ryzen AI Max+ 395), each model with its own fp32 runtime (balbal-v1: inference/runtime_onnx.py + KenLM). omniASR LLM 7B was not timed on CPU. Stored / active: the acoustic memory of balbal-v1 and Duyu-small is stored in full but only a few of its rows are read per frame.

* MihuAI's official runs used their INT8 releases.

‡ MihuAI turkish-stt-v2 has the same architecture and size as v1, so it shares v1's speed run: balbal-v1 is about 6.4× faster than both.

† Whisper falls into repetition loops on some clips (Turbo 1, base 4), for which the judge's normalization does not finish; those clips are counted as fully wrong.

How sure are these differences?

Paired bootstrap over utterances (10,000 resamples), balbal-v1 minus the other model, in percentage points with 95% intervals. Negative = balbal-v1 is better; an interval that excludes 0 is a significant difference.

vs	Mu-Bench official WER	Mu-Bench sigWER	FLEURS WER
Duyu-small	−3.78 [−4.80, −2.77]	−10.05 [−13.24, −6.74]	−0.08 [−0.48, +0.33] (tie)
MihuAI turkish-stt	−12.67 [−14.14, −11.23]	−24.35 [−27.90, −20.92]	−1.62 [−2.12, −1.13]
MihuAI turkish-stt-v2	+1.02 [+0.03, +2.00]	+4.02 [+0.95, +7.09]	+1.27 [+0.84, +1.70]
omniASR LLM 7B	−9.18 [−10.83, −7.64]	−19.50 [−23.40, −15.60]	+1.23 [+0.70, +1.76]
Whisper large-v3-turbo	−12.73 [−15.05, −10.51]	−21.87 [−25.89, −17.85]	+2.47 [+1.96, +2.97]
Whisper base	−28.96 [−30.91, −27.03]	−43.50 [−47.28, −39.60]	−19.46 [−20.48, −18.46]

On phone calls (Mu-Bench) balbal-v1 is significantly better than every open model in the table except MihuAI turkish-stt-v2, which is more accurate (official WER 10.91 vs 11.93, sigWER 21.16 vs 25.18) but about 6.4× slower on the same CPU. On read speech (FLEURS) it ties Duyu-small, beats MihuAI v1 and Whisper base, and is behind MihuAI v2 and the much larger Whisper large-v3-turbo and omniASR LLM 7B.

Commercial APIs on Mu-Bench

System	Mu-Bench official WER	sigWER
Google Chirp-3	8.03	14.30
Google Gemini-3.5-Transcribe	8.68	18.20
ElevenLabs Scribe-v2	9.43	19.15
balbal-v1 (15.4M active, on a CPU)	11.93	25.18
OpenAI GPT-4o-Mini-Transcribe	12.82	28.61
Deepgram Nova-3	13.37	24.35
Microsoft Azure-Speech-v1	14.02	30.73
Qwen3-ASR-1.7B (Modal)	14.76	30.61
Qwen3-ASR-1.7B (Baseten)	15.03	31.91
xAI Grok-STT	15.94	34.04

Commercial systems: the published Mu-Bench leaderboard (tr-TR). balbal-v1: our run of the same official harness and GPT-4.1 judge. Deepgram Nova-3 has slightly fewer meaning-changing errors (sigWER 24.35 vs 25.18) despite its higher WER.

CPU threads

Same 100 clips, batch 1, ONNX runtime + KenLM (inference/runtime_onnx.py), one fresh process per setting, AMD Ryzen AI Max+ 395.

Threads	Speed (× real time)	Short clips (≤ 3 s), median latency	Model load
1	30×	81 ms	1.4 s
2	44×	48 ms	1.5 s
4	58×	43 ms	1.5 s
8	68–77× (96× on an idle machine)	33–44 ms	1.0–1.3 s

These were measured in one session while other jobs ran on the machine, so they are lower than the 96× headline (measured idle); the scaling between settings is what they show. Peak memory of the whole process stays under 0.8 GB.

Files

conformer.pt                  PyTorch state dict (acoustic model + refiner + memory)
model_config.yaml             NeMo model config
refiner.json                  refiner / memory config
decoder.json                  decoder settings
tokenizer/                    SentencePiece model + vocab
lm/model.binary               6-gram token KenLM (trie)
lm/tokenizer.model            same SentencePiece model
onnx/model-int8.onnx          8-bit ONNX of balbal-v1 (69 MB, see "Quantized ONNX")
onnx/model-int4.onnx          4-bit ONNX of a quantization-aware fine-tune, no acoustic memory (34 MB)

This folder is the MODEL_DIR the code expects.

Usage

Linux x86-64, Python 3.12.

git clone https://github.com/erayyap/balbal && cd balbal
huggingface-cli download erayyapagci/balbal-v1 --local-dir balbal-v1
python3.12 -m venv .venv && . .venv/bin/activate
python inference/setup_env.py --device cpu --model-dir balbal-v1   # pinned torch + NeMo, builds the decoder, runs checks
python inference/transcribe.py --model-dir balbal-v1 speech.wav --device cpu

For an ONNX CPU runtime without torch/NeMo, see inference/export_onnx.py and inference/runtime_onnx.py in the code repository.

Run the commands from the code checkout (cd balbal).

Quantized ONNX (8-bit and 4-bit)

Two smaller graphs for CPU, onnxruntime only (no torch/NeMo): 16 kHz mono float32 audio in, natural-log CTC probabilities [frames, 1025] out (40 ms frames; class 1024 = blank). Decode them with the same KenLM decoder (decoder.json, lm/) or greedily.

File	Size	What it is	Acoustic memory
onnx/model-int8.onnx	69 MB	balbal-v1, every MatMul and the memory table in 8-bit (dynamic, per channel)	kept (8-bit)
onnx/model-int4.onnx	34 MB	4-bit quantization-aware fine-tune of balbal-v1 (2,000 updates), acoustic memory removed; encoder feed-forward and pointwise layers in 4-bit (blocks of 16, MatMulNBits), attention/refiner/head in 8-bit	removed

WER / CER (%), deployed KenLM decoder, balbal-normalizer (same clips as above; Mu-Bench here is not the official GPT-4.1 scoring):

Graph	DEV (721)	FLEURS tr test	Mu-Bench tr-TR	Peak RAM*
fp32 (conformer.pt exported)	10.03 / 3.65	6.59 / 1.86	11.79 / 6.02	~510 MB
8-bit	10.18 / 3.73	6.60 / 1.86	11.88 / 6.00	~420 MB
4-bit	12.50 / 4.26	8.88 / 2.41	14.15 / 6.82	~390 MB

*onnxruntime 1.30, 4 threads, Python process including onnxruntime, clips up to 30 s, without the KenLM (lm/model.binary, 124 MB, memory-mapped). Setting enable_cpu_mem_arena = False lowers the 8-bit peak to ~325 MB. The 8-bit graph is within noise of fp32; the 4-bit graph trades about 2.5 WER points for size.

import onnxruntime as ort, soundfile as sf
session = ort.InferenceSession("balbal-v1/onnx/model-int8.onnx", providers=["CPUExecutionProvider"])
wave, sr = sf.read("speech.wav", dtype="float32")          # mono, 16 kHz
log_probs = session.run(None, {"audio": wave[None]})[0]   # [frames, 1025]

Steering mode

balbal-v1 can be steered toward a sentence from another source, for example the text of a larger ASR system. A small offset is added to the acoustic-memory reads of each refinement pass and optimised for a few Adam steps on CTC(sentence), only on frames where balbal is unsure (max probability < 0.9). The weights stay frozen and the offset is thrown away after each clip. The model only moves where the audio allows it: acoustically implausible sentences are largely refused, and a sentence much shorter than what balbal hears is ignored.

With the text of Whisper-large-v3-turbo as the sentence (WER with balbal-normalizer, KenLM decoding):

Test set	balbal-v1	Whisper-large-v3-turbo alone	balbal-v1 steered by Turbo
FLEURS tr test	6.53	4.11	3.97
General dev set (Turkish)	10.03	8.36	7.20
Mu-Bench tr-TR	11.79	20.73	11.08
LibriSpeech test-clean (English)	22.91	3.85	4.75

• Where both models are good (FLEURS, general dev set), steering beats either one alone.
• Where balbal-v1 is weak (English: it is a Turkish model), steering brings it to nearly the steering model's level: 22.91 → 4.75, against 3.85 for Whisper alone.
• Where the steering model is weak (Mu-Bench phone calls, where Whisper makes almost twice balbal's errors), steering does not hurt: balbal keeps what it hears clearly. This gain is not statistically significant (paired bootstrap 95% interval [-1.58, +0.06]).

Two settings (steering/steer.json in the code repository):

Mode	Use it for	Settings
weak	a sentence that may be wrong (another ASR system, a guess)	3 steps, lr 0.03, only unsure frames
strong	a sentence from a trusted source	6 steps, lr 0.03
auto	runs Whisper-large-v3-turbo itself and steers toward its text with weak	needs transformers; a GPU is advised

Steering toward the ground truth

Given the correct transcript, steering nearly reproduces it, and it does not simply copy a wrong one. On 200 held-out clips:

Sentence given to steering	weak	strong
none (plain balbal-v1)	13.16	13.16
the correct transcript	4.04	1.74
the correct transcript with 15% of its words made wrong	7.03	8.02

With the correct transcript, strong cuts the errors by 87% (13.16 → 1.74). With the corrupted transcript, the result (7–8 WER) stays well below the 15% errors in the sentence: many of the wrong words are refused because the audio does not support them.

This makes steering usable as a check that a transcript matches its audio. Steer with strong toward the transcript; if the steered output still disagrees with it, or the sentence is refused (guarded), the audio and the text probably do not match (a wrong label, a cut or broken recording).

Steering itself takes about 0.25 s per clip on a CPU; Whisper on a CPU takes tens of seconds per clip.

python steering/steer_transcribe.py --model-dir balbal-v1 --mode auto speech.wav          # pip install transformers
python steering/steer_transcribe.py --model-dir balbal-v1 --mode weak --text "Yarın saat 15'te toplantı var mı?" speech.wav
python steering/steer_transcribe.py --model-dir balbal-v1 --mode strong --text "..." speech.wav

Each line of output is JSON with text (steered), plain_text (without steering), target (the sentence with digits spelled as words), guarded (true = the sentence was refused) and timings. Keep case and punctuation in the sentence.

License

The weights, tokenizer and language model in this repository are released under the balbal Community License 1.0 (see LICENSE):

• Free for research and any non-commercial use, including fine-tuning and sharing adapted versions.
• Free for small businesses' commercial use: under US$500k revenue in the last 12 months, fewer than 25 people and under US$250k raised from outside investors. You may use it to run your own business, including on your own customer calls.
• Not covered, at any size: selling or providing speech recognition, transcription, voice agents or other voice AI to others (as a product, API or platform) built on balbal-v1. The license grants no rights beyond the uses above.

Copies downloaded earlier under CC BY-NC 4.0 can keep being used under those terms. The code in github.com/erayyap/balbal is licensed separately under Apache 2.0.

The acoustic encoder was initialized from NVIDIA's stt_en_conformer_ctc_small (CC BY 4.0).

Citation

@misc{yapagci2026balbal,
  author       = {Eray Yapa{\u{g}}c{\i}},
  title        = {balbal-v1: A Small Turkish Speech Recognition Model},
  year         = {2026},
  month        = oct,
  howpublished = {\url{https://erayy.com/balbal-blog}},
  note         = {Code: \url{https://github.com/erayyap/balbal}}
}

---

Türkçe özet

balbal-v1, CPU'da çalışan küçük bir Türkçe konuşma tanıma modeli: Conformer-CTC, 3 geçişli döngülü düzeltici ve ürün-anahtar akustik bellek, 6-gram token KenLM ile çözülür (45.4M saklanan / 15.4M aktif parametre). FLEURS tr test %6.54 WER (balbal-normalizer), Mu-Bench tr-TR %11.93 (resmi puanlama). Aynı kliplerde FLEURS WER: Duyu-small 6,62, MihuAI turkish-stt 8,16, MihuAI turkish-stt-v2 5,26 (balbal-v1 6,54). Mu-Bench resmi WER / anlamlı hata oranı (sigWER): balbal-v1 %11,93 / %25,18, Duyu-small %15,70 / %35,22, MihuAI v1 %24,59 / %49,53, MihuAI v2 %10,91 / %21,16. Aynı CPU'da hız (8 thread, gerçek zamanın katı): balbal-v1 96x, Duyu-small 27x, MihuAI 15x (v2 aynı mimari ve boyutta; balbal-v1 yaklaşık 6,4 kat hızlı). Tek CPU thread'inde bile gerçek zamanın 30 katı; 3 saniyelik bir telefon cümlesi tek thread'de ~80 ms'de yazıya dökülüyor. Mu-Bench resmi puanlamasında ticari API'lerle: Google Chirp-3 %8,03, Gemini %8,68, ElevenLabs Scribe-v2 %9,43, balbal-v1 %11,93, OpenAI GPT-4o-mini %12,82, Deepgram Nova-3 %13,37, Microsoft Azure %14,02, Qwen3-ASR 1.7B %14,76. Tüm değerlendirme transkriptleri: https://huggingface.co/datasets/erayyapagci/balbal-v1-evals Aynı Mu-Bench resmi puanlamasıyla: omniASR LLM 7B %21,10 / %44,68, Whisper large-v3-turbo %24,65 / %47,04, Whisper base %40,88 / %68,68.

Küçük ONNX sürümleri: onnx/model-int8.onnx (69 MB, balbal-v1'in 8-bit hali; DEV 10,18 WER, fp32 10,03; akustik bellek dahil) ve onnx/model-int4.onnx (34 MB, 4-bit için eğitilmiş, akustik belleği çıkarılmış sürüm; DEV 12,50 WER). Yalnızca onnxruntime ile CPU'da çalışırlar.

Yönlendirme (steering): model, başka bir kaynaktan gelen bir cümleye doğru yönlendirilebilir (ör. daha büyük bir ASR sisteminin metni). Ağırlıklar donuk kalır; yalnızca modelin emin olmadığı karelerde bellek okumalarına küçük bir ofset eklenir, ses izin vermeyen cümleler büyük ölçüde reddedilir. Whisper-large-v3-turbo metniyle FLEURS tr testte WER 6,53 → 3,97 (yalnız Turbo 4,11), Mu-Bench'te 11,79 → 11,08 (yalnız Turbo 20,73), İngilizce LibriSpeech test-clean'de 22,91 → 4,75 (yalnız Turbo 3,85; balbal-normalizer). balbal'ın zayıf olduğu yerde yönlendirme, yönlendiren modelin seviyesine yaklaştırıyor. Kullanım: steering/steer_transcribe.py (--mode auto|weak|strong). Doğru transkripte yönlendirince (strong) 200 ayrılmış klipte WER 13,16 → 1,74; kelimelerinin %15'i bozulmuş bir transkriptte ise sonuç 7–8 WER'de kalıyor, yani sesin desteklemediği yanlış kelimelerin çoğu reddediliyor. Bu sayede steering, bir transkriptin sesiyle uyuşup uyuşmadığını kontrol etmek için de kullanılabilir.

Ağırlıklar balbal Community License 1.0 ile lisanslıdır (LICENSE): araştırma ve ticari olmayan her kullanım serbest; küçük işletmeler (son 12 ayda 500 bin dolardan az gelir, 25 kişiden az çalışan, dış yatırımcıdan 250 bin dolardan az yatırım) modeli kendi işlerinde ticari olarak da ücretsiz kullanabilir. Modelle başkalarına konuşma tanıma, transkripsiyon ya da sesli asistan ürünü, API'si veya platformu sunmak ise şirketin boyutu ne olursa olsun lisansın kapsamı dışındadır; lisans bu kullanımlar dışında bir hak tanımaz. Kod ayrı olarak Apache 2.0 lisanslıdır: github.com/erayyap/balbal. Teknik yazı: https://erayy.com/balbal-blog

Challenger: challenger-b2a

Model	DEV WER / CER	FLEURS test WER / CER	Mu-Bench WER / CER	CPU speed
challenger-b2a	8.72 / 3.24	5.07 / 1.48	11.43 / 5.54	38.3×
challenger-b2a (no LM)	11.52 / 3.81	7.09 / 1.94	11.69 / 5.90	44.1×
balbal-v1	10.21 / 3.72	6.54 / 1.85	11.78 / 6.01	96.4×

A new challenger is approaching.