Model bake-off: keep Qwen3.5-0.8B, LFM2.5 is worse in Russian #20

2026-07-31T10:08:20+02:00

claude commented

2026-07-31 10:08:20 +02:00

One commit, doc only. Vikunja #278 and #250.

Verdict: keep Qwen3.5-0.8B. Nothing changes in the config.

Same fixture, same server flags, one model at a time on an otherwise idle box.

	Qwen3.5-0.8B	LFM2.5-1.2B Instruct	LFM2.5-1.2B Thinking
intent accuracy	60.5%	52.6%	36.8%
full accuracy	36.8%	32.9%	21.1%
Russian	22/61	18/61	10/61
English	6/15	7/15	6/15
route errors	0	0	0
p50 latency	1.05s	2.47s	2.42s

Your Russian worry was right, and it is one specific failure. LFM2.5 hears almost any Russian imperative or short phrase as a reminder. перезапусти докер, закрой жалюзи, доброе утро, спасибо тебе, заметка: продлить домен — all filed as reminders. English is a wash (7/15 against 6/15), so the entire deficit is Russian. It also returns empty or wrong fact keys in Russian (поужинал → empty, сходил в душ → water), and leaked German once.

The tool-calling edge did not show up. act is the closest thing the fixture has to tool-calling and LFM2.5 was worse at it.

It is also 2.4x slower despite being the model you would switch to for speed-adjacent reasons. That alone would be hard to justify at equal accuracy, and the accuracy is not equal.

The Thinking variant is not viable at all: 36.8%, no latency saving for the thinking it does, and with thinking turned off it collapsed to 28.9% with 2 route errors.

To be fair to LFM2.5: no load failures, no grammar violations, zero route errors in either shippable config. The output contract held perfectly. The problem is judgement in Russian, not format.

One harness bug found, not fixed: the report hardcodes the label (0.8B) for every config no matter which gguf is loaded, so two runs from different models look identically named. Model identity was verified per run through /v1/models, so these numbers are correct — but that label needs fixing before anyone else runs this.

One commit, doc only. Vikunja #278 and #250. **Verdict: keep Qwen3.5-0.8B. Nothing changes in the config.** Same fixture, same server flags, one model at a time on an otherwise idle box. | | Qwen3.5-0.8B | LFM2.5-1.2B Instruct | LFM2.5-1.2B Thinking | |---|---|---|---| | intent accuracy | **60.5%** | 52.6% | 36.8% | | full accuracy | **36.8%** | 32.9% | 21.1% | | **Russian** | **22/61** | 18/61 | 10/61 | | English | 6/15 | **7/15** | 6/15 | | route errors | 0 | 0 | 0 | | p50 latency | **1.05s** | 2.47s | 2.42s | **Your Russian worry was right, and it is one specific failure.** LFM2.5 hears almost any Russian imperative or short phrase as a reminder. `перезапусти докер`, `закрой жалюзи`, `доброе утро`, `спасибо тебе`, `заметка: продлить домен` — all filed as reminders. English is a wash (7/15 against 6/15), so **the entire deficit is Russian**. It also returns empty or wrong fact keys in Russian (`поужинал` → empty, `сходил в душ` → `water`), and leaked German once. **The tool-calling edge did not show up.** `act` is the closest thing the fixture has to tool-calling and LFM2.5 was worse at it. It is also **2.4x slower** despite being the model you would switch to for speed-adjacent reasons. That alone would be hard to justify at equal accuracy, and the accuracy is not equal. **The Thinking variant is not viable at all:** 36.8%, no latency saving for the thinking it does, and with thinking turned off it collapsed to 28.9% with 2 route errors. To be fair to LFM2.5: no load failures, no grammar violations, zero route errors in either shippable config. The output contract held perfectly. The problem is judgement in Russian, not format. **One harness bug found, not fixed:** the report hardcodes the label `(0.8B)` for every config no matter which gguf is loaded, so two runs from different models look identically named. Model identity was verified per run through `/v1/models`, so these numbers are correct — but that label needs fixing before anyone else runs this.

claude changed target branch from master to overnight/recall-margin

2026-07-31 10:39:24 +02:00

claude added 2 commits 2026-07-31 10:39:24 +02:00

Bake off LFM2.5 against Qwen3.5-0.8B on the RU routing fixture 4ca68d2f3f

Vikunja #278 / #250. Keep Qwen: LFM2.5-1.2B loses 8 points of intent
accuracy, all of it Russian, and runs 2.4x slower.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Merge commit '4ca68d2' into overnight-jul31 1db0fcfcd0

kami commented

2026-07-31 18:51:52 +02:00

right - md files worth the single commit. /s

kami commented

2026-07-31 20:22:02 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:22:04 +02:00

kami referenced this issue from a commit

2026-08-04 16:12:36 +02:00

Merge task/476 into the entity-reference branch (V-524)

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#20