escape control characters inside the string, not around it (V-44)

Qwen3-1.7B pretty-prints its JSON: it opens the object and writes three
newlines before the first key. escapeRawControls rewrote those structural
newlines into a literal backslash-n, which is legal nowhere outside a string,
so the object stopped parsing and came back as errBrokenJSON.

The comment claimed escaping unconditionally could not turn valid JSON into
anything else, on the grounds that JSON permits no control character outside a
string. It permits three: newline, tab and return are whitespace between
tokens, and that is what pretty-printing is made of.

Measured on the talk fixture against the resident model: 31 of 36 conversational
cases were failing generations and answered from the stub. Every chat reply and
every knowledge answer the resident model wrote was being discarded. Now 25/36
pass every check, 0 errors, and the 15 nudges stay at 15/15.
This commit is contained in:
2026-08-05 14:02:53 +04:00
parent 9e15ff36aa
commit 4dbeca5a2e
2 changed files with 63 additions and 6 deletions
+33
View File
@@ -91,6 +91,39 @@ func TestEscapeRawControlsLeavesValidJSONAlone(t *testing.T) {
}
}
// Pretty-printed JSON is what Qwen3-1.7B writes: it opens the object and puts
// three newlines before the first key. Escaping those structural newlines made
// the object unparseable, so 31 of 36 conversational cases in the talk fixture
// answered from the stub (Vikunja #44, measured 2026-08-05).
func TestParseResponseMoodReadsPrettyPrintedJSON(t *testing.T) {
raw := "{\n\n\n \"response\": \"Хорошо настроение.\",\n \"mood\": \"neutral\"\n}"
text, mood, err := parseResponseMood(raw)
if err != nil {
t.Fatalf("err = %v, want nil — this is valid JSON, not a failed generation", err)
}
if want := "Хорошо настроение."; text != want {
t.Errorf("response = %q, want %q", text, want)
}
if mood != "neutral" {
t.Errorf("mood = %q, want neutral", mood)
}
}
// Both at once: structural newlines outside the strings, a raw one inside.
func TestParseResponseMoodRepairsInsideAndKeepsOutside(t *testing.T) {
raw := "{\n\t\"response\": \"первая\nвторая\",\n\t\"mood\": \"tired\"\n}"
text, mood, err := parseResponseMood(raw)
if err != nil {
t.Fatalf("err = %v, want nil", err)
}
if want := "первая\nвторая"; text != want {
t.Errorf("response = %q, want %q", text, want)
}
if mood != "tired" {
t.Errorf("mood = %q, want tired", mood)
}
}
// A tab and a bare control byte take the same path as the newline.
func TestParseResponseMoodRepairsOtherControls(t *testing.T) {
raw := "{\"response\": \"таб\tи \x01байт\", \"mood\": \"tired\"}"