claude
431e052e6f
Add MatchText field to NormalizedInput — a lossy lexical matching view derived from ingress text: TrimSpace → NFKC → lowercase → collapse Unicode whitespace. Does NOT fold ё→е, strip punctuation, strip wake words, rewrite numbers, or invoke morphology. Both ingress sites (voice + text) construct MatchText at entry. No existing consumer reads MatchText yet — it is dark data for future opt-in migration. Vikunja: #725
33 lines
1.0 KiB
Go
33 lines
1.0 KiB
Go
package router
|
|
|
|
import (
|
|
"strings"
|
|
"unicode"
|
|
|
|
"golang.org/x/text/unicode/norm"
|
|
)
|
|
|
|
// NormalizeMatchText derives a lossy lexical matching view from raw ingress text.
|
|
// The result is intended ONLY for consumers that explicitly opt into
|
|
// case-insensitive, whitespace-collapsed matching. It must not be used for
|
|
// user-visible text, entity identity, quoted content, free-form arguments,
|
|
// or persisted utterances without proving the transformation is safe for
|
|
// that consumer.
|
|
//
|
|
// Pipeline: TrimSpace → Unicode NFKC → lowercase → collapse Unicode whitespace.
|
|
// Does NOT fold ё→е, strip punctuation, strip wake words, rewrite numbers,
|
|
// or invoke morphology.
|
|
func NormalizeMatchText(in string) string {
|
|
out := strings.TrimSpace(in)
|
|
out = norm.NFKC.String(out)
|
|
out = strings.ToLower(out)
|
|
// Collapse runs of Unicode whitespace (including non-breaking space,
|
|
// thin space, ideographic space, etc.) to a single ASCII space.
|
|
out = strings.Join(strings.FieldsFunc(out, func(r rune) bool {
|
|
return unicode.IsSpace(r)
|
|
}), " ")
|
|
return out
|
|
}
|
|
|
|
|