// Package checks holds the deterministic, $0 verdicts over one chunk's text: the output sanitizer // (contamination classes + the export-contract normalization), the cheap style flaggers and the // defect-class checkers, the excision coverage gate and the post-reflow regression guard. // // Every check here is a PURE function of its inputs — no store, no clock, no randomness, no LLM — // so a resumed run re-derives an identical verdict without re-billing. The package reports FACTS // (this class fired, this metric was breached); mapping a fact onto a chunk disposition is the // driver's job, so no disposition/flag vocabulary is imported here. All pair- and target-specific // patterns and word lists arrive as DATA (a *lang.Pack for the pair, lang.TargetChecks for the // target); a book with no pack runs the language-specific checkers inert rather than skipping them. package checks import ( "fmt" "sort" "strings" "unicode" "textmachine/backend/internal/lang" "textmachine/backend/internal/text" ) // cheapgates.go: four cheap, deterministic post-check flaggers on the FINAL chunk text (04-unhappy // §6/§9, plan v3). They are OBSERVABILITY, never hard gates (like the glossary post-check's default // flagger mode): a hit is recorded in the retrieval-state and surfaced in the report / chapter // passport, but it never changes a chunk disposition or costs an LLM call. All four are pure and // deterministic (no time/rand, sorted detail) so a resume re-derives identical counts. // // 1. dialogue-dash linter (Rosenthal §47–52): direct speech marked with straight quotes or a // hyphen/en-dash instead of the em-dash on a new line, or a chunk mixing the two styles. // 2. yofikator: inconsistent ё — the SAME word spelled both with ё and with е (Пётр/Петр), or a // brief ё-policy (all-yo / all-e) violated. // 3. translit-interjection blocklist: untranslated JP/EN fillers («ара-ара», «маа», «хмф», // «нани», «ауч», «упс») left in the Russian output (04-unhappy §9); per-project Allowlist. // 4. 万/億 magnitude gate: a CJK myriad/hundred-million magnitude in the source whose order is not // represented on the Russian side (三万 → «три миллиона» is a 100× error, 04-unhappy §9). // // FALSE POSITIVES are the main risk (quotations, stylization, homographs), so each rule is tuned // for PRECISION over recall: it fires only on a high-confidence signal and stays silent on the // ambiguous middle. The unit tests assert both firing AND non-firing. // CheapGateVersion versions the rules above. It is folded into the job snapshot (like // classifierVersion): editing any rule is a loud --resnapshot, so the reported style-flag counts // never shift silently between runs under the same snapshot. [D15.2 note: this is a VERDICT // version — once content-addressed resume lands it moves to verdictSnapshotID and a rule edit // re-classifies free instead of re-paying the book.] // // v2 (D20.4, package №3) closes three adversarial-review false positives, so the bump is LOUD by // design (counts shift): (1) a chevron CITATION at line start no longer counts as dialogue-style // mixing — only the «…», — attribution shape does; (2) a source 万/億 magnitude is no longer // false-flagged against a STRAY output integer (a year, a count) — only a mismatching magnitude // WORD triggers, bare integers can merely confirm; (3) same mechanism covers a 万-in-a-name + // unrelated output number. No live book has run under v1 (D18 acceptance pending), so the re-pin is free. // // v3 (WS5, R4) adds the defect-class checkers DC1 (时辰 double-hour units), DC2 (千万/数十万 magnitude // scale) and DC6 (register negative-list — the zh-ru pack, checkers_zh_ru.go). They are observability // (never a disposition), tuned precision-over-recall; the bump is a loud --resnapshot as the discipline // requires (a rule/pack edit shifts recorded counts). They fire 0 on a non-zh source / non-register text. // Ш-2 (see text/norm.go): the cheap style/DC checkers classify via unicode predicates + NFC folding, so a // toolchain Unicode bump that shifts a class is a loud --resnapshot rather than a silent count change. // // v4 (pack-13) adds three GENERAL observability checkers (never a disposition, tuned precision over // recall): a 成-percent scale checker (六成六=66% mis-rendered as a decimal fraction), a Latin-residue // checker (a whole Latin word left in the Russian output), and a broken-word checker (a Russian word // ending in the impossible «-йть»). All are language-general — no book-specific word lists. Editing a rule // shifts the recorded counts, so the version bump is a loud --resnapshot; they fire 0 on a non-zh source / // clean Russian output (the golden fixture stays style_flags=0). const CheapGateVersion = "cheapgate-v4-percent-latin-brokenword+u" + unicode.Version // CheapGateConfig carries the brief-derived knobs: the ё-policy and the per-project Allowlist of // surfaces that look like a blocklisted interjection but are legitimate here (e.g. a character // named «Ара»). Both come from book.yaml. type CheapGateConfig struct { YoPolicy string // "auto" (inconsistency only) | "all-yo" | "all-e" Allowlist map[string]bool // lower-cased surfaces exempt from the interjection blocklist // RegressionEnabled turns on the post-reflow regression guard (D38, regressionguard.go): an // OPT-IN observability flagger (draft→final length collapse + number drift) folded into this // result. Off → the two regression fields stay 0 and the output is byte-identical to before. RegressionEnabled bool // checkers is the compiled WS5/pack-13 checker spec (pair-14 data-out): the pair's DETECTION patterns + // lookup tables (from the pair langpack) plus the target-general lists (from embedded target data), // resolved once per run. nil for a bare config or a book with no data → the language-specific checkers // run inert (fire 0, the no-pack golden path); the general Latin-residue check needs no spec. Checkers *Checkers } // CheapGateResult is the per-chunk outcome: a count per flagger plus human-readable detail lines // (deterministic order) for the report. Total() is what the passport surfaces. type CheapGateResult struct { DialogueDash int `json:"dialogue_dash,omitempty"` YoInconsistent int `json:"yo,omitempty"` TranslitInterj int `json:"translit_interj,omitempty"` NumberMagnitude int `json:"number_magnitude,omitempty"` // LengthCollapse / NumberDrift are the opt-in post-reflow regression guard (D38, // regressionguard.go), folded into this observability result. They stay 0 unless // cfg.RegressionEnabled, so a book that does not enable the guard serialises identically. LengthCollapse int `json:"length_collapse,omitempty"` NumberDrift int `json:"number_drift,omitempty"` // DC1TimeUnits / DC2Magnitude / DC6Register are the WS5 defect-class checkers (checkers_zh_ru.go), // observability like the others. Zero on a non-zh source / non-register final (the golden fixture). DC1TimeUnits int `json:"dc1_time_units,omitempty"` DC2Magnitude int `json:"dc2_magnitude,omitempty"` DC6Register int `json:"dc6_register,omitempty"` // PercentScale / LatinResidue / BrokenWord are the pack-13 general checkers (checkers_zh_ru.go), // observability like the others. Zero on a clean Russian output / non-zh source. PercentScale int `json:"percent_scale,omitempty"` LatinResidue int `json:"latin_residue,omitempty"` BrokenWord int `json:"broken_word,omitempty"` Detail []string `json:"detail,omitempty"` } func (c CheapGateResult) Total() int { return c.DialogueDash + c.YoInconsistent + c.TranslitInterj + c.NumberMagnitude + c.LengthCollapse + c.NumberDrift + c.DC1TimeUnits + c.DC2Magnitude + c.DC6Register + c.PercentScale + c.LatinResidue + c.BrokenWord } // RunCheapGates runs the four always-on style flaggers over one chunk's source and FINAL text, plus // (opt-in) the draft→final regression guard. `draft` is the first-stage translator output (== final // when there is no distinct reflow stage, so the guard then trivially never fires). func RunCheapGates(source, draft, final string, cfg CheapGateConfig) CheapGateResult { var r CheapGateResult n, det := lintDialogueDash(final) r.DialogueDash, r.Detail = n, append(r.Detail, det...) n, det = cfg.Checkers.lintYofikation(final, cfg.YoPolicy) r.YoInconsistent = n r.Detail = append(r.Detail, det...) n, det = lintTranslitInterjections(final, cfg.Allowlist) r.TranslitInterj = n r.Detail = append(r.Detail, det...) n, det = cfg.Checkers.lintNumberMagnitude(source, final) r.NumberMagnitude = n r.Detail = append(r.Detail, det...) // WS5 defect-class checkers (DC1/DC2/DC6, checkers.go) — src↔target observability flaggers. All patterns // + tables are pair langpack DATA (pair-14 data-out); the spec is nil/inert for a no-pack book → fire 0. n, det = cfg.Checkers.lintTimeUnits(source, final) r.DC1TimeUnits = n r.Detail = append(r.Detail, det...) n, det = cfg.Checkers.lintMagnitudeScale(source, final) r.DC2Magnitude = n r.Detail = append(r.Detail, det...) n, det = cfg.Checkers.lintRegisterLexicon(final) r.DC6Register = n r.Detail = append(r.Detail, det...) // pack-13 general checkers (checkers.go): percent scale (pair data), Latin residue (language-general), // broken word (target data). n, det = cfg.Checkers.lintPercentScale(source, final) r.PercentScale = n r.Detail = append(r.Detail, det...) n, det = lintLatinResidue(final, cfg.Allowlist) r.LatinResidue = n r.Detail = append(r.Detail, det...) n, det = cfg.Checkers.lintBrokenWord(final) r.BrokenWord = n r.Detail = append(r.Detail, det...) if cfg.RegressionEnabled { rg := runRegressionGuard(draft, final) r.LengthCollapse = rg.LengthCollapse r.NumberDrift = rg.NumberDrift r.Detail = append(r.Detail, rg.Detail...) } return r } // --- 1. dialogue-dash linter ---------------------------------------------------- // lintDialogueDash flags direct-speech lines opened with the wrong marker. Russian direct speech // takes an EM-dash «—» at the line start; MTL output leaves a straight ASCII quote or a plain // hyphen/en-dash. It fires per offending line, and additionally when a chunk MIXES em-dash speech // with chevron-quote «…» speech (a within-chapter style clash). Precision guards: a hyphen only // counts as a mis-set dash when followed by a space and a letter (so a hyphenated word wrap or a // «- 1» list item does not fire); chevron lines count only under mixing (a chunk that uses «…» for // speech throughout may be a deliberate style, but mixing it with dashes is an inconsistency). func lintDialogueDash(text string) (int, []string) { var emDash, hyphenLike, chevronSpeech int var detail []string for _, line := range strings.Split(text, "\n") { t := strings.TrimLeft(line, " \t ") rs := []rune(t) if len(rs) == 0 { continue } switch rs[0] { case '—': // em-dash: the correct Russian dialogue marker — count only true speech shape if dialogueShape(rs[1:]) { // «— 15 минут спустя» (dash+space+digit, a scene break) is NOT dialogue (self-review) emDash++ } case '"': // straight ASCII quote leading a line → Russian typography never uses these → MTL artifact if quoteShape(rs[1:]) { detail = append(detail, "direct speech opened with a straight quote \" instead of a dash: "+preview(t)) hyphenLike++ // counted in the offending total } case '-', '–': // hyphen / en-dash where an em-dash belongs if dialogueShape(rs[1:]) { detail = append(detail, "a line using a hyphen/short dash instead of the long «—»: "+preview(t)) hyphenLike++ } case '«': // chevron-led line counts toward mixing ONLY as chevron DIALOGUE (not a citation/title) if chevronSpeechShape(rs) { // D20.4 FP: a «-citation at line start no longer counts as style mixing chevronSpeech++ } } } n := hyphenLike if emDash > 0 && chevronSpeech > 0 { n += chevronSpeech detail = append(detail, fmt.Sprintf("mixed direct-speech styles in the chunk: %d lines with «—» and %d with «…»", emDash, chevronSpeech)) } return n, detail } // chevronSpeechShape reports whether a chevron-led line is a chevron DIALOGUE turn rather than a // citation, title or quoted term (which also open with «). The high-precision signal is the Russian // dialogue-attribution join: a closing » directly followed by a comma, then (spaces) an em/en/hyphen // dash — «Реплика», — сказал он. A quotation/definition uses «X» — … (a dash WITHOUT the comma) or // ends mid-sentence, so it does NOT match, which kills the D20.4 false positive of a «-citation at // line start read as «style mixing». It deliberately MISSES exclamatory «Реплика!» — and unattributed // «Реплика.» chevron dialogue (precision over recall — the gate header's stated bias). func chevronSpeechShape(rs []rune) bool { i := 1 // rs[0] is '«' for i < len(rs) && (rs[i] == ' ' || rs[i] == ' ') { i++ } if i >= len(rs) || !unicode.IsLetter(rs[i]) { return false // «» empty or «123…» — not spoken content } for ; i < len(rs); i++ { if rs[i] != '»' { continue } j := i + 1 if j >= len(rs) || rs[j] != ',' { // the attribution comma must sit right after the closing » continue } j++ for j < len(rs) && (rs[j] == ' ' || rs[j] == ' ') { j++ } if j < len(rs) && (rs[j] == '—' || rs[j] == '–' || rs[j] == '-') { return true } } return false } // dialogueShape reports whether the runes after a leading marker look like spoken text: an optional // space then a letter. It filters out non-dialogue leading dashes (word wraps, «-1», bare marks). func dialogueShape(after []rune) bool { i := 0 for i < len(after) && (after[i] == ' ' || after[i] == ' ') { i++ } if i == 0 { return false // a marker glued to the next glyph (a hyphenated fragment), not «— реплика» } return i < len(after) && unicode.IsLetter(after[i]) } // quoteShape reports whether the runes after a leading quote look like spoken text: an OPTIONAL // space then a letter («"Привет»). Unlike a dash, a quote sits directly on the first word, so no // space is required. func quoteShape(after []rune) bool { i := 0 for i < len(after) && (after[i] == ' ' || after[i] == ' ') { i++ } return i < len(after) && unicode.IsLetter(after[i]) } // --- 2. yofikator --------------------------------------------------------------- // lintYofikation flags inconsistent ё. Default ("auto"): the same word appears BOTH with ё and, as // a separate token, with its exact ё→е form (Пётр/Петр) — excluding the target's homograph whitelist // (е-spellings that are DISTINCT words from their ё-counterpart, все≠всё …; TARGET data, lang.TargetChecks // "yo_homograph" — pair-14 data-out, so the ё↔е FOLD stays as generic orthography and only the wordlist is // data). Policy "all-e": any ё present is a violation. Policy "all-yo": same signal as auto. Full "every // word that SHOULD have ё" enforcement needs a ё-dictionary (deferred, B-tier). Inert if no target data. func (c *Checkers) lintYofikation(out, policy string) (int, []string) { if c == nil { return 0, nil } words := text.TokenizeCyrillic(out) if policy == "all-e" { seen := map[string]bool{} var det []string n := 0 for _, w := range words { if strings.ContainsRune(w, 'ё') && !seen[w] { seen[w] = true n++ det = append(det, "ё under the all-e policy: "+w) } } sort.Strings(det) return n, det } // auto / all-yo: detect a word present in BOTH its ё-form and its е-form. present := map[string]bool{} for _, w := range words { present[w] = true } seenPair := map[string]bool{} var det []string n := 0 for _, w := range words { if !strings.ContainsRune(w, 'ё') { continue } eForm := strings.ReplaceAll(w, "ё", "е") if eForm == w || c.yoHomograph[eForm] { continue // no ё, or a distinct-word homograph (все/всё) — not an inconsistency } if present[eForm] && !seenPair[w] { seenPair[w] = true n++ det = append(det, fmt.Sprintf("inconsistent ё: %q and %q", w, eForm)) } } sort.Strings(det) return n, det } // --- 3. translit-interjection blocklist ----------------------------------------- // translitInterjections is the starter blocklist (04-unhappy §9): JP/EN fillers that must be // translated/adapted, not transliterated. Only surfaces that are NOT ordinary Russian words are // listed (self-review majors): «ара» (a macaw, also a name) and «уму» (dative of «ум») were removed // as high-false-positive collisions; «ара-ара» (the reduplicated JP filler) stays. A residual name // collision is handled by the per-project Allowlist. Lower-cased. // // NOTE (D20.4): only the EXACT hyphenated reduplications enumerated here fire. Other reduplicated // fillers a translit-leak might emit («уху-уху», «эхе-хе», «ня-ня», «фу-фу») are NOT caught — a // generic «X-X»-reduplication rule was rejected as too false-positive-prone (legitimate Russian // reduplications: «еле-еле», «чуть-чуть», «крепко-накрепко»). Extend this list per corpus finding // rather than by heuristic; the residual leak is an accepted recall gap (precision over recall). var translitInterjections = []string{ "ара-ара", "маа", "хмф", "нани", "ауч", "упс", "кья", "десу", } // lintTranslitInterjections counts whole-word (letter-bounded, case-insensitive) occurrences of a // blocklisted interjection in the output, minus any surface on the per-project Allowlist. Whole-word // matching keeps «ара» from firing inside «характер»; the Allowlist exempts legitimate uses. func lintTranslitInterjections(out string, Allowlist map[string]bool) (int, []string) { low := []rune(strings.ToLower(out)) counts := map[string]int{} for _, interj := range translitInterjections { if Allowlist[interj] { continue } f := []rune(interj) for i := 0; i+len(f) <= len(low); i++ { if !text.RunesEqual(low[i:i+len(f)], f) { continue } if (i == 0 || !isWordRune(low[i-1])) && (i+len(f) == len(low) || !isWordRune(low[i+len(f)])) { counts[interj]++ } } } n := 0 surfaces := make([]string, 0, len(counts)) for k, c := range counts { n += c surfaces = append(surfaces, k) } sort.Strings(surfaces) var det []string for _, s := range surfaces { det = append(det, fmt.Sprintf("untranslated interjection %q ×%d", s, counts[s])) } return n, det } // isWordRune reports whether r is part of a word for boundary checks (letter or a hyphen inside a // compound interjection like «ара-ара»). The hyphen inclusion means «ара-ара» matches as one unit // and its inner «ара» does not double-count against a hyphen boundary. func isWordRune(r rune) bool { return unicode.IsLetter(r) || r == '-' } // --- 4. 万/億 magnitude gate ----------------------------------------------------- // lintNumberMagnitude flags a source CJK myriad/hundred-million magnitude whose ORDER of magnitude // is not represented on the Russian side (04-unhappy §9: 三万 → «три миллиона» is a 100× error). It // parses each CJK numeral RUN that carries a big marker (万/萬/億/亿/兆) into an order of magnitude, // then checks the output's magnitude coverage (Russian magnitude words expanded to a [base,base+2] // range for their possible multiplier, plus Arabic-number orders). It fires only when the source's // TOP order is outside every output range — a conservative, multiplier-tolerant signal that leaves // 三億→«триста миллионов» (8 within миллион's [6,8]) silent while catching 三万→«три миллиона». func (c *Checkers) lintNumberMagnitude(source, final string) (int, []string) { if c == nil || len(c.magnitudeStem) == 0 { return 0, nil // no target magnitude-word data → the gate can never confirm coverage; stay inert } srcOrders := cjkMagnitudeOrders(source) if len(srcOrders) == 0 { return 0, nil } maxSrc := 0 for _, o := range srcOrders { if o > maxSrc { maxSrc = o } } if maxSrc < 4 { // only gate on 万+ magnitudes (the 万/億 concern) return 0, nil } // A bare output integer (a year, a count, a page number) may CONFIRM the magnitude but must never // TRIGGER a flag: an unrelated number of a different order is not evidence the 万/億 was // mistranslated (D20.4 FPs: «万 as part of a name + an unrelated number in the output», «a paraphrased // magnitude + a year»). So an Arabic figure of the RIGHT order suppresses; a mismatching one is ignored. for _, o := range arabicNumberOrders(final) { if o == maxSrc { return 0, nil // an Arabic figure of the source order confirms coverage (30000 for 三万) } } wordRanges := c.magnitudeWordRanges(final) for _, rg := range wordRanges { if maxSrc >= rg[0] && maxSrc <= rg[1] { return 0, nil // a magnitude WORD covers the source order (三万 → «тридцать тысяч») } } // Only a mismatching magnitude WORD is evidence of an order-of-magnitude error (三万 → «три миллиона»). Absent // any magnitude word, stay silent: the magnitude was rephrased as prose, or the number in the // output is unrelated (the two D20.4 FPs above). Accepted recall cost: a dropped-magnitude error // rendered as a BARE integer of a smaller order (三万 → «300») is now also silent — indistinguishable // offline from an unrelated stray integer without number alignment (Ф2). Precision over recall. if len(wordRanges) == 0 { return 0, nil } return 1, []string{fmt.Sprintf("the source magnitude 10^%d (万/億) is not reflected in the translation's orders of magnitude — a possible magnitude error (e.g. 三万→«три миллиона»)", maxSrc)} } // isCJKNumeralRune reports whether r can form a CJK numeral expression: an ASCII digit or a shared // lang.CJKSection numeral (Zero∪Digit∪Unit∪Magnitude, pair-14 data-out — the gate no longer keeps its own // rune set). A maximal run of these is one candidate number. func isCJKNumeralRune(r rune) bool { return (r >= '0' && r <= '9') || lang.DefaultCJKSection().IsNumeralRune(r) } // cjkMagnitudeOrders returns the base-10 order of every CJK numeral run in text that contains a big // marker (万/億/兆). Runs without a big marker are ignored (the gate is about myriad-scale orders of magnitude). func cjkMagnitudeOrders(text string) []int { var orders []int rs := []rune(text) for i := 0; i < len(rs); { if !isCJKNumeralRune(rs[i]) { i++ continue } j := i for j < len(rs) && isCJKNumeralRune(rs[j]) { j++ } run := string(rs[i:j]) // A run counts only when it carries a big marker AND has an explicit DIGIT coefficient before // it (self-review major): this excludes the common web-novel IDIOMS that are not magnitudes — // 万一 (in case), 万分 (extremely), 万物 (all things), 千万 (by all means), 亿万 (myriads) — where 万/億 // is not preceded by a digit. It also drops bare-unit magnitudes (十万/百万) — an accepted recall // trade for not false-flagging the far more frequent idioms. if containsBigMarker(run) && hasDigitBeforeBigMarker(run) { if v, ok := parseCJKNumber(run); ok && v > 0 { orders = append(orders, orderOf(v)) } } i = j } return orders } // isBigMarker / containsBigMarker read the shared lang.CJKSection magnitude set (万/萬/億/亿/兆, pair-14 // data-out) instead of a hard-coded rune string. func isBigMarker(r rune) bool { _, ok := lang.DefaultCJKSection().BigUnit(r); return ok } func containsBigMarker(run string) bool { for _, r := range run { if isBigMarker(r) { return true } } return false } // hasDigitBeforeBigMarker reports whether a digit (一-九 / 两 / 0-9) appears before the FIRST big // marker (万/億/兆) in the run — the signature of a real magnitude expression (三万) vs an idiom (万一). func hasDigitBeforeBigMarker(run string) bool { sec := lang.DefaultCJKSection() for _, r := range run { if isBigMarker(r) { return false // hit a big marker with no digit before it → idiom / bare unit } if _, isDigit := sec.Digit[r]; (r >= '0' && r <= '9') || isDigit { return true } } return false } // parseCJKNumber parses a CJK numeral expression (mixed with Arabic digits) into its integer value. // Standard section algorithm: small units (十百千) scale the pending coefficient into the current // <10^4 section; big units (万億兆) flush the section times the big unit into the total. Returns // ok=false on a shape it cannot parse (conservative — an unparseable run does not flag). func parseCJKNumber(s string) (int64, bool) { sec := lang.DefaultCJKSection() var total, section, cur int64 sawBig := false for _, r := range s { switch { case r >= '0' && r <= '9': cur = cur*10 + int64(r-'0') case sec.Zero[r]: cur = cur * 10 default: if d, ok := sec.Digit[r]; ok { cur = cur*10 + int64(d) // positional accumulation (一二→12), matching the Arabic-digit branch (self-review) continue } if u, ok := sec.Unit[r]; ok { if cur == 0 { cur = 1 } section += cur * int64(u) cur = 0 continue } if u, ok := sec.BigUnit(r); ok { sawBig = true section += cur if section == 0 { section = 1 } total += section * u section = 0 cur = 0 continue } return 0, false // an unexpected rune } } if !sawBig { return 0, false // no 万/億/兆 → not a magnitude expression this gate cares about } return total + section + cur, true } func orderOf(v int64) int { o := 0 for v >= 10 { v /= 10 o++ } return o } // magnitudeWordRanges returns the covered [minOrder,maxOrder] ranges implied by Russian magnitude // WORDS in the output. A word covers [base, base+2] because an unseen multiplier can lift it up to // two orders (триста миллионов = 3·10^8, base 6 → order 8). Arabic integers are handled separately // by the caller (they may only CONFIRM coverage, never trigger a mismatch — D20.4), so they are NOT // folded in here. func (c *Checkers) magnitudeWordRanges(text string) [][2]int { low := strings.ToLower(text) var ranges [][2]int for stem, base := range c.magnitudeStem { // target-general data (pair-14 data-out) if strings.Contains(low, stem) { ranges = append(ranges, [2]int{base, base + 2}) } } return ranges } // arabicNumberOrders returns the order of each Arabic integer in text, stitching grouping // separators (space / NBSP / comma) between runs of exactly three digits so "30 000" reads as one // 5-digit number, not "30" and "000". func arabicNumberOrders(text string) []int { rs := []rune(text) var orders []int for i := 0; i < len(rs); { if !isASCIIDigit(rs[i]) { i++ continue } // Leading group. j := i for j < len(rs) && isASCIIDigit(rs[j]) { j++ } digits := j - i // Stitch " ddd" / ",ddd" groups. for j < len(rs) { if (rs[j] == ' ' || rs[j] == ' ' || rs[j] == ',') && j+3 < len(rs)+1 { k := j + 1 g := 0 for k < len(rs) && isASCIIDigit(rs[k]) { k++ g++ } if g == 3 { digits += 3 j = k continue } } break } orders = append(orders, digits-1) i = j } return orders } func isASCIIDigit(r rune) bool { return r >= '0' && r <= '9' } // preview trims a long line for the human detail (deterministic). func preview(s string) string { rs := []rune(s) if len(rs) > 48 { return string(rs[:48]) + "…" } return s }