// Package langscreen answers ONE question about a model's completion: is it written in the language we // asked for? It is the isolated module D39.93 п.1 ratified — pure functions over (text, the target's // declared scripts), returning a NAMED verdict and the score behind it. Nothing here reads a config, a // stage, a book or a language pair, and nothing here decides what to DO about a verdict: the driver maps it // to a disposition, which is what keeps the flag vocabulary in one place. // // WHY IT IS NOT A LANGUAGE DETECTOR, and the distinction is the owner's (D39.92 п.1). A full LID is at best // a spare flagger OUTSIDE the verdict path; what the engine needs is the CLOSED-SET question «did it come // back in a language other than the one we asked for», whose null hypothesis is «this is the target» and // whose attractors are known (English, the source language, the target's siblings). This module answers the // SCRIPT half of that question, which is the half that is decidable without a model. // // WHAT IT DELIBERATELY DOES NOT ANSWER. The owner's four failure modes are echo · a switch to another // language · drift in the middle · meta/degradation, and running them through one detector was named as a // design error. So: // - ECHO of the SOURCE is not here. It is a different measurement over the SOURCE's scripts, it already // ships (pipeline.sourceScriptShare), and it is measured to catch 28 of 30 real events at zero false // positives with a 12.9× margin (pipeline's off-target corpus, backlog row 113). This module exists for // the residue that measurement cannot see BY CONSTRUCTION: an output with no source-script rune in it — // the two fluent ENGLISH drafts of the same corpus. // - DRIFT IN THE MIDDLE is not here. The score is whole-output, so a paragraph that changes language // inside a long chunk stays under the threshold. A segmented worst-of rule was considered and rejected // on this corpus (D39.92 п.1(б)): it has no positive examples to calibrate on and multiplies the false // positive rate by the number of segments. // - SIBLING languages are not here, and cannot be: Bulgarian is written in the Russian alphabet, so no // script rule separates them. That threat has a registry entry and a named discriminator for the day it // is observed (the density of у/э/ё, `13-tech-debt-anchors.md` §Б-108) rather than a mechanism nobody // can calibrate. // // ⚠ AND THE ONE LIMIT THAT WILL BITE A NEW PAIR. The rule is «what share of the letters are in a script the // target declares», so a target whose script set is a SUPERSET of another language's cannot tell them apart // this way: a purely Chinese output for a →ja book scores 1.000, because ja declares han. That is a // property of the question, not a defect of the code, and closing it needs a datum this layer does not have // (which of a language's scripts must be PRESENT, not merely allowed). The limit is about SUPERSETS and // nothing else: a →en book declares Latin and IS screened normally, and a →ja one is screened against // everything that is not han/kana. What →ja cannot be told apart from is Chinese specifically. package langscreen import ( "fmt" "unicode" ) // Verdict is the CLOSED set of answers this module gives. Closed on purpose (D39.92 п.1): the driver maps // each one to a disposition, and a value the driver has never heard of is a silent skip. type Verdict string const ( // OnTarget: enough of the letters are in a script the target declares. OnTarget Verdict = "on_target" // OffTarget: the completion is written in some other script — «вернуло не на том языке, что просили». OffTarget Verdict = "off_target" // Abstain: the module declines to answer. Either the target declares no scripts (the module is inert // by ABSENCE, exactly like the rest of layer 7 — a pair whose data nobody has authored is never judged // by a rule nobody calibrated), or the text is too short to carry evidence. Abstain Verdict = "abstain" ) // Result is one screening: the verdict, the score it was reached on, and the size of the evidence. The // score and the count travel WITH the verdict because a bare verdict is unactionable in a log — «off // target» over 0.49 on 210 letters and over 0.00 on 4700 are different events for an operator. type Result struct { Verdict Verdict // Share is the fraction of LETTERS written in one of the target's scripts, in [0,1]. Zero when the // module abstained. Share float64 // Letters is how many letters the screen actually weighed. Non-letters — digits, punctuation, the // spacing of a formatted reply — are excluded on purpose: they are shared by every language and would // dilute the ratio differently for different chunks. Letters int } // MinLetters is the evidence floor: below it the screen ABSTAINS rather than guessing. Calibrated, not // picked: on the stand's whole corpus (1162 shipping completions) the SHORTEST off-target output is 346 // letters, so a floor at 200 cannot hide a real event, while it does excuse the three healthy completions // that fall under it — a chapter heading, a one-line remainder — which carry too few letters for any ratio // to mean anything. const MinLetters = 200 // OnTargetFloor is the share below which a completion is off-target. It is a CONSTANT placed in an empty // interval, not a tuned parameter, and that distinction is the whole of D39.92 п.1(а): the Neyman-Pearson // guarantee the design was asked for cannot be bought on this corpus (114 independent positions; the rule // of three gives ≤2.594%, twenty-six times weaker than the 0.1% asked), and calibrating a threshold on the // same sample it is measured over doubles the data. So it is versioned as a contract instead. // // The interval it sits in, measured on the corpus (pipeline's off-target fixture, backlog row 113): every // off-target completion scores 0.0000 and the WORST healthy one scores 0.9767, with nothing in between. // 0.50 is that interval's middle, giving a 1.95× margin on the healthy side and no near-miss at all on the // other. const OnTargetFloor = 0.50 // Version identifies the RULE, so a run's verdicts stay attributable to the rule that produced them. It // belongs to the screen rather than to the caller for the reason the sanitizer's does: the caller folds it, // but only this file knows when the answer changed. // // ⚠ IT IS DERIVED FROM THE CONSTANTS, NOT WRITTEN BESIDE THEM, and that is the whole difference between a // mechanism and a discipline. A hand-written literal moves when somebody remembers to move it: planted, the // floor slid 0.50 → 0.45 and the entire battery stayed green, because both constants are pinned to an // INTERVAL (they must sit in the corpus's empty band) and 0.45 is still in it. The snapshot did not move // either, so a RESUMED chunk would have been re-verdicted under a threshold nobody re-billed for — the // exact class classifierVersion exists to make loud. Now editing either number moves this string by // construction, and the loud --resnapshot follows it. // // It is a var rather than a const because a const cannot be formatted; classifierVersion follows suit. var Version = fmt.Sprintf("langscreen-v1-target-script-share+floor%.2f+min%d", OnTargetFloor, MinLetters) // Screen answers whether text is written in the target's script. targetScripts is the target language's // DECLARED script set (lang.LangScripts) — data, so a pair the repository does not carry is one row away // and needs no edit here. // // Deterministic and allocation-free: one pass over the runes, no regexp, no map. func Screen(text string, targetScripts []*unicode.RangeTable) Result { if len(targetScripts) == 0 { // The target declares no script. The honest answer is «I was given nothing to judge by», never // «off target» — a target with no data must be INERT, not condemned (the layer-7 rule, D39.64 П1). return Result{Verdict: Abstain} } letters, onScript := 0, 0 for _, r := range text { if !unicode.IsLetter(r) { continue } letters++ if unicode.In(r, targetScripts...) { onScript++ } } if letters < MinLetters { return Result{Verdict: Abstain, Letters: letters} } share := float64(onScript) / float64(letters) v := OnTarget if share < OnTargetFloor { v = OffTarget } return Result{Verdict: v, Share: share, Letters: letters} }