137 lines
8.3 KiB
Go
137 lines
8.3 KiB
Go
// Package langscreen answers ONE question about a model's completion: is it written in the language we
|
||
// asked for? It is the isolated module D39.93 п.1 ratified — pure functions over (text, the target's
|
||
// declared scripts), returning a NAMED verdict and the score behind it. Nothing here reads a config, a
|
||
// stage, a book or a language pair, and nothing here decides what to DO about a verdict: the driver maps it
|
||
// to a disposition, which is what keeps the flag vocabulary in one place.
|
||
//
|
||
// WHY IT IS NOT A LANGUAGE DETECTOR, and the distinction is the owner's (D39.92 п.1). A full LID is at best
|
||
// a spare flagger OUTSIDE the verdict path; what the engine needs is the CLOSED-SET question «did it come
|
||
// back in a language other than the one we asked for», whose null hypothesis is «this is the target» and
|
||
// whose attractors are known (English, the source language, the target's siblings). This module answers the
|
||
// SCRIPT half of that question, which is the half that is decidable without a model.
|
||
//
|
||
// WHAT IT DELIBERATELY DOES NOT ANSWER. The owner's four failure modes are echo · a switch to another
|
||
// language · drift in the middle · meta/degradation, and running them through one detector was named as a
|
||
// design error. So:
|
||
// - ECHO of the SOURCE is not here. It is a different measurement over the SOURCE's scripts, it already
|
||
// ships (pipeline.sourceScriptShare), and it is measured to catch 28 of 30 real events at zero false
|
||
// positives with a 12.9× margin (pipeline's off-target corpus, backlog row 113). This module exists for
|
||
// the residue that measurement cannot see BY CONSTRUCTION: an output with no source-script rune in it —
|
||
// the two fluent ENGLISH drafts of the same corpus.
|
||
// - DRIFT IN THE MIDDLE is not here. The score is whole-output, so a paragraph that changes language
|
||
// inside a long chunk stays under the threshold. A segmented worst-of rule was considered and rejected
|
||
// on this corpus (D39.92 п.1(б)): it has no positive examples to calibrate on and multiplies the false
|
||
// positive rate by the number of segments.
|
||
// - SIBLING languages are not here, and cannot be: Bulgarian is written in the Russian alphabet, so no
|
||
// script rule separates them. That threat has a registry entry and a named discriminator for the day it
|
||
// is observed (the density of у/э/ё, `13-tech-debt-anchors.md` §Б-108) rather than a mechanism nobody
|
||
// can calibrate.
|
||
//
|
||
// ⚠ AND THE ONE LIMIT THAT WILL BITE A NEW PAIR. The rule is «what share of the letters are in a script the
|
||
// target declares», so a target whose script set is a SUPERSET of another language's cannot tell them apart
|
||
// this way: a purely Chinese output for a →ja book scores 1.000, because ja declares han. That is a
|
||
// property of the question, not a defect of the code, and closing it needs a datum this layer does not have
|
||
// (which of a language's scripts must be PRESENT, not merely allowed). The limit is about SUPERSETS and
|
||
// nothing else: a →en book declares Latin and IS screened normally, and a →ja one is screened against
|
||
// everything that is not han/kana. What →ja cannot be told apart from is Chinese specifically.
|
||
package langscreen
|
||
|
||
import (
|
||
"fmt"
|
||
"unicode"
|
||
)
|
||
|
||
// Verdict is the CLOSED set of answers this module gives. Closed on purpose (D39.92 п.1): the driver maps
|
||
// each one to a disposition, and a value the driver has never heard of is a silent skip.
|
||
type Verdict string
|
||
|
||
const (
|
||
// OnTarget: enough of the letters are in a script the target declares.
|
||
OnTarget Verdict = "on_target"
|
||
// OffTarget: the completion is written in some other script — «вернуло не на том языке, что просили».
|
||
OffTarget Verdict = "off_target"
|
||
// Abstain: the module declines to answer. Either the target declares no scripts (the module is inert
|
||
// by ABSENCE, exactly like the rest of layer 7 — a pair whose data nobody has authored is never judged
|
||
// by a rule nobody calibrated), or the text is too short to carry evidence.
|
||
Abstain Verdict = "abstain"
|
||
)
|
||
|
||
// Result is one screening: the verdict, the score it was reached on, and the size of the evidence. The
|
||
// score and the count travel WITH the verdict because a bare verdict is unactionable in a log — «off
|
||
// target» over 0.49 on 210 letters and over 0.00 on 4700 are different events for an operator.
|
||
type Result struct {
|
||
Verdict Verdict
|
||
// Share is the fraction of LETTERS written in one of the target's scripts, in [0,1]. Zero when the
|
||
// module abstained.
|
||
Share float64
|
||
// Letters is how many letters the screen actually weighed. Non-letters — digits, punctuation, the
|
||
// spacing of a formatted reply — are excluded on purpose: they are shared by every language and would
|
||
// dilute the ratio differently for different chunks.
|
||
Letters int
|
||
}
|
||
|
||
// MinLetters is the evidence floor: below it the screen ABSTAINS rather than guessing. Calibrated, not
|
||
// picked: on the stand's whole corpus (1162 shipping completions) the SHORTEST off-target output is 346
|
||
// letters, so a floor at 200 cannot hide a real event, while it does excuse the three healthy completions
|
||
// that fall under it — a chapter heading, a one-line remainder — which carry too few letters for any ratio
|
||
// to mean anything.
|
||
const MinLetters = 200
|
||
|
||
// OnTargetFloor is the share below which a completion is off-target. It is a CONSTANT placed in an empty
|
||
// interval, not a tuned parameter, and that distinction is the whole of D39.92 п.1(а): the Neyman-Pearson
|
||
// guarantee the design was asked for cannot be bought on this corpus (114 independent positions; the rule
|
||
// of three gives ≤2.594%, twenty-six times weaker than the 0.1% asked), and calibrating a threshold on the
|
||
// same sample it is measured over doubles the data. So it is versioned as a contract instead.
|
||
//
|
||
// The interval it sits in, measured on the corpus (pipeline's off-target fixture, backlog row 113): every
|
||
// off-target completion scores 0.0000 and the WORST healthy one scores 0.9767, with nothing in between.
|
||
// 0.50 is that interval's middle, giving a 1.95× margin on the healthy side and no near-miss at all on the
|
||
// other.
|
||
const OnTargetFloor = 0.50
|
||
|
||
// Version identifies the RULE, so a run's verdicts stay attributable to the rule that produced them. It
|
||
// belongs to the screen rather than to the caller for the reason the sanitizer's does: the caller folds it,
|
||
// but only this file knows when the answer changed.
|
||
//
|
||
// ⚠ IT IS DERIVED FROM THE CONSTANTS, NOT WRITTEN BESIDE THEM, and that is the whole difference between a
|
||
// mechanism and a discipline. A hand-written literal moves when somebody remembers to move it: planted, the
|
||
// floor slid 0.50 → 0.45 and the entire battery stayed green, because both constants are pinned to an
|
||
// INTERVAL (they must sit in the corpus's empty band) and 0.45 is still in it. The snapshot did not move
|
||
// either, so a RESUMED chunk would have been re-verdicted under a threshold nobody re-billed for — the
|
||
// exact class classifierVersion exists to make loud. Now editing either number moves this string by
|
||
// construction, and the loud --resnapshot follows it.
|
||
//
|
||
// It is a var rather than a const because a const cannot be formatted; classifierVersion follows suit.
|
||
var Version = fmt.Sprintf("langscreen-v1-target-script-share+floor%.2f+min%d", OnTargetFloor, MinLetters)
|
||
|
||
// Screen answers whether text is written in the target's script. targetScripts is the target language's
|
||
// DECLARED script set (lang.LangScripts) — data, so a pair the repository does not carry is one row away
|
||
// and needs no edit here.
|
||
//
|
||
// Deterministic and allocation-free: one pass over the runes, no regexp, no map.
|
||
func Screen(text string, targetScripts []*unicode.RangeTable) Result {
|
||
if len(targetScripts) == 0 {
|
||
// The target declares no script. The honest answer is «I was given nothing to judge by», never
|
||
// «off target» — a target with no data must be INERT, not condemned (the layer-7 rule, D39.64 П1).
|
||
return Result{Verdict: Abstain}
|
||
}
|
||
letters, onScript := 0, 0
|
||
for _, r := range text {
|
||
if !unicode.IsLetter(r) {
|
||
continue
|
||
}
|
||
letters++
|
||
if unicode.In(r, targetScripts...) {
|
||
onScript++
|
||
}
|
||
}
|
||
if letters < MinLetters {
|
||
return Result{Verdict: Abstain, Letters: letters}
|
||
}
|
||
share := float64(onScript) / float64(letters)
|
||
v := OnTarget
|
||
if share < OnTargetFloor {
|
||
v = OffTarget
|
||
}
|
||
return Result{Verdict: v, Share: share, Letters: letters}
|
||
}
|