textmachine/backend/internal/membank/mempostcheck.go

package membank

import (
	"strings"
	"unicode"

	"textmachine/backend/internal/lang"
	"textmachine/backend/internal/text"
)

// mempostcheck.go: the post-check (registry E1/E2 / research/13 Q5) — the main, almost
// free detector of silent degradation. For every injected record whose SRC actually
// fired in THIS chunk, it asks one question: did an accepted DST form appear in the
// output? A miss says the injected rendering is ABSENT and nothing more — "the model
// ignored the glossary" (soft-following loses 17–36% of terms, 2310.05824), whether the
// injected dst was right or wrong.
//
// BLIND SPOT, by the shape of the check: a wrong dst the model OBEYED (the owner's fear,
// 2510.00829) leaves the injected form PRESENT, so `if present { continue }` below is taken
// and no miss is written — the more faithfully a model executes a wrong injection, the
// quieter this gets. A zero confirmed-miss count therefore says the signed rows that fired
// on a TRUSTED key were EXECUTED — not that they were executed CORRECTLY, and nothing about
// the rest: an unsigned row's miss and a signed one demoted by a single-Han key both go to
// Unverified and are not counted. Judging the VALUE of an injected rendering needs a
// source-side check that does not live here (backlog row 286). Pinned by
// TestWrongDstObeyedIsNotAMiss.
//
// It is a post-CHECK, never a blind post-REPLACE (E2):
// forcing a dictionary form into an oblique Russian case breaks agreement (46% of
// constrained-model errors in en-cs are agreement, 2106.12398), so v1 only FLAGS.
//
// Decl-awareness is load-bearing (research/14 §2, quantified): a naive \b-regexp on the
// base form gives 18–36% FALSE flags on a hard chunk — inflected forms rendered
// CORRECTLY ("Вэйчжуане", "Бородатого Вана", "Дэна", "сюцая") that the boundary regexp
// misses. So the check matches against the STORED decl forms (filled at term-commit),
// not the bare lemma. BUT stored-decl reliability is itself a function of decl
// COMPLETENESS — an OOV translit name whose forms were under-filled re-introduces the
// same false flags. That is why v1 keeps the post-check a FLAGGER (observable, in the
// retrieval-state), NOT a hard disposition gate, until the false-flag rate is MEASURED
// on real inflected Russian (E1 — memory_e1_test.go). A hard gate flips on only after
// the owner validates precision (mirrors the coverage gate's opt-in, D12 Q4).
//
// Sticky records are DELIBERATELY excluded: a sticky entry's src is NOT in this chunk
// (that is why scene-inertia carries it), so its dst is legitimately absent from the
// output — post-checking it would false-flag every pronominal chunk. Only records whose
// key exactly fired here are expected in the output.

// PostcheckMiss is one flagged term: an injected record whose src fired but whose dst
// (any accepted form) is absent from the output.
type PostcheckMiss struct {
	Src  string `json:"src"`
	Dst  string `json:"dst"`
	Disp string `json:"disp"` // the INJECTION trust: confirmed | ambiguous (A2: ambiguous injections force a post-check)
	// Demoted marks a CONFIRMED injection whose miss was routed to Unverified observability (not counted)
	// because its firing key is a single Han rune (#10) — weak substring evidence, not a real consistency
	// failure. It disambiguates the otherwise-contradictory disp:"confirmed" that would sit in the Unverified
	// list with ConfirmedCount()==0: the demotion is a KEY-evidence axis, orthogonal to the injection trust
	// Disp records, so it is a separate flag, not an overload of Disp. omitempty — absent on every non-demoted
	// miss, so the wire and the golden are byte-identical unless a demotion actually occurs.
	Demoted bool `json:"demoted,omitempty"`
}

// PostcheckResult separates the two kinds of miss BY TYPE rather than by the discipline of whoever calls
// the checker (pack-20 / D39.42 п.4). The rule itself is unchanged and load-bearing: only a CONFIRMED
// miss is a consistency failure. An AMBIGUOUS miss is an auto/draft candidate the model was ENTITLED to
// reject, and counting it would invert the contract — punishing a model that correctly refused an
// unverified suggestion while passing one that obeyed a wrong CONFIRMED injection (external-review major
// #1). What changes is that the rule is now impossible to get wrong at a call site: there is no combined
// slice to filter, so a caller cannot forget to. The Unverified misses are still carried, in full, as
// observability (A2's forced post-check) — they are simply not a count.
//
// Unverified now holds TWO populations, and "entitled to reject" applies to only one: (a) genuinely
// ambiguous/unsigned injections the model WAS entitled to refuse; and (b) CONFIRMED injections DEMOTED here
// by a weak single-Han firing key (#10, PostcheckMiss.Demoted). For (b) "entitled to reject" is false — the
// injection was signed; it is demoted because the KEY evidence (a one-rune substring) is weak, not because
// the model could refuse it. Both are observability and neither counts, but the Demoted flag keeps them
// distinguishable in the detail so (b) is not misread as a licensed refusal.
type PostcheckResult struct {
	Confirmed  []PostcheckMiss
	Unverified []PostcheckMiss
	// Shown / Followed are the $0 observability channel of the unsigned wire: how many unsigned rows this
	// chunk could be JUDGED on, and how many of those the model went along with. Without the denominator
	// "3 deviations" is unreadable — 3 out of 3 is a channel nobody follows, 3 out of 90 is ordinary
	// variance.
	//
	// ⚠ "JUDGED ON" IS NARROWER THAN "PUT IN FRONT OF THE MODEL", and the difference is the STICKY skip
	// above. A sticky carry is rendered into the injection block like any other record — the model does see
	// it — but its src is not in this chunk, so its dst is legitimately absent from the output; counting it
	// would make every pronominal chunk read as a channel nobody follows. What is left is exactly the
	// population Followed is a fraction of, and it is smaller than the set of rows shown.
	//
	// The other two skips do not narrow it on any real selection: MaterializeBank gives a row with no dst
	// no source surfaces at all, so it can neither fire nor be carried, and a row-less record only exists
	// when a caller assembles PickedEntry by hand. Both are defensive, and naming them as reasons the count
	// is small would misdescribe it as badly as the old wording did.
	//
	// ⚠ WHAT THEY MEASURE NARROWED WITH ROW 134, and the numbers did not change shape. They were born on a
	// wire that marked an unsigned row ⟨проверить⟩ and told the model it was free to reject it, so a low
	// ratio read as «the model is exercising judgement it was granted». D39.104 п.2 took the mark off: an
	// unsigned row is now shown as law, nothing invites a refusal, and a deviation is just a deviation.
	// The pair is still the only channel that says whether an unsigned rendering survives into the text.
	Shown    int
	Followed int
}

// ConfirmedCount is the actionable consistency-failure count — the ONLY number allowed to gate.
func (r PostcheckResult) ConfirmedCount() int { return len(r.Confirmed) }

// All returns every miss, confirmed first, for the human-facing detail blob.
func (r PostcheckResult) All() []PostcheckMiss {
	if len(r.Unverified) == 0 {
		return r.Confirmed
	}
	out := make([]PostcheckMiss, 0, len(r.Confirmed)+len(r.Unverified))
	return append(append(out, r.Confirmed...), r.Unverified...)
}

// Empty reports whether nothing at all was missed.
func (r PostcheckResult) Empty() bool { return len(r.Confirmed) == 0 && len(r.Unverified) == 0 }

// Postcheck runs the decl-aware check over the injected records against the model
// output. Pure and deterministic. Returns the misses split by trust (empty = all fired terms present),
// plus the shown/followed counters of the unverified channel.
func (b *Bank) Postcheck(injected []PickedEntry, output string) PostcheckResult {
	nout := text.NormalizeTargetForm(output)
	var res PostcheckResult
	noutRunes := []rune(nout)
	outWords := lang.TokenizeWords(nout) // once for the whole chunk; the §3 stem branch reuses it per term
	for _, p := range injected {
		if !p.valid() { // a caller-built zero record carries no row — nothing to check (pack-16 tail)
			continue
		}
		if p.Sticky { // sticky context is not expected in the output
			continue
		}
		if strings.TrimSpace(p.entry.dst) == "" { // a ruby candidate with no dst yet — nothing to check
			continue
		}
		present := dstFormPresent(p.entry, noutRunes, outWords, b.stemmer)
		if p.Disp != Confirmed {
			res.Shown++
			if present {
				res.Followed++
			}
		}
		if present {
			continue
		}
		m := PostcheckMiss{Src: p.entry.src, Dst: p.entry.dst, Disp: string(p.Disp)}
		switch {
		case p.Disp == Confirmed && singleHanKeyFired(p.Via):
			m.Demoted = true // a confirmed miss on a single-Han substring key → Unverified, not a counted failure (#10)
			res.Unverified = append(res.Unverified, m)
		case p.Disp == Confirmed:
			res.Confirmed = append(res.Confirmed, m)
		default:
			res.Unverified = append(res.Unverified, m)
		}
	}
	return res
}

// singleHanKeyFired reports whether the firing key is a SINGLE Han ideograph (#10, D39.39). Han has no word
// segmentation, so a one-rune key matches by substring (转 inside 转身) — weak evidence the entity occurred —
// so a dst-absent miss on it is demoted to Unverified observability, not a hard CONFIRMED miss (precision over
// recall). Multi-rune Han is NOT demoted — not because it gets a word boundary (it does not: Han is never
// boundary-checked here, memory.go:536) but because a ≥2-rune substring collides far less, so the evidence is
// statistically stronger. Han-only until the B6 tokenizer (B6 = backlog row 81).
func singleHanKeyFired(via string) bool {
	r := []rune(via)
	return len(r) == 1 && unicode.Is(unicode.Han, r[0])
}

// dstFormPresent reports whether any accepted dst form of the entry appears in the
// normalized output as a WHOLE WORD (bounded by non-letters or string edges), not a bare
// substring. The whole-word rule closes self-review #3: a DROPPED short translit name
// («Ван») must not be masked by an unrelated common word that merely contains its letters
// («караВАН», «диВАН», «ИВАН» — a different person). Decl-aware (the research/14
// requirement): the stored decl forms carry the inflections a boundary regexp would miss.
//
// The accepted set is the UNION of the base dst AND the stored decl forms (D24.4 — D24.3 is
// the max_tokens floor, a different fix; the union is the post-check verdict change):
// the base translation is itself an approved form, so it is ALWAYS checked — not only as a
// fallback when decl is empty. Checking it only on empty decl false-flagged 37/55
// acceptance-stage-A misses (nominative_gap): an approved dst present in the NOMINATIVE
// while `decl.forms` listed only oblique cases (方源→Фан Юань present in the output, decl
// carried only genitive/dative). Requiring the base form to be duplicated into decl is
// redundant input every future book would trip on.
//
// PRECISION/RECALL TRADEOFF, not a free win (adversarial review, pkg 5): enlarging the
// accepted set is monotonic — it can only turn misses into passes, never the reverse. On
// the measured stage-A data those removed misses are all FALSE positives on correctly
// rendered nominatives (precision up, the 37/55). The one way it can COST recall is narrow
// but real: an entry whose decl OMITS the nominative AND whose base dst is a common word
// that recurs elsewhere while the term itself was DROPPED for its firing occurrence
// (剑→меч dropped as «клинок», yet an unrelated «меч» sits elsewhere) — this bag-of-words
// presence check then passes and masks the drop. That class is rare on real data (translit
// names do not recur coincidentally) and TOLERABLE only while this stays a FLAGGER: a
// hard-gate promotion (postcheck_gate) MUST re-measure recall on common-noun terms, not
// assume it — the "recall unaffected" phrasing in the D24.4 rationale is imprecise here.
// The remaining inflection_gap (18/55: a plural rendered but only the singular seeded) is a
// SEED completeness fix (Polygon), not a code one. NB target ALIASES: the accepted set is base
// dst ∪ decl forms — an alternative TARGET rendering (a synonym/nickname the model may legitimately
// use) is NOT matched here, and adding one is a SEED concern (seed it as a decl form / alias), not
// code, on the same footing as the inflection_gap. noutRunes is the normalized output pre-decomposed.
func dstFormPresent(e *entry, noutRunes []rune, outWords []string, stemmer lang.TargetStemmer) bool {
	if base := text.NormalizeTargetForm(e.dst); base != "" {
		if containsWholeWord(noutRunes, []rune(base)) {
			return true
		}
		// §3: accept an OBLIQUE case of the base when the seed listed no decl forms (the measured 142/142-null
		// noise). A component-wise stem match — «горы Цинмао» for «гора Цинмао», «мечом» for «меч» — with no
		// assumption about which word inflects. Inert for a target with no decl_suffix registry.
		if declinedFormPresent(outWords, base, stemmer) {
			return true
		}
	}
	for _, f := range e.declForms {
		if f != "" && containsWholeWord(noutRunes, []rune(f)) {
			return true
		}
	}
	return false
}

// declinedFormPresent reports whether the base rendering appears in the tokenized output with each of its
// words possibly inflected — a window of outWords the same length as base's words where every component
// shares its stem. Conservative (needs the whole phrase present, in order) and inert without a stemmer.
func declinedFormPresent(outWords []string, base string, stemmer lang.TargetStemmer) bool {
	if !stemmer.Enabled() {
		return false
	}
	bw := lang.TokenizeWords(base)
	n := len(bw)
	if n == 0 || n > len(outWords) {
		return false
	}
	for i := 0; i+n <= len(outWords); i++ {
		all := true
		for k := 0; k < n; k++ {
			if !stemmer.SameStem(outWords[i+k], bw[k]) {
				all = false
				break
			}
		}
		if all {
			return true
		}
	}
	return false
}

// containsWholeWord reports whether form occurs in hay with a letter-boundary on both
// ends (the char before the match is not a letter or is the start; likewise after). Runs
// on runes so Cyrillic boundaries are correct. O(len(hay)·len(form)) — fine, the
// post-check runs once per chunk over a handful of short forms.
func containsWholeWord(hay, form []rune) bool {
	n := len(form)
	if n == 0 || n > len(hay) {
		return false
	}
	for i := 0; i+n <= len(hay); i++ {
		if !text.RunesEqual(hay[i:i+n], form) {
			continue
		}
		if (i == 0 || !unicode.IsLetter(hay[i-1])) && (i+n == len(hay) || !unicode.IsLetter(hay[i+n])) {
			return true
		}
	}
	return false
}

// SpoilerLeak is one rendering the spoiler window REJECTED for this chapter that appeared in the output
// anyway — the self-inflicted spoiler the windows exist to prevent (D21 п.3 / research/15 §2.2).
type SpoilerLeak struct {
	Src string `json:"src"`
	Dst string `json:"dst"`
	// Since/Until are the window that excluded the row, so the operator sees at a glance whether the
	// leak is a premature reveal (before since_ch) or a stale one (after until_ch).
	SinceCh int `json:"since_ch"`
	UntilCh int `json:"until_ch"`
}

// SpoilerLeaks checks the chunk's REJECTED records against the model output: the row's key fired here,
// the window said the chapter must not know this rendering yet, and the rendering is in the output all
// the same. It is the reveal half of D21 п.3, built on the machinery that already exists — the selection
// records its rejects, and dstFormPresent already answers "is this rendering here" decl-aware — so it
// needs no schema and no second matcher.
//
// SCOPE, ratified (D39.55): "a leak on a FIRED key". Two classes are therefore OUT, and out on purpose
// rather than by oversight:
//
//   - a STICKY carry rejected by the window (Sticky==true). Its key did NOT fire in this chunk, so the
//     model was not looking at the entity here; counting it would silently widen the ratified scope on
//     the back of the pack-19 fix that put those carries into Rejected at all;
//   - a SOURCE-ANCHORED reveal — an identity twist whose source surface never occurs (the text says
//     "the stranger", not the name). Nothing fires, so nothing is rejected, so there is nothing to
//     check. Closing that class needs a target-side index of post-reveal renderings, which is a
//     different mechanism and not this pack's.
//
// Pure and deterministic (rejected order, which Select fixes). Observability only — never a disposition.
func (b *Bank) SpoilerLeaks(rejected []PickedEntry, output string) []SpoilerLeak {
	norm := text.NormalizeTargetForm(output)
	nout := []rune(norm)
	outWords := lang.TokenizeWords(norm)
	var out []SpoilerLeak
	for _, p := range rejected {
		if !p.valid() || p.Sticky {
			continue
		}
		if strings.TrimSpace(p.entry.dst) == "" {
			continue // nothing to leak
		}
		if !dstFormPresent(p.entry, nout, outWords, b.stemmer) {
			continue
		}
		out = append(out, SpoilerLeak{
			Src: p.entry.src, Dst: p.entry.dst,
			SinceCh: p.entry.sinceCh, UntilCh: p.entry.untilCh,
		})
	}
	return out
}