79 lines
4 KiB
Go
79 lines
4 KiB
Go
package terminology
|
|
|
|
import "strings"
|
|
|
|
// evidence.go: the EVIDENCE view of a candidate — every context the source holds for a surface, as opposed
|
|
// to the subset AttachKWIC hands the model.
|
|
//
|
|
// ⛔ WHY THIS IS NOT KWIC, AND WHY THE TWO MUST NOT SHARE A KNOB. KWIC answers "how much context does the
|
|
// model need to translate this well", and it is a PROMPT-SHAPE question: it is capped (maxPer), it takes
|
|
// the FIRST occurrences in (chapter, chunk, offset) order, and the day a stratified selection lands it
|
|
// will take a different subset of the same text. EvidenceWindows answers a different question — "what does
|
|
// the book say about this surface" — so it is uncapped and selection-free by construction.
|
|
//
|
|
// The distinction is money. A fingerprint built over the KWIC subset would move for every book on earth
|
|
// the moment anyone turned kwic_per_term, changed the selection rule, or widened the model's context, and
|
|
// each of those moves would re-buy a whole bank out of a LIFETIME role budget. Built over every
|
|
// occurrence, it moves only when the SOURCE around the surface moves — which is the event that genuinely
|
|
// invalidates a decision.
|
|
|
|
// EvidenceWindows returns the ±width-rune context around EVERY occurrence of key in nsource, in document
|
|
// order, non-overlapping — the same window shape KWIC shows, without the cap and without the selection.
|
|
//
|
|
// nsource is the book's NORMALIZED source (the same space keys live in), so a caller must not pass raw
|
|
// text: a surface would then be looked up in one space and matched in another.
|
|
//
|
|
// An empty result means the surface does not occur in the source at all, which is honest rather than
|
|
// empty-by-accident: it is what a draft-invented term looks like, and a caller that hashes it gets a
|
|
// value distinct from every attested surface.
|
|
//
|
|
// ⛔ IT DOES NOT REUSE window(), AND THAT IS ABOUT COST RATHER THAN TASTE. window() re-derives the rune
|
|
// view of its input on every call — `[]rune(s)` plus two prefix counts — which is free at the size its own
|
|
// comment declares ("the strings are chunk-sized") and quadratic here, where the input is the WHOLE book
|
|
// and the call happens once per occurrence. Measured on a synthetic source at the occurrence density of the
|
|
// bought cold run: a source four times longer took FIFTEEN times as long, and a million-rune book would
|
|
// have spent minutes — plus tens of gigabytes of garbage — inside a $0 pass that runs before every
|
|
// purchase, whether or not it ends up saving anything. So the source is converted ONCE per call and the
|
|
// windows are cut from that view.
|
|
//
|
|
// ⚠ The window CONTENT is identical to window()'s by construction: same ±width in runes, same clamp at the
|
|
// ends, same TrimSpace. That equality is pinned rather than asserted here, because a fingerprint built on
|
|
// a subtly different window would be a silent, book-wide re-purchase the first time anything compared them.
|
|
func EvidenceWindows(nsource, key string, width int) []string {
|
|
if key == "" || width <= 0 {
|
|
return nil
|
|
}
|
|
runes := []rune(nsource)
|
|
// Byte offset → rune index, built by ONE forward pass. `range` over a string visits rune STARTS, which
|
|
// is exactly the set of offsets this function indexes: strings.Index of a valid UTF-8 key inside valid
|
|
// UTF-8 text returns a rune-aligned offset, and so does that offset plus the key's byte length.
|
|
// Continuation bytes are left at zero on purpose — indexing one would be a bug in the caller, not a
|
|
// case to handle here, and giving them a plausible value would hide it.
|
|
runeAt := make([]int, len(nsource)+1)
|
|
n := 0
|
|
for bi := range nsource {
|
|
runeAt[bi] = n
|
|
n++
|
|
}
|
|
runeAt[len(nsource)] = n
|
|
|
|
var out []string
|
|
for from := 0; from < len(nsource); {
|
|
i := strings.Index(nsource[from:], key)
|
|
if i < 0 {
|
|
break
|
|
}
|
|
at := from + i
|
|
lo := runeAt[at] - width
|
|
if lo < 0 {
|
|
lo = 0
|
|
}
|
|
hi := runeAt[at+len(key)] + width
|
|
if hi > len(runes) {
|
|
hi = len(runes)
|
|
}
|
|
out = append(out, strings.TrimSpace(string(runes[lo:hi])))
|
|
from = at + len(key)
|
|
}
|
|
return out
|
|
}
|