heaven/textmachine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Land R1 wave driver-switch (D39.17): parallel draft/edit waves with bank-mining stop, editor never re-renders draft, Sh-1/Sh-2, per-unit read-models; rubezh-2 clean bill, three MINOR mining-wiring fixes to pre-rerun; pack-11 complete

Browse source
This commit is contained in:
Claude (backend session) 2026-07-19 20:11:05 +03:00
parent a263365be3
commit 89d5bf9d6a
36 changed files with 1825 additions and 564 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

32
backend/internal/config/book.go
View file

@ -45,21 +45,32 @@ type Book struct {
StyleAllowlist []string `yaml:"style_allowlist"`
// Wiring: paths are resolved relative to the book.yaml location.
Pipeline string `yaml:"pipeline"`
ModelsFile string `yaml:"models"`
SourceFile string `yaml:"source_file"` // Фаза 0: один файл = один чанк
Pipeline string `yaml:"pipeline"`
ModelsFile string `yaml:"models"`
SourceFile string `yaml:"source_file"` // Фаза 0: один файл = один чанк
// Encoding of the txt source: auto|utf8|gb18030 ("" defaults to auto). Real zh .txt are often
// GB18030 (the 蛊真人 acceptance book is), which the auto-detect handles; declare it explicitly
// to skip detection. Ignored for epub (its documents carry their own charset). See ingest.go.
Encoding string `yaml:"encoding"`
ProjectDB string `yaml:"project_db"` // default: <book_id>.db рядом с book.yaml
Encoding string `yaml:"encoding"`
ProjectDB string `yaml:"project_db"` // default: <book_id>.db рядом с book.yaml
// GlossarySeed is the optional path to the manual glossary seed YAML (memory v2,
// шаг 4). Its curated approved/draft terms + the classified ruby readings are the
// deterministic inputs the glossary is REPLACED from each run; editing it changes the
// materialized approved set → a loud --resnapshot (F1). Empty = ruby-seed only (or an
// empty glossary → the injection is inert, a safe no-op).
GlossarySeed string `yaml:"glossary_seed"`
Ceilings BookCap `yaml:"ceilings"`
GlossarySeed string `yaml:"glossary_seed"`
// LangpackRoot is the optional root of the language-data packs (configs/langpacks/, D39.15/16): the
// miner reads configs/langpacks/<src>/ (source morphology) + configs/langpacks/<src>-<tgt>/ (Palladius)
// from here. Resolved relative to book.yaml. Empty ⇒ no pack (the miner is inert / W1.5 auto-continues);
// set ⇒ the runner loads the pack in W0, failing LOUD when the pair's catalog dir exists but a file is
// missing/corrupt, and folds pack.Version() into the snapshot so a pack edit is a loud --resnapshot (R1).
LangpackRoot string `yaml:"langpack_root"`
// MinedDelta is the optional path to the owner-curated mined-term delta YAML (the W1.5 sign artifact,
// R1). Its terms are loaded as Source:"mined" (NOT Source:"seed"), so they fold into the ENRICHED bank
// version but NOT the base — adding them moves only snapshot_W2 («переоплата ОДНА»). Same seedTerm
// schema as glossary_seed; resolved relative to book.yaml. Empty = no mined terms.
MinedDelta string `yaml:"mined_delta"`
Ceilings BookCap `yaml:"ceilings"`
}
// BookCap are the ledger admission limits (Р7: потолок $ на книгу/день).
@ -90,6 +101,8 @@ func LoadBook(path string) (*Book, error) {
b.ModelsFile = resolve(b.ModelsFile)
b.SourceFile = resolve(b.SourceFile)
b.GlossarySeed = resolve(b.GlossarySeed)
b.LangpackRoot = resolve(b.LangpackRoot)
b.MinedDelta = resolve(b.MinedDelta)
if b.ProjectDB == "" {
b.ProjectDB = filepath.Join(dir, b.BookID+".db")
} else {
@ -123,6 +136,11 @@ func LoadBook(path string) (*Book, error) {
bad("glossary_seed %s is not readable: %v", b.GlossarySeed, err)
}
}
if b.MinedDelta != "" {
if _, err := os.Stat(b.MinedDelta); err != nil {
bad("mined_delta %s is not readable: %v", b.MinedDelta, err)
}
}
if b.Encoding == "" {
b.Encoding = "auto"
}

33
backend/internal/config/pipeline.go
View file

@ -29,6 +29,32 @@ type Pipeline struct {
Gates Gates `yaml:"gates"`
Escal Escalation `yaml:"escalation"`
Fanout Fanout `yaml:"fanout"`
Waves Waves `yaml:"waves"`
Mining Mining `yaml:"mining"`
}
// Waves configures the wave executor's parallelism (WS1 §1б / R1): how many worker goroutines fan out
// over the draft chunks (W1) and edit units (W2). It is a TRANSPORT axis — it changes NOTHING on the wire
// (each unit of work renders identical bytes regardless of which goroutine runs it), so it is deliberately
// NOT folded into the snapshot (folding it would make a worker-count edit a spurious --resnapshot / whole-
// book re-bill). The inner per-model cap stays RateLimit.MaxConcurrency (models.yaml). Workers ≤ 0 defaults
// to 1: a wave-STRUCTURED but sequential run (W1 all drafts → W1.5 → W2 all units), deterministic and safe;
// prod sets it higher (the COGS sim assumed 8). A run with workers=1 is byte-identical in results to
// workers=N (only request_log/wire ORDER differs — the golden capture sorts to absorb that).
type Waves struct {
Workers int `yaml:"workers"`
}
// Mining configures the W1.5 bank-mining stop (WS3 / R1): the general-zh contrast corpus the WHICH-detector
// scores candidates against. It is OFF unless ContrastPath is set AND a language pack is loaded (book
// langpack_root): the miner then runs at the W1.5 boundary over the W1 drafts, emits the seed-delta +
// signature map, and STOPS for owner sign (or auto-continues on an empty delta). ContrastPath is the jieba-
// style word-freq artifact (large, deployment-specific, NOT in git), resolved relative to pipeline.yaml. It
// is NOT snapshot-folded: mining produces Source:mined PROPOSALS (status:auto, inert until owner-approved),
// so it touches no existing checkpoint's wire or verdict — only the langpack VERSION (which shapes the
// proposals) is folded, via the runner's pack.Version() (§8). Empty ContrastPath ⇒ W1.5 auto-continues.
type Mining struct {
ContrastPath string `yaml:"contrast_path"`
}
// Segmentation is the WS2 OUTPUT-token chunking budget (слой 1, L2-budget-wrong-unit fix): the
@ -309,6 +335,8 @@ func LoadPipeline(path string, models *Models) (*Pipeline, error) {
p.Stages[i].Prompts[pair] = resolvePrompt(pr)
}
}
// The mining contrast artifact path is resolved like the prompts (relative to pipeline.yaml).
p.Mining.ContrastPath = resolvePrompt(p.Mining.ContrastPath)
var problems []string
bad := func(format string, a ...any) { problems = append(problems, fmt.Sprintf(format, a...)) }
@ -328,6 +356,11 @@ func LoadPipeline(path string, models *Models) (*Pipeline, error) {
if p.Fanout.Candidates <= 0 {
p.Fanout.Candidates = 1
}
// Wave workers default to 1 (a wave-structured but sequential, deterministic run). A negative value
// is a config typo, not a request — clamp loudly-neutral to 1 rather than fail (transport axis).
if p.Waves.Workers <= 0 {
p.Waves.Workers = 1
}
// WS2 segmentation defaults (ratified zh-ru, §2в): a config that omits the block gets the
// output-token budget + independently-re-derived fertility. Zero/negative → default.
if p.Segmentation.DraftBudgetOut <= 0 {

2
backend/internal/pipeline/banknote.go
View file

@ -51,7 +51,7 @@ var bankTypeOK = map[string]bool{"name": true, "place": true, "title": true, "te
// with Python's unicode \s around the pipe is exact on the term corpus.)
var bankFieldSplit = regexp.MustCompile(`\t| {2,}|\s*\|\s*`)
// bankEntry is one parsed candidate line (evidence for W1.5 §C; NOT written to the store until owner
// bankEntry is one parsed candidate line (evidence for the bank-mining stop (§C); NOT written to the store until owner
// signoff). Dst is the model's proposed translation — the direct dst delivery the co-occurrence
// miner could not extract (蛊→гу, non-seed 龙公→Лун Гун).
type bankEntry struct {

63
backend/internal/pipeline/bookrun.go
View file

@ -100,12 +100,12 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
return nil, err
}
// W0: eager-build every reachable client BEFORE any wave goroutine, so the clients map is
// the precompute pass: eager-build every reachable client BEFORE any wave goroutine, so the clients map is
// read-only in the waves (r.client is lock-free, a miss is loud — closes the D12 lazy-init race).
if err := r.buildClients(); err != nil {
return nil, err
}
// W0: build the per-model rate-guards (transport axis, read-only in the waves; a no-op until a
// the precompute pass: build the per-model rate-guards (transport axis, read-only in the waves; a no-op until a
// model configures rate_limit — WS1 §1б, the mistral-arm precondition WS6).
r.buildRateGuards()
@ -123,17 +123,6 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
return nil, err
}
// Snapshot is book-level (brief + stage plan + memory), computed once and
// upserted before the loop; every job pins to it. memoryVersion() now reflects the
// materialized bank, so a changed approved glossary re-pins loudly (F1).
snapID, snapPayload, err := r.snapshotID()
if err != nil {
return nil, err
}
if err := r.Store.UpsertSnapshot(snapID, r.Book.BriefHash(), snapPayload); err != nil {
return nil, fmt.Errorf("pipeline: upsert snapshot %.12s: %w", snapID, err)
}
chunks := SplitChunks(doc.Chapters, r.segBudget())
if len(chunks) == 0 {
return nil, fmt.Errorf("pipeline: source file %s produced no chunks after normalization", r.Book.SourceFile)
@ -141,41 +130,23 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
// Масштаб прогона — одной строкой на stderr (боль smoke-прогона: N/M и знаменатель
// прогресса не появлялись в логах вообще, только в stdout/status).
r.Log.InfoContext(ctx, "book run started", "book", r.Book.BookID, "snapshot", snapID[:12],
"chapters", len(doc.Chapters), "chunks", len(chunks), "stages_per_chunk", len(r.Pipeline.Stages))
r.Log.InfoContext(ctx, "book run started", "book", r.Book.BookID,
"chapters", len(doc.Chapters), "chunks", len(chunks), "stages", len(r.Pipeline.Stages))
res := &BookResult{BookID: r.Book.BookID}
// W0: pre-compute the sticky-chain memory selection for EVERY chunk (WS1 §1б, precomputeSticky). The
// the precompute pass: pre-compute the sticky-chain memory selection for EVERY chunk (WS1 §1б, precomputeSticky). The
// sticky window (A5 scene-inertia — the recent chunks' exact matches in the current chapter, reset at
// a chapter boundary) is a CROSS-CHUNK sequential dependency, so it is computed once here in W0 and
// consumed per-chunk below. This is byte-identical to the old inline computation (Select is $0 and
// pure; the golden test proves the injected bytes did not move) — and it is the exact precompute the
// parallel wave executor will consume (the cross-chunk state cannot be recomputed inside a wave).
stickySel := precomputeSticky(chunks, r.memory, r.Pipeline.Context.GlossaryTokenBudget)
prevChapter := 0
for i, ch := range chunks {
if ch.Chapter != prevChapter {
prevChapter = ch.Chapter
// Пульс прогресса на границе главы: done/total + деньги ЭТОГО прогона —
// оператор длинной книги видит движение, не открывая status.
r.Log.InfoContext(ctx, "chapter started", "chapter", ch.Chapter,
"chunks_done", i, "chunks_total", len(chunks), "run_usd", fmt.Sprintf("%.6f", res.TotalUSD))
}
outcome, err := r.translateChunk(ctx, snapID, ch, stickySel[i])
if err != nil {
// Infra failure: abort. Chunks already committed are checkpointed;
// resume continues from here at $0 for the done work.
return res, err
}
res.Chunks = append(res.Chunks, *outcome)
res.TotalUSD += outcome.CostUSD
if outcome.Disposition == DispFlagged {
res.Flagged++
}
}
r.Log.InfoContext(ctx, "book run finished", "book", r.Book.BookID,
"chunks", len(res.Chunks), "flagged", res.Flagged, "run_usd", fmt.Sprintf("%.6f", res.TotalUSD))
return res, nil
// a chapter boundary) is a CROSS-CHUNK sequential dependency, so it is computed once here in the precompute pass and
// consumed by the parallel draft-wave workers (it cannot be recomputed inside a wave). Byte-identical to the
// retired inline computation (Select is $0 and pure — the golden test proves the injected bytes hold).
// The draft wave selects over the BASE bank (mined-excluded) so its injection is byte-identical across
// a bank-mining enrichment (the review-confirmed «переоплата ОДНА» fix); the editor uses the enriched bank.
stickySel := precomputeSticky(chunks, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget)
// The wave executor (R1, waverun.go): the draft wave (draft ∥) → the bank-mining stop → the edit wave (edit ∥). It computes +
// upserts the per-wave snapshots (draft-wave snapshot base-bank / edit-wave snapshot enriched) itself and pins each
// wave's jobs to its own — a the bank-mining stop enrichment moves only edit-wave snapshot, keeping draft-wave checkpoints valid
// («переоплата ОДНА»). Returns a *WaveSignatureStop when the bank-mining stop stops for owner sign.
return r.translateBookWaves(ctx, chunks, stickySel)
}
// unionSticky merges the recent chunks' exact-matched ids into one sticky_prev, carrying

4
backend/internal/pipeline/cheapgates.go
View file

@ -43,7 +43,9 @@ import (
// scale) and DC6 (register negative-list — the zh-ru pack, checkers_zh_ru.go). They are observability
// (never a disposition), tuned precision-over-recall; the bump is a loud --resnapshot as the discipline
// requires (a rule/pack edit shifts recorded counts). They fire 0 on a non-zh source / non-register text.
const cheapGateVersion = "cheapgate-v3-dc-checkers"
// Ш-2 (see memnorm.go): the cheap style/DC checkers classify via unicode predicates + NFC folding, so a
// toolchain Unicode bump that shifts a class is a loud --resnapshot rather than a silent count change.
const cheapGateVersion = "cheapgate-v3-dc-checkers+u" + unicode.Version
// cheapGateConfig carries the brief-derived knobs: the ё-policy and the per-project allowlist of
// surfaces that look like a blocklisted interjection but are legitimate here (e.g. a character

8
backend/internal/pipeline/chunker.go
View file

@ -48,8 +48,8 @@ type Chunk struct {
OversizedSentence bool
// EditUnitID is the 0-based book-global id of the coarse EDIT unit this draft chunk belongs to
// (WS2 decoupling: draft = small chunk, edit = large unit). An edit unit is a greedy grouping of
// WHOLE draft chunks up to EditCeilingOut (per chapter), so W2's editor reads a unit's draft as
// the concatenation of its member chunks' W1 outputs — clean reconstruction by construction (no
// WHOLE draft chunks up to EditCeilingOut (per chapter), so the edit wave's editor reads a unit's draft as
// the concatenation of its member chunks' the draft wave outputs — clean reconstruction by construction (no
// straddle; verified: grouping whole chunks reproduces the paragraph-packed count 37 with 0
// straddle on the rerun corpus). Draft chunks never cross an edit-unit boundary.
EditUnitID int
@ -80,7 +80,7 @@ func (s SegBudget) estOut(cjk, other int) float64 {
// cover/nav page) do NOT consume a chapter number, so numbering stays dense over
// real content (Фаза-0 backward compat). Fully deterministic. Each chunk carries its
// EstOut + OversizedSentence flag + EditUnitID (WS2); the edit-unit id is monotone across
// the book so a W2 unit is uniquely addressable.
// the book so a the edit wave unit is uniquely addressable.
func SplitChunks(chapters []string, seg SegBudget) []Chunk {
var out []Chunk
chapterNo := 0
@ -155,7 +155,7 @@ func chapterDraftChunks(chapterNo int, paras []string, seg SegBudget) []Chunk {
// WHOLE draft chunks until the accumulated est_out would exceed EditCeilingOut — and stamps each
// chunk's EditUnitID (monotone across the book via *nextID). Because units are formed from whole
// draft chunks, a unit's source/draft span is exactly the concatenation of its member chunks (no
// straddle) and W2 reconstructs the unit's draft losslessly. A single draft chunk larger than the
// straddle) and the edit wave reconstructs the unit's draft losslessly. A single draft chunk larger than the
// ceiling (an OversizedSentence, or a chapter whose first chunk already exceeds the ceiling) is its
// own unit — grouping never splits a chunk (never-split-below-chunk mirrors never-split-paragraph).
func assignEditUnits(chunks []Chunk, seg SegBudget, nextID *int) {

177
backend/internal/pipeline/chunkrun.go
View file

@ -1,9 +1,7 @@
package pipeline
import (
"context"
"encoding/json"
"fmt"
"maps"
"slices"
"strings"
@ -79,181 +77,6 @@ func (r *Runner) classifyOutput(role, source, output, finish string, isFinal boo
return cls, ""
}
// translateChunk drives ONE chunk through the stage list. Stages run in order,
// each feeding the next; the FIRST flagged stage stops the chunk — later stages
// are recorded `skipped` (no paid edit over a garbage draft, D2). It also runs the
// memory bank v2 hot path: it Selects the glossary injection once ($0, deterministic),
// injects it into the TRANSLATOR stage, post-checks the translator output (E1), and
// persists the per-chunk retrieval-state (observability). Returns the chunk outcome, the
// exact-matched entity ids (the next chunk's sticky_prev, A5), and an error only on an
// infra failure. stickyPrev is the prior chunks' exact matches in this chapter.
func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, memSel memorySelection) (*ChunkOutcome, error) {
out := &ChunkOutcome{Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Disposition: DispOK}
prev := ""
flagged := false
var flagReason FlagReason
// recovered carries a cosmetic sanitizer strip's cleaned export forward from the flagging
// stage (D35.4a): non-empty only when the FINAL stage flagged FlagSanitizerStripped, "" for a
// dropped substantive flag. Captured from the FIRST flagging stage (the sanitizer flags only
// the final stage, so an upstream substantive flag leaves it "" and the chunk exports empty).
recovered := ""
// draftText is the FIRST (translator) stage's output — the pre-reflow baseline for the opt-in
// regression guard's draft→final comparison (== final in a single-stage pipeline).
draftText := ""
// bankTelem carries the translator draft's banknote telemetry (WS4 point 10) into the per-chunk
// retrieval-state. Zero for every channel-off / non-banknote chunk.
var bankTelem bankFlags
// Hot path: the glossary selection for this chunk is PRE-COMPUTED in W0 (precomputeSticky, WS1 §1б)
// and passed in — the sticky chain is a cross-chunk dependency the parallel waves cannot recompute.
// It is a pure deterministic function of (chunk, sticky_prev, frozen bank), so a resumed chunk
// reproduces the identical injected bytes. Serialize the per-role injection blocks via the
// role→renderer registry (D39 слой 7).
injectionByRole := map[string]string{}
if r.memory != nil {
for role, render := range roleInjectionRenderers {
injectionByRole[role] = render(memSel.injected) // pure → per-role result is map-order-independent
}
// The disposition-gated suppressor refused to let a lower-trust longer key eat a higher-trust
// nested one (D39 слой 4): the approved term survives, but surface the seed-hygiene collision
// loudly so an operator can reconcile the draft-nesting-over-approved seed (research/13 §7).
if len(memSel.trustGated) > 0 {
r.Log.WarnContext(ctx, "memory: lower-trust longer key refused from suppressing a higher-trust nested key (approved term preserved; reconcile the seed)",
"chapter", ch.Chapter, "chunk", ch.ChunkIdx, "trust_gated", len(memSel.trustGated),
"first", memSel.trustGated[0].Suppressor+"⊃"+memSel.trustGated[0].Protected)
}
}
for stageIdx, st := range r.Pipeline.Stages {
if flagged {
// An earlier stage flagged → this stage is not attempted or billed.
detail := fmt.Sprintf("skipped: an upstream stage was flagged (%s)", flagReason)
if err := r.Store.UpsertChunkStatus(store.ChunkStatus{
BookID: r.Book.BookID, Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Stage: st.Name,
SnapshotID: snapID, Disposition: string(DispSkipped), FlagReason: string(flagReason),
Detail: detail,
}); err != nil {
return out, fmt.Errorf("pipeline: record skipped chunk_status: %w", err)
}
out.Stages = append(out.Stages, StageResult{
Stage: st.Name, Role: st.Role, Model: st.Model,
Disposition: DispSkipped, FlagReason: flagReason, Detail: detail,
})
continue
}
// Resolve the per-role memory injection from the registry (D39 слой 7): the TRANSLATOR gets the
// src→dst glossary block (keys matched against the source chunk); the BILINGUAL editor (D30.1)
// gets the CONFIRMED dst forms as target-consistency constraints; every other role gets none
// (empty string → the plain 2-message layout). The injection is a message, so it enters this
// stage's request_hash automatically (a resumed chunk reproduces it deterministically).
injection := injectionByRole[st.Role]
sr, err := r.runStage(ctx, st, stageIdx, snapID, ch, prev, injection)
if err != nil {
return out, err
}
out.Stages = append(out.Stages, *sr)
out.CostUSD += sr.CostUSD
if st.Role == roleTranslator {
bankTelem = sr.BankFlags // translator draft's banknote telemetry (WS4 point 10)
// FL-2: the resume fast-path (resumeFromChunkStatus) serves final_hash — the banknote-CLEANED
// derived checkpoint — and never re-parses the raw block, so it leaves BankFlags at the zero
// value. Restore the telemetry the fresh run persisted instead of letting persistRetrievalState
// zero the three banknote columns on every resume of a ready chunk (the comment below promises
// an identical row). Guarded on an EMPTY BankFlags so the attempt-loop path — which re-derives
// the telemetry correctly from the raw checkpoint — is never overwritten. Channel-off ⇒ always
// zero anyway, so the store read is skipped.
if bankTelem == (bankFlags{}) && sr.FromResume && r.Pipeline.Gates.Banknote.Enabled {
if prev, err := r.Store.GetRetrievalState(r.Book.BookID, ch.Chapter, ch.ChunkIdx); err == nil && prev != nil {
bankTelem = bankFlags{
NLines: prev.NBanknoteLines,
ParseFail: prev.BanknoteParseFail != 0,
Truncated: prev.BanknoteTruncated != 0,
}
}
}
}
if sr.Disposition == DispFlagged {
flagged = true
flagReason = sr.FlagReason
recovered = sr.RecoveredText
continue
}
prev = sr.Text
if stageIdx == 0 {
draftText = sr.Text // the translator draft, before any reflow (regression-guard baseline)
}
}
// Post-check the FINAL, exported output (E1): the reader sees the last stage's text
// (the editor's), not the translator draft — the monolingual editor can drift an
// approved term the translator got right, so checking the draft alone would miss it.
// Runs on the fresh OR fully-resumed chunk (both carry the final text through `prev`),
// so it is resume-reproducible. In the default FLAGGER mode a miss is recorded only in
// the retrieval-state (observability); with the opt-in gate a miss flags the chunk
// (glossary_miss). Skipped when a stage already flagged (no usable output to check).
var postMisses []postcheckMiss
outputChecked := false
if r.memory != nil && !flagged && prev != "" {
outputChecked = true
postMisses = r.memory.postcheck(memSel.injected, prev)
// The gate flips ONLY on CONFIRMED (approved) misses — an AMBIGUOUS miss is an
// unverified candidate the model may legitimately reject, so it must not discard a
// correct translation (external-review major #1).
if r.Pipeline.Gates.Glossary.PostcheckGate && countConfirmedMisses(postMisses) > 0 {
flagged = true
flagReason = FlagGlossaryMiss
r.Log.WarnContext(ctx, "glossary post-check gate flagged the chunk",
"chapter", ch.Chapter, "chunk", ch.ChunkIdx, "confirmed_misses", countConfirmedMisses(postMisses))
}
}
// Cheap deterministic style/number flaggers on the FINAL text (cheapgates.go): observability
// only, never a disposition. Runs on the same fresh-or-resumed output as the post-check, so it
// is resume-reproducible; skipped when a stage flagged (no usable output). Source is ch.Text
// (the 万/億 magnitude gate compares source↔output). These are READABILITY flaggers whose rules
// bake in Russian conventions (em-dash dialogue, ёфикатор, ru magnitude words), so they are gated
// behind a Russian target (D39 слой 7, L8-readability-gates-target-blind) — a no-op for any other
// pair, which would false-flag. Prod zh→ru is unaffected (target=ru).
var cheap cheapGateResult
if !flagged && prev != "" && isRuTarget(r.Book.TargetLang) {
cheap = runCheapGates(ch.Text, draftText, prev, r.cheapGateConfig())
if cheap.total() > 0 {
r.Log.InfoContext(ctx, "cheap style gates flagged the chunk (observability, not a gate)",
"chapter", ch.Chapter, "chunk", ch.ChunkIdx, "style_flags", cheap.total(),
"dialogue_dash", cheap.DialogueDash, "yo", cheap.YoInconsistent,
"translit_interj", cheap.TranslitInterj, "number_magnitude", cheap.NumberMagnitude)
}
}
// Persist the per-chunk retrieval-state (registry gate #4: convert silent memory
// degradation into a loud, visible record) plus the cheap style-flag counts. Deterministic +
// idempotent, so a resume reproduces the identical row — the memory/style counts re-derive from
// the deterministic selection + post-check, and the banknote columns are carried forward on resume
// (FL-2 above) since the raw block is not re-parsed on the fast-path. Only when a glossary is
// materialized (always true in a normal run — seedGlossary sets an at-least-empty bank).
if r.memory != nil {
if err := r.persistRetrievalState(snapID, ch, memSel, postMisses, outputChecked, cheap, bankTelem); err != nil {
return out, err
}
}
// The export contract (D39 слой 6): NORMALISE every final text before it ships — clean OR
// flagged-stripped — so a cosmetic Unicode artifact (fullwidth glyph, U+3000 indent, stray CJK
// punctuation, a leaked combining stress mark) is fixed uniformly, not only on a flagged chunk
// (L5-normalization-fused-to-flag-path). A cosmetic sanitizer strip exports its cleaned remainder
// (recovered != ""); every other flag exports "" (the contaminated output never reaches TM/export,
// D2 / D35.4a), and exportNormalize("") == "". Pure ephemeral projection — no wire/checkpoint touch.
if flagged {
out.Disposition = DispFlagged
out.FlagReason = flagReason
out.FinalText = exportNormalize(recovered)
} else {
out.FinalText = exportNormalize(prev)
}
return out, nil
}
// injectionRenderer serializes a chunk's selected memory records into a role's injection message.
type injectionRenderer func(injected []pickedEntry) string

4
backend/internal/pipeline/disposition.go
View file

@ -111,7 +111,9 @@ const (
// re-runs a skipped stage (fresh spend), ok→flagged burns a fresh escalation hop.
// Folded into the snapshot exactly like coverageGateVersion, so such a change is a
// loud --resnapshot, not a silent divergence (external-review finding).
const classifierVersion = "classify-v1-refusal+echo015+loop"
// Ш-2 (see memnorm.go): unicode.Version is woven in so a toolchain Unicode bump that shifts the
// echo/refusal normalization or character classification re-verdicts a resumed chunk LOUDLY (--resnapshot).
const classifierVersion = "classify-v1-refusal+echo015+loop+u" + unicode.Version
// decodeErrorFinish is the finish_reason the runner stores on a billed-but-
// unreadable 2xx (BilledDecodeError). classify() recognises it so a RESUMED

9
backend/internal/pipeline/escalation.go
View file

@ -109,6 +109,15 @@ func (r *Runner) maybeEscalate(ctx context.Context, st config.Stage, snapID stri
}
mayHop := fbExists != nil
if !mayHop {
// Fresh hop: serialize the budget admission THROUGH the paid settle across the parallel the draft wave draft
// workers (R1, escMu). escalationBudgetRemains is a non-atomic read-then-act over EscalationSpentUSD,
// so without this N concurrent draft chunks could each read spent<budget and all be admitted →
// overshoot by up to N-1 hops. Holding escMu until this function returns (past runAttempt's settle)
// makes a later worker read the UPDATED spend, restoring the sequential soft-cap (overshoot ≤ 1 hop).
// The $0 replay path above (fbExists != nil) stays lock-free; escalations are the rare content-failure
// exception, so the contention is negligible.
r.escMu.Lock()
defer r.escMu.Unlock()
if mayHop, err = r.escalationBudgetRemains(); err != nil {
return out, err
}

98
backend/internal/pipeline/export.go
View file

@ -101,53 +101,57 @@ func (r *Runner) Export(pairs bool) (*BookExport, error) {
if n := len(r.Pipeline.Stages); n > 0 {
lastStage = r.Pipeline.Stages[n-1].Name
}
// Index the final-stage rows by chunk key (the exported text + verdict live there) and record the
// snapshot ids the rows carry (for the drift check).
// Index the final-stage rows by their addressing key (the exported text + verdict live there) and
// record the snapshot ids they carry. Under the R1 wave model the SHIPPING stage is the editor, run
// per EDIT UNIT — so its chunk_status rows exist only at each unit's LEADER (chapter, firstChunkIdx),
// NOT per draft chunk. finalRow therefore holds one entry per output unit (per draft chunk for a
// draft-only pipeline). All these rows carry the FINAL wave's snapshot (edit-wave snapshot for an edit
// pipeline, draft-wave snapshot for draft-only).
finalRow := map[chunkKey]store.ChunkStatus{}
snapSeen := map[string]bool{}
for _, cs := range statuses {
if cs.Stage != lastStage {
continue // the exported text and the chunk's final verdict live on the final stage's row
continue // the exported text and the unit's final verdict live on the final stage's row
}
finalRow[chunkKey{cs.Chapter, cs.ChunkIdx}] = cs
if cs.SnapshotID != "" {
snapSeen[cs.SnapshotID] = true
}
}
// F4: the $0 source manifest is the authoritative chunk set — join it so a PENDING chunk is explicit
// and a GHOST (deleted-source) row is not exported as if live.
// F4: the $0 source manifest is authoritative. Under the wave model the export granularity is the EDIT
// UNIT (the editor collapses a unit's member chunks into ONE final text), so join the manifest into
// output units — each a (chapter, firstChunkIdx) key + the unit's joined source (the --pairs column
// aligned byte-for-byte to how the edit wave concatenated the members). A draft-only pipeline has one output per
// draft chunk (the draft ships). Iterating units keeps a PENDING unit explicit and a GHOST row visible.
manifest, err := r.bookChunks()
if err != nil {
return nil, err
}
units := r.outputUnits(manifest)
exp := &BookExport{BookID: r.Book.BookID, Chunks: []ChunkExport{}}
inManifest := map[chunkKey]bool{}
for _, ch := range manifest {
k := chunkKey{ch.Chapter, ch.ChunkIdx}
inManifest[k] = true
for _, u := range units {
key := chunkKey{u.Chapter, u.FirstChunkIdx}
inManifest[key] = true
exp.TotalChunks++
cs, ok := finalRow[k]
cs, ok := finalRow[key]
if !ok {
pend := ChunkExport{Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Disposition: exportPending}
pend := ChunkExport{Chapter: u.Chapter, ChunkIdx: u.FirstChunkIdx, Disposition: exportPending}
if pairs {
pend.Source = ch.Text
pend.Source = u.sourceText()
}
exp.PendingChunks++
exp.Chunks = append(exp.Chunks, pend)
continue
}
ce, err := r.chunkExport(cs, gateOn, missByChunk[k])
ce, err := r.chunkExport(cs, gateOn, missByChunk[key])
if err != nil {
return nil, err
}
if pairs {
ce.Source = ch.Text // the src↔target column for the DC1/DC2 FP-measure (--pairs)
ce.Source = u.sourceText() // the unit src↔target column for the DC1/DC2 FP-measure (--pairs)
}
exp.Chunks = append(exp.Chunks, ce)
}
// F4: GHOST rows — a stored final row whose chunk is NOT in the current manifest (source shrunk since
// the run). Dropped (never exported as if live) but counted + WARNed so the drop is loud.
// F4: GHOST rows — a stored final row whose unit-leader key is NOT in the current manifest units
// (source shrunk since the run). Dropped (never exported as if live) but counted + WARNed.
for k := range finalRow {
if !inManifest[k] {
exp.GhostRows++
@ -158,23 +162,51 @@ func (r *Runner) Export(pairs bool) (*BookExport, error) {
"book", r.Book.BookID, "ghost_rows", exp.GhostRows)
}
// F3: config/gate drift — only meaningful on a single stored snapshot (multi-snapshot drift is
// already visible in the per-chunk snapshot_id). Compute the CURRENT config's projected snapshot
// read-only and compare; surface a mismatch as a flag + WARN, never silently.
if len(snapSeen) == 1 {
var stored string
for s := range snapSeen {
stored = s
// F3: config/gate drift — the stored rows carry PER-WAVE snapshots (draft rows → draft-wave snapshot, edit rows
// → edit-wave snapshot), so compare EACH wave against its current projection (like status). A change in EITHER
// wave means `translate` would --resnapshot (a draft-config change re-pins the draft wave and cascades
// to the edit via content-hash). Only meaningful when a wave's rows carry a single snapshot (a
// mid-book multi-snapshot drift is already visible in the per-row snapshot_id).
draftStageNames := stageNameSet(r.waveStagesIndexed(waveDraft))
editStageNames := stageNameSet(r.waveStagesIndexed(waveEdit))
draftSnaps, editSnaps := map[string]bool{}, map[string]bool{}
for _, cs := range statuses {
if cs.SnapshotID == "" {
continue
}
if cur, serr := r.currentSnapshotProjected(); serr != nil {
r.Log.Warn("export: config-drift check failed; drift state unknown (reported as none)", "err", serr)
} else if cur != stored {
exp.ConfigDrift = true
exp.CurrentSnapshot = cur
r.Log.Warn("export: CONFIG-DRIFT — current config renders a different snapshot than the stored rows; the gate/stage re-derivation may not match the run (translate would require --resnapshot)",
"book", r.Book.BookID, "stored", stored, "current", cur)
switch {
case draftStageNames[cs.Stage]:
draftSnaps[cs.SnapshotID] = true
case editStageNames[cs.Stage]:
editSnaps[cs.SnapshotID] = true
}
}
if err := r.projectStoredMemory(); err != nil {
r.Log.Warn("export: config-drift check failed; drift state unknown (reported as none)", "err", err)
} else {
checkWave := func(snaps map[string]bool, w wave) {
if len(snaps) != 1 {
return
}
var stored string
for s := range snaps {
stored = s
}
cur, _, serr := r.snapshotIDForWave(w)
if serr != nil {
r.Log.Warn("export: config-drift check failed for a wave; drift state unknown", "err", serr)
return
}
if cur != stored {
exp.ConfigDrift = true
exp.CurrentSnapshot = cur
r.Log.Warn("export: CONFIG-DRIFT — current config renders a different snapshot than the stored rows; the gate/stage re-derivation may not match the run (translate would require --resnapshot)",
"book", r.Book.BookID, "stored", stored, "current", cur, "wave", w)
}
}
checkWave(draftSnaps, waveDraft)
checkWave(editSnaps, waveEdit)
}
return exp, nil
}

133
backend/internal/pipeline/golden_test.go
View file

@ -10,6 +10,7 @@ import (
"net/http/httptest"
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
@ -35,21 +36,23 @@ import (
//
// D28.2 fixture expansion (D30.9 diff-class, this package): three ratified behaviours now
// PINNED by golden, not only by the dedicated unit tests —
// • FLOOR: fake-model floors max_tokens to 4000, fake-fallback to 6000 (per-model
// - FLOOR: fake-model floors max_tokens to 4000, fake-fallback to 6000 (per-model
// capabilities.min_max_tokens) → the wire max_tokens and the resolved capability move,
// and the escalation HOP takes ITS OWN model's floor (ch2 hop = 6000, primary = 4000);
// • UNION: ch5 fires an OBLIQUE-ONLY decl term (老人→старик) whose FINAL text carries the
// - UNION: ch5 fires an OBLIQUE-ONLY decl term (老人→старик) whose FINAL text carries the
// NOMINATIVE «старик» → postcheck_miss=0 only because the base-dst∪decl union checks the
// base (revert the union → this chunk false-flags a miss);
// • SANITIZER (SUBSTANTIVE): ch6 edit leaks a service preamble → the output-sanitizer gate (ON
// - SANITIZER (SUBSTANTIVE): ch6 edit leaks a service preamble → the output-sanitizer gate (ON
// in the fixture pipeline) flags it FlagSanitizerDefect and the edit is DROPPED (final_text "").
//
// D38 infra-pack cells (pin the cosmetic strip-and-export path, D35.4a — final_hash points at a
// derived $0 sanitized checkpoint so the export contract yields the cleaned text; resume re-serves
// it identically at $0):
// • SANITIZER (COSMETIC markdown): ch7 edit leaks a leading «### Глава 7» → flagged
// - SANITIZER (COSMETIC markdown): ch7 edit leaks a leading «### Глава 7» → flagged
// sanitizer_stripped, final_text = «Глава 7\n\n…» (the «### » stripped), a derived checkpoint holds it;
// • SANITIZER (COSMETIC CJK): ch8 edit leaks a sparse «特产» → flagged sanitizer_stripped, final_text
// - SANITIZER (COSMETIC CJK): ch8 edit leaks a sparse «特产» → flagged sanitizer_stripped, final_text
// = the run removed and the seam tidied.
//
// A cell with AMBIGUOUS>0 (an auto/draft candidate forcing a post-check) remains an OPTIONAL
// future extension, deliberately NOT added here.
//
@ -64,9 +67,9 @@ const (
goldenEchoMarker = "響動計画"
goldenRefusalMarker = "拒絶計画"
goldenUnionMarker = "老人の秘密" // ch5 — oblique-only decl union case
goldenSanitizerMarker = "序文漏洩" // ch6 — sanitizer preamble-leak case (SUBSTANTIVE → dropped)
goldenMarkdownMarker = "見出漏洩" // ch7 — cosmetic markdown «### Глава» leak (STRIPPED + exported)
goldenCJKMarker = "漢字漏洩" // ch8 — cosmetic CJK-leak «特产» (STRIPPED + exported)
goldenSanitizerMarker = "序文漏洩" // ch6 — sanitizer preamble-leak case (SUBSTANTIVE → dropped)
goldenMarkdownMarker = "見出漏洩" // ch7 — cosmetic markdown «### Глава» leak (STRIPPED + exported)
goldenCJKMarker = "漢字漏洩" // ch8 — cosmetic CJK-leak «特产» (STRIPPED + exported)
)
// goldenRespond is the deterministic mock provider brain. The response text is a pure
@ -178,14 +181,35 @@ func captureGolden(t *testing.T, label string, r *Runner, res *BookResult, wireB
fl := func(f float64) string { return strconv.FormatFloat(f, 'f', -1, 64) }
w("==== run %s ====", label)
snapID, payload, err := r.snapshotID()
// R1: the driver pins each wave's jobs/rows to its OWN snapshot (draft rows → the draft-wave snapshot
// over the base bank, edit rows → the edit-wave snapshot over the enriched bank), so the golden pins
// BOTH — and the base/enriched split that keeps «переоплата ОДНА».
draftSnap, draftSnapPayload, err := r.snapshotIDForWave(waveDraft)
if err != nil {
t.Fatal(err)
}
w("snapshot_id: %s", snapID)
w("snapshot_payload: %s", payload)
editSnap, editSnapPayload, err := r.snapshotIDForWave(waveEdit)
if err != nil {
t.Fatal(err)
}
w("snapshot_draft: %s", draftSnap)
w("snapshot_draft_payload: %s", draftSnapPayload)
w("snapshot_edit: %s", editSnap)
w("snapshot_edit_payload: %s", editSnapPayload)
w("brief_hash: %s", r.Book.BriefHash())
w("memory_version: %s", r.memoryVersion())
w("base_memory_version: %s", r.baseMemoryVersion())
// stageWaveSnap maps a stage NAME to its wave's snapshot, so each stored row's snap_match compares
// against the RIGHT per-wave snapshot (a draft row against the draft-wave snapshot, an edit row against
// the edit-wave one). retrieval_state rows are draft-basis (written in the draft wave), so they compare
// against the draft-wave snapshot.
stageWaveSnap := map[string]string{}
for _, ws := range r.waveStagesIndexed(waveDraft) {
stageWaveSnap[ws.st.Name] = draftSnap
}
for _, ws := range r.waveStagesIndexed(waveEdit) {
stageWaveSnap[ws.st.Name] = editSnap
}
w("-- book result --")
w("chunks=%d flagged=%d exit=%d total_usd=%s", len(res.Chunks), res.Flagged, res.ExitCode(), fl(res.TotalUSD))
@ -217,7 +241,7 @@ func captureGolden(t *testing.T, label string, r *Runner, res *BookResult, wireB
})
for _, cs := range css {
w("ch%d/%d %s snap_match=%t content_hash=%s disp=%s flag=%q attempts=%d final_hash=%s cost=%s escalated=%t esc_model=%q detail=%q",
cs.Chapter, cs.ChunkIdx, cs.Stage, cs.SnapshotID == snapID, cs.ContentHash, cs.Disposition,
cs.Chapter, cs.ChunkIdx, cs.Stage, cs.SnapshotID == stageWaveSnap[cs.Stage], cs.ContentHash, cs.Disposition,
cs.FlagReason, cs.Attempts, cs.FinalHash, fl(cs.CostUSD), cs.Escalated, cs.EscalationModel, cs.Detail)
}
@ -246,7 +270,7 @@ func captureGolden(t *testing.T, label string, r *Runner, res *BookResult, wireB
})
for _, rs := range rss {
w("ch%d/%d snap_match=%t exact=%d sticky=%d ambiguous=%d spoiler=%d evicted=%d postcheck_miss=%d style_flags=%d trust_gated=%d",
rs.Chapter, rs.ChunkIdx, rs.SnapshotID == snapID, rs.NExactHits, rs.NSticky, rs.NAmbiguousFlagged,
rs.Chapter, rs.ChunkIdx, rs.SnapshotID == draftSnap, rs.NExactHits, rs.NSticky, rs.NAmbiguousFlagged,
rs.NSpoilerBlocked, rs.NEvicted, rs.NPostcheckMiss, rs.NStyleFlags, rs.NTrustGatedSuppress)
w(" injected_ids=%s postcheck_detail=%s style_detail=%s trust_gate_detail=%s", rs.InjectedIDs, rs.PostcheckDetail, rs.StyleDetail, rs.TrustGateDetail)
}
@ -304,6 +328,12 @@ func TestGoldenDeterminism(t *testing.T) {
capture += captureGolden(t, "2 (resume)", r2, res2, nil)
if os.Getenv("TM_UPDATE_GOLDEN") == "1" {
// Ш-1: print the masked structural diff BEFORE writing, so a re-capture is safe-by-construction —
// a version-only re-capture shows 0 verdict/wire changes; a ratified structural change (the R1 wave
// switch) shows them for a human to confirm (attached to the landing report).
if prev, rerr := os.ReadFile(goldenFile); rerr == nil {
t.Logf("%s", goldenMaskedDiff(string(prev), capture))
}
if err := os.WriteFile(goldenFile, []byte(capture), 0o644); err != nil {
t.Fatal(err)
}
@ -321,6 +351,83 @@ func TestGoldenDeterminism(t *testing.T) {
}
}
// goldenHexRE matches a hash-like run (≥12 hex chars) — snapshot ids, request/content/final hashes, the
// memory versions — everything that moves on a --resnapshot without a verdict change.
var goldenHexRE = regexp.MustCompile(`[0-9a-f]{12,}`)
// goldenWireIdxRE matches a wire-body line's call index prefix ("[12] ") — normalised so a call reorder
// (the wave driver runs all drafts, then all edits) is not counted as a wire change.
var goldenWireIdxRE = regexp.MustCompile(`^\[\d+\] `)
// maskGoldenLines masks the volatile hash/version fields of a golden capture: a snapshot PAYLOAD line
// (metadata whose version strings + field set move on a toolchain/table/schema bump) is blanked whole; every
// other line has its hash-like runs replaced. What survives is the verdict/wire content — final texts,
// dispositions, flags, counts, and the request BODIES (whose bytes are the wire).
func maskGoldenLines(s string) []string {
lines := strings.Split(s, "\n")
for i, ln := range lines {
if strings.HasPrefix(ln, "snapshot_draft_payload:") || strings.HasPrefix(ln, "snapshot_edit_payload:") || strings.HasPrefix(ln, "snapshot_payload:") {
lines[i] = "snapshot_payload: ⟨masked⟩"
continue
}
// Normalise the wire-body call index: the wave driver reorders the calls (all drafts, then all
// edits) vs the sequential interleave, so a byte-identical request body carries a different [N].
// Masking the index lets a reordered-but-identical body match, so the diff shows only GENUINE
// wire changes (the concatenated edit input of a multi-chunk unit), not the reorder.
ln = goldenWireIdxRE.ReplaceAllString(ln, "[⟨i⟩] ")
lines[i] = goldenHexRE.ReplaceAllString(ln, "⟨hex⟩")
}
return lines
}
// multisetDiff returns the lines of `a` that are not covered by `b` (multiset semantics), i.e. lines added
// or changed going b→a. Order-insensitive, so it survives the row insert/delete a structural change makes.
func multisetDiff(a, b []string) []string {
counts := map[string]int{}
for _, ln := range b {
counts[ln]++
}
var out []string
for _, ln := range a {
if counts[ln] > 0 {
counts[ln]--
continue
}
out = append(out, ln)
}
return out
}
// goldenMaskedDiff (Ш-1) makes a golden re-capture SAFE-BY-CONSTRUCTION: before overwriting the golden it
// masks every volatile hash/version field in BOTH the old golden and the new capture, then reports how many
// lines changed ONLY in masked fields ("version-only") vs in real verdict/wire content. A clean toolchain/
// version re-capture reports ZERO verdict/wire changes; a ratified structural change (the R1 wave switch)
// reports them so a human confirms each is intended (the masked diff attached to the landing report). The
// masked lines are compared as multisets, so row insertions/deletions (the wave switch drops per-unit edit
// rows) do not smear the count across every following line.
func goldenMaskedDiff(oldGolden, newCapture string) string {
rawAdded := multisetDiff(strings.Split(newCapture, "\n"), strings.Split(oldGolden, "\n"))
rawRemoved := multisetDiff(strings.Split(oldGolden, "\n"), strings.Split(newCapture, "\n"))
maskedAdded := multisetDiff(maskGoldenLines(newCapture), maskGoldenLines(oldGolden))
maskedRemoved := multisetDiff(maskGoldenLines(oldGolden), maskGoldenLines(newCapture))
verdictChanges := len(maskedAdded) + len(maskedRemoved)
versionOnly := (len(rawAdded) + len(rawRemoved)) - verdictChanges
var b strings.Builder
fmt.Fprintf(&b, "golden masked-diff (Ш-1): %d verdict/wire change line(s), %d version-only line(s)\n", verdictChanges, versionOnly)
show := maskedAdded
if len(maskedRemoved) > len(show) {
show = maskedRemoved
}
for i, ln := range show {
if i >= 30 {
fmt.Fprintf(&b, " … (+%d more masked changes)\n", len(show)-i)
break
}
fmt.Fprintf(&b, " ~ %s\n", ln)
}
return b.String()
}
// diffHint points a human at the first diverging line — the full capture is too big
// for a useful t.Fatalf dump.
func diffHint(want, got string) string {

9
backend/internal/pipeline/memnorm.go
View file

@ -40,7 +40,14 @@ var trad2simpRaw string
// editing/completing the data file also invalidates — drift-proof (D8): you cannot
// forget to bump a version when you change the table, because the table's bytes ARE
// the version.
const memoryNormAlgoVersion = "memnorm-v3-nfkc+apos+stripignorable+trad+kana+lower/nfc+dash+apos+stripignorable+lower+yofold"
// Ш-2 (D39.16, folded here in R1): unicode.Version is woven into the algorithm tag so a Unicode-table
// bump (a go toolchain upgrade — NFKC/NFC via x/text, unicode.Is(Cf), the kana range, significantLen's
// unicode.Han/Cyrillic predicates) changes this version → the snapshot moves → a LOUD --resnapshot,
// never a silent match-behaviour change on already-checkpointed chunks. TRIPWIRE: do NOT bump the
// toolchain past the pinned Unicode edition without a --resnapshot (we are on go1.26.4 / Unicode 15.0.0).
// (x/text/norm carries its OWN Unicode edition independent of stdlib unicode.Version — a standalone x/text
// dependency bump must be treated like a toolchain bump, i.e. a deliberate --resnapshot; residual noted.)
const memoryNormAlgoVersion = "memnorm-v3-nfkc+apos+stripignorable+trad+kana+lower/nfc+dash+apos+stripignorable+lower+yofold+u" + unicode.Version
var (
// trad2simp is the parsed Traditional→Simplified single-char map (data/trad2simp.txt).

18
backend/internal/pipeline/memory.go
View file

@ -131,11 +131,11 @@ type MemoryBank struct {
ac *ahoCorasick
keyOwners map[string][]int // normalized key → indices of entries that contributed it
// enrichedVersion folds ALL foldable rows incl Source:mined — the memory version of the EDIT
// wave (snapshot_W2). baseVersion EXCLUDES Source:mined — the memory version of the DRAFT wave
// (snapshot_W1), so a W1.5 bank-enrichment (adding mined rows) moves ONLY snapshot_W2, keeping
// W1 checkpoints valid (WS1 §1в, «переоплата ОДНА»). The two are DOMAIN-SEPARATED (a "base-excl-
// wave (edit-wave snapshot). baseVersion EXCLUDES Source:mined — the memory version of the DRAFT wave
// (draft-wave snapshot), so a the bank-mining stop bank-enrichment (adding mined rows) moves ONLY edit-wave snapshot, keeping
// draft-wave checkpoints valid (WS1 §1в, «переоплата ОДНА»). The two are DOMAIN-SEPARATED (a "base-excl-
// mined" tag in the base fold), so base ≠ enriched for EVERY row set — even a mined-free book (a
// W1 job must never content-address to a W2 checkpoint). The load-bearing invariant is not
// the draft wave job must never content-address to a the edit wave checkpoint). The load-bearing invariant is not
// equality but STABILITY: base stays fixed as mined rows are added; only enriched moves.
enrichedVersion string
baseVersion string
@ -182,7 +182,7 @@ type memorySelection struct {
// Version is the ENRICHED memory version (all approved rows incl mined) — the edit-wave / whole-book
// memory component. BaseVersion excludes Source:mined (the draft-wave component). They are
// DOMAIN-SEPARATED — base ≠ enriched for ANY row set, including a mined-free book (never assert
// equality) — but base stays STABLE when a W1.5 pass adds mined rows, which is what keeps W1
// equality) — but base stays STABLE when a the bank-mining stop pass adds mined rows, which is what keeps the draft wave
// checkpoints valid («переоплата ОДНА»).
func (b *MemoryBank) Version() string { return b.enrichedVersion }
func (b *MemoryBank) BaseVersion() string { return b.baseVersion }
@ -256,8 +256,8 @@ func materializeMemory(rows []store.GlossaryEntry, gateOn bool) *MemoryBank {
}
sort.Strings(allKeys) // deterministic automaton construction
b.ac = buildAC(allKeys)
b.enrichedVersion = computeMemoryVersion(rows, gateOn) // all approved incl mined (W2)
b.baseVersion = computeMemoryVersionScoped(rows, gateOn, true) // excl Source:mined (W1)
b.enrichedVersion = computeMemoryVersion(rows, gateOn) // all approved incl mined (the edit wave)
b.baseVersion = computeMemoryVersionScoped(rows, gateOn, true) // excl Source:mined (the draft wave)
return b
}
@ -278,8 +278,8 @@ func computeMemoryVersion(rows []store.GlossaryEntry, gateOn bool) string {
// computeMemoryVersionScoped is computeMemoryVersion with an explicit Source-scope: excludeMined
// drops every Source:"mined" row from the fold (WS1 §1в base-bank-version). The DRAFT-wave snapshot
// (snapshot_W1) folds base (excludeMined=true) so a W1.5 pass that ADDS mined rows moves ONLY the
// enriched (edit-wave) version — keeping W1 checkpoints valid, «переоплата ОДНА». Deterministic (a
// (draft-wave snapshot) folds base (excludeMined=true) so a the bank-mining stop pass that ADDS mined rows moves ONLY the
// enriched (edit-wave) version — keeping draft-wave checkpoints valid, «переоплата ОДНА». Deterministic (a
// Source filter over the same ORDER BY-stable rows). excludeMined=false is BYTE-IDENTICAL to the
// pre-split fold (no extra field), so the enriched hash / existing snapshots do not move; the
// excludeMined=true variant adds a domain separator so base and enriched never collide.

14
backend/internal/pipeline/memory_wave_test.go
View file

@ -7,31 +7,31 @@ import (
)
// TestBaseEnrichedMemoryVersionSplit pins the WS1 §1в per-wave bank-version mechanism: the BASE
// version (draft wave / snapshot_W1) excludes Source:"mined" rows, so adding a mined approved row
// after W1 moves ONLY the ENRICHED version (edit wave / snapshot_W2) — keeping W1 checkpoints valid
// («переоплата ОДНА»). A broken split (base folding mined) would move snapshot_W1 and re-bill W1.
// version (draft wave / draft-wave snapshot) excludes Source:"mined" rows, so adding a mined approved row
// after the draft wave moves ONLY the ENRICHED version (edit wave / snapshot_W2) — keeping the draft wave checkpoints valid
// («переоплата ОДНА»). A broken split (base folding mined) would move draft-wave snapshot and re-bill the draft wave.
func TestBaseEnrichedMemoryVersionSplit(t *testing.T) {
seedRows := []store.GlossaryEntry{
{Src: "方源", Dst: "Фан Юань", Status: "approved", Source: "seed"},
{Src: "古月", Dst: "Гу Юэ", Status: "approved", Source: "ruby"},
}
base0 := materializeMemory(seedRows, false)
// Same rows PLUS a mined approved row (as a W1.5 pass would add).
// Same rows PLUS a mined approved row (as a the draft wave.5 pass would add).
enrichedRows := append([]store.GlossaryEntry{}, seedRows...)
enrichedRows = append(enrichedRows, store.GlossaryEntry{Src: "蛊", Dst: "гу", Status: "approved", Source: "mined"})
bank1 := materializeMemory(enrichedRows, false)
// 1) Base version is UNCHANGED by the mined addition (draft wave stays valid).
if base0.BaseVersion() != bank1.BaseVersion() {
t.Fatalf("base version moved on a mined-row addition — snapshot_W1 would re-bill W1:\n before %s\n after %s",
t.Fatalf("base version moved on a mined-row addition — draft-wave snapshot would re-bill the draft wave:\n before %s\n after %s",
base0.BaseVersion(), bank1.BaseVersion())
}
// 2) Enriched version DID move (edit wave sees the new mined canon).
if base0.Version() == bank1.Version() {
t.Fatalf("enriched version did NOT move on a mined-row addition — the edit wave would miss the mined canon")
}
// 3) Base ≠ enriched even before any mined row (domain-separated), so a W1 job can never
// accidentally content-address to a W2 checkpoint.
// 3) Base ≠ enriched even before any mined row (domain-separated), so a the draft wave job can never
// accidentally content-address to a the edit wave checkpoint.
if base0.BaseVersion() == base0.Version() {
t.Fatalf("base and enriched versions collide over identical rows — they must be domain-separated")
}

10
backend/internal/pipeline/memseed.go
View file

@ -506,7 +506,7 @@ func hasAliasSurface(aliases []store.GlossaryAlias, surface string) bool {
// loud error listing every problem (WS3 (д): "сухой прогон фейл-лаудов реального loadGlossarySeed по
// дельте"). $0, no store, no LLM — it runs the same loader translate would, so a delta that lints clean
// here is guaranteed loadable, and a shared-key livelock / duplicate / missing-dst is caught BEFORE the
// W1.5 reseed instead of aborting a paid run.
// bank-mining reseed instead of aborting a paid run.
func SeedLint(path string) error {
entries, err := loadGlossarySeed(path)
if err != nil {
@ -518,13 +518,13 @@ func SeedLint(path string) error {
return nil
}
// --- mined → candidates (WS3 W1.5 mined-write path) -----------------------------
// --- mined → candidates (WS3 bank-mining mined-write path) -----------------------------
// minedToCandidates converts the miner's WHICH proposals (MineBank) into store candidates for the W1.5
// minedToCandidates converts the miner's WHICH proposals (MineBank) into store candidates for the the bank-mining stop
// reseed (WS3 / §1в mined-write path). It is the DEDICATED mined path the plan requires: it stamps
// Source:"mined" (mirroring rubyToCandidates' Source:"ruby"), NOT the Source:"seed" that loadGlossarySeed
// hardcodes — so a mined row is EXCLUDED from the base-bank-version (BaseVersion) and the W1.5 bank
// enrichment moves ONLY snapshot_W2, keeping W1 checkpoints valid («переоплата ОДНА», §1в/F2). Every
// hardcodes — so a mined row is EXCLUDED from the base-bank-version (BaseVersion) and the the bank-mining stop bank
// enrichment moves ONLY snapshot_W2, keeping the draft wave checkpoints valid («переоплата ОДНА», §1в/F2). Every
// candidate is status=auto with NO dst (the WHAT is delivered by the banknote/owner sign, WS4): inert on
// the hot path until a human/batch promotes it, never a blind name-lock. A src already covered by the
// manual seed is skipped (the curated entry wins), exactly like rubyToCandidates. Deterministic (MineBank

6
backend/internal/pipeline/miner_emit.go
View file

@ -15,7 +15,7 @@ import (
// which carries §B2 fragment noise), clusters aliases (tier-1), and emits one MinedTerm per NEW entity.
//
// Discipline: the miner NEVER writes `approved` — every emitted term is a `draft`/`auto` PROPOSAL for the
// owner to sign (§C2). In default B the WHAT (dst) is delivered by the banknote (WS4) at the W1.5 sign
// owner to sign (§C2). In default B the WHAT (dst) is delivered by the banknote (WS4) at the bank-mining sign
// boundary, so the miner emits WHICH-only candidates (status:auto, no dst — inert until a dst + owner
// promotion). A candidate that clusters with an existing seed surface is an alias-of-existing and is left
// for the owner to attach (not emitted as a new entity), so the delta is genuinely new terms.
@ -31,7 +31,7 @@ const emitRankCap = 200
// MinedTerm is one WHICH candidate the miner proposes (a seed-delta entry). The miner never writes a dst
// (default B) or `approved`; the caller persists it via the mined-write path (Source:"mined", status
// auto) and the owner signs it at W1.5.
// auto) and the owner signs it at the bank-mining stop.
type MinedTerm struct {
Src string
Type string // name|place|title|term (the candidate's first pattern type; "" → term)
@ -157,7 +157,7 @@ func MineBank(chunks []MinerChunk, contrast *Contrast, seed []store.GlossaryEntr
// MinedDeltaYAML serializes the mined delta into the seedTerm YAML schema (§C2-7) — the owner-sign
// artifact and the `tmctl seed-lint` input. Every term is status:auto with NO dst (WHICH-only, default
// B; the WHAT is delivered by the banknote at W1.5). It reuses the EXISTING seedTerm schema (no new
// B; the WHAT is delivered by the banknote at the bank-mining stop). It reuses the EXISTING seedTerm schema (no new
// fields — §3(б)); evidence / zones live in the sidecar sign-map, not the seed. The output is guaranteed
// loadable by loadGlossarySeed (status:auto may lack a dst); seed-lint proves it against the real loader.
func MinedDeltaYAML(mined []MinedTerm) (string, error) {

6
backend/internal/pipeline/miner_test.go
View file

@ -325,19 +325,19 @@ func TestMinedToCandidatesStampsMinedSource(t *testing.T) {
func TestMinedDeltaStaysBaseBankStable(t *testing.T) {
// The mined delta (Source:"mined") must NOT move BaseVersion — the load-bearing «переоплата ОДНА»
// invariant (§1в): a W1.5 mined-row addition moves only the enriched version. (Sibling of the
// invariant (§1в): a the draft wave.5 mined-row addition moves only the enriched version. (Sibling of the
// Block-A TestBaseEnrichedMemoryVersionSplit, here through the real mined-write path.)
seed := []store.GlossaryEntry{{Src: "方源", Dst: "Фан Юань", Status: "approved", Source: "seed"}}
base0 := materializeMemory(seed, false)
mined := minedToCandidates([]MinedTerm{{Src: "龙公", Type: "name", SinceCh: 1, Freq: 5}}, map[string]bool{"方源": true})
// A mined candidate is status=auto (not folded even into enriched when gate off) — promote it to
// approved to exercise the split, as the owner sign would at W1.5.
// approved to exercise the split, as the owner sign would at the draft wave.5.
mined[0].Status = "approved"
mined[0].Dst = "Лун Гун"
enriched := append(append([]store.GlossaryEntry{}, seed...), mined...)
bank1 := materializeMemory(enriched, false)
if base0.BaseVersion() != bank1.BaseVersion() {
t.Fatalf("base-bank-version moved on a Source:mined addition — snapshot_W1 would re-bill W1")
t.Fatalf("base-bank-version moved on a Source:mined addition — draft-wave snapshot would re-bill the draft wave")
}
if base0.Version() == bank1.Version() {
t.Fatalf("enriched version must move on a mined approved addition")

91
backend/internal/pipeline/mining.go Normal file
View file

@ -0,0 +1,91 @@
package pipeline
import (
"context"
"fmt"
"os"
"textmachine/backend/internal/store"
)
// mining.go: bank-mining stop boundary (WS3 wired live, R1). Between the drafts done and the edit wave, the
// miner scores WHICH-candidates over the the draft wave source (the detector is offline — it reads the SOURCE, not the
// drafts; drafts only mark that a chapter was reached) against the general-zh contrast, emits an
// alias-clustered seed-delta (default B, WHICH-only — the dst is the owner's to attach via the banknote at
// sign time), and — on a NON-EMPTY delta — writes the owner SIGNATURE MAP and STOPS before the edit wave. The owner
// reviews it, promotes terms into the mined-delta file (approved + dst), and re-runs: seedGlossary loads
// those as Source:mined (moving only edit-wave snapshot), the draft wave resumes at $0, the bank-mining stop re-mines (the now-seeded terms are
// excluded → the delta empties), and the edit wave runs. Mining is OFF (auto-continue) unless a langpack AND a contrast
// artifact are both configured — so every $0 test / the golden fixture (no contrast) auto-continues.
// signatureMapPath is where the bank-mining stop writes the mined-delta YAML for owner sign — beside the project DB, so it
// travels with the book state (never in git, like the DB). Deterministic (no time/rand).
func (r *Runner) signatureMapPath() string {
return r.Book.ProjectDB + ".mined-signature.yaml"
}
// runBankMiningStop executes the bank-mining stop. Returns stopped=true (with the signature map written and
// r.lastMinedCount set) when the miner proposes a non-empty delta; stopped=false (auto-continue) when mining
// is unconfigured or the delta is empty. $0 to providers (the detector is deterministic + offline).
func (r *Runner) runBankMiningStop(ctx context.Context, chunks []Chunk) (stopped bool, err error) {
if r.pack == nil || r.Pipeline.Mining.ContrastPath == "" {
return false, nil // mining not configured → auto-continue to the edit wave
}
f, err := os.Open(r.Pipeline.Mining.ContrastPath)
if err != nil {
return false, fmt.Errorf("pipeline: the bank-mining stop open mining contrast %s: %w", r.Pipeline.Mining.ContrastPath, err)
}
defer f.Close()
contrast, err := LoadContrast(f)
if err != nil {
return false, fmt.Errorf("pipeline: the bank-mining stop load mining contrast %s: %w", r.Pipeline.Mining.ContrastPath, err)
}
// The miner works over the memnorm-normalized SOURCE of every chunk (the candidate space matches the
// glossary/GT space). Deterministic; the annotation/blurb rule is handled inside MineBank's filters.
minerChunks := make([]MinerChunk, len(chunks))
for i, ch := range chunks {
minerChunks[i] = MinerChunk{Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, NSource: normalizeSourceKey(ch.Text)}
}
seed, err := r.Store.GlossaryForBook(r.Book.BookID)
if err != nil {
return false, fmt.Errorf("pipeline: the bank-mining stop read glossary for mining: %w", err)
}
mined := MineBank(minerChunks, contrast, seed, frozenMinerConfig(), r.pack)
r.lastMinedCount = len(mined)
if len(mined) == 0 {
r.Log.InfoContext(ctx, "bank-mining: empty delta, auto-continuing to the edit wave", "book", r.Book.BookID)
return false, nil
}
// Non-empty delta → write the owner signature map (the mined seed-delta YAML) and STOP before the edit wave.
yamlDelta, err := MinedDeltaYAML(mined)
if err != nil {
return false, fmt.Errorf("pipeline: the bank-mining stop marshal mined delta: %w", err)
}
if err := os.WriteFile(r.signatureMapPath(), []byte(yamlDelta), 0o644); err != nil {
return false, fmt.Errorf("pipeline: the bank-mining stop write signature map %s: %w", r.signatureMapPath(), err)
}
r.Log.WarnContext(ctx, "bank-mining: new terms await owner signature; run STOPPED before the edit wave (review the signature map, promote terms into the mined-delta file, then resume)",
"book", r.Book.BookID, "terms", len(mined), "signature_map", r.signatureMapPath())
return true, nil
}
// loadMinedDelta reads the owner-curated mined-delta YAML (book.MinedDelta) and stamps every entry
// Source:"mined" — NOT via loadGlossarySeed (which hardcodes Source:"seed", memseed.go, moving the base
// bank / draft-wave snapshot). This is the mined-write path (plan §1(в), F2): the mined terms land in the ENRICHED
// bank version but NOT the base, so adding them moves ONLY edit-wave snapshot («переоплата ОДНА»). Reuses
// loadGlossarySeed's parser/validation, then re-stamps the Source. Empty path → nil (no mined terms).
func (r *Runner) loadMinedDelta() ([]store.GlossaryEntry, error) {
if r.Book.MinedDelta == "" {
return nil, nil
}
entries, err := loadGlossarySeed(r.Book.MinedDelta)
if err != nil {
return nil, fmt.Errorf("pipeline: load mined-delta %s: %w", r.Book.MinedDelta, err)
}
for i := range entries {
entries[i].Source = "mined" // override the seed loader's Source:seed → mined (base-excluded)
}
return entries, nil
}

38
backend/internal/pipeline/quality.go
View file

@ -102,8 +102,25 @@ func (r *Runner) QualityReport() (*QualityReport, error) {
if err != nil {
return nil, err
}
// Total = the SHIPPING units (the manifest re-chunk, $0), matching status/export + the per-unit
// BookResult (R1): under the wave model the editor's final text is per EDIT UNIT, so ProcessedChunks
// (the lastStage rows, one per unit leader) and TotalChunks agree at unit granularity. The per-unit
// KPI/echo/strip signals land on the leader's edit row; a non-leader member's ChunkQuality row carries
// only its draft-side signals (trust-gated) with a 0 structural KPI — observability, never a gate.
chunks, err := r.bookChunks()
if err != nil {
return nil, err
}
units := r.outputUnits(chunks)
// GHOST guard (parity with Export/Status): a stored row whose unit-leader key is NOT in the current
// manifest (source shrank since the run) is a ghost — dropping it keeps ProcessedChunks ≤ TotalChunks
// and the echo/strip rates over live units only, instead of mixing in stale leader rows.
inManifest := map[chunkKey]bool{}
for _, u := range units {
inManifest[chunkKey{u.Chapter, u.FirstChunkIdx}] = true
}
rep := &QualityReport{BookID: r.Book.BookID}
rep := &QualityReport{BookID: r.Book.BookID, TotalChunks: len(units)}
byChunk := map[chunkKey]*ChunkQuality{}
order := []chunkKey{}
chunkOf := func(k chunkKey) *ChunkQuality {
@ -124,7 +141,17 @@ func (r *Runner) QualityReport() (*QualityReport, error) {
withheld := map[chunkKey]bool{}
// Aggregate the stored retrieval-state signals (glossary consistency, style breakdown, trust-gated).
// A retrieval_state row is keyed per DRAFT chunk, so it is live iff its chunk is still in the manifest;
// a chunk that left the source is a ghost. (A leader row also carries the unit's post-check; if the
// leader survives, so does the unit.)
liveChunks := map[chunkKey]bool{}
for _, ch := range chunks {
liveChunks[chunkKey{ch.Chapter, ch.ChunkIdx}] = true
}
for _, rs := range states {
if !liveChunks[chunkKey{rs.Chapter, rs.ChunkIdx}] {
continue // ghost retrieval_state row (chunk dropped from source)
}
q := chunkOf(chunkKey{rs.Chapter, rs.ChunkIdx})
q.GlossaryMisses += rs.NPostcheckMiss
q.TrustGated += rs.NTrustGatedSuppress
@ -151,15 +178,10 @@ func (r *Runner) QualityReport() (*QualityReport, error) {
if n := len(r.Pipeline.Stages); n > 0 {
lastStage = r.Pipeline.Stages[n-1].Name
}
seen := map[chunkKey]bool{}
for _, cs := range statuses {
k := chunkKey{cs.Chapter, cs.ChunkIdx}
if !seen[k] {
seen[k] = true
rep.TotalChunks++
}
if cs.Stage != lastStage {
continue // the exported text and the final verdict live on the final stage's row
if cs.Stage != lastStage || !inManifest[k] {
continue // the final verdict lives on the final stage's row (per unit); drop ghost leader rows
}
// Every chunk that REACHED the final stage has exactly one lastStage row (ok, cosmetic-strip,
// or skipped-because-an-upstream-stage-flagged). This is the common rate denominator so the

4
backend/internal/pipeline/ratelimit.go
View file

@ -12,7 +12,7 @@ import (
// parallelism (a token-bucket / concurrency cap, tier-dependent — 00-provider-quirks §Транспорт),
// while grok is 0%. A model caps its own wave concurrency (max_concurrency) and, optionally, paces
// call STARTS (min_interval). This is a TRANSPORT axis: it never touches the request bytes, so it is
// wire-neutral and NOT snapshot-folded. Guards are built once in W0, read-only in the waves; each
// wire-neutral and NOT snapshot-folded. Guards are built once in the precompute pass, read-only in the waves; each
// guard's internals (a buffered-channel semaphore + a mutex-protected pacer) are concurrency-safe,
// so acquiring from N wave goroutines is race-clean.
@ -79,7 +79,7 @@ func (g *rateGuard) acquire(ctx context.Context) (func(), error) {
return release, nil
}
// buildRateGuards constructs a guard for every reachable model in W0 (read-only in the waves).
// buildRateGuards constructs a guard for every reachable model in the precompute pass (read-only in the waves).
// A model with no rate_limit config gets an unlimited (no-op) guard, so rateGuard(model) is always
// non-nil and lookup-safe.
func (r *Runner) buildRateGuards() {

10
backend/internal/pipeline/render.go
View file

@ -47,13 +47,19 @@ import (
// (ru) tokens via fertility (est_out), draft chunks decoupled from coarse EDIT units (grouped whole
// chunks to EditCeilingOut), oversized-sentence flag added. The budget itself is ALSO folded via
// segmentationSnap (snapshot.go), so this version covers only the algorithm shape.
const chunkerVersion = "chunker-v5-output-budget-editunit"
// Ш-2 EXTENSION (see memnorm.go; beyond the owner-named memnorm/classifier/style set — justified: same
// silent-drift class): tokenClassCounts (chunker.go) classifies source runes via unicode.Han/Hiragana/
// Katakana/Hangul to compute est_out, which SETS chunk boundaries → the wire. A toolchain Unicode bump that
// reclassifies a rune would silently re-chunk the book; weaving unicode.Version makes it a loud --resnapshot.
const chunkerVersion = "chunker-v5-output-budget-editunit+u" + unicode.Version
// estimatorVersion versions EstimateTokens: его выход входит в max_tokens и
// через него в request-hash, поэтому перекалибровка весов — тоже явная
// инвалидация через snapshot, а не тихий промах всех чекпоинтов (находка
// ревью: code-only правка эстиматора пере-оплатила бы полкниги).
const estimatorVersion = "estimator-v0"
// Ш-2 EXTENSION (see memnorm.go): EstimateTokens classifies runes via the same unicode ranges to size
// max_tokens (∈ request_hash), so a toolchain Unicode reclassification shifts the wire — folded loudly.
const estimatorVersion = "estimator-v0+u" + unicode.Version
// maxTokensPolicyVersion versions the attempt→max_tokens scaling
// (maxTokensForAttempt, disposition.go). attempt-0 budget is already covered by

82
backend/internal/pipeline/runner.go
View file

@ -5,8 +5,11 @@ import (
"fmt"
"log/slog"
"os"
"path/filepath"
"sync"
"textmachine/backend/internal/config"
"textmachine/backend/internal/lang"
"textmachine/backend/internal/ledger"
"textmachine/backend/internal/llm"
"textmachine/backend/internal/store"
@ -44,7 +47,7 @@ type Runner struct {
clients map[string]llm.LLMClient
templates map[string]*PromptTemplate
// rateGuards is the per-model wave-concurrency guard set (WS1 §1б), built once in W0 and
// rateGuards is the per-model wave-concurrency guard set (WS1 §1б), built once in the precompute pass and
// read-only in the waves — a transport axis, never snapshot-folded. nil until buildRateGuards.
rateGuards map[string]*rateGuard
@ -54,6 +57,36 @@ type Runner struct {
// nil until materialized (report path / no glossary) → memoryVersion() falls back
// to the empty-materialization hash, a stable constant.
memory *MemoryBank
// baseMemory is the DRAFT-wave glossary bank: the BASE rows only (Source∈{seed,ruby,auto}, EXCLUDING
// Source:mined). The draft wave's injection MUST be selected over this — not the enriched `memory` —
// so it is BYTE-IDENTICAL across a bank-mining enrichment, matching the draft-wave snapshot which folds
// baseMemoryVersion (mined-excluded). Otherwise signing a mined term would change the draft wire (the
// injection is a message folded into request_hash) and silently re-bill the draft wave on the owner
// re-run — even though the base snapshot is unchanged (the review-confirmed «переоплата ОДНА» hole).
// When the book has NO mined rows (every $0 test / the golden), it is the SAME object as `memory`
// (identical content) — no double materialization, the injection is unchanged. The editor keeps the
// enriched `memory`. nil ⇔ memory is nil (materialized together in seedGlossary).
baseMemory *MemoryBank
// pack is the book's language-data pack (internal/lang), loaded once in openRunner from
// book.LangpackRoot (D39.15/16). The the bank-mining stop bank-miner reads its tables; pack.Version() is folded into
// the snapshot (a pack edit is a loud --resnapshot). nil when the book declares no langpack_root, or
// its pair has no catalog dir — then the miner is inert (the bank-mining stop auto-continues) and the fold is omitted.
pack *lang.Pack
// escMu serializes the single-hop escalation budget admission across the PARALLEL draft workers
// (R1). escalationBudgetRemains is a non-atomic read-then-act over EscalationSpentUSD, so N concurrent
// draft chunks could each read spent<budget and all be admitted → the premium soft-cap overshoots by
// up to N-1 hops. Holding escMu across the budget check AND the fresh hop's settle makes the cap exact
// (a later chunk reads the updated spend). Escalations are the rare content-failure exception, so the
// contention is negligible; the $0 checkpoint-replay of an already-paid hop stays lock-free.
escMu sync.Mutex
// lastMinedCount is the size of the most recent the bank-mining stop mined delta (set by runBankMiningStop) — carried into the
// WaveSignatureStop the driver returns so the CLI can report "N terms await signature". Single-writer
// (runBankMiningStop runs between the parallel waves, not inside one), so no synchronization is needed.
lastMinedCount int
}
// NewRunner loads all three configs rooted at book.yaml and opens the project
@ -144,9 +177,46 @@ func openRunner(bookPath string, logger *slog.Logger, forWrite bool) (*Runner, e
st.Close()
return nil, err
}
if err := r.loadLangPack(); err != nil {
st.Close()
return nil, err
}
return r, nil
}
// loadLangPack resolves the book's language-data pack (D39.15/16), loaded on BOTH the write path and the
// read-only path so status/export/report reproduce the identical snapshot (pack.Version() is folded). The
// contract mirrors the owner directive R1: a pair WITH a catalog directory (configs/langpacks/<src>-<tgt>/)
// loads FAIL-LOUD (a missing/corrupt file stops the run, never a silently-empty miner); a pair WITHOUT a
// catalog — or a book with no langpack_root — runs with a nil pack (the miner is inert, the bank-mining stop auto-continues,
// and the snapshot fold is omitted). Presence of the pair directory is the "catalog exists" signal.
func (r *Runner) loadLangPack() error {
if r.Book.LangpackRoot == "" {
return nil // no langpack declared → nil pack, miner inert
}
pairDir := filepath.Join(r.Book.LangpackRoot, r.Book.LangPair())
if fi, err := os.Stat(pairDir); err != nil || !fi.IsDir() {
// No catalog for this pair → nil-and-run (a ja book against a zh-only root just runs без майнинга).
return nil
}
pack, err := lang.Load(r.Book.LangpackRoot, r.Book.SourceLang, r.Book.TargetLang)
if err != nil {
return fmt.Errorf("pipeline: load langpack for %s: %w", r.Book.LangPair(), err)
}
r.pack = pack
return nil
}
// packVersion is the snapshot-folded language-pack version: pack.Version() when a pack is loaded, "" when
// not (omitted from the snapshot so a no-pack book is byte-stable and never re-billed for a feature it does
// not use). A pack DATA edit changes Version() → the snapshot moves → a loud --resnapshot (R1, drift-proof).
func (r *Runner) packVersion() string {
if r.pack != nil {
return r.pack.Version()
}
return ""
}
func (r *Runner) Close() error { return r.Store.Close() }
func (r *Runner) loadTemplates() error {
@ -186,7 +256,7 @@ func (r *Runner) segBudget() SegBudget {
// reachableModels enumerates every model a run can call: the union of the stage models and their
// single-hop escalate_to fallbacks (stagerun.go calls r.client ONLY with model∈{st.Model, EscalateTo}).
// This set is complete and static for a book, which is what lets W0 pre-build every client (a third
// This set is complete and static for a book, which is what lets the precompute pass pre-build every client (a third
// model axis — channel B / annotator — must extend this to stay race-free). Deterministic order.
func (r *Runner) reachableModels() []string {
seen := map[string]bool{}
@ -204,7 +274,7 @@ func (r *Runner) reachableModels() []string {
return out
}
// buildClients EAGER-constructs every reachable LLM client BEFORE any wave goroutine starts (W0).
// buildClients EAGER-constructs every reachable LLM client BEFORE any wave goroutine starts (the precompute pass).
// After this the clients map is READ-ONLY in the waves, so r.client is lock-free and a miss is a
// loud error, not a lazy build under a data race — closing the runner.go lazy-init race (D12) and
// tripwiring a future un-enumerated model axis. Idempotent; BuildClient needs only the provider
@ -216,7 +286,7 @@ func (r *Runner) buildClients() error {
}
c, err := BuildClient(r.Models, m, r.Log)
if err != nil {
return fmt.Errorf("pipeline: eager-build client for model %q (W0): %w", m, err)
return fmt.Errorf("pipeline: eager-build client for model %q (the precompute pass): %w", m, err)
}
r.clients[m] = c
}
@ -224,11 +294,11 @@ func (r *Runner) buildClients() error {
}
// client returns the PRE-BUILT client for a model. Read-only (no lazy build, no lock): buildClients
// constructed every reachable client in W0, so the map is only read in the waves; a miss means an
// constructed every reachable client in the precompute pass, so the map is only read in the waves; a miss means an
// un-enumerated model reached the wire and is a loud error, never a silent lazy build under a race.
func (r *Runner) client(model string) (llm.LLMClient, error) {
if c, ok := r.clients[model]; ok {
return c, nil
}
return nil, fmt.Errorf("pipeline: no pre-built client for model %q — W0 buildClients enumerates stage models ∪ escalate_to; a model outside that set reached the wire, add it to the eager set to keep the waves race-free", model)
return nil, fmt.Errorf("pipeline: no pre-built client for model %q — the precompute pass buildClients enumerates stage models ∪ escalate_to; a model outside that set reached the wire, add it to the eager set to keep the waves race-free", model)
}

34
backend/internal/pipeline/runner_test.go
View file

@ -102,16 +102,17 @@ func maxTokensOf(t *testing.T, body string) int {
const fakeCallUSD = (800*1.0 + 200*0.1 + 500*2.0) / 1e6
type projectOpts struct {
source string
epub []epubChapter // when set, the source is an epub (source.epub) instead of txt
spine []string // spine order for epub
regenerate int
minMaxTokens int
bookUSD float64
gatesYAML string // optional gates:/... block appended to pipeline.yaml
glossarySeed string // optional glossary seed YAML content; "" = no glossary
postcheckGate bool // enable the memory post-check hard gate (assumes gatesYAML is empty)
banknote bool // enable the banknote channel gate (WS4; assumes gatesYAML/postcheckGate empty)
source string
epub []epubChapter // when set, the source is an epub (source.epub) instead of txt
spine []string // spine order for epub
regenerate int
minMaxTokens int
bookUSD float64
gatesYAML string // optional gates:/... block appended to pipeline.yaml
glossarySeed string // optional glossary seed YAML content; "" = no glossary
postcheckGate bool // enable the memory post-check hard gate (assumes gatesYAML is empty)
banknote bool // enable the banknote channel gate (WS4; assumes gatesYAML/postcheckGate empty)
waveWorkers int // wave-executor parallelism (0 → default 1, a deterministic sequential-structured run)
}
func setupProjectOpts(t *testing.T, providerURL string, o projectOpts) string {
@ -159,10 +160,11 @@ version: 1
defaults: { max_output_ratio: 2.0, min_max_tokens: %d }
retries: { regenerate_before_escalate: %d }
context: { glossary_injection: selective, glossary_token_budget: 800 }
waves: { workers: %d }
stages:
- { name: draft, role: translator, model: fake-model, prompt: prompts/translator.md, prompt_version: v-test, temperature: 0.3, reasoning: "off" }
- { name: edit, role: editor, model: fake-model, prompt: prompts/editor.md, prompt_version: v-test, temperature: 0.4, reasoning: "off" }
%s`, o.minMaxTokens, o.regenerate, gatesBlock))
%s`, o.minMaxTokens, o.regenerate, o.waveWorkers, gatesBlock))
sourceName := "source.txt"
if len(o.epub) > 0 {
@ -375,8 +377,14 @@ func TestRunnerSnapshotPinning(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if rec.count() != 4 {
t.Fatalf("resnapshot run must re-call both stages, calls=%d", rec.count())
// R1 per-wave --resnapshot is SURGICAL, not all-or-nothing: the changed TRANSLATOR prompt moves only
// snapshot_W1, so the DRAFT wave re-pins and re-calls (+1). The EDITOR resumes: snapshot_W2 is unchanged
// AND its wire input (the mock draft "ЧЕРНОВИК ПЕРЕВОДА" is prompt-independent) is byte-identical, so
// its content-addressed checkpoint hits — re-billing a byte-identical wire request is exactly the D15
// waste the per-wave snapshot avoids. Total = 2 (fresh) + 1 (draft re-call) = 3. In production a draft
// prompt change WOULD change the draft output, cascading to the editor via its content-hash (→ 4).
if rec.count() != 3 {
t.Fatalf("resnapshot run must re-call only the changed (draft) wave; the editor resumes on an unchanged wire, calls=%d", rec.count())
}
if res.TotalUSD <= 0 {
t.Fatal("re-translation must be billed")

30
backend/internal/pipeline/seeding.go
View file

@ -43,7 +43,16 @@ func (r *Runner) seedGlossary(ctx context.Context) error {
r.Log.WarnContext(ctx, "ruby kana-alias skipped as a homophone collision (kana form left unmatchable; disambiguate in the seed if needed)",
"skipped", strings.Join(skipped, "; "))
}
entries = append(entries, rubyToCandidates(ruby, manualSrcs)...)
// Mined-write path (R1, plan §1(в)/F2): the owner-curated mined-delta file joins the seed as
// Source:"mined" (loadMinedDelta re-stamps the seed loader's Source), so its terms fold into the
// ENRICHED bank version but NOT the base — a re-run that adds signed mined terms moves ONLY snapshot_W2.
// Loaded AFTER the seed/ruby so manualSrcs already reflects the curated seed. A `mined` term that
// duplicates a seed src is caught by approvedSharedKeyCollisions below like any other collision.
minedDelta, err := r.loadMinedDelta()
if err != nil {
return err
}
entries = append(entries, minedDelta...)
for i := range entries {
entries[i].BookID = r.Book.BookID
}
@ -61,6 +70,25 @@ func (r *Runner) seedGlossary(ctx context.Context) error {
return fmt.Errorf("pipeline: read glossary for %s: %w", r.Book.BookID, err)
}
r.memory = materializeMemory(rows, r.Pipeline.Gates.Glossary.PostcheckGate)
// The DRAFT wave selects over a BASE-scoped bank (Source:mined excluded) so its injection is
// byte-identical across a bank-mining enrichment — matching the draft-wave snapshot (baseMemoryVersion),
// which keeps «переоплата ОДНА» honest at the WIRE level, not only the version-hash level. Only the
// editor sees mined terms (the enriched `memory`). When there are no mined rows (every $0 test / the
// golden) the base bank IS the enriched one — share the object, no double materialization, no drift.
baseRows := rows[:0:0]
hasMined := false
for _, row := range rows {
if row.Source == "mined" {
hasMined = true
continue
}
baseRows = append(baseRows, row)
}
if hasMined {
r.baseMemory = materializeMemory(baseRows, r.Pipeline.Gates.Glossary.PostcheckGate)
} else {
r.baseMemory = r.memory
}
if cols := injectivityCollisions(rows); len(cols) > 0 {
r.Log.WarnContext(ctx, "glossary approved dst-collisions (B2: two source terms share one Russian surface — the reader cannot tell them apart)",
"collisions", strings.Join(cols, "; "))

2
backend/internal/pipeline/seedlint_test.go
View file

@ -51,7 +51,7 @@ func TestSeedLintCatchesDefects(t *testing.T) {
func TestSeedLintEmittedMinedDelta(t *testing.T) {
// The emitted mined delta must be loadable by the REAL loader (0 fail-louds) — the pre-condition for
// the W1.5 reseed.
// the bank-mining reseed.
chunks := mchunks(strings.Repeat("龙公来到青茅山。龙公很强。青茅山很高。", 4))
seed := []store.GlossaryEntry{{Src: "青茅山", Dst: "гора Цинмао", Type: "place", Status: "approved", Source: "seed"}}
mined := MineBank(chunks, smallContrast(t), seed, frozenMinerConfig(), testLangPack(t))

45
backend/internal/pipeline/snapshot.go
View file

@ -148,8 +148,8 @@ func (r *Runner) memoryVersion() string {
type wave int
const (
waveW1 wave = iota // DRAFT wave: translator-role stages + the BASE bank version (excl Source:mined)
waveW2 // EDIT wave: non-translator stages + the ENRICHED bank version (all approved incl mined)
waveDraft wave = iota // DRAFT wave: translator-role stages + the BASE bank version (excl Source:mined)
waveEdit // EDIT wave: non-translator stages + the ENRICHED bank version (all approved incl mined)
)
// snapshotID materializes the BOOK-GLOBAL job-context snapshot (§3.2): brief_hash, chunker version,
@ -161,35 +161,47 @@ func (r *Runner) snapshotID() (id, payload string, err error) {
}
// snapshotIDForWave computes the PER-WAVE snapshot (WS1 §1в, ADDITIVE — the wave executor residual pins
// each wave's jobs to it; the sequential driver still uses snapshotID). W1 folds the DRAFT stages
// (translator role) + the BASE bank version (Source∈{seed,ruby,auto}, EXCL mined); W2 folds the EDIT
// stages (non-translator) + the ENRICHED bank version (all approved incl Source:mined). A W1.5 bank
// enrichment (adding mined rows) moves ONLY the enriched version → ONLY snapshot_W2, keeping W1
// each wave's jobs to it; the sequential driver still uses snapshotID). the draft wave folds the DRAFT stages
// (translator role) + the BASE bank version (Source∈{seed,ruby,auto}, EXCL mined); the edit wave folds the EDIT
// stages (non-translator) + the ENRICHED bank version (all approved incl Source:mined). A the bank-mining stop bank
// enrichment (adding mined rows) moves ONLY the enriched version → ONLY edit-wave snapshot, keeping the draft wave
// checkpoints valid («переоплата ОДНА»). Byte-consistent with snapshotID's building — the SAME payload
// struct, only the stage subset and the memory version differ.
func (r *Runner) snapshotIDForWave(w wave) (id, payload string, err error) {
memVer := r.memoryVersion() // enriched (W2)
if w == waveW1 {
memVer := r.memoryVersion() // enriched (the edit wave)
if w == waveDraft {
memVer = r.baseMemoryVersion()
}
return r.buildSnapshotID(waveStages(r.Pipeline.Stages, w), memVer)
}
// waveStages partitions the pipeline stages by role for a wave: W1 = the translator-role (draft) stages,
// W2 = every non-translator (editor/other) stage. Order preserved. The prod C1 core is one draft + one
// finalStageWave is the wave that owns the SHIPPING (last) stage: the edit wave when the last stage is an editor/other
// role (the normal 2-stage pipeline — the edit unit ships), the draft wave when the pipeline is draft-only (the draft
// itself ships). The wave-aware read-models (export/status) use it to compare a stored final-stage row
// against the correct per-wave snapshot and to know whether the shipping unit is an edit unit or a chunk.
func (r *Runner) finalStageWave() wave {
n := len(r.Pipeline.Stages)
if n > 0 && r.Pipeline.Stages[n-1].Role != roleTranslator {
return waveEdit
}
return waveDraft
}
// waveStages partitions the pipeline stages by role for a wave: the draft wave = the translator-role (draft) stages,
// the edit wave = every non-translator (editor/other) stage. Order preserved. The prod C1 core is one draft + one
// edit, so this yields [draft] and [edit] respectively.
func waveStages(stages []config.Stage, w wave) []config.Stage {
var out []config.Stage
for _, st := range stages {
isDraft := st.Role == roleTranslator
if (w == waveW1) == isDraft {
if (w == waveDraft) == isDraft {
out = append(out, st)
}
}
return out
}
// baseMemoryVersion is the DRAFT-wave (W1) memory component: the BASE bank version (excl Source:mined).
// baseMemoryVersion is the DRAFT-wave (the draft wave) memory component: the BASE bank version (excl Source:mined).
// nil bank → the empty-materialization base hash (a stable constant), mirroring memoryVersion().
func (r *Runner) baseMemoryVersion() string {
if r.memory != nil {
@ -294,6 +306,14 @@ func (r *Runner) buildSnapshotID(stages []config.Stage, memVersion string) (id,
// src→dst layout (WS2 §2в) and the gender-constraint render (WS5). A format change shifts the
// injected bytes → the wire, so folding it separately keeps a flip a loud --resnapshot.
RenderFormatVersion string `json:"render_format_version"`
// LangpackVersion — the loaded language-data pack's content hash (internal/lang, D39.15/16),
// folded ONLY when a pack is present (omitempty otherwise, so a no-pack book is byte-stable and
// never re-billed for a feature it does not use). The pack feeds the the bank-mining stop bank-miner; a pack DATA
// edit changes what it proposes, so folding its version makes a pack edit a loud --resnapshot,
// drift-proof by mechanism (the pack's bytes ARE its version). Wire-neutral for existing
// checkpoints (the miner produces Source:mined proposals, inert until owner-approved), but folded
// per the R1 directive so the run's snapshot durably records which pack version produced it.
LangpackVersion string `json:"langpack_version,omitempty"`
// MemoryVersion — content-hash детерминированно материализованной инъекти-
// руемой памяти (approved-глоссарий+резюме+series-bible). Не bump-счётчик
// (D8): забыть пересчитать нельзя, а забытый bump переоткрыл бы класс тихой
@ -341,6 +361,7 @@ func (r *Runner) buildSnapshotID(stages []config.Stage, memVersion string) (id,
},
Segmentation: r.segmentationSnapshot(),
RenderFormatVersion: renderFormatVersion,
LangpackVersion: r.packVersion(),
MemoryVersion: memVersion,
PostcheckGate: r.Pipeline.Gates.Glossary.PostcheckGate,
Coverage: r.coverageSnapshot(),

32
backend/internal/pipeline/snapshot_wave_test.go
View file

@ -21,35 +21,35 @@ func TestSnapshotIDForWave(t *testing.T) {
}
r.memory = materializeMemory(seed, false)
w1id, w1p, err := r.snapshotIDForWave(waveW1)
w1id, w1p, err := r.snapshotIDForWave(waveDraft)
if err != nil {
t.Fatal(err)
}
w2id, w2p, err := r.snapshotIDForWave(waveW2)
w2id, w2p, err := r.snapshotIDForWave(waveEdit)
if err != nil {
t.Fatal(err)
}
// W1 and W2 differ (different stage subset AND different bank version).
// the draft wave and the edit wave differ (different stage subset AND different bank version).
if w1id == w2id {
t.Fatalf("snapshot_W1 must differ from snapshot_W2 (stage subset + bank version)")
t.Fatalf("draft-wave snapshot must differ from snapshot_W2 (stage subset + bank version)")
}
// The stage PARTITION: W1 folds the draft (translator) stage, W2 folds the edit (editor) stage.
// The stage PARTITION: the draft wave folds the draft (translator) stage, the edit wave folds the edit (editor) stage.
if !strings.Contains(w1p, `"name":"draft"`) || strings.Contains(w1p, `"name":"edit"`) {
t.Fatalf("snapshot_W1 must fold ONLY the draft stage, payload: %s", w1p)
t.Fatalf("draft-wave snapshot must fold ONLY the draft stage, payload: %s", w1p)
}
if !strings.Contains(w2p, `"name":"edit"`) || strings.Contains(w2p, `"name":"draft"`) {
t.Fatalf("snapshot_W2 must fold ONLY the edit stage, payload: %s", w2p)
}
// «Переоплата ОДНА» at the snapshot level (§1в): a W1.5 mined-approved addition moves ONLY W2.
// «Переоплата ОДНА» at the snapshot level (§1в): a the draft wave.5 mined-approved addition moves ONLY the edit wave.
enriched := append(append([]store.GlossaryEntry{}, seed...), store.GlossaryEntry{
Src: "蛊", Dst: "гу", Status: "approved", Source: "mined",
})
r.memory = materializeMemory(enriched, false)
w1id2, _, _ := r.snapshotIDForWave(waveW1)
w2id2, _, _ := r.snapshotIDForWave(waveW2)
w1id2, _, _ := r.snapshotIDForWave(waveDraft)
w2id2, _, _ := r.snapshotIDForWave(waveEdit)
if w1id2 != w1id {
t.Fatalf("snapshot_W1 moved on a mined-approved addition — W1 checkpoints would re-bill («переоплата ОДНА» broken)")
t.Fatalf("draft-wave snapshot moved on a mined-approved addition — the draft wave checkpoints would re-bill («переоплата ОДНА» broken)")
}
if w2id2 == w2id {
t.Fatalf("snapshot_W2 did NOT move on a mined-approved addition — the edit wave would miss the mined canon")
@ -60,12 +60,12 @@ func TestSnapshotIDForWave(t *testing.T) {
func TestWaveStagesPartition(t *testing.T) {
r := newRunner(t, setupProject(t, "http://127.0.0.1:1"))
defer r.Close()
w1 := waveStages(r.Pipeline.Stages, waveW1)
w2 := waveStages(r.Pipeline.Stages, waveW2)
if len(w1) != 1 || w1[0].Role != roleTranslator {
t.Fatalf("W1 must be the translator stage(s), got %+v", w1)
draftStages := waveStages(r.Pipeline.Stages, waveDraft)
editStages := waveStages(r.Pipeline.Stages, waveEdit)
if len(draftStages) != 1 || draftStages[0].Role != roleTranslator {
t.Fatalf("the draft wave must be the translator stage(s), got %+v", draftStages)
}
if len(w2) != 1 || w2[0].Role != roleEditor {
t.Fatalf("W2 must be the editor stage(s), got %+v", w2)
if len(editStages) != 1 || editStages[0].Role != roleEditor {
t.Fatalf("the edit wave must be the editor stage(s), got %+v", editStages)
}
}

175
backend/internal/pipeline/status.go
View file

@ -193,16 +193,12 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
return nil, err
}
byChunk := map[chunkKey][]store.ChunkStatus{}
snapSeen := map[string]bool{}
for _, cs := range statuses {
byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}] = append(byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}], cs)
if cs.SnapshotID != "" {
snapSeen[cs.SnapshotID] = true
}
}
// Per-chunk post-check misses (retrieval_state) — the glossary-consistency signal that is
// re-derived each run and NOT a chunk_status disposition (default flagger mode).
// Post-check misses + style flags (retrieval_state) — the unit-level signal the the edit-wave editor merged onto
// its LEADER chunk's row (a non-leader member carries only its draft injection, post-check=0).
states, err := r.Store.RetrievalStatesForBook(r.Book.BookID)
if err != nil {
return nil, err
@ -214,40 +210,51 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
styleByChunk[chunkKey{rs.Chapter, rs.ChunkIdx}] = rs.NStyleFlags
}
stagesTotal := len(r.Pipeline.Stages)
// The post-check GATE (opt-in) flags a chunk at the CHUNK level AFTER the stage loop, so it
// is NEVER written as a chunk_status row (every stage stays DispOK). Without accounting for
// it here, status would report a gate-flagged chunk as done/pass while `tmctl translate`
// exits 2 (finding #2). When the gate is ON, promote an otherwise-done chunk with a CONFIRMED
// post-check miss to flagged/glossary_miss so status matches the run.
// The shipping granularity is the OUTPUT UNIT (edit unit for an edit pipeline, draft chunk for a
// draft-only one) — status projects it like export + the per-unit BookResult. A unit is DONE when every
// member draft AND the unit's edit resolved ok, so its expected-ok count is len(members)·|draft stages| +
// |edit stages| (a non-leader member has draft rows only; the single edit row lives at the leader).
draftStages := r.waveStagesIndexed(waveDraft)
editStages := r.waveStagesIndexed(waveEdit)
draftStageNames, editStageNames := stageNameSet(draftStages), stageNameSet(editStages)
units := r.outputUnits(chunks)
// The post-check GATE (opt-in) flags a unit at the CHUNK level AFTER the stage loop, so it is NEVER
// written as a chunk_status row (every stage stays DispOK). Without accounting for it here, status
// would report a gate-flagged unit as done/pass while `tmctl translate` exits 2 (finding #2).
gateOn := r.Pipeline.Gates.Glossary.PostcheckGate
rep := &StatusReport{BookID: r.Book.BookID, TotalChunks: len(chunks)}
rep := &StatusReport{BookID: r.Book.BookID, TotalChunks: len(units)}
passports := map[int]*ChapterPassport{}
var chapterOrder []int
var processedCost float64 // spend of PROCESSED chunks (done+flagged) — the projection base (finding #7)
var processedCost float64 // spend of PROCESSED units (done+flagged) — the projection base (finding #7)
for _, ch := range chunks {
p := passports[ch.Chapter]
for _, u := range units {
p := passports[u.Chapter]
if p == nil {
p = &ChapterPassport{Chapter: ch.Chapter, Verdict: "pass"}
passports[ch.Chapter] = p
chapterOrder = append(chapterOrder, ch.Chapter)
p = &ChapterPassport{Chapter: u.Chapter, Verdict: "pass"}
passports[u.Chapter] = p
chapterOrder = append(chapterOrder, u.Chapter)
}
p.ChunksTotal++
key := chunkKey{ch.Chapter, ch.ChunkIdx}
state, reason, cost, escalated, skipped := resolveChunkState(byChunk[key], stagesTotal)
leader := chunkKey{u.Chapter, u.FirstChunkIdx}
var rows []store.ChunkStatus
for _, m := range u.Members {
rows = append(rows, byChunk[chunkKey{m.Chapter, m.ChunkIdx}]...)
}
expected := len(u.Members)*len(draftStages) + len(editStages)
state, reason, cost, escalated, skipped := resolveChunkState(rows, expected)
p.CostUSD += cost
p.StagesSkipped += skipped
miss := missByChunk[key]
miss := missByChunk[leader] // the unit's post-check ran once, at the leader row
p.PostcheckMisses += miss
rep.PostcheckMisses += miss
style := styleByChunk[key]
style := styleByChunk[leader]
p.StyleFlags += style
rep.StyleFlags += style
if gateOn && state == ChunkDone && miss > 0 {
// Matches translateChunk: the gate flags only an otherwise-ok chunk. Not re-drivable
// (the miss re-derives from the unchanged glossary; a fix is a glossary edit =
// --resnapshot, not a redrive) — counted separately so the CLI advises the right action.
// Matches the wave editor: the gate flags only an otherwise-ok unit. Not re-drivable (the miss
// re-derives from the unchanged glossary; a fix is a glossary edit = --resnapshot, not a
// redrive) — counted separately so the CLI advises the right action.
state, reason = ChunkFlagged, string(FlagGlossaryMiss)
rep.GlossaryMissFlagged++
}
@ -273,11 +280,12 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
p.ChunksPending++
}
if state == ChunkDone || state == ChunkFlagged {
processedCost += cost // a fully-attempted chunk's cost feeds the projection
processedCost += cost // a fully-attempted unit's cost feeds the projection
}
}
// Chapter verdicts (exp07 chapter rule: 0 flagged = pass, 1 = attention, ≥2 = fail).
// Chapter verdicts (exp07 chapter rule: 0 flagged = pass, 1 = attention, ≥2 = fail — over UNITS: a
// flagged edit unit counts once, not per member chunk).
sort.Ints(chapterOrder)
for _, n := range chapterOrder {
p := passports[n]
@ -295,31 +303,59 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
if rep.TotalChunks > 0 {
rep.PercentDone = 100 * float64(rep.Done) / float64(rep.TotalChunks)
}
if len(snapSeen) == 1 {
for s := range snapSeen {
// Per-wave snapshot drift (finding #3, wave-aware): the rows carry draft-wave snapshot (draft) + edit-wave snapshot
// (edit), so a normal book has TWO snapshots — that is NOT drift. SnapshotDrift means a disagreement
// WITHIN a wave (a config changed mid-book without a full re-pin). ConfigDrift compares each wave's
// single stored snapshot against the CURRENT projection of THAT wave (materialize the stored glossary
// once, then project both). rep.Snapshot reports the SHIPPING wave's snapshot.
draftSnaps, editSnaps := map[string]bool{}, map[string]bool{}
for _, cs := range statuses {
if cs.SnapshotID == "" {
continue
}
switch {
case draftStageNames[cs.Stage]:
draftSnaps[cs.SnapshotID] = true
case editStageNames[cs.Stage]:
editSnaps[cs.SnapshotID] = true
}
}
rep.SnapshotDrift = len(draftSnaps) > 1 || len(editSnaps) > 1
finalSnaps := editSnaps
if r.finalStageWave() == waveDraft {
finalSnaps = draftSnaps
}
if len(finalSnaps) == 1 {
for s := range finalSnaps {
rep.Snapshot = s
}
} else if len(snapSeen) > 1 {
rep.SnapshotDrift = true
}
// Config-drift (finding #3): compute the CURRENT config's snapshot READ-ONLY (materialize
// the STORED glossary — no re-seed, no write) and compare to the single snapshot the stored
// rows carry. A wire/verdict-affecting config edit since the last run (a prompt bump, a gate
// flip — this very package bumps the editor prompt_version) is otherwise hidden behind
// "done/pass" until translate forces a --resnapshot. Only meaningful when a single snapshot
// is on record (multi-snapshot is already flagged as SnapshotDrift). Note: a change to the
// seed FILE that was not re-run is NOT detected here (the stored glossary is what status
// projects); it surfaces on the next translate's re-seed.
if rep.Snapshot != "" {
if curSnap, serr := r.currentSnapshotProjected(); serr != nil {
// Не молчать: провал проекции раньше тихо читался как «дрифта нет», и
// оператор верил чистому статусу при реально изменённом конфиге (аудит
// цепочек). Drift остаётся false (проекция неизвестна), но с WARN.
r.Log.WarnContext(ctx, "config-drift check failed; drift state unknown (reported as none)", "err", serr)
} else if curSnap != rep.Snapshot {
rep.ConfigDrift = true
rep.CurrentSnapshot = curSnap
if !rep.SnapshotDrift && (len(draftSnaps) > 0 || len(editSnaps) > 0) {
if err := r.projectStoredMemory(); err != nil {
// Не молчать: провал проекции раньше тихо читался как «дрифта нет» (аудит цепочек).
r.Log.WarnContext(ctx, "config-drift check failed; drift state unknown (reported as none)", "err", err)
} else {
checkWave := func(snaps map[string]bool, w wave) {
if len(snaps) != 1 {
return
}
var stored string
for s := range snaps {
stored = s
}
cur, _, serr := r.snapshotIDForWave(w)
if serr != nil {
r.Log.WarnContext(ctx, "config-drift check failed for a wave; drift state unknown", "err", serr)
return
}
if cur != stored {
rep.ConfigDrift = true
rep.CurrentSnapshot = cur
}
}
checkWave(draftSnaps, waveDraft)
checkWave(editSnaps, waveEdit)
}
}
@ -361,21 +397,17 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
return rep, nil
}
// currentSnapshotProjected computes the CURRENT config's snapshotID from the STORED glossary
// (read-only: materialize what is persisted, no re-seed, no write) — the single projection
// Status uses for ConfigDrift and Redrive uses to guard its destructive reset. Side effect
// (deliberate, on the read path): it sets r.memory to the stored-glossary materialization, the
// same as Status did inline. A seed-FILE edit not yet re-run is NOT reflected here (the STORED
// glossary is projected, not the seed file) — that drift surfaces on the next translate's
// projectStoredMemory materializes r.memory from the STORED glossary (read-only: no re-seed, no write) —
// the side effect the wave-snapshot projections need. A seed-FILE edit not yet re-run is NOT reflected here
// (the STORED glossary is projected, not the seed file) — that drift surfaces on the next translate's
// re-seed, exactly as it does for `translate` itself.
func (r *Runner) currentSnapshotProjected() (string, error) {
func (r *Runner) projectStoredMemory() error {
rows, err := r.Store.GlossaryForBook(r.Book.BookID)
if err != nil {
return "", err
return err
}
r.memory = materializeMemory(rows, r.Pipeline.Gates.Glossary.PostcheckGate)
id, _, err := r.snapshotID()
return id, err
return nil
}
// RedriveSelector picks the flagged chunks to re-attack. -1 on Chapter/ChunkIdx means "any"
@ -508,13 +540,26 @@ func (r *Runner) Redrive(ctx context.Context, sel RedriveSelector) (*RedriveSumm
// (the operator explicitly accepts the re-pin/re-pay) — but ONLY the snapshot COMPARISON is skipped,
// never the seed-error-before-reset protection above.
if !r.Resnapshot {
curSnap, _, serr := r.snapshotID()
if serr != nil {
return summary, nil, fmt.Errorf("pipeline: redrive snapshot check: %w", serr)
// Per-wave drift (R1): a stored row carries its WAVE's snapshot (draft rows → draft-wave snapshot, edit
// rows → edit-wave snapshot), never the whole-pipeline one — so compare each row against the current
// projection of ITS wave (from the SEEDED r.memory seedGlossary just set, matching what the re-run
// will render). A whole-pipeline comparison would abort every redrive spuriously.
w1cur, _, e1 := r.snapshotIDForWave(waveDraft)
if e1 != nil {
return summary, nil, fmt.Errorf("pipeline: redrive draft-wave snapshot check: %w", e1)
}
w2cur, _, e2 := r.snapshotIDForWave(waveEdit)
if e2 != nil {
return summary, nil, fmt.Errorf("pipeline: redrive edit-wave snapshot check: %w", e2)
}
draftStageNames := stageNameSet(r.waveStagesIndexed(waveDraft))
for _, cs := range statuses {
if cs.SnapshotID != "" && cs.SnapshotID != curSnap {
return summary, nil, fmt.Errorf("pipeline: redrive прерван — строки книги под снапшотом %.12s, а текущий конфиг/сид рендерит %.12s (конфиг/промпты/капабилити/сид-глоссарий изменились): сброс сейчас пере-оплатил бы книгу и уничтожил бы флаг-телеметрию; повторите `translate --resnapshot`, чтобы принять пере-оплату явно, либо верните конфиг/сид", cs.SnapshotID, curSnap)
cur := w2cur
if draftStageNames[cs.Stage] {
cur = w1cur
}
if cs.SnapshotID != "" && cs.SnapshotID != cur {
return summary, nil, fmt.Errorf("pipeline: redrive прерван — строки книги под снапшотом %.12s, а текущий конфиг/сид рендерит %.12s (конфиг/промпты/капабилити/сид-глоссарий изменились): сброс сейчас пере-оплатил бы книгу и уничтожил бы флаг-телеметрию; повторите `translate --resnapshot`, чтобы принять пере-оплату явно, либо верните конфиг/сид", cs.SnapshotID, cur)
}
}
}

238
backend/internal/pipeline/testdata/golden/capture.golden vendored
View file

File diff suppressed because one or more lines are too long

56
backend/internal/pipeline/wave.go
View file

@ -1,10 +1,60 @@
package pipeline
// wave.go: the W0 (deterministic, $0) pre-compute of the wave runner (WS1 §1б). This is the FIRST piece
import "strings"
// unitJoinSeparator joins a the edit wave edit-unit's member draft chunks into the editor's input draft AND (in
// export --pairs) their source chunks into the unit's source column. It MUST be identical on both sides so
// the exported src↔dst pair aligns byte-for-byte (export map §4). "\n\n" mirrors the chunker's own paragraph
// separator (chunkDraftChunks joins paragraphs with "\n\n"), so a re-joined unit reads as clean prose.
const unitJoinSeparator = "\n\n"
// editUnit is one the edit wave unit of edit work (WS2 decoupling: draft = small chunk, edit = coarse unit). It groups
// the WHOLE draft chunks sharing a Chunk.EditUnitID (assignEditUnits stamps them per chapter), so the unit's
// source/draft is exactly the concatenation of its members — no straddle, lossless reconstruction. It is
// addressed by (Chapter, FirstChunkIdx): FirstChunkIdx is the lowest ChunkIdx among the members, the
// LEADER chunk whose (chapter, chunk_idx, "edit") key carries the unit's single edit chunk_status row.
type editUnit struct {
EditUnitID int
Chapter int
FirstChunkIdx int
Members []Chunk // in ChunkIdx order (append order from SplitChunks is already sorted)
}
// sourceText is the unit's source: the members' source joined with unitJoinSeparator (the editor's
// bilingual {{text}} and the fresh the edit wave memory.Select both run over this whole-unit source).
func (u editUnit) sourceText() string {
parts := make([]string, len(u.Members))
for i, m := range u.Members {
parts[i] = m.Text
}
return strings.Join(parts, unitJoinSeparator)
}
// buildEditUnits groups the ordered draft chunks into edit units by EditUnitID (WS2). SplitChunks emits
// chunks in (chapter, chunk_idx) order and stamps a monotone book-global EditUnitID, so a single ordered
// pass yields units in book order with members in ChunkIdx order. Pure and deterministic.
func buildEditUnits(chunks []Chunk) []editUnit {
var units []editUnit
byID := map[int]int{} // EditUnitID → index into units
for _, ch := range chunks {
if idx, ok := byID[ch.EditUnitID]; ok {
units[idx].Members = append(units[idx].Members, ch)
continue
}
byID[ch.EditUnitID] = len(units)
units = append(units, editUnit{
EditUnitID: ch.EditUnitID, Chapter: ch.Chapter, FirstChunkIdx: ch.ChunkIdx,
Members: []Chunk{ch},
})
}
return units
}
// wave.go: the the precompute pass (deterministic, $0) pre-compute of the wave runner (WS1 §1б). This is the FIRST piece
// of the wave executor built ahead of the driver switch (R1 residual): the sticky-chain memory selection
// is a CROSS-CHUNK sequential dependency (a chunk's sticky_prev is the union of the prior chunks' exact
// matches in the same chapter), so the parallel waves cannot recompute it per-chunk — it must be
// precomputed once in W0. precomputeSticky replicates EXACTLY the sequential driver's inline computation
// precomputed once in the precompute pass. precomputeSticky replicates EXACTLY the sequential driver's inline computation
// (bookrun.go's per-chapter reset + memory.Select + sticky advance), and the sequential driver now
// CONSUMES it — so it is byte-identical by construction, and the golden test proves the injection bytes
// did not move. When the driver switches to waves (residual), the waves consume this same precompute.
@ -13,7 +63,7 @@ package pipeline
// model output, so precomputing it reproduces byte-for-byte the injection the sequential runner rendered
// (parallelism does not change the bytes — the order is fixed and the input carries no model state).
// precomputeSticky computes the per-chunk memory selection for EVERY chunk in one ordered W0 pass
// precomputeSticky computes the per-chunk memory selection for EVERY chunk in one ordered the precompute pass pass
// (WS1 §1б): the sticky window is reset at each chapter boundary (a new chapter is a scene change) and
// advanced by each chunk's exact matches, exactly as the sequential loop did inline. Returns a slice
// aligned index-for-index with `chunks`. A nil bank (report path / no glossary) yields zero-valued

535
backend/internal/pipeline/waverun.go Normal file
View file

@ -0,0 +1,535 @@
package pipeline
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync"
"textmachine/backend/internal/config"
"textmachine/backend/internal/store"
)
// waverun.go: the R1 WAVE EXECUTOR — the driver switch from the sequential per-chunk-all-stages loop
// (bookrun.go/chunkrun.go, retired) to the precompute pass ($0) → the draft wave (draft stage ∥ over draft CHUNKS) →
// the bank-mining stop → the edit wave (edit stage ∥ over edit UNITS). The load-bearing decoupling (plan §1/§2):
// the DRAFT is the fine per-chunk unit (COGS/coverage/alignment), the EDIT is the coarse per-unit unit
// (a chapter-scale grouping of whole draft chunks — cross-chunk cohesion, D39 п.4). the edit wave reads a unit's draft
// as the concatenation of its members' draft-wave outputs and does a FRESH memory.Select over the whole-unit source;
// it NEVER re-renders the draft stage (the несущий invariant — a re-render over the enriched bank would move
// the draft request_hash and re-bill the draft wave). Money stays single-writer-safe (Reserve/Settle serialize); the
// waves add only read-only shared state (clients/templates/memory/rate-guards built in the precompute pass). Per-wave
// snapshots (draft-wave snapshot base-bank, edit-wave snapshot enriched) keep «переоплата ОДНА»: a the bank-mining stop enrichment moves
// only edit-wave snapshot, so draft-wave checkpoints stay valid.
// wavedStage carries a wave stage with its GLOBAL index in Pipeline.Stages — runStage needs the global
// index for isFinal (the LAST stage is the shipping stage the sanitizer runs on) and for the >0 sizing
// path (a later stage sizes max_tokens from its input draft, not the source, D2.5).
type wavedStage struct {
st config.Stage
idx int
}
// waveStagesIndexed partitions Pipeline.Stages by role for a wave, carrying each stage's global index
// (mirrors waveStages but retains the index runStage needs). the draft wave = translator (draft) stages; the edit wave = every
// non-translator (editor/other) stage. Order preserved.
func (r *Runner) waveStagesIndexed(w wave) []wavedStage {
var out []wavedStage
for i, st := range r.Pipeline.Stages {
isDraft := st.Role == roleTranslator
if (w == waveDraft) == isDraft {
out = append(out, wavedStage{st: st, idx: i})
}
}
return out
}
// stageNameSet is the set of a wave's stage names — used by the read-models to classify a stored
// chunk_status row by wave (its snapshot belongs to that wave's per-wave snapshot).
func stageNameSet(staged []wavedStage) map[string]bool {
m := make(map[string]bool, len(staged))
for _, ws := range staged {
m[ws.st.Name] = true
}
return m
}
// outputUnits is the SHIPPING granularity the read-models (export/status) project — matching the per-unit
// BookResult: edit units (member draft chunks grouped) for an edit pipeline, or one singleton unit per
// draft chunk for a draft-only pipeline (the draft itself ships). One helper so export and status agree.
func (r *Runner) outputUnits(chunks []Chunk) []editUnit {
if r.finalStageWave() == waveEdit {
return buildEditUnits(chunks)
}
out := make([]editUnit, len(chunks))
for i, ch := range chunks {
out[i] = editUnit{EditUnitID: ch.EditUnitID, Chapter: ch.Chapter, FirstChunkIdx: ch.ChunkIdx, Members: []Chunk{ch}}
}
return out
}
// WaveSignatureStop is the typed sentinel the CLI surfaces when bank-mining proposes a non-empty seed-delta:
// the run completes the draft wave, writes the owner signature map, and STOPS before the edit wave (plan §1(в) default — an explicit
// stop boundary, not a mid-run append). The owner reviews the signature map, curates the mined-delta file
// (promoting terms to approved with a dst), and re-runs: the draft wave resumes at $0, the bank-mining stop re-mines (the now-seeded
// terms are excluded → the delta empties → auto-continue), and the edit wave runs under edit-wave snapshot. NOT an infra
// failure — a deliberate human-in-the-loop pause, carried up like CompletedWithFlags.
type WaveSignatureStop struct {
Terms int
SignaturePath string
}
func (e *WaveSignatureStop) Error() string {
return fmt.Sprintf("bank-mining proposed %d new term(s) — review + sign %s, then resume (the edit wave)", e.Terms, e.SignaturePath)
}
// translateBookWaves is the wave driver (R1). It runs the draft wave (draft ∥), the bank-mining stop, the edit wave (edit ∥) and
// assembles the BookResult. chunks + stickySel come from the precompute pass (SplitChunks + precomputeSticky), computed by
// the caller (TranslateBook) after ingest/seed/eager-build. Returns a *WaveSignatureStop when the bank-mining stop stops.
func (r *Runner) translateBookWaves(ctx context.Context, chunks []Chunk, stickySel []memorySelection) (*BookResult, error) {
draftStages := r.waveStagesIndexed(waveDraft)
editStages := r.waveStagesIndexed(waveEdit)
workers := r.Pipeline.Waves.Workers
editWave := len(editStages) > 0 // a real editor wave exists; else the draft IS the shipping output (draft-only)
// --- the draft wave: draft stage(s) ∥ over draft chunks under draft-wave snapshot (base-bank version) ---
draftSnapshot, draftPayload, err := r.snapshotIDForWave(waveDraft)
if err != nil {
return nil, err
}
if err := r.Store.UpsertSnapshot(draftSnapshot, r.Book.BriefHash(), draftPayload); err != nil {
return nil, fmt.Errorf("pipeline: upsert draft-wave snapshot %.12s: %w", draftSnapshot, err)
}
r.Log.InfoContext(ctx, "draft wave started", "snapshot", draftSnapshot[:12],
"chunks", len(chunks), "workers", workers, "draft_stages", len(draftStages))
draftResults := make([]stageSeqResult, len(chunks))
if err := r.runWave(ctx, workers, len(chunks), func(ctx context.Context, i int) error {
d, err := r.runDraftChunk(ctx, draftSnapshot, chunks[i], stickySel[i], draftStages, !editWave)
if err != nil {
return err
}
draftResults[i] = d
return nil
}); err != nil {
return nil, err
}
// --- the bank-mining stop: bank-mining stop boundary (auto-continues when mining is not configured) ---
if stopped, err := r.runBankMiningStop(ctx, chunks); err != nil {
return nil, err
} else if stopped {
return nil, &WaveSignatureStop{Terms: r.lastMinedCount, SignaturePath: r.signatureMapPath()}
}
res := &BookResult{BookID: r.Book.BookID}
// --- Draft-only pipeline: the draft IS the shipping output; assemble per draft chunk (no edit units) ---
if !editWave {
for i, ch := range chunks {
oc := draftOnlyOutcome(ch, draftResults[i])
res.Chunks = append(res.Chunks, oc)
res.TotalUSD += oc.CostUSD
if oc.Disposition == DispFlagged {
res.Flagged++
}
}
r.Log.InfoContext(ctx, "book run finished (draft-only)", "book", r.Book.BookID,
"chunks", len(res.Chunks), "flagged", res.Flagged, "run_usd", fmt.Sprintf("%.6f", res.TotalUSD))
return res, nil
}
// --- the edit wave: edit stage(s) ∥ over edit units under edit-wave snapshot (enriched-bank version) ---
editSnapshot, editPayload, err := r.snapshotIDForWave(waveEdit)
if err != nil {
return nil, err
}
if err := r.Store.UpsertSnapshot(editSnapshot, r.Book.BriefHash(), editPayload); err != nil {
return nil, fmt.Errorf("pipeline: upsert edit-wave snapshot %.12s: %w", editSnapshot, err)
}
units := buildEditUnits(chunks)
draftByKey := make(map[chunkKey]stageSeqResult, len(chunks))
for i, ch := range chunks {
draftByKey[chunkKey{ch.Chapter, ch.ChunkIdx}] = draftResults[i]
}
r.Log.InfoContext(ctx, "edit wave started", "snapshot", editSnapshot[:12],
"units", len(units), "workers", workers, "edit_stages", len(editStages))
unitOutcomes := make([]*ChunkOutcome, len(units))
if err := r.runWave(ctx, workers, len(units), func(ctx context.Context, i int) error {
oc, err := r.runEditUnit(ctx, editSnapshot, units[i], draftByKey, editStages)
if err != nil {
return err
}
unitOutcomes[i] = oc
return nil
}); err != nil {
return nil, err
}
for _, oc := range unitOutcomes {
res.Chunks = append(res.Chunks, *oc)
res.TotalUSD += oc.CostUSD
if oc.Disposition == DispFlagged {
res.Flagged++
}
}
r.Log.InfoContext(ctx, "book run finished", "book", r.Book.BookID,
"units", len(res.Chunks), "flagged", res.Flagged, "run_usd", fmt.Sprintf("%.6f", res.TotalUSD))
return res, nil
}
// runWave fans `n` items out over `workers` goroutines, calling work(ctx, i) for each index. The first
// error cancels the rest (via a derived context) and is returned; work items are independent (a wave has no
// shared mutable input), and money/store writes serialize through the single-writer pool, so this is
// race-clean. Results are written by the callback into a caller-owned slice indexed by i, so the result
// ORDER is deterministic (item index) regardless of completion order — the golden capture sorts the wire/log
// side-effects separately. workers ≤ 0 is treated as 1 (LoadPipeline already clamps).
func (r *Runner) runWave(parent context.Context, workers, n int, work func(ctx context.Context, i int) error) error {
if workers < 1 {
workers = 1
}
if n == 0 {
return nil
}
ctx, cancel := context.WithCancel(parent)
defer cancel()
idxCh := make(chan int)
var wg sync.WaitGroup
var mu sync.Mutex
var firstErr error
fail := func(err error) {
mu.Lock()
if firstErr == nil {
firstErr = err
cancel()
}
mu.Unlock()
}
for w := 0; w < workers; w++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := range idxCh {
if err := work(ctx, i); err != nil {
fail(err)
return
}
}
}()
}
for i := 0; i < n; i++ {
select {
case idxCh <- i:
case <-ctx.Done():
}
}
close(idxCh)
wg.Wait()
return firstErr
}
// stageSeqResult is the outcome of running one chunk/unit through an ORDERED list of stages (one wave):
// the per-stage results, the final OK stage's text, and the terminal flag state.
type stageSeqResult struct {
stages []StageResult
finalText string // the last OK stage's Text (empty until a stage runs; unused when flagged)
flagged bool
flagReason FlagReason
recovered string // sanitizer-stripped export text from the flagging stage (D35.4a)
cost float64 // THIS run's spend across the sequence
bankTelem bankFlags // the translator-role stage's banknote telemetry (WS4 point 10)
}
// runStageSequence runs `staged` in order over one chunk/unit, feeding each stage's output to the next as
// `prev` (starting from startPrev), stopping the sequence at the FIRST flagged stage (later stages are
// recorded `skipped` — no paid edit over a garbage draft, D2). It is the generalized inner loop of the
// retired translateChunk, reused by BOTH waves (draft over a chunk, the edit-wave editor over a unit). injectionByRole
// maps a stage role to its already-rendered memory-injection message (the translator's src→dst glossary in
// the draft wave, the editor's CONFIRMED-dst constraint block in the edit wave). Deterministic; content-verified resume via runStage.
func (r *Runner) runStageSequence(ctx context.Context, staged []wavedStage, snapID string, ch Chunk, startPrev string, injectionByRole map[string]string) (stageSeqResult, error) {
var res stageSeqResult
prev := startPrev
flagged := false
var flagReason FlagReason
recovered := ""
for _, ws := range staged {
st := ws.st
if flagged {
detail := fmt.Sprintf("skipped: an upstream stage was flagged (%s)", flagReason)
if err := r.Store.UpsertChunkStatus(store.ChunkStatus{
BookID: r.Book.BookID, Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Stage: st.Name,
SnapshotID: snapID, Disposition: string(DispSkipped), FlagReason: string(flagReason), Detail: detail,
}); err != nil {
return res, fmt.Errorf("pipeline: record skipped chunk_status: %w", err)
}
res.stages = append(res.stages, StageResult{
Stage: st.Name, Role: st.Role, Model: st.Model,
Disposition: DispSkipped, FlagReason: flagReason, Detail: detail,
})
continue
}
sr, err := r.runStage(ctx, st, ws.idx, snapID, ch, prev, injectionByRole[st.Role])
if err != nil {
return res, err
}
res.stages = append(res.stages, *sr)
res.cost += sr.CostUSD
if st.Role == roleTranslator {
res.bankTelem = sr.BankFlags
// FL-2: the resume fast-path serves the banknote-cleaned final_hash and leaves BankFlags zero;
// restore the telemetry the fresh run persisted so persistRetrievalState does not zero the three
// banknote columns on every resume of a ready chunk (guarded on an empty BankFlags so the
// attempt-loop path — which re-derives telemetry from the raw checkpoint — is never overwritten).
if res.bankTelem == (bankFlags{}) && sr.FromResume && r.Pipeline.Gates.Banknote.Enabled {
if prevRS, err := r.Store.GetRetrievalState(r.Book.BookID, ch.Chapter, ch.ChunkIdx); err == nil && prevRS != nil {
res.bankTelem = bankFlags{
NLines: prevRS.NBanknoteLines, ParseFail: prevRS.BanknoteParseFail != 0, Truncated: prevRS.BanknoteTruncated != 0,
}
}
}
}
if sr.Disposition == DispFlagged {
flagged = true
flagReason = sr.FlagReason
recovered = sr.RecoveredText
continue
}
prev = sr.Text
}
res.finalText = prev
res.flagged = flagged
res.flagReason = flagReason
res.recovered = recovered
return res, nil
}
// runDraftChunk runs the draft stage(s) over one chunk under the draft-wave snapshot (plan §1). It renders
// the translator's src→dst glossary injection from the precomputed per-chunk selection over the BASE bank
// (baseMemory — mined-excluded, so the injection is stable across a bank-mining enrichment), runs the
// sequence, and persists the draft retrieval_state (injection counts + banknote telemetry). When this is the
// FINAL wave (a draft-only pipeline with no editor), the draft IS the shipping text, so the post-check +
// cheap gates run here over the draft; otherwise they belong to the edit wave (the edit unit's final output).
func (r *Runner) runDraftChunk(ctx context.Context, draftSnapshot string, ch Chunk, memSel memorySelection, draftStages []wavedStage, isFinalWave bool) (stageSeqResult, error) {
injectionByRole := map[string]string{}
if r.baseMemory != nil {
// Serialize the per-role injection blocks via the role→renderer registry (D39 слой 7) from the
// precomputed base-bank selection: the translator gets its src→dst glossary block; any other
// role's block is rendered too but consumed only if a stage of that role runs in this wave.
for role, render := range roleInjectionRenderers {
injectionByRole[role] = render(memSel.injected) // pure → map-order-independent
}
if len(memSel.trustGated) > 0 {
r.Log.WarnContext(ctx, "memory: lower-trust longer key refused from suppressing a higher-trust nested key (approved term preserved; reconcile the seed)",
"chapter", ch.Chapter, "chunk", ch.ChunkIdx, "trust_gated", len(memSel.trustGated),
"first", memSel.trustGated[0].Suppressor+"⊃"+memSel.trustGated[0].Protected)
}
}
seq, err := r.runStageSequence(ctx, draftStages, draftSnapshot, ch, "", injectionByRole)
if err != nil {
return seq, err
}
var postMisses []postcheckMiss
outputChecked := false
var cheap cheapGateResult
if isFinalWave && r.baseMemory != nil && !seq.flagged && seq.finalText != "" {
// Draft-only pipeline: the draft is the shipping output → run the FINAL-output checks here, over the
// SAME base bank whose terms the draft was injected with (post-check parity with the injection).
outputChecked = true
postMisses = r.baseMemory.postcheck(memSel.injected, seq.finalText)
if r.Pipeline.Gates.Glossary.PostcheckGate && countConfirmedMisses(postMisses) > 0 {
seq.flagged = true
seq.flagReason = FlagGlossaryMiss
r.Log.WarnContext(ctx, "glossary post-check gate flagged the chunk",
"chapter", ch.Chapter, "chunk", ch.ChunkIdx, "confirmed_misses", countConfirmedMisses(postMisses))
}
}
if isFinalWave && !seq.flagged && seq.finalText != "" && isRuTarget(r.Book.TargetLang) {
cheap = runCheapGates(ch.Text, seq.finalText, seq.finalText, r.cheapGateConfig())
}
if r.baseMemory != nil {
if err := r.persistRetrievalState(draftSnapshot, ch, memSel, postMisses, outputChecked, cheap, seq.bankTelem); err != nil {
return seq, err
}
}
return seq, nil
}
// runEditUnit runs the editor stage(s) over one edit unit under the edit-wave snapshot (plan §1/§2). It reads
// the unit's draft as the concatenation of its member chunks' draft outputs (NEVER re-rendering the draft
// stage — the несущий invariant), does a FRESH memory.Select over the WHOLE-unit source over the ENRICHED
// bank (sticky degenerate at unit scope → nil), and runs the editor over (unit source, unit draft). A unit whose
// ANY member draft flagged is flagged (skip edit — no paid edit over a partial draft, D2 flag+skip); the
// good members' draft-wave spend stays honestly committed. Post-check + cheap gates run over the edit output at
// unit granularity, merged into the LEADER chunk's retrieval_state row.
func (r *Runner) runEditUnit(ctx context.Context, editSnapshot string, unit editUnit, draftByKey map[chunkKey]stageSeqResult, editStages []wavedStage) (*ChunkOutcome, error) {
leader := Chunk{Chapter: unit.Chapter, ChunkIdx: unit.FirstChunkIdx, Text: unit.sourceText()}
out := &ChunkOutcome{Chapter: unit.Chapter, ChunkIdx: unit.FirstChunkIdx, Disposition: DispOK}
var draftStages []StageResult
var draftCost float64
var draftParts []string
memberFlagged := false
var memberFlagReason FlagReason
memberRecovered := ""
for _, m := range unit.Members {
d, ok := draftByKey[chunkKey{m.Chapter, m.ChunkIdx}]
if !ok {
return nil, fmt.Errorf("pipeline: edit unit ch%d chunk%d: missing draft result (wave sequencing bug)", m.Chapter, m.ChunkIdx)
}
draftStages = append(draftStages, d.stages...)
draftCost += d.cost
if d.flagged {
if !memberFlagged {
memberFlagged = true
memberFlagReason = d.flagReason
memberRecovered = d.recovered
}
continue
}
draftParts = append(draftParts, d.finalText)
}
out.CostUSD = draftCost
if memberFlagged {
// A member draft flagged → the unit's edit is skipped (no paid edit over a partial draft). Record a
// skipped edit chunk_status at the leader and export the recovered draft (or "" for a dropped flag).
skipped, err := r.recordSkippedStages(ctx, editStages, editSnapshot, leader, memberFlagReason)
if err != nil {
return nil, err
}
out.Stages = append(draftStages, skipped...)
out.Disposition = DispFlagged
out.FlagReason = memberFlagReason
out.FinalText = exportNormalize(memberRecovered)
return out, nil
}
unitDraft := strings.Join(draftParts, unitJoinSeparator)
var editSel memorySelection
injectionByRole := map[string]string{}
if r.memory != nil {
// FRESH Select over the WHOLE-unit source over the ENRICHED bank (§1(б)/F3): NOT the the precompute pass per-chunk
// memSel (that was over the base bank). Sticky is degenerate at unit scope (a unit == a chapter or a
// sub-chapter split, so the intra-chapter sticky window does not apply) → nil sticky_prev.
editSel = r.memory.Select(leader.Text, unit.Chapter, nil, r.Pipeline.Context.GlossaryTokenBudget)
// Render the per-role injection from the ENRICHED unit selection via the registry (D39 слой 7): the
// editor gets its CONFIRMED-dst constraint block; other roles' blocks are rendered but consumed
// only if a stage of that role runs in the edit wave.
for role, render := range roleInjectionRenderers {
injectionByRole[role] = render(editSel.injected)
}
}
seq, err := r.runStageSequence(ctx, editStages, editSnapshot, leader, unitDraft, injectionByRole)
if err != nil {
return nil, err
}
out.Stages = append(draftStages, seq.stages...)
out.CostUSD += seq.cost
finalText, flagged, flagReason, recovered := seq.finalText, seq.flagged, seq.flagReason, seq.recovered
var postMisses []postcheckMiss
outputChecked := false
if r.memory != nil && !flagged && finalText != "" {
outputChecked = true
postMisses = r.memory.postcheck(editSel.injected, finalText)
if r.Pipeline.Gates.Glossary.PostcheckGate && countConfirmedMisses(postMisses) > 0 {
flagged = true
flagReason = FlagGlossaryMiss
r.Log.WarnContext(ctx, "glossary post-check gate flagged the edit unit",
"chapter", unit.Chapter, "leader_chunk", unit.FirstChunkIdx, "confirmed_misses", countConfirmedMisses(postMisses))
}
}
var cheap cheapGateResult
if !flagged && finalText != "" && isRuTarget(r.Book.TargetLang) {
cheap = runCheapGates(leader.Text, unitDraft, finalText, r.cheapGateConfig())
if cheap.total() > 0 {
r.Log.InfoContext(ctx, "cheap style gates flagged the edit unit (observability, not a gate)",
"chapter", unit.Chapter, "leader_chunk", unit.FirstChunkIdx, "style_flags", cheap.total())
}
}
if r.memory != nil {
if err := r.mergeUnitRetrievalState(unit.Chapter, unit.FirstChunkIdx, postMisses, outputChecked, cheap); err != nil {
return nil, err
}
}
if flagged {
out.Disposition = DispFlagged
out.FlagReason = flagReason
out.FinalText = exportNormalize(recovered)
} else {
out.FinalText = exportNormalize(finalText)
}
return out, nil
}
// recordSkippedStages writes a `skipped` chunk_status row (+ a skipped StageResult) for each stage in a
// wave, at chunk `ch` under `snapID` — used when a unit's member draft flagged, so the unit's edit is
// skipped (mirrors runStageSequence's flagged-skip path but for a pre-decided skip).
func (r *Runner) recordSkippedStages(ctx context.Context, staged []wavedStage, snapID string, ch Chunk, flagReason FlagReason) ([]StageResult, error) {
detail := fmt.Sprintf("skipped: a member draft chunk of this edit unit was flagged (%s)", flagReason)
var out []StageResult
for _, ws := range staged {
st := ws.st
if err := r.Store.UpsertChunkStatus(store.ChunkStatus{
BookID: r.Book.BookID, Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Stage: st.Name,
SnapshotID: snapID, Disposition: string(DispSkipped), FlagReason: string(flagReason), Detail: detail,
}); err != nil {
return nil, fmt.Errorf("pipeline: record skipped edit chunk_status: %w", err)
}
out = append(out, StageResult{
Stage: st.Name, Role: st.Role, Model: st.Model,
Disposition: DispSkipped, FlagReason: flagReason, Detail: detail,
})
}
return out, nil
}
// mergeUnitRetrievalState folds a edit unit's post-check + cheap-gate observability into the LEADER chunk's
// retrieval_state row — the row the draft wave already wrote with the leader chunk's draft-injection counts + banknote
// telemetry. A read-modify-write (single-writer safe; each unit owns a distinct leader chunk_idx) preserves
// the draft fields and adds the unit-level post-check/style fields, so the read-models aggregate injection
// per draft chunk and post-check/style per unit. A missing row (r.memory nil path) is a no-op.
func (r *Runner) mergeUnitRetrievalState(chapter, firstChunkIdx int, misses []postcheckMiss, outputChecked bool, cheap cheapGateResult) error {
rs, err := r.Store.GetRetrievalState(r.Book.BookID, chapter, firstChunkIdx)
if err != nil {
return fmt.Errorf("pipeline: read leader retrieval_state ch%d/chunk%d: %w", chapter, firstChunkIdx, err)
}
if rs == nil {
return nil // no draft row (no memory / no draft) → nothing to merge onto
}
rs.NStyleFlags = cheap.total()
rs.StyleDetail = ""
if cheap.total() > 0 {
if b, err := json.Marshal(cheap); err == nil {
rs.StyleDetail = string(b)
}
}
rs.NPostcheckMiss = 0
rs.PostcheckDetail = ""
if outputChecked {
rs.NPostcheckMiss = countConfirmedMisses(misses)
if len(misses) > 0 {
if b, err := json.Marshal(misses); err == nil {
rs.PostcheckDetail = string(b)
}
}
}
return r.Store.UpsertRetrievalState(*rs)
}
// draftOnlyOutcome assembles a ChunkOutcome for a draft-only pipeline (no editor): the draft is the shipping
// output, so the per-chunk draft result maps straight to the outcome (post-check/cheap already ran in
// runDraftChunk as the final wave). Mirrors the export contract: a cosmetic strip exports its recovered
// text; every other flag exports "".
func draftOnlyOutcome(ch Chunk, d stageSeqResult) ChunkOutcome {
oc := ChunkOutcome{Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Stages: d.stages, CostUSD: d.cost, Disposition: DispOK}
if d.flagged {
oc.Disposition = DispFlagged
oc.FlagReason = d.flagReason
oc.FinalText = exportNormalize(d.recovered)
} else {
oc.FinalText = exportNormalize(d.finalText)
}
return oc
}

368
backend/internal/pipeline/waverun_test.go Normal file
View file

@ -0,0 +1,368 @@
package pipeline
import (
"context"
"errors"
"math"
"os"
"path/filepath"
"strings"
"testing"
"textmachine/backend/internal/obs"
)
// waverun_test.go: the R1 wave-driver behaviours the 1-chunk-per-chapter fixtures cannot exercise — a
// MULTI-CHUNK edit unit (the editor collapses several draft chunks into one edit call over their
// concatenation, addressed by the leader), parallel-worker money conservation, a flagged member draft
// skipping the unit's edit, and the bank-mining stop gate.
// cjkSource builds a single-chapter ja source of `paras` paragraphs, each `hanPerPara` Han chars + a full
// stop, separated by a blank line — enough to make the chunker split it into several draft chunks.
func cjkSource(paras, hanPerPara int) string {
p := make([]string, paras)
for i := range p {
p[i] = strings.Repeat("文", hanPerPara) + "。"
}
return strings.Join(p, "\n\n")
}
// TestWaveMultiChunkEditUnit is the load-bearing R1 pin: a chapter that splits into TWO draft chunks but
// groups into ONE edit unit. The editor must run ONCE over the concatenation of the two members' drafts
// (never re-rendering either draft), the unit's outcome is a single ChunkOutcome at the leader, and export
// projects one row — not two "pending" chunks.
func TestWaveMultiChunkEditUnit(t *testing.T) {
rec := &reqRec{}
srv := newJSONProvider(rec, draftEdit)
defer srv.Close()
// para1 est_out ≈ 1677 (≤ draft budget 1797 → its own chunk); para2 est_out ≈ 1318 forces a second
// chunk; together ≈ 2995 ≤ edit ceiling 3200 → ONE edit unit with both members.
bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: cjkSource(1, 1400) + "\n\n" + strings.Repeat("字", 1100) + "。", regenerate: 0})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
// Confirm the fixture actually produces the 2-chunk / 1-unit shape the test targets.
r0 := newRunner(t, bookPath)
manifest, err := r0.bookChunks()
r0.Close()
if err != nil {
t.Fatal(err)
}
units := buildEditUnits(manifest)
if len(manifest) != 2 || len(units) != 1 || len(units[0].Members) != 2 {
t.Fatalf("fixture must be 2 draft chunks in 1 edit unit, got %d chunks / %d units", len(manifest), len(units))
}
r1 := newRunner(t, bookPath)
res, err := r1.TranslateBook(ctx)
if err != nil {
t.Fatal(err)
}
r1.Close()
// One outcome PER UNIT, at the leader chunk, carrying both member drafts + the single edit stage.
if len(res.Chunks) != 1 {
t.Fatalf("a 2-chunk unit must yield ONE unit outcome, got %d", len(res.Chunks))
}
oc := res.Chunks[0]
if oc.ChunkIdx != 0 {
t.Fatalf("the unit outcome must address the leader chunk (0), got %d", oc.ChunkIdx)
}
if len(oc.Stages) != 3 {
t.Fatalf("the unit outcome must carry 2 member drafts + 1 edit, got %d stages", len(oc.Stages))
}
if oc.Stages[0].Role != roleTranslator || oc.Stages[1].Role != roleTranslator || oc.Stages[2].Role != roleEditor {
t.Fatalf("stages must be [draft, draft, edit], got %s/%s/%s", oc.Stages[0].Role, oc.Stages[1].Role, oc.Stages[2].Role)
}
if oc.FinalText != "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД" {
t.Fatalf("the unit's final text is the single edit output, got %q", oc.FinalText)
}
// EXACTLY 3 provider calls: 2 drafts + 1 edit — the editor did NOT re-render either draft.
if rec.count() != 3 {
t.Fatalf("multi-chunk unit must be 2 draft + 1 edit calls, got %d", rec.count())
}
// The editor's request carried the CONCATENATION of both members' drafts (joined by the unit separator).
var editBody string
for _, b := range rec.all() {
if isEditBody(b) {
editBody = b
}
}
// Both members' drafts appear in the editor's input (the body JSON-escapes the join, so count the
// draft marker rather than match the raw separator).
if n := strings.Count(editBody, "ЧЕРНОВИК ПЕРЕВОДА"); n != 2 {
t.Fatalf("the editor must read BOTH member drafts concatenated, found %d in the edit body", n)
}
// Export projects ONE unit row (not 2, and not a phantom "pending" for the non-leader member).
r2 := newRunner(t, bookPath)
exp, err := r2.Export(true)
if err != nil {
t.Fatal(err)
}
if exp.TotalChunks != 1 || exp.PendingChunks != 0 || len(exp.Chunks) != 1 {
t.Fatalf("export must be 1 unit / 0 pending, got total=%d pending=%d rows=%d", exp.TotalChunks, exp.PendingChunks, len(exp.Chunks))
}
if exp.Chunks[0].FinalText != "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД" {
t.Fatalf("export unit text mismatch: %q", exp.Chunks[0].FinalText)
}
// --pairs source is the members' joined source (the src↔target pair aligns to the editor's input).
if !strings.Contains(exp.Chunks[0].Source, unitJoinSeparator) {
t.Fatalf("export --pairs source must be the joined unit source, got %q", exp.Chunks[0].Source[:min(40, len(exp.Chunks[0].Source))])
}
// Status counts the unit as ONE done chunk (not two).
st, err := r2.Status(ctx)
if err != nil {
t.Fatal(err)
}
if st.TotalChunks != 1 || st.Done != 1 {
t.Fatalf("status must be 1 unit done, got total=%d done=%d", st.TotalChunks, st.Done)
}
r2.Close()
// Resume is $0 (all checkpoints hit; the editor re-derives its content-addressed id for free).
r3 := newRunner(t, bookPath)
defer r3.Close()
before := rec.count()
res2, err := r3.TranslateBook(ctx)
if err != nil {
t.Fatal(err)
}
if rec.count() != before || res2.TotalUSD != 0 {
t.Fatalf("resume must be $0 with no new calls: calls=%d usd=%v", rec.count()-before, res2.TotalUSD)
}
}
// TestWaveParallelWorkersMoneyConserved runs the wave driver with several parallel workers over a book of
// many chunks and asserts the money invariant survives concurrency: committed spend == the sum of the
// settled checkpoints (the single-writer store serializes Reserve/Settle), and the per-unit result is
// complete + deterministic. Run under `-race` this also proves the waves share only read-only state.
func TestWaveParallelWorkersMoneyConserved(t *testing.T) {
rec := &reqRec{}
srv := newJSONProvider(rec, draftEdit)
defer srv.Close()
// 5 paragraphs of ~1400 Han each → 5 draft chunks, and since any two exceed the 3200 edit ceiling,
// 5 edit units → 10 wave work items fanned over 4 workers.
bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: cjkSource(5, 1400), regenerate: 0, waveWorkers: 4, bookUSD: 100})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r1 := newRunner(t, bookPath)
res, err := r1.TranslateBook(ctx)
if err != nil {
t.Fatal(err)
}
if len(res.Chunks) != 5 {
t.Fatalf("5 chapters-worth of single-chunk units expected, got %d", len(res.Chunks))
}
for _, oc := range res.Chunks {
if oc.Disposition != DispOK || oc.FinalText != "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД" {
t.Fatalf("every parallel unit must be ok+edited, got %s / %q", oc.Disposition, oc.FinalText)
}
}
// Money invariant under parallelism: the ledger's committed spend matches the run's own aggregate and
// no reservation leaked — the single-writer store serialized every Reserve/Settle despite N workers
// (the store-level atomicity + kill-9 durability is separately pinned by store/kill9_test.go).
committed, reserved, err := r1.Store.SpentUSD("test-book")
if err != nil {
t.Fatal(err)
}
r1.Close()
if reserved != 0 {
t.Fatalf("no reservation may leak after a clean parallel run, reserved=%v", reserved)
}
if math.Abs(res.TotalUSD-committed) > 1e-9 {
t.Fatalf("run total %v != ledger committed %v under parallel workers — a settle raced/leaked", res.TotalUSD, committed)
}
}
// TestWaveEditUnitFlaggedMemberDraft: when ONE member draft of a multi-chunk unit flags, the whole unit is
// flagged and its edit is SKIPPED (no paid edit over a partial draft), while the good member's draft money
// stays committed. There is exactly ONE edit call fewer than a clean run.
func TestWaveEditUnitFlaggedMemberDraft(t *testing.T) {
rec := &reqRec{}
// The second paragraph carries a marker; the mock returns an untranslated CJK echo for it → cjk_artifact.
const echoMarker = "禁"
respond := func(body string) (string, string) {
if isEditBody(body) {
return "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД", "stop"
}
if strings.Contains(body, echoMarker) {
return "这是完全没有翻译的中文内容。", "stop" // fully-CJK → classify flags cjk_artifact
}
return "ЧЕРНОВИК ПЕРЕВОДА", "stop"
}
srv := newJSONProvider(rec, respond)
defer srv.Close()
src := strings.Repeat("文", 1400) + "。\n\n" + strings.Repeat(echoMarker, 1100) + "。"
bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: src, regenerate: 0})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r := newRunner(t, bookPath)
defer r.Close()
res, err := r.TranslateBook(ctx)
if err != nil {
t.Fatal(err)
}
if len(res.Chunks) != 1 {
t.Fatalf("still one unit outcome, got %d", len(res.Chunks))
}
oc := res.Chunks[0]
if oc.Disposition != DispFlagged || oc.FlagReason != FlagCJKArtifact {
t.Fatalf("a flagged member draft must flag the unit (cjk_artifact), got %s/%s", oc.Disposition, oc.FlagReason)
}
if oc.FinalText != "" {
t.Fatalf("a flagged unit exports nothing, got %q", oc.FinalText)
}
// 2 draft calls (one ok, one echo), ZERO edit calls — the unit's edit was skipped over the partial draft.
if rec.count() != 2 {
t.Fatalf("a partial-draft unit runs 2 drafts + 0 edit, got %d calls", rec.count())
}
}
// TestWaveEscalationBudgetSerializedUnderParallelism pins the escMu: two draft chunks escalate
// CONCURRENTLY under a budget that admits only ONE hop. The escalation soft-cap is a non-atomic
// read-then-act over EscalationSpentUSD, so without escMu both parallel workers could read spent<budget and
// both be admitted (overshoot by a whole hop). escMu serializes the admission through the paid settle, so
// exactly one hop fires — the same guarantee the sequential TestRunnerEscalationBudgetExhaustionDeniesLaterHop
// gives, now under parallel workers. (Run under -race this also proves the shared spend path is race-clean.)
func TestWaveEscalationBudgetSerializedUnderParallelism(t *testing.T) {
rec := &reqRec{}
srv := newJSONProvider(rec, echoOrClean) // primary echoes (cjk_artifact); fake-fallback cleans
defer srv.Close()
bookPath := setupTwoChapterEscalation(t, srv.URL, 0.001) // budget 0.001 < one hop (≈0.00182)
// Bump to parallel workers so both chapters' drafts escalate at once.
pipePath := filepath.Join(filepath.Dir(bookPath), "pipeline.yaml")
raw, err := os.ReadFile(pipePath)
if err != nil {
t.Fatal(err)
}
writeFile(t, pipePath, strings.Replace(string(raw), "core: C1", "core: C1\nwaves: { workers: 4 }", 1))
r := newRunner(t, bookPath)
defer r.Close()
res, err := r.TranslateBook(context.Background())
if err != nil {
t.Fatal(err)
}
esc, err := r.Store.EscalationSpentUSD("test-book")
if err != nil {
t.Fatal(err)
}
if diff := esc - fakeCallUSD; diff > 1e-12 || diff < -1e-12 {
t.Fatalf("escMu must serialize escalation admission under parallel workers: want exactly one hop (%.6f), got %.6f", fakeCallUSD, esc)
}
// Exactly one unit escalated-OK and one stayed flagged (WHICH one wins the single hop is
// scheduling-dependent, so assert the totals, not the identity).
escalatedOK, flagged := 0, 0
for _, oc := range res.Chunks {
if oc.Disposition == DispOK && len(oc.Stages) > 0 && oc.Stages[0].Escalated {
escalatedOK++
}
if oc.Disposition == DispFlagged {
flagged++
}
}
if escalatedOK != 1 || flagged != 1 {
t.Fatalf("exactly one unit escalates-OK and one stays flagged, got ok=%d flagged=%d", escalatedOK, flagged)
}
}
// TestWaveMinedSignDoesNotRebillDraft is the «переоплата ОДНА» pin AT THE INJECTION LEVEL (the checkpoint-
// review found the split was implemented only at the version-hash level): signing a mined term and re-running
// must move ONLY the edit wave, leaving the draft wave's checkpoints byte-stable. The draft selects over the
// BASE bank (mined-excluded), so a mined addition does NOT change the draft's injected wire → its content_hash
// / final_hash / snapshot are unchanged → the draft resumes at $0. The editor selects over the ENRICHED bank,
// so the mined term DOES move the edit-wave snapshot (the single, --resnapshot-gated overpay). Reverting the
// fix (draft over the enriched bank) makes the draft content_hash change here → the test fails.
func TestWaveMinedSignDoesNotRebillDraft(t *testing.T) {
rec := &reqRec{}
srv := newJSONProvider(rec, draftEdit)
defer srv.Close()
// Source carries a BASE seed term (魔法学院) and a to-be-mined term (方源); both are ≥2-char Han keys.
src := "方源走进了魔法学院的图书馆。"
seed := "terms:\n - src: 魔法学院\n dst: Академия магии\n status: approved\n"
bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: src, glossarySeed: seed, regenerate: 0})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r1 := newRunner(t, bookPath)
if _, err := r1.TranslateBook(ctx); err != nil {
t.Fatal(err)
}
draftBefore, err := r1.Store.GetChunkStatus("test-book", 1, 0, "draft")
if err != nil || draftBefore == nil {
t.Fatalf("draft chunk_status after run 1: %v / %v", draftBefore, err)
}
editBefore, err := r1.Store.GetChunkStatus("test-book", 1, 0, "edit")
if err != nil || editBefore == nil {
t.Fatalf("edit chunk_status after run 1: %v / %v", editBefore, err)
}
r1.Close()
// Owner signs 方源 → Фан Юань (approved) into a mined-delta file (loaded as Source:mined) and re-runs
// with --resnapshot (the edit wave's enriched snapshot legitimately moves; the draft's must not).
dir := filepath.Dir(bookPath)
writeFile(t, filepath.Join(dir, "mined-delta.yaml"), "terms:\n - src: 方源\n dst: Фан Юань\n status: approved\n")
rawBook, err := os.ReadFile(bookPath)
if err != nil {
t.Fatal(err)
}
writeFile(t, bookPath, strings.Replace(string(rawBook), "pipeline: pipeline.yaml", "pipeline: pipeline.yaml\nmined_delta: mined-delta.yaml", 1))
r2 := newRunner(t, bookPath)
defer r2.Close()
r2.Resnapshot = true
if _, err := r2.TranslateBook(ctx); err != nil {
t.Fatal(err)
}
draftAfter, err := r2.Store.GetChunkStatus("test-book", 1, 0, "draft")
if err != nil || draftAfter == nil {
t.Fatalf("draft chunk_status after run 2: %v / %v", draftAfter, err)
}
editAfter, err := r2.Store.GetChunkStatus("test-book", 1, 0, "edit")
if err != nil || editAfter == nil {
t.Fatalf("edit chunk_status after run 2: %v / %v", editAfter, err)
}
// «переоплата ОДНА»: the DRAFT wave is byte-stable across the mined sign — same snapshot, same rendered
// content, same authoritative checkpoint → it resumed at $0 (a mined term never entered the draft wire).
if draftAfter.SnapshotID != draftBefore.SnapshotID {
t.Fatalf("draft-wave snapshot moved on a mined sign (%.12s → %.12s) — base bank not excluding mined", draftBefore.SnapshotID, draftAfter.SnapshotID)
}
if draftAfter.ContentHash != draftBefore.ContentHash {
t.Fatalf("draft injection CHANGED on a mined sign — the draft selects over the ENRICHED bank (regression): «переоплата ОДНА» broken, the whole draft wave re-bills")
}
if draftAfter.FinalHash != draftBefore.FinalHash {
t.Fatalf("draft checkpoint re-created on a mined sign (%.12s → %.12s) — the draft was re-billed", draftBefore.FinalHash, draftAfter.FinalHash)
}
// The single, gated overpay DID land on the edit wave: the enriched snapshot moved.
if editAfter.SnapshotID == editBefore.SnapshotID {
t.Fatalf("edit-wave snapshot did NOT move on a mined sign — the enriched bank is not folding the mined term")
}
}
// TestWaveMiningUnconfiguredAutoContinues: with no langpack / no contrast (every existing fixture), the
// bank-mining stop is a no-op — the driver runs straight through to the edit wave and writes no signature
// map. Guards that the mining wiring never perturbs a normal $0 run.
func TestWaveMiningUnconfiguredAutoContinues(t *testing.T) {
rec := &reqRec{}
srv := newJSONProvider(rec, draftEdit)
defer srv.Close()
bookPath := setupProjectOpts(t, srv.URL, projectOpts{regenerate: 0})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r := newRunner(t, bookPath)
defer r.Close()
if r.pack != nil {
t.Fatal("no langpack_root → pack must be nil")
}
stopped, err := r.runBankMiningStop(ctx, nil)
if err != nil || stopped {
t.Fatalf("unconfigured mining must auto-continue, got stopped=%v err=%v", stopped, err)
}
if _, err := r.TranslateBook(ctx); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(r.signatureMapPath()); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("no signature map may be written when mining is unconfigured (err=%v)", err)
}
}

4
docs/PROGRESS.md
View file

@ -9,6 +9,10 @@
> - **Ждём от владельца:** тачпойнты exp16 (карта подписи/пол · мини-голд алиасов · precision@30, `books/gu-zhenren/exp16/`, ~30–40 мин — для сид-дельты к пере-прогону) · развилки плана §10 (W1.5-UX · DC5-стих-политика · Edit-ceiling · и др.) · реплика ja→ru (тест общности §B5) · **ре-чек прайса DeepSeek у катовера слагов 24.07** (до него платных deepseek-прогонов нет) · чтение пере-прогона (после R1+resnapshot) · старые висящие: планка запуска (лучше-фана/гибрид/издательский) · билингв-якорь пилота (D25.9-Q1) · контаминация пилот-корпуса (D27.4) · publishable/waiver (D25.1) · FN-bound L3 (D25.4) · юр-пакет · провенанс 12-*-доков.
> - Архивы хроники: `archive/PROGRESS-2026-07-04-10.md` (D31) · `archive/PROGRESS-2026-07-10-13.md` (D39.6-гигиена). Записи ниже — живая эра D39.
## Оркестратор №7 — R1 драйвер-свитч ПРИНЯТ и залендён (D39.17), ПАК-11 ЗАВЕРШЁН, 19.07
Бэкенд-сессия сдала волновой исполнитель (`waverun.go`/`mining.go` нов): precompute→черновик∥→банк-майнинг-стоп→редактура∥; редактор читает concat-черновик по final_hash, НИКОГДА не пере-рендерит draft; `translateChunk` удалён. + wiring майнера (langpack fail-loud/nil + pack.Version-фолд + mined-write) + Ш-2 (unicode.Version) + Ш-1 (masked-diff) + golden-рерайт (8 глав→8 единиц) + read-модели per-unit + именование без W0-W2. **Приёмка: прямая (build/race сам, 8 волновых пинов вкл. кусачий регресс-пин переоплаты-ОДНА, golden байт-стабилен, построчное чтение waverun=план §1/research19 §B3-бис) + ПОЛНЫЙ рубеж-2** (5 линз вкл. явную синк-с-ресёрчем — владелец просил; refute-by-default, $0): **4 находки / 1 refuted / 3 выжили — ВСЕ MINOR/NOTE, 0 CRIT/MAJOR; все несущие линзы (деньги/секвенирование/детерминизм/СИНК/снапшот) ПУСТЫ.** Сессия сама поймала MAJOR (инъекция черновика над enriched вместо base → тихий re-bill при mined-подписи; фикс r.baseMemory + регресс-пин). **Фикс-лист (3, стенд-only mining / трипваер — не блокируют, ГЕЙТ до живого mining пере-прогона):** R1-FL-A (CLI не даёт WaveSignatureStop distinct exit-код → exit 1; коммент/отчёт оверклейм) · R1-FL-B (mining-стоп очищается только при пустой дельте, reject-механизма нет → отклонённый терм livelock'ит стоп; связано с W1.5-UX §10-1) · R1-FL-C (NOTE: x/text/norm своя Unicode-редакция не фолдится — расширить Ш-2). Девиации приняты (сургичный per-wave resnapshot, escMu-сериализация эскалаций). **ПАК-11 ЗАВЕРШЁН** (Block A+continuation+арх-проход+R1); `BACKEND_PLAN11` отработан→архив. **Очередь:** пере-прогон-преп (R1-FL-A/B/C + единый resnapshot D30.9 §8) → пере-прогон 3–10 глав (армы + live mining с подписью) → чтение (планка 2 претензии).
## Оркестратор №7 — подпись карты exp16 + находка «Бай Нинбин = предел DC-3», 19.07
Владелец подписал карту exp16 (`books/gu-zhenren/exp16/signature_map.md`, вне git): **dst-канон принят** (banknote-dst + сидовый dst для Палладий-канон-записей; мусорные co-occ «справочно» — игнор, это доказанный провал WHAT §3.3). Правки владельца: 南疆→«Южные земли» (не транслит), 青丝蛊→«Шёлковый гу», 长生 без изм. (=«бессмертие», авто-dst был брак). Пол: остальные персонажи — male по evidence (ожидает финального «ок» владельца), **白凝冰 (Бай Нинбин) = `hidden`**. **НАХОДКА (предел дефект-класса gender-enforce D39.8):** Бай Нинбин ЛОМАЕТ статическое поле пола — пол МЕНЯЕТСЯ по книге (безгендерный→мужчина→магически женщина) И расщеплён по перспективе (внешне ж / сам себя мыслит мужчиной). Ни `gender`, ни окно `since_ch/until_ch` этого не выражают → **детерминированный DC-3 НЕ трогает (правильно, `hidden`→Ф2/редактор); местоимения рендерит редактор ПО СЦЕНЕ.** Следствия: (1) валидирует D39.16-решение отложить hidden-чекер в Ф2; (2) **требование к Ф2/editor-контексту: нужна НАРРАТИВНАЯ гендер-аннотация (since_ch-смены + перспектива), не булево поле** — вход дизайна Ф2-аннотатора; (3) **ожидание пере-прогона (честно владельцу): его претензия «мечтал/мечтала» по Бай Нинбину — редакторская-на-понимание, детерминированным фиксом НЕ закрывается; зависит от того, поймёт ли редактор сцену (может частично на прямых фазах, не гарантировано на расщеплённых).** Гарантированные детерм-фиксы пере-прогона — чистые male-персонажи + 时辰-юниты + числа + регистр.

2
docs/README.md
View file

@ -16,7 +16,7 @@
- `experiments/` — эмпирика «Полигона»: `00-provider-quirks` (читать перед любым вызовом провайдера), `01-token-calibration`, `02-refusal-benchmark`, `03-local-stand`, `04-editor-quality`, `06-local-extraction`, `07-coverage-precision`, `08-cost-model-v2` (актуальная денежная модель), `09-pilot-protocol` (пилот Ф2.5 + поправки D13), `10-explicit-benchmark` (18+ violence-рука канала B), `11-erotica-benchmark` (erotica по трём парам/регистрам — закрытие D14.4, D22).
- `research/` — фактура исследований 04–05.07: `01–10` базовые, `11-gap-*` добор критиком, `12-*` режимы отказа/отзывы/таксономии (+ два внешних материала с провенанс-шапками), `13` валидация памяти, `14` адаптивная память, `15` голос и состояние (принят, D21), **`16` ридер-IDE (принят с ревью-шапкой, D29)**, **`17` внешняя критика GPT-5.6 (принят с ревью-шапкой, D25)** — у 16/17 читать шапку прежде тела. **`18` рычаги качества (два отчёта, D36)** · **`19` нарезка+когезия+контракт t/e (D39.1)** · **`20` банк-майнинг W1.5 (D39.6)** — у всех ревью-шапки. ⚠ Часть под superseded-баннерами (01/02/03/04/05/09 и gap-1/2/5) — **читай баннер прежде содержимого**.
- `PROGRESS.md` — **журнал** (CURRENT-STATE сверху, ниже хронология; НЕ источник решений).
- **Активные хендофф-промты (пост-D39.16, 19.07):** [`BACKEND_PLAN11_SESSION_PROMPT.md`](BACKEND_PLAN11_SESSION_PROMPT.md) (**СТАРТОВЫЙ — R1-продолжение [драйвер-свитч + FL-фиксы + фолд `pack.Version()`/загрузка langpack при wiring майнера] по плану [`11-implementation-plan.md`](architecture/11-implementation-plan.md); D39.12/14/16**) · [`ORCHESTRATOR_SESSION_PROMPT.md`](ORCHESTRATOR_SESSION_PROMPT.md) (хендофф №5 оркестратора) · [`POLYGON_PACKAGE4_SESSION_PROMPT.md`](POLYGON_PACKAGE4_SESSION_PROMPT.md) (residual пилот/18+/echo). Закрытые — в `archive/prompts/` (свежие: арх-cleanup→D39.16, дизайн-синтез→D39.12, exp16→D39.10, Q4a→D39.9, exp15→D39.7).
- **Активные хендофф-промты (пост-D39.17, 19.07):** [`ORCHESTRATOR_SESSION_PROMPT.md`](ORCHESTRATOR_SESSION_PROMPT.md) (хендофф №5 оркестратора; статус-баннер — пост-D39.17) · [`POLYGON_PACKAGE4_SESSION_PROMPT.md`](POLYGON_PACKAGE4_SESSION_PROMPT.md) (residual пилот/18+/echo). **ПАК-11 ЗАВЕРШЁН** — след. шаг = пере-прогон-преп (3 mining-wiring-фикса R1-FL-A/B/C + единый resnapshot; промт пишет оркестратор). Закрытые — в `archive/prompts/` (свежие: пак-11/R1→D39.17, арх-cleanup→D39.16, дизайн-синтез→D39.12).
- `archive/` — закрытые сессионные промты (только история, инструкции оттуда не исполнять).
## Статус (2026-07-19, пост-D39.16 — стройка пере-прогонного стека)

15
docs/architecture/05-decisions-log.md
View file

File diff suppressed because one or more lines are too long

2
docs/BACKEND_PLAN11_SESSION_PROMPT.md → docs/archive/prompts/BACKEND_PLAN11_SESSION_PROMPT_2026-07-19.md
View file

@ -1,3 +1,5 @@
> **⟶ ОТРАБОТАН ПОЛНОСТЬЮ → пак-11 ЗАВЕРШЁН (Block A D39.13 · continuation D39.14 · арх-проход D39.16 · R1 драйвер-свитч D39.17). Волновой исполнитель залендён, рубеж-2 clean bill. Остаток — 3 MINOR/NOTE mining-wiring-фикса (R1-FL-A/B/C) → пере-прогон-преп. Архивная копия.**
> **⚠ АКТУАЛИЗИРОВАН ПОД R1-ПРОДОЛЖЕНИЕ (19.07, пост-D39.14; предыдущие версии — в git-истории).** Сессия №1
> сдала Block A (`85f5b9e`, D39.13); сессия №2 сдала R2–R5 + R1-аддитив (D39.14). **Осталось: R1 драйвер-свитч +
> 3-пунктовый фикс-лист рубежа-2 (ниже).** Дизайны — `docs/archive/reports/PACK11_CONT_REPORT_2026-07-19.md §R1` + `docs/archive/reports/PACK11_REPORT_2026-07-19.md §Резидуал`