heaven/textmachine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Land the contract-blockers pack: phase progress counters, chapter manifest with cut-tagged unit ids, machine bank-stop table, bank export, and the run ceiling flag, hardened by three review rounds

Browse source
This commit is contained in:
Claude (backend session) 2026-08-09 04:46:38 +03:00
parent 92baed671a
commit 0e69bc16c8
26 changed files with 2819 additions and 80 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
backend/README.md
View file

@ -8,7 +8,7 @@
| Пакет | Что делает | Ключевые файлы |
|---|---|---|
| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3). main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` |
| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) / `manifest` (**$0-производитель персиста манифеста глав/чанков**, строка 100: нужен ДО первого прогона — дерево глав разобранной, но не запущенной книги; `--json` печатает сам документ). Денежный аргумент прогона: `--ceiling-usd` (строка 145) перекрывает книжный `ceilings.book_usd` ТОЛЬКО на этот прогон, в book.yaml не пишется, ноль/отрицательное = отказ запуска. main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` |
| `internal/llm` | OpenAI-совместимый транспорт + retry/backoff, **capability-слой** (budget_field/temperature/reasoning per-модель), провайдеры openai/local (no-proxy)/anthropic (DEPRECATED-референс); `failover.go` удалён паком-17 — маршрутизация лейблами живёт в config/pipeline (канал B, один хоп `chain[0]`, fail-closed) | `httpllm.go`, `capability.go`, `provider_*.go` |
| `internal/ledger` | Цены по usage (вкл. reasoning/cache-поля), `PriceForResponse` по фактической модели | `pricing.go` |
| `internal/store` | SQLite (modernc, CGO-free), цепочка миграций `schema_version` (⚠ ALTER-шаги v8+ не идемпотентны вопреки шапке — бэклог-строка 49а), reserve/settle+checkpoint, chunk_status, глоссарий (+подписной цикл терминолога), ruby, retrieval_state, request_log; `OpenReadOnly` — без flock/миграций/recovery для status/report/export | `ledger.go`, `migrate.go`, `glossary.go`, `store.go` |
@ -21,7 +21,7 @@
| `internal/membank` | Банк памяти v2, memmatch-v4: Aho-Corasick, спойлер-окна, disposition, **trust-gated longest-match** (draft-длиннее НЕ съедает вложенный approved; отказ = громкая телеметрия — терм-дрейф закрыт в КОДЕ, D39.2-T1), post-check, рендереры инъекции, загрузка/валидация сида (`SeedLint`) | `memory.go`, `memseed.go`, `mempostcheck.go` |
| `internal/miner` | Детерминированный оффлайн-майнер банка (WS3, $0): `MineBank` над нормализованными чанками + контраст-корпус → `[]Term`, `DeltaYAML` — сид-дельта на подпись владельца; все пар-данные приходят `*lang.Pack` | `miner.go`, `miner_*.go` |
| `internal/checks` | Детерминированные $0-вердикты над текстом чанка, **target-aware ПО ДАННЫМ (D39.64): `Checkers` компилируется из цель-данных, `TargetActive()`/`TargetScriptNonLatin()` гейтят всё — цель без данных инертна, Go-веток по паре нет**: санитайзер v7 (fold-first, CJK/Hangul-leak, CJK-глосс-whitelist) + `ExportNormalize` (**export-contract слой 6**, no-op без цель-данных), cheap-гейты и DC-чекеры (пар-данные из пака), coverage-гейт эксцизии, regression-guard. Флаг-вокабуляр НЕ импортирует: возвращает факт, диспозицию назначает драйвер | `sanitizer.go`, `cheapgates.go`, `checkers.go`, `coverage.go` |
| `internal/pipeline` | Композит-корень (драйвер): сетап `runner.go` → снапшот `snapshot.go` → сид `seeding.go` → цикл книги `bookrun.go` → волны `waverun.go` → петля чанка `chunkrun.go` → стадия `stagerun.go` → эскалация с ре-гейтом `escalation.go` → resume `resume.go`; classify/disposition; **реестр `roleInjectionRenderers`** (новая роль = данные+рендерер, не правка switch); `export.go`/`quality.go`/`status.go` — read-only проекции (общая derivation member-drop = `memberDrops`) | `bookrun.go`, `waverun.go`, `stagerun.go`, `export.go`, `disposition.go` |
| `internal/pipeline` | Композит-корень (драйвер): сетап `runner.go` → снапшот `snapshot.go` → сид `seeding.go` → цикл книги `bookrun.go` → волны `waverun.go` → петля чанка `chunkrun.go` → стадия `stagerun.go` → эскалация с ре-гейтом `escalation.go` → resume `resume.go`; classify/disposition; **реестр `roleInjectionRenderers`** (новая роль = данные+рендерер, не правка switch); `export.go`/`quality.go`/`status.go` — read-only проекции (общая derivation member-drop = `memberDrops`); **read-out файлы для платформы** (D39.85 — приватный SQLite движка ей закрыт): `manifest.go` (структура глав/чанков + стабильный id главы, строка 100), `bankexport.go` (весь банк тремя статусами, 125), машинная стоп-таблица в `mining.go` (101); все три пишутся через `artifact.go` (write-then-rename — их читают ПОКА идёт прогон) | `bookrun.go`, `waverun.go`, `stagerun.go`, `export.go`, `manifest.go`, `disposition.go` |
| `internal/obs` | trace_id, структурные логи (contextHandler несёт book/chapter/chunk/stage/role из ctx), safego | |
## Инварианты — ЛОМАТЬ НЕЛЬЗЯ (каждый закреплён тестами)
@ -64,6 +64,8 @@ go run ./cmd/tmctl translate --config example/book.yaml # реальные
go run ./cmd/tmctl status --config example/book.yaml --json # $0, N/M+паспорта+деньги, живой прогон ок
go run ./cmd/tmctl report --config example/book.yaml # $0, quality-report (KPI/rates)
go run ./cmd/tmctl export --config example/book.yaml # $0, экспорт-JSON для полигона (--plaintext для человека)
go run ./cmd/tmctl manifest --config example/book.yaml # $0, пере-строить персист манифеста глав/чанков (--json — сам документ)
go run ./cmd/tmctl translate --config example/book.yaml --ceiling-usd 0.5 # потолок ТОЛЬКО на этот прогон, book.yaml не пишется
# live-conformance (реальные провайдеры, платно, вне CI):
set -a; . ./.env; set +a; TM_LIVE=1 go test -tags live -run TestLive -v ./internal/pipeline/
```

2
backend/cmd/tmctl/backup_test.go
View file

@ -65,7 +65,7 @@ func TestFakeTranslatePathCreatesNoBackup(t *testing.T) {
defer srv.Close()
bookPath := setupCLIProject(t, srv.URL)
if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false); err != nil {
if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false, 0); err != nil {
t.Fatalf("fake translate: %v", err)
}
if _, err := os.Stat(filepath.Join(filepath.Dir(bookPath), "backups")); err == nil {

30
backend/cmd/tmctl/invocation.go
View file

@ -30,6 +30,11 @@ type invocation struct {
// «флаг бинарный» (owner) — and deliberately not a book.yaml key: whether a human is available to
// sign right now is a property of the invocation, not of the book.
verifyBank bool
// ceilingUSD is the BOOK ceiling for THIS RUN ONLY (row 145 / D39.110), 0 when the flag was absent.
// It is an invocation property for the same reason verifyBank is: the amount a caller is willing to
// spend on one run belongs to that run, not to the book's data — writing it into book.yaml would make
// a caller's number a permanent record in the engine's config and mix the zones (D39.81/D39.85).
ceilingUSD float64
sel pipeline.RedriveSelector
}
@ -88,7 +93,7 @@ func (v *rebillConsentValue) IsBoolFlag() bool { return true }
// main()'s default branch → exit 1, leaving 2 exclusive to flagged chunks.
func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
if len(args) < 1 {
return invocation{}, fmt.Errorf("usage: tmctl <translate|report|status|export|redrive> --config book.yaml")
return invocation{}, fmt.Errorf("usage: tmctl <translate|report|status|export|redrive|manifest> --config book.yaml")
}
cmd, rest := args[0], args[1:]
@ -98,7 +103,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
resnapshot := fs.Bool("resnapshot", false, "re-pin existing jobs to the current config snapshot (re-translates already-paid chunks — explicit consent)")
acceptRebill := &rebillConsentValue{}
fs.Var(acceptRebill, "accept-rebill", "translate/redrive: consent to the projected RE-PAYMENT of already-billed work (D20.2-Q2). Bare accepts the whole projected amount; --accept-rebill=1.50 accepts it only up to $1.50")
asJSON := fs.Bool("json", false, "status: emit the projection as JSON (stable disposition/flag_reason enums) for CI/IDE")
asJSON := fs.Bool("json", false, "status: emit the projection as JSON (stable disposition/flag_reason enums) for CI/IDE; manifest: emit the manifest document instead of a summary")
asPlaintext := fs.Bool("plaintext", false, "export: emit the concatenated human text instead of the default stable JSON")
asPairs := fs.Bool("pairs", false, "export: include the source text per chunk (src↔target column for the DC1/DC2 FP-measure, WS5)")
chapter := fs.Int("chapter", -1, "redrive: restrict to this chapter (default: any)")
@ -106,6 +111,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
reason := fs.String("reason", "", "redrive: restrict to this flag_reason (default: any)")
dryRun := fs.Bool("dry-run", false, "redrive: report what would be re-attacked without touching anything")
verifyBank := fs.Bool("verify-bank", false, "translate/redrive: STOP at the bank-mining boundary and print the bank table for review instead of continuing into the edit wave with an unsigned bank (default: continue)")
ceilingUSD := fs.Float64("ceiling-usd", 0, "translate/redrive: the book USD ceiling in force for THIS RUN ONLY — it OVERRIDES book.yaml `ceilings.book_usd` and is never written back. It caps the book's CUMULATIVE committed+reserved spend, not this run's increment, and must be > 0")
seed := fs.String("seed", "", "seed-lint: path to the glossary seed YAML to validate ($0, no --config)")
if err := fs.Parse(rest); err != nil {
return invocation{}, err
@ -120,6 +126,25 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
if *cfgPath == "" {
return invocation{}, fmt.Errorf("--config book.yaml is required")
}
// The run ceiling is refused rather than defaulted when it is present but not a spendable amount, so
// the flag can only ever ADD a bound, never remove one (D39.110: «выключено» is not a state).
//
// Р7's own validator (config.LoadBook) is untouched and is NOT the same rule: it demands at least ONE
// of book_usd/day_usd, so a book may legitimately declare a daily ceiling only — and then the book
// ceiling this flag overrides is zero, i.e. absent, and passing the flag introduces one for this run.
// fs.Visit reports only flags actually PRESENT on the command line, which is what separates
// "--ceiling-usd 0" (a refusal) from an absent flag (the book's own ceiling stands). Placed AFTER the
// --config check so the frozen validation order is untouched: a missing --config is still the first
// thing a caller is told about.
ceilingGiven := false
fs.Visit(func(f *flag.Flag) {
if f.Name == "ceiling-usd" {
ceilingGiven = true
}
})
if ceilingGiven && (math.IsNaN(*ceilingUSD) || math.IsInf(*ceilingUSD, 0) || *ceilingUSD <= 0) {
return invocation{}, fmt.Errorf("--ceiling-usd must be a finite amount greater than zero (it is the ceiling for THIS run and overrides book.yaml ceilings.book_usd); got %v — omit the flag to run under the book's own ceiling", *ceilingUSD)
}
// The optional-value trap (see IsBoolFlag): `--accept-rebill 1.50` leaves 1.50 as a stray argument
// and the consent unlimited. Refuse loud rather than bill the difference. Scoped to exactly that
// spelling, so the historically-tolerated stray argument stays tolerated everywhere else.
@ -129,6 +154,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
return invocation{
cmd: cmd, cfgPath: *cfgPath, resnapshot: *resnapshot, acceptRebill: acceptRebill.c,
asJSON: *asJSON, asPlaintext: *asPlaintext, asPairs: *asPairs, verifyBank: *verifyBank,
ceilingUSD: *ceilingUSD,
sel: pipeline.RedriveSelector{
Chapter: *chapter, ChunkIdx: *chunk, Reason: *reason, DryRun: *dryRun,
},

37
backend/cmd/tmctl/invocation_test.go
View file

@ -17,9 +17,10 @@ import (
func TestParseNoArgsUsage(t *testing.T) {
_, err := parseInvocation(nil, &bytes.Buffer{})
// The command list gained `export` (D39 слой 6 read-only surface, a deliberate contract extension);
// the rest of the usage text stays frozen.
if err == nil || err.Error() != "usage: tmctl <translate|report|status|export|redrive> --config book.yaml" {
// The command list gained `export` (D39 слой 6 read-only surface) and then `manifest` (backlog row 100
// — the $0 producer of the chapter/chunk manifest); both are deliberate contract extensions and the
// rest of the usage text stays frozen.
if err == nil || err.Error() != "usage: tmctl <translate|report|status|export|redrive|manifest> --config book.yaml" {
t.Fatalf("usage error text is frozen, got: %v", err)
}
}
@ -206,3 +207,33 @@ func TestParseVerifyBank(t *testing.T) {
t.Fatal("--verify-bank must not accept a non-boolean value")
}
}
// --- backlog row 145: the per-run ceiling flag ----------------------------------------------------
// TestParseCeilingUSD pins the flag's whole contract: absent means "the book's own ceiling stands", a
// positive amount is carried through, and every value that is not a spendable amount is REFUSED rather
// than defaulted. The refusal is the load-bearing half — Р7 forbids a ledger with no ceiling, and a
// silently-ignored `--ceiling-usd 0` would be exactly that with a flag in front of it.
func TestParseCeilingUSD(t *testing.T) {
inv, err := parseInvocation([]string{"translate", "--config", "b.yaml"}, &bytes.Buffer{})
if err != nil || inv.ceilingUSD != 0 {
t.Fatalf("an absent flag must leave the ceiling unset, got %v / %v", inv.ceilingUSD, err)
}
inv, err = parseInvocation([]string{"translate", "--config", "b.yaml", "--ceiling-usd=1.25"}, &bytes.Buffer{})
if err != nil || inv.ceilingUSD != 1.25 {
t.Fatalf("--ceiling-usd=1.25 → %v / %v", inv.ceilingUSD, err)
}
// The space spelling binds too (unlike --accept-rebill, which is a bool-flag with an optional value).
inv, err = parseInvocation([]string{"redrive", "--config", "b.yaml", "--ceiling-usd", "0.5"}, &bytes.Buffer{})
if err != nil || inv.ceilingUSD != 0.5 {
t.Fatalf("--ceiling-usd 0.5 → %v / %v", inv.ceilingUSD, err)
}
for _, bad := range []string{"0", "-1", "NaN", "Inf"} {
var diag bytes.Buffer
if _, err := parseInvocation([]string{"translate", "--config", "b.yaml", "--ceiling-usd=" + bad}, &diag); err == nil {
t.Fatalf("--ceiling-usd=%s must be refused: a run ceiling that is not a spendable amount is not a ceiling", bad)
} else if code := exitCode(err); code != 1 {
t.Fatalf("--ceiling-usd=%s must exit 1, got %d", bad, code)
}
}
}

63
backend/cmd/tmctl/main.go
View file

@ -1,4 +1,4 @@
// tmctl is the TextMachine CLI: translate / report / status / redrive / backup.
// tmctl is the TextMachine CLI: translate / report / status / export / redrive / manifest / backup.
// main.go — thin wiring (package №4): argument parsing — invocation.go,
// output renderers — render.go, .env — dotenv.go; here just the
// «parse → env → ctx → fetch → render» wiring and exit-code mapping.
@ -77,7 +77,7 @@ func run() error {
if err := preflightBackup(inv.cfgPath, os.Stdout); err != nil {
return err
}
return translate(ctx, inv.cfgPath, inv.resnapshot, inv.acceptRebill, inv.verifyBank)
return translate(ctx, inv.cfgPath, inv.resnapshot, inv.acceptRebill, inv.verifyBank, inv.ceilingUSD)
case "report":
return report(inv.cfgPath)
case "status":
@ -91,21 +91,31 @@ func run() error {
return err
}
}
return redrive(ctx, inv.cfgPath, inv.resnapshot, inv.acceptRebill, inv.sel, inv.verifyBank)
return redrive(ctx, inv.cfgPath, inv.resnapshot, inv.acceptRebill, inv.sel, inv.verifyBank, inv.ceilingUSD)
case "manifest":
return manifestCmd(inv.cfgPath, inv.asJSON)
case "backup":
return backupCmd(inv.cfgPath, os.Stdout)
case "seed-lint":
return seedLint(inv.seedPath)
default:
return fmt.Errorf("unknown command %q (want translate|report|status|export|redrive|backup|seed-lint)", inv.cmd)
return fmt.Errorf("unknown command %q (want translate|report|status|export|redrive|manifest|backup|seed-lint)", inv.cmd)
}
}
// translate runs the book. The two money flags are ORTHOGONAL (D20.2-Q2): --resnapshot grants
// permission to re-pin jobs onto the current config snapshot, while --accept-rebill[=usd] consents to
// the AMOUNT that re-pin would re-pay. Over the book's consent threshold the run stops with the sum
// before reserving anything — a flag that names no money cannot carry a Р6 consent to a spend.
func translate(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill pipeline.RebillConsent, verifyBank bool) error {
// translate runs the book. The three money flags are ORTHOGONAL (D20.2-Q2 / row 145): --resnapshot
// grants permission to re-pin jobs onto the current config snapshot, --accept-rebill[=usd] consents to
// the AMOUNT that re-pin would re-pay, and --ceiling-usd replaces the BOOK ceiling for this process.
//
// ⚠ --ceiling-usd is NOT a per-run budget. The ledger admits against the book's CUMULATIVE spend
// (store/ledger.go: `bookTotal + estimate > BookUSD`, where bookTotal is committed+reserved across every
// run of the book), so the flag says "this book may reach $X while I am running", not "this run may
// spend $X". A caller that wants to grant an increment has to pass already-committed + increment, and a
// value below what the book has already spent denies the very first reservation.
//
// Over the book's consent threshold the run stops with the sum before reserving anything — a flag that
// names no money cannot carry a Р6 consent to a spend.
func translate(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill pipeline.RebillConsent, verifyBank bool, ceilingUSD float64) error {
r, err := pipeline.NewRunner(cfgPath, obs.NewLogger())
if err != nil {
return err
@ -114,6 +124,7 @@ func translate(ctx context.Context, cfgPath string, resnapshot bool, acceptRebil
r.Resnapshot = resnapshot
r.AcceptRebill = acceptRebill
r.VerifyBank = verifyBank
r.CeilingUSD = ceilingUSD
res, err := r.TranslateBook(ctx)
if err != nil {
@ -215,6 +226,34 @@ func status(ctx context.Context, cfgPath string, asJSON bool) error {
return renderStatusHuman(os.Stdout, rep, cfgPath)
}
// manifestCmd rebuilds and persists the book's chapter/chunk manifest (backlog row 100). $0 and key-less
// like the other read commands: it ingests + cuts the source and writes a sidecar beside the project DB —
// no LLM, no provider key, no wave.
//
// It opens the store exactly as `status` does (NewReadOnlyRunner): read-only, no flock — EXCEPT on the
// first touch of a project, where that path falls back to a full Open and therefore CREATES and migrates
// the database. That is the state this command is most often run in ("parsed, never run"), so the
// side effect is the common case rather than the corner one, and it is the same one `status` has always
// had. It also needs the project directory to be writable, unlike status/report/export.
//
// It exists because the manifest is needed BEFORE any run: a book that has been accepted and cut but
// never translated still has a chapter tree, and every other producer of that structure is a paid path.
// A translate rewrites the manifest itself (bookrun.go), so this command is for the not-yet-run case and
// for refreshing the artifact by hand after a source edit.
func manifestCmd(cfgPath string, asJSON bool) error {
r, err := pipeline.NewReadOnlyRunner(cfgPath, obs.NewLogger())
if err != nil {
return err
}
defer r.Close()
m, err := r.BuildAndPersistManifest()
if err != nil {
return err
}
return renderManifest(os.Stdout, m, r.ManifestPath(), asJSON)
}
// redrive re-attacks the FLAGGED chunks matching the selector (D15.3): it resets their terminal
// flag (chunk_status + checkpoints of the flagged/skipped stages) and re-runs the durable loop
// with a FRESH retry/escalation budget, never touching DispOK work. On --dry-run it only reports
@ -225,7 +264,10 @@ func status(ctx context.Context, cfgPath string, asJSON bool) error {
// silently ignored — Redrive already honoured r.Resnapshot, only the CLI never set it).
// --accept-rebill[=usd] is the separate consent to the AMOUNT such a re-pin re-pays (D20.2-Q2): over
// the threshold Redrive refuses BEFORE its destructive reset, so the flag telemetry survives a refusal.
func redrive(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill pipeline.RebillConsent, sel pipeline.RedriveSelector, verifyBank bool) error {
// --ceiling-usd applies here for the same reason it applies to translate: a redrive re-attacks flagged
// chunks with real provider calls, so leaving it on the book's ceiling alone would be a hole in exactly
// the surface row 145 exists to close.
func redrive(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill pipeline.RebillConsent, sel pipeline.RedriveSelector, verifyBank bool, ceilingUSD float64) error {
r, err := pipeline.NewRunner(cfgPath, obs.NewLogger())
if err != nil {
return err
@ -234,6 +276,7 @@ func redrive(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill
r.Resnapshot = resnapshot
r.AcceptRebill = acceptRebill
r.VerifyBank = verifyBank
r.CeilingUSD = ceilingUSD
summary, res, err := r.Redrive(ctx, sel)
if err != nil {

69
backend/cmd/tmctl/rebill_cli_test.go
View file

@ -98,7 +98,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) {
bookPath := setupCLIProject(t, srv.URL)
ctx := context.Background()
if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false); err != nil {
if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0); err != nil {
t.Fatalf("first run: %v", err)
}
if calls != 1 {
@ -114,7 +114,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) {
writeCLIFile(t, pipePath, strings.Replace(string(raw), "prompt_version: v-cli", "prompt_version: v-cli-drift", 1))
// --resnapshot WITHOUT consent → refused, nothing called.
err = translate(ctx, bookPath, true, pipeline.RebillConsent{}, false)
err = translate(ctx, bookPath, true, pipeline.RebillConsent{}, false, 0)
if err == nil {
t.Fatal("`translate --resnapshot` without --accept-rebill must be refused over the threshold")
}
@ -127,7 +127,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) {
// A ceiling BELOW the projection is still a refusal — proving the CAPPED value is wired, not just
// the boolean.
if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true, Capped: true, CapUSD: 0.0001}, false); err == nil {
if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true, Capped: true, CapUSD: 0.0001}, false, 0); err == nil {
t.Fatal("a ceiling below the projection must be refused through the CLI entry point too")
}
if calls != 1 {
@ -135,7 +135,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) {
}
// Consent given → the run proceeds and really re-pays.
if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true}, false); err != nil {
if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true}, false, 0); err != nil {
t.Fatalf("`translate --resnapshot --accept-rebill` must proceed: %v", err)
}
if calls != 2 {
@ -160,7 +160,7 @@ func TestRedriveWiresAcceptRebill(t *testing.T) {
ctx := context.Background()
// The refusal text flags the only chunk (exit 2 is a sentinel, not an infra failure).
if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false); exitCode(err) != 2 {
if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0); exitCode(err) != 2 {
t.Fatalf("setup: the refusal fixture must complete with flags (exit 2), got %v", err)
}
callsAfterRun1 := calls
@ -173,7 +173,7 @@ func TestRedriveWiresAcceptRebill(t *testing.T) {
writeCLIFile(t, pipePath, strings.Replace(string(raw), "prompt_version: v-cli", "prompt_version: v-cli-drift", 1))
sel := pipeline.RedriveSelector{Chapter: -1, ChunkIdx: -1}
if err := redrive(ctx, bookPath, true, pipeline.RebillConsent{}, sel, false); err == nil {
if err := redrive(ctx, bookPath, true, pipeline.RebillConsent{}, sel, false, 0); err == nil {
t.Fatal("`redrive --resnapshot` without consent must be refused over the threshold")
} else if !strings.Contains(err.Error(), "--accept-rebill") {
t.Errorf("the redrive refusal must name the flag; got: %v", err)
@ -184,10 +184,65 @@ func TestRedriveWiresAcceptRebill(t *testing.T) {
// Consent given → the redrive resets and re-attacks (the mock still refuses, so it ends flagged again
// — what matters is that it RAN and billed).
if err := redrive(ctx, bookPath, true, pipeline.RebillConsent{Given: true}, sel, false); exitCode(err) != 2 {
if err := redrive(ctx, bookPath, true, pipeline.RebillConsent{Given: true}, sel, false, 0); exitCode(err) != 2 {
t.Fatalf("`redrive --resnapshot --accept-rebill` must run (and end flagged), got %v", err)
}
if calls <= callsAfterRun1 {
t.Fatalf("the consented redrive must re-call the provider, calls=%d", calls)
}
}
// TestTranslateAndRedriveWireTheRunCeiling is the same class as TestTranslateWiresAcceptRebill, for the
// flag row 145 added: it exists to change what the RUN does, not what the parser returns.
//
// It closes an audit finding — deleting `r.CeilingUSD = ceilingUSD` from either command left the whole
// backend suite green, because TestParseCeilingUSD stops at the parsed invocation and the pipeline-level
// ceiling tests set the field on the Runner directly. Neither of them touches the wire between the two.
func TestTranslateAndRedriveWireTheRunCeiling(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = io.ReadAll(r.Body)
tb, _ := json.Marshal("ЧЕРНОВИК ПЕРЕВОДА")
fmt.Fprintf(w, `{"id":"fake","model":"fake-model","choices":[{"message":{"content":%s},"finish_reason":"stop"}],
"usage":{"prompt_tokens":1000,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":200}}}`, tb)
}))
defer srv.Close()
ctx := context.Background()
// A ceiling far below anything the book can cost. The book's own ceiling is comfortable, so ONLY the
// argument can produce this stop — and only if the command actually hands it to the runner.
const tiny = 0.000001
err := translate(ctx, setupCLIProject(t, srv.URL), false, pipeline.RebillConsent{}, false, tiny)
if err == nil || !strings.Contains(err.Error(), "ceiling reached") {
t.Fatalf("translate must carry --ceiling-usd into the run; got %v", err)
}
if !strings.Contains(err.Error(), "--ceiling-usd") {
t.Fatalf("the stop must name the ceiling in force: %v", err)
}
// Same for redrive: it re-attacks with real provider calls, so an unwired flag there is a hole in
// exactly the surface the row closes. A redrive with no targets never reserves and therefore could not
// show a ceiling at all — so the book is first driven to a FLAGGED chunk, which gives the redrive
// something to re-attack and makes the reservation (and the ceiling) reachable.
refuse := true
srv2 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
text := "ЧЕРНОВИК ПЕРЕВОДА"
if refuse && !strings.Contains(string(body), "Черновик перевода для редактуры") {
text = "Извините, я не могу перевести это." // soft refusal → the chunk is flagged
}
tb, _ := json.Marshal(text)
fmt.Fprintf(w, `{"id":"fake","model":"fake-model","choices":[{"message":{"content":%s},"finish_reason":"stop"}],
"usage":{"prompt_tokens":1000,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":200}}}`, tb)
}))
defer srv2.Close()
bookPath := setupCLIProject(t, srv2.URL)
if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0); exitCode(err) != 2 {
t.Fatalf("setup: the run must finish WITH a flagged chunk (exit 2), got %v", err)
}
refuse = false
sel := pipeline.RedriveSelector{Chapter: -1, ChunkIdx: -1}
err = redrive(ctx, bookPath, false, pipeline.RebillConsent{}, sel, false, tiny)
if err == nil || !strings.Contains(err.Error(), "ceiling reached") {
t.Fatalf("redrive must carry --ceiling-usd into the re-attack; got %v", err)
}
}

38
backend/cmd/tmctl/render.go
View file

@ -84,6 +84,9 @@ func renderSignatureStop(w io.Writer, s *pipeline.WaveSignatureStop) {
if s.TablePath != "" {
fmt.Fprintf(w, "Full table: %s\n", s.TablePath)
}
if s.TableJSONPath != "" {
fmt.Fprintf(w, "Machine table: %s\n", s.TableJSONPath)
}
fmt.Fprintln(w, "Next: for EACH term either promote it into the mined-delta file (approved + dst), OR decline it in")
fmt.Fprintln(w, "the mined-rejects file (book.yaml: mined_rejects), then re-run `tmctl translate`. The stop clears")
fmt.Fprintln(w, "once every proposed term is promoted or rejected (then the delta is empty → auto-continue to the edit wave).")
@ -137,7 +140,7 @@ func renderBankStopRows(w io.Writer, rows []pipeline.BankStopRow) {
if len(r.Variants) > 1 {
// The disagreement is the reason to sign: one term coming back several ways is exactly what a
// canon exists to close, and hiding it behind one winner throws that reason away.
fmt.Fprintf(w, " drafts: %s\n", trunc(strings.Join(r.Variants, " | "), 90))
fmt.Fprintf(w, " drafts: %s\n", trunc(strings.Join(r.VariantLabels(), " | "), 90))
}
if len(r.Contexts) > 0 {
fmt.Fprintf(w, " ctx: %s\n", trunc(r.Contexts[0], 90))
@ -516,6 +519,24 @@ func flagParen(r string) string {
return " (" + r + ")"
}
// renderManifest prints the persisted chapter/chunk manifest (row 100). The default is a human summary —
// a 2283-chapter tree is not something anyone reads in a terminal — and --json emits the document itself,
// byte-identical to the sidecar, so a consumer can pipe it instead of knowing where the file lives.
func renderManifest(w io.Writer, m *pipeline.BookManifest, path string, asJSON bool) error {
if asJSON {
enc := json.NewEncoder(w)
enc.SetIndent("", " ")
return enc.Encode(m)
}
fmt.Fprintf(w, "=== MANIFEST: %s ===\n", m.BookID)
fmt.Fprintf(w, "chapters=%d units=%d chunks=%d\n", m.ChaptersTotal, m.UnitsTotal, m.ChunksTotal)
fmt.Fprintf(w, "source: %d bytes (sha256 %.12s) · encoding %s · %s→%s\n",
m.SourceBytes, m.SourceSHA256, m.Encoding, m.SourceLang, m.TargetLang)
fmt.Fprintf(w, "cut by: %s · validity key %.12s\n", m.ChunkerVersion, m.Key)
fmt.Fprintf(w, "written: %s\n", path)
return nil
}
// renderStatusJSON emits the StatusReport as indented JSON (stable enums — the
// ratified CI/IDE contract, D12/D15.3) and maps flagged>0 to the exit-2 sentinel.
func renderStatusJSON(w io.Writer, rep *pipeline.StatusReport) error {
@ -560,6 +581,12 @@ func renderStatusHuman(w io.Writer, rep *pipeline.StatusReport, cfgPath string)
fmt.Fprintf(w, "=== STATUS: %s (snapshot %s)%s ===\n", rep.BookID, snap, drift)
fmt.Fprintf(w, "Progress: %d/%d units (%.1f%%) — done=%d in_progress=%d flagged=%d pending=%d\n",
rep.Done, rep.TotalUnits, rep.PercentDone, rep.Done, rep.InProgress, rep.Flagged, rep.Pending)
// Per-wave progress (row 99). The line above needs BOTH waves to have resolved a unit ok, and the edit
// wave does not start before the bank stop — so it sits at 0/N for the whole draft wave and says nothing
// about the drafting that IS happening. This line is the split; "resolved" there includes flagged units
// (see PhaseProgress), which is why it can exceed the done count above.
fmt.Fprintf(w, "Phases (units resolved by wave): draft %s · edit %s\n",
waveCell(rep.Progress.Draft), waveCell(rep.Progress.Edit))
fmt.Fprintf(w, "Escalations: %d · post-check misses (confirmed): %d · style-flags (observability): %d\n",
rep.Escalations, rep.PostcheckMisses, rep.StyleFlags)
// Printed only when the bank actually holds unsigned rows, so a book translating entirely against signed
@ -614,6 +641,15 @@ func renderStatusHuman(w io.Writer, rep *pipeline.StatusReport, cfgPath string)
return nil
}
// waveCell renders one wave counter, or "—" for a wave this pipeline does not have (a draft-only
// pipeline has no edit wave, and printing "0/0" for it would read as work that never starts).
func waveCell(c pipeline.WaveCounter) string {
if c.Total == 0 {
return "—"
}
return fmt.Sprintf("%d/%d", c.Done, c.Total)
}
// dashIfEmpty renders "" as "—" for a table cell.
// dashIfEmpty renders an empty (or blank) cell as an em dash. ONE definition: pack-20 briefly added a
// second, near-identical `dash` 400 lines above, and the two disagreed on whitespace — the kind of split

2
backend/cmd/tmctl/render_test.go
View file

@ -262,7 +262,7 @@ func TestRenderSignatureStopShowsTheTable(t *testing.T) {
var rows []pipeline.BankStopRow
rows = append(rows, pipeline.BankStopRow{
Src: "方源", Dst: "Фан Юань", Origin: "both", Type: "name", Freq: 12, Spread: 2,
Variants: []string{"Фан Юань ×3", "Фань Юань ×1"}, Contexts: []string{"方源来到青茅山"},
Variants: []pipeline.BankStopVariant{{Dst: "Фан Юань", Chunks: 3}, {Dst: "Фань Юань", Chunks: 1}}, Contexts: []string{"方源来到青茅山"},
})
for i := 0; i < 25; i++ {
rows = append(rows, pipeline.BankStopRow{Src: fmt.Sprintf("术%d", i), Origin: "mined", Freq: 5})

32
backend/docs/pipeline.puml
View file

@ -1,5 +1,5 @@
@startuml pipeline
title TextMachine — поток перевода книги (2026-07-25, синхронизировано с D1–D39.25, пост-пак-16: волновой исполнитель)\nПометки: [OK] = построено · [FLAG] = построено за enabled:false · [Ф2] = планово
title TextMachine — поток перевода книги (2026-08-08, синхронизировано с D1–D39.25, пост-пак-16: волновой исполнитель; + блокеры контракта 99/100/101/125/145)\nПометки: [OK] = построено · [FLAG] = построено за enabled:false · [Ф2] = планово
skinparam wrapWidth 260
start
@ -15,6 +15,16 @@ GB18030-детект/конверсия, ruby-захват, NormalizeSource;
:Чанкинг [OK]: draft-чанки по ВЫХОДНОМУ токен-бюджету
(SegBudget пары: фертильность CJK/other), edit-единицы декаплены ·
precompute sticky по BASE-банку одним упорядоченным проходом;
:МАНИФЕСТ глав и чанков [OK] (строка 100): сайдкар
рядом с БД — project_db + .manifest.json, атомарная запись ·
СТАБИЛЬНЫЙ id главы = хеш ИНГЕСТИРОВАННОГО текста главы
(переживает пере-чанковку и правку соседней главы) ·
id ЮНИТА, наоборот, умирает с разрезом: глава + тег разреза + leader-индекс ·
ключ годности = исходник+чанкер+сегментация+форма пайплайна+пак+embed+norm ·
пишет любой write-путь, до прогона — команда tmctl manifest;
:ЭКСПОРТ БАНКА [OK] (строка 125): project_db + .bank.json —
весь банк тремя статусами, id из ключа уникальности
(не автоинкремент glossary.id) · пишется на границах стадий;
note right
Снапшот per-wave [OK]: draft-волна пинит BASE-банк,
edit-волна — ENRICHED (обогащение двигает ТОЛЬКО edit —
@ -29,7 +39,9 @@ partition "ВОЛНА ДРАФТА [OK] (∥ workers, по draft-чанкам)"
+ src→dst-инъекция глоссария из precomputed selection
(Aho-Corasick, trust-gate, спойлер-hard-reject, sticky);
:runStage [OK]: чекпойнт-хит → $0-реклассификация ·
иначе Reserve (потолки книга/день) → вызов →
иначе Reserve (потолки книга/день · потолок ПРОГОНА
--ceiling-usd перекрывает книжный book_usd, строка 145,
в хеши не входит) → вызов →
Settle+чекпойнт ОДНОЙ tx → classify ПОСЛЕ денег;
:Банкнота ⟦TM-BANK-v1⟧ [FLAG]: срез с RAW-драфта
до classify · чистый драфт → derived-чекпойнт;
@ -56,6 +68,10 @@ if (Майнинг вкл. И дельта непуста? [OK]) then (да)
:$0-экраны: канон-конфликты · КОНФЛИКТЫ СВОИХ ЖЕ
консолидаций · label-mismatch → стоп-таблица
(сигналы топ-варианта · invented · уверенность · конвенций);
:Сайдкары стопа [OK]: .bank-stop.txt — человеку ·
.bank-stop.json — машине (строка 101): ПОЛНАЯ таблица,
списки списками, «роль промолчала» = null, не 0 ·
плюс обновление экспорта банка;
:signature map → прогон ОСТАНОВЛЕН (exit 3);
:Владелец: каждый терм → promote в mined_delta (approved+dst)
ИЛИ decline в mined_rejects;
@ -88,10 +104,20 @@ partition "ВОЛНА РЕДАКТУРЫ [OK] (∥ по edit-единицам, E
}
:Read-модели $0 [OK]: tmctl status (wave-aware) ·
ПОФАЗНЫЙ прогресс draft N/M и edit N/M в юнитах (строка 99) —
сквозной счётчик стоит на нуле всю черновую волну ·
quality-report (структурный KPI + residual-скан
RepairCandidates = замер остатка для решения о repair) ·
очередь флагнутых → человек (redrive точечно);
:Экспорт $0 [OK]: манифест-ре-чанк + final-строки chunk_status,
note right
Источник манифеста для status и export по умолчанию —
ПЕРСИСТ (строка 100), а не пере-нарезка: 1.4 с CPU → 0.05-0.12 с
на книге 23 МБ. Протухший ключ = откат на полную нарезку,
подать устаревший знаменатель нельзя по построению.
Пере-нарезку сохраняют пути, которым нужен ТЕКСТ:
export --pairs, контент-ветка ре-билла, quality-report.
end note
:Экспорт $0 [OK]: манифест + final-строки chunk_status,
checks.ExportNormalize байт-в-байт с translate,
--pairs (source-колонка), счётчики pending/ghost/ConfigDrift;
note right

22
backend/internal/chunk/chunker.go
View file

@ -103,21 +103,37 @@ func (s SegBudget) EstOut(cjk, other int) float64 {
// EstOut + OversizedSentence flag + EditUnitID (WS2); the edit-unit id is monotone across
// the book so a the edit wave unit is uniquely addressable.
func SplitChunks(chapters []string, seg SegBudget, heading *lang.HeadingRule, abbrevs map[string]bool) []Chunk {
chunks, _ := SplitChunksWithChapters(chapters, seg, heading, abbrevs)
return chunks
}
// SplitChunksWithChapters is SplitChunks plus the INGESTED text of every chapter that consumed a number,
// index-aligned to the chapter numbers it emitted (kept[0] is chapter 1). It exists so a caller that has
// to identify a chapter — the persisted manifest, backlog row 100 — gets the chapter's own text from the
// ONE place that knows which chapters consume a number, instead of re-deriving that rule and drifting
// from it the first time an "empty chapter" edge case changes.
//
// The text returned is the chapter AS INGESTED — before stripHeading. That is deliberate: an identity
// built on it survives a heading-rule edit (a data change in the pair pack) and a chunker/budget change,
// because neither touches what ingest produced. Nothing else here changes: SplitChunks is this function
// with the second result dropped, so every existing caller is byte-identical.
func SplitChunksWithChapters(chapters []string, seg SegBudget, heading *lang.HeadingRule, abbrevs map[string]bool) (chunks []Chunk, kept []string) {
var out []Chunk
chapterNo := 0
editUnitID := 0
for _, chapText := range chapters {
for _, ingested := range chapters {
// Title policy (pack-13): detect a leading structural header, render it deterministically and
// STRIP its marker from the text the model sees. A nil rule (no pack / no heading.txt) or a chapter
// with no header is a NO-OP — headingText is "" and chapText is unchanged, so a book that does not
// opt in produces byte-identical chunks. Runs BEFORE splitParagraphs so the stripped subtitle is
// re-paragraphed normally.
headingText, chapText := stripHeading(chapText, heading)
headingText, chapText := stripHeading(ingested, heading)
paras := splitParagraphs(chapText)
if len(paras) == 0 {
continue // an empty chapter does not consume a chapter number
}
chapterNo++
kept = append(kept, ingested)
chapterChunks := chapterDraftChunks(chapterNo, paras, seg, abbrevs)
if headingText != "" && len(chapterChunks) > 0 {
chapterChunks[0].Heading = headingText // the chapter's first chunk carries the deterministic title
@ -125,7 +141,7 @@ func SplitChunks(chapters []string, seg SegBudget, heading *lang.HeadingRule, ab
assignEditUnits(chapterChunks, seg, &editUnitID)
out = append(out, chapterChunks...)
}
return out
return out, kept
}
// stripHeading detects a chapter-leading structural header via the pair's HeadingRule, returning the

70
backend/internal/chunk/chunker_manifest_test.go Normal file
View file

@ -0,0 +1,70 @@
package chunk
import (
"reflect"
"strings"
"testing"
"textmachine/backend/internal/lang"
)
// chunker_manifest_test.go: the seam SplitChunksWithChapters adds for the persisted manifest (backlog
// row 100). Two properties matter and neither is obvious from the signature: the split itself must not
// have changed, and the chapter texts returned must be index-aligned to the chapter NUMBERS the split
// emitted — a chapter that consumes no number must not consume a slot either, or every chapter after the
// first empty one is identified by its neighbour's text.
var manifestSeg = SegBudget{DraftBudgetOut: 1797, EditCeilingOut: 3200, FertCJK: 1.1978, FertOther: 0.3852}
func TestSplitChunksWithChaptersLeavesTheSplitUnchanged(t *testing.T) {
chapters := []string{
"Первая глава книги.",
" \n\t ", // whitespace-only: consumes NO chapter number
"Вторая глава книги.",
"",
strings.Repeat("Длинный абзац для нарезки. ", 200),
}
want := SplitChunks(chapters, manifestSeg, nil, nil)
got, kept := SplitChunksWithChapters(chapters, manifestSeg, nil, nil)
if !reflect.DeepEqual(got, want) {
t.Fatal("SplitChunksWithChapters must produce the SAME cut as SplitChunks — the chunk boundaries decide what goes on the wire")
}
if len(kept) != 3 {
t.Fatalf("three chapters yield text, so three consume a number; got %d: %q", len(kept), kept)
}
// Index alignment: kept[n-1] must be the text of chapter n as the chunker numbered it.
if kept[0] != chapters[0] || kept[1] != chapters[2] || kept[2] != chapters[4] {
t.Fatalf("kept texts are not aligned to the emitted chapter numbers: %q", kept)
}
// …and every emitted chapter number has a slot.
highest := 0
for _, c := range got {
if c.Chapter > highest {
highest = c.Chapter
}
}
if highest != len(kept) {
t.Fatalf("the split emitted chapters up to %d but returned %d chapter texts", highest, len(kept))
}
}
// TestSplitChunksWithChaptersReturnsPreStripText: the text handed back is the chapter AS INGESTED, with
// its source header still in it. That is what makes an identity built on it survive a heading-rule edit —
// the rule changes what is STRIPPED and what title is rendered, not what ingest produced. Asserted with a
// rule ON and OFF over the same chapter, so the invariance is measured rather than assumed.
func TestSplitChunksWithChaptersReturnsPreStripText(t *testing.T) {
chapters := []string{"第一节：подзаголовок\nтело главы"}
rule := &lang.HeadingRule{Marker: "第", Units: map[rune]bool{'节': true}, Template: "Глава {n}"}
withRule, keptWith := SplitChunksWithChapters(chapters, manifestSeg, rule, nil)
withoutRule, keptWithout := SplitChunksWithChapters(chapters, manifestSeg, nil, nil)
if len(withRule) == 0 || len(withoutRule) == 0 {
t.Fatalf("fixture drifted: %d/%d chunks", len(withRule), len(withoutRule))
}
if withRule[0].Heading != "Глава 1" || strings.Contains(withRule[0].Text, "第一节") {
t.Fatalf("precondition: the rule must render a title and strip the marker, got heading=%q text=%q", withRule[0].Heading, withRule[0].Text)
}
if keptWith[0] != chapters[0] || keptWithout[0] != chapters[0] {
t.Fatalf("the chapter text must be returned AS INGESTED under both rules, got %q / %q", keptWith[0], keptWithout[0])
}
}

60
backend/internal/pipeline/artifact.go Normal file
View file

@ -0,0 +1,60 @@
package pipeline
import (
"fmt"
"os"
"path/filepath"
)
// artifact.go: the shared write discipline of the engine's READ-OUT FILES — the sidecars beside the
// project DB that another process (the platform, D39.81/D39.85) reads while this one runs: the
// chapter/chunk manifest (row 100), the machine bank-stop table (row 101), the bank export (row 125).
//
// They have one property in common that the older sidecars did not have to care about: they are read
// CONCURRENTLY with the run that writes them. A plain os.WriteFile truncates first, so a reader that
// opens the file in that window gets a valid path, zero bytes and a parse error indistinguishable from
// a corrupt artifact. Write-then-rename makes every read see either the previous document or the next
// one, whole — the same guarantee the store gets from its transactions, at file granularity.
// writeFileAtomic replaces path with data through a temp file in the SAME directory + rename. Same
// directory is load-bearing: rename is atomic only within a filesystem, and the project dir is where the
// reader is looking anyway. Every RETURNED failure removes its temp file, so a failed write leaves the
// previous document in place and nothing beside it; a kill -9 mid-write can still leave one dot-prefixed
// temp file, which is inert (no reader looks at it) and is not worth a startup sweep.
func writeFileAtomic(path string, data []byte) error {
dir, base := filepath.Dir(path), filepath.Base(path)
f, err := os.CreateTemp(dir, "."+base+".tmp-*")
if err != nil {
return fmt.Errorf("pipeline: create temp for %s: %w", path, err)
}
tmp := f.Name()
if _, err := f.Write(data); err != nil {
f.Close()
_ = os.Remove(tmp)
return fmt.Errorf("pipeline: write %s: %w", tmp, err)
}
// Sync before the rename: without it the rename's metadata can reach disk ahead of the bytes, so a host
// crash can leave a present-but-truncated document under the real name. The manifest would survive that
// (a parse failure degrades to the re-chunk), but the bank export and the stop table have no such
// validation — a consumer would read a short document as a complete one.
if err := f.Sync(); err != nil {
f.Close()
_ = os.Remove(tmp)
return fmt.Errorf("pipeline: sync %s: %w", tmp, err)
}
if err := f.Close(); err != nil {
_ = os.Remove(tmp)
return fmt.Errorf("pipeline: close %s: %w", tmp, err)
}
// 0o644 like the other sidecars: CreateTemp makes 0o600, and an artifact another process reads must
// not depend on that process running as the same user.
if err := os.Chmod(tmp, 0o644); err != nil {
_ = os.Remove(tmp)
return fmt.Errorf("pipeline: chmod %s: %w", tmp, err)
}
if err := os.Rename(tmp, path); err != nil {
_ = os.Remove(tmp)
return fmt.Errorf("pipeline: replace %s: %w", path, err)
}
return nil
}

161
backend/internal/pipeline/bankexport.go Normal file
View file

@ -0,0 +1,161 @@
package pipeline
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"sort"
"strconv"
"strings"
"textmachine/backend/internal/store"
)
// bankexport.go: the BANK EXPORT artifact (backlog row 125) — the engine's read-out of the whole memory
// bank, all three statuses, as a file beside the project DB.
//
// It exists because the bank lives in the engine's PRIVATE SQLite and the platform must not read that
// store (D39.85), so a read channel for it had no carrier at all. It is deliberately NOT the same thing
// as the bank-stop table (row 101): that one is the set of DECISIONS put in front of the owner at one
// stop — proposals, evidence, why-this-rendering — and it exists only for a run that mined something.
// This one is the STATE of the bank as it stands: every row the book has, including seed rows nobody
// ever had to decide on, and it exists for every book with a bank whether anything was mined or not.
//
// It is a projection, never a source: nothing reads it back into the engine. Signing still goes through
// the ratified stop mechanics (promote into the mined-delta file / decline in the mined-rejects file),
// because the pipeline REPLACES a book's whole glossary from its deterministic inputs on every run and a
// row edit would be erased by the next one.
// bankExportVersion versions the SHAPE of the document (not its content), so a consumer that has to
// tolerate the field set changing has something to branch on.
const bankExportVersion = "tm-bank-v1"
// BankExport is the whole-bank read-out.
type BankExport struct {
Version string `json:"bank_version"`
BookID string `json:"book_id"`
// Total / Signed are whole-bank counters (not counters of some page): rows in the file, and rows of
// those whose status is `approved` — the only status that is canon on the wire.
Total int `json:"total"`
Signed int `json:"signed"`
Terms []BankExportTerm `json:"terms"`
}
// BankExportTerm is one bank row.
//
// The field names follow the ENGINE's own vocabulary for the term's surfaces (`src`/`dst`) and call the
// provenance column `origin`, because the engine's own column name for provenance is `source` and using
// that word here would make `source` mean the term's text in one schema and its provenance in the other.
type BankExportTerm struct {
// ID is a STABLE key derived from the row's uniqueness key (src, sense, since_ch, until_ch) — NOT
// glossary.id. That column is a fresh autoincrement on every bank replace (store/migrate.go v5), and
// the bank is replaced on every run, so an exported autoincrement would re-point under a consumer
// between two reads of the same unchanged term. Derived, it survives exactly as long as the term does.
ID string `json:"id"`
Src string `json:"src"`
Dst string `json:"dst"` // "" for a candidate with no proposed rendering yet
// Kind is the engine's `type` column and is LEGITIMATELY empty: a ruby candidate that is neither a
// name nor a place carries none (membank/memseed.go). It is exported as the empty string rather than
// omitted — "the engine did not decide" is a state the row still needs signing in, and a missing field
// would invite a consumer to invent one.
Kind string `json:"kind"`
// Status is the three-valued signing state auto|draft|approved; only `approved` is injected as canon.
Status string `json:"status"`
// Origin is the provenance: which path created the row (seed|ruby|mined). It is an axis INDEPENDENT
// of Status — a mined row can be approved, a seed row can be draft.
Origin string `json:"origin"`
Sense string `json:"sense"`
SinceChapter int `json:"since_chapter"`
UntilChapter int `json:"until_chapter"`
Aliases []string `json:"aliases"`
}
// bankExportPath is where the read-out lands: beside the project DB, like the signature map, the stop
// tables and the auto-bank. That is the book's state directory — it travels with the DB and never enters
// git — and it is the one place a reader already has to know about.
func (r *Runner) bankExportPath() string { return r.Book.ProjectDB + ".bank.json" }
// bankTermID is the stable derived id (see BankExportTerm.ID).
//
// The four parts are LENGTH-PREFIXED rather than separator-joined. The first version used U+001F and
// asserted in a comment that it "cannot occur in any of them" — an assumption about data the engine never
// validates: `src` and `sense` are free text copied from a seed YAML, and nothing on the load path
// rejects a control character. Length-prefixing makes the encoding injective by construction, so the
// claim does not have to be true (and does not have to be re-checked when a new seed source appears).
// 16 hex chars of SHA-256 is 64 bits, which for a bank of thousands of rows makes a collision an
// irrelevance rather than a risk taken on purpose.
func bankTermID(src, sense string, since, until int) string {
var b strings.Builder
for _, part := range []string{src, sense, strconv.Itoa(since), strconv.Itoa(until)} {
fmt.Fprintf(&b, "%d:%s", len(part), part)
}
sum := sha256.Sum256([]byte(b.String()))
return hex.EncodeToString(sum[:])[:16]
}
// projectBankExport turns stored rows into the document. Deterministic: rows are sorted by the same
// uniqueness key the id is derived from, so two runs over an unchanged bank produce byte-identical files
// (a consumer can diff them, and a no-op run is visibly a no-op).
func projectBankExport(bookID string, rows []store.GlossaryEntry) BankExport {
out := BankExport{Version: bankExportVersion, BookID: bookID, Total: len(rows), Terms: make([]BankExportTerm, 0, len(rows))}
for _, e := range rows {
if e.Status == "approved" {
out.Signed++
}
aliases := make([]string, 0, len(e.Aliases))
for _, a := range e.Aliases {
aliases = append(aliases, a.Alias)
}
sort.Strings(aliases) // the store returns them in insert order; a read-out must not depend on it
out.Terms = append(out.Terms, BankExportTerm{
ID: bankTermID(e.Src, e.Sense, e.SinceCh, e.UntilCh),
Kind: e.Type, Src: e.Src, Dst: e.Dst, Status: e.Status, Origin: e.Source,
Sense: e.Sense, SinceChapter: e.SinceCh, UntilChapter: e.UntilCh, Aliases: aliases,
})
}
sort.SliceStable(out.Terms, func(i, j int) bool {
a, b := out.Terms[i], out.Terms[j]
switch {
case a.Src != b.Src:
return a.Src < b.Src
case a.Sense != b.Sense:
return a.Sense < b.Sense
case a.SinceChapter != b.SinceChapter:
return a.SinceChapter < b.SinceChapter
default:
return a.UntilChapter < b.UntilChapter
}
})
return out
}
// exportBank writes the read-out. `at` names the boundary that triggered it and rides the log line, so a
// stale artifact can be traced to the moment it was last written rather than guessed at.
//
// FAILURE DISCIPLINE: a read-out that cannot be written is loud but never fatal. Killing a paid run
// because a projection of state the run already holds could not be serialized would trade real money for
// an artifact that the next boundary rewrites anyway.
func (r *Runner) exportBank(ctx context.Context, at string) {
rows, err := r.Store.GlossaryForBook(r.Book.BookID)
if err != nil {
r.Log.WarnContext(ctx, "bank export: could not read the bank; the export artifact was NOT refreshed and may be stale",
"at", at, "path", r.bankExportPath(), "err", err)
return
}
exp := projectBankExport(r.Book.BookID, rows)
body, err := json.MarshalIndent(exp, "", " ")
if err != nil {
r.Log.WarnContext(ctx, "bank export: could not marshal the bank; the export artifact was NOT refreshed and may be stale",
"at", at, "path", r.bankExportPath(), "err", err)
return
}
if err := writeFileAtomic(r.bankExportPath(), append(body, '\n')); err != nil {
r.Log.WarnContext(ctx, "bank export: could not write the export artifact; it was NOT refreshed and may be stale",
"at", at, "path", r.bankExportPath(), "err", err)
return
}
r.Log.InfoContext(ctx, "bank export refreshed", "at", at, "path", r.bankExportPath(),
"terms", exp.Total, "approved", exp.Signed)
}

23
backend/internal/pipeline/bookrun.go
View file

@ -109,6 +109,10 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
// Ingest reads + normalizes the source (txt/epub) into ordered per-chapter text
// and captures ruby readings (ingest.go), decoding the txt source per book.encoding
// (auto/utf8/gb18030). Offline and deterministic ($0, no LLM).
// Fingerprint the source BEFORE reading it (row 100): the manifest written below must be stamped with
// the identity of the bytes it was actually cut from, and the cut takes seconds on a large book.
// nil = it could not be taken, which costs this run its manifest and nothing else.
srcBefore := r.sourceFingerprintBeforeIngest(ctx)
doc, err := chunk.IngestEncoded(r.Book.SourceFile, r.Book.Encoding, r.Book.SourceLang)
if err != nil {
return nil, err
@ -136,11 +140,19 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
if err := r.seedGlossary(ctx); err != nil {
return nil, err
}
// The bank read-out (backlog row 125), at the FIRST boundary where a bank exists. Doing it here rather
// than only at the stops means a book that never mines anything — or a run that dies before the first
// wave — still leaves a readable bank behind, which is the state a reader has to be able to see.
r.exportBank(ctx, "run-start/seeded")
chunks := chunk.SplitChunks(doc.Chapters, r.segBudget(), r.headingRule(), r.sentenceAbbrevs())
chunks, chapterTexts := chunk.SplitChunksWithChapters(doc.Chapters, r.segBudget(), r.headingRule(), r.sentenceAbbrevs())
if len(chunks) == 0 {
return nil, fmt.Errorf("pipeline: source file %s produced no chunks after normalization", r.Book.SourceFile)
}
// The chapter/chunk manifest (backlog row 100): persisted HERE, where the split that every paid byte
// is addressed against was just computed, so the artifact and the run can never describe different
// books. Loud but not fatal — see persistManifest.
r.persistManifest(ctx, srcBefore, chapterTexts, chunks)
// the precompute pass: the banknote parser's source index (backlog 19). Built here, in the composite
// root, because the rule "a bank line must name something the book contains" needs the whole chunk
// manifest, and building it anywhere else would let a call path exist without it.
@ -173,5 +185,12 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
// upserts the per-wave snapshots (draft-wave snapshot base-bank / edit-wave snapshot enriched) itself and pins each
// wave's jobs to its own — a the bank-mining stop enrichment moves only edit-wave snapshot, keeping draft-wave checkpoints valid
// («re-paid ONCE»). Returns a *WaveSignatureStop when the bank-mining stop stops for owner sign.
return r.translateBookWaves(ctx, chunks, stickySel)
res, err := r.translateBookWaves(ctx, chunks, stickySel)
if err == nil {
// End of the run (row 125). The stop paths refresh the read-out themselves, so this is the boundary
// they do not cover: a run that finished, whose last bank change was the auto-continue re-seed or a
// signed mined-delta loaded at start.
r.exportBank(ctx, "run-finished")
}
return res, err
}

1313
backend/internal/pipeline/contractblockers_test.go Normal file
View file

File diff suppressed because it is too large Load diff

17
backend/internal/pipeline/export.go
View file

@ -91,7 +91,11 @@ func (r *Runner) Export(pairs bool) (*BookExport, error) {
// output units — each a (chapter, firstChunkIdx) key + the unit's joined source (the --pairs column
// aligned byte-for-byte to how the edit wave concatenated the members). A draft-only pipeline has one output per
// draft chunk (the draft ships). Iterating units keeps a PENDING unit explicit and a GHOST row visible.
manifest, err := r.bookChunks()
// The persisted manifest serves the DEFAULT export (row 100): the projection joins stored rows against
// positions and prepends the chapter title, both of which the manifest carries. `--pairs` is the one
// mode that emits the SOURCE column, so it demands the full re-chunk — a text-free manifest would ship
// an empty src column to the FP-measure, which is worse than being slow.
manifest, err := r.exportChunks(pairs)
if err != nil {
return nil, err
}
@ -140,6 +144,17 @@ func (r *Runner) Export(pairs bool) (*BookExport, error) {
return exp, nil
}
// exportChunks resolves the manifest the export projects over: the persisted one when the export does
// not need source text, the full ingest+split when it does (`--pairs`). One helper so the "who needs
// text" rule lives in one place instead of being re-decided at each call.
func (r *Runner) exportChunks(pairs bool) ([]chunk.Chunk, error) {
if pairs {
return r.bookChunks()
}
chunks, _, err := r.readModelChunks()
return chunks, err
}
// exportGateMisses returns the CONFIRMED post-check miss count per chunk, or an empty map when the
// glossary post-check gate is off.
//

531
backend/internal/pipeline/manifest.go Normal file
View file

@ -0,0 +1,531 @@
package pipeline
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"os"
"strconv"
"textmachine/backend/internal/chunk"
"textmachine/backend/internal/lang"
"textmachine/backend/internal/text"
)
// manifest.go: the PERSISTED chapter/chunk manifest (backlog row 100).
//
// Two problems, one artifact. (1) Every $0 read re-ingested and re-cut the whole source to get its
// honest N/M denominator — 1.4–1.5 s of CPU per call on a 23 MB book, paid again on every poll of every
// book. (2) The chapter tree a reader needs has no carrier at all: the engine addresses chapters by a
// DENSE ordinal (chapters that yield no text do not consume a number, chunker.go), so editing the source
// renumbers every later chapter and any bookmark built on the number silently moves to another chapter.
//
// The artifact is a FILE beside the project DB, not a table, for the same reason the bank export is one
// (bankexport.go): the reader is another process which must not open the engine's private store
// (D39.85). It is also a projection and never a source — nothing reads it back into a decision the
// engine makes; a stale one is DETECTED and ignored, never trusted.
//
// STALENESS is decided by a KEY, not by the mtime and not by a flag. Everything whose change moves the
// chunk boundaries or the chapter set is hashed into it, so a manifest written under a different chunker,
// budget, langpack, embedded data set, encoding or source file cannot be mistaken for a current one. A
// write path rebuilds it unconditionally (so `translate --resnapshot`, which is the loud "the chunker
// moved" command, rebuilds it like every other run); a read path that finds a key mismatch falls back to
// the full re-chunk it did before this file existed. There is no state in which a wrong manifest is used.
// manifestVersion versions the DOCUMENT SHAPE. It also rides the key, so a shape change invalidates every
// stored manifest rather than being half-read by a binary expecting the other layout.
//
// v2 (acceptance V2-4): unit ids gained the cut tag — «<chapter>:<cutTag>:<idx>» instead of
// «<chapter>:<idx>». A v1 sidecar is structurally valid and would keep validating, handing a reader ids
// in the OLD form whose whole defect was that they outlive the cut. The bump is what refuses it.
const manifestVersion = "tm-manifest-v2"
// BookManifest is the persisted chapter/chunk structure of a book.
type BookManifest struct {
Version string `json:"manifest_version"`
BookID string `json:"book_id"`
// Key is the validity key (manifestKey). A reader that only wants "is this still the same book, cut
// the same way" compares this one string.
Key string `json:"key"`
// The key's INPUTS, spelled out beside it. The key alone answers "is it current"; these answer "what
// changed" when it is not, which is the question an operator staring at an invalidated tree asks.
ChunkerVersion string `json:"chunker_version"`
SourceSHA256 string `json:"source_sha256"`
SourceBytes int64 `json:"source_bytes"`
SourceLang string `json:"source_lang"`
TargetLang string `json:"target_lang"`
Encoding string `json:"encoding"`
Chapters []ManifestChapter `json:"chapters"`
ChaptersTotal int `json:"chapters_total"`
UnitsTotal int `json:"units_total"`
ChunksTotal int `json:"chunks_total"`
}
// ManifestChapter is one chapter: its stable identity, its DISPLAY ordinal, and its units.
type ManifestChapter struct {
// ID is the engine's STABLE key for this chapter: 16 hex chars of SHA-256 over the chapter's INGESTED
// text (see manifestChapterID). It is stable across a chunker/budget/heading-rule change and across
// edits to OTHER chapters — which is the whole point, since Number is not: numbering is dense, so
// inserting a chapter shifts every later ordinal while every later id stays put.
ID string `json:"id"`
// Number is the dense 1-based ordinal the engine addresses chunk_status rows by. It is a POSITION,
// not a key — see ID.
Number int `json:"number"`
// Heading is the title the engine RENDERS for this chapter from the pair's heading rule («Глава N»),
// or "" when the book has no rule or the chapter has no structural header. ⚠ It is the engine's own
// deterministic render — the source marker is stripped from the text the model sees and this string is
// glued back onto the chapter's first output unit at export (chunk.ApplyHeading). It is NOT a label
// carried by the book's own data, and a consumer must not present it as one. Which label a reader
// should see is an open contract question (companion §4, K-2/K-3), not settled here.
Heading string `json:"heading"`
UnitsTotal int `json:"units_total"`
ChunksTotal int `json:"chunks_total"`
Units []ManifestUnit `json:"units"`
}
// ManifestUnit is one OUTPUT unit — the granularity the engine ships and every read model counts in.
type ManifestUnit struct {
// ID is stable only as long as the CUT is, and it SAYS SO in its own bytes: it is the chapter id, the
// cut tag, and the unit's leader chunk index (manifestUnitID). A chunker/budget/pipeline-shape change
// re-cuts the chapter into different units, mints a new cut tag, and therefore mints a new id for
// EVERY unit of the book — including a chapter's first unit, whose leader index is 0 under any cut and
// which would otherwise carry an id that outlives the text it named.
//
// So: chapter identity survives a re-chunk, unit identity does not, and a consumer can tell the two
// apart by whether the id it holds still appears in the current manifest.
ID string `json:"id"`
// FirstChunkIdx is the LEADER chunk index — the (chapter, chunk_idx) the unit's shipping row is
// addressed at in chunk_status, and therefore the join key for every progress read.
FirstChunkIdx int `json:"first_chunk_idx"`
// ChunkCount is how many draft chunks the unit groups; members are the CONSECUTIVE indices
// [FirstChunkIdx, FirstChunkIdx+ChunkCount) of this chapter (chunk.assignEditUnits groups consecutive
// chunks, which is what makes storing a count instead of a list lossless).
ChunkCount int `json:"chunk_count"`
// EditUnitID is the engine's book-global edit-unit id for this unit.
EditUnitID int `json:"edit_unit_id"`
}
// manifestPath is where the manifest lands: beside the project DB, like every other book-state sidecar.
func (r *Runner) manifestPath() string { return r.Book.ProjectDB + ".manifest.json" }
// manifestChapterID is a chapter's stable identity: the first 64 bits of SHA-256 over its INGESTED text.
//
// WHY CONTENT AND NOT POSITION. The alternative — hashing the book id and the ordinal — is the ordinal
// with extra steps: inserting a chapter would re-point every later id at a different chapter's content,
// which is precisely the "open tabs and bookmarks quietly move" damage the contract companion names (§5).
// Content-derived, an edit to chapter 5 mints a new id for chapter 5 alone and every other chapter keeps
// the id it had, whatever its number became.
//
// DUPLICATES: two chapters with byte-identical text would hash the same, so the second and later
// occurrences carry an occurrence suffix. Deterministic for a given reading order, and honest about its
// own edge — deleting the FIRST of a duplicate pair promotes the second, changing its id.
func manifestChapterID(ingested string, occurrence int) string {
sum := sha256.Sum256([]byte(ingested))
id := hex.EncodeToString(sum[:])[:16]
if occurrence > 1 {
id += "-" + strconv.Itoa(occurrence)
}
return id
}
// manifestUnitID is a unit's identity: its chapter's id, the CUT it belongs to, and the leader chunk
// index.
//
// The cut tag is load-bearing and was missing in the first version, which made the declared boundary
// FALSE rather than merely coarse. Without it a unit id is «<chapter>:<firstChunkIdx>», and a chapter's
// FIRST unit is always index 0 — so «<chapter>:0» survived every re-chunk unchanged while pointing at a
// different span of text. A consumer that persists unit ids reads "the id is still there" as "my anchor
// is still valid", and lands on the wrong text with no signal. With the tag, ANY re-cut mints new unit
// ids across the whole book, which is exactly what ManifestUnit.ID's boundary says it does.
func manifestUnitID(chapterID, cutTag string, firstChunkIdx int) string {
return chapterID + ":" + cutTag + ":" + strconv.Itoa(firstChunkIdx)
}
// cutTag identifies the CUT — everything that decides where chunk boundaries fall and how chunks group
// into output units. It is a SHORT hash because it rides inside every unit id; the full identity of the
// manifest is the validity key, which folds these same inputs plus the source bytes and the book identity.
//
// The last three are the acceptance V2-3 catch and are the same class as the defect the tag exists to
// close: the langpack decides the heading rule (a header-only chapter can stop consuming a number), the
// embedded data feeds the CJK numerals and sentence terminators that place boundaries, and the
// normalization decides what the chunker is even splitting. Any of them moving re-cuts the book while
// chunker_version and the budget stand still — and a manifest rebuilt afterwards would stamp the same
// unit ids onto different text.
func (r *Runner) cutTag() string {
payload := struct {
Chunker string `json:"chunker_version"`
Segmentation segmentationSnap `json:"segmentation"`
ShippingWave string `json:"shipping_wave"`
Langpack string `json:"langpack_version,omitempty"`
Embedded string `json:"embedded_version"`
Norm string `json:"norm_version"`
}{chunkerVersion, r.segmentationSnapshot(), r.shippingWaveTag(),
r.packVersion(), lang.EmbeddedVersion(), text.NormVersion()}
data, err := json.Marshal(payload)
if err != nil {
panic(fmt.Sprintf("pipeline: manifest cut tag marshal: %v", err))
}
sum := sha256.Sum256(data)
return hex.EncodeToString(sum[:])[:8]
}
// shippingWaveTag names which wave owns the shipping stage. The UNIT decomposition depends on it
// (outputUnits groups whole chunks into edit units for an edit pipeline and emits one unit per chunk for
// a draft-only one), so it belongs to the cut for both the validity key and the unit id.
func (r *Runner) shippingWaveTag() string {
if r.finalStageWave() == waveDraft {
return "draft"
}
return "edit"
}
// sourceFingerprint is the source file's identity at one moment: the SHA-256 of its BYTES as they sit on
// disk — before decoding, before normalization — plus its size. Taken before AND after the split (see
// writeManifest), because a manifest whose STRUCTURE came from one version of the file and whose KEY came
// from another would validate forever while describing a book that no longer exists.
type sourceFingerprint struct {
SHA string
Bytes int64
}
// sourceSHA256 fingerprints the source file. Streamed, so a 23 MB book costs a buffer and not a copy of
// itself (measured: ~20 ms on the acceptance book, against ~1.4 s for the ingest+split it replaces).
func sourceSHA256(path string) (sourceFingerprint, error) {
f, err := os.Open(path)
if err != nil {
return sourceFingerprint{}, err
}
defer f.Close()
h := sha256.New()
n, err := io.Copy(h, f)
if err != nil {
return sourceFingerprint{}, err
}
return sourceFingerprint{SHA: hex.EncodeToString(h.Sum(nil)), Bytes: n}, nil
}
// sourceFingerprintBeforeIngest takes the fingerprint a manifest write must be built against, and returns
// nil (with a warning, never an error) when it cannot: the manifest is an accelerator, and a run must not
// die because its accelerator could not be stamped.
func (r *Runner) sourceFingerprintBeforeIngest(ctx context.Context) *sourceFingerprint {
fp, err := sourceSHA256(r.Book.SourceFile)
if err != nil {
r.Log.WarnContext(ctx, "manifest: could not fingerprint the source before reading it; the chapter/chunk manifest will not be written this run",
"source", r.Book.SourceFile, "err", err)
return nil
}
return &fp
}
// manifestKey hashes everything whose change makes a stored manifest wrong.
//
// The list is the set of inputs the chunk manifest is a function of: the source bytes, how they are
// decoded (encoding) and which language's rules segment them, the chunker's own version, the
// segmentation budget (a re-chunk), the language pack (its heading rule decides both the rendered
// heading and, for a header-only chapter, whether the chapter survives at all) and the embedded language
// data (CJK numerals/terminators feed chapter and sentence boundaries; source abbreviations feed the
// sentence splitter). text.NormVersion rides too — normalization decides what "the chapter's text" even
// is, and the chapter ids are hashes of exactly that.
//
// INGEST CODE ITSELF is covered by convention rather than by a version of its own: the project already
// treats an ingest-rule change as a chunkerVersion bump ("This is an ingest rule → covered by
// chunkerVersion", text/source.go). That convention is inherited here, not invented — but it IS a
// convention, so an ingest edit that forgets the bump leaves a stale manifest looking current, exactly as
// it leaves stale checkpoints looking current today.
func (r *Runner) manifestKey(src sourceFingerprint) string {
// The UNIT decomposition of the document is a function of the pipeline SHAPE, not only of the cut:
// outputUnits groups whole chunks into edit units for an edit pipeline and emits one unit per chunk
// for a draft-only one (waverun.go). Dropping the editor stage moves no other input here, so without
// this the stored units[] would keep validating while describing groups the run no longer ships.
payload := struct {
Version string `json:"manifest_version"`
BookID string `json:"book_id"`
SourceSHA string `json:"source_sha256"`
SourceBytes int64 `json:"source_bytes"`
Encoding string `json:"encoding"`
SourceLang string `json:"source_lang"`
TargetLang string `json:"target_lang"`
Chunker string `json:"chunker_version"`
Segmentation segmentationSnap `json:"segmentation"`
Langpack string `json:"langpack_version,omitempty"`
Embedded string `json:"embedded_version"`
Norm string `json:"norm_version"`
ShippingWave string `json:"shipping_wave"`
}{
Version: manifestVersion, BookID: r.Book.BookID,
SourceSHA: src.SHA, SourceBytes: src.Bytes,
Encoding: r.Book.Encoding, SourceLang: r.Book.SourceLang, TargetLang: r.Book.TargetLang,
Chunker: chunkerVersion, Segmentation: r.segmentationSnapshot(),
Langpack: r.packVersion(), Embedded: lang.EmbeddedVersion(), Norm: text.NormVersion(),
ShippingWave: r.shippingWaveTag(),
}
data, err := json.Marshal(payload)
if err != nil {
// A struct of scalars and a fixed nested struct cannot fail to marshal; a key that could silently
// become "" would make every manifest look valid, so refuse to produce one instead.
panic(fmt.Sprintf("pipeline: manifest key marshal: %v", err))
}
sum := sha256.Sum256(data)
return hex.EncodeToString(sum[:])
}
// buildManifest projects a fresh split into the document. chapterTexts is index-aligned to chapter
// numbers (chunk.SplitChunksWithChapters), chunks is the split itself. Pure and deterministic.
func (r *Runner) buildManifest(chapterTexts []string, chunks []chunk.Chunk, src sourceFingerprint) *BookManifest {
m := &BookManifest{
Version: manifestVersion, BookID: r.Book.BookID,
Key: r.manifestKey(src),
ChunkerVersion: chunkerVersion,
SourceSHA256: src.SHA, SourceBytes: src.Bytes,
SourceLang: r.Book.SourceLang, TargetLang: r.Book.TargetLang, Encoding: r.Book.Encoding,
ChunksTotal: len(chunks),
}
// Chapter ids first, so a unit can name its chapter's id. The occurrence counter makes duplicate
// chapter texts distinguishable (manifestChapterID).
occurrence := map[string]int{}
ids := make([]string, len(chapterTexts))
for i, txt := range chapterTexts {
occurrence[txt]++
ids[i] = manifestChapterID(txt, occurrence[txt])
}
// A chapter's title lives on its FIRST chunk only (chunker.go), so it is collected off the chunk list
// rather than off the unit list — a unit is a group of chunks and knowing which of them is the
// chapter's opener is the chunker's business, not this projection's.
headings := map[int]string{}
for _, c := range chunks {
if c.Heading != "" {
headings[c.Chapter] = c.Heading
}
}
cut := r.cutTag() // rides every unit id: a re-cut must mint new ones (manifestUnitID)
at := map[int]int{} // chapter number → index into m.Chapters (units arrive in book order)
for _, u := range r.outputUnits(chunks) {
i, ok := at[u.Chapter]
if !ok {
// Chapter numbers are dense and index-aligned to chapterTexts by construction (the chunker
// appends a text exactly when it consumes a number). If that ever stops holding, fall back to a
// number-derived id and say so: an EMPTY id would make every unit id of every such chapter
// collide on ":<idx>", which is worse than an id that is merely position-based.
id := ""
if k := u.Chapter - 1; k >= 0 && k < len(ids) {
id = ids[k]
} else {
id = "n" + strconv.Itoa(u.Chapter)
r.Log.Warn("manifest: no ingested text for a chapter the split emitted — its id falls back to the (unstable) chapter number",
"book", r.Book.BookID, "chapter", u.Chapter, "chapter_texts", len(ids))
}
m.Chapters = append(m.Chapters, ManifestChapter{ID: id, Number: u.Chapter, Heading: headings[u.Chapter]})
i = len(m.Chapters) - 1
at[u.Chapter] = i
}
c := &m.Chapters[i]
c.UnitsTotal++
c.ChunksTotal += len(u.Members)
c.Units = append(c.Units, ManifestUnit{
ID: manifestUnitID(c.ID, cut, u.FirstChunkIdx),
FirstChunkIdx: u.FirstChunkIdx, ChunkCount: len(u.Members), EditUnitID: u.EditUnitID,
})
m.UnitsTotal++
}
m.ChaptersTotal = len(m.Chapters)
return m
}
// persistManifest writes the manifest for a fresh split. LOUD BUT NOT FATAL: the manifest is a read
// accelerator and a reader's tree, not an input to anything the run decides — failing a paid run over it
// would trade money for an artifact the next run rewrites. A reader that finds no manifest (or a stale
// one) falls back to the full re-chunk, so the failure degrades to the pre-existing behaviour.
func (r *Runner) persistManifest(ctx context.Context, before *sourceFingerprint, chapterTexts []string, chunks []chunk.Chunk) {
if before == nil {
return // sourceFingerprintBeforeIngest already said why
}
if _, err := r.writeManifest(*before, chapterTexts, chunks); err != nil {
r.Log.WarnContext(ctx, "manifest: could not persist the chapter/chunk manifest; read paths fall back to re-chunking the source and the chapter tree may be stale",
"path", r.manifestPath(), "err", err)
}
}
// writeManifest is persistManifest's fallible half — separate so a caller that MUST know (the manifest
// command) can report the failure instead of logging it. It returns the document it wrote, so a caller
// that also wants to render it does not build and hash the book a second time.
//
// `before` is the fingerprint taken BEFORE the source was read, and the re-fingerprint here is what makes
// the validity key mean what it claims. Hashing only afterwards would stamp the structure derived from
// one version of the file with the identity of another: the ingest+split of a 23 MB book takes ~1.4 s,
// and a source that is rewritten inside that window (a re-upload, an operator's edit) would produce a
// document that describes the OLD cut, matches its own key forever, and can never be detected as stale.
// A source that moved under the read is therefore not written at all — the next write path produces a
// consistent one, and until then readers take the full re-chunk.
func (r *Runner) writeManifest(before sourceFingerprint, chapterTexts []string, chunks []chunk.Chunk) (*BookManifest, error) {
after, err := sourceSHA256(r.Book.SourceFile)
if err != nil {
return nil, fmt.Errorf("pipeline: re-hash source %s for the manifest: %w", r.Book.SourceFile, err)
}
if after != before {
return nil, fmt.Errorf("pipeline: the source %s changed while it was being read (%.12s/%d bytes → %.12s/%d bytes): the manifest is NOT written, because a structure cut from the old bytes stamped with the new hash would validate forever",
r.Book.SourceFile, before.SHA, before.Bytes, after.SHA, after.Bytes)
}
m := r.buildManifest(chapterTexts, chunks, before)
body, err := json.MarshalIndent(m, "", " ")
if err != nil {
return nil, fmt.Errorf("pipeline: marshal the manifest: %w", err)
}
if err := writeFileAtomic(r.manifestPath(), append(body, '\n')); err != nil {
return nil, err
}
return m, nil
}
// BuildAndPersistManifest re-derives the split and persists the manifest ($0: ingest + chunker, no LLM,
// no provider key, no wave). It is the producer behind `tmctl manifest` — the command a caller
// runs to get a chapter tree for a book that has never been translated, which is a state the library has
// to be able to show and which no other engine surface produces.
func (r *Runner) BuildAndPersistManifest() (*BookManifest, error) {
// Fingerprint BEFORE the read, so writeManifest can prove the structure and the identity came from the
// same bytes. Here a fingerprint failure IS an error: producing the artifact is the command's job.
before, err := sourceSHA256(r.Book.SourceFile)
if err != nil {
return nil, fmt.Errorf("pipeline: hash source %s for the manifest: %w", r.Book.SourceFile, err)
}
doc, err := chunk.IngestEncoded(r.Book.SourceFile, r.Book.Encoding, r.Book.SourceLang)
if err != nil {
return nil, err
}
chunks, chapterTexts := chunk.SplitChunksWithChapters(doc.Chapters, r.segBudget(), r.headingRule(), r.sentenceAbbrevs())
if len(chunks) == 0 {
return nil, fmt.Errorf("pipeline: source file %s produced no chunks after normalization", r.Book.SourceFile)
}
return r.writeManifest(before, chapterTexts, chunks)
}
// ManifestPath exposes where the artifact lives, for a CLI that has to tell the operator.
func (r *Runner) ManifestPath() string { return r.manifestPath() }
// loadManifest returns the persisted manifest IF it is present and still describes this book, cut this
// way. Every other outcome — absent, unreadable, malformed, wrong shape, stale key — returns nil, and the
// caller re-chunks. A read failure is deliberately not an error to the caller: the manifest is an
// accelerator, and a broken accelerator must degrade to the slow path, never to a wrong answer.
func (r *Runner) loadManifest() *BookManifest {
raw, err := os.ReadFile(r.manifestPath())
if err != nil {
if !errors.Is(err, fs.ErrNotExist) {
r.Log.Warn("manifest: could not read the manifest; falling back to re-chunking the source", "path", r.manifestPath(), "err", err)
}
return nil
}
var m BookManifest
if err := json.Unmarshal(raw, &m); err != nil {
r.Log.Warn("manifest: the manifest is not readable JSON; falling back to re-chunking the source", "path", r.manifestPath(), "err", err)
return nil
}
if m.Version != manifestVersion {
r.Log.Warn("manifest: the manifest was written in another document version; falling back to re-chunking the source",
"path", r.manifestPath(), "stored", m.Version, "want", manifestVersion)
return nil
}
// Internal sanity BEFORE the key: the key is a stored string, so a hand-edited sidecar (they sit in the
// directory operators open) keeps matching while its counters say anything at all. A negative capacity
// panics `make`, and an absurd one allocates the machine out of memory — inside a $0 read command.
if !m.selfConsistent() {
r.Log.Warn("manifest: the manifest's own counters do not describe its own contents; falling back to re-chunking the source",
"path", r.manifestPath(), "chapters_total", m.ChaptersTotal, "units_total", m.UnitsTotal, "chunks_total", m.ChunksTotal)
return nil
}
src, err := sourceSHA256(r.Book.SourceFile)
if err != nil {
r.Log.Warn("manifest: could not hash the source to validate the manifest; falling back to re-chunking the source", "err", err)
return nil
}
if want := r.manifestKey(src); m.Key != want {
// This is the normal, expected path after a source edit or a chunker/pack/budget change — say what
// happened at INFO-level volume rather than as a fault, but say it: a chapter tree whose ids just
// changed is a fact a reader has to be told about (a re-anchor, not a refresh).
r.Log.Info("manifest: the stored manifest is stale (source or cut changed); re-chunking the source and rebuilding it on the next write path",
"path", r.manifestPath(), "stored_key", m.Key, "current_key", want)
return nil
}
return &m
}
// selfConsistent reports whether the document's own counters describe its own contents. It is a guard
// against a HAND-EDITED file, not against a corrupt one (a corrupt one fails the JSON parse or the key):
// the counters are read straight into slice capacities and loop bounds below, so a negative or absurd
// number would panic or exhaust memory inside a read-only command.
func (m *BookManifest) selfConsistent() bool {
if m.ChaptersTotal != len(m.Chapters) {
return false
}
units, chunks := 0, 0
for _, c := range m.Chapters {
if c.Number < 1 || c.UnitsTotal != len(c.Units) {
return false
}
for _, u := range c.Units {
if u.ChunkCount < 1 || u.ChunkCount > manifestMaxChunks || u.FirstChunkIdx < 0 {
return false
}
chunks += u.ChunkCount
}
units += len(c.Units)
}
// The UPPER bound matters as much as the lower one: a file edited so that ChunksTotal and the per-unit
// counts agree at an absurd value passes every equality above and then allocates that many chunks.
if chunks > manifestMaxChunks {
return false
}
return m.UnitsTotal == units && m.ChunksTotal == chunks
}
// manifestMaxChunks bounds what a manifest may claim to describe. It is a sanity ceiling, not a product
// limit: the 23 MB acceptance book cuts into ~5 000 chunks, so this is three orders of magnitude above
// any real book and far below the value at which the reconstruction would exhaust memory.
const manifestMaxChunks = 10_000_000
// chunks reconstructs the manifest's chunk list. Safe to call only on a document that passed
// selfConsistent (loadManifest is the only producer).
//
// ⚠ THE CHUNKS CARRY NO SOURCE TEXT. The manifest stores STRUCTURE — a 23 MB book's text is the source
// file, and duplicating it into a sidecar would trade the CPU this file saves for the same amount of
// disk and write time. Every field the read models actually join on is here (chapter, chunk index, edit
// unit, the chapter title). Three read-side consumers genuinely need the text and therefore take the full
// re-chunk instead: `export --pairs` (it emits the source column), the re-bill projection's content
// re-render (it re-renders injected bytes), and `QualityReport` (`quality.go` — its residual scan is a
// comparison against the unit's source, so `tmctl report` still re-cuts the book).
func (m *BookManifest) chunks() []chunk.Chunk {
out := make([]chunk.Chunk, 0, m.ChunksTotal)
for _, c := range m.Chapters {
for _, u := range c.Units {
for i := 0; i < u.ChunkCount; i++ {
ch := chunk.Chunk{
Chapter: c.Number, ChunkIdx: u.FirstChunkIdx + i, EditUnitID: u.EditUnitID,
}
if ch.ChunkIdx == 0 {
ch.Heading = c.Heading // only a chapter's first chunk carries the title (chunker.go)
}
out = append(out, ch)
}
}
}
return out
}
// readModelChunks is what the $0 read models join against: the persisted manifest when it is current, the
// full ingest+split otherwise.
//
// The second result is a PROVIDER of the same chunks WITH their source text, not a flag — a flag that
// every caller discarded would document a guard the code does not keep. On the manifest path it is the
// full re-chunk (paid only if some branch actually needs text); on the fallback path it hands back the
// slice that was just materialized, so a read that already re-cut the book never re-cuts it twice.
func (r *Runner) readModelChunks() (chunks []chunk.Chunk, withText func() ([]chunk.Chunk, error), err error) {
if m := r.loadManifest(); m != nil {
return m.chunks(), r.bookChunks, nil
}
full, err := r.bookChunks()
return full, func() ([]chunk.Chunk, error) { return full, err }, err
}

197
backend/internal/pipeline/mining.go
View file

@ -2,6 +2,7 @@ package pipeline
import (
"context"
"encoding/json"
"errors"
"fmt"
"io/fs"
@ -48,6 +49,11 @@ func (r *Runner) runBankMiningStop(ctx context.Context, chunks []chunk.Chunk, dr
return false, fmt.Errorf("pipeline: --verify-bank was passed but this book cannot mine a bank: it needs BOTH a langpack (book.yaml `langpack_root`, currently %s) and a mining contrast artifact (pipeline.yaml `mining.contrast_path`, currently %q); without them there is no candidate list to verify",
packStateLabel(r.pack), r.Pipeline.Mining.ContrastPath)
}
// A book that cannot mine has no signing table — and if one is lying beside its DB from when it
// could (a langpack or contrast path removed from the config), it would outlive every run and keep
// offering terms to a screen forever. Clear it, but only when it exists: a book that never mines
// must not acquire a sidecar it has no use for.
r.clearBankStopTableJSON(ctx, "mining is not configured for this book")
return false, nil // mining not configured → auto-continue to the edit wave
}
f, err := os.Open(r.Pipeline.Mining.ContrastPath)
@ -146,6 +152,17 @@ func (r *Runner) runBankMiningStop(ctx context.Context, chunks []chunk.Chunk, dr
} else {
r.Log.InfoContext(ctx, msg, args...)
}
// The MACHINE table is rewritten EMPTY here, and that is the difference between it and the human one
// (row 101). An empty delta means "nothing awaits signature"; leaving the previous run's rows on disk
// would let a signing screen re-offer terms the owner has just signed, with no field in the document
// to tell that state from a live one. The text sidecar keeps its prior behaviour deliberately — it is
// the document a human re-reads, and a human knows which run they are looking at.
//
// Written unconditionally here (unlike the not-configured path above): mining DID run, so "nothing
// awaits signature" is this run's own finding and the artifact should state it.
if werr := r.writeBankStopTableJSON(nil); werr != nil {
r.Log.WarnContext(ctx, "bank-mining: could not clear the machine stop table sidecar; it still holds the PREVIOUS run's rows", "err", werr)
}
return false, nil
}
consolidated, classified, tres, err := r.runTerminologist(ctx, draftSnapshot, cands)
@ -178,6 +195,13 @@ func (r *Runner) runBankMiningStop(ctx context.Context, chunks []chunk.Chunk, dr
if werr := os.WriteFile(r.bankStopTablePath(), []byte(renderBankStopTable(rows)), 0o644); werr != nil {
r.Log.WarnContext(ctx, "bank-mining: could not write the stop table sidecar (the signature map is unaffected)", "err", werr)
}
// …and the SAME table as a machine surface (backlog row 101). Same rows, same order, same fields — the
// text sidecar stays the human document and this one is what a signing screen is built from, so the two
// can never describe the bank differently. Same failure discipline as the text sidecar: a sidecar that
// cannot be written must not take a paid run down with it.
if werr := r.writeBankStopTableJSON(rows); werr != nil {
r.Log.WarnContext(ctx, "bank-mining: could not write the machine stop table sidecar (the signature map and the text table are unaffected)", "err", werr)
}
// THE FLAG (D39.42 п.5, owner's words: «можно запустить перевод так, чтоб сессия не останавливалась и
// не запрашивала верификацию банка, а просто как намайнит и закончит — шла в редактуру»). Default is
@ -197,9 +221,13 @@ func (r *Runner) runBankMiningStop(ctx context.Context, chunks []chunk.Chunk, dr
if err := r.seedGlossary(ctx); err != nil {
return false, fmt.Errorf("pipeline: re-seed the bank with the auto rows: %w", err)
}
// The bank just changed (row 125): the auto rows are IN it now, so the read-out has to say so
// before the edit wave starts translating against them.
r.exportBank(ctx, "bank-mining/auto-continue")
r.Log.InfoContext(ctx, "bank-mining: auto-continuing with an UNSIGNED bank (pass --verify-bank to stop and review it)",
append([]any{"book", r.Book.BookID, "terms", len(mined), "terms_with_proposed_dst", withDst,
"auto_bank", r.autoBankPath(), "signature_map", r.signatureMapPath(), "table", r.bankStopTablePath()},
"auto_bank", r.autoBankPath(), "signature_map", r.signatureMapPath(),
"table", r.bankStopTablePath(), "table_json", r.bankStopTableJSONPath()},
emissionArgs(emission, cands)...)...)
return false, nil
}
@ -209,6 +237,9 @@ func (r *Runner) runBankMiningStop(ctx context.Context, chunks []chunk.Chunk, dr
return false, nil
}
r.lastBankStopRows = rows
// The signature stop is the boundary the signing screen reads at (row 125): refresh the bank read-out
// so the state behind the decisions is the state as of this stop, not as of the last run.
r.exportBank(ctx, "bank-mining/signature-stop")
r.Log.WarnContext(ctx, "bank-mining: new terms await owner signature; run STOPPED before the edit wave (review the signature map, then for EACH term either promote it into the mined-delta file OR decline it in the mined-rejects file, then resume — the stop clears once every proposed term is promoted or rejected)",
append([]any{"book", r.Book.BookID, "terms", len(mined), "terms_with_proposed_dst", withDst,
"signature_map", r.signatureMapPath()}, emissionArgs(emission, cands)...)...)
@ -250,6 +281,118 @@ func (r *Runner) bankStopTablePath() string {
return r.Book.ProjectDB + ".bank-stop.txt"
}
// bankStopTableJSONPath is the MACHINE sidecar of the same table (backlog row 101). It sits beside the
// text one for the same reason that one sits beside the signature map: a loadable seed YAML cannot carry
// evidence, and a table meant to be parsed cannot be a column layout. Deliberately a SEPARATE file rather
// than a replacement — the text table is the document a human reads at 3 a.m. with `less`.
func (r *Runner) bankStopTableJSONPath() string {
return r.Book.ProjectDB + ".bank-stop.json"
}
// bankStopTableVersion versions the SHAPE of the machine table, so a consumer that must tolerate a field
// set changing under it has something to branch on. It is not a snapshot input: the table is evidence
// produced from stored answers, never a paid byte (the same standing the parse rule has, see above).
const bankStopTableVersion = "tm-bank-stop-v1"
// bankStopTableFile is the machine table's document shape.
type bankStopTableFile struct {
Version string `json:"table_version"`
BookID string `json:"book_id"`
Terms int `json:"terms"`
Rows []bankStopRowJSON `json:"rows"`
}
// bankStopRowJSON is ONE row of the machine table — the same fields the text table prints, with the two
// places where a human-facing rendering would lose information made explicit:
//
// - Conf is a POINTER: the role's own confidence is absent for a term it never mentioned, and "absent"
// and "0% sure" are opposites (mining.go confOrAbsent). The text table omits the word `confidence`
// for the first case; JSON says null, and a consumer that reads 0 for both would sort the sheet
// exactly wrong.
// - every list is emitted as a list, never a joined string, because the text table's separators (", "
// / " | " / "; ") occur inside real renderings and a consumer cannot split them back safely.
//
// It is a DTO rather than JSON tags on BankStopRow on purpose: BankStopRow is the CLI's in-memory row and
// its field names are free to follow the code, while these names are a published surface.
type bankStopRowJSON struct {
Src string `json:"src"`
Dst string `json:"dst"` // "" when nothing was consolidated for this term
Origin string `json:"origin"`
Type string `json:"type"`
Freq int `json:"freq"`
Spread int `json:"spread"`
Conventions int `json:"conventions"`
Conf *int `json:"conf"`
Invented bool `json:"invented"`
Signals []string `json:"signals"`
Contradicts []string `json:"contradicts"`
Variants []BankStopVariant `json:"variants"`
Evidence []string `json:"evidence"`
Contexts []string `json:"contexts"`
}
// bankStopTableJSON projects the rows into the machine document. Deterministic: the row order is the
// caller's (source-key order, the reference order the text sidecar also uses), and every empty list is
// rendered as `[]` rather than `null` so a consumer has one shape to handle instead of two.
func bankStopTableJSON(bookID string, rows []BankStopRow) bankStopTableFile {
out := bankStopTableFile{Version: bankStopTableVersion, BookID: bookID, Terms: len(rows), Rows: make([]bankStopRowJSON, 0, len(rows))}
for _, row := range rows {
jr := bankStopRowJSON{
Src: row.Src, Dst: row.Dst, Origin: row.Origin, Type: row.Type,
Freq: row.Freq, Spread: row.Spread, Conventions: row.Conventions, Invented: row.Invented,
Signals: emptyIfNil(row.Signals), Contradicts: emptyIfNil(row.Contradicts),
Variants: emptyVariantsIfNil(row.Variants), Evidence: emptyIfNil(row.Evidence), Contexts: emptyIfNil(row.Contexts),
}
if row.Conf >= 0 {
conf := row.Conf
jr.Conf = &conf
}
out.Rows = append(out.Rows, jr)
}
return out
}
// emptyVariantsIfNil is emptyIfNil for the variant list — one helper per element type keeps the render
// deterministic without reaching for generics, the same trade render.go's sortedKeys pair already makes.
func emptyVariantsIfNil(v []BankStopVariant) []BankStopVariant {
if v == nil {
return []BankStopVariant{}
}
return v
}
// emptyIfNil renders a nil slice as an empty one (see bankStopRowJSON).
func emptyIfNil(s []string) []string {
if s == nil {
return []string{}
}
return s
}
// clearBankStopTableJSON empties an EXISTING machine stop table, and does nothing when there is none —
// the difference between "this book has nothing awaiting signature" (a statement worth making) and "this
// book has no signing surface at all" (nothing to say, and a file nobody asked for).
func (r *Runner) clearBankStopTableJSON(ctx context.Context, why string) {
if _, err := os.Stat(r.bankStopTableJSONPath()); err != nil {
return // absent (or unreadable — then rewriting it is not this path's business either)
}
if werr := r.writeBankStopTableJSON(nil); werr != nil {
r.Log.WarnContext(ctx, "bank-mining: could not clear the stale machine stop table sidecar; it still offers terms for signature",
"path", r.bankStopTableJSONPath(), "reason", why, "err", werr)
return
}
r.Log.InfoContext(ctx, "bank-mining: cleared the machine stop table sidecar", "path", r.bankStopTableJSONPath(), "reason", why)
}
// writeBankStopTableJSON serializes and atomically replaces the machine table sidecar.
func (r *Runner) writeBankStopTableJSON(rows []BankStopRow) error {
body, err := json.MarshalIndent(bankStopTableJSON(r.Book.BookID, rows), "", " ")
if err != nil {
return fmt.Errorf("pipeline: marshal the machine bank-stop table: %w", err)
}
return writeFileAtomic(r.bankStopTableJSONPath(), append(body, '\n'))
}
// BankStopRow is one row of the bank-verification table the stop shows the operator — the table D39.36
// specified and the CLI never had (it printed a term count and a path). Exported because the CLI renders it.
type BankStopRow struct {
@ -257,10 +400,10 @@ type BankStopRow struct {
Dst string // the consolidated rendering, or "" when nothing was consolidated
Origin string // mined | banknote | both — WHICH channel found it
Type string
Freq int // occurrences in the source
Spread int // how many DISTINCT renderings the drafts produced (the disagreement signal)
Variants []string // "rendering ×N", best-ranked first
Contexts []string // source KWIC
Freq int // occurrences in the source
Spread int // how many DISTINCT renderings the drafts produced (the disagreement signal)
Variants []BankStopVariant // the renderings the drafts produced, best-ranked first
Contexts []string // source KWIC
Evidence []string
// The §G3 arbitration record: until this pack, «why does this term have THIS dst» was unanswerable from
// the artifacts (research/24 §A7). Every field below is read off work the ranking already did.
@ -281,6 +424,42 @@ type BankStopRow struct {
Contradicts []string
}
// BankStopVariant is ONE rendering the drafts produced, kept in its PARTS rather than as the sentence a
// table prints. The renderers format it (Label); the machine sidecar ships the parts. Before this the row
// carried the pre-rendered «<dst> ×<n> (proposed for <via>)» string and nothing else, so the machine
// surface could only be un-parsed by splitting on «×» and on a parenthesis — both of which occur inside
// real renderings.
// Its JSON tags ARE the published shape of the machine sidecar: unlike the row, whose field names in the
// file are deliberately independent of the code's, a variant's three parts are called the same thing on
// both sides, and a second type to say so would only be a conversion waiting to drift.
type BankStopVariant struct {
Dst string `json:"dst"`
Chunks int `json:"chunks"` // how many draft chunks proposed it
// Via names the ALIAS this rendering was proposed for, "" for a direct proposal. Load-bearing rather
// than decoration: an alias-routed rendering must never silently become the cluster owner's dst
// (proposalsFromCandidates), so a surface that offers one has to be able to say whose it was.
Via string `json:"via,omitempty"`
}
// Label is the one rendering of a variant every human-facing table uses, so the capped stdout view and
// the text sidecar cannot describe a variant differently.
func (v BankStopVariant) Label() string {
s := fmt.Sprintf("%s ×%d", v.Dst, v.Chunks)
if v.Via != "" {
s += " (proposed for " + v.Via + ")"
}
return s
}
// VariantLabels renders a row's variants for a text table.
func (r BankStopRow) VariantLabels() []string {
out := make([]string, 0, len(r.Variants))
for _, v := range r.Variants {
out = append(out, v.Label())
}
return out
}
// bankStopRows projects the merged candidates into the operator table, best-ranked variants first.
// Deterministic: cands is key-ordered and nothing here iterates a map for output.
func bankStopRows(cands []terminology.Candidate, consolidated map[string]string, tres terminologyResult) []BankStopRow {
@ -294,11 +473,7 @@ func bankStopRows(cands []terminology.Candidate, consolidated map[string]string,
Conf: confOrAbsent(tres.Conf, c.Key), Contradicts: tres.Contradictions[c.Src],
}
for i, v := range c.Variants {
label := fmt.Sprintf("%s ×%d", v.Dst, v.Chunks)
if v.Via != "" {
label += " (proposed for " + v.Via + ")"
}
row.Variants = append(row.Variants, label)
row.Variants = append(row.Variants, BankStopVariant{Dst: v.Dst, Chunks: v.Chunks, Via: v.Via})
if i == 0 {
row.Signals = v.Signals
}
@ -355,7 +530,7 @@ func renderBankStopTable(rows []BankStopRow) string {
fmt.Fprintf(&b, " CONTRADICTS this run's own: %s\n", strings.Join(r.Contradicts, "; "))
}
if len(r.Variants) > 0 {
fmt.Fprintf(&b, " drafts: %s\n", strings.Join(r.Variants, " | "))
fmt.Fprintf(&b, " drafts: %s\n", strings.Join(r.VariantLabels(), " | "))
}
if len(r.Evidence) > 0 {
fmt.Fprintf(&b, " evidence: %s\n", strings.Join(r.Evidence, ", "))

3
backend/internal/pipeline/miningstop_join_test.go
View file

@ -11,6 +11,7 @@ import (
"strings"
"testing"
"textmachine/backend/internal/chunk"
"textmachine/backend/internal/chunk/chunktest"
"textmachine/backend/internal/config"
"textmachine/backend/internal/lang"
@ -1594,7 +1595,7 @@ func TestPointwiseReEditOnlyPaysForTheUnitsTheTermTouches(t *testing.T) {
if err != nil {
t.Fatal(err)
}
proj, err := rp.projectRebill(statuses, chunks)
proj, err := rp.projectRebill(statuses, chunks, func() ([]chunk.Chunk, error) { return chunks, nil })
if err != nil {
t.Fatal(err)
}

23
backend/internal/pipeline/rebill.go
View file

@ -92,12 +92,19 @@ type RebillProjection struct {
//
// The wave snapshots are rendered LAZILY, so a draft-only pipeline never renders an edit-wave snapshot
// it has no stages for.
func (r *Runner) projectRebill(statuses []store.ChunkStatus, chunks []chunk.Chunk) (RebillProjection, error) {
//
// `manifest` supplies only the POSITIONS still in the book (the membership test above), which is why it
// may be the text-free projection of the persisted manifest (manifest.go). `withText` supplies the same
// chunks WITH their source, and is called only on the content-aware branch below — the one that has to
// re-render injected bytes and therefore genuinely needs the text. Splitting the two is what lets a
// status read skip a 1.4 s re-chunk it almost never needs, without ever computing a content hash over an
// empty string.
func (r *Runner) projectRebill(statuses []store.ChunkStatus, manifest []chunk.Chunk, withText func() ([]chunk.Chunk, error)) (RebillProjection, error) {
var p RebillProjection
decider := newRepinDecider(r)
var contentHashes map[chunkKey]map[string]string
live := make(map[chunkKey]bool, len(chunks))
for _, ch := range chunks {
live := make(map[chunkKey]bool, len(manifest))
for _, ch := range manifest {
live[chunkKey{ch.Chapter, ch.ChunkIdx}] = true
}
draftNames := stageNameSet(r.waveStagesIndexed(waveDraft))
@ -144,7 +151,11 @@ func (r *Runner) projectRebill(statuses []store.ChunkStatus, chunks []chunk.Chun
}
if bankOnly {
if contentHashes == nil { // rendered once, lazily: a book with no bank move never pays for it
contentHashes = r.renderedContentHashes(chunks, precomputeSticky(chunks, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget))
full, ferr := withText()
if ferr != nil {
return p, fmt.Errorf("pipeline: re-chunk the source for the re-bill content check: %w", ferr)
}
contentHashes = r.renderedContentHashes(full, precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget))
}
if h, ok := contentHashes[chunkKey{cs.Chapter, cs.ChunkIdx}][cs.Stage]; ok && h == cs.ContentHash {
p.Repinned++
@ -217,7 +228,9 @@ func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk) e
if err != nil {
return fmt.Errorf("pipeline: read chunk_status for the re-bill projection: %w", err)
}
proj, err := r.projectRebill(statuses, chunks)
// A write path already holds the real split, so the lazy provider just hands it back — no second
// ingest, and no branch where the consent gate could see text-free chunks.
proj, err := r.projectRebill(statuses, chunks, func() ([]chunk.Chunk, error) { return chunks, nil })
if err != nil {
return err
}

12
backend/internal/pipeline/rebill_test.go
View file

@ -290,7 +290,7 @@ func TestRebillProjectionIsPerWave(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if proj, err := r1.projectRebill(statusesBefore, chunksOf(t, r1)); err != nil || proj.Rows != 0 {
if proj, err := r1.projectRebill(statusesBefore, chunksOf(t, r1), func() ([]chunk.Chunk, error) { return chunksOf(t, r1), nil }); err != nil || proj.Rows != 0 {
t.Fatalf("a freshly-run book projects no re-bill; each row must be judged against ITS OWN wave, got %+v (err=%v)", proj, err)
}
r1.Close()
@ -312,7 +312,7 @@ func TestRebillProjectionIsPerWave(t *testing.T) {
if err != nil {
t.Fatal(err)
}
proj, err := r2.projectRebill(statuses, chunksOf(t, r2))
proj, err := r2.projectRebill(statuses, chunksOf(t, r2), func() ([]chunk.Chunk, error) { return chunksOf(t, r2), nil })
if err != nil {
t.Fatal(err)
}
@ -358,7 +358,7 @@ func TestRebillProjectionExcludesSkippedAndUnchanged(t *testing.T) {
t.Fatal("setup: the refusal fixture must leave a skipped edit row")
}
// No drift yet: nothing is superseded, so nothing is projected.
if proj, err := r1.projectRebill(statuses, chunksOf(t, r1)); err != nil || proj.Rows != 0 || proj.USD != 0 {
if proj, err := r1.projectRebill(statuses, chunksOf(t, r1), func() ([]chunk.Chunk, error) { return chunksOf(t, r1), nil }); err != nil || proj.Rows != 0 || proj.USD != 0 {
t.Fatalf("an undrifted book must project no re-bill, got %+v (err=%v)", proj, err)
}
r1.Close()
@ -370,7 +370,7 @@ func TestRebillProjectionExcludesSkippedAndUnchanged(t *testing.T) {
if err := r2.seedGlossary(ctx); err != nil {
t.Fatal(err)
}
proj, err := r2.projectRebill(statuses, chunksOf(t, r2))
proj, err := r2.projectRebill(statuses, chunksOf(t, r2), func() ([]chunk.Chunk, error) { return chunksOf(t, r2), nil })
if err != nil {
t.Fatal(err)
}
@ -408,7 +408,7 @@ func TestRebillProjectionIgnoresRetiredStages(t *testing.T) {
if err != nil {
t.Fatal(err)
}
proj, err := r.projectRebill(statuses, chunksOf(t, r))
proj, err := r.projectRebill(statuses, chunksOf(t, r), func() ([]chunk.Chunk, error) { return chunksOf(t, r), nil })
if err != nil {
t.Fatal(err)
}
@ -450,7 +450,7 @@ func TestRebillProjectionIgnoresVanishedChunks(t *testing.T) {
if err := r2.seedGlossary(ctx); err != nil {
t.Fatal(err)
}
proj, err := r2.projectRebill(statuses, chunksOf(t, r2))
proj, err := r2.projectRebill(statuses, chunksOf(t, r2), func() ([]chunk.Chunk, error) { return chunksOf(t, r2), nil })
if err != nil {
t.Fatal(err)
}

30
backend/internal/pipeline/runner.go
View file

@ -61,6 +61,21 @@ type Runner struct {
// carries it forward marked-unverified, and finishes — «просто как намайнит и закончит». It is an
// operator MODE, not a wire or snapshot axis: it changes when a human is asked, never what is sent.
VerifyBank bool
// CeilingUSD is the BOOK ceiling for THIS RUN ONLY (tmctl --ceiling-usd, backlog row 145 / D39.110),
// 0 = unset. It OVERRIDES book.yaml's ceilings.book_usd for the duration of the process and is never
// written back: the caller's number stays a property of the run, so a platform can cap a run without
// editing the engine's book config (the zone mixing D39.81/D39.85 forbid).
//
// ⚠ It is a BOOK ceiling, not a run BUDGET: the ledger compares it against the book's cumulative
// committed+reserved (store/ledger.go), so a caller granting "another $2" passes already-spent + $2,
// and a value below what the book already spent denies the first reservation of the run. That is the
// same quantity book.yaml's ceilings.book_usd is, which is what makes overriding it coherent — the
// alternative (a second, per-process budget axis) would be a new money mechanism, not this row. Like VerifyBank it is an
// operator axis, not a wire one — the ceilings are in NEITHER BriefHash (config/book.go: "Wiring
// fields (paths, ceilings, db) deliberately excluded") NOR the snapshot payload (snapshot.go), so
// setting it moves no request_hash and re-bills nothing. The DAY ceiling is deliberately not
// overridable: it is an account-wide guard, not a property of one run.
CeilingUSD float64
clients map[string]llm.LLMClient
templates map[string]*PromptTemplate
@ -330,6 +345,21 @@ func (r *Runner) packVersion() string {
func (r *Runner) Close() error { return r.Store.Close() }
// bookCeilingUSD is the ONE definition of "the book ceiling in force for this process" (row 145): the
// run-scoped override when the invocation named one, the book's own `ceilings.book_usd` otherwise. Every
// surface that admits a spend or reports the cap reads it here, so the ledger can never admit against one
// number while the operator is shown another. A run-scoped override is only ever an amount > 0 (the CLI
// refuses zero/negative/non-finite), so the flag can only ever ADD a bound, never remove one — which is
// what leaves Р7 intact without amending config.LoadBook's validator. Р7 is a DIFFERENT rule: it demands
// at least one of book_usd/day_usd, so a book may legitimately declare a DAILY ceiling only, this returns
// 0 (no book ceiling, exactly as before the flag existed), and passing the flag introduces one for the run.
func (r *Runner) bookCeilingUSD() float64 {
if r.CeilingUSD > 0 {
return r.CeilingUSD
}
return r.Book.Ceilings.BookUSD
}
func (r *Runner) loadTemplates() error {
for _, st := range r.Pipeline.Stages {
// The path was resolved at load — by the pair/role convention, or by the stage's deliberate

10
backend/internal/pipeline/runner_test.go
View file

@ -120,6 +120,9 @@ type projectOpts struct {
// glossaryTokenBudget overrides context.glossary_token_budget. 0 keeps the fixture's historical 800, so
// every pre-existing project is byte-identical; a tiny value is how the EVICTION path is reachable at all.
glossaryTokenBudget int
// draftOnly drops the editor stage, so the DRAFT is the shipping output (the pipeline shape the wave
// executor calls draft-only). false keeps the historical two-stage fixture byte-identical.
draftOnly bool
}
func setupProjectOpts(t *testing.T, providerURL string, o projectOpts) string {
@ -164,6 +167,10 @@ models:
if o.banknote {
gatesBlock += "\ngates:\n banknote:\n enabled: true\n"
}
editStage := " - { name: edit, role: editor, model: fake-model, prompt_override: prompts/editor.md, prompt_version: v-test, temperature: 0.4, reasoning: \"off\" }\n"
if o.draftOnly {
editStage = ""
}
writeFile(t, filepath.Join(dir, "pipeline.yaml"), fmt.Sprintf(`
core: C1
version: 1
@ -173,8 +180,7 @@ context: { glossary_injection: selective, glossary_token_budget: %d }
waves: { workers: %d }
stages:
- { name: draft, role: translator, model: fake-model, prompt_override: prompts/translator.md, prompt_version: v-test, temperature: 0.3, reasoning: "off" }
- { name: edit, role: editor, model: fake-model, prompt_override: prompts/editor.md, prompt_version: v-test, temperature: 0.4, reasoning: "off" }
%s`, o.minMaxTokens, o.regenerate, o.glossaryTokenBudget, o.waveWorkers, gatesBlock))
%s%s`, o.minMaxTokens, o.regenerate, o.glossaryTokenBudget, o.waveWorkers, editStage, gatesBlock))
sourceName := "source.txt"
if len(o.epub) > 0 {

10
backend/internal/pipeline/stagerun.go
View file

@ -477,7 +477,7 @@ func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID
estimate := ledger.EstimateUSD(price, promptEst, maxTokens, reasoningBudget)
resv, verdict, err := r.Store.Reserve(r.Book.BookID, estimate, store.Ceilings{
BookUSD: r.Book.Ceilings.BookUSD, DayUSD: r.Book.Ceilings.DayUSD,
BookUSD: r.bookCeilingUSD(), DayUSD: r.Book.Ceilings.DayUSD,
})
if err != nil {
r.setJobStatus(ctx, job.ID, "failed")
@ -502,7 +502,13 @@ func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID
committed, reserved, estimate, ch.Chapter, ch.ChunkIdx, st.Name)
}
if verdict == store.ReserveDeniedBook {
return att, fmt.Errorf("pipeline: book USD ceiling reached ($%g)%s — raise ceilings.book_usd or stop: %w", r.Book.Ceilings.BookUSD, money, errReserveCeiling)
// Name WHICH ceiling stopped the run: a caller who passed --ceiling-usd and is told to "raise
// ceilings.book_usd" would edit a file that is not in force (row 145).
raise := "raise ceilings.book_usd"
if r.CeilingUSD > 0 {
raise = fmt.Sprintf("this run's --ceiling-usd=%g overrides the book's ceilings.book_usd=%g — re-run with a higher --ceiling-usd", r.CeilingUSD, r.Book.Ceilings.BookUSD)
}
return att, fmt.Errorf("pipeline: book USD ceiling reached ($%g)%s — %s or stop: %w", r.bookCeilingUSD(), money, raise, errReserveCeiling)
}
return att, fmt.Errorf("pipeline: daily USD ceiling reached ($%g)%s: %w", r.Book.Ceilings.DayUSD, money, errReserveCeiling)
}

128
backend/internal/pipeline/status.go
View file

@ -32,19 +32,62 @@ const (
ChunkPending ChunkState = "pending" // no stage attempted yet
)
// WaveCounter is one wave's done/total pair, in OUTPUT UNITS — the same denominator the whole
// projection counts in (TotalUnits), so the two phases and the unit totals can never be read against
// different scales. Total is 0 for a wave the pipeline does not have (a draft-only pipeline has no edit
// wave), which is how a consumer tells "no such phase" from "none of it is done yet".
type WaveCounter struct {
Done int `json:"done"`
Total int `json:"total"`
}
// PhaseProgress is progress PER WAVE (backlog row 99). The end-to-end unit counter (`Done`) requires
// BOTH every member draft AND the unit's edit to have resolved ok, and the edit wave does not start
// before the bank stop — so it reads 0 for the whole draft wave, which is exactly the indicator the row
// was raised about. These two counters split that single number by wave.
//
// DONE HERE MEANS RESOLVED, NOT OK. A unit counts for a wave once every chunk_status row that wave owes
// it exists — ok, flagged or skipped alike. Two reasons, both load-bearing:
//
// - a flagged unit is FINISHED as far as work goes (nothing re-attempts it without an explicit
// `tmctl redrive`), so excluding it would leave a progress bar permanently short of its own
// denominator on any book with a single flagged chunk;
// - the ok/flagged split is already carried, unconflated, by Done/Flagged/GlossaryMissFlagged below.
//
// Consequence to read deliberately, ON AN EDIT PIPELINE: Edit.Done ≥ Done, and the gap is every unit the
// edit wave RESOLVED but the unit-level verdict did not call done — a unit the post-check GATE flagged
// (its edit row is ok; the gate flips the UNIT after the stage loop), a c-lite unit whose edit shipped
// while a member draft flagged, and a unit whose edit was recorded skipped because every member flagged.
// All three are finished work, which is what these counters count; "attention needed" is the other
// counters' job. On a DRAFT-ONLY pipeline the relation does not hold in that direction at all — Edit is
// the zero counter and Draft is the one to compare against Done.
//
// A stage RENAME is the one state where the two readings disagree in the other direction: stored rows
// carry the old stage name, which is in neither wave's name set, so a wave counter stays short of its
// denominator while `Done` (which counts ok rows by disposition, not by name) can still read full. The
// same rename already makes `export` report every unit pending; `ConfigDrift` is what flags the condition.
type PhaseProgress struct {
Draft WaveCounter `json:"draft"`
Edit WaveCounter `json:"edit"`
}
// ChapterPassport is the per-chapter quality passport (D12): unit counts, the worst flag
// reason, a pass|attention|fail verdict (exp07 chapter rule) and the chapter's spend.
type ChapterPassport struct {
Chapter int `json:"chapter"`
UnitsTotal int `json:"units_total"`
UnitsDone int `json:"units_done"`
UnitsFlagged int `json:"units_flagged"`
UnitsInProgress int `json:"units_in_progress"`
UnitsPending int `json:"units_pending"`
StagesSkipped int `json:"stages_skipped"` // per-stage skip count (downstream of a flag)
Escalations int `json:"escalations"`
PostcheckMisses int `json:"postcheck_misses"`
StyleFlags int `json:"style_flags"` // cheap style/number gate hits (observability, not a disposition)
Chapter int `json:"chapter"`
UnitsTotal int `json:"units_total"`
// Progress is this chapter's per-wave split of UnitsTotal (row 99). It exists for the same reason the
// book-level one does, one level down: a chapter tree whose per-chapter counter is the end-to-end one
// shows every chapter at zero for the whole draft wave (contract companion §4, K-10).
Progress PhaseProgress `json:"progress"`
UnitsDone int `json:"units_done"`
UnitsFlagged int `json:"units_flagged"`
UnitsInProgress int `json:"units_in_progress"`
UnitsPending int `json:"units_pending"`
StagesSkipped int `json:"stages_skipped"` // per-stage skip count (downstream of a flag)
Escalations int `json:"escalations"`
PostcheckMisses int `json:"postcheck_misses"`
StyleFlags int `json:"style_flags"` // cheap style/number gate hits (observability, not a disposition)
// RepairApplied counts this chapter's units whose SHIPPED text carries an applied repair (pack-16). It is
// DERIVED from the durable final_hash namespace, never stored, so it survives a resume like the rest of
// the projection. omitempty: a chapter with no repairs renders exactly as before the field existed.
@ -83,6 +126,11 @@ type StatusReport struct {
Flagged int `json:"flagged"`
Pending int `json:"pending"`
PercentDone float64 `json:"percent_done"` // 100·done/total, unit-count only (no synthetic time-bar)
// Progress is the PER-WAVE split of TotalUnits (backlog row 99) — see PhaseProgress for what "done"
// means there and why it is not the same predicate as Done above. No percentage is derived from it
// here on purpose: how two phases combine into one number a reader sees is a product decision, not a
// projection fact.
Progress PhaseProgress `json:"progress"`
// GlossaryMissFlagged counts chunks the post-check GATE promoted to flagged/glossary_miss (a
// CONFIRMED miss on an otherwise-ok chunk). A SUBSET of Flagged and NOT re-drivable: the miss
@ -299,7 +347,11 @@ func resolveChunkState(rows []store.ChunkStatus, stagesTotal int) chunkStateReso
// no LLM call and replays no checkpoint — the D12 query-handler analogue. Safe to run whenever
// the project file is not held by a running tmctl (the same exclusive-lock rule as `report`).
func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
chunks, err := r.bookChunks()
// The manifest fast path (row 100): status joins stored rows against POSITIONS, never against source
// text, so the persisted structure is a complete substitute for the re-chunk here. The one branch that
// does need text — the re-bill content check — takes the full split through the provider below, and
// only when a bank-only snapshot move actually put it on the path.
chunks, withText, err := r.readModelChunks()
if err != nil {
return nil, err
}
@ -339,6 +391,14 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
// would report a gate-flagged unit as done/pass while `tmctl translate` exits 2 (finding #2).
gateOn := r.Pipeline.Gates.Glossary.PostcheckGate
rep := &StatusReport{BookID: r.Book.BookID, TotalUnits: len(units)}
// Per-wave denominators (row 99): a wave the pipeline does not run has a total of 0 rather than a
// total it can never reach — a draft-only pipeline must not show "edit 0/4276" forever.
if len(draftStages) > 0 {
rep.Progress.Draft.Total = len(units)
}
if len(editStages) > 0 {
rep.Progress.Edit.Total = len(units)
}
// The unsigned-bank exposure, same definition as the quality report's (a row with a rendering that is
// not approved). Read-only, like everything else here; a read failure says so rather than reporting 0.
if rows, gerr := r.Store.GlossaryForBook(r.Book.BookID); gerr != nil {
@ -361,10 +421,40 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
chapterOrder = append(chapterOrder, u.Chapter)
}
p.UnitsTotal++
if len(draftStages) > 0 {
p.Progress.Draft.Total++
}
if len(editStages) > 0 {
p.Progress.Edit.Total++
}
leader := chunkKey{u.Chapter, u.FirstChunkIdx}
expected := len(u.Members)*len(draftStages) + len(editStages)
res := resolveChunkState(unitRows(u, byChunk), expected)
// Collected once: three passes over the unit's rows follow, and on a 5000-unit book re-walking the
// member map for each of them is pure churn.
rows := unitRows(u, byChunk)
res := resolveChunkState(rows, expected)
state, reason := res.State, res.Reason
// Per-wave resolution (row 99), off the SAME rows the unit state is folded from. A wave owes a unit
// a fixed number of rows — |members|·|draft stages| in the draft wave, |edit stages| at the leader in
// the edit one — and chunk_status is keyed (book, chapter, chunk, stage), so counting the rows that
// wave wrote IS the resolution test: a row exists only once the wave decided (ok/flagged/skipped).
draftRows, editRows := 0, 0
for _, cs := range rows {
switch {
case draftStageNames[cs.Stage]:
draftRows++
case editStageNames[cs.Stage]:
editRows++
}
}
if len(draftStages) > 0 && draftRows >= len(u.Members)*len(draftStages) {
rep.Progress.Draft.Done++
p.Progress.Draft.Done++
}
if len(editStages) > 0 && editRows >= len(editStages) {
rep.Progress.Edit.Done++
p.Progress.Edit.Done++
}
p.CostUSD += res.CostUSD
p.StagesSkipped += res.Skipped
miss := missByChunk[leader] // the unit's post-check ran once, at the leader row
@ -375,7 +465,7 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
rep.StyleFlags += style
// Applied repairs (pack-16): the unit's shipping row points at a repair export. Read off the durable
// final_hash namespace — no counter column, so a resumed run reports the same number.
for _, row := range unitRows(u, byChunk) {
for _, row := range rows {
if strings.HasPrefix(row.FinalHash, repairDerivedNS+":") {
p.RepairApplied++
break
@ -491,7 +581,7 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
// re-bills too, and that is precisely the case the boolean pair leaves unpriced. $0 and read-only:
// projectRebill only re-renders the wave snapshots and sums stored costs.
if len(statuses) > 0 {
if proj, perr := r.projectRebill(statuses, chunks); perr != nil {
if proj, perr := r.projectRebill(statuses, chunks, withText); perr != nil {
// Same discipline as the drift check above: a failed projection is reported, never
// silently rendered as "nothing to re-pay".
r.Log.WarnContext(ctx, "re-bill projection failed; the re-payment cost of the drift is unknown (reported as none)", "err", perr)
@ -506,7 +596,10 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
return nil, err
}
rep.CommittedUSD, rep.ReservedUSD = committed, reserved
rep.BookCeilingUSD = r.Book.Ceilings.BookUSD
// The ceiling IN FORCE (row 145): a read path never carries a run-scoped override, so this is the
// book's own number there — but reading it through the single definition means status can never quote
// a ceiling the ledger is not admitting against.
rep.BookCeilingUSD = r.bookCeilingUSD()
if rep.BookCeilingUSD > 0 {
rep.CeilingPct = 100 * (committed + reserved) / rep.BookCeilingUSD
}
@ -681,6 +774,11 @@ func (r *Runner) Redrive(ctx context.Context, sel RedriveSelector) (*RedriveSumm
if err := r.seedGlossary(ctx); err != nil {
return summary, nil, fmt.Errorf("pipeline: redrive seed glossary: %w", err)
}
// The bank read-out (row 125) follows the re-seed HERE, not only inside TranslateBook. Every guard
// below can abort the redrive after this point — the drift guard, the re-payment consent, the reset
// itself — and each of those exits leaves the bank re-seeded in the store while the exported file
// still described the previous one. A reader would then hold a bank the engine no longer has.
r.exportBank(ctx, "redrive/re-seeded")
// Config/seed-drift guard BEFORE the destructive reset (external-review 1c): if the current config
// drifted from the snapshot the stored rows carry, TranslateBook would fail loud in runStage — but
// the reset (ResetChunkStages) would already have deleted the flagged rows/checkpoints, so a later

10
backend/internal/pipeline/waverun.go
View file

@ -89,7 +89,10 @@ type WaveSignatureStop struct {
// the CLI can print a capped view without re-reading the file (the cap exists because emitRankCap is
// 200 and a 200-term dump is not a review surface).
TablePath string
Rows []BankStopRow
// TableJSONPath is the MACHINE form of that same table (row 101) — what a signing screen is built
// from. Named beside TablePath rather than instead of it: the two are one table in two renderings.
TableJSONPath string
Rows []BankStopRow
}
func (e *WaveSignatureStop) Error() string {
@ -133,7 +136,10 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s
if stopped, err := r.runBankMiningStop(ctx, chunks, draftSnapshot, editWave); err != nil {
return nil, err
} else if stopped {
return nil, &WaveSignatureStop{Terms: r.lastMinedCount, SignaturePath: r.signatureMapPath(), TablePath: r.bankStopTablePath(), Rows: r.lastBankStopRows}
return nil, &WaveSignatureStop{
Terms: r.lastMinedCount, SignaturePath: r.signatureMapPath(),
TablePath: r.bankStopTablePath(), TableJSONPath: r.bankStopTableJSONPath(), Rows: r.lastBankStopRows,
}
}
res := &BookResult{BookID: r.Book.BookID}