diff --git a/backend/README.md b/backend/README.md index 67481cff..23a5227b 100644 --- a/backend/README.md +++ b/backend/README.md @@ -8,7 +8,7 @@ | Пакет | Что делает | Ключевые файлы | |---|---|---| -| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3). main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` | +| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) / `manifest` (**$0-производитель персиста манифеста глав/чанков**, строка 100: нужен ДО первого прогона — дерево глав разобранной, но не запущенной книги; `--json` печатает сам документ). Денежный аргумент прогона: `--ceiling-usd` (строка 145) перекрывает книжный `ceilings.book_usd` ТОЛЬКО на этот прогон, в book.yaml не пишется, ноль/отрицательное = отказ запуска. main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` | | `internal/llm` | OpenAI-совместимый транспорт + retry/backoff, **capability-слой** (budget_field/temperature/reasoning per-модель), провайдеры openai/local (no-proxy)/anthropic (DEPRECATED-референс); `failover.go` удалён паком-17 — маршрутизация лейблами живёт в config/pipeline (канал B, один хоп `chain[0]`, fail-closed) | `httpllm.go`, `capability.go`, `provider_*.go` | | `internal/ledger` | Цены по usage (вкл. reasoning/cache-поля), `PriceForResponse` по фактической модели | `pricing.go` | | `internal/store` | SQLite (modernc, CGO-free), цепочка миграций `schema_version` (⚠ ALTER-шаги v8+ не идемпотентны вопреки шапке — бэклог-строка 49а), reserve/settle+checkpoint, chunk_status, глоссарий (+подписной цикл терминолога), ruby, retrieval_state, request_log; `OpenReadOnly` — без flock/миграций/recovery для status/report/export | `ledger.go`, `migrate.go`, `glossary.go`, `store.go` | @@ -21,7 +21,7 @@ | `internal/membank` | Банк памяти v2, memmatch-v4: Aho-Corasick, спойлер-окна, disposition, **trust-gated longest-match** (draft-длиннее НЕ съедает вложенный approved; отказ = громкая телеметрия — терм-дрейф закрыт в КОДЕ, D39.2-T1), post-check, рендереры инъекции, загрузка/валидация сида (`SeedLint`) | `memory.go`, `memseed.go`, `mempostcheck.go` | | `internal/miner` | Детерминированный оффлайн-майнер банка (WS3, $0): `MineBank` над нормализованными чанками + контраст-корпус → `[]Term`, `DeltaYAML` — сид-дельта на подпись владельца; все пар-данные приходят `*lang.Pack` | `miner.go`, `miner_*.go` | | `internal/checks` | Детерминированные $0-вердикты над текстом чанка, **target-aware ПО ДАННЫМ (D39.64): `Checkers` компилируется из цель-данных, `TargetActive()`/`TargetScriptNonLatin()` гейтят всё — цель без данных инертна, Go-веток по паре нет**: санитайзер v7 (fold-first, CJK/Hangul-leak, CJK-глосс-whitelist) + `ExportNormalize` (**export-contract слой 6**, no-op без цель-данных), cheap-гейты и DC-чекеры (пар-данные из пака), coverage-гейт эксцизии, regression-guard. Флаг-вокабуляр НЕ импортирует: возвращает факт, диспозицию назначает драйвер | `sanitizer.go`, `cheapgates.go`, `checkers.go`, `coverage.go` | -| `internal/pipeline` | Композит-корень (драйвер): сетап `runner.go` → снапшот `snapshot.go` → сид `seeding.go` → цикл книги `bookrun.go` → волны `waverun.go` → петля чанка `chunkrun.go` → стадия `stagerun.go` → эскалация с ре-гейтом `escalation.go` → resume `resume.go`; classify/disposition; **реестр `roleInjectionRenderers`** (новая роль = данные+рендерер, не правка switch); `export.go`/`quality.go`/`status.go` — read-only проекции (общая derivation member-drop = `memberDrops`) | `bookrun.go`, `waverun.go`, `stagerun.go`, `export.go`, `disposition.go` | +| `internal/pipeline` | Композит-корень (драйвер): сетап `runner.go` → снапшот `snapshot.go` → сид `seeding.go` → цикл книги `bookrun.go` → волны `waverun.go` → петля чанка `chunkrun.go` → стадия `stagerun.go` → эскалация с ре-гейтом `escalation.go` → resume `resume.go`; classify/disposition; **реестр `roleInjectionRenderers`** (новая роль = данные+рендерер, не правка switch); `export.go`/`quality.go`/`status.go` — read-only проекции (общая derivation member-drop = `memberDrops`); **read-out файлы для платформы** (D39.85 — приватный SQLite движка ей закрыт): `manifest.go` (структура глав/чанков + стабильный id главы, строка 100), `bankexport.go` (весь банк тремя статусами, 125), машинная стоп-таблица в `mining.go` (101); все три пишутся через `artifact.go` (write-then-rename — их читают ПОКА идёт прогон) | `bookrun.go`, `waverun.go`, `stagerun.go`, `export.go`, `manifest.go`, `disposition.go` | | `internal/obs` | trace_id, структурные логи (contextHandler несёт book/chapter/chunk/stage/role из ctx), safego | | ## Инварианты — ЛОМАТЬ НЕЛЬЗЯ (каждый закреплён тестами) @@ -64,6 +64,8 @@ go run ./cmd/tmctl translate --config example/book.yaml # реальные go run ./cmd/tmctl status --config example/book.yaml --json # $0, N/M+паспорта+деньги, живой прогон ок go run ./cmd/tmctl report --config example/book.yaml # $0, quality-report (KPI/rates) go run ./cmd/tmctl export --config example/book.yaml # $0, экспорт-JSON для полигона (--plaintext для человека) +go run ./cmd/tmctl manifest --config example/book.yaml # $0, пере-строить персист манифеста глав/чанков (--json — сам документ) +go run ./cmd/tmctl translate --config example/book.yaml --ceiling-usd 0.5 # потолок ТОЛЬКО на этот прогон, book.yaml не пишется # live-conformance (реальные провайдеры, платно, вне CI): set -a; . ./.env; set +a; TM_LIVE=1 go test -tags live -run TestLive -v ./internal/pipeline/ ``` diff --git a/backend/cmd/tmctl/backup_test.go b/backend/cmd/tmctl/backup_test.go index 686e2514..c9a869bc 100644 --- a/backend/cmd/tmctl/backup_test.go +++ b/backend/cmd/tmctl/backup_test.go @@ -65,7 +65,7 @@ func TestFakeTranslatePathCreatesNoBackup(t *testing.T) { defer srv.Close() bookPath := setupCLIProject(t, srv.URL) - if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false); err != nil { + if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false, 0); err != nil { t.Fatalf("fake translate: %v", err) } if _, err := os.Stat(filepath.Join(filepath.Dir(bookPath), "backups")); err == nil { diff --git a/backend/cmd/tmctl/invocation.go b/backend/cmd/tmctl/invocation.go index 7b3a310b..646d261a 100644 --- a/backend/cmd/tmctl/invocation.go +++ b/backend/cmd/tmctl/invocation.go @@ -30,6 +30,11 @@ type invocation struct { // «флаг бинарный» (owner) — and deliberately not a book.yaml key: whether a human is available to // sign right now is a property of the invocation, not of the book. verifyBank bool + // ceilingUSD is the BOOK ceiling for THIS RUN ONLY (row 145 / D39.110), 0 when the flag was absent. + // It is an invocation property for the same reason verifyBank is: the amount a caller is willing to + // spend on one run belongs to that run, not to the book's data — writing it into book.yaml would make + // a caller's number a permanent record in the engine's config and mix the zones (D39.81/D39.85). + ceilingUSD float64 sel pipeline.RedriveSelector } @@ -88,7 +93,7 @@ func (v *rebillConsentValue) IsBoolFlag() bool { return true } // main()'s default branch → exit 1, leaving 2 exclusive to flagged chunks. func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { if len(args) < 1 { - return invocation{}, fmt.Errorf("usage: tmctl --config book.yaml") + return invocation{}, fmt.Errorf("usage: tmctl --config book.yaml") } cmd, rest := args[0], args[1:] @@ -98,7 +103,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { resnapshot := fs.Bool("resnapshot", false, "re-pin existing jobs to the current config snapshot (re-translates already-paid chunks — explicit consent)") acceptRebill := &rebillConsentValue{} fs.Var(acceptRebill, "accept-rebill", "translate/redrive: consent to the projected RE-PAYMENT of already-billed work (D20.2-Q2). Bare accepts the whole projected amount; --accept-rebill=1.50 accepts it only up to $1.50") - asJSON := fs.Bool("json", false, "status: emit the projection as JSON (stable disposition/flag_reason enums) for CI/IDE") + asJSON := fs.Bool("json", false, "status: emit the projection as JSON (stable disposition/flag_reason enums) for CI/IDE; manifest: emit the manifest document instead of a summary") asPlaintext := fs.Bool("plaintext", false, "export: emit the concatenated human text instead of the default stable JSON") asPairs := fs.Bool("pairs", false, "export: include the source text per chunk (src↔target column for the DC1/DC2 FP-measure, WS5)") chapter := fs.Int("chapter", -1, "redrive: restrict to this chapter (default: any)") @@ -106,6 +111,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { reason := fs.String("reason", "", "redrive: restrict to this flag_reason (default: any)") dryRun := fs.Bool("dry-run", false, "redrive: report what would be re-attacked without touching anything") verifyBank := fs.Bool("verify-bank", false, "translate/redrive: STOP at the bank-mining boundary and print the bank table for review instead of continuing into the edit wave with an unsigned bank (default: continue)") + ceilingUSD := fs.Float64("ceiling-usd", 0, "translate/redrive: the book USD ceiling in force for THIS RUN ONLY — it OVERRIDES book.yaml `ceilings.book_usd` and is never written back. It caps the book's CUMULATIVE committed+reserved spend, not this run's increment, and must be > 0") seed := fs.String("seed", "", "seed-lint: path to the glossary seed YAML to validate ($0, no --config)") if err := fs.Parse(rest); err != nil { return invocation{}, err @@ -120,6 +126,25 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { if *cfgPath == "" { return invocation{}, fmt.Errorf("--config book.yaml is required") } + // The run ceiling is refused rather than defaulted when it is present but not a spendable amount, so + // the flag can only ever ADD a bound, never remove one (D39.110: «выключено» is not a state). + // + // Р7's own validator (config.LoadBook) is untouched and is NOT the same rule: it demands at least ONE + // of book_usd/day_usd, so a book may legitimately declare a daily ceiling only — and then the book + // ceiling this flag overrides is zero, i.e. absent, and passing the flag introduces one for this run. + // fs.Visit reports only flags actually PRESENT on the command line, which is what separates + // "--ceiling-usd 0" (a refusal) from an absent flag (the book's own ceiling stands). Placed AFTER the + // --config check so the frozen validation order is untouched: a missing --config is still the first + // thing a caller is told about. + ceilingGiven := false + fs.Visit(func(f *flag.Flag) { + if f.Name == "ceiling-usd" { + ceilingGiven = true + } + }) + if ceilingGiven && (math.IsNaN(*ceilingUSD) || math.IsInf(*ceilingUSD, 0) || *ceilingUSD <= 0) { + return invocation{}, fmt.Errorf("--ceiling-usd must be a finite amount greater than zero (it is the ceiling for THIS run and overrides book.yaml ceilings.book_usd); got %v — omit the flag to run under the book's own ceiling", *ceilingUSD) + } // The optional-value trap (see IsBoolFlag): `--accept-rebill 1.50` leaves 1.50 as a stray argument // and the consent unlimited. Refuse loud rather than bill the difference. Scoped to exactly that // spelling, so the historically-tolerated stray argument stays tolerated everywhere else. @@ -129,6 +154,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { return invocation{ cmd: cmd, cfgPath: *cfgPath, resnapshot: *resnapshot, acceptRebill: acceptRebill.c, asJSON: *asJSON, asPlaintext: *asPlaintext, asPairs: *asPairs, verifyBank: *verifyBank, + ceilingUSD: *ceilingUSD, sel: pipeline.RedriveSelector{ Chapter: *chapter, ChunkIdx: *chunk, Reason: *reason, DryRun: *dryRun, }, diff --git a/backend/cmd/tmctl/invocation_test.go b/backend/cmd/tmctl/invocation_test.go index 738b4cd4..d25bae9a 100644 --- a/backend/cmd/tmctl/invocation_test.go +++ b/backend/cmd/tmctl/invocation_test.go @@ -17,9 +17,10 @@ import ( func TestParseNoArgsUsage(t *testing.T) { _, err := parseInvocation(nil, &bytes.Buffer{}) - // The command list gained `export` (D39 слой 6 read-only surface, a deliberate contract extension); - // the rest of the usage text stays frozen. - if err == nil || err.Error() != "usage: tmctl --config book.yaml" { + // The command list gained `export` (D39 слой 6 read-only surface) and then `manifest` (backlog row 100 + // — the $0 producer of the chapter/chunk manifest); both are deliberate contract extensions and the + // rest of the usage text stays frozen. + if err == nil || err.Error() != "usage: tmctl --config book.yaml" { t.Fatalf("usage error text is frozen, got: %v", err) } } @@ -206,3 +207,33 @@ func TestParseVerifyBank(t *testing.T) { t.Fatal("--verify-bank must not accept a non-boolean value") } } + +// --- backlog row 145: the per-run ceiling flag ---------------------------------------------------- + +// TestParseCeilingUSD pins the flag's whole contract: absent means "the book's own ceiling stands", a +// positive amount is carried through, and every value that is not a spendable amount is REFUSED rather +// than defaulted. The refusal is the load-bearing half — Р7 forbids a ledger with no ceiling, and a +// silently-ignored `--ceiling-usd 0` would be exactly that with a flag in front of it. +func TestParseCeilingUSD(t *testing.T) { + inv, err := parseInvocation([]string{"translate", "--config", "b.yaml"}, &bytes.Buffer{}) + if err != nil || inv.ceilingUSD != 0 { + t.Fatalf("an absent flag must leave the ceiling unset, got %v / %v", inv.ceilingUSD, err) + } + inv, err = parseInvocation([]string{"translate", "--config", "b.yaml", "--ceiling-usd=1.25"}, &bytes.Buffer{}) + if err != nil || inv.ceilingUSD != 1.25 { + t.Fatalf("--ceiling-usd=1.25 → %v / %v", inv.ceilingUSD, err) + } + // The space spelling binds too (unlike --accept-rebill, which is a bool-flag with an optional value). + inv, err = parseInvocation([]string{"redrive", "--config", "b.yaml", "--ceiling-usd", "0.5"}, &bytes.Buffer{}) + if err != nil || inv.ceilingUSD != 0.5 { + t.Fatalf("--ceiling-usd 0.5 → %v / %v", inv.ceilingUSD, err) + } + for _, bad := range []string{"0", "-1", "NaN", "Inf"} { + var diag bytes.Buffer + if _, err := parseInvocation([]string{"translate", "--config", "b.yaml", "--ceiling-usd=" + bad}, &diag); err == nil { + t.Fatalf("--ceiling-usd=%s must be refused: a run ceiling that is not a spendable amount is not a ceiling", bad) + } else if code := exitCode(err); code != 1 { + t.Fatalf("--ceiling-usd=%s must exit 1, got %d", bad, code) + } + } +} diff --git a/backend/cmd/tmctl/main.go b/backend/cmd/tmctl/main.go index 8e9a39d0..437bc1a3 100644 --- a/backend/cmd/tmctl/main.go +++ b/backend/cmd/tmctl/main.go @@ -1,4 +1,4 @@ -// tmctl is the TextMachine CLI: translate / report / status / redrive / backup. +// tmctl is the TextMachine CLI: translate / report / status / export / redrive / manifest / backup. // main.go — thin wiring (package №4): argument parsing — invocation.go, // output renderers — render.go, .env — dotenv.go; here just the // «parse → env → ctx → fetch → render» wiring and exit-code mapping. @@ -77,7 +77,7 @@ func run() error { if err := preflightBackup(inv.cfgPath, os.Stdout); err != nil { return err } - return translate(ctx, inv.cfgPath, inv.resnapshot, inv.acceptRebill, inv.verifyBank) + return translate(ctx, inv.cfgPath, inv.resnapshot, inv.acceptRebill, inv.verifyBank, inv.ceilingUSD) case "report": return report(inv.cfgPath) case "status": @@ -91,21 +91,31 @@ func run() error { return err } } - return redrive(ctx, inv.cfgPath, inv.resnapshot, inv.acceptRebill, inv.sel, inv.verifyBank) + return redrive(ctx, inv.cfgPath, inv.resnapshot, inv.acceptRebill, inv.sel, inv.verifyBank, inv.ceilingUSD) + case "manifest": + return manifestCmd(inv.cfgPath, inv.asJSON) case "backup": return backupCmd(inv.cfgPath, os.Stdout) case "seed-lint": return seedLint(inv.seedPath) default: - return fmt.Errorf("unknown command %q (want translate|report|status|export|redrive|backup|seed-lint)", inv.cmd) + return fmt.Errorf("unknown command %q (want translate|report|status|export|redrive|manifest|backup|seed-lint)", inv.cmd) } } -// translate runs the book. The two money flags are ORTHOGONAL (D20.2-Q2): --resnapshot grants -// permission to re-pin jobs onto the current config snapshot, while --accept-rebill[=usd] consents to -// the AMOUNT that re-pin would re-pay. Over the book's consent threshold the run stops with the sum -// before reserving anything — a flag that names no money cannot carry a Р6 consent to a spend. -func translate(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill pipeline.RebillConsent, verifyBank bool) error { +// translate runs the book. The three money flags are ORTHOGONAL (D20.2-Q2 / row 145): --resnapshot +// grants permission to re-pin jobs onto the current config snapshot, --accept-rebill[=usd] consents to +// the AMOUNT that re-pin would re-pay, and --ceiling-usd replaces the BOOK ceiling for this process. +// +// ⚠ --ceiling-usd is NOT a per-run budget. The ledger admits against the book's CUMULATIVE spend +// (store/ledger.go: `bookTotal + estimate > BookUSD`, where bookTotal is committed+reserved across every +// run of the book), so the flag says "this book may reach $X while I am running", not "this run may +// spend $X". A caller that wants to grant an increment has to pass already-committed + increment, and a +// value below what the book has already spent denies the very first reservation. +// +// Over the book's consent threshold the run stops with the sum before reserving anything — a flag that +// names no money cannot carry a Р6 consent to a spend. +func translate(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill pipeline.RebillConsent, verifyBank bool, ceilingUSD float64) error { r, err := pipeline.NewRunner(cfgPath, obs.NewLogger()) if err != nil { return err @@ -114,6 +124,7 @@ func translate(ctx context.Context, cfgPath string, resnapshot bool, acceptRebil r.Resnapshot = resnapshot r.AcceptRebill = acceptRebill r.VerifyBank = verifyBank + r.CeilingUSD = ceilingUSD res, err := r.TranslateBook(ctx) if err != nil { @@ -215,6 +226,34 @@ func status(ctx context.Context, cfgPath string, asJSON bool) error { return renderStatusHuman(os.Stdout, rep, cfgPath) } +// manifestCmd rebuilds and persists the book's chapter/chunk manifest (backlog row 100). $0 and key-less +// like the other read commands: it ingests + cuts the source and writes a sidecar beside the project DB — +// no LLM, no provider key, no wave. +// +// It opens the store exactly as `status` does (NewReadOnlyRunner): read-only, no flock — EXCEPT on the +// first touch of a project, where that path falls back to a full Open and therefore CREATES and migrates +// the database. That is the state this command is most often run in ("parsed, never run"), so the +// side effect is the common case rather than the corner one, and it is the same one `status` has always +// had. It also needs the project directory to be writable, unlike status/report/export. +// +// It exists because the manifest is needed BEFORE any run: a book that has been accepted and cut but +// never translated still has a chapter tree, and every other producer of that structure is a paid path. +// A translate rewrites the manifest itself (bookrun.go), so this command is for the not-yet-run case and +// for refreshing the artifact by hand after a source edit. +func manifestCmd(cfgPath string, asJSON bool) error { + r, err := pipeline.NewReadOnlyRunner(cfgPath, obs.NewLogger()) + if err != nil { + return err + } + defer r.Close() + + m, err := r.BuildAndPersistManifest() + if err != nil { + return err + } + return renderManifest(os.Stdout, m, r.ManifestPath(), asJSON) +} + // redrive re-attacks the FLAGGED chunks matching the selector (D15.3): it resets their terminal // flag (chunk_status + checkpoints of the flagged/skipped stages) and re-runs the durable loop // with a FRESH retry/escalation budget, never touching DispOK work. On --dry-run it only reports @@ -225,7 +264,10 @@ func status(ctx context.Context, cfgPath string, asJSON bool) error { // silently ignored — Redrive already honoured r.Resnapshot, only the CLI never set it). // --accept-rebill[=usd] is the separate consent to the AMOUNT such a re-pin re-pays (D20.2-Q2): over // the threshold Redrive refuses BEFORE its destructive reset, so the flag telemetry survives a refusal. -func redrive(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill pipeline.RebillConsent, sel pipeline.RedriveSelector, verifyBank bool) error { +// --ceiling-usd applies here for the same reason it applies to translate: a redrive re-attacks flagged +// chunks with real provider calls, so leaving it on the book's ceiling alone would be a hole in exactly +// the surface row 145 exists to close. +func redrive(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill pipeline.RebillConsent, sel pipeline.RedriveSelector, verifyBank bool, ceilingUSD float64) error { r, err := pipeline.NewRunner(cfgPath, obs.NewLogger()) if err != nil { return err @@ -234,6 +276,7 @@ func redrive(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill r.Resnapshot = resnapshot r.AcceptRebill = acceptRebill r.VerifyBank = verifyBank + r.CeilingUSD = ceilingUSD summary, res, err := r.Redrive(ctx, sel) if err != nil { diff --git a/backend/cmd/tmctl/rebill_cli_test.go b/backend/cmd/tmctl/rebill_cli_test.go index d69d9008..8ea70612 100644 --- a/backend/cmd/tmctl/rebill_cli_test.go +++ b/backend/cmd/tmctl/rebill_cli_test.go @@ -98,7 +98,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) { bookPath := setupCLIProject(t, srv.URL) ctx := context.Background() - if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false); err != nil { + if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0); err != nil { t.Fatalf("first run: %v", err) } if calls != 1 { @@ -114,7 +114,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) { writeCLIFile(t, pipePath, strings.Replace(string(raw), "prompt_version: v-cli", "prompt_version: v-cli-drift", 1)) // --resnapshot WITHOUT consent → refused, nothing called. - err = translate(ctx, bookPath, true, pipeline.RebillConsent{}, false) + err = translate(ctx, bookPath, true, pipeline.RebillConsent{}, false, 0) if err == nil { t.Fatal("`translate --resnapshot` without --accept-rebill must be refused over the threshold") } @@ -127,7 +127,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) { // A ceiling BELOW the projection is still a refusal — proving the CAPPED value is wired, not just // the boolean. - if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true, Capped: true, CapUSD: 0.0001}, false); err == nil { + if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true, Capped: true, CapUSD: 0.0001}, false, 0); err == nil { t.Fatal("a ceiling below the projection must be refused through the CLI entry point too") } if calls != 1 { @@ -135,7 +135,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) { } // Consent given → the run proceeds and really re-pays. - if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true}, false); err != nil { + if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true}, false, 0); err != nil { t.Fatalf("`translate --resnapshot --accept-rebill` must proceed: %v", err) } if calls != 2 { @@ -160,7 +160,7 @@ func TestRedriveWiresAcceptRebill(t *testing.T) { ctx := context.Background() // The refusal text flags the only chunk (exit 2 is a sentinel, not an infra failure). - if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false); exitCode(err) != 2 { + if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0); exitCode(err) != 2 { t.Fatalf("setup: the refusal fixture must complete with flags (exit 2), got %v", err) } callsAfterRun1 := calls @@ -173,7 +173,7 @@ func TestRedriveWiresAcceptRebill(t *testing.T) { writeCLIFile(t, pipePath, strings.Replace(string(raw), "prompt_version: v-cli", "prompt_version: v-cli-drift", 1)) sel := pipeline.RedriveSelector{Chapter: -1, ChunkIdx: -1} - if err := redrive(ctx, bookPath, true, pipeline.RebillConsent{}, sel, false); err == nil { + if err := redrive(ctx, bookPath, true, pipeline.RebillConsent{}, sel, false, 0); err == nil { t.Fatal("`redrive --resnapshot` without consent must be refused over the threshold") } else if !strings.Contains(err.Error(), "--accept-rebill") { t.Errorf("the redrive refusal must name the flag; got: %v", err) @@ -184,10 +184,65 @@ func TestRedriveWiresAcceptRebill(t *testing.T) { // Consent given → the redrive resets and re-attacks (the mock still refuses, so it ends flagged again // — what matters is that it RAN and billed). - if err := redrive(ctx, bookPath, true, pipeline.RebillConsent{Given: true}, sel, false); exitCode(err) != 2 { + if err := redrive(ctx, bookPath, true, pipeline.RebillConsent{Given: true}, sel, false, 0); exitCode(err) != 2 { t.Fatalf("`redrive --resnapshot --accept-rebill` must run (and end flagged), got %v", err) } if calls <= callsAfterRun1 { t.Fatalf("the consented redrive must re-call the provider, calls=%d", calls) } } + +// TestTranslateAndRedriveWireTheRunCeiling is the same class as TestTranslateWiresAcceptRebill, for the +// flag row 145 added: it exists to change what the RUN does, not what the parser returns. +// +// It closes an audit finding — deleting `r.CeilingUSD = ceilingUSD` from either command left the whole +// backend suite green, because TestParseCeilingUSD stops at the parsed invocation and the pipeline-level +// ceiling tests set the field on the Runner directly. Neither of them touches the wire between the two. +func TestTranslateAndRedriveWireTheRunCeiling(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = io.ReadAll(r.Body) + tb, _ := json.Marshal("ЧЕРНОВИК ПЕРЕВОДА") + fmt.Fprintf(w, `{"id":"fake","model":"fake-model","choices":[{"message":{"content":%s},"finish_reason":"stop"}], + "usage":{"prompt_tokens":1000,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":200}}}`, tb) + })) + defer srv.Close() + ctx := context.Background() + + // A ceiling far below anything the book can cost. The book's own ceiling is comfortable, so ONLY the + // argument can produce this stop — and only if the command actually hands it to the runner. + const tiny = 0.000001 + err := translate(ctx, setupCLIProject(t, srv.URL), false, pipeline.RebillConsent{}, false, tiny) + if err == nil || !strings.Contains(err.Error(), "ceiling reached") { + t.Fatalf("translate must carry --ceiling-usd into the run; got %v", err) + } + if !strings.Contains(err.Error(), "--ceiling-usd") { + t.Fatalf("the stop must name the ceiling in force: %v", err) + } + + // Same for redrive: it re-attacks with real provider calls, so an unwired flag there is a hole in + // exactly the surface the row closes. A redrive with no targets never reserves and therefore could not + // show a ceiling at all — so the book is first driven to a FLAGGED chunk, which gives the redrive + // something to re-attack and makes the reservation (and the ceiling) reachable. + refuse := true + srv2 := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(r.Body) + text := "ЧЕРНОВИК ПЕРЕВОДА" + if refuse && !strings.Contains(string(body), "Черновик перевода для редактуры") { + text = "Извините, я не могу перевести это." // soft refusal → the chunk is flagged + } + tb, _ := json.Marshal(text) + fmt.Fprintf(w, `{"id":"fake","model":"fake-model","choices":[{"message":{"content":%s},"finish_reason":"stop"}], + "usage":{"prompt_tokens":1000,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":200}}}`, tb) + })) + defer srv2.Close() + bookPath := setupCLIProject(t, srv2.URL) + if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0); exitCode(err) != 2 { + t.Fatalf("setup: the run must finish WITH a flagged chunk (exit 2), got %v", err) + } + refuse = false + sel := pipeline.RedriveSelector{Chapter: -1, ChunkIdx: -1} + err = redrive(ctx, bookPath, false, pipeline.RebillConsent{}, sel, false, tiny) + if err == nil || !strings.Contains(err.Error(), "ceiling reached") { + t.Fatalf("redrive must carry --ceiling-usd into the re-attack; got %v", err) + } +} diff --git a/backend/cmd/tmctl/render.go b/backend/cmd/tmctl/render.go index 894e3f20..d3186595 100644 --- a/backend/cmd/tmctl/render.go +++ b/backend/cmd/tmctl/render.go @@ -84,6 +84,9 @@ func renderSignatureStop(w io.Writer, s *pipeline.WaveSignatureStop) { if s.TablePath != "" { fmt.Fprintf(w, "Full table: %s\n", s.TablePath) } + if s.TableJSONPath != "" { + fmt.Fprintf(w, "Machine table: %s\n", s.TableJSONPath) + } fmt.Fprintln(w, "Next: for EACH term either promote it into the mined-delta file (approved + dst), OR decline it in") fmt.Fprintln(w, "the mined-rejects file (book.yaml: mined_rejects), then re-run `tmctl translate`. The stop clears") fmt.Fprintln(w, "once every proposed term is promoted or rejected (then the delta is empty → auto-continue to the edit wave).") @@ -137,7 +140,7 @@ func renderBankStopRows(w io.Writer, rows []pipeline.BankStopRow) { if len(r.Variants) > 1 { // The disagreement is the reason to sign: one term coming back several ways is exactly what a // canon exists to close, and hiding it behind one winner throws that reason away. - fmt.Fprintf(w, " drafts: %s\n", trunc(strings.Join(r.Variants, " | "), 90)) + fmt.Fprintf(w, " drafts: %s\n", trunc(strings.Join(r.VariantLabels(), " | "), 90)) } if len(r.Contexts) > 0 { fmt.Fprintf(w, " ctx: %s\n", trunc(r.Contexts[0], 90)) @@ -516,6 +519,24 @@ func flagParen(r string) string { return " (" + r + ")" } +// renderManifest prints the persisted chapter/chunk manifest (row 100). The default is a human summary — +// a 2283-chapter tree is not something anyone reads in a terminal — and --json emits the document itself, +// byte-identical to the sidecar, so a consumer can pipe it instead of knowing where the file lives. +func renderManifest(w io.Writer, m *pipeline.BookManifest, path string, asJSON bool) error { + if asJSON { + enc := json.NewEncoder(w) + enc.SetIndent("", " ") + return enc.Encode(m) + } + fmt.Fprintf(w, "=== MANIFEST: %s ===\n", m.BookID) + fmt.Fprintf(w, "chapters=%d units=%d chunks=%d\n", m.ChaptersTotal, m.UnitsTotal, m.ChunksTotal) + fmt.Fprintf(w, "source: %d bytes (sha256 %.12s) · encoding %s · %s→%s\n", + m.SourceBytes, m.SourceSHA256, m.Encoding, m.SourceLang, m.TargetLang) + fmt.Fprintf(w, "cut by: %s · validity key %.12s\n", m.ChunkerVersion, m.Key) + fmt.Fprintf(w, "written: %s\n", path) + return nil +} + // renderStatusJSON emits the StatusReport as indented JSON (stable enums — the // ratified CI/IDE contract, D12/D15.3) and maps flagged>0 to the exit-2 sentinel. func renderStatusJSON(w io.Writer, rep *pipeline.StatusReport) error { @@ -560,6 +581,12 @@ func renderStatusHuman(w io.Writer, rep *pipeline.StatusReport, cfgPath string) fmt.Fprintf(w, "=== STATUS: %s (snapshot %s)%s ===\n", rep.BookID, snap, drift) fmt.Fprintf(w, "Progress: %d/%d units (%.1f%%) — done=%d in_progress=%d flagged=%d pending=%d\n", rep.Done, rep.TotalUnits, rep.PercentDone, rep.Done, rep.InProgress, rep.Flagged, rep.Pending) + // Per-wave progress (row 99). The line above needs BOTH waves to have resolved a unit ok, and the edit + // wave does not start before the bank stop — so it sits at 0/N for the whole draft wave and says nothing + // about the drafting that IS happening. This line is the split; "resolved" there includes flagged units + // (see PhaseProgress), which is why it can exceed the done count above. + fmt.Fprintf(w, "Phases (units resolved by wave): draft %s · edit %s\n", + waveCell(rep.Progress.Draft), waveCell(rep.Progress.Edit)) fmt.Fprintf(w, "Escalations: %d · post-check misses (confirmed): %d · style-flags (observability): %d\n", rep.Escalations, rep.PostcheckMisses, rep.StyleFlags) // Printed only when the bank actually holds unsigned rows, so a book translating entirely against signed @@ -614,6 +641,15 @@ func renderStatusHuman(w io.Writer, rep *pipeline.StatusReport, cfgPath string) return nil } +// waveCell renders one wave counter, or "—" for a wave this pipeline does not have (a draft-only +// pipeline has no edit wave, and printing "0/0" for it would read as work that never starts). +func waveCell(c pipeline.WaveCounter) string { + if c.Total == 0 { + return "—" + } + return fmt.Sprintf("%d/%d", c.Done, c.Total) +} + // dashIfEmpty renders "" as "—" for a table cell. // dashIfEmpty renders an empty (or blank) cell as an em dash. ONE definition: pack-20 briefly added a // second, near-identical `dash` 400 lines above, and the two disagreed on whitespace — the kind of split diff --git a/backend/cmd/tmctl/render_test.go b/backend/cmd/tmctl/render_test.go index 02ba529c..6094e7ef 100644 --- a/backend/cmd/tmctl/render_test.go +++ b/backend/cmd/tmctl/render_test.go @@ -262,7 +262,7 @@ func TestRenderSignatureStopShowsTheTable(t *testing.T) { var rows []pipeline.BankStopRow rows = append(rows, pipeline.BankStopRow{ Src: "方源", Dst: "Фан Юань", Origin: "both", Type: "name", Freq: 12, Spread: 2, - Variants: []string{"Фан Юань ×3", "Фань Юань ×1"}, Contexts: []string{"方源来到青茅山"}, + Variants: []pipeline.BankStopVariant{{Dst: "Фан Юань", Chunks: 3}, {Dst: "Фань Юань", Chunks: 1}}, Contexts: []string{"方源来到青茅山"}, }) for i := 0; i < 25; i++ { rows = append(rows, pipeline.BankStopRow{Src: fmt.Sprintf("术%d", i), Origin: "mined", Freq: 5}) diff --git a/backend/docs/pipeline.puml b/backend/docs/pipeline.puml index fbb791f2..50921bae 100644 --- a/backend/docs/pipeline.puml +++ b/backend/docs/pipeline.puml @@ -1,5 +1,5 @@ @startuml pipeline -title TextMachine — поток перевода книги (2026-07-25, синхронизировано с D1–D39.25, пост-пак-16: волновой исполнитель)\nПометки: [OK] = построено · [FLAG] = построено за enabled:false · [Ф2] = планово +title TextMachine — поток перевода книги (2026-08-08, синхронизировано с D1–D39.25, пост-пак-16: волновой исполнитель; + блокеры контракта 99/100/101/125/145)\nПометки: [OK] = построено · [FLAG] = построено за enabled:false · [Ф2] = планово skinparam wrapWidth 260 start @@ -15,6 +15,16 @@ GB18030-детект/конверсия, ruby-захват, NormalizeSource; :Чанкинг [OK]: draft-чанки по ВЫХОДНОМУ токен-бюджету (SegBudget пары: фертильность CJK/other), edit-единицы декаплены · precompute sticky по BASE-банку одним упорядоченным проходом; +:МАНИФЕСТ глав и чанков [OK] (строка 100): сайдкар +рядом с БД — project_db + .manifest.json, атомарная запись · +СТАБИЛЬНЫЙ id главы = хеш ИНГЕСТИРОВАННОГО текста главы +(переживает пере-чанковку и правку соседней главы) · +id ЮНИТА, наоборот, умирает с разрезом: глава + тег разреза + leader-индекс · +ключ годности = исходник+чанкер+сегментация+форма пайплайна+пак+embed+norm · +пишет любой write-путь, до прогона — команда tmctl manifest; +:ЭКСПОРТ БАНКА [OK] (строка 125): project_db + .bank.json — +весь банк тремя статусами, id из ключа уникальности +(не автоинкремент glossary.id) · пишется на границах стадий; note right Снапшот per-wave [OK]: draft-волна пинит BASE-банк, edit-волна — ENRICHED (обогащение двигает ТОЛЬКО edit — @@ -29,7 +39,9 @@ partition "ВОЛНА ДРАФТА [OK] (∥ workers, по draft-чанкам)" + src→dst-инъекция глоссария из precomputed selection (Aho-Corasick, trust-gate, спойлер-hard-reject, sticky); :runStage [OK]: чекпойнт-хит → $0-реклассификация · - иначе Reserve (потолки книга/день) → вызов → + иначе Reserve (потолки книга/день · потолок ПРОГОНА + --ceiling-usd перекрывает книжный book_usd, строка 145, + в хеши не входит) → вызов → Settle+чекпойнт ОДНОЙ tx → classify ПОСЛЕ денег; :Банкнота ⟦TM-BANK-v1⟧ [FLAG]: срез с RAW-драфта до classify · чистый драфт → derived-чекпойнт; @@ -56,6 +68,10 @@ if (Майнинг вкл. И дельта непуста? [OK]) then (да) :$0-экраны: канон-конфликты · КОНФЛИКТЫ СВОИХ ЖЕ консолидаций · label-mismatch → стоп-таблица (сигналы топ-варианта · invented · уверенность · конвенций); + :Сайдкары стопа [OK]: .bank-stop.txt — человеку · + .bank-stop.json — машине (строка 101): ПОЛНАЯ таблица, + списки списками, «роль промолчала» = null, не 0 · + плюс обновление экспорта банка; :signature map → прогон ОСТАНОВЛЕН (exit 3); :Владелец: каждый терм → promote в mined_delta (approved+dst) ИЛИ decline в mined_rejects; @@ -88,10 +104,20 @@ partition "ВОЛНА РЕДАКТУРЫ [OK] (∥ по edit-единицам, E } :Read-модели $0 [OK]: tmctl status (wave-aware) · +ПОФАЗНЫЙ прогресс draft N/M и edit N/M в юнитах (строка 99) — +сквозной счётчик стоит на нуле всю черновую волну · quality-report (структурный KPI + residual-скан RepairCandidates = замер остатка для решения о repair) · очередь флагнутых → человек (redrive точечно); -:Экспорт $0 [OK]: манифест-ре-чанк + final-строки chunk_status, +note right + Источник манифеста для status и export по умолчанию — + ПЕРСИСТ (строка 100), а не пере-нарезка: 1.4 с CPU → 0.05-0.12 с + на книге 23 МБ. Протухший ключ = откат на полную нарезку, + подать устаревший знаменатель нельзя по построению. + Пере-нарезку сохраняют пути, которым нужен ТЕКСТ: + export --pairs, контент-ветка ре-билла, quality-report. +end note +:Экспорт $0 [OK]: манифест + final-строки chunk_status, checks.ExportNormalize байт-в-байт с translate, --pairs (source-колонка), счётчики pending/ghost/ConfigDrift; note right diff --git a/backend/internal/chunk/chunker.go b/backend/internal/chunk/chunker.go index 552d3625..2f34bc3a 100644 --- a/backend/internal/chunk/chunker.go +++ b/backend/internal/chunk/chunker.go @@ -103,21 +103,37 @@ func (s SegBudget) EstOut(cjk, other int) float64 { // EstOut + OversizedSentence flag + EditUnitID (WS2); the edit-unit id is monotone across // the book so a the edit wave unit is uniquely addressable. func SplitChunks(chapters []string, seg SegBudget, heading *lang.HeadingRule, abbrevs map[string]bool) []Chunk { + chunks, _ := SplitChunksWithChapters(chapters, seg, heading, abbrevs) + return chunks +} + +// SplitChunksWithChapters is SplitChunks plus the INGESTED text of every chapter that consumed a number, +// index-aligned to the chapter numbers it emitted (kept[0] is chapter 1). It exists so a caller that has +// to identify a chapter — the persisted manifest, backlog row 100 — gets the chapter's own text from the +// ONE place that knows which chapters consume a number, instead of re-deriving that rule and drifting +// from it the first time an "empty chapter" edge case changes. +// +// The text returned is the chapter AS INGESTED — before stripHeading. That is deliberate: an identity +// built on it survives a heading-rule edit (a data change in the pair pack) and a chunker/budget change, +// because neither touches what ingest produced. Nothing else here changes: SplitChunks is this function +// with the second result dropped, so every existing caller is byte-identical. +func SplitChunksWithChapters(chapters []string, seg SegBudget, heading *lang.HeadingRule, abbrevs map[string]bool) (chunks []Chunk, kept []string) { var out []Chunk chapterNo := 0 editUnitID := 0 - for _, chapText := range chapters { + for _, ingested := range chapters { // Title policy (pack-13): detect a leading structural header, render it deterministically and // STRIP its marker from the text the model sees. A nil rule (no pack / no heading.txt) or a chapter // with no header is a NO-OP — headingText is "" and chapText is unchanged, so a book that does not // opt in produces byte-identical chunks. Runs BEFORE splitParagraphs so the stripped subtitle is // re-paragraphed normally. - headingText, chapText := stripHeading(chapText, heading) + headingText, chapText := stripHeading(ingested, heading) paras := splitParagraphs(chapText) if len(paras) == 0 { continue // an empty chapter does not consume a chapter number } chapterNo++ + kept = append(kept, ingested) chapterChunks := chapterDraftChunks(chapterNo, paras, seg, abbrevs) if headingText != "" && len(chapterChunks) > 0 { chapterChunks[0].Heading = headingText // the chapter's first chunk carries the deterministic title @@ -125,7 +141,7 @@ func SplitChunks(chapters []string, seg SegBudget, heading *lang.HeadingRule, ab assignEditUnits(chapterChunks, seg, &editUnitID) out = append(out, chapterChunks...) } - return out + return out, kept } // stripHeading detects a chapter-leading structural header via the pair's HeadingRule, returning the diff --git a/backend/internal/chunk/chunker_manifest_test.go b/backend/internal/chunk/chunker_manifest_test.go new file mode 100644 index 00000000..674a4034 --- /dev/null +++ b/backend/internal/chunk/chunker_manifest_test.go @@ -0,0 +1,70 @@ +package chunk + +import ( + "reflect" + "strings" + "testing" + + "textmachine/backend/internal/lang" +) + +// chunker_manifest_test.go: the seam SplitChunksWithChapters adds for the persisted manifest (backlog +// row 100). Two properties matter and neither is obvious from the signature: the split itself must not +// have changed, and the chapter texts returned must be index-aligned to the chapter NUMBERS the split +// emitted — a chapter that consumes no number must not consume a slot either, or every chapter after the +// first empty one is identified by its neighbour's text. + +var manifestSeg = SegBudget{DraftBudgetOut: 1797, EditCeilingOut: 3200, FertCJK: 1.1978, FertOther: 0.3852} + +func TestSplitChunksWithChaptersLeavesTheSplitUnchanged(t *testing.T) { + chapters := []string{ + "Первая глава книги.", + " \n\t ", // whitespace-only: consumes NO chapter number + "Вторая глава книги.", + "", + strings.Repeat("Длинный абзац для нарезки. ", 200), + } + want := SplitChunks(chapters, manifestSeg, nil, nil) + got, kept := SplitChunksWithChapters(chapters, manifestSeg, nil, nil) + if !reflect.DeepEqual(got, want) { + t.Fatal("SplitChunksWithChapters must produce the SAME cut as SplitChunks — the chunk boundaries decide what goes on the wire") + } + if len(kept) != 3 { + t.Fatalf("three chapters yield text, so three consume a number; got %d: %q", len(kept), kept) + } + // Index alignment: kept[n-1] must be the text of chapter n as the chunker numbered it. + if kept[0] != chapters[0] || kept[1] != chapters[2] || kept[2] != chapters[4] { + t.Fatalf("kept texts are not aligned to the emitted chapter numbers: %q", kept) + } + // …and every emitted chapter number has a slot. + highest := 0 + for _, c := range got { + if c.Chapter > highest { + highest = c.Chapter + } + } + if highest != len(kept) { + t.Fatalf("the split emitted chapters up to %d but returned %d chapter texts", highest, len(kept)) + } +} + +// TestSplitChunksWithChaptersReturnsPreStripText: the text handed back is the chapter AS INGESTED, with +// its source header still in it. That is what makes an identity built on it survive a heading-rule edit — +// the rule changes what is STRIPPED and what title is rendered, not what ingest produced. Asserted with a +// rule ON and OFF over the same chapter, so the invariance is measured rather than assumed. +func TestSplitChunksWithChaptersReturnsPreStripText(t *testing.T) { + chapters := []string{"第一节：подзаголовок\nтело главы"} + rule := &lang.HeadingRule{Marker: "第", Units: map[rune]bool{'节': true}, Template: "Глава {n}"} + + withRule, keptWith := SplitChunksWithChapters(chapters, manifestSeg, rule, nil) + withoutRule, keptWithout := SplitChunksWithChapters(chapters, manifestSeg, nil, nil) + if len(withRule) == 0 || len(withoutRule) == 0 { + t.Fatalf("fixture drifted: %d/%d chunks", len(withRule), len(withoutRule)) + } + if withRule[0].Heading != "Глава 1" || strings.Contains(withRule[0].Text, "第一节") { + t.Fatalf("precondition: the rule must render a title and strip the marker, got heading=%q text=%q", withRule[0].Heading, withRule[0].Text) + } + if keptWith[0] != chapters[0] || keptWithout[0] != chapters[0] { + t.Fatalf("the chapter text must be returned AS INGESTED under both rules, got %q / %q", keptWith[0], keptWithout[0]) + } +} diff --git a/backend/internal/pipeline/artifact.go b/backend/internal/pipeline/artifact.go new file mode 100644 index 00000000..8cf9bfc5 --- /dev/null +++ b/backend/internal/pipeline/artifact.go @@ -0,0 +1,60 @@ +package pipeline + +import ( + "fmt" + "os" + "path/filepath" +) + +// artifact.go: the shared write discipline of the engine's READ-OUT FILES — the sidecars beside the +// project DB that another process (the platform, D39.81/D39.85) reads while this one runs: the +// chapter/chunk manifest (row 100), the machine bank-stop table (row 101), the bank export (row 125). +// +// They have one property in common that the older sidecars did not have to care about: they are read +// CONCURRENTLY with the run that writes them. A plain os.WriteFile truncates first, so a reader that +// opens the file in that window gets a valid path, zero bytes and a parse error indistinguishable from +// a corrupt artifact. Write-then-rename makes every read see either the previous document or the next +// one, whole — the same guarantee the store gets from its transactions, at file granularity. + +// writeFileAtomic replaces path with data through a temp file in the SAME directory + rename. Same +// directory is load-bearing: rename is atomic only within a filesystem, and the project dir is where the +// reader is looking anyway. Every RETURNED failure removes its temp file, so a failed write leaves the +// previous document in place and nothing beside it; a kill -9 mid-write can still leave one dot-prefixed +// temp file, which is inert (no reader looks at it) and is not worth a startup sweep. +func writeFileAtomic(path string, data []byte) error { + dir, base := filepath.Dir(path), filepath.Base(path) + f, err := os.CreateTemp(dir, "."+base+".tmp-*") + if err != nil { + return fmt.Errorf("pipeline: create temp for %s: %w", path, err) + } + tmp := f.Name() + if _, err := f.Write(data); err != nil { + f.Close() + _ = os.Remove(tmp) + return fmt.Errorf("pipeline: write %s: %w", tmp, err) + } + // Sync before the rename: without it the rename's metadata can reach disk ahead of the bytes, so a host + // crash can leave a present-but-truncated document under the real name. The manifest would survive that + // (a parse failure degrades to the re-chunk), but the bank export and the stop table have no such + // validation — a consumer would read a short document as a complete one. + if err := f.Sync(); err != nil { + f.Close() + _ = os.Remove(tmp) + return fmt.Errorf("pipeline: sync %s: %w", tmp, err) + } + if err := f.Close(); err != nil { + _ = os.Remove(tmp) + return fmt.Errorf("pipeline: close %s: %w", tmp, err) + } + // 0o644 like the other sidecars: CreateTemp makes 0o600, and an artifact another process reads must + // not depend on that process running as the same user. + if err := os.Chmod(tmp, 0o644); err != nil { + _ = os.Remove(tmp) + return fmt.Errorf("pipeline: chmod %s: %w", tmp, err) + } + if err := os.Rename(tmp, path); err != nil { + _ = os.Remove(tmp) + return fmt.Errorf("pipeline: replace %s: %w", path, err) + } + return nil +} diff --git a/backend/internal/pipeline/bankexport.go b/backend/internal/pipeline/bankexport.go new file mode 100644 index 00000000..10a6ee6a --- /dev/null +++ b/backend/internal/pipeline/bankexport.go @@ -0,0 +1,161 @@ +package pipeline + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "sort" + "strconv" + "strings" + + "textmachine/backend/internal/store" +) + +// bankexport.go: the BANK EXPORT artifact (backlog row 125) — the engine's read-out of the whole memory +// bank, all three statuses, as a file beside the project DB. +// +// It exists because the bank lives in the engine's PRIVATE SQLite and the platform must not read that +// store (D39.85), so a read channel for it had no carrier at all. It is deliberately NOT the same thing +// as the bank-stop table (row 101): that one is the set of DECISIONS put in front of the owner at one +// stop — proposals, evidence, why-this-rendering — and it exists only for a run that mined something. +// This one is the STATE of the bank as it stands: every row the book has, including seed rows nobody +// ever had to decide on, and it exists for every book with a bank whether anything was mined or not. +// +// It is a projection, never a source: nothing reads it back into the engine. Signing still goes through +// the ratified stop mechanics (promote into the mined-delta file / decline in the mined-rejects file), +// because the pipeline REPLACES a book's whole glossary from its deterministic inputs on every run and a +// row edit would be erased by the next one. + +// bankExportVersion versions the SHAPE of the document (not its content), so a consumer that has to +// tolerate the field set changing has something to branch on. +const bankExportVersion = "tm-bank-v1" + +// BankExport is the whole-bank read-out. +type BankExport struct { + Version string `json:"bank_version"` + BookID string `json:"book_id"` + // Total / Signed are whole-bank counters (not counters of some page): rows in the file, and rows of + // those whose status is `approved` — the only status that is canon on the wire. + Total int `json:"total"` + Signed int `json:"signed"` + Terms []BankExportTerm `json:"terms"` +} + +// BankExportTerm is one bank row. +// +// The field names follow the ENGINE's own vocabulary for the term's surfaces (`src`/`dst`) and call the +// provenance column `origin`, because the engine's own column name for provenance is `source` and using +// that word here would make `source` mean the term's text in one schema and its provenance in the other. +type BankExportTerm struct { + // ID is a STABLE key derived from the row's uniqueness key (src, sense, since_ch, until_ch) — NOT + // glossary.id. That column is a fresh autoincrement on every bank replace (store/migrate.go v5), and + // the bank is replaced on every run, so an exported autoincrement would re-point under a consumer + // between two reads of the same unchanged term. Derived, it survives exactly as long as the term does. + ID string `json:"id"` + Src string `json:"src"` + Dst string `json:"dst"` // "" for a candidate with no proposed rendering yet + // Kind is the engine's `type` column and is LEGITIMATELY empty: a ruby candidate that is neither a + // name nor a place carries none (membank/memseed.go). It is exported as the empty string rather than + // omitted — "the engine did not decide" is a state the row still needs signing in, and a missing field + // would invite a consumer to invent one. + Kind string `json:"kind"` + // Status is the three-valued signing state auto|draft|approved; only `approved` is injected as canon. + Status string `json:"status"` + // Origin is the provenance: which path created the row (seed|ruby|mined). It is an axis INDEPENDENT + // of Status — a mined row can be approved, a seed row can be draft. + Origin string `json:"origin"` + Sense string `json:"sense"` + SinceChapter int `json:"since_chapter"` + UntilChapter int `json:"until_chapter"` + Aliases []string `json:"aliases"` +} + +// bankExportPath is where the read-out lands: beside the project DB, like the signature map, the stop +// tables and the auto-bank. That is the book's state directory — it travels with the DB and never enters +// git — and it is the one place a reader already has to know about. +func (r *Runner) bankExportPath() string { return r.Book.ProjectDB + ".bank.json" } + +// bankTermID is the stable derived id (see BankExportTerm.ID). +// +// The four parts are LENGTH-PREFIXED rather than separator-joined. The first version used U+001F and +// asserted in a comment that it "cannot occur in any of them" — an assumption about data the engine never +// validates: `src` and `sense` are free text copied from a seed YAML, and nothing on the load path +// rejects a control character. Length-prefixing makes the encoding injective by construction, so the +// claim does not have to be true (and does not have to be re-checked when a new seed source appears). +// 16 hex chars of SHA-256 is 64 bits, which for a bank of thousands of rows makes a collision an +// irrelevance rather than a risk taken on purpose. +func bankTermID(src, sense string, since, until int) string { + var b strings.Builder + for _, part := range []string{src, sense, strconv.Itoa(since), strconv.Itoa(until)} { + fmt.Fprintf(&b, "%d:%s", len(part), part) + } + sum := sha256.Sum256([]byte(b.String())) + return hex.EncodeToString(sum[:])[:16] +} + +// projectBankExport turns stored rows into the document. Deterministic: rows are sorted by the same +// uniqueness key the id is derived from, so two runs over an unchanged bank produce byte-identical files +// (a consumer can diff them, and a no-op run is visibly a no-op). +func projectBankExport(bookID string, rows []store.GlossaryEntry) BankExport { + out := BankExport{Version: bankExportVersion, BookID: bookID, Total: len(rows), Terms: make([]BankExportTerm, 0, len(rows))} + for _, e := range rows { + if e.Status == "approved" { + out.Signed++ + } + aliases := make([]string, 0, len(e.Aliases)) + for _, a := range e.Aliases { + aliases = append(aliases, a.Alias) + } + sort.Strings(aliases) // the store returns them in insert order; a read-out must not depend on it + out.Terms = append(out.Terms, BankExportTerm{ + ID: bankTermID(e.Src, e.Sense, e.SinceCh, e.UntilCh), + Kind: e.Type, Src: e.Src, Dst: e.Dst, Status: e.Status, Origin: e.Source, + Sense: e.Sense, SinceChapter: e.SinceCh, UntilChapter: e.UntilCh, Aliases: aliases, + }) + } + sort.SliceStable(out.Terms, func(i, j int) bool { + a, b := out.Terms[i], out.Terms[j] + switch { + case a.Src != b.Src: + return a.Src < b.Src + case a.Sense != b.Sense: + return a.Sense < b.Sense + case a.SinceChapter != b.SinceChapter: + return a.SinceChapter < b.SinceChapter + default: + return a.UntilChapter < b.UntilChapter + } + }) + return out +} + +// exportBank writes the read-out. `at` names the boundary that triggered it and rides the log line, so a +// stale artifact can be traced to the moment it was last written rather than guessed at. +// +// FAILURE DISCIPLINE: a read-out that cannot be written is loud but never fatal. Killing a paid run +// because a projection of state the run already holds could not be serialized would trade real money for +// an artifact that the next boundary rewrites anyway. +func (r *Runner) exportBank(ctx context.Context, at string) { + rows, err := r.Store.GlossaryForBook(r.Book.BookID) + if err != nil { + r.Log.WarnContext(ctx, "bank export: could not read the bank; the export artifact was NOT refreshed and may be stale", + "at", at, "path", r.bankExportPath(), "err", err) + return + } + exp := projectBankExport(r.Book.BookID, rows) + body, err := json.MarshalIndent(exp, "", " ") + if err != nil { + r.Log.WarnContext(ctx, "bank export: could not marshal the bank; the export artifact was NOT refreshed and may be stale", + "at", at, "path", r.bankExportPath(), "err", err) + return + } + if err := writeFileAtomic(r.bankExportPath(), append(body, '\n')); err != nil { + r.Log.WarnContext(ctx, "bank export: could not write the export artifact; it was NOT refreshed and may be stale", + "at", at, "path", r.bankExportPath(), "err", err) + return + } + r.Log.InfoContext(ctx, "bank export refreshed", "at", at, "path", r.bankExportPath(), + "terms", exp.Total, "approved", exp.Signed) +} diff --git a/backend/internal/pipeline/bookrun.go b/backend/internal/pipeline/bookrun.go index 096175d3..842679bd 100644 --- a/backend/internal/pipeline/bookrun.go +++ b/backend/internal/pipeline/bookrun.go @@ -109,6 +109,10 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) { // Ingest reads + normalizes the source (txt/epub) into ordered per-chapter text // and captures ruby readings (ingest.go), decoding the txt source per book.encoding // (auto/utf8/gb18030). Offline and deterministic ($0, no LLM). + // Fingerprint the source BEFORE reading it (row 100): the manifest written below must be stamped with + // the identity of the bytes it was actually cut from, and the cut takes seconds on a large book. + // nil = it could not be taken, which costs this run its manifest and nothing else. + srcBefore := r.sourceFingerprintBeforeIngest(ctx) doc, err := chunk.IngestEncoded(r.Book.SourceFile, r.Book.Encoding, r.Book.SourceLang) if err != nil { return nil, err @@ -136,11 +140,19 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) { if err := r.seedGlossary(ctx); err != nil { return nil, err } + // The bank read-out (backlog row 125), at the FIRST boundary where a bank exists. Doing it here rather + // than only at the stops means a book that never mines anything — or a run that dies before the first + // wave — still leaves a readable bank behind, which is the state a reader has to be able to see. + r.exportBank(ctx, "run-start/seeded") - chunks := chunk.SplitChunks(doc.Chapters, r.segBudget(), r.headingRule(), r.sentenceAbbrevs()) + chunks, chapterTexts := chunk.SplitChunksWithChapters(doc.Chapters, r.segBudget(), r.headingRule(), r.sentenceAbbrevs()) if len(chunks) == 0 { return nil, fmt.Errorf("pipeline: source file %s produced no chunks after normalization", r.Book.SourceFile) } + // The chapter/chunk manifest (backlog row 100): persisted HERE, where the split that every paid byte + // is addressed against was just computed, so the artifact and the run can never describe different + // books. Loud but not fatal — see persistManifest. + r.persistManifest(ctx, srcBefore, chapterTexts, chunks) // the precompute pass: the banknote parser's source index (backlog 19). Built here, in the composite // root, because the rule "a bank line must name something the book contains" needs the whole chunk // manifest, and building it anywhere else would let a call path exist without it. @@ -173,5 +185,12 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) { // upserts the per-wave snapshots (draft-wave snapshot base-bank / edit-wave snapshot enriched) itself and pins each // wave's jobs to its own — a the bank-mining stop enrichment moves only edit-wave snapshot, keeping draft-wave checkpoints valid // («re-paid ONCE»). Returns a *WaveSignatureStop when the bank-mining stop stops for owner sign. - return r.translateBookWaves(ctx, chunks, stickySel) + res, err := r.translateBookWaves(ctx, chunks, stickySel) + if err == nil { + // End of the run (row 125). The stop paths refresh the read-out themselves, so this is the boundary + // they do not cover: a run that finished, whose last bank change was the auto-continue re-seed or a + // signed mined-delta loaded at start. + r.exportBank(ctx, "run-finished") + } + return res, err } diff --git a/backend/internal/pipeline/contractblockers_test.go b/backend/internal/pipeline/contractblockers_test.go new file mode 100644 index 00000000..e0f53768 --- /dev/null +++ b/backend/internal/pipeline/contractblockers_test.go @@ -0,0 +1,1313 @@ +package pipeline + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "reflect" + "strconv" + "strings" + "testing" + + "textmachine/backend/internal/chunk" + "textmachine/backend/internal/obs" +) + +// contractblockers_test.go: the five engine surfaces the API contract is blocked on — per-phase progress +// (backlog row 99), the persisted chapter/chunk manifest (100), the machine bank-stop table (101), the +// bank export (125) and the per-run ceiling (145). Every test here runs against the mock provider or a +// pure projection: no paid call exists in this file. + +// --- row 99: per-phase progress ------------------------------------------------------------------ + +// TestPhaseProgressSplitsTheWaves pins the defect the row was raised on: the end-to-end unit counter needs +// BOTH waves to have resolved a unit ok, so a book whose drafts are all done but whose edits are not reads +// 0/N — indistinguishable from a book where nothing has happened. The per-wave counters have to tell those +// two states apart. +func TestPhaseProgressSplitsTheWaves(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ\fВТОРАЯ"}) + ctx := context.Background() + + r := newRunner(t, bookPath) + if _, err := r.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + rep, err := r.Status(ctx) + if err != nil { + t.Fatal(err) + } + if rep.Progress.Draft != (WaveCounter{Done: 2, Total: 2}) || rep.Progress.Edit != (WaveCounter{Done: 2, Total: 2}) { + t.Fatalf("a finished book must read full on BOTH waves, got %+v", rep.Progress) + } + + // Now remove the edit row of chapter 1 — the exact shape of a run stopped between the waves (the bank + // stop sits there, and it is the state the row's "0% for the whole draft wave" complaint describes). + if err := r.Store.ResetChunkStages(r.Book.BookID, 1, 0, []string{"edit"}); err != nil { + t.Fatal(err) + } + rep, err = r.Status(ctx) + if err != nil { + t.Fatal(err) + } + r.Close() + if rep.Done != 1 { + t.Fatalf("the end-to-end counter must still be 1 (the row's defect), got %d", rep.Done) + } + if rep.Progress.Draft != (WaveCounter{Done: 2, Total: 2}) { + t.Fatalf("the draft wave finished BOTH units and must say so, got %+v", rep.Progress.Draft) + } + if rep.Progress.Edit != (WaveCounter{Done: 1, Total: 2}) { + t.Fatalf("the edit wave resolved one of two units, got %+v", rep.Progress.Edit) + } + // …and per chapter (contract companion K-10: a chapter tree built on the end-to-end counter shows every + // chapter at zero for the whole draft wave). + if len(rep.Chapters) != 2 { + t.Fatalf("want 2 chapter passports, got %d", len(rep.Chapters)) + } + if rep.Chapters[0].Progress.Draft != (WaveCounter{Done: 1, Total: 1}) || rep.Chapters[0].Progress.Edit != (WaveCounter{Done: 0, Total: 1}) { + t.Fatalf("chapter 1 passport progress = %+v", rep.Chapters[0].Progress) + } + if rep.Chapters[1].Progress.Edit != (WaveCounter{Done: 1, Total: 1}) { + t.Fatalf("chapter 2 passport progress = %+v", rep.Chapters[1].Progress) + } +} + +// TestPhaseProgressCountsAFlaggedUnitAsResolved pins the semantics the counters are documented with: a +// wave's counter reaches its own denominator on a book with flagged units, because a flagged unit is +// finished work. Counting only ok units would leave a progress bar permanently short of 100% on any book +// with one bad chunk, and the ok/flagged split is carried unconflated by Done/Flagged. +func TestPhaseProgressCountsAFlaggedUnitAsResolved(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, func(body string) (string, string) { + if strings.Contains(body, "ОТКАЗНАЯГЛАВА") && !isEditBody(body) { + return "Извините, я не могу перевести это.", "stop" // soft refusal → the unit is flagged + } + return draftEdit(body) + }) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ОТКАЗНАЯГЛАВА\fНОРМАЛЬНАЯГЛАВА", regenerate: 0}) + ctx := context.Background() + + r := newRunner(t, bookPath) + defer r.Close() + if _, err := r.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + rep, err := r.Status(ctx) + if err != nil { + t.Fatal(err) + } + if rep.Done != 1 || rep.Flagged != 1 { + t.Fatalf("fixture drifted: want 1 done + 1 flagged, got %+v", rep) + } + if rep.Progress.Draft != (WaveCounter{Done: 2, Total: 2}) || rep.Progress.Edit != (WaveCounter{Done: 2, Total: 2}) { + t.Fatalf("both waves resolved both units (one ok, one flagged/skipped) and must read full: %+v", rep.Progress) + } +} + +// TestPhaseProgressHasNoEditWaveOnADraftOnlyPipeline: a pipeline with no editor must report an edit total +// of 0, not a total it can never reach. A consumer reads "this phase does not exist" from the zero +// denominator; "0/N forever" would be a progress bar that is broken by design. +func TestPhaseProgressHasNoEditWaveOnADraftOnlyPipeline(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ\fВТОРАЯ", draftOnly: true}) + ctx := context.Background() + + r := newRunner(t, bookPath) + defer r.Close() + if _, err := r.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + rep, err := r.Status(ctx) + if err != nil { + t.Fatal(err) + } + if rep.Progress.Draft != (WaveCounter{Done: 2, Total: 2}) { + t.Fatalf("draft counter = %+v", rep.Progress.Draft) + } + if rep.Progress.Edit != (WaveCounter{}) { + t.Fatalf("a draft-only pipeline has no edit wave; want a zero counter, got %+v", rep.Progress.Edit) + } +} + +// --- row 100: the persisted manifest ------------------------------------------------------------- + +// TestManifestServesTheReadModelsIdentically is the load-bearing claim of the manifest: the projections +// built from it must be the SAME projections, not merely similar ones. It runs each read model twice — +// once served by the manifest, once with the manifest removed so the code re-ingests and re-cuts — and +// compares the whole structure. +func TestManifestServesTheReadModelsIdentically(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ\fВТОРАЯ\fТРЕТЬЯ"}) + ctx := context.Background() + + r := newRunner(t, bookPath) + defer r.Close() + if _, err := r.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + if r.loadManifest() == nil { + t.Fatal("a translate must leave a valid manifest behind — the read paths depend on it") + } + + withManifest, err := r.Status(ctx) + if err != nil { + t.Fatal(err) + } + expWith, err := r.Export(false) + if err != nil { + t.Fatal(err) + } + if err := os.Remove(r.manifestPath()); err != nil { + t.Fatal(err) + } + if r.loadManifest() != nil { + t.Fatal("a removed manifest must not be served from anywhere") + } + withoutManifest, err := r.Status(ctx) + if err != nil { + t.Fatal(err) + } + expWithout, err := r.Export(false) + if err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(withManifest, withoutManifest) { + t.Fatalf("the manifest path must project the SAME status report\nwith: %+v\nwithout: %+v", withManifest, withoutManifest) + } + if !reflect.DeepEqual(expWith, expWithout) { + t.Fatalf("export differs between the manifest path and the re-chunk path") + } + // `--pairs` is the mode the manifest deliberately does NOT serve (it stores structure, not text). Pin + // that it still carries a source column, so the fast path can never silently empty the FP-measure's + // src side — the one way a text-free manifest could do real damage. + pairs, err := r.Export(true) + if err != nil { + t.Fatal(err) + } + for _, ce := range pairs.Chunks { + if strings.TrimSpace(ce.Source) == "" { + t.Fatalf("--pairs must still emit the source column: %+v", ce) + } + } +} + +// TestManifestIsIgnoredWhenTheSourceChanges: staleness is decided by the validity key, so a source edit +// makes the stored manifest unusable BEFORE it can hand a read model a wrong denominator. This is the +// property that lets the fast path exist at all — the accelerator has to degrade to the slow path, never +// to a stale answer. +func TestManifestIsIgnoredWhenTheSourceChanges(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ\fВТОРАЯ"}) + ctx := context.Background() + + r := newRunner(t, bookPath) + defer r.Close() + if _, err := r.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + if r.loadManifest() == nil { + t.Fatal("precondition: a fresh manifest must be valid") + } + // A third chapter appears in the source. The stored manifest still says two. + if err := os.WriteFile(r.Book.SourceFile, []byte("ПЕРВАЯ\fВТОРАЯ\fТРЕТЬЯ"), 0o644); err != nil { + t.Fatal(err) + } + if m := r.loadManifest(); m != nil { + t.Fatalf("a manifest written for another source must be refused, got %d chapters", m.ChaptersTotal) + } + rep, err := r.Status(ctx) + if err != nil { + t.Fatal(err) + } + if rep.TotalUnits != 3 { + t.Fatalf("status must fall back to the re-chunk and see the new chapter, got %d units", rep.TotalUnits) + } +} + +// TestManifestChapterIDSurvivesAReCutAndAnEditElsewhere is the identity guarantee the row demands, and its +// declared boundary, pinned as behaviour rather than prose: +// +// - a different CUT (chunker budget) leaves every chapter id untouched — the id is a function of the +// chapter's ingested text, which the cut does not touch; +// - editing ONE chapter re-mints that chapter's id and leaves the others alone, INCLUDING chapters whose +// dense ordinal shifts — which is exactly what a number-based id cannot do. +func TestManifestChapterIDSurvivesAReCutAndAnEditElsewhere(t *testing.T) { + chapters := []string{"ПЕРВАЯ ГЛАВА", "ВТОРАЯ ГЛАВА", "ТРЕТЬЯ ГЛАВА"} + small := chunk.SegBudget{DraftBudgetOut: 8, EditCeilingOut: 16, FertCJK: 1.2, FertOther: 0.4} + large := chunk.SegBudget{DraftBudgetOut: 4000, EditCeilingOut: 8000, FertCJK: 1.2, FertOther: 0.4} + + idsOf := func(chs []string, seg chunk.SegBudget) []string { + chunks, kept := chunk.SplitChunksWithChapters(chs, seg, nil, nil) + if len(chunks) == 0 { + t.Fatalf("fixture produced no chunks") + } + occurrence := map[string]int{} + out := make([]string, len(kept)) + for i, txt := range kept { + occurrence[txt]++ + out[i] = manifestChapterID(txt, occurrence[txt]) + } + return out + } + + base := idsOf(chapters, small) + if got := idsOf(chapters, large); !reflect.DeepEqual(base, got) { + t.Fatalf("a re-cut must not move chapter ids:\nsmall budget: %v\nlarge budget: %v", base, got) + } + + // An INSERTION before chapter 2: every later chapter's NUMBER shifts by one, every later chapter's id + // must not move. + inserted := []string{"ПЕРВАЯ ГЛАВА", "НОВАЯ ГЛАВА", "ВТОРАЯ ГЛАВА", "ТРЕТЬЯ ГЛАВА"} + got := idsOf(inserted, small) + if len(got) != 4 { + t.Fatalf("want 4 chapters, got %d", len(got)) + } + if got[0] != base[0] || got[2] != base[1] || got[3] != base[2] { + t.Fatalf("an insertion must renumber, not re-identify:\nbefore: %v\nafter: %v", base, got) + } + // …and an edit INSIDE a chapter re-mints exactly that chapter. + edited := []string{"ПЕРВАЯ ГЛАВА", "ВТОРАЯ ГЛАВА исправленная", "ТРЕТЬЯ ГЛАВА"} + got = idsOf(edited, small) + if got[0] != base[0] || got[2] != base[2] { + t.Fatalf("an edit must touch only its own chapter's id:\nbefore: %v\nafter: %v", base, got) + } + if got[1] == base[1] { + t.Fatal("an edited chapter must get a new id — a bookmark into it is no longer pointing at the same text") + } +} + +// TestManifestChunksReproduceTheCut pins the reconstruction: a chunk list rebuilt from the stored +// structure must have the same positions, the same edit-unit grouping and the same chapter titles as the +// split it was written from. A drift here would show up as a silently wrong denominator, which is the one +// failure mode a read model cannot detect for itself. +func TestManifestChunksReproduceTheCut(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + // A long chapter so the cut produces several chunks and more than one edit unit per chapter. + long := strings.Repeat("Это длинный абзац русского текста для нарезки. ", 60) + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: long + "\fВТОРАЯ ГЛАВА", minMaxTokens: 64}) + r := newRunner(t, bookPath) + defer r.Close() + + full, err := r.bookChunks() + if err != nil { + t.Fatal(err) + } + _, chapterTexts := chunk.SplitChunksWithChapters(mustIngestChapters(t, r), r.segBudget(), r.headingRule(), r.sentenceAbbrevs()) + m := r.buildManifest(chapterTexts, full, sourceFingerprint{SHA: "sha-fixture", Bytes: 42}) + got := m.chunks() + if len(got) != len(full) { + t.Fatalf("manifest reconstructs %d chunks, the cut has %d", len(got), len(full)) + } + for i := range full { + if got[i].Chapter != full[i].Chapter || got[i].ChunkIdx != full[i].ChunkIdx || + got[i].EditUnitID != full[i].EditUnitID || got[i].Heading != full[i].Heading { + t.Fatalf("chunk %d differs:\nmanifest: %+v\ncut: %+v", i, got[i], full[i]) + } + } + if m.ChunksTotal != len(full) || m.UnitsTotal != len(r.outputUnits(full)) { + t.Fatalf("manifest totals wrong: chunks=%d units=%d (cut: %d/%d)", + m.ChunksTotal, m.UnitsTotal, len(full), len(r.outputUnits(full))) + } + // Every unit id must be unique — it is the key a reader addresses a unit by. + seen := map[string]bool{} + for _, c := range m.Chapters { + for _, u := range c.Units { + if seen[u.ID] { + t.Fatalf("duplicate unit id %q", u.ID) + } + seen[u.ID] = true + } + } +} + +// TestManifestCommandProducesATreeBeforeAnyRun: the chapter tree has to exist for a book that has been cut +// but never translated (a library shows it as parsed-not-started), and every other producer of that +// structure is a paid path. $0: no provider is reachable in this test at all. +func TestManifestCommandProducesATreeBeforeAnyRun(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ\fВТОРАЯ\fТРЕТЬЯ"}) + srv.Close() // nothing here may reach a provider + + r, err := NewReadOnlyRunner(bookPath, obs.NewLogger()) + if err != nil { + t.Fatal(err) + } + defer r.Close() + m, err := r.BuildAndPersistManifest() + if err != nil { + t.Fatal(err) + } + if m.ChaptersTotal != 3 || m.UnitsTotal != 3 { + t.Fatalf("want a 3-chapter tree, got chapters=%d units=%d", m.ChaptersTotal, m.UnitsTotal) + } + for i, c := range m.Chapters { + if c.ID == "" || c.Number != i+1 || len(c.Units) == 0 { + t.Fatalf("chapter %d is not addressable: %+v", i, c) + } + } + raw, err := os.ReadFile(r.ManifestPath()) + if err != nil { + t.Fatal(err) + } + var stored BookManifest + if err := json.Unmarshal(raw, &stored); err != nil { + t.Fatalf("the persisted manifest must be readable JSON: %v", err) + } + if !reflect.DeepEqual(&stored, m) { + t.Fatal("the persisted document and the returned one must be the same document") + } + if rec.count() != 0 { + t.Fatalf("the manifest command must make ZERO provider calls, made %d", rec.count()) + } +} + +// mustIngestChapters re-reads the book's ingested chapters for a test that needs both halves of the split. +func mustIngestChapters(t *testing.T, r *Runner) []string { + t.Helper() + doc, err := chunk.IngestEncoded(r.Book.SourceFile, r.Book.Encoding, r.Book.SourceLang) + if err != nil { + t.Fatal(err) + } + return doc.Chapters +} + +// --- row 101: the machine bank-stop table --------------------------------------------------------- + +// TestBankStopMachineTableCarriesTheWholeTable pins row 101's two claims at once: the machine sidecar +// holds the FULL table (stdout is capped at 20 and the text sidecar is a column layout no consumer can +// parse back), and it carries the arbitration fields the fix-pack added — signals, invented, confidence, +// conventions, contradictions — with the one distinction a rendering loses: "the role said nothing" is +// null, not zero. +func TestBankStopMachineTableCarriesTheWholeTable(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, func(body string) (string, string) { + if isTerminologyBody(body) { + return "方源\tГуюэ Фан Юань\t40", "stop" // a rendering no draft proposed, with a confidence + } + return "Фан Юань пришёл." + "\n" + bankSeparator + "\n方源\tФан Юань\tname\n方源\tфан юань\tname", "stop" + }) + defer srv.Close() + r := newVerifyRunner(t, setupMiningStopProject(t, srv.URL, miningStopOpts{terminology: true})) + defer r.Close() + stop := runToSignatureStop(t, r) + + if stop.TableJSONPath == "" { + t.Fatal("the stop must tell the operator where the machine table is") + } + var doc bankStopTableFile + mustParseJSON(t, stop.TableJSONPath, &doc) + if doc.Version != bankStopTableVersion || doc.BookID != r.Book.BookID { + t.Fatalf("machine table header wrong: %+v", doc) + } + // The FULL table: as many rows as the in-memory table the CLI caps for stdout. + if doc.Terms != len(stop.Rows) || len(doc.Rows) != len(stop.Rows) { + t.Fatalf("the machine table must hold every row (%d), got terms=%d rows=%d", len(stop.Rows), doc.Terms, len(doc.Rows)) + } + var fang *bankStopRowJSON + for i := range doc.Rows { + if doc.Rows[i].Src == "方源" { + fang = &doc.Rows[i] + } + } + if fang == nil { + t.Fatalf("方源 missing from the machine table: %+v", doc.Rows) + } + if fang.Conf == nil || *fang.Conf != 40 { + t.Fatalf("the role's stated confidence must survive as a number, got %v", fang.Conf) + } + if !fang.Invented || fang.Conventions != 1 || fang.Spread < 2 || len(fang.Signals) == 0 || len(fang.Contexts) == 0 { + t.Fatalf("the arbitration record is incomplete: %+v", *fang) + } + // Variants ship in PARTS, not as the sentence the text table prints: «×» and the parenthesis of + // "(proposed for …)" both occur inside real renderings, so a joined label cannot be split back. + if len(fang.Variants) == 0 || fang.Variants[0].Dst == "" || fang.Variants[0].Chunks <= 0 { + t.Fatalf("a variant must arrive as {dst, chunks, via}, got %+v", fang.Variants) + } + if strings.Contains(fang.Variants[0].Dst, "×") { + t.Fatalf("the machine surface must not carry the rendered label: %+v", fang.Variants[0]) + } + // A row the role never mentioned carries null, not 0 — the two are opposites on a review sheet. + sawAbsent := false + for _, row := range doc.Rows { + if row.Src != "方源" && row.Conf == nil { + sawAbsent = true + } + } + if !sawAbsent { + t.Fatalf("fixture drifted: no row without a stated confidence, so the null/zero distinction is untested: %+v", doc.Rows) + } + // Rows must arrive in the same order the reference text sidecar uses, so the two documents can be read + // against each other line by line. + for i := range doc.Rows { + if doc.Rows[i].Src != stop.Rows[i].Src { + t.Fatalf("row %d: machine table %q, in-memory table %q — the two orders must agree", i, doc.Rows[i].Src, stop.Rows[i].Src) + } + } + // …and the bank read-out (row 125) is refreshed AT the stop, which is the boundary a signing screen + // reads at. The two artifacts are distinct: this one is the decisions, that one is the state. + // This fixture seeds nothing and the mined terms are precisely what is AWAITING signature, so the bank + // legitimately holds zero rows here — the artifact still has to exist and be self-consistent, because + // "the bank is empty" is an answer a signing screen needs and a missing file is not. + var bank BankExport + mustParseJSON(t, r.bankExportPath(), &bank) + if bank.BookID != r.Book.BookID || bank.Total != len(bank.Terms) || bank.Version != bankExportVersion { + t.Fatalf("the bank read-out at the stop is not self-consistent: %+v", bank) + } +} + +// --- row 125: the bank export -------------------------------------------------------------------- + +// TestBankExportCarriesEveryStatusWithStableIDs pins what the artifact is for: the WHOLE bank, all three +// statuses, with an id a consumer can hold across runs. The id must NOT be glossary.id — that column is a +// fresh autoincrement on every bank replace, so an exported one would re-point under a reader between two +// reads of an unchanged term. +func TestBankExportCarriesEveryStatusWithStableIDs(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + seed := `terms: + - src: 花海 + dst: Море цветов + type: place + status: approved + - src: 方源 + dst: Фан Юань + type: name + status: draft +` + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ", glossarySeed: seed}) + ctx := context.Background() + + r := newRunner(t, bookPath) + if _, err := r.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + first := readBankExport(t, r.bankExportPath()) + r.Close() + + if first.Total != 2 || first.Signed != 1 { + t.Fatalf("want 2 terms / 1 approved, got %+v", first) + } + byStatus := map[string]BankExportTerm{} + for _, tm := range first.Terms { + byStatus[tm.Status] = tm + } + if byStatus["approved"].Src != "花海" || byStatus["approved"].Kind != "place" || byStatus["approved"].Origin != "seed" { + t.Fatalf("the approved row lost a field: %+v", byStatus["approved"]) + } + if byStatus["draft"].Src != "方源" || byStatus["draft"].Dst != "Фан Юань" { + t.Fatalf("the draft row must be exported too — a boolean `signed` would merge it with the auto rows: %+v", byStatus["draft"]) + } + + // A second run REPLACES the bank (the pipeline rebuilds it from its deterministic inputs, so every + // glossary.id is new). The exported ids must not move. + r2 := newRunner(t, bookPath) + if _, err := r2.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + second := readBankExport(t, r2.bankExportPath()) + r2.Close() + if !reflect.DeepEqual(first, second) { + t.Fatalf("an unchanged bank must export byte-identically across runs:\nrun1: %+v\nrun2: %+v", first, second) + } + // …and the id is the derived one, not the row's autoincrement primary key. + for _, tm := range first.Terms { + if want := bankTermID(tm.Src, tm.Sense, tm.SinceChapter, tm.UntilChapter); tm.ID != want { + t.Fatalf("term %q carries id %q, want the key-derived %q", tm.Src, tm.ID, want) + } + } +} + +// readBankExport loads the artifact, failing the test if it is absent or malformed. +func readBankExport(t *testing.T, path string) BankExport { + t.Helper() + raw, err := os.ReadFile(path) + if err != nil { + t.Fatalf("the bank export artifact must exist after a run: %v", err) + } + var exp BankExport + if err := json.Unmarshal(raw, &exp); err != nil { + t.Fatalf("the bank export must be readable JSON: %v", err) + } + if exp.Version != bankExportVersion { + t.Fatalf("bank export version = %q, want %q", exp.Version, bankExportVersion) + } + return exp +} + +// --- row 145: the per-run ceiling ---------------------------------------------------------------- + +// TestRunCeilingArgumentOverridesTheBookAndBookYAMLIsNotWritten: the argument caps THIS run, the book's own +// ceiling is left where it was, and the config file is untouched — the whole point of the row is that a +// caller's number must not become a permanent record in the engine's data. +func TestRunCeilingArgumentOverridesTheBookCeiling(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ\fВТОРАЯ\fТРЕТЬЯ", bookUSD: 10.0}) + before, err := os.ReadFile(bookPath) + if err != nil { + t.Fatal(err) + } + ctx := context.Background() + + r := newRunner(t, bookPath) + r.CeilingUSD = fakeCallUSD * 1.5 // enough for one call, not for the book + _, err = r.TranslateBook(ctx) + if err == nil { + t.Fatal("a run ceiling below the book's cost must stop the run") + } + if !strings.Contains(err.Error(), "ceiling reached") { + t.Fatalf("want a ceiling stop, got %v", err) + } + if !strings.Contains(err.Error(), "--ceiling-usd") { + t.Fatalf("the stop must name the ceiling actually in force, not the book's: %v", err) + } + if r.Book.Ceilings.BookUSD != 10.0 { + t.Fatalf("the book's own ceiling must be untouched, got %v", r.Book.Ceilings.BookUSD) + } + r.Close() + after, err := os.ReadFile(bookPath) + if err != nil { + t.Fatal(err) + } + if string(before) != string(after) { + t.Fatal("book.yaml must not be written by a run-scoped ceiling") + } +} + +// TestRunCeilingMovesNoHashAndResumesWhenRaised is the row's cost claim and its resume claim in one run, +// because they are the same run: the ceiling is in NEITHER BriefHash NOR either wave snapshot, so a run +// stopped by it resumes at $0 under a higher one instead of re-buying what it already paid for. +func TestRunCeilingMovesNoHashAndResumesWhenRaised(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ\fВТОРАЯ\fТРЕТЬЯ", bookUSD: 10.0}) + ctx := context.Background() + + // --- the hashes do not move --- + probe := newRunner(t, bookPath) + briefBefore := probe.Book.BriefHash() + if err := probe.seedGlossary(ctx); err != nil { + t.Fatal(err) + } + draftBefore, _, err := probe.snapshotIDForWave(waveDraft) + if err != nil { + t.Fatal(err) + } + editBefore, _, err := probe.snapshotIDForWave(waveEdit) + if err != nil { + t.Fatal(err) + } + probe.CeilingUSD = 0.01 + draftAfter, _, err := probe.snapshotIDForWave(waveDraft) + if err != nil { + t.Fatal(err) + } + editAfter, _, err := probe.snapshotIDForWave(waveEdit) + if err != nil { + t.Fatal(err) + } + if probe.Book.BriefHash() != briefBefore || draftAfter != draftBefore || editAfter != editBefore { + t.Fatal("a run-scoped ceiling must move NO hash — otherwise capping a run silently re-bills the book") + } + probe.Close() + + // --- a stopped run resumes for free under a raised ceiling --- + r1 := newRunner(t, bookPath) + r1.CeilingUSD = fakeCallUSD * 2.5 // two calls fit, the book needs six + if _, err := r1.TranslateBook(ctx); err == nil { + t.Fatal("the tight ceiling must stop the run") + } + paidCalls := rec.count() + r1.Close() + if paidCalls == 0 { + t.Fatal("fixture drifted: the tight run must have paid for something before stopping") + } + + r2 := newRunner(t, bookPath) + defer r2.Close() + r2.CeilingUSD = 10.0 + res, err := r2.TranslateBook(ctx) + if err != nil { + t.Fatalf("raising the ceiling must let the run continue: %v", err) + } + if res.Flagged != 0 { + t.Fatalf("the resumed run must finish clean, got %d flagged", res.Flagged) + } + // The already-paid attempts must have been REPLAYED, not re-bought: the second run's provider calls are + // only the ones the first never made. + rep, err := r2.Status(ctx) + if err != nil { + t.Fatal(err) + } + if rep.Done != 3 { + t.Fatalf("want 3 done units, got %d", rep.Done) + } + total := rec.count() + if total != 6 { + t.Fatalf("a book of 3 units × 2 stages must cost exactly 6 provider calls across both runs, got %d", total) + } +} + +// TestCeilingsAreAbsentFromEveryHashInput is the grep-proof half of the cost claim, executed rather than +// asserted in prose: the rendered snapshot payload — the thing every request_hash folds — must not mention +// a ceiling at all, under any spelling. +func TestCeilingsAreAbsentFromEveryHashInput(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + r := newRunner(t, setupProjectOpts(t, srv.URL, projectOpts{bookUSD: 7.25})) + defer r.Close() + if err := r.seedGlossary(context.Background()); err != nil { + t.Fatal(err) + } + for _, w := range []wave{waveDraft, waveEdit} { + _, payload, err := r.snapshotIDForWave(w) + if err != nil { + t.Fatal(err) + } + // Names, not substrings: `edit_ceiling_out` is the SEGMENTATION budget and legitimately carries the + // word "ceiling" — matching on it would have made this test pass for the wrong reason forever. + for _, forbidden := range []string{"ceilings", "book_usd", "day_usd", "7.25"} { + if strings.Contains(payload, forbidden) { + t.Fatalf("the snapshot payload of wave %v carries %q — a ceiling edit would re-bill the book:\n%s", w, forbidden, payload) + } + } + } + // The brief canon is the other hash a ceiling could hide in, and it is a hash rather than a document — + // so the executed proof is that moving the ceilings does not move it. + before := r.Book.BriefHash() + r.Book.Ceilings.BookUSD, r.Book.Ceilings.DayUSD = 999, 999 + if after := r.Book.BriefHash(); after != before { + t.Fatalf("BriefHash moved when the ceilings did (%s → %s) — every RequestHash of the book would move with it", before, after) + } +} + +// --- row 101 + 125 + 100 together: the artifacts a run leaves behind ----------------------------- + +// TestRunLeavesTheContractArtifacts walks the artifacts a caller of the engine is told to read and asserts +// each one exists and parses. It is the cheapest possible guard against the failure this pack is most +// exposed to: an artifact that is written on a path no test takes. +func TestRunLeavesTheContractArtifacts(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + r := newRunner(t, setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ\fВТОРАЯ"})) + defer r.Close() + if _, err := r.TranslateBook(context.Background()); err != nil { + t.Fatal(err) + } + var m BookManifest + mustParseJSON(t, r.manifestPath(), &m) + if m.ChaptersTotal != 2 { + t.Fatalf("manifest chapters = %d", m.ChaptersTotal) + } + var b BankExport + mustParseJSON(t, r.bankExportPath(), &b) + if b.BookID != r.Book.BookID { + t.Fatalf("bank export book_id = %q", b.BookID) + } + // The bank-stop table is written only by a run that MINES — this fixture has no contrast artifact, so + // its absence here is correct and is covered by the mining fixture instead. + if _, err := os.Stat(r.bankStopTableJSONPath()); err == nil { + t.Fatal("a book that mines nothing must not leave a stop table behind") + } +} + +func mustParseJSON(t *testing.T, path string, v any) { + t.Helper() + raw, err := os.ReadFile(path) + if err != nil { + t.Fatalf("artifact %s must exist: %v", filepath.Base(path), err) + } + if err := json.Unmarshal(raw, v); err != nil { + t.Fatalf("artifact %s must be readable JSON: %v", filepath.Base(path), err) + } +} + +// --- the fix-pack round: defects the adversarial review found ------------------------------------ + +// TestManifestIsNotWrittenWhenTheSourceMovesUnderTheRead closes the review's top finding. The structure +// is derived from the bytes ingest read; the validity key was stamped from a SECOND read of the same +// path. A source rewritten between the two (a re-upload, an operator's edit — the ingest+split of a 23 MB +// book takes ~1.4 s) produced a document describing the OLD cut under the NEW hash, which then validated +// forever and could never be detected as stale — the exact state manifest.go claims cannot exist. +func TestManifestIsNotWrittenWhenTheSourceMovesUnderTheRead(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ\fВТОРАЯ"}) + r := newRunner(t, bookPath) + defer r.Close() + + // The fingerprint and the cut both come from the ORIGINAL source… + before, err := sourceSHA256(r.Book.SourceFile) + if err != nil { + t.Fatal(err) + } + chunks, chapterTexts := chunk.SplitChunksWithChapters(mustIngestChapters(t, r), r.segBudget(), r.headingRule(), r.sentenceAbbrevs()) + // …and the file changes before the write lands. + if err := os.WriteFile(r.Book.SourceFile, []byte("ПЕРВАЯ\fВТОРАЯ\fТРЕТЬЯ"), 0o644); err != nil { + t.Fatal(err) + } + if _, err := r.writeManifest(before, chapterTexts, chunks); err == nil { + t.Fatal("a manifest cut from bytes that are no longer on disk must NOT be written — it would validate forever against the new hash") + } + if _, serr := os.Stat(r.manifestPath()); serr == nil { + t.Fatal("nothing may be left on disk when the source moved under the read") + } + // And the read path is unaffected: no manifest → full re-chunk → the NEW chapter count. + rep, err := r.Status(context.Background()) + if err != nil { + t.Fatal(err) + } + if rep.TotalUnits != 3 { + t.Fatalf("status must see the current source, got %d units", rep.TotalUnits) + } +} + +// TestManifestKeyFollowsThePipelineShape closes the review's second finding: the document's units[] is a +// projection of `outputUnits`, which groups whole chunks into edit units for an edit pipeline and emits +// one unit per chunk for a draft-only one. Nothing else in the key moves when the editor stage is +// dropped, so without the shipping wave in the key the stored unit tree would keep validating while +// describing groups the run no longer ships. +func TestManifestKeyFollowsThePipelineShape(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: strings.Repeat("Абзац для нарезки. ", 200) + "\fВТОРАЯ", minMaxTokens: 64}) + r := newRunner(t, bookPath) + if _, err := r.BuildAndPersistManifest(); err != nil { + t.Fatal(err) + } + if r.loadManifest() == nil { + t.Fatal("precondition: a fresh manifest must be valid") + } + r.Close() + + // Drop the editor stage — the cut is untouched, the UNIT decomposition is not. + pipePath := filepath.Join(filepath.Dir(bookPath), "pipeline.yaml") + raw, err := os.ReadFile(pipePath) + if err != nil { + t.Fatal(err) + } + var kept []string + for _, line := range strings.Split(string(raw), "\n") { + if !strings.Contains(line, "name: edit,") { + kept = append(kept, line) + } + } + if len(kept) == len(strings.Split(string(raw), "\n")) { + t.Fatal("fixture drifted: no editor stage line to drop") + } + if err := os.WriteFile(pipePath, []byte(strings.Join(kept, "\n")), 0o644); err != nil { + t.Fatal(err) + } + r2 := newRunner(t, bookPath) + defer r2.Close() + if r2.finalStageWave() != waveDraft { + t.Fatal("fixture drifted: the pipeline is still an edit pipeline") + } + if m := r2.loadManifest(); m != nil { + t.Fatalf("a manifest whose unit tree was built for another pipeline shape must be refused, got %d units", m.UnitsTotal) + } +} + +// TestManifestWithBrokenCountersIsRefused: the key is a stored string, so a hand-edited sidecar keeps +// matching while its counters say anything at all — and those counters are read straight into a slice +// capacity and a loop bound. A negative one panics `make` inside a $0 read command. +func TestManifestWithBrokenCountersIsRefused(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + r := newRunner(t, setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ\fВТОРАЯ"})) + defer r.Close() + if _, err := r.BuildAndPersistManifest(); err != nil { + t.Fatal(err) + } + raw, err := os.ReadFile(r.manifestPath()) + if err != nil { + t.Fatal(err) + } + var m BookManifest + if err := json.Unmarshal(raw, &m); err != nil { + t.Fatal(err) + } + for name, mutate := range map[string]func(*BookManifest){ + "negative chunk total": func(x *BookManifest) { x.ChunksTotal = -1 }, + "absurd chunk count": func(x *BookManifest) { x.Chapters[0].Units[0].ChunkCount = 1 << 40 }, + "lying chapter total": func(x *BookManifest) { x.ChaptersTotal = 99 }, + "lying unit total": func(x *BookManifest) { x.UnitsTotal = 0 }, + } { + broken := m + broken.Chapters = append([]ManifestChapter(nil), m.Chapters...) + broken.Chapters[0].Units = append([]ManifestUnit(nil), m.Chapters[0].Units...) + mutate(&broken) + body, err := json.Marshal(broken) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(r.manifestPath(), body, 0o644); err != nil { + t.Fatal(err) + } + if got := r.loadManifest(); got != nil { + t.Fatalf("%s: a manifest whose counters do not describe its own contents must be refused", name) + } + // …and the read model still answers, off the full re-chunk. + if _, err := r.Status(context.Background()); err != nil { + t.Fatalf("%s: status must survive a broken sidecar: %v", name, err) + } + } +} + +// TestBankStopMachineTableIsClearedWhenNothingAwaitsSignature closes the review's finding that the +// machine table was written only on a NON-empty delta: after the owner signed every proposed term, the +// previous run's rows stayed on disk byte-identical, so a signing screen built on the file — the artifact's +// whole purpose — would re-offer terms that were already signed. +func TestBankStopMachineTableIsClearedWhenNothingAwaitsSignature(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, func(body string) (string, string) { + if isEditBody(body) { + return "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД", "stop" + } + return "Фан Юань пришёл к горе Цинмао." + "\n" + bankBlockForMining, "stop" + }) + defer srv.Close() + + bookPath := setupMiningStopProject(t, srv.URL, miningStopOpts{}) + r1 := newVerifyRunner(t, bookPath) + stop := runToSignatureStop(t, r1) + proposed := readSignatureMap(t, r1) + r1.Close() + + var first bankStopTableFile + mustParseJSON(t, stop.TableJSONPath, &first) + if first.Terms == 0 { + t.Fatal("test premise broken: the first run wrote an empty table") + } + + // The owner declines every proposed term — the delta empties and the stop clears. Same project, so the + // same sidecar path. + dir := filepath.Dir(bookPath) + var rejects strings.Builder + rejects.WriteString("rejects:\n") + for _, line := range strings.Split(proposed, "\n") { + if s, ok := strings.CutPrefix(strings.TrimSpace(line), "- src: "); ok { + rejects.WriteString(" - src: " + s + "\n") + } + } + writeFile(t, filepath.Join(dir, "mined-rejects.yaml"), rejects.String()) + raw, err := os.ReadFile(bookPath) + if err != nil { + t.Fatal(err) + } + writeFile(t, bookPath, string(raw)+"\nmined_rejects: mined-rejects.yaml\n") + + r2 := newVerifyRunner(t, bookPath) + defer r2.Close() + if _, err := r2.TranslateBook(context.Background()); err != nil { + t.Fatalf("a fully-declined delta must clear the stop, got %v", err) + } + var second bankStopTableFile + mustParseJSON(t, r2.bankStopTableJSONPath(), &second) + if second.Terms != 0 || len(second.Rows) != 0 { + t.Fatalf("nothing awaits signature, so the machine table must say so rather than keep the previous run's %d row(s); got %d", first.Terms, second.Terms) + } +} + +// TestManifestUnitIDDiesWithTheCut closes the acceptance finding F1. The declared boundary is "chapter +// ids survive a re-chunk, unit ids do not" — and the first version made that statement FALSE for exactly +// the unit a reader is most likely to hold: a chapter's FIRST unit has leader index 0 under every cut, so +// «:0» outlived any re-chunk while naming a different span. A consumer that persists unit ids +// would read "the id is still in the manifest" as "my anchor is still valid". +func TestManifestUnitIDDiesWithTheCut(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + // A chapter long enough to hold several units, so the cut visibly changes when the budget does. + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: strings.Repeat("Это довольно длинный абзац русского текста. ", 300)}) + dir := filepath.Dir(bookPath) + + idsUnder := func(pairYAML string) (chapterIDs, unitIDs []string) { + t.Helper() + writeFile(t, filepath.Join(dir, "pairs", "ja-ru.yaml"), pairYAML) + r := newRunner(t, bookPath) + defer r.Close() + m, err := r.BuildAndPersistManifest() + if err != nil { + t.Fatal(err) + } + for _, c := range m.Chapters { + chapterIDs = append(chapterIDs, c.ID) + for _, u := range c.Units { + unitIDs = append(unitIDs, u.ID) + } + } + return chapterIDs, unitIDs + } + + coarse := "pair: ja-ru\nprompts_root: prompts\nsegmentation:\n draft_budget_out: 4000\n edit_ceiling_out: 8000\n fertility: { cjk: 1.1978, other: 0.3852 }\n" + fine := "pair: ja-ru\nprompts_root: prompts\nsegmentation:\n draft_budget_out: 200\n edit_ceiling_out: 400\n fertility: { cjk: 1.1978, other: 0.3852 }\n" + + chaptersCoarse, unitsCoarse := idsUnder(coarse) + chaptersFine, unitsFine := idsUnder(fine) + + if len(unitsCoarse) == len(unitsFine) { + t.Fatalf("fixture drifted: the two budgets produced the same number of units (%d) — the cut did not change", len(unitsCoarse)) + } + if !reflect.DeepEqual(chaptersCoarse, chaptersFine) { + t.Fatalf("a re-cut must NOT move chapter ids:\ncoarse: %v\nfine: %v", chaptersCoarse, chaptersFine) + } + // The load-bearing assertion: NO unit id may survive the re-cut, the chapter's first one included. + held := map[string]bool{} + for _, id := range unitsCoarse { + held[id] = true + } + for _, id := range unitsFine { + if held[id] { + t.Fatalf("unit id %q survived a re-cut — a consumer holding it would anchor onto different text", id) + } + } +} + +// TestBankExportIsRefreshedWhenARedriveReSeedsAndAborts closes acceptance finding F2: a redrive re-seeds +// the bank BEFORE its destructive reset, and every guard after that point can abort the command — leaving +// the store holding one bank and the exported file describing another. +func TestBankExportIsRefreshedWhenARedriveReSeedsAndAborts(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, func(body string) (string, string) { + if strings.Contains(body, "ОТКАЗНАЯГЛАВА") && !isEditBody(body) { + return "Извините, я не могу перевести это.", "stop" + } + return draftEdit(body) + }) + defer srv.Close() + seed := "terms:\n - src: 花海\n dst: Море цветов\n type: place\n status: approved\n" + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ОТКАЗНАЯГЛАВА\fНОРМАЛЬНАЯГЛАВА", regenerate: 0, glossarySeed: seed}) + ctx := context.Background() + + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + r1.Close() + + // The seed grows between the runs, and the redrive is made to ABORT after its re-seed by pointing the + // config at a snapshot the stored rows do not carry (no --resnapshot ⇒ the drift guard refuses). + writeFile(t, filepath.Join(filepath.Dir(bookPath), "glossary-seed.yaml"), + seed+" - src: 方源\n dst: Фан Юань\n type: name\n status: approved\n") + r2 := newRunner(t, bookPath) + defer r2.Close() + if _, _, err := r2.Redrive(ctx, RedriveSelector{Chapter: -1, ChunkIdx: -1}); err == nil { + t.Fatal("test premise broken: the redrive was expected to abort on the drift guard") + } + exp := readBankExport(t, r2.bankExportPath()) + if exp.Total != 2 { + t.Fatalf("the redrive re-seeded the bank to 2 terms; the export must describe the bank the engine now holds, got %d: %+v", exp.Total, exp.Terms) + } +} + +// TestBankStopMachineTableIsClearedWhenMiningIsUnconfigured closes acceptance finding F3: a signing table +// left beside the DB by a run that COULD mine must not outlive the configuration that produced it, or it +// offers terms for signature forever. +func TestBankStopMachineTableIsClearedWhenMiningIsUnconfigured(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + r := newRunner(t, setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ"})) + defer r.Close() + + // A table from when the book could mine. + if err := r.writeBankStopTableJSON([]BankStopRow{{Src: "方源", Dst: "Фан Юань", Origin: "mined", Freq: 5}}); err != nil { + t.Fatal(err) + } + // This fixture has no contrast artifact, so the stop takes the not-configured path. + if _, err := r.TranslateBook(context.Background()); err != nil { + t.Fatal(err) + } + var doc bankStopTableFile + mustParseJSON(t, r.bankStopTableJSONPath(), &doc) + if doc.Terms != 0 || len(doc.Rows) != 0 { + t.Fatalf("a book that cannot mine must not keep offering %d term(s) for signature", doc.Terms) + } +} + +// TestBankStopMachineTableIsNotCreatedForABookThatNeverMines is the other half of F3's fix: clearing must +// not become "every book acquires a signing sidecar it has no use for". +func TestBankStopMachineTableIsNotCreatedForABookThatNeverMines(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + r := newRunner(t, setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ"})) + defer r.Close() + if _, err := r.TranslateBook(context.Background()); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(r.bankStopTableJSONPath()); err == nil { + t.Fatal("a book with no mining configured must leave no signing table behind") + } +} + +// TestManifestWithAbsurdButConsistentCountersIsRefused closes acceptance finding V2-2: the upper bound in +// selfConsistent was written but nothing exercised it — a verifier removed it and the whole battery +// stayed green. The counters in TestManifestWithBrokenCountersIsRefused are all INCONSISTENT, so they die +// on the equality checks and never reach the bound; this one is internally consistent and absurd, which +// is the only shape the bound is for. +// +// The assertion deliberately stops at loadManifest: with the bound removed the reconstruction would +// allocate the claimed number of chunks, and a test that reached it would take the machine down instead +// of failing. +func TestManifestWithAbsurdButConsistentCountersIsRefused(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + r := newRunner(t, setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ"})) + defer r.Close() + if _, err := r.BuildAndPersistManifest(); err != nil { + t.Fatal(err) + } + raw, err := os.ReadFile(r.manifestPath()) + if err != nil { + t.Fatal(err) + } + var m BookManifest + if err := json.Unmarshal(raw, &m); err != nil { + t.Fatal(err) + } + // One chapter, one unit, a chunk count just past the bound — and every counter agreeing with it, so + // the equality checks pass and only the bound can refuse. + absurd := manifestMaxChunks + 1 + m.Chapters = m.Chapters[:1] + m.Chapters[0].Units = m.Chapters[0].Units[:1] + m.Chapters[0].Units[0].ChunkCount = absurd + m.Chapters[0].UnitsTotal, m.Chapters[0].ChunksTotal = 1, absurd + m.ChaptersTotal, m.UnitsTotal, m.ChunksTotal = 1, 1, absurd + if !m.selfConsistentExceptTheBound() { + t.Fatal("test premise broken: the mutated document must be internally consistent, so that ONLY the bound rejects it") + } + body, err := json.Marshal(m) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(r.manifestPath(), body, 0o644); err != nil { + t.Fatal(err) + } + if got := r.loadManifest(); got != nil { + t.Fatalf("a manifest claiming %d chunks must be refused before anything allocates them", absurd) + } +} + +// selfConsistentExceptTheBound re-checks only the EQUALITIES selfConsistent enforces, so the test above +// can assert its own premise ("the equalities pass, only the bound refuses") instead of assuming it. +func (m *BookManifest) selfConsistentExceptTheBound() bool { + if m.ChaptersTotal != len(m.Chapters) { + return false + } + units, chunks := 0, 0 + for _, c := range m.Chapters { + if c.Number < 1 || c.UnitsTotal != len(c.Units) { + return false + } + for _, u := range c.Units { + if u.ChunkCount < 1 || u.FirstChunkIdx < 0 { + return false + } + chunks += u.ChunkCount + } + units += len(c.Units) + } + return m.UnitsTotal == units && m.ChunksTotal == chunks +} + +// TestManifestOfAnOlderDocumentVersionIsRefused closes acceptance finding V2-4: a sidecar written before +// unit ids gained the cut tag is structurally valid and would keep validating, handing a reader ids in the +// old form — the very form whose defect (a chapter's first unit outliving every re-cut) the tag closes. +func TestManifestOfAnOlderDocumentVersionIsRefused(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + r := newRunner(t, setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ\fВТОРАЯ"})) + defer r.Close() + if _, err := r.BuildAndPersistManifest(); err != nil { + t.Fatal(err) + } + raw, err := os.ReadFile(r.manifestPath()) + if err != nil { + t.Fatal(err) + } + var m BookManifest + if err := json.Unmarshal(raw, &m); err != nil { + t.Fatal(err) + } + m.Version = "tm-manifest-v1" + for ci := range m.Chapters { + for ui := range m.Chapters[ci].Units { + // …in the old id form, so the fixture is a real v1 document rather than a relabelled v2 one. + m.Chapters[ci].Units[ui].ID = m.Chapters[ci].ID + ":" + strconv.Itoa(m.Chapters[ci].Units[ui].FirstChunkIdx) + } + } + body, err := json.Marshal(m) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(r.manifestPath(), body, 0o644); err != nil { + t.Fatal(err) + } + if got := r.loadManifest(); got != nil { + t.Fatal("a v1 sidecar carries unit ids that outlive their cut; the version bump must refuse it") + } +} + +// TestBankTermIDIsUnambiguous closes acceptance finding V2-1 by execution rather than by a comment. The +// first version joined the uniqueness key with U+001F and asserted the separator "cannot occur in any of +// them" — an assumption about free text the engine never validates. These two rows are DIFFERENT terms +// whose separator-joined encodings are byte-identical, so under the old scheme they shared one id and a +// consumer addressing either would hit the other. +func TestBankTermIDIsUnambiguous(t *testing.T) { + a := bankTermID("a\x1fb", "", 0, 0) // separator-join: "a␟b␟␟0␟0" + b := bankTermID("a", "b", 0, 0) // separator-join: "a␟b␟0␟0" → collides once the empty sense folds + if a == b { + t.Fatalf("two different bank keys must not share an id (%q); the encoding has to be injective, not merely conventional", a) + } + // The same class one field over, and a control character the engine has no rule against. + if bankTermID("x", "1\x1f2", 0, 0) == bankTermID("x\x1f1", "2", 0, 0) { + t.Fatal("a control character in `sense` must not be able to impersonate a different (src, sense) split") + } + // Determinism across runs is not asserted here — it is what + // TestBankExportCarriesEveryStatusWithStableIDs measures end-to-end (two runs, byte-identical export). +} + +// TestCutTagFollowsTheLangpack closes acceptance finding V2-3 on the axis that CAN be varied at runtime. +// The langpack decides the heading rule, and the heading rule can decide whether a header-only chapter +// consumes a number at all — so it re-cuts the book while chunker_version and the budget stand still. A +// manifest rebuilt after such a change must not stamp the previous cut's unit ids onto different text. +// +// The other two inputs folded by the same fix — the embedded language data and the normalization version — +// are compile-time constants of the binary and cannot be varied from a test, exactly like chunkerVersion, +// which the segmentation test cannot vary either. They are asserted by construction (they are in the +// payload) and not by execution; this is stated rather than implied. +func TestCutTagFollowsTheLangpack(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + r := newRunner(t, setupMiningStopProject(t, srv.URL, miningStopOpts{})) + defer r.Close() + if r.pack == nil { + t.Fatal("fixture drifted: this project must load a real langpack") + } + withPack := r.cutTag() + r.pack = nil // the same book, cut by a binary whose pair pack is gone + if withoutPack := r.cutTag(); withoutPack == withPack { + t.Fatalf("the cut tag must move with the langpack — the heading rule can change which chapters exist; got %q both ways", withPack) + } +} + +// TestManifestFastPathIsActuallyTaken closes an audit finding: NOTHING pinned the fast path itself. +// TestManifestServesTheReadModelsIdentically compares "with manifest" against "without manifest", so +// deleting the fast path made both sides the slow path and the test passed trivially — the whole benefit +// of row 100 requirement (б) rested on a wall-clock measurement in a report. +// +// The observable difference is not timing but TEXT: the manifest stores structure, so the fast path +// returns chunks with no source text, and the fallback returns the full split. That is the property the +// rest of the design is built on (which read paths may use it, which must not), so it is the one to pin. +func TestManifestFastPathIsActuallyTaken(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + r := newRunner(t, setupProjectOpts(t, srv.URL, projectOpts{source: "ПЕРВАЯ ГЛАВА\fВТОРАЯ ГЛАВА"})) + defer r.Close() + if _, err := r.BuildAndPersistManifest(); err != nil { + t.Fatal(err) + } + + fast, withText, err := r.readModelChunks() + if err != nil { + t.Fatal(err) + } + if len(fast) == 0 { + t.Fatal("no chunks") + } + for i, c := range fast { + if c.Text != "" { + t.Fatalf("chunk %d carries source text — the manifest fast path was NOT taken, the source was re-ingested and re-cut", i) + } + } + // …and the provider beside it still hands back text, so a caller that needs it is not stranded. + full, err := withText() + if err != nil { + t.Fatal(err) + } + if len(full) != len(fast) || full[0].Text == "" { + t.Fatalf("the text provider must return the SAME cut, with text: %d vs %d chunks, first text %q", len(full), len(fast), full[0].Text) + } + + // Without a manifest the slow path runs, and its chunks DO carry text — the two branches are visibly + // different, which is what makes the assertion above meaningful rather than vacuous. + if err := os.Remove(r.manifestPath()); err != nil { + t.Fatal(err) + } + slow, _, err := r.readModelChunks() + if err != nil { + t.Fatal(err) + } + if len(slow) != len(fast) || slow[0].Text == "" { + t.Fatalf("the fallback must re-cut the book WITH text: %d vs %d chunks, first text %q", len(slow), len(fast), slow[0].Text) + } +} + +// TestWriteFileAtomicLeavesNoLitter pins what a single-process test CAN observe about the artifact write: +// the replacement lands whole and no temp file survives beside it. The property the helper exists for — +// a concurrent reader never seeing a half-written document — is not observable from one process and is +// declared in the report rather than claimed as tested. +func TestWriteFileAtomicLeavesNoLitter(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "artifact.json") + if err := writeFileAtomic(path, []byte("first")); err != nil { + t.Fatal(err) + } + if err := writeFileAtomic(path, []byte("second")); err != nil { + t.Fatal(err) + } + got, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if string(got) != "second" { + t.Fatalf("content = %q, want the replacement", got) + } + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatal(err) + } + if len(entries) != 1 || entries[0].Name() != "artifact.json" { + names := make([]string, 0, len(entries)) + for _, e := range entries { + names = append(names, e.Name()) + } + t.Fatalf("a completed write must leave exactly the artifact, got %v", names) + } + // 0644, not CreateTemp's 0600: another process has to be able to read it. + fi, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + if fi.Mode().Perm() != 0o644 { + t.Fatalf("mode = %v, want 0644 (a reader in another process must not depend on running as this user)", fi.Mode().Perm()) + } +} diff --git a/backend/internal/pipeline/export.go b/backend/internal/pipeline/export.go index 91716bbd..c68ae472 100644 --- a/backend/internal/pipeline/export.go +++ b/backend/internal/pipeline/export.go @@ -91,7 +91,11 @@ func (r *Runner) Export(pairs bool) (*BookExport, error) { // output units — each a (chapter, firstChunkIdx) key + the unit's joined source (the --pairs column // aligned byte-for-byte to how the edit wave concatenated the members). A draft-only pipeline has one output per // draft chunk (the draft ships). Iterating units keeps a PENDING unit explicit and a GHOST row visible. - manifest, err := r.bookChunks() + // The persisted manifest serves the DEFAULT export (row 100): the projection joins stored rows against + // positions and prepends the chapter title, both of which the manifest carries. `--pairs` is the one + // mode that emits the SOURCE column, so it demands the full re-chunk — a text-free manifest would ship + // an empty src column to the FP-measure, which is worse than being slow. + manifest, err := r.exportChunks(pairs) if err != nil { return nil, err } @@ -140,6 +144,17 @@ func (r *Runner) Export(pairs bool) (*BookExport, error) { return exp, nil } +// exportChunks resolves the manifest the export projects over: the persisted one when the export does +// not need source text, the full ingest+split when it does (`--pairs`). One helper so the "who needs +// text" rule lives in one place instead of being re-decided at each call. +func (r *Runner) exportChunks(pairs bool) ([]chunk.Chunk, error) { + if pairs { + return r.bookChunks() + } + chunks, _, err := r.readModelChunks() + return chunks, err +} + // exportGateMisses returns the CONFIRMED post-check miss count per chunk, or an empty map when the // glossary post-check gate is off. // diff --git a/backend/internal/pipeline/manifest.go b/backend/internal/pipeline/manifest.go new file mode 100644 index 00000000..d704c357 --- /dev/null +++ b/backend/internal/pipeline/manifest.go @@ -0,0 +1,531 @@ +package pipeline + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "io/fs" + "os" + "strconv" + + "textmachine/backend/internal/chunk" + "textmachine/backend/internal/lang" + "textmachine/backend/internal/text" +) + +// manifest.go: the PERSISTED chapter/chunk manifest (backlog row 100). +// +// Two problems, one artifact. (1) Every $0 read re-ingested and re-cut the whole source to get its +// honest N/M denominator — 1.4–1.5 s of CPU per call on a 23 MB book, paid again on every poll of every +// book. (2) The chapter tree a reader needs has no carrier at all: the engine addresses chapters by a +// DENSE ordinal (chapters that yield no text do not consume a number, chunker.go), so editing the source +// renumbers every later chapter and any bookmark built on the number silently moves to another chapter. +// +// The artifact is a FILE beside the project DB, not a table, for the same reason the bank export is one +// (bankexport.go): the reader is another process which must not open the engine's private store +// (D39.85). It is also a projection and never a source — nothing reads it back into a decision the +// engine makes; a stale one is DETECTED and ignored, never trusted. +// +// STALENESS is decided by a KEY, not by the mtime and not by a flag. Everything whose change moves the +// chunk boundaries or the chapter set is hashed into it, so a manifest written under a different chunker, +// budget, langpack, embedded data set, encoding or source file cannot be mistaken for a current one. A +// write path rebuilds it unconditionally (so `translate --resnapshot`, which is the loud "the chunker +// moved" command, rebuilds it like every other run); a read path that finds a key mismatch falls back to +// the full re-chunk it did before this file existed. There is no state in which a wrong manifest is used. + +// manifestVersion versions the DOCUMENT SHAPE. It also rides the key, so a shape change invalidates every +// stored manifest rather than being half-read by a binary expecting the other layout. +// +// v2 (acceptance V2-4): unit ids gained the cut tag — «::» instead of +// «:». A v1 sidecar is structurally valid and would keep validating, handing a reader ids +// in the OLD form whose whole defect was that they outlive the cut. The bump is what refuses it. +const manifestVersion = "tm-manifest-v2" + +// BookManifest is the persisted chapter/chunk structure of a book. +type BookManifest struct { + Version string `json:"manifest_version"` + BookID string `json:"book_id"` + // Key is the validity key (manifestKey). A reader that only wants "is this still the same book, cut + // the same way" compares this one string. + Key string `json:"key"` + // The key's INPUTS, spelled out beside it. The key alone answers "is it current"; these answer "what + // changed" when it is not, which is the question an operator staring at an invalidated tree asks. + ChunkerVersion string `json:"chunker_version"` + SourceSHA256 string `json:"source_sha256"` + SourceBytes int64 `json:"source_bytes"` + SourceLang string `json:"source_lang"` + TargetLang string `json:"target_lang"` + Encoding string `json:"encoding"` + + Chapters []ManifestChapter `json:"chapters"` + ChaptersTotal int `json:"chapters_total"` + UnitsTotal int `json:"units_total"` + ChunksTotal int `json:"chunks_total"` +} + +// ManifestChapter is one chapter: its stable identity, its DISPLAY ordinal, and its units. +type ManifestChapter struct { + // ID is the engine's STABLE key for this chapter: 16 hex chars of SHA-256 over the chapter's INGESTED + // text (see manifestChapterID). It is stable across a chunker/budget/heading-rule change and across + // edits to OTHER chapters — which is the whole point, since Number is not: numbering is dense, so + // inserting a chapter shifts every later ordinal while every later id stays put. + ID string `json:"id"` + // Number is the dense 1-based ordinal the engine addresses chunk_status rows by. It is a POSITION, + // not a key — see ID. + Number int `json:"number"` + // Heading is the title the engine RENDERS for this chapter from the pair's heading rule («Глава N»), + // or "" when the book has no rule or the chapter has no structural header. ⚠ It is the engine's own + // deterministic render — the source marker is stripped from the text the model sees and this string is + // glued back onto the chapter's first output unit at export (chunk.ApplyHeading). It is NOT a label + // carried by the book's own data, and a consumer must not present it as one. Which label a reader + // should see is an open contract question (companion §4, K-2/K-3), not settled here. + Heading string `json:"heading"` + UnitsTotal int `json:"units_total"` + ChunksTotal int `json:"chunks_total"` + Units []ManifestUnit `json:"units"` +} + +// ManifestUnit is one OUTPUT unit — the granularity the engine ships and every read model counts in. +type ManifestUnit struct { + // ID is stable only as long as the CUT is, and it SAYS SO in its own bytes: it is the chapter id, the + // cut tag, and the unit's leader chunk index (manifestUnitID). A chunker/budget/pipeline-shape change + // re-cuts the chapter into different units, mints a new cut tag, and therefore mints a new id for + // EVERY unit of the book — including a chapter's first unit, whose leader index is 0 under any cut and + // which would otherwise carry an id that outlives the text it named. + // + // So: chapter identity survives a re-chunk, unit identity does not, and a consumer can tell the two + // apart by whether the id it holds still appears in the current manifest. + ID string `json:"id"` + // FirstChunkIdx is the LEADER chunk index — the (chapter, chunk_idx) the unit's shipping row is + // addressed at in chunk_status, and therefore the join key for every progress read. + FirstChunkIdx int `json:"first_chunk_idx"` + // ChunkCount is how many draft chunks the unit groups; members are the CONSECUTIVE indices + // [FirstChunkIdx, FirstChunkIdx+ChunkCount) of this chapter (chunk.assignEditUnits groups consecutive + // chunks, which is what makes storing a count instead of a list lossless). + ChunkCount int `json:"chunk_count"` + // EditUnitID is the engine's book-global edit-unit id for this unit. + EditUnitID int `json:"edit_unit_id"` +} + +// manifestPath is where the manifest lands: beside the project DB, like every other book-state sidecar. +func (r *Runner) manifestPath() string { return r.Book.ProjectDB + ".manifest.json" } + +// manifestChapterID is a chapter's stable identity: the first 64 bits of SHA-256 over its INGESTED text. +// +// WHY CONTENT AND NOT POSITION. The alternative — hashing the book id and the ordinal — is the ordinal +// with extra steps: inserting a chapter would re-point every later id at a different chapter's content, +// which is precisely the "open tabs and bookmarks quietly move" damage the contract companion names (§5). +// Content-derived, an edit to chapter 5 mints a new id for chapter 5 alone and every other chapter keeps +// the id it had, whatever its number became. +// +// DUPLICATES: two chapters with byte-identical text would hash the same, so the second and later +// occurrences carry an occurrence suffix. Deterministic for a given reading order, and honest about its +// own edge — deleting the FIRST of a duplicate pair promotes the second, changing its id. +func manifestChapterID(ingested string, occurrence int) string { + sum := sha256.Sum256([]byte(ingested)) + id := hex.EncodeToString(sum[:])[:16] + if occurrence > 1 { + id += "-" + strconv.Itoa(occurrence) + } + return id +} + +// manifestUnitID is a unit's identity: its chapter's id, the CUT it belongs to, and the leader chunk +// index. +// +// The cut tag is load-bearing and was missing in the first version, which made the declared boundary +// FALSE rather than merely coarse. Without it a unit id is «:», and a chapter's +// FIRST unit is always index 0 — so «:0» survived every re-chunk unchanged while pointing at a +// different span of text. A consumer that persists unit ids reads "the id is still there" as "my anchor +// is still valid", and lands on the wrong text with no signal. With the tag, ANY re-cut mints new unit +// ids across the whole book, which is exactly what ManifestUnit.ID's boundary says it does. +func manifestUnitID(chapterID, cutTag string, firstChunkIdx int) string { + return chapterID + ":" + cutTag + ":" + strconv.Itoa(firstChunkIdx) +} + +// cutTag identifies the CUT — everything that decides where chunk boundaries fall and how chunks group +// into output units. It is a SHORT hash because it rides inside every unit id; the full identity of the +// manifest is the validity key, which folds these same inputs plus the source bytes and the book identity. +// +// The last three are the acceptance V2-3 catch and are the same class as the defect the tag exists to +// close: the langpack decides the heading rule (a header-only chapter can stop consuming a number), the +// embedded data feeds the CJK numerals and sentence terminators that place boundaries, and the +// normalization decides what the chunker is even splitting. Any of them moving re-cuts the book while +// chunker_version and the budget stand still — and a manifest rebuilt afterwards would stamp the same +// unit ids onto different text. +func (r *Runner) cutTag() string { + payload := struct { + Chunker string `json:"chunker_version"` + Segmentation segmentationSnap `json:"segmentation"` + ShippingWave string `json:"shipping_wave"` + Langpack string `json:"langpack_version,omitempty"` + Embedded string `json:"embedded_version"` + Norm string `json:"norm_version"` + }{chunkerVersion, r.segmentationSnapshot(), r.shippingWaveTag(), + r.packVersion(), lang.EmbeddedVersion(), text.NormVersion()} + data, err := json.Marshal(payload) + if err != nil { + panic(fmt.Sprintf("pipeline: manifest cut tag marshal: %v", err)) + } + sum := sha256.Sum256(data) + return hex.EncodeToString(sum[:])[:8] +} + +// shippingWaveTag names which wave owns the shipping stage. The UNIT decomposition depends on it +// (outputUnits groups whole chunks into edit units for an edit pipeline and emits one unit per chunk for +// a draft-only one), so it belongs to the cut for both the validity key and the unit id. +func (r *Runner) shippingWaveTag() string { + if r.finalStageWave() == waveDraft { + return "draft" + } + return "edit" +} + +// sourceFingerprint is the source file's identity at one moment: the SHA-256 of its BYTES as they sit on +// disk — before decoding, before normalization — plus its size. Taken before AND after the split (see +// writeManifest), because a manifest whose STRUCTURE came from one version of the file and whose KEY came +// from another would validate forever while describing a book that no longer exists. +type sourceFingerprint struct { + SHA string + Bytes int64 +} + +// sourceSHA256 fingerprints the source file. Streamed, so a 23 MB book costs a buffer and not a copy of +// itself (measured: ~20 ms on the acceptance book, against ~1.4 s for the ingest+split it replaces). +func sourceSHA256(path string) (sourceFingerprint, error) { + f, err := os.Open(path) + if err != nil { + return sourceFingerprint{}, err + } + defer f.Close() + h := sha256.New() + n, err := io.Copy(h, f) + if err != nil { + return sourceFingerprint{}, err + } + return sourceFingerprint{SHA: hex.EncodeToString(h.Sum(nil)), Bytes: n}, nil +} + +// sourceFingerprintBeforeIngest takes the fingerprint a manifest write must be built against, and returns +// nil (with a warning, never an error) when it cannot: the manifest is an accelerator, and a run must not +// die because its accelerator could not be stamped. +func (r *Runner) sourceFingerprintBeforeIngest(ctx context.Context) *sourceFingerprint { + fp, err := sourceSHA256(r.Book.SourceFile) + if err != nil { + r.Log.WarnContext(ctx, "manifest: could not fingerprint the source before reading it; the chapter/chunk manifest will not be written this run", + "source", r.Book.SourceFile, "err", err) + return nil + } + return &fp +} + +// manifestKey hashes everything whose change makes a stored manifest wrong. +// +// The list is the set of inputs the chunk manifest is a function of: the source bytes, how they are +// decoded (encoding) and which language's rules segment them, the chunker's own version, the +// segmentation budget (a re-chunk), the language pack (its heading rule decides both the rendered +// heading and, for a header-only chapter, whether the chapter survives at all) and the embedded language +// data (CJK numerals/terminators feed chapter and sentence boundaries; source abbreviations feed the +// sentence splitter). text.NormVersion rides too — normalization decides what "the chapter's text" even +// is, and the chapter ids are hashes of exactly that. +// +// INGEST CODE ITSELF is covered by convention rather than by a version of its own: the project already +// treats an ingest-rule change as a chunkerVersion bump ("This is an ingest rule → covered by +// chunkerVersion", text/source.go). That convention is inherited here, not invented — but it IS a +// convention, so an ingest edit that forgets the bump leaves a stale manifest looking current, exactly as +// it leaves stale checkpoints looking current today. +func (r *Runner) manifestKey(src sourceFingerprint) string { + // The UNIT decomposition of the document is a function of the pipeline SHAPE, not only of the cut: + // outputUnits groups whole chunks into edit units for an edit pipeline and emits one unit per chunk + // for a draft-only one (waverun.go). Dropping the editor stage moves no other input here, so without + // this the stored units[] would keep validating while describing groups the run no longer ships. + payload := struct { + Version string `json:"manifest_version"` + BookID string `json:"book_id"` + SourceSHA string `json:"source_sha256"` + SourceBytes int64 `json:"source_bytes"` + Encoding string `json:"encoding"` + SourceLang string `json:"source_lang"` + TargetLang string `json:"target_lang"` + Chunker string `json:"chunker_version"` + Segmentation segmentationSnap `json:"segmentation"` + Langpack string `json:"langpack_version,omitempty"` + Embedded string `json:"embedded_version"` + Norm string `json:"norm_version"` + ShippingWave string `json:"shipping_wave"` + }{ + Version: manifestVersion, BookID: r.Book.BookID, + SourceSHA: src.SHA, SourceBytes: src.Bytes, + Encoding: r.Book.Encoding, SourceLang: r.Book.SourceLang, TargetLang: r.Book.TargetLang, + Chunker: chunkerVersion, Segmentation: r.segmentationSnapshot(), + Langpack: r.packVersion(), Embedded: lang.EmbeddedVersion(), Norm: text.NormVersion(), + ShippingWave: r.shippingWaveTag(), + } + data, err := json.Marshal(payload) + if err != nil { + // A struct of scalars and a fixed nested struct cannot fail to marshal; a key that could silently + // become "" would make every manifest look valid, so refuse to produce one instead. + panic(fmt.Sprintf("pipeline: manifest key marshal: %v", err)) + } + sum := sha256.Sum256(data) + return hex.EncodeToString(sum[:]) +} + +// buildManifest projects a fresh split into the document. chapterTexts is index-aligned to chapter +// numbers (chunk.SplitChunksWithChapters), chunks is the split itself. Pure and deterministic. +func (r *Runner) buildManifest(chapterTexts []string, chunks []chunk.Chunk, src sourceFingerprint) *BookManifest { + m := &BookManifest{ + Version: manifestVersion, BookID: r.Book.BookID, + Key: r.manifestKey(src), + ChunkerVersion: chunkerVersion, + SourceSHA256: src.SHA, SourceBytes: src.Bytes, + SourceLang: r.Book.SourceLang, TargetLang: r.Book.TargetLang, Encoding: r.Book.Encoding, + ChunksTotal: len(chunks), + } + // Chapter ids first, so a unit can name its chapter's id. The occurrence counter makes duplicate + // chapter texts distinguishable (manifestChapterID). + occurrence := map[string]int{} + ids := make([]string, len(chapterTexts)) + for i, txt := range chapterTexts { + occurrence[txt]++ + ids[i] = manifestChapterID(txt, occurrence[txt]) + } + // A chapter's title lives on its FIRST chunk only (chunker.go), so it is collected off the chunk list + // rather than off the unit list — a unit is a group of chunks and knowing which of them is the + // chapter's opener is the chunker's business, not this projection's. + headings := map[int]string{} + for _, c := range chunks { + if c.Heading != "" { + headings[c.Chapter] = c.Heading + } + } + cut := r.cutTag() // rides every unit id: a re-cut must mint new ones (manifestUnitID) + at := map[int]int{} // chapter number → index into m.Chapters (units arrive in book order) + for _, u := range r.outputUnits(chunks) { + i, ok := at[u.Chapter] + if !ok { + // Chapter numbers are dense and index-aligned to chapterTexts by construction (the chunker + // appends a text exactly when it consumes a number). If that ever stops holding, fall back to a + // number-derived id and say so: an EMPTY id would make every unit id of every such chapter + // collide on ":", which is worse than an id that is merely position-based. + id := "" + if k := u.Chapter - 1; k >= 0 && k < len(ids) { + id = ids[k] + } else { + id = "n" + strconv.Itoa(u.Chapter) + r.Log.Warn("manifest: no ingested text for a chapter the split emitted — its id falls back to the (unstable) chapter number", + "book", r.Book.BookID, "chapter", u.Chapter, "chapter_texts", len(ids)) + } + m.Chapters = append(m.Chapters, ManifestChapter{ID: id, Number: u.Chapter, Heading: headings[u.Chapter]}) + i = len(m.Chapters) - 1 + at[u.Chapter] = i + } + c := &m.Chapters[i] + c.UnitsTotal++ + c.ChunksTotal += len(u.Members) + c.Units = append(c.Units, ManifestUnit{ + ID: manifestUnitID(c.ID, cut, u.FirstChunkIdx), + FirstChunkIdx: u.FirstChunkIdx, ChunkCount: len(u.Members), EditUnitID: u.EditUnitID, + }) + m.UnitsTotal++ + } + m.ChaptersTotal = len(m.Chapters) + return m +} + +// persistManifest writes the manifest for a fresh split. LOUD BUT NOT FATAL: the manifest is a read +// accelerator and a reader's tree, not an input to anything the run decides — failing a paid run over it +// would trade money for an artifact the next run rewrites. A reader that finds no manifest (or a stale +// one) falls back to the full re-chunk, so the failure degrades to the pre-existing behaviour. +func (r *Runner) persistManifest(ctx context.Context, before *sourceFingerprint, chapterTexts []string, chunks []chunk.Chunk) { + if before == nil { + return // sourceFingerprintBeforeIngest already said why + } + if _, err := r.writeManifest(*before, chapterTexts, chunks); err != nil { + r.Log.WarnContext(ctx, "manifest: could not persist the chapter/chunk manifest; read paths fall back to re-chunking the source and the chapter tree may be stale", + "path", r.manifestPath(), "err", err) + } +} + +// writeManifest is persistManifest's fallible half — separate so a caller that MUST know (the manifest +// command) can report the failure instead of logging it. It returns the document it wrote, so a caller +// that also wants to render it does not build and hash the book a second time. +// +// `before` is the fingerprint taken BEFORE the source was read, and the re-fingerprint here is what makes +// the validity key mean what it claims. Hashing only afterwards would stamp the structure derived from +// one version of the file with the identity of another: the ingest+split of a 23 MB book takes ~1.4 s, +// and a source that is rewritten inside that window (a re-upload, an operator's edit) would produce a +// document that describes the OLD cut, matches its own key forever, and can never be detected as stale. +// A source that moved under the read is therefore not written at all — the next write path produces a +// consistent one, and until then readers take the full re-chunk. +func (r *Runner) writeManifest(before sourceFingerprint, chapterTexts []string, chunks []chunk.Chunk) (*BookManifest, error) { + after, err := sourceSHA256(r.Book.SourceFile) + if err != nil { + return nil, fmt.Errorf("pipeline: re-hash source %s for the manifest: %w", r.Book.SourceFile, err) + } + if after != before { + return nil, fmt.Errorf("pipeline: the source %s changed while it was being read (%.12s/%d bytes → %.12s/%d bytes): the manifest is NOT written, because a structure cut from the old bytes stamped with the new hash would validate forever", + r.Book.SourceFile, before.SHA, before.Bytes, after.SHA, after.Bytes) + } + m := r.buildManifest(chapterTexts, chunks, before) + body, err := json.MarshalIndent(m, "", " ") + if err != nil { + return nil, fmt.Errorf("pipeline: marshal the manifest: %w", err) + } + if err := writeFileAtomic(r.manifestPath(), append(body, '\n')); err != nil { + return nil, err + } + return m, nil +} + +// BuildAndPersistManifest re-derives the split and persists the manifest ($0: ingest + chunker, no LLM, +// no provider key, no wave). It is the producer behind `tmctl manifest` — the command a caller +// runs to get a chapter tree for a book that has never been translated, which is a state the library has +// to be able to show and which no other engine surface produces. +func (r *Runner) BuildAndPersistManifest() (*BookManifest, error) { + // Fingerprint BEFORE the read, so writeManifest can prove the structure and the identity came from the + // same bytes. Here a fingerprint failure IS an error: producing the artifact is the command's job. + before, err := sourceSHA256(r.Book.SourceFile) + if err != nil { + return nil, fmt.Errorf("pipeline: hash source %s for the manifest: %w", r.Book.SourceFile, err) + } + doc, err := chunk.IngestEncoded(r.Book.SourceFile, r.Book.Encoding, r.Book.SourceLang) + if err != nil { + return nil, err + } + chunks, chapterTexts := chunk.SplitChunksWithChapters(doc.Chapters, r.segBudget(), r.headingRule(), r.sentenceAbbrevs()) + if len(chunks) == 0 { + return nil, fmt.Errorf("pipeline: source file %s produced no chunks after normalization", r.Book.SourceFile) + } + return r.writeManifest(before, chapterTexts, chunks) +} + +// ManifestPath exposes where the artifact lives, for a CLI that has to tell the operator. +func (r *Runner) ManifestPath() string { return r.manifestPath() } + +// loadManifest returns the persisted manifest IF it is present and still describes this book, cut this +// way. Every other outcome — absent, unreadable, malformed, wrong shape, stale key — returns nil, and the +// caller re-chunks. A read failure is deliberately not an error to the caller: the manifest is an +// accelerator, and a broken accelerator must degrade to the slow path, never to a wrong answer. +func (r *Runner) loadManifest() *BookManifest { + raw, err := os.ReadFile(r.manifestPath()) + if err != nil { + if !errors.Is(err, fs.ErrNotExist) { + r.Log.Warn("manifest: could not read the manifest; falling back to re-chunking the source", "path", r.manifestPath(), "err", err) + } + return nil + } + var m BookManifest + if err := json.Unmarshal(raw, &m); err != nil { + r.Log.Warn("manifest: the manifest is not readable JSON; falling back to re-chunking the source", "path", r.manifestPath(), "err", err) + return nil + } + if m.Version != manifestVersion { + r.Log.Warn("manifest: the manifest was written in another document version; falling back to re-chunking the source", + "path", r.manifestPath(), "stored", m.Version, "want", manifestVersion) + return nil + } + // Internal sanity BEFORE the key: the key is a stored string, so a hand-edited sidecar (they sit in the + // directory operators open) keeps matching while its counters say anything at all. A negative capacity + // panics `make`, and an absurd one allocates the machine out of memory — inside a $0 read command. + if !m.selfConsistent() { + r.Log.Warn("manifest: the manifest's own counters do not describe its own contents; falling back to re-chunking the source", + "path", r.manifestPath(), "chapters_total", m.ChaptersTotal, "units_total", m.UnitsTotal, "chunks_total", m.ChunksTotal) + return nil + } + src, err := sourceSHA256(r.Book.SourceFile) + if err != nil { + r.Log.Warn("manifest: could not hash the source to validate the manifest; falling back to re-chunking the source", "err", err) + return nil + } + if want := r.manifestKey(src); m.Key != want { + // This is the normal, expected path after a source edit or a chunker/pack/budget change — say what + // happened at INFO-level volume rather than as a fault, but say it: a chapter tree whose ids just + // changed is a fact a reader has to be told about (a re-anchor, not a refresh). + r.Log.Info("manifest: the stored manifest is stale (source or cut changed); re-chunking the source and rebuilding it on the next write path", + "path", r.manifestPath(), "stored_key", m.Key, "current_key", want) + return nil + } + return &m +} + +// selfConsistent reports whether the document's own counters describe its own contents. It is a guard +// against a HAND-EDITED file, not against a corrupt one (a corrupt one fails the JSON parse or the key): +// the counters are read straight into slice capacities and loop bounds below, so a negative or absurd +// number would panic or exhaust memory inside a read-only command. +func (m *BookManifest) selfConsistent() bool { + if m.ChaptersTotal != len(m.Chapters) { + return false + } + units, chunks := 0, 0 + for _, c := range m.Chapters { + if c.Number < 1 || c.UnitsTotal != len(c.Units) { + return false + } + for _, u := range c.Units { + if u.ChunkCount < 1 || u.ChunkCount > manifestMaxChunks || u.FirstChunkIdx < 0 { + return false + } + chunks += u.ChunkCount + } + units += len(c.Units) + } + // The UPPER bound matters as much as the lower one: a file edited so that ChunksTotal and the per-unit + // counts agree at an absurd value passes every equality above and then allocates that many chunks. + if chunks > manifestMaxChunks { + return false + } + return m.UnitsTotal == units && m.ChunksTotal == chunks +} + +// manifestMaxChunks bounds what a manifest may claim to describe. It is a sanity ceiling, not a product +// limit: the 23 MB acceptance book cuts into ~5 000 chunks, so this is three orders of magnitude above +// any real book and far below the value at which the reconstruction would exhaust memory. +const manifestMaxChunks = 10_000_000 + +// chunks reconstructs the manifest's chunk list. Safe to call only on a document that passed +// selfConsistent (loadManifest is the only producer). +// +// ⚠ THE CHUNKS CARRY NO SOURCE TEXT. The manifest stores STRUCTURE — a 23 MB book's text is the source +// file, and duplicating it into a sidecar would trade the CPU this file saves for the same amount of +// disk and write time. Every field the read models actually join on is here (chapter, chunk index, edit +// unit, the chapter title). Three read-side consumers genuinely need the text and therefore take the full +// re-chunk instead: `export --pairs` (it emits the source column), the re-bill projection's content +// re-render (it re-renders injected bytes), and `QualityReport` (`quality.go` — its residual scan is a +// comparison against the unit's source, so `tmctl report` still re-cuts the book). +func (m *BookManifest) chunks() []chunk.Chunk { + out := make([]chunk.Chunk, 0, m.ChunksTotal) + for _, c := range m.Chapters { + for _, u := range c.Units { + for i := 0; i < u.ChunkCount; i++ { + ch := chunk.Chunk{ + Chapter: c.Number, ChunkIdx: u.FirstChunkIdx + i, EditUnitID: u.EditUnitID, + } + if ch.ChunkIdx == 0 { + ch.Heading = c.Heading // only a chapter's first chunk carries the title (chunker.go) + } + out = append(out, ch) + } + } + } + return out +} + +// readModelChunks is what the $0 read models join against: the persisted manifest when it is current, the +// full ingest+split otherwise. +// +// The second result is a PROVIDER of the same chunks WITH their source text, not a flag — a flag that +// every caller discarded would document a guard the code does not keep. On the manifest path it is the +// full re-chunk (paid only if some branch actually needs text); on the fallback path it hands back the +// slice that was just materialized, so a read that already re-cut the book never re-cuts it twice. +func (r *Runner) readModelChunks() (chunks []chunk.Chunk, withText func() ([]chunk.Chunk, error), err error) { + if m := r.loadManifest(); m != nil { + return m.chunks(), r.bookChunks, nil + } + full, err := r.bookChunks() + return full, func() ([]chunk.Chunk, error) { return full, err }, err +} diff --git a/backend/internal/pipeline/mining.go b/backend/internal/pipeline/mining.go index fcb394cb..285e7fc3 100644 --- a/backend/internal/pipeline/mining.go +++ b/backend/internal/pipeline/mining.go @@ -2,6 +2,7 @@ package pipeline import ( "context" + "encoding/json" "errors" "fmt" "io/fs" @@ -48,6 +49,11 @@ func (r *Runner) runBankMiningStop(ctx context.Context, chunks []chunk.Chunk, dr return false, fmt.Errorf("pipeline: --verify-bank was passed but this book cannot mine a bank: it needs BOTH a langpack (book.yaml `langpack_root`, currently %s) and a mining contrast artifact (pipeline.yaml `mining.contrast_path`, currently %q); without them there is no candidate list to verify", packStateLabel(r.pack), r.Pipeline.Mining.ContrastPath) } + // A book that cannot mine has no signing table — and if one is lying beside its DB from when it + // could (a langpack or contrast path removed from the config), it would outlive every run and keep + // offering terms to a screen forever. Clear it, but only when it exists: a book that never mines + // must not acquire a sidecar it has no use for. + r.clearBankStopTableJSON(ctx, "mining is not configured for this book") return false, nil // mining not configured → auto-continue to the edit wave } f, err := os.Open(r.Pipeline.Mining.ContrastPath) @@ -146,6 +152,17 @@ func (r *Runner) runBankMiningStop(ctx context.Context, chunks []chunk.Chunk, dr } else { r.Log.InfoContext(ctx, msg, args...) } + // The MACHINE table is rewritten EMPTY here, and that is the difference between it and the human one + // (row 101). An empty delta means "nothing awaits signature"; leaving the previous run's rows on disk + // would let a signing screen re-offer terms the owner has just signed, with no field in the document + // to tell that state from a live one. The text sidecar keeps its prior behaviour deliberately — it is + // the document a human re-reads, and a human knows which run they are looking at. + // + // Written unconditionally here (unlike the not-configured path above): mining DID run, so "nothing + // awaits signature" is this run's own finding and the artifact should state it. + if werr := r.writeBankStopTableJSON(nil); werr != nil { + r.Log.WarnContext(ctx, "bank-mining: could not clear the machine stop table sidecar; it still holds the PREVIOUS run's rows", "err", werr) + } return false, nil } consolidated, classified, tres, err := r.runTerminologist(ctx, draftSnapshot, cands) @@ -178,6 +195,13 @@ func (r *Runner) runBankMiningStop(ctx context.Context, chunks []chunk.Chunk, dr if werr := os.WriteFile(r.bankStopTablePath(), []byte(renderBankStopTable(rows)), 0o644); werr != nil { r.Log.WarnContext(ctx, "bank-mining: could not write the stop table sidecar (the signature map is unaffected)", "err", werr) } + // …and the SAME table as a machine surface (backlog row 101). Same rows, same order, same fields — the + // text sidecar stays the human document and this one is what a signing screen is built from, so the two + // can never describe the bank differently. Same failure discipline as the text sidecar: a sidecar that + // cannot be written must not take a paid run down with it. + if werr := r.writeBankStopTableJSON(rows); werr != nil { + r.Log.WarnContext(ctx, "bank-mining: could not write the machine stop table sidecar (the signature map and the text table are unaffected)", "err", werr) + } // THE FLAG (D39.42 п.5, owner's words: «можно запустить перевод так, чтоб сессия не останавливалась и // не запрашивала верификацию банка, а просто как намайнит и закончит — шла в редактуру»). Default is @@ -197,9 +221,13 @@ func (r *Runner) runBankMiningStop(ctx context.Context, chunks []chunk.Chunk, dr if err := r.seedGlossary(ctx); err != nil { return false, fmt.Errorf("pipeline: re-seed the bank with the auto rows: %w", err) } + // The bank just changed (row 125): the auto rows are IN it now, so the read-out has to say so + // before the edit wave starts translating against them. + r.exportBank(ctx, "bank-mining/auto-continue") r.Log.InfoContext(ctx, "bank-mining: auto-continuing with an UNSIGNED bank (pass --verify-bank to stop and review it)", append([]any{"book", r.Book.BookID, "terms", len(mined), "terms_with_proposed_dst", withDst, - "auto_bank", r.autoBankPath(), "signature_map", r.signatureMapPath(), "table", r.bankStopTablePath()}, + "auto_bank", r.autoBankPath(), "signature_map", r.signatureMapPath(), + "table", r.bankStopTablePath(), "table_json", r.bankStopTableJSONPath()}, emissionArgs(emission, cands)...)...) return false, nil } @@ -209,6 +237,9 @@ func (r *Runner) runBankMiningStop(ctx context.Context, chunks []chunk.Chunk, dr return false, nil } r.lastBankStopRows = rows + // The signature stop is the boundary the signing screen reads at (row 125): refresh the bank read-out + // so the state behind the decisions is the state as of this stop, not as of the last run. + r.exportBank(ctx, "bank-mining/signature-stop") r.Log.WarnContext(ctx, "bank-mining: new terms await owner signature; run STOPPED before the edit wave (review the signature map, then for EACH term either promote it into the mined-delta file OR decline it in the mined-rejects file, then resume — the stop clears once every proposed term is promoted or rejected)", append([]any{"book", r.Book.BookID, "terms", len(mined), "terms_with_proposed_dst", withDst, "signature_map", r.signatureMapPath()}, emissionArgs(emission, cands)...)...) @@ -250,6 +281,118 @@ func (r *Runner) bankStopTablePath() string { return r.Book.ProjectDB + ".bank-stop.txt" } +// bankStopTableJSONPath is the MACHINE sidecar of the same table (backlog row 101). It sits beside the +// text one for the same reason that one sits beside the signature map: a loadable seed YAML cannot carry +// evidence, and a table meant to be parsed cannot be a column layout. Deliberately a SEPARATE file rather +// than a replacement — the text table is the document a human reads at 3 a.m. with `less`. +func (r *Runner) bankStopTableJSONPath() string { + return r.Book.ProjectDB + ".bank-stop.json" +} + +// bankStopTableVersion versions the SHAPE of the machine table, so a consumer that must tolerate a field +// set changing under it has something to branch on. It is not a snapshot input: the table is evidence +// produced from stored answers, never a paid byte (the same standing the parse rule has, see above). +const bankStopTableVersion = "tm-bank-stop-v1" + +// bankStopTableFile is the machine table's document shape. +type bankStopTableFile struct { + Version string `json:"table_version"` + BookID string `json:"book_id"` + Terms int `json:"terms"` + Rows []bankStopRowJSON `json:"rows"` +} + +// bankStopRowJSON is ONE row of the machine table — the same fields the text table prints, with the two +// places where a human-facing rendering would lose information made explicit: +// +// - Conf is a POINTER: the role's own confidence is absent for a term it never mentioned, and "absent" +// and "0% sure" are opposites (mining.go confOrAbsent). The text table omits the word `confidence` +// for the first case; JSON says null, and a consumer that reads 0 for both would sort the sheet +// exactly wrong. +// - every list is emitted as a list, never a joined string, because the text table's separators (", " +// / " | " / "; ") occur inside real renderings and a consumer cannot split them back safely. +// +// It is a DTO rather than JSON tags on BankStopRow on purpose: BankStopRow is the CLI's in-memory row and +// its field names are free to follow the code, while these names are a published surface. +type bankStopRowJSON struct { + Src string `json:"src"` + Dst string `json:"dst"` // "" when nothing was consolidated for this term + Origin string `json:"origin"` + Type string `json:"type"` + Freq int `json:"freq"` + Spread int `json:"spread"` + Conventions int `json:"conventions"` + Conf *int `json:"conf"` + Invented bool `json:"invented"` + Signals []string `json:"signals"` + Contradicts []string `json:"contradicts"` + Variants []BankStopVariant `json:"variants"` + Evidence []string `json:"evidence"` + Contexts []string `json:"contexts"` +} + +// bankStopTableJSON projects the rows into the machine document. Deterministic: the row order is the +// caller's (source-key order, the reference order the text sidecar also uses), and every empty list is +// rendered as `[]` rather than `null` so a consumer has one shape to handle instead of two. +func bankStopTableJSON(bookID string, rows []BankStopRow) bankStopTableFile { + out := bankStopTableFile{Version: bankStopTableVersion, BookID: bookID, Terms: len(rows), Rows: make([]bankStopRowJSON, 0, len(rows))} + for _, row := range rows { + jr := bankStopRowJSON{ + Src: row.Src, Dst: row.Dst, Origin: row.Origin, Type: row.Type, + Freq: row.Freq, Spread: row.Spread, Conventions: row.Conventions, Invented: row.Invented, + Signals: emptyIfNil(row.Signals), Contradicts: emptyIfNil(row.Contradicts), + Variants: emptyVariantsIfNil(row.Variants), Evidence: emptyIfNil(row.Evidence), Contexts: emptyIfNil(row.Contexts), + } + if row.Conf >= 0 { + conf := row.Conf + jr.Conf = &conf + } + out.Rows = append(out.Rows, jr) + } + return out +} + +// emptyVariantsIfNil is emptyIfNil for the variant list — one helper per element type keeps the render +// deterministic without reaching for generics, the same trade render.go's sortedKeys pair already makes. +func emptyVariantsIfNil(v []BankStopVariant) []BankStopVariant { + if v == nil { + return []BankStopVariant{} + } + return v +} + +// emptyIfNil renders a nil slice as an empty one (see bankStopRowJSON). +func emptyIfNil(s []string) []string { + if s == nil { + return []string{} + } + return s +} + +// clearBankStopTableJSON empties an EXISTING machine stop table, and does nothing when there is none — +// the difference between "this book has nothing awaiting signature" (a statement worth making) and "this +// book has no signing surface at all" (nothing to say, and a file nobody asked for). +func (r *Runner) clearBankStopTableJSON(ctx context.Context, why string) { + if _, err := os.Stat(r.bankStopTableJSONPath()); err != nil { + return // absent (or unreadable — then rewriting it is not this path's business either) + } + if werr := r.writeBankStopTableJSON(nil); werr != nil { + r.Log.WarnContext(ctx, "bank-mining: could not clear the stale machine stop table sidecar; it still offers terms for signature", + "path", r.bankStopTableJSONPath(), "reason", why, "err", werr) + return + } + r.Log.InfoContext(ctx, "bank-mining: cleared the machine stop table sidecar", "path", r.bankStopTableJSONPath(), "reason", why) +} + +// writeBankStopTableJSON serializes and atomically replaces the machine table sidecar. +func (r *Runner) writeBankStopTableJSON(rows []BankStopRow) error { + body, err := json.MarshalIndent(bankStopTableJSON(r.Book.BookID, rows), "", " ") + if err != nil { + return fmt.Errorf("pipeline: marshal the machine bank-stop table: %w", err) + } + return writeFileAtomic(r.bankStopTableJSONPath(), append(body, '\n')) +} + // BankStopRow is one row of the bank-verification table the stop shows the operator — the table D39.36 // specified and the CLI never had (it printed a term count and a path). Exported because the CLI renders it. type BankStopRow struct { @@ -257,10 +400,10 @@ type BankStopRow struct { Dst string // the consolidated rendering, or "" when nothing was consolidated Origin string // mined | banknote | both — WHICH channel found it Type string - Freq int // occurrences in the source - Spread int // how many DISTINCT renderings the drafts produced (the disagreement signal) - Variants []string // "rendering ×N", best-ranked first - Contexts []string // source KWIC + Freq int // occurrences in the source + Spread int // how many DISTINCT renderings the drafts produced (the disagreement signal) + Variants []BankStopVariant // the renderings the drafts produced, best-ranked first + Contexts []string // source KWIC Evidence []string // The §G3 arbitration record: until this pack, «why does this term have THIS dst» was unanswerable from // the artifacts (research/24 §A7). Every field below is read off work the ranking already did. @@ -281,6 +424,42 @@ type BankStopRow struct { Contradicts []string } +// BankStopVariant is ONE rendering the drafts produced, kept in its PARTS rather than as the sentence a +// table prints. The renderers format it (Label); the machine sidecar ships the parts. Before this the row +// carried the pre-rendered « × (proposed for )» string and nothing else, so the machine +// surface could only be un-parsed by splitting on «×» and on a parenthesis — both of which occur inside +// real renderings. +// Its JSON tags ARE the published shape of the machine sidecar: unlike the row, whose field names in the +// file are deliberately independent of the code's, a variant's three parts are called the same thing on +// both sides, and a second type to say so would only be a conversion waiting to drift. +type BankStopVariant struct { + Dst string `json:"dst"` + Chunks int `json:"chunks"` // how many draft chunks proposed it + // Via names the ALIAS this rendering was proposed for, "" for a direct proposal. Load-bearing rather + // than decoration: an alias-routed rendering must never silently become the cluster owner's dst + // (proposalsFromCandidates), so a surface that offers one has to be able to say whose it was. + Via string `json:"via,omitempty"` +} + +// Label is the one rendering of a variant every human-facing table uses, so the capped stdout view and +// the text sidecar cannot describe a variant differently. +func (v BankStopVariant) Label() string { + s := fmt.Sprintf("%s ×%d", v.Dst, v.Chunks) + if v.Via != "" { + s += " (proposed for " + v.Via + ")" + } + return s +} + +// VariantLabels renders a row's variants for a text table. +func (r BankStopRow) VariantLabels() []string { + out := make([]string, 0, len(r.Variants)) + for _, v := range r.Variants { + out = append(out, v.Label()) + } + return out +} + // bankStopRows projects the merged candidates into the operator table, best-ranked variants first. // Deterministic: cands is key-ordered and nothing here iterates a map for output. func bankStopRows(cands []terminology.Candidate, consolidated map[string]string, tres terminologyResult) []BankStopRow { @@ -294,11 +473,7 @@ func bankStopRows(cands []terminology.Candidate, consolidated map[string]string, Conf: confOrAbsent(tres.Conf, c.Key), Contradicts: tres.Contradictions[c.Src], } for i, v := range c.Variants { - label := fmt.Sprintf("%s ×%d", v.Dst, v.Chunks) - if v.Via != "" { - label += " (proposed for " + v.Via + ")" - } - row.Variants = append(row.Variants, label) + row.Variants = append(row.Variants, BankStopVariant{Dst: v.Dst, Chunks: v.Chunks, Via: v.Via}) if i == 0 { row.Signals = v.Signals } @@ -355,7 +530,7 @@ func renderBankStopTable(rows []BankStopRow) string { fmt.Fprintf(&b, " CONTRADICTS this run's own: %s\n", strings.Join(r.Contradicts, "; ")) } if len(r.Variants) > 0 { - fmt.Fprintf(&b, " drafts: %s\n", strings.Join(r.Variants, " | ")) + fmt.Fprintf(&b, " drafts: %s\n", strings.Join(r.VariantLabels(), " | ")) } if len(r.Evidence) > 0 { fmt.Fprintf(&b, " evidence: %s\n", strings.Join(r.Evidence, ", ")) diff --git a/backend/internal/pipeline/miningstop_join_test.go b/backend/internal/pipeline/miningstop_join_test.go index 0f439992..41b58197 100644 --- a/backend/internal/pipeline/miningstop_join_test.go +++ b/backend/internal/pipeline/miningstop_join_test.go @@ -11,6 +11,7 @@ import ( "strings" "testing" + "textmachine/backend/internal/chunk" "textmachine/backend/internal/chunk/chunktest" "textmachine/backend/internal/config" "textmachine/backend/internal/lang" @@ -1594,7 +1595,7 @@ func TestPointwiseReEditOnlyPaysForTheUnitsTheTermTouches(t *testing.T) { if err != nil { t.Fatal(err) } - proj, err := rp.projectRebill(statuses, chunks) + proj, err := rp.projectRebill(statuses, chunks, func() ([]chunk.Chunk, error) { return chunks, nil }) if err != nil { t.Fatal(err) } diff --git a/backend/internal/pipeline/rebill.go b/backend/internal/pipeline/rebill.go index 93afb464..cc3245f3 100644 --- a/backend/internal/pipeline/rebill.go +++ b/backend/internal/pipeline/rebill.go @@ -92,12 +92,19 @@ type RebillProjection struct { // // The wave snapshots are rendered LAZILY, so a draft-only pipeline never renders an edit-wave snapshot // it has no stages for. -func (r *Runner) projectRebill(statuses []store.ChunkStatus, chunks []chunk.Chunk) (RebillProjection, error) { +// +// `manifest` supplies only the POSITIONS still in the book (the membership test above), which is why it +// may be the text-free projection of the persisted manifest (manifest.go). `withText` supplies the same +// chunks WITH their source, and is called only on the content-aware branch below — the one that has to +// re-render injected bytes and therefore genuinely needs the text. Splitting the two is what lets a +// status read skip a 1.4 s re-chunk it almost never needs, without ever computing a content hash over an +// empty string. +func (r *Runner) projectRebill(statuses []store.ChunkStatus, manifest []chunk.Chunk, withText func() ([]chunk.Chunk, error)) (RebillProjection, error) { var p RebillProjection decider := newRepinDecider(r) var contentHashes map[chunkKey]map[string]string - live := make(map[chunkKey]bool, len(chunks)) - for _, ch := range chunks { + live := make(map[chunkKey]bool, len(manifest)) + for _, ch := range manifest { live[chunkKey{ch.Chapter, ch.ChunkIdx}] = true } draftNames := stageNameSet(r.waveStagesIndexed(waveDraft)) @@ -144,7 +151,11 @@ func (r *Runner) projectRebill(statuses []store.ChunkStatus, chunks []chunk.Chun } if bankOnly { if contentHashes == nil { // rendered once, lazily: a book with no bank move never pays for it - contentHashes = r.renderedContentHashes(chunks, precomputeSticky(chunks, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget)) + full, ferr := withText() + if ferr != nil { + return p, fmt.Errorf("pipeline: re-chunk the source for the re-bill content check: %w", ferr) + } + contentHashes = r.renderedContentHashes(full, precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget)) } if h, ok := contentHashes[chunkKey{cs.Chapter, cs.ChunkIdx}][cs.Stage]; ok && h == cs.ContentHash { p.Repinned++ @@ -217,7 +228,9 @@ func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk) e if err != nil { return fmt.Errorf("pipeline: read chunk_status for the re-bill projection: %w", err) } - proj, err := r.projectRebill(statuses, chunks) + // A write path already holds the real split, so the lazy provider just hands it back — no second + // ingest, and no branch where the consent gate could see text-free chunks. + proj, err := r.projectRebill(statuses, chunks, func() ([]chunk.Chunk, error) { return chunks, nil }) if err != nil { return err } diff --git a/backend/internal/pipeline/rebill_test.go b/backend/internal/pipeline/rebill_test.go index 283f94e7..6bc31525 100644 --- a/backend/internal/pipeline/rebill_test.go +++ b/backend/internal/pipeline/rebill_test.go @@ -290,7 +290,7 @@ func TestRebillProjectionIsPerWave(t *testing.T) { if err != nil { t.Fatal(err) } - if proj, err := r1.projectRebill(statusesBefore, chunksOf(t, r1)); err != nil || proj.Rows != 0 { + if proj, err := r1.projectRebill(statusesBefore, chunksOf(t, r1), func() ([]chunk.Chunk, error) { return chunksOf(t, r1), nil }); err != nil || proj.Rows != 0 { t.Fatalf("a freshly-run book projects no re-bill; each row must be judged against ITS OWN wave, got %+v (err=%v)", proj, err) } r1.Close() @@ -312,7 +312,7 @@ func TestRebillProjectionIsPerWave(t *testing.T) { if err != nil { t.Fatal(err) } - proj, err := r2.projectRebill(statuses, chunksOf(t, r2)) + proj, err := r2.projectRebill(statuses, chunksOf(t, r2), func() ([]chunk.Chunk, error) { return chunksOf(t, r2), nil }) if err != nil { t.Fatal(err) } @@ -358,7 +358,7 @@ func TestRebillProjectionExcludesSkippedAndUnchanged(t *testing.T) { t.Fatal("setup: the refusal fixture must leave a skipped edit row") } // No drift yet: nothing is superseded, so nothing is projected. - if proj, err := r1.projectRebill(statuses, chunksOf(t, r1)); err != nil || proj.Rows != 0 || proj.USD != 0 { + if proj, err := r1.projectRebill(statuses, chunksOf(t, r1), func() ([]chunk.Chunk, error) { return chunksOf(t, r1), nil }); err != nil || proj.Rows != 0 || proj.USD != 0 { t.Fatalf("an undrifted book must project no re-bill, got %+v (err=%v)", proj, err) } r1.Close() @@ -370,7 +370,7 @@ func TestRebillProjectionExcludesSkippedAndUnchanged(t *testing.T) { if err := r2.seedGlossary(ctx); err != nil { t.Fatal(err) } - proj, err := r2.projectRebill(statuses, chunksOf(t, r2)) + proj, err := r2.projectRebill(statuses, chunksOf(t, r2), func() ([]chunk.Chunk, error) { return chunksOf(t, r2), nil }) if err != nil { t.Fatal(err) } @@ -408,7 +408,7 @@ func TestRebillProjectionIgnoresRetiredStages(t *testing.T) { if err != nil { t.Fatal(err) } - proj, err := r.projectRebill(statuses, chunksOf(t, r)) + proj, err := r.projectRebill(statuses, chunksOf(t, r), func() ([]chunk.Chunk, error) { return chunksOf(t, r), nil }) if err != nil { t.Fatal(err) } @@ -450,7 +450,7 @@ func TestRebillProjectionIgnoresVanishedChunks(t *testing.T) { if err := r2.seedGlossary(ctx); err != nil { t.Fatal(err) } - proj, err := r2.projectRebill(statuses, chunksOf(t, r2)) + proj, err := r2.projectRebill(statuses, chunksOf(t, r2), func() ([]chunk.Chunk, error) { return chunksOf(t, r2), nil }) if err != nil { t.Fatal(err) } diff --git a/backend/internal/pipeline/runner.go b/backend/internal/pipeline/runner.go index 6f35544d..746294d8 100644 --- a/backend/internal/pipeline/runner.go +++ b/backend/internal/pipeline/runner.go @@ -61,6 +61,21 @@ type Runner struct { // carries it forward marked-unverified, and finishes — «просто как намайнит и закончит». It is an // operator MODE, not a wire or snapshot axis: it changes when a human is asked, never what is sent. VerifyBank bool + // CeilingUSD is the BOOK ceiling for THIS RUN ONLY (tmctl --ceiling-usd, backlog row 145 / D39.110), + // 0 = unset. It OVERRIDES book.yaml's ceilings.book_usd for the duration of the process and is never + // written back: the caller's number stays a property of the run, so a platform can cap a run without + // editing the engine's book config (the zone mixing D39.81/D39.85 forbid). + // + // ⚠ It is a BOOK ceiling, not a run BUDGET: the ledger compares it against the book's cumulative + // committed+reserved (store/ledger.go), so a caller granting "another $2" passes already-spent + $2, + // and a value below what the book already spent denies the first reservation of the run. That is the + // same quantity book.yaml's ceilings.book_usd is, which is what makes overriding it coherent — the + // alternative (a second, per-process budget axis) would be a new money mechanism, not this row. Like VerifyBank it is an + // operator axis, not a wire one — the ceilings are in NEITHER BriefHash (config/book.go: "Wiring + // fields (paths, ceilings, db) deliberately excluded") NOR the snapshot payload (snapshot.go), so + // setting it moves no request_hash and re-bills nothing. The DAY ceiling is deliberately not + // overridable: it is an account-wide guard, not a property of one run. + CeilingUSD float64 clients map[string]llm.LLMClient templates map[string]*PromptTemplate @@ -330,6 +345,21 @@ func (r *Runner) packVersion() string { func (r *Runner) Close() error { return r.Store.Close() } +// bookCeilingUSD is the ONE definition of "the book ceiling in force for this process" (row 145): the +// run-scoped override when the invocation named one, the book's own `ceilings.book_usd` otherwise. Every +// surface that admits a spend or reports the cap reads it here, so the ledger can never admit against one +// number while the operator is shown another. A run-scoped override is only ever an amount > 0 (the CLI +// refuses zero/negative/non-finite), so the flag can only ever ADD a bound, never remove one — which is +// what leaves Р7 intact without amending config.LoadBook's validator. Р7 is a DIFFERENT rule: it demands +// at least one of book_usd/day_usd, so a book may legitimately declare a DAILY ceiling only, this returns +// 0 (no book ceiling, exactly as before the flag existed), and passing the flag introduces one for the run. +func (r *Runner) bookCeilingUSD() float64 { + if r.CeilingUSD > 0 { + return r.CeilingUSD + } + return r.Book.Ceilings.BookUSD +} + func (r *Runner) loadTemplates() error { for _, st := range r.Pipeline.Stages { // The path was resolved at load — by the pair/role convention, or by the stage's deliberate diff --git a/backend/internal/pipeline/runner_test.go b/backend/internal/pipeline/runner_test.go index 59670e58..efc713f2 100644 --- a/backend/internal/pipeline/runner_test.go +++ b/backend/internal/pipeline/runner_test.go @@ -120,6 +120,9 @@ type projectOpts struct { // glossaryTokenBudget overrides context.glossary_token_budget. 0 keeps the fixture's historical 800, so // every pre-existing project is byte-identical; a tiny value is how the EVICTION path is reachable at all. glossaryTokenBudget int + // draftOnly drops the editor stage, so the DRAFT is the shipping output (the pipeline shape the wave + // executor calls draft-only). false keeps the historical two-stage fixture byte-identical. + draftOnly bool } func setupProjectOpts(t *testing.T, providerURL string, o projectOpts) string { @@ -164,6 +167,10 @@ models: if o.banknote { gatesBlock += "\ngates:\n banknote:\n enabled: true\n" } + editStage := " - { name: edit, role: editor, model: fake-model, prompt_override: prompts/editor.md, prompt_version: v-test, temperature: 0.4, reasoning: \"off\" }\n" + if o.draftOnly { + editStage = "" + } writeFile(t, filepath.Join(dir, "pipeline.yaml"), fmt.Sprintf(` core: C1 version: 1 @@ -173,8 +180,7 @@ context: { glossary_injection: selective, glossary_token_budget: %d } waves: { workers: %d } stages: - { name: draft, role: translator, model: fake-model, prompt_override: prompts/translator.md, prompt_version: v-test, temperature: 0.3, reasoning: "off" } - - { name: edit, role: editor, model: fake-model, prompt_override: prompts/editor.md, prompt_version: v-test, temperature: 0.4, reasoning: "off" } -%s`, o.minMaxTokens, o.regenerate, o.glossaryTokenBudget, o.waveWorkers, gatesBlock)) +%s%s`, o.minMaxTokens, o.regenerate, o.glossaryTokenBudget, o.waveWorkers, editStage, gatesBlock)) sourceName := "source.txt" if len(o.epub) > 0 { diff --git a/backend/internal/pipeline/stagerun.go b/backend/internal/pipeline/stagerun.go index c0b08fa6..624f9a5f 100644 --- a/backend/internal/pipeline/stagerun.go +++ b/backend/internal/pipeline/stagerun.go @@ -477,7 +477,7 @@ func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID estimate := ledger.EstimateUSD(price, promptEst, maxTokens, reasoningBudget) resv, verdict, err := r.Store.Reserve(r.Book.BookID, estimate, store.Ceilings{ - BookUSD: r.Book.Ceilings.BookUSD, DayUSD: r.Book.Ceilings.DayUSD, + BookUSD: r.bookCeilingUSD(), DayUSD: r.Book.Ceilings.DayUSD, }) if err != nil { r.setJobStatus(ctx, job.ID, "failed") @@ -502,7 +502,13 @@ func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID committed, reserved, estimate, ch.Chapter, ch.ChunkIdx, st.Name) } if verdict == store.ReserveDeniedBook { - return att, fmt.Errorf("pipeline: book USD ceiling reached ($%g)%s — raise ceilings.book_usd or stop: %w", r.Book.Ceilings.BookUSD, money, errReserveCeiling) + // Name WHICH ceiling stopped the run: a caller who passed --ceiling-usd and is told to "raise + // ceilings.book_usd" would edit a file that is not in force (row 145). + raise := "raise ceilings.book_usd" + if r.CeilingUSD > 0 { + raise = fmt.Sprintf("this run's --ceiling-usd=%g overrides the book's ceilings.book_usd=%g — re-run with a higher --ceiling-usd", r.CeilingUSD, r.Book.Ceilings.BookUSD) + } + return att, fmt.Errorf("pipeline: book USD ceiling reached ($%g)%s — %s or stop: %w", r.bookCeilingUSD(), money, raise, errReserveCeiling) } return att, fmt.Errorf("pipeline: daily USD ceiling reached ($%g)%s: %w", r.Book.Ceilings.DayUSD, money, errReserveCeiling) } diff --git a/backend/internal/pipeline/status.go b/backend/internal/pipeline/status.go index 899302ba..cde09131 100644 --- a/backend/internal/pipeline/status.go +++ b/backend/internal/pipeline/status.go @@ -32,19 +32,62 @@ const ( ChunkPending ChunkState = "pending" // no stage attempted yet ) +// WaveCounter is one wave's done/total pair, in OUTPUT UNITS — the same denominator the whole +// projection counts in (TotalUnits), so the two phases and the unit totals can never be read against +// different scales. Total is 0 for a wave the pipeline does not have (a draft-only pipeline has no edit +// wave), which is how a consumer tells "no such phase" from "none of it is done yet". +type WaveCounter struct { + Done int `json:"done"` + Total int `json:"total"` +} + +// PhaseProgress is progress PER WAVE (backlog row 99). The end-to-end unit counter (`Done`) requires +// BOTH every member draft AND the unit's edit to have resolved ok, and the edit wave does not start +// before the bank stop — so it reads 0 for the whole draft wave, which is exactly the indicator the row +// was raised about. These two counters split that single number by wave. +// +// DONE HERE MEANS RESOLVED, NOT OK. A unit counts for a wave once every chunk_status row that wave owes +// it exists — ok, flagged or skipped alike. Two reasons, both load-bearing: +// +// - a flagged unit is FINISHED as far as work goes (nothing re-attempts it without an explicit +// `tmctl redrive`), so excluding it would leave a progress bar permanently short of its own +// denominator on any book with a single flagged chunk; +// - the ok/flagged split is already carried, unconflated, by Done/Flagged/GlossaryMissFlagged below. +// +// Consequence to read deliberately, ON AN EDIT PIPELINE: Edit.Done ≥ Done, and the gap is every unit the +// edit wave RESOLVED but the unit-level verdict did not call done — a unit the post-check GATE flagged +// (its edit row is ok; the gate flips the UNIT after the stage loop), a c-lite unit whose edit shipped +// while a member draft flagged, and a unit whose edit was recorded skipped because every member flagged. +// All three are finished work, which is what these counters count; "attention needed" is the other +// counters' job. On a DRAFT-ONLY pipeline the relation does not hold in that direction at all — Edit is +// the zero counter and Draft is the one to compare against Done. +// +// A stage RENAME is the one state where the two readings disagree in the other direction: stored rows +// carry the old stage name, which is in neither wave's name set, so a wave counter stays short of its +// denominator while `Done` (which counts ok rows by disposition, not by name) can still read full. The +// same rename already makes `export` report every unit pending; `ConfigDrift` is what flags the condition. +type PhaseProgress struct { + Draft WaveCounter `json:"draft"` + Edit WaveCounter `json:"edit"` +} + // ChapterPassport is the per-chapter quality passport (D12): unit counts, the worst flag // reason, a pass|attention|fail verdict (exp07 chapter rule) and the chapter's spend. type ChapterPassport struct { - Chapter int `json:"chapter"` - UnitsTotal int `json:"units_total"` - UnitsDone int `json:"units_done"` - UnitsFlagged int `json:"units_flagged"` - UnitsInProgress int `json:"units_in_progress"` - UnitsPending int `json:"units_pending"` - StagesSkipped int `json:"stages_skipped"` // per-stage skip count (downstream of a flag) - Escalations int `json:"escalations"` - PostcheckMisses int `json:"postcheck_misses"` - StyleFlags int `json:"style_flags"` // cheap style/number gate hits (observability, not a disposition) + Chapter int `json:"chapter"` + UnitsTotal int `json:"units_total"` + // Progress is this chapter's per-wave split of UnitsTotal (row 99). It exists for the same reason the + // book-level one does, one level down: a chapter tree whose per-chapter counter is the end-to-end one + // shows every chapter at zero for the whole draft wave (contract companion §4, K-10). + Progress PhaseProgress `json:"progress"` + UnitsDone int `json:"units_done"` + UnitsFlagged int `json:"units_flagged"` + UnitsInProgress int `json:"units_in_progress"` + UnitsPending int `json:"units_pending"` + StagesSkipped int `json:"stages_skipped"` // per-stage skip count (downstream of a flag) + Escalations int `json:"escalations"` + PostcheckMisses int `json:"postcheck_misses"` + StyleFlags int `json:"style_flags"` // cheap style/number gate hits (observability, not a disposition) // RepairApplied counts this chapter's units whose SHIPPED text carries an applied repair (pack-16). It is // DERIVED from the durable final_hash namespace, never stored, so it survives a resume like the rest of // the projection. omitempty: a chapter with no repairs renders exactly as before the field existed. @@ -83,6 +126,11 @@ type StatusReport struct { Flagged int `json:"flagged"` Pending int `json:"pending"` PercentDone float64 `json:"percent_done"` // 100·done/total, unit-count only (no synthetic time-bar) + // Progress is the PER-WAVE split of TotalUnits (backlog row 99) — see PhaseProgress for what "done" + // means there and why it is not the same predicate as Done above. No percentage is derived from it + // here on purpose: how two phases combine into one number a reader sees is a product decision, not a + // projection fact. + Progress PhaseProgress `json:"progress"` // GlossaryMissFlagged counts chunks the post-check GATE promoted to flagged/glossary_miss (a // CONFIRMED miss on an otherwise-ok chunk). A SUBSET of Flagged and NOT re-drivable: the miss @@ -299,7 +347,11 @@ func resolveChunkState(rows []store.ChunkStatus, stagesTotal int) chunkStateReso // no LLM call and replays no checkpoint — the D12 query-handler analogue. Safe to run whenever // the project file is not held by a running tmctl (the same exclusive-lock rule as `report`). func (r *Runner) Status(ctx context.Context) (*StatusReport, error) { - chunks, err := r.bookChunks() + // The manifest fast path (row 100): status joins stored rows against POSITIONS, never against source + // text, so the persisted structure is a complete substitute for the re-chunk here. The one branch that + // does need text — the re-bill content check — takes the full split through the provider below, and + // only when a bank-only snapshot move actually put it on the path. + chunks, withText, err := r.readModelChunks() if err != nil { return nil, err } @@ -339,6 +391,14 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) { // would report a gate-flagged unit as done/pass while `tmctl translate` exits 2 (finding #2). gateOn := r.Pipeline.Gates.Glossary.PostcheckGate rep := &StatusReport{BookID: r.Book.BookID, TotalUnits: len(units)} + // Per-wave denominators (row 99): a wave the pipeline does not run has a total of 0 rather than a + // total it can never reach — a draft-only pipeline must not show "edit 0/4276" forever. + if len(draftStages) > 0 { + rep.Progress.Draft.Total = len(units) + } + if len(editStages) > 0 { + rep.Progress.Edit.Total = len(units) + } // The unsigned-bank exposure, same definition as the quality report's (a row with a rendering that is // not approved). Read-only, like everything else here; a read failure says so rather than reporting 0. if rows, gerr := r.Store.GlossaryForBook(r.Book.BookID); gerr != nil { @@ -361,10 +421,40 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) { chapterOrder = append(chapterOrder, u.Chapter) } p.UnitsTotal++ + if len(draftStages) > 0 { + p.Progress.Draft.Total++ + } + if len(editStages) > 0 { + p.Progress.Edit.Total++ + } leader := chunkKey{u.Chapter, u.FirstChunkIdx} expected := len(u.Members)*len(draftStages) + len(editStages) - res := resolveChunkState(unitRows(u, byChunk), expected) + // Collected once: three passes over the unit's rows follow, and on a 5000-unit book re-walking the + // member map for each of them is pure churn. + rows := unitRows(u, byChunk) + res := resolveChunkState(rows, expected) state, reason := res.State, res.Reason + // Per-wave resolution (row 99), off the SAME rows the unit state is folded from. A wave owes a unit + // a fixed number of rows — |members|·|draft stages| in the draft wave, |edit stages| at the leader in + // the edit one — and chunk_status is keyed (book, chapter, chunk, stage), so counting the rows that + // wave wrote IS the resolution test: a row exists only once the wave decided (ok/flagged/skipped). + draftRows, editRows := 0, 0 + for _, cs := range rows { + switch { + case draftStageNames[cs.Stage]: + draftRows++ + case editStageNames[cs.Stage]: + editRows++ + } + } + if len(draftStages) > 0 && draftRows >= len(u.Members)*len(draftStages) { + rep.Progress.Draft.Done++ + p.Progress.Draft.Done++ + } + if len(editStages) > 0 && editRows >= len(editStages) { + rep.Progress.Edit.Done++ + p.Progress.Edit.Done++ + } p.CostUSD += res.CostUSD p.StagesSkipped += res.Skipped miss := missByChunk[leader] // the unit's post-check ran once, at the leader row @@ -375,7 +465,7 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) { rep.StyleFlags += style // Applied repairs (pack-16): the unit's shipping row points at a repair export. Read off the durable // final_hash namespace — no counter column, so a resumed run reports the same number. - for _, row := range unitRows(u, byChunk) { + for _, row := range rows { if strings.HasPrefix(row.FinalHash, repairDerivedNS+":") { p.RepairApplied++ break @@ -491,7 +581,7 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) { // re-bills too, and that is precisely the case the boolean pair leaves unpriced. $0 and read-only: // projectRebill only re-renders the wave snapshots and sums stored costs. if len(statuses) > 0 { - if proj, perr := r.projectRebill(statuses, chunks); perr != nil { + if proj, perr := r.projectRebill(statuses, chunks, withText); perr != nil { // Same discipline as the drift check above: a failed projection is reported, never // silently rendered as "nothing to re-pay". r.Log.WarnContext(ctx, "re-bill projection failed; the re-payment cost of the drift is unknown (reported as none)", "err", perr) @@ -506,7 +596,10 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) { return nil, err } rep.CommittedUSD, rep.ReservedUSD = committed, reserved - rep.BookCeilingUSD = r.Book.Ceilings.BookUSD + // The ceiling IN FORCE (row 145): a read path never carries a run-scoped override, so this is the + // book's own number there — but reading it through the single definition means status can never quote + // a ceiling the ledger is not admitting against. + rep.BookCeilingUSD = r.bookCeilingUSD() if rep.BookCeilingUSD > 0 { rep.CeilingPct = 100 * (committed + reserved) / rep.BookCeilingUSD } @@ -681,6 +774,11 @@ func (r *Runner) Redrive(ctx context.Context, sel RedriveSelector) (*RedriveSumm if err := r.seedGlossary(ctx); err != nil { return summary, nil, fmt.Errorf("pipeline: redrive seed glossary: %w", err) } + // The bank read-out (row 125) follows the re-seed HERE, not only inside TranslateBook. Every guard + // below can abort the redrive after this point — the drift guard, the re-payment consent, the reset + // itself — and each of those exits leaves the bank re-seeded in the store while the exported file + // still described the previous one. A reader would then hold a bank the engine no longer has. + r.exportBank(ctx, "redrive/re-seeded") // Config/seed-drift guard BEFORE the destructive reset (external-review 1c): if the current config // drifted from the snapshot the stored rows carry, TranslateBook would fail loud in runStage — but // the reset (ResetChunkStages) would already have deleted the flagged rows/checkpoints, so a later diff --git a/backend/internal/pipeline/waverun.go b/backend/internal/pipeline/waverun.go index e9e70ce2..7d2dde6e 100644 --- a/backend/internal/pipeline/waverun.go +++ b/backend/internal/pipeline/waverun.go @@ -89,7 +89,10 @@ type WaveSignatureStop struct { // the CLI can print a capped view without re-reading the file (the cap exists because emitRankCap is // 200 and a 200-term dump is not a review surface). TablePath string - Rows []BankStopRow + // TableJSONPath is the MACHINE form of that same table (row 101) — what a signing screen is built + // from. Named beside TablePath rather than instead of it: the two are one table in two renderings. + TableJSONPath string + Rows []BankStopRow } func (e *WaveSignatureStop) Error() string { @@ -133,7 +136,10 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s if stopped, err := r.runBankMiningStop(ctx, chunks, draftSnapshot, editWave); err != nil { return nil, err } else if stopped { - return nil, &WaveSignatureStop{Terms: r.lastMinedCount, SignaturePath: r.signatureMapPath(), TablePath: r.bankStopTablePath(), Rows: r.lastBankStopRows} + return nil, &WaveSignatureStop{ + Terms: r.lastMinedCount, SignaturePath: r.signatureMapPath(), + TablePath: r.bankStopTablePath(), TableJSONPath: r.bankStopTableJSONPath(), Rows: r.lastBankStopRows, + } } res := &BookResult{BookID: r.Book.BookID}