diff --git a/backend/README.md b/backend/README.md index f9f1551b..b36472e6 100644 --- a/backend/README.md +++ b/backend/README.md @@ -8,7 +8,7 @@ | Пакет | Что делает | Ключевые файлы | |---|---|---| -| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) / `manifest` (**$0-производитель персиста манифеста глав/чанков**, строка 100: нужен ДО первого прогона — дерево глав разобранной, но не запущенной книги; `--json` печатает сам документ) / `migrate` (**$0-write-open без прогона**, строка 174: read-only команды требуют ТОЧНОГО совпадения схемы и не мигрируют, поэтому апгрейд бинаря запирал все существующие книги — платформа зовёт `status --json` перед каждым спавном, а write-команда, которая мигрировала бы, не наступала никогда. Грузит ТОЛЬКО `book.yaml` — ни цен, ни ключей на $0-пути (строка 146); restore point берёт ПОД ЛОКОМ и только когда шаг реально применяется (имя `<метка>-pre-migrate.db` — вне секундного неймспейса платного пути, строка 173); несовпадение схемы у read-only путей — типизированный отказ **exit 13** с машинным токеном `schema_mismatch found=N expected=M`). Денежный аргумент прогона: `--ceiling-usd` (строка 145) перекрывает книжный `ceilings.book_usd` ТОЛЬКО на этот прогон, в book.yaml не пишется, ноль/отрицательное = отказ запуска. main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` | +| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) / `manifest` (**$0-производитель персиста манифеста глав/чанков**, строка 100: нужен ДО первого прогона — дерево глав разобранной, но не запущенной книги; `--json` печатает сам документ) / `migrate` (**$0-write-open без прогона**, строка 174: read-only команды требуют ТОЧНОГО совпадения схемы и не мигрируют, поэтому апгрейд бинаря запирал все существующие книги — платформа зовёт `status --json` перед каждым спавном, а write-команда, которая мигрировала бы, не наступала никогда. Грузит ТОЛЬКО `book.yaml` — ни цен, ни ключей на $0-пути (строка 146); restore point берёт ПОД ЛОКОМ и только когда шаг реально применяется (имя `<метка>-pre-migrate.db` — вне секундного неймспейса платного пути, строка 173); несовпадение схемы у read-only путей — типизированный отказ **exit 13** с машинным токеном `schema_mismatch found=N expected=M`). **ДВА потолка прогона, и они ортогональны.** `--ceiling-usd` (строка 145) — ДЕНЬГИ: перекрывает книжный `ceilings.book_usd` ТОЛЬКО на этот прогон, в book.yaml не пишется, ноль/отрицательное = отказ запуска; каппит КУМУЛЯТИВНУЮ трату книги, а не приращение прогона. `--max-units` (D39.165 §1б) — ОБЪЁМ: не больше N ВЫХОДНЫХ ЮНИТОВ (гранулярность `units_total` манифеста, та самая, в которой платформа продаёт главы) будет ОПЛАЧЕНО этим прогоном. Юниты, которые прогон отдаёт за $0 (резюм, ре-пин), едут бесплатно и потолок не тратят; ретраи и эскалации — тоже нет, они внутри юнита. Принимает только `translate`. **Остановка по объёму — ЗАВЕРШЕНИЕ (exit 0), а не пауза:** словарь кодов выхода не расширялся, различение живёт в отчёте прогона и в логе. Отчёт различает ДОСТАВКУ (юнит, которого не было) и ПЕРЕ-ДЕЛКУ (уже доставленный юнит под сдвинутым снапшотом) — покупка, целиком ушедшая в переделку, обязана читаться как переделка. ⚠ На книге, которая МАЙНИТ банк, вторая покупка требует `--resnapshot` (авто-банк растёт между покупками и двигает edit-снапшот) — прогон предупреждает об этом в логе. main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` | | `internal/llm` | OpenAI-совместимый транспорт + retry/backoff, **capability-слой** (budget_field/temperature/reasoning per-модель), провайдеры openai/local (no-proxy)/anthropic (DEPRECATED-референс); `failover.go` удалён паком-17 — маршрутизация лейблами живёт в config/pipeline (канал B, один хоп `chain[0]`, fail-closed) | `httpllm.go`, `capability.go`, `provider_*.go` | | `internal/ledger` | Цены по usage (вкл. reasoning/cache-поля), `PriceForResponse` по фактической модели | `pricing.go` | | `internal/store` | SQLite (modernc, CGO-free), цепочка миграций `schema_version` (⚠ ALTER-шаги v8+ не идемпотентны вопреки шапке — бэклог-строка 49а), reserve/settle+checkpoint, chunk_status, глоссарий (+подписной цикл терминолога), ruby, retrieval_state, request_log; `OpenReadOnly` — без flock/миграций/recovery для status/report/export (схема не совпала — типизированный `*store.SchemaMismatchError{Found,Expected}`, обе стороны); `Migrate` (+шов `beforeApply` под локом: restore point берётся там) — поверхность деплой-шага строки 174, `SchemaHead` — та самая `Expected`; write-open БД новее бинаря теперь ОТКАЗ, а не тихое открытие | `ledger.go`, `migrate.go`, `glossary.go`, `store.go` | @@ -66,7 +66,8 @@ go run ./cmd/tmctl report --config example/book.yaml # $0, quality-repor go run ./cmd/tmctl export --config example/book.yaml # $0, экспорт-JSON для полигона (--plaintext для человека) go run ./cmd/tmctl manifest --config example/book.yaml # $0, пере-строить персист манифеста глав/чанков (--json — сам документ) go run ./cmd/tmctl migrate --config example/book.yaml # $0, довести схему проекта до головы бинаря (деплой-шаг: стоп прогонов → НОВЫЙ бинарь на место → migrate ИМ по каждой книге без открытых попыток → прогоны в работу) -go run ./cmd/tmctl translate --config example/book.yaml --ceiling-usd 0.5 # потолок ТОЛЬКО на этот прогон, book.yaml не пишется +go run ./cmd/tmctl translate --config example/book.yaml --ceiling-usd 0.5 # ДЕНЕЖНЫЙ потолок ТОЛЬКО на этот прогон, book.yaml не пишется +go run ./cmd/tmctl translate --config example/book.yaml --max-units 10 # ОБЪЁМНЫЙ потолок: оплатить не больше 10 выходных юнитов; стоп = завершение (exit 0) # live-conformance (реальные провайдеры, платно, вне CI): set -a; . ./.env; set +a; TM_LIVE=1 go test -tags live -run TestLive -v ./internal/pipeline/ ``` diff --git a/backend/cmd/tmctl/backup_test.go b/backend/cmd/tmctl/backup_test.go index c9a869bc..ff8744b8 100644 --- a/backend/cmd/tmctl/backup_test.go +++ b/backend/cmd/tmctl/backup_test.go @@ -65,7 +65,7 @@ func TestFakeTranslatePathCreatesNoBackup(t *testing.T) { defer srv.Close() bookPath := setupCLIProject(t, srv.URL) - if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false, 0); err != nil { + if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false, 0, 0); err != nil { t.Fatalf("fake translate: %v", err) } if _, err := os.Stat(filepath.Join(filepath.Dir(bookPath), "backups")); err == nil { diff --git a/backend/cmd/tmctl/invocation.go b/backend/cmd/tmctl/invocation.go index 9cd21866..bf55115a 100644 --- a/backend/cmd/tmctl/invocation.go +++ b/backend/cmd/tmctl/invocation.go @@ -45,7 +45,11 @@ type invocation struct { // spend on one run belongs to that run, not to the book's data — writing it into book.yaml would make // a caller's number a permanent record in the engine's config and mix the zones (D39.81/D39.85). ceilingUSD float64 - sel pipeline.RedriveSelector + // maxUnits is the VOLUME ceiling for THIS RUN (D39.165 §1б), 0 when the flag was absent. It is an + // invocation property for exactly the reason ceilingUSD is — how much of the book a caller is buying + // right now belongs to the call, not to the book's data. + maxUnits int + sel pipeline.RedriveSelector } // rebillConsentValue parses `--accept-rebill[=usd]` (D20.2-Q2): the OPTIONAL-VALUE form of Р6, where a @@ -127,6 +131,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { dryRun := fs.Bool("dry-run", false, "redrive: report what would be re-attacked without touching anything; bank-apply: print the projection of the decisions and write NOTHING") verifyBank := fs.Bool("verify-bank", false, "translate/redrive: STOP at the bank-mining boundary and print the bank table when the delta holds terms no earlier stop has shown (D39.144: the flag trips on novelty; a resumed run goes on and undecided terms ride to the editor marked). Default: never stop") ceilingUSD := fs.Float64("ceiling-usd", 0, "translate/redrive: the book USD ceiling in force for THIS RUN ONLY — it OVERRIDES book.yaml `ceilings.book_usd` and is never written back. It caps the book's CUMULATIVE committed+reserved spend, not this run's increment, and must be > 0") + maxUnits := fs.Int("max-units", 0, "translate: the VOLUME ceiling for THIS RUN — at most N OUTPUT UNITS (the manifest's units_total granularity) may be PAID for. Units this run resolves for free (resumed or re-pinned) ride along and do not count, and neither do retries or escalation hops inside a unit. ORTHOGONAL to --ceiling-usd, which caps the book's CUMULATIVE money: this caps THIS run's work. Stopping on it is a COMPLETION (exit 0), not a pause") seed := fs.String("seed", "", "seed-lint: path to the glossary seed YAML to validate ($0, no --config)") keysFile := fs.String("keys-file", "", "translate: path to the DEPLOYMENT's provider-key file (KEY=VALUE lines). Loaded FIRST, so it wins over the .env beside book.yaml; a named file that cannot be read is a refusal, never a silent skip") decisions := fs.String("decisions", "", "bank-apply: path to the JSON decision document to apply to the book's memory bank ($0)") @@ -181,6 +186,31 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { if given["dry-run"] && cmd != "bank-apply" && cmd != "redrive" { return invocation{}, fmt.Errorf("--dry-run is accepted by `bank-apply` and `redrive` only, not by %q: %q has no projection mode, and running it with this flag would spend real money on a call its caller believes is free", cmd, cmd) } + // The volume ceiling follows the same rule as every other flag above: a command that does not act on it + // REFUSES it rather than ignoring it. Silently accepting it here would be the expensive spelling — + // `redrive --max-units 10` re-attacks flagged chunks and then translates the WHOLE book through + // TranslateBook, so a caller who believed the flag bounded the work would be billed for everything + // left. Scoped to `translate`, which is the one verb whose volume a purchase actually buys. + if given["max-units"] { + if cmd != "translate" { + // The reason has to be TRUE of the command it addresses, which is the discipline the + // --keys-file refusal above already carries a test for. `redrive` is the case that makes it + // matter: it DOES drive a book wave — it calls TranslateBook after its reset — so telling it + // "you have no book wave to bound" would teach the reader a false fact about the engine, and + // the next session wiring this flag would believe it. + if cmd == "redrive" { + return invocation{}, fmt.Errorf("--max-units is accepted by `translate` only, not by `redrive`: redrive DOES drive a book wave (it re-attacks the flagged chunks and then translates through TranslateBook), but the destructive reset it performs first is bounded by its own selector and by nothing else — a units cap here would bound the second half of the command while reading as a cap on the whole. Bound the re-attack with --chapter/--chunk/--reason, and the translation that follows with a separate `translate --max-units`") + } + return invocation{}, fmt.Errorf("--max-units is accepted by `translate` only, not by %q: it caps the OUTPUT UNITS a run pays for, and %q neither runs nor pays for them — accepting the flag would tell a caller their work was capped when it was not", cmd, cmd) + } + // Refused rather than defaulted when present-but-not-a-volume, the same rule --ceiling-usd follows + // (D39.110: "off" is not a state). Zero is the ABSENT value, so admitting `--max-units 0` would turn + // an unset variable in a deployment's unit file into an unbounded run that the caller believes is + // bounded — and a negative is not a quantity of book at all. + if *maxUnits <= 0 { + return invocation{}, fmt.Errorf("--max-units must be a positive number of output units (it is the volume ceiling for THIS run); got %d — omit the flag to run the whole book", *maxUnits) + } + } // seed-lint validates a standalone seed YAML — it takes --seed, not --config (no book/store/keys). if cmd == "seed-lint" { if *seed == "" { @@ -218,7 +248,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { return invocation{ cmd: cmd, cfgPath: *cfgPath, resnapshot: *resnapshot, acceptRebill: acceptRebill.c, asJSON: *asJSON, asPlaintext: *asPlaintext, asPairs: *asPairs, verifyBank: *verifyBank, - ceilingUSD: *ceilingUSD, keysFile: *keysFile, decisionsPath: *decisions, dryRun: *dryRun, + ceilingUSD: *ceilingUSD, maxUnits: *maxUnits, keysFile: *keysFile, decisionsPath: *decisions, dryRun: *dryRun, sel: pipeline.RedriveSelector{ Chapter: *chapter, ChunkIdx: *chunk, Reason: *reason, DryRun: *dryRun, }, diff --git a/backend/cmd/tmctl/invocation_test.go b/backend/cmd/tmctl/invocation_test.go index 044c3d00..b8911ba2 100644 --- a/backend/cmd/tmctl/invocation_test.go +++ b/backend/cmd/tmctl/invocation_test.go @@ -299,3 +299,64 @@ func TestDispatchCommandsCoversTheSwitch(t *testing.T) { t.Errorf("the dispatch switch has %d commands, dispatchCommands lists %d — the two must be one list", found, len(dispatchCommands)) } } + +// TestMaxUnitsIsRefusedWhereItWouldDoNothing pins the volume ceiling onto the rule the other four flags +// already follow: a command that does not act on a flag REFUSES it rather than ignoring it. +// +// The silent version of this one costs book rather than dollars, and `redrive` is the expensive case. It +// re-attacks the flagged chunks and then runs TranslateBook over the WHOLE book, so `redrive --max-units +// 10` accepted-and-ignored would bill a caller for every unit the book has left while their own logs said +// they had capped the work at ten. +func TestMaxUnitsIsRefusedWhereItWouldDoNothing(t *testing.T) { + for _, cmd := range dispatchCommands { + if cmd == "translate" { + continue + } + args := []string{cmd, "--config", "book.yaml", "--max-units", "10"} + if cmd == "bank-apply" { + args = append(args, "--decisions", "d.json") + } + _, err := parseInvocation(args, &bytes.Buffer{}) + if err == nil { + t.Errorf("%s accepted --max-units; a flag a command does not act on must be refused, never ignored", cmd) + continue + } + if !strings.Contains(err.Error(), "--max-units is accepted by `translate` only") { + t.Errorf("%s: the refusal must name the flag and the one verb that takes it, got: %v", cmd, err) + } + // And the REASON must be true of the command it addresses — the discipline the --keys-file + // refusal already carries a test for. The first draft of this refusal told every command it + // "does not run a book wave whose volume this could bound", which is false of `redrive`: it + // calls TranslateBook after its reset. A refusal that misdescribes its reader teaches them the + // wrong thing about the engine, and the next session to wire this flag inherits it. + if strings.Contains(err.Error(), "neither runs nor pays for them") && cmd == "redrive" { + t.Errorf("redrive DOES drive a book wave (it calls TranslateBook); its refusal must not claim otherwise, got: %v", err) + } + } + // redrive gets its own reason, and that reason has to say the true thing. + _, err := parseInvocation([]string{"redrive", "--config", "book.yaml", "--max-units", "10"}, &bytes.Buffer{}) + if err == nil || !strings.Contains(err.Error(), "redrive DOES drive a book wave") { + t.Fatalf("redrive's refusal must name the real reason (the reset is bounded by its selector, not by units), got: %v", err) + } +} + +// TestMaxUnitsRefusesANonVolume pins the same "«off» is not a state" rule --ceiling-usd has needed since +// D39.110: a PRESENT flag with a meaningless value is a refusal, because the live trigger is an unset +// variable in a deployment's unit file, and defaulting it to «unbounded» would hand a caller a whole book +// while their own configuration said ten units. +func TestMaxUnitsRefusesANonVolume(t *testing.T) { + for _, v := range []string{"0", "-1"} { + _, err := parseInvocation([]string{"translate", "--config", "book.yaml", "--max-units", v}, &bytes.Buffer{}) + if err == nil || !strings.Contains(err.Error(), "--max-units must be a positive number of output units") { + t.Fatalf("--max-units %s: want a refusal naming the rule, got %v", v, err) + } + } + // Absent is the ONE spelling that means "the whole book", and it must stay silent. + inv, err := parseInvocation([]string{"translate", "--config", "book.yaml"}, &bytes.Buffer{}) + if err != nil { + t.Fatalf("an absent --max-units must parse cleanly: %v", err) + } + if inv.maxUnits != 0 { + t.Fatalf("an absent --max-units must leave the run unbounded, got %d", inv.maxUnits) + } +} diff --git a/backend/cmd/tmctl/main.go b/backend/cmd/tmctl/main.go index ab5979a4..1bf71dc5 100644 --- a/backend/cmd/tmctl/main.go +++ b/backend/cmd/tmctl/main.go @@ -219,7 +219,7 @@ func run() error { if err := preflightBackup(inv.cfgPath, os.Stdout); err != nil { return err } - return translate(ctx, inv.cfgPath, inv.resnapshot, inv.acceptRebill, inv.verifyBank, inv.ceilingUSD) + return translate(ctx, inv.cfgPath, inv.resnapshot, inv.acceptRebill, inv.verifyBank, inv.ceilingUSD, inv.maxUnits) case "report": return report(inv.cfgPath) case "status": @@ -261,7 +261,17 @@ func run() error { // // Over the book's consent threshold the run stops with the sum before reserving anything — a flag that // names no money cannot carry a Р6 consent to a spend. -func translate(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill pipeline.RebillConsent, verifyBank bool, ceilingUSD float64) error { +// +// --max-units is the FOURTH flag and the second CEILING, and it is orthogonal to --ceiling-usd in the way +// that matters most (D39.165 §1б): that one bounds the book's CUMULATIVE money, this one bounds THIS +// run's WORK, in output units — the granularity the manifest publishes and a purchase of chapters +// converts into exactly. Both can be in force; whichever binds first stops the run, and the two stops are +// different answers. Money stops the run in the middle of what it meant to do — exit 4, resumable, "add +// money". Volume means the run did precisely what was bought — exit 0, an ordinary completion, and the +// exit-code dictionary above is not touched. Which one stopped it is said in the run's report and its log +// (pipeline.VolumeStop), never by a new code: the platform reads an unknown exit code as a FAILURE, so a +// user who received exactly what they paid for would have been shown a service error. +func translate(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill pipeline.RebillConsent, verifyBank bool, ceilingUSD float64, maxUnits int) error { r, err := pipeline.NewRunner(cfgPath, obs.NewLogger()) if err != nil { return err @@ -271,6 +281,7 @@ func translate(ctx context.Context, cfgPath string, resnapshot bool, acceptRebil r.AcceptRebill = acceptRebill r.VerifyBank = verifyBank r.CeilingUSD = ceilingUSD + r.MaxUnits = maxUnits res, err := r.TranslateBook(ctx) if err != nil { diff --git a/backend/cmd/tmctl/migrate_cli_test.go b/backend/cmd/tmctl/migrate_cli_test.go index 8de41109..eb68a728 100644 --- a/backend/cmd/tmctl/migrate_cli_test.go +++ b/backend/cmd/tmctl/migrate_cli_test.go @@ -253,7 +253,7 @@ func TestMigrateNeedsNeitherPricesNorModels(t *testing.T) { func TestMigrateKeepsTheMoneyAndRepeatsAsANoOp(t *testing.T) { bookPath := setupCLIProject(t, mockProvider(t).URL) dbPath := projectDBOf(bookPath) - if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false, 0); err != nil { + if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false, 0, 0); err != nil { t.Fatalf("setup run: %v", err) } diff --git a/backend/cmd/tmctl/rebill_cli_test.go b/backend/cmd/tmctl/rebill_cli_test.go index 8ea70612..d8b78ff7 100644 --- a/backend/cmd/tmctl/rebill_cli_test.go +++ b/backend/cmd/tmctl/rebill_cli_test.go @@ -10,6 +10,7 @@ import ( "os" "path/filepath" "strings" + "sync" "testing" "time" @@ -98,7 +99,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) { bookPath := setupCLIProject(t, srv.URL) ctx := context.Background() - if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0); err != nil { + if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0, 0); err != nil { t.Fatalf("first run: %v", err) } if calls != 1 { @@ -114,7 +115,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) { writeCLIFile(t, pipePath, strings.Replace(string(raw), "prompt_version: v-cli", "prompt_version: v-cli-drift", 1)) // --resnapshot WITHOUT consent → refused, nothing called. - err = translate(ctx, bookPath, true, pipeline.RebillConsent{}, false, 0) + err = translate(ctx, bookPath, true, pipeline.RebillConsent{}, false, 0, 0) if err == nil { t.Fatal("`translate --resnapshot` without --accept-rebill must be refused over the threshold") } @@ -127,7 +128,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) { // A ceiling BELOW the projection is still a refusal — proving the CAPPED value is wired, not just // the boolean. - if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true, Capped: true, CapUSD: 0.0001}, false, 0); err == nil { + if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true, Capped: true, CapUSD: 0.0001}, false, 0, 0); err == nil { t.Fatal("a ceiling below the projection must be refused through the CLI entry point too") } if calls != 1 { @@ -135,7 +136,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) { } // Consent given → the run proceeds and really re-pays. - if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true}, false, 0); err != nil { + if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true}, false, 0, 0); err != nil { t.Fatalf("`translate --resnapshot --accept-rebill` must proceed: %v", err) } if calls != 2 { @@ -160,7 +161,7 @@ func TestRedriveWiresAcceptRebill(t *testing.T) { ctx := context.Background() // The refusal text flags the only chunk (exit 2 is a sentinel, not an infra failure). - if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0); exitCode(err) != 2 { + if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0, 0); exitCode(err) != 2 { t.Fatalf("setup: the refusal fixture must complete with flags (exit 2), got %v", err) } callsAfterRun1 := calls @@ -211,7 +212,7 @@ func TestTranslateAndRedriveWireTheRunCeiling(t *testing.T) { // A ceiling far below anything the book can cost. The book's own ceiling is comfortable, so ONLY the // argument can produce this stop — and only if the command actually hands it to the runner. const tiny = 0.000001 - err := translate(ctx, setupCLIProject(t, srv.URL), false, pipeline.RebillConsent{}, false, tiny) + err := translate(ctx, setupCLIProject(t, srv.URL), false, pipeline.RebillConsent{}, false, tiny, 0) if err == nil || !strings.Contains(err.Error(), "ceiling reached") { t.Fatalf("translate must carry --ceiling-usd into the run; got %v", err) } @@ -236,7 +237,7 @@ func TestTranslateAndRedriveWireTheRunCeiling(t *testing.T) { })) defer srv2.Close() bookPath := setupCLIProject(t, srv2.URL) - if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0); exitCode(err) != 2 { + if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0, 0); exitCode(err) != 2 { t.Fatalf("setup: the run must finish WITH a flagged chunk (exit 2), got %v", err) } refuse = false @@ -246,3 +247,76 @@ func TestTranslateAndRedriveWireTheRunCeiling(t *testing.T) { t.Fatalf("redrive must carry --ceiling-usd into the re-attack; got %v", err) } } + +// setupCLIProjectMulti is setupCLIProject with a source long enough — and a draft cut fine enough — to +// produce SEVERAL output units, which is what a volume ceiling needs in order to bite at all. +func setupCLIProjectMulti(t *testing.T, providerURL string) string { + t.Helper() + bookPath := setupCLIProject(t, providerURL) + dir := filepath.Dir(bookPath) + writeCLIFile(t, filepath.Join(dir, "source.txt"), + "静かな図書館の朝。鈴木は本を読んだ。外では雨が降っていた。彼は窓を見た。時間は静かに過ぎた。夜になった。") + writeCLIFile(t, filepath.Join(dir, "pipeline.yaml"), ` +core: C1 +version: 1 +defaults: { max_output_ratio: 2.0, min_max_tokens: 512 } +retries: { regenerate_before_escalate: 0 } +context: { glossary_injection: selective, glossary_token_budget: 800 } +segmentation: + draft_budget_out: 24 + edit_ceiling_out: 48 + fertility: { cjk: 1.1978, other: 0.3852 } +stages: + - { name: draft, role: translator, model: fake-model, prompt_override: prompts/translator.md, prompt_version: v-cli, temperature: 0.3, reasoning: "off" } +`) + return bookPath +} + +// TestTranslateWiresMaxUnits is the same kind of pin as TestTranslateWiresAcceptRebill above, for the +// other run ceiling: it proves the flag changes what the RUN DOES, not merely what the parser returns. +// +// Every other proof of the volume ceiling sets Runner.MaxUnits directly, which leaves exactly one line +// unverified — `r.MaxUnits = maxUnits` in main.go's translate(). That line is the entire seam between the +// flag a platform passes and the engine that honours it; dropping it would leave every volume test in the +// pipeline package green while `tmctl translate --max-units 2` translated the whole book. +func TestTranslateWiresMaxUnits(t *testing.T) { + var mu sync.Mutex + calls := 0 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = io.ReadAll(r.Body) + mu.Lock() + calls++ + mu.Unlock() + tb, _ := json.Marshal("ЧЕРНОВИК ПЕРЕВОДА") + fmt.Fprintf(w, `{"id":"fake","model":"fake-model","choices":[{"message":{"content":%s},"finish_reason":"stop"}], + "usage":{"prompt_tokens":1000,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":200}}}`, tb) + })) + defer srv.Close() + bookPath := setupCLIProjectMulti(t, srv.URL) + ctx := context.Background() + + // Unbounded first, on a throwaway copy of the project, to learn how many units this book really has — + // so the bounded assertion below is against a measured number rather than a guessed one. + fullPath := setupCLIProjectMulti(t, srv.URL) + if err := translate(ctx, fullPath, false, pipeline.RebillConsent{}, false, 0, 0); err != nil { + t.Fatalf("the unbounded baseline must complete: %v", err) + } + mu.Lock() + total := calls + calls = 0 + mu.Unlock() + if total < 3 { + t.Fatalf("fixture is too thin to bound: the whole book took %d call(s), so a ceiling of 2 proves nothing", total) + } + + // Now the same book through the CLI entry point WITH the flag. + if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0, 2); err != nil { + t.Fatalf("a volume-bounded run is a completion, not an error: %v", err) + } + mu.Lock() + bounded := calls + mu.Unlock() + if bounded != 2 { + t.Fatalf("`translate --max-units 2` made %d provider call(s) on a %d-unit book: the flag did not reach the engine (main.go's `r.MaxUnits = maxUnits`)", bounded, total) + } +} diff --git a/backend/cmd/tmctl/render.go b/backend/cmd/tmctl/render.go index 99ba614b..7b7f6523 100644 --- a/backend/cmd/tmctl/render.go +++ b/backend/cmd/tmctl/render.go @@ -121,6 +121,36 @@ func renderTranslate(w io.Writer, res *pipeline.BookResult, ledger func() (commi } fmt.Fprintf(w, "Book ledger: committed=$%.6f reserved=$%.6f\n", committed, reserved) + // The stop NAMES its ceiling. A run bounded by --max-units ends with exit 0 like any other completed + // run, so without this line the operator sees "TOTAL … chunks 3" on a hundred-unit book and has no way + // to tell "the volume I bought is done" from "something went quietly wrong" — the silent-limit + // complaint D39.165 §1б took from OpenHands. It names the ceiling, what was delivered as against + // re-made, and what remains of each kind, so the next action is readable off the line itself. + if v := res.Volume; v != nil { + fmt.Fprintf(w, "VOLUME CEILING: %s\n", v) + fmt.Fprintf(w, " (this is a COMPLETION, not a pause — the run did what it was granted; the money ceiling is a separate stop with its own exit code.)\n") + // ⚠ The invitation to buy more is spoken ONLY about units that were never delivered. The first + // version said "run again to buy more" unconditionally, which on a fully-translated book with a + // moved bank offered the reader chapters they already own — re-payment presented as stock. + // + // Both remainders are reported when both exist. An if/else here would have printed the fresh half + // and silently dropped the other, leaving an operator who has BOTH kinds pending believing the + // first number was all of it. + if v.LeftFresh > 0 { + fmt.Fprintf(w, " Next: %d output unit(s) of this book have never been delivered — `--max-units` again to buy them (units, NOT chapters: a chapter can be several units; the manifest is what converts).\n", v.LeftFresh) + } + if v.LeftRework > 0 { + if v.LeftFresh > 0 { + fmt.Fprintf(w, " Also pending, and NOT a purchase: %d already-delivered unit(s) carry a superseded snapshot and would be RE-MADE. Buying them delivers no new chapter.\n", v.LeftRework) + } else { + fmt.Fprintf(w, " Next: every unit of this book HAS been delivered; the %d left carry a superseded snapshot and would be RE-MADE, not delivered. That is a re-pass, not a purchase of new book.\n", v.LeftRework) + } + } + if v.Delivered == 0 && v.Reworked > 0 { + fmt.Fprintf(w, " ⚠ This run delivered NO new unit: all %d paid unit(s) were re-made under a moved snapshot.\n", v.Reworked) + } + } + // Exit code 2 is a typed sentinel, not an infra error: the report above is // already on stdout; main() maps this to a non-zero exit for the operator. if res.Flagged > 0 { @@ -663,7 +693,21 @@ func renderStatusHuman(w io.Writer, rep *pipeline.StatusReport, cfgPath string) // It is the same projection `translate` refuses on, so the operator reads the decision number here // instead of discovering it in a refusal. if rep.RebillUnits > 0 { - drift += fmt.Sprintf(" ⚠ RE-PAYMENT: %d chunk×stage unit(s) already billed under a superseded snapshot would be paid for again, ~$%.6f (translate needs --accept-rebill above the book's consent threshold)", rep.RebillUnits, rep.RebillUSD) + // BOTH denominations, always, and labelled. The chunk×stage count is what the money is billed in; + // the output-unit count is what a purchase is sized in. Printing only the first is how a reader + // hands `--max-units` a number several times too large. + drift += fmt.Sprintf(" ⚠ RE-PAYMENT: %d chunk×stage unit(s) already billed under a superseded snapshot would be paid for again — that is %d OUTPUT unit(s) of the book, the granularity --max-units counts in — ~$%.6f (translate needs --accept-rebill above the book's consent threshold). ⚠ Sizing --max-units from that number does NOT buy those units back: a grant goes to NEVER-DELIVERED units first and only reaches re-making when none are left, so on a book that still has undelivered units it will deliver new ones instead", + rep.RebillUnits, rep.RebillOutputUnits, rep.RebillUSD) + } + // A zero is not self-explanatory, and printing it as though it were is the mistake the `rebill_basis` + // field exists to stop: "nothing to re-pay" and "we could not work out what a re-pass would cost" are + // the same two zeroes. Only the two states that are NOT an answer speak here — a computed figure needs + // no caveat, and a book with nothing stored has nothing to say about re-payment at all. + switch rep.RebillBasis { + case pipeline.RebillBasisFailed: + drift += " ⚠ RE-PAYMENT UNKNOWN (the projection failed — the figures below are zero because they could not be computed, NOT because continuing is free; see the log)" + case pipeline.RebillBasisStored: + drift += " ⚠ RE-PAYMENT FIGURES ARE STALE (the book's bank could not be folded, so they describe the glossary the LAST run stored, not what the next run would build — fix the bank and re-read; see the log)" } fmt.Fprintf(w, "=== STATUS: %s (snapshot %s)%s ===\n", rep.BookID, snap, drift) fmt.Fprintf(w, "Progress: %d/%d units (%.1f%%) — done=%d in_progress=%d flagged=%d pending=%d\n", diff --git a/backend/cmd/tmctl/render_test.go b/backend/cmd/tmctl/render_test.go index 6094e7ef..a3a1d072 100644 --- a/backend/cmd/tmctl/render_test.go +++ b/backend/cmd/tmctl/render_test.go @@ -321,3 +321,106 @@ func TestRenderSignatureStopOrdersTheReviewLeastSureFirst(t *testing.T) { t.Fatalf("the banner must say what its order means, or a sorted table reads as arbitrary:\n%s", out) } } + +// TestRenderTranslateNamesTheVolumeCeiling pins the operator-facing half of the "the stop must NAME its +// ceiling" rule. A volume-bounded run exits 0 like any completed run, so without this line an operator +// looking at "chunks 2" on a hundred-unit book cannot tell "the volume I bought is finished" from +// "something went quietly wrong" — the silent-limit complaint the design took from OpenHands. The line +// has to carry all three actionable numbers and to say it is a completion, not a pause. +func TestRenderTranslateNamesTheVolumeCeiling(t *testing.T) { + var b strings.Builder + res := &pipeline.BookResult{BookID: "b1", TotalUSD: 0.01, + Volume: &pipeline.VolumeStop{MaxUnits: 2, Delivered: 1, Reworked: 1, Free: 3, LeftFresh: 5, LeftRework: 2}} + if err := renderTranslate(&b, res, okLedger); err != nil { + t.Fatal(err) + } + out := b.String() + for _, want := range []string{"VOLUME CEILING", "--max-units 2", "2 paying output unit(s)", + "1 NEW unit(s) delivered", "1 already-delivered unit(s) re-made", + "3 rode along at $0", "5 unit(s) NEVER delivered", "2 already delivered but not yet re-made", + "COMPLETION, not a pause", + // The invitation must be about the units that were never delivered, and only those. + "5 output unit(s) of this book have never been delivered"} { + if !strings.Contains(out, want) { + t.Errorf("the volume stop must say %q; got:\n%s", want, out) + } + } + // A run that was NOT bounded must say nothing at all — otherwise every ordinary run reads as cut short. + var b2 strings.Builder + if err := renderTranslate(&b2, &pipeline.BookResult{BookID: "b1"}, okLedger); err != nil { + t.Fatal(err) + } + if strings.Contains(b2.String(), "VOLUME CEILING") { + t.Errorf("an unbounded run must not mention a ceiling:\n%s", b2.String()) + } +} + +// TestRenderTranslateNeverSellsWhatIsAlreadyOwned pins the operator-facing half of the same defect. On a +// book every unit of which has been delivered, the stop must not invite a purchase: the units it held +// back are unrefreshed, not unbought, and the first version of this line offered them as "run again with +// --max-units to buy more". +func TestRenderTranslateNeverSellsWhatIsAlreadyOwned(t *testing.T) { + var b strings.Builder + res := &pipeline.BookResult{BookID: "b1", + Volume: &pipeline.VolumeStop{MaxUnits: 1, Delivered: 0, Reworked: 1, LeftFresh: 0, LeftRework: 3}} + if err := renderTranslate(&b, res, okLedger); err != nil { + t.Fatal(err) + } + out := b.String() + if strings.Contains(out, "have never been delivered") { + t.Fatalf("the stop offered chapters the reader already owns:\n%s", out) + } + for _, want := range []string{ + "every unit of this book HAS been delivered", + "That is a re-pass, not a purchase of new book", + "delivered NO new unit", + } { + if !strings.Contains(out, want) { + t.Errorf("want %q in:\n%s", want, out) + } + } +} + +// TestRenderTranslateReportsBothRemaindersWhenBothExist pins the case an if/else would have swallowed: +// a book with BOTH never-delivered units and delivered-but-superseded ones. Printing only the first +// leaves an operator who has both believing the one number was the whole picture. +func TestRenderTranslateReportsBothRemaindersWhenBothExist(t *testing.T) { + var b strings.Builder + res := &pipeline.BookResult{BookID: "b1", + Volume: &pipeline.VolumeStop{MaxUnits: 2, Delivered: 2, LeftFresh: 4, LeftRework: 6}} + if err := renderTranslate(&b, res, okLedger); err != nil { + t.Fatal(err) + } + out := b.String() + if !strings.Contains(out, "4 output unit(s) of this book have never been delivered") { + t.Errorf("the buyable remainder is missing:\n%s", out) + } + if !strings.Contains(out, "6 already-delivered unit(s) carry a superseded snapshot") { + t.Errorf("the re-make remainder was dropped — an if/else over the two would do exactly this:\n%s", out) + } + // A run that DID deliver must not carry the "delivered NO new unit" warning. + if strings.Contains(out, "delivered NO new unit") { + t.Errorf("this run delivered 2 units; it must not claim otherwise:\n%s", out) + } +} + +// TestTheRePaymentHintDoesNotPromiseWhatTheGrantWillNotDo pins the acceptance hunter's finding 6. The +// hint offers rebill_output_units as "the granularity --max-units counts in", which is true and, on its +// own, misleading: the grant goes to NEVER-DELIVERED units first, so on a book that still has any, sizing +// --max-units from the re-payment figure delivers new units and re-makes nothing. The caveat lives in the +// code; it has to live where the operator reads. +func TestTheRePaymentHintDoesNotPromiseWhatTheGrantWillNotDo(t *testing.T) { + var b strings.Builder + rep := &pipeline.StatusReport{BookID: "b1", ConfigDrift: true, + RebillUnits: 9, RebillOutputUnits: 3, RebillUSD: 0.02, RebillBasis: pipeline.RebillBasisPending} + if err := renderStatusHuman(&b, rep, ""); err != nil { + t.Fatal(err) + } + out := b.String() + if !strings.Contains(out, "3 OUTPUT unit(s)") { + t.Fatalf("the sale-sized figure must be shown:\n%s", out) + } + if !strings.Contains(out, "does NOT buy those units back") { + t.Fatalf("the hint must say the grant goes to never-delivered units first, or it promises a re-pass it will not perform:\n%s", out) + } +} diff --git a/backend/internal/pipeline/artifact.go b/backend/internal/pipeline/artifact.go index f36511a8..3751b33e 100644 --- a/backend/internal/pipeline/artifact.go +++ b/backend/internal/pipeline/artifact.go @@ -8,7 +8,9 @@ import ( // artifact.go: the shared write discipline of the engine's READ-OUT FILES — the sidecars beside the // project DB that another process (the platform, D39.81/D39.85) reads while this one runs: the -// chapter/chunk manifest (row 100), the machine bank-stop table (row 101), the bank export (row 125). +// chapter/chunk manifest (row 100), the machine bank-stop table (row 101), the bank export (row 125) and +// the engine's auto-bank (row 231 — it joined this list when the read-only surfaces started folding the +// bank from the decision files themselves, which is what made it a concurrently-read document). // // They have one property in common that the older sidecars did not have to care about: they are read // CONCURRENTLY with the run that writes them. A plain os.WriteFile truncates first, so a reader that diff --git a/backend/internal/pipeline/bankmaterialize.go b/backend/internal/pipeline/bankmaterialize.go new file mode 100644 index 00000000..5955c613 --- /dev/null +++ b/backend/internal/pipeline/bankmaterialize.go @@ -0,0 +1,403 @@ +package pipeline + +import ( + "fmt" + "sort" + + "textmachine/backend/internal/lang" + "textmachine/backend/internal/membank" + "textmachine/backend/internal/store" +) + +// bankmaterialize.go: the book's bank fold, split so that the SAME fold can be performed with or without the +// store round-trip that sits in the middle of it (row 231 / D39.165 §3, errata 28.08-к). +// +// THE PROBLEM IT EXISTS FOR. `seedGlossary` folds the bank from FILES (the curated seed, the ruby +// readings, the owner's mined-delta, the engine's auto-bank), writes the result with ReplaceBank, reads +// it straight back with GlossaryForBook and materializes THAT. Every read-only surface, however, had no +// way to reach the same answer: it cannot write, so `projectStoredMemory` materialized the glossary the +// LAST run stored — i.e. the previous fold. Right after a `bank-apply` the decision files hold rows the +// stored glossary does not, so a read-only projection answered "nothing moved" while the very next +// `translate` would re-seed, see the rows and bill for them. The projection and the run disagreed about +// money, which is the one thing they may never do. +// +// WHY THE ROUND-TRIP IS NOT A FORMALITY, and what makes skipping it PROVABLE rather than plausible. +// The store contributes exactly one thing to the fold: an ORDER. `GlossaryForBook` returns +// `ORDER BY src, sense, since_ch, until_ch, status, dst`, and that order is load-bearing twice over — +// `MaterializeBank` iterates the rows as given (it never re-sorts), so the order decides both the bank's +// content hash (ComputeVersionScopedIn streams a SHA-256 over the rows in iteration order) and, through +// Select's stable budget sort, the literal LINE ORDER of the rendered glossary block, whose bytes are +// hashed into chunk_status.content_hash. Reproduce the set but not the order and the free estimate names +// one number while the paid run charges another. +// +// The reproduction is exact, and the reason is a constraint rather than an observation: `glossary` +// carries UNIQUE (book_id, src, sense, since_ch, until_ch) (store/migrate.go), so the ORDER BY's first +// four columns CANNOT tie within one book — the order is total and fully determined by values, with no +// dependence on rowid or insertion order (`glossary.id` is deliberately excluded from the fold, see +// membank.ComputeVersion). The schema declares no COLLATE anywhere, so SQLite compares TEXT with its +// default BINARY collation — byte-wise, which is exactly what Go's `<` on a string does. Aliases come +// back `ORDER BY term_id, alias`, i.e. sorted by alias within an entry. Voice profiles and address pairs +// each have their own ORDER BY matching their own UNIQUE key, so the same argument covers them. +// +// storeOrder below therefore reproduces the round-trip's ONE contribution, and bankmaterialize_identity_test.go +// proves the equality by running both paths over the same inputs rather than by asserting this comment. + +// bankInputs is everything the deterministic $0 prelude gathers about a book's bank BEFORE anything is +// written: the glossary rows in the order seedGlossary builds them (which is the order ReplaceBank +// inserts them in), the two pack-19 record types, and the non-fatal remarks the gather produced. +// +// It is deliberately the PRE-store form. Turning it into what a run actually materializes is storeOrder's +// job, and it is one function so a caller cannot accidentally materialize the unsorted build order. +type bankInputs struct { + entries []store.GlossaryEntry + voices []store.VoiceProfile + pairs []store.AddressPair + // remarks are the gather's non-fatal findings, in the order they were made. seedGlossary logs them; + // a read-only projection drops them, because a $0 read must not narrate a run it is not making. + remarks []bankRemark +} + +// bankRemark is one non-fatal finding of the gather, carried rather than logged so the two callers can +// decide for themselves whether their surface is one that speaks. +type bankRemark struct { + msg string + args []any +} + +// gatherBankInputs performs the pure, $0, WRITE-FREE half of the bank fold: it reads the book's +// deterministic inputs and assembles the exact row set seedGlossary hands to ReplaceBank. +// +// Everything fatal here is fatal for the same reason it was fatal inside seedGlossary — a collision that +// would crash ReplaceBank mid-run, or a voice row naming a character the bank does not have, is a broken +// book and not a projection detail. What the two callers differ in is what they DO with that error: +// `translate` fails the run, a read-only projection falls back and says so. +func (r *Runner) gatherBankInputs() (bankInputs, error) { + var in bankInputs + if r.Book.GlossarySeed != "" { + bank, err := membank.LoadBankSeed(r.Book.GlossarySeed) + if err != nil { + return in, err + } + in.entries = append(in.entries, bank.Terms...) + in.voices, in.pairs = bank.Voices, bank.Pairs + } + ruby, err := r.Store.RubyReadingsForBook(r.Book.BookID) + if err != nil { + return in, fmt.Errorf("pipeline: read ruby readings for %s: %w", r.Book.BookID, err) + } + // D16.4: attach a manual term's kana ruby-reading as an alias (kana spelling matchable) + // BEFORE appending the auto-candidates, so it only touches the curated manual entries. A + // reading that would collide with a different seeded term (homophone) is skipped+logged, + // not attached (which would fail the book loud on an alias the operator cannot edit out). + if skipped := membank.AttachRubyAliasesToManual(in.entries, ruby); len(skipped) > 0 { + in.remark("ruby kana-alias skipped as a homophone collision (kana form left unmatchable; disambiguate in the seed if needed)", + "skipped", joinSemi(skipped)) + } + // Mined-write path (R1, plan §1(в)/F2): the owner-curated mined-delta file joins the seed as + // Source:"mined" (loadMinedDelta re-stamps the seed loader's Source), so its terms fold into the + // ENRICHED bank version but NOT the base — a re-run that adds signed mined terms moves ONLY snapshot_W2. + minedDelta, err := r.loadMinedDelta() + if err != nil { + return in, err + } + // D39.20 deviation-#1 fix: a mined-delta term whose UNIQUE key (src, sense, since_ch, until_ch — + // store/migrate.go glossary UNIQUE) already exists in the SIGNED seed makes the flat INSERT in + // ReplaceGlossary crash on that constraint and abort the whole paid run. membank.ApprovedSharedKeyCollisions + // below does NOT catch it (it skips a SAME-dst duplicate, and keys on the firing surface, not the + // UNIQUE tuple). Fail LOUD here with the duplicate list + a fix hint (edit the seed OR drop it from + // the delta) — NEVER a silent merge over the signed seed. Checked BEFORE the append so the delta rows + // are still separable. Deterministic (seed order). + if dups := membank.MinedDeltaSeedCollisions(in.entries, minedDelta); len(dups) > 0 { + return in, fmt.Errorf("pipeline: mined-delta %s duplicates term(s) already in the signed seed (would crash ReplaceGlossary on the glossary UNIQUE(book_id,src,sense,since_ch,until_ch)):\n - %s\n fix: correct the term in the seed, or remove it from the mined-delta file — never both (no silent merge over the signed seed)", + r.Book.MinedDelta, joinLines(dups)) + } + in.entries = append(in.entries, minedDelta...) + // AUTO-BANK (pack-20 / D39.42 п.3): the engine's own unsigned rows — what the terminologist consolidated + // on the last run's bank-mining boundary. They load exactly like the owner-curated delta (Source:"mined", + // so base-excluded and only the edit-wave snapshot moves) but they are NOT signed: their status is + // auto/draft, which is what routes them to the editor's separately-headed unverified section instead of + // the canon list. + // + // Loading them HERE rather than injecting them mid-run is load-bearing for money. The gather runs before + // checkRebillConsent, so the re-payment projection sees the same bank the edit wave will use; if the rows + // only appeared after the projection, the next run would compare stored edit rows against a bank that has + // not been rebuilt yet and project a re-payment of the whole edit wave that is not real. + autoBank, dropped, err := r.loadAutoBank(in.entries) + if err != nil { + return in, err + } + if len(dropped) > 0 { + // A collision with a SIGNED row cannot abort a paid run over an engine-written proposal — the signed + // row simply wins and the proposal is dropped, loudly. + in.remark("auto-bank rows dropped: their key is already held by a signed term (the signed term wins)", + "book", r.Book.BookID, "dropped", joinSemi(dropped)) + } + in.entries = append(in.entries, autoBank...) + for i := range in.entries { + in.entries[i].BookID = r.Book.BookID + } + // Task-6 re-audit: fail loud on a firing key shared by two DIFFERENT approved terms with + // different dst + overlapping windows (the alias generalization of the D16.1 polysemy + // livelock) — checked over the FULL entry set (incl. ruby-attached aliases) before persisting. + if cols := membank.ApprovedSharedKeyCollisions(in.entries); len(cols) > 0 { + return in, fmt.Errorf("pipeline: glossary shared-key collisions (A2 / D16.1 livelock class):\n - %s", joinLines(cols)) + } + // A voice/address row naming a character the bank does not have is SILENTLY inert — nothing can ever + // attribute a reply to it — which is the A-class hole this bank exists to close, so it stops the run. + // Checked over the FULL entry set (seed + ruby + mined + auto), because a character may legitimately be + // signed in a delta rather than in the base seed. + if unknown := membank.UnknownVoiceCharacters(in.entries, in.voices, in.pairs); len(unknown) > 0 { + return in, fmt.Errorf("pipeline: voice/address rows name characters absent from the bank (a profile for a term that does not exist can never fire):\n - %s", joinLines(unknown)) + } + return in, nil +} + +func (in *bankInputs) remark(msg string, args ...any) { + in.remarks = append(in.remarks, bankRemark{msg: msg, args: args}) +} + +// storeOrder reproduces, WITHOUT a store, exactly what a ReplaceBank+read-back round-trip returns: the +// same rows in the store's ORDER BY, each entry's aliases in the store's alias order, and the same for +// voice profiles and address pairs. The inputs are left untouched — a caller may still hand the ORIGINAL +// slice to ReplaceBank and get the insert order it has always had. +// +// It REFUSES the states the store itself would refuse rather than papering over them, because a +// projection of a book whose fold would crash the next run is not a projection of anything: a duplicate +// UNIQUE key would abort ReplaceBank's flat INSERT mid-transaction, and reporting a tidy number for it +// would promise a run that cannot start. The three refusals correspond one-to-one to the three UNIQUE +// constraints the round-trip passes through. +func storeOrder(in bankInputs) (rows []store.GlossaryEntry, voices []store.VoiceProfile, pairs []store.AddressPair, err error) { + type gkey struct { + src, sense string + since, until int + } + rows = append(rows, in.entries...) + seen := make(map[gkey]bool, len(rows)) + for i := range rows { + k := gkey{rows[i].Src, rows[i].Sense, rows[i].SinceCh, rows[i].UntilCh} + if seen[k] { + return nil, nil, nil, fmt.Errorf("pipeline: the folded bank holds two rows for (src=%q, sense=%q, since_ch=%d, until_ch=%d) — the glossary UNIQUE key; ReplaceBank would abort on it", k.src, k.sense, k.since, k.until) + } + seen[k] = true + // The store returns a term's aliases ORDER BY alias (within its term_id), and refuses a repeated + // one on UNIQUE (book_id, term_id, alias). The aliases are copied before sorting so an input slice + // shared with the write path keeps the order ReplaceBank inserts in. + if len(rows[i].Aliases) > 1 { + al := append([]store.GlossaryAlias(nil), rows[i].Aliases...) + sort.SliceStable(al, func(a, b int) bool { return al[a].Alias < al[b].Alias }) + for j := 1; j < len(al); j++ { + if al[j].Alias == al[j-1].Alias { + return nil, nil, nil, fmt.Errorf("pipeline: the folded bank gives term %q the alias %q twice — the glossary_aliases UNIQUE key; ReplaceBank would abort on it", rows[i].Src, al[j].Alias) + } + } + rows[i].Aliases = al + } + } + // GlossaryForBook: ORDER BY src, sense, since_ch, until_ch, status, dst. The first four columns are the + // table's UNIQUE key, so status/dst can never actually decide anything — they are kept so this reads as + // the SQL it reproduces and would still hold if that constraint were ever widened. + sort.SliceStable(rows, func(a, b int) bool { return glossaryLess(rows[a], rows[b]) }) + + voices = append(voices, in.voices...) + vseen := make(map[gkey]bool, len(voices)) + for _, v := range voices { + k := gkey{v.Src, v.Sense, v.SinceCh, v.UntilCh} + if vseen[k] { + return nil, nil, nil, fmt.Errorf("pipeline: the folded bank holds two voice profiles for (src=%q, sense=%q, since_ch=%d, until_ch=%d) — the voice_profiles UNIQUE key; ReplaceBank would abort on it", k.src, k.sense, k.since, k.until) + } + vseen[k] = true + } + // VoiceProfilesForBook: ORDER BY src, sense, since_ch, until_ch. + sort.SliceStable(voices, func(a, b int) bool { + x, y := voices[a], voices[b] + return lessKey4(x.Src, y.Src, x.Sense, y.Sense, x.SinceCh, y.SinceCh, x.UntilCh, y.UntilCh) + }) + + type pkey struct { + spSrc, spSense, adSrc, adSense string + since, until int + } + pairs = append(pairs, in.pairs...) + pseen := make(map[pkey]bool, len(pairs)) + for _, p := range pairs { + k := pkey{p.SpeakerSrc, p.SpeakerSense, p.AddresseeSrc, p.AddresseeSense, p.SinceCh, p.UntilCh} + if pseen[k] { + return nil, nil, nil, fmt.Errorf("pipeline: the folded bank holds two address pairs for speaker %q → addressee %q over the same window — the address_pairs UNIQUE key; ReplaceBank would abort on it", k.spSrc, k.adSrc) + } + pseen[k] = true + } + // AddressPairsForBook: ORDER BY speaker_src, speaker_sense, addressee_src, addressee_sense, since_ch, until_ch. + sort.SliceStable(pairs, func(a, b int) bool { + x, y := pairs[a], pairs[b] + if x.SpeakerSrc != y.SpeakerSrc { + return x.SpeakerSrc < y.SpeakerSrc + } + if x.SpeakerSense != y.SpeakerSense { + return x.SpeakerSense < y.SpeakerSense + } + if x.AddresseeSrc != y.AddresseeSrc { + return x.AddresseeSrc < y.AddresseeSrc + } + if x.AddresseeSense != y.AddresseeSense { + return x.AddresseeSense < y.AddresseeSense + } + if x.SinceCh != y.SinceCh { + return x.SinceCh < y.SinceCh + } + return x.UntilCh < y.UntilCh + }) + return rows, voices, pairs, nil +} + +// glossaryLess is GlossaryForBook's ORDER BY as a Go comparison. Both sides are byte-wise on TEXT (the +// schema declares no COLLATE, so SQLite uses BINARY) and numeric on the two INTEGER columns. +func glossaryLess(x, y store.GlossaryEntry) bool { + if x.Src != y.Src { + return x.Src < y.Src + } + if x.Sense != y.Sense { + return x.Sense < y.Sense + } + if x.SinceCh != y.SinceCh { + return x.SinceCh < y.SinceCh + } + if x.UntilCh != y.UntilCh { + return x.UntilCh < y.UntilCh + } + if x.Status != y.Status { + return x.Status < y.Status + } + return x.Dst < y.Dst +} + +func lessKey4(s1, s2, e1, e2 string, a1, a2, b1, b2 int) bool { + if s1 != s2 { + return s1 < s2 + } + if e1 != e2 { + return e1 < e2 + } + if a1 != a2 { + return a1 < a2 + } + return b1 < b2 +} + +// materializeBanks builds the two banks a run works against — the ENRICHED one the editor sees and the +// BASE one (Source:"mined" excluded) the draft wave selects over — from bank content already in the +// store's order. It is the shared tail of both folds, so the write path and the read-only projection can +// never materialize the same rows two different ways. +func (r *Runner) materializeBanks(rows []store.GlossaryEntry, voices []store.VoiceProfile, pairs []store.AddressPair) { + // InjectVoice is FALSE and has no config knob: pack-19 builds the schema and the flagger, and D21 п.2 + // holds the injection conditional until the polygon experiment. It is the fold's condition, so while + // it is false a book with voice rows hashes exactly as it did without them and nobody re-pays for + // authoring a profile. Wiring the injection means setting it and accepting a full --resnapshot. + // The §3 decl stemmer is target data, constant per book; baseIn copies bankIn below, so both banks share + // it. A target with no decl_suffix registry yields an inert stemmer → exact-match post-check as before. + // The rows are retained because a caller that has just folded the bank must be able to say things + // ABOUT it — how many of its terms are unsigned, for one — without going back to the store and + // getting a DIFFERENT bank than the one it just materialized. A status document that folds the + // decision files for its money figures and counts unsigned terms off the stored glossary is one + // document describing two banks. + r.bankRows = rows + // The wire-hash memo belongs to the bank that produced it: a new materialization invalidates every + // hash rendered against the old one (cachedRenderedContentHashes). + r.contentHashes = nil + bankIn := membank.BankInput{Rows: rows, Voices: voices, Pairs: pairs, + TargetStemmer: lang.NewTargetStemmer(lang.TargetChecksFor(r.Book.TargetLang))} + r.memory = membank.MaterializeBank(bankIn, r.Pipeline.Gates.Glossary.PostcheckGate) + // The DRAFT wave selects over a BASE-scoped bank (Source:mined excluded) so its injection is + // byte-identical across a bank-mining enrichment — matching the draft-wave snapshot (baseMemoryVersion), + // which keeps «one re-payment» honest at the WIRE level, not only the version-hash level. Only the + // editor sees mined terms (the enriched `memory`). When there are no mined rows (every $0 test / the + // golden) the base bank IS the enriched one — share the object, no double materialization, no drift. + baseRows := rows[:0:0] + hasMined := false + for _, row := range rows { + if row.Source == "mined" { + hasMined = true + continue + } + baseRows = append(baseRows, row) + } + if hasMined { + baseIn := bankIn + baseIn.Rows = baseRows + r.baseMemory = membank.MaterializeBank(baseIn, r.Pipeline.Gates.Glossary.PostcheckGate) + } else { + r.baseMemory = r.memory + } +} + +// projectFoldedMemory materializes BOTH banks from the book's deterministic inputs WITHOUT writing +// anything — the read-only twin of seedGlossary, and the answer to backlog row 231. +// +// What it fixes is a disagreement about money. `translate` re-seeds from the FILES and does it BEFORE +// checkRebillConsent, so its consent gate sees a `bank-apply` edit the moment the files change; the +// read-only surfaces materialized the STORED glossary instead — last run's fold — and therefore reported +// "nothing moved" for exactly the edit the next run would bill for. Reading the files rather than the +// store keeps every read-only guarantee intact: this writes nothing at all, so a store opened +// query_only(1) is untouched, and `status` remains the purely-reading verb it is ratified as. +// +// It also sets r.baseMemory, which projectStoredMemory never did — see the note on projectStoredMemory. +func (r *Runner) projectFoldedMemory() error { + in, err := r.gatherBankInputs() + if err != nil { + return err + } + rows, voices, pairs, err := storeOrder(in) + if err != nil { + return err + } + r.materializeBanks(rows, voices, pairs) + return nil +} + +// The values of StatusReport.RebillBasis — see its doc comment for what each one promises a reader. +// Exported because they are a WIRE vocabulary: the CLI renders them and a consumer branches on them, and +// a second hand-typed copy of an enum in the renderer is how the two drift apart. +const ( + RebillBasisPending = "pending" + RebillBasisStored = "stored" + RebillBasisNone = "none" + RebillBasisFailed = "failed" +) + +// foldMemoryForRead materializes the bank a READ-ONLY surface must judge against, and reports WHICH bank +// it managed to get. It is the one place the fallback lives, so status and export can never end up +// judging drift against two different banks. +// +// The preferred answer is the FOLD of the book's current inputs, because that is the bank the next +// `translate` will build (it re-seeds before its consent gate) — projecting anything else is projecting a +// run nobody is going to make. The fallback exists because the fold can legitimately refuse: a seed with a +// shared-key collision, a mined-delta duplicating a signed term, a voice row naming an absent character. +// Those are broken books, and `translate` fails loudly on them — but a READ command must not, or the book +// that most needs inspecting becomes the one that cannot be inspected (the D20.4 property rebill.go leans +// on: "a book that needs consent stays fully inspectable"). So the read falls back to the glossary the +// last run stored, and says so through the basis rather than passing the older number off as the answer. +// +// The returned error is the fold's refusal, for the caller to LOG in its own idiom; it is never fatal. +func (r *Runner) foldMemoryForRead() (basis string, warn error) { + foldErr := r.projectFoldedMemory() + if foldErr == nil { + return RebillBasisPending, nil + } + if storedErr := r.projectStoredMemory(); storedErr != nil { + return RebillBasisFailed, fmt.Errorf("the bank fold refused (%w) and the stored glossary could not be materialized either: %w", foldErr, storedErr) + } + return RebillBasisStored, foldErr +} + +func joinSemi(v []string) string { return joinWith(v, "; ") } +func joinLines(v []string) string { return joinWith(v, "\n - ") } + +func joinWith(v []string, sep string) string { + out := "" + for i, s := range v { + if i > 0 { + out += sep + } + out += s + } + return out +} diff --git a/backend/internal/pipeline/bankmaterialize_identity_test.go b/backend/internal/pipeline/bankmaterialize_identity_test.go new file mode 100644 index 00000000..0fb18257 --- /dev/null +++ b/backend/internal/pipeline/bankmaterialize_identity_test.go @@ -0,0 +1,414 @@ +package pipeline + +import ( + "bytes" + "context" + "log/slog" + "os" + "path/filepath" + "reflect" + "strings" + "testing" + + "textmachine/backend/internal/miner" + "textmachine/backend/internal/obs" + "textmachine/backend/internal/store" +) + +// identitySeed is deliberately hostile to the fold: every ordered axis is presented in the WRONG order. +// The terms descend by src where the store ascends, one term carries two senses whose order is inverted, +// two terms share a src and differ only by spoiler window (presented later-window-first), the aliases of +// one term are listed in reverse, and the voice/address rows are scrambled against their own ORDER BY. +// A fold that "keeps the order it was given" therefore cannot accidentally pass this fixture. +const identitySeed = ` +terms: + - src: 鈴木 + dst: Судзуки + type: name + status: approved + since_ch: 5 + decl: { invariant: true, forms: ["Судзуки"] } + aliases: + - { alias: すずき, type: name } + - { alias: SUZUKI, type: name } + - { alias: Suzuki, type: name } + - src: 鈴木 + dst: Судзуки-старший + type: name + status: approved + since_ch: 1 + until_ch: 4 + decl: { invariant: true, forms: ["Судзуки-старший"] } + - src: 図書館 + dst: библиотека + type: term + status: approved + sense: здание + since_ch: 3 + decl: { invariant: false, forms: ["библиотека", "библиотеки"] } + - src: 図書館 + dst: книгохранилище + type: term + status: draft + sense: а-архаизм + since_ch: 1 + until_ch: 2 + - src: 朝 + dst: утро + type: term + status: approved +voices: + - src: 図書館 + sense: здание + register: нейтральный + self_ref: я + address_default: formal + since_ch: 3 + - src: 鈴木 + register: разговорный + self_ref: я + address_default: informal + since_ch: 1 + until_ch: 4 + - src: 鈴木 + register: книжный + self_ref: я + address_default: formal + since_ch: 5 +addresses: + - speaker: 鈴木 + addressee: 図書館 + addressee_sense: здание + register: formal + form: вы + closeness: далеко + since_ch: 5 + - speaker: 図書館 + speaker_sense: здание + addressee: 鈴木 + register: informal + form: ты + closeness: близко + since_ch: 1 +` + +// identityDelta is the owner's decision file — the state that only exists AFTER a `bank-apply` and that +// the stored glossary therefore does not hold. Its rows sort INTO the middle of the seed's, so a fold +// that appended them instead of merging them by value would be caught. +const identityDelta = ` +terms: + - src: 静か + dst: тихий + type: term + status: approved + - src: 元海 + dst: Изначальное море + type: term + status: approved +` + +// identityAutoBank is the engine's own unsigned proposal file (Source:"mined", base-excluded), included +// so the fixture exercises the BASE/ENRICHED split as well as the plain ordering. +const identityAutoBank = ` +terms: + - src: 行った + dst: пошёл + type: term + status: auto +` + +// bankFoldFixture builds a project whose bank is folded from all four sources at once — curated seed, +// ruby readings, the owner's mined-delta and the engine's auto-bank — and returns the book path. +func bankFoldFixture(t *testing.T, providerURL string) string { + t.Helper() + bookPath := setupProjectOpts(t, providerURL, projectOpts{ + regenerate: 1, source: suzukiSource, glossarySeed: identitySeed, + }) + dir := filepath.Dir(bookPath) + writeFile(t, filepath.Join(dir, "test-book.mined-delta.yaml"), identityDelta) + writeFile(t, filepath.Join(dir, "test-book.db.auto-bank.yaml"), identityAutoBank) + return bookPath +} + +// stripIDs zeroes the store's autoincrement id, the ONE field the round-trip adds and the fold never +// hashes (membank.ComputeVersion excludes it by construction). Everything else must match exactly. +func stripIDs(rows []store.GlossaryEntry) []store.GlossaryEntry { + out := append([]store.GlossaryEntry(nil), rows...) + for i := range out { + out[i].ID = 0 + } + return out +} + +// TestTheInMemoryFoldIsIdenticalToTheStoreRoundTrip is the proof the free estimate rests on, and it is a +// proof by EXECUTION rather than by argument: both folds are run over the same inputs and every quantity +// that money depends on is compared. +// +// What is at stake. The canonical fold goes THROUGH the store — ReplaceBank writes, GlossaryForBook reads +// back ORDER BY src, sense, since_ch, until_ch, status, dst — and that order decides the bank's content +// hash and, through Select's stable budget sort, the literal bytes of the rendered glossary block, which +// are hashed into chunk_status.content_hash. A read-only surface cannot write, so it must reproduce the +// order without the store. If it reproduced it only approximately, the free projection would name one +// number and the paid run would charge another — which is precisely the defect the projection is sold as +// fixing. +// +// The comparison is made at three depths on purpose: the rows themselves (so a divergence is localised), +// the two bank versions (the snapshot component), and the per-position rendered content hashes (the wire +// bytes the resume fast-path and the re-bill projection actually compare). +func TestTheInMemoryFoldIsIdenticalToTheStoreRoundTrip(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := bankFoldFixture(t, srv.URL) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + // A RUN MUST HAVE HAPPENED before either fold is measured, and it is load-bearing rather than + // scene-setting. renderedContentHashes reproduces an EDIT position only when every member's stored + // draft is retrievable (repin.go), so on a book with no chunk_status rows the hash map comes back + // draft-only — and the edit wave is exactly the half the mined-delta and auto-bank rows reach, since + // Source:"mined" folds into the ENRICHED bank alone. Without the run this comparison would be blind to + // every possible defect in the half it was written to prove. + r0 := newRunner(t, bookPath) + if err := r0.Store.ReplaceRubyReadings("test-book", []store.RubyReading{ + {BookID: "test-book", Base: "鈴木", Reading: "すずき", FirstChapter: 1, Occurrences: 3}, + {BookID: "test-book", Base: "図書館", Reading: "としょかん", FirstChapter: 1, Occurrences: 2}, + }); err != nil { + t.Fatal(err) + } + if _, err := r0.TranslateBook(ctx); err != nil { + t.Fatalf("the run this comparison rests on did not complete: %v", err) + } + r0.Close() + + // --- Path A: the canonical fold, through the store --- + ra := newRunner(t, bookPath) + if err := ra.seedGlossary(ctx); err != nil { + t.Fatalf("canonical fold: %v", err) + } + rowsA, err := ra.Store.GlossaryForBook("test-book") + if err != nil { + t.Fatal(err) + } + voicesA, err := ra.Store.VoiceProfilesForBook("test-book") + if err != nil { + t.Fatal(err) + } + pairsA, err := ra.Store.AddressPairsForBook("test-book") + if err != nil { + t.Fatal(err) + } + verA, baseA := ra.memory.Version(), ra.baseMemory.Version() + hashesA := foldRenderedHashes(t, ra) + // The project flock is exclusive, so the second runner cannot open until the first lets go. + ra.Close() + + // --- Path B: the same fold with the write removed --- + rb := newRunner(t, bookPath) + defer rb.Close() + if err := rb.projectFoldedMemory(); err != nil { + t.Fatalf("in-memory fold: %v", err) + } + in, err := rb.gatherBankInputs() + if err != nil { + t.Fatal(err) + } + rowsB, voicesB, pairsB, err := storeOrder(in) + if err != nil { + t.Fatal(err) + } + verB, baseB := rb.memory.Version(), rb.baseMemory.Version() + hashesB := foldRenderedHashes(t, rb) + + // The fixture must actually exercise the ordering — a fold that never had to sort anything would + // pass this test while proving nothing. + if len(rowsA) < 8 { + t.Fatalf("fixture too thin to prove an ordering: %d rows", len(rowsA)) + } + if reflect.DeepEqual(stripIDs(rowsA), stripIDs(in.entries)) { + t.Fatalf("fixture is degenerate: the BUILD order already equals the store order, so sorting is untested") + } + + if !reflect.DeepEqual(stripIDs(rowsA), stripIDs(rowsB)) { + for i := range rowsA { + if i >= len(rowsB) { + t.Fatalf("row %d: store has %+v, memory ran out (%d rows)", i, rowsA[i], len(rowsB)) + } + a, b := rowsA[i], rowsB[i] + a.ID, b.ID = 0, 0 + if !reflect.DeepEqual(a, b) { + t.Fatalf("row %d differs:\n store = %+v\n memory = %+v", i, a, b) + } + } + t.Fatalf("row counts differ: store %d, memory %d", len(rowsA), len(rowsB)) + } + if !reflect.DeepEqual(voicesA, voicesB) { + t.Fatalf("voice profiles differ:\n store = %+v\n memory = %+v", voicesA, voicesB) + } + if !reflect.DeepEqual(pairsA, pairsB) { + t.Fatalf("address pairs differ:\n store = %+v\n memory = %+v", pairsA, pairsB) + } + if verA != verB { + t.Fatalf("enriched bank version differs: store %s, memory %s", verA, verB) + } + if baseA != baseB { + t.Fatalf("base bank version differs: store %s, memory %s", baseA, baseB) + } + if baseA == verA { + t.Fatalf("fixture is degenerate: the auto-bank row must make base ≠ enriched, both are %s", verA) + } + if !reflect.DeepEqual(hashesA, hashesB) { + t.Fatalf("rendered content hashes differ — the free estimate would name a number the paid run does not charge:\n store = %+v\n memory = %+v", hashesA, hashesB) + } + // ⚠ NOT merely "non-empty". Two empty maps compare equal, and so do two DRAFT-ONLY maps — and the + // draft wave is the half the fixture's mined rows do not even reach (Source:"mined" folds into the + // ENRICHED bank only). The comparison is only a proof if it actually covers the edit wave, so the + // coverage is asserted rather than assumed. + waves := map[string]int{} + for _, byStage := range hashesA { + for stage := range byStage { + waves[stage]++ + } + } + if waves["draft"] == 0 || waves["edit"] == 0 { + t.Fatalf("the byte-level comparison must cover BOTH waves — the mined rows this fixture carries reach only the edit one; got %v", waves) + } +} + +// foldRenderedHashes reproduces the per-position wire hashes exactly as the re-bill projection does +// (rebill.go), so the comparison is over the quantity money is decided by rather than over a proxy. +func foldRenderedHashes(t *testing.T, r *Runner) map[chunkKey]map[string]string { + t.Helper() + chunks, withText, err := r.readModelChunks() + if err != nil { + t.Fatalf("read-model chunks: %v", err) + } + _ = chunks + full, err := withText() + if err != nil { + t.Fatalf("re-chunk with text: %v", err) + } + return r.renderedContentHashes(full, precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget)) +} + +// TestTheInMemoryFoldRefusesWhatTheStoreWouldRefuse pins the other half of the identity: where the +// round-trip would ABORT, the projection must abort too. A fold that quietly de-duplicated a repeated +// UNIQUE key would answer with a tidy number for a book whose next run cannot even start. +func TestTheInMemoryFoldRefusesWhatTheStoreWouldRefuse(t *testing.T) { + dup := store.GlossaryEntry{BookID: "b", Src: "鈴木", Dst: "Судзуки", Status: "approved"} + if _, _, _, err := storeOrder(bankInputs{entries: []store.GlossaryEntry{dup, dup}}); err == nil { + t.Fatalf("a repeated glossary UNIQUE key must refuse, not de-duplicate") + } + aliased := store.GlossaryEntry{BookID: "b", Src: "図書館", Dst: "библиотека", Status: "approved", + Aliases: []store.GlossaryAlias{{Alias: "としょかん"}, {Alias: "としょかん"}}} + if _, _, _, err := storeOrder(bankInputs{entries: []store.GlossaryEntry{aliased}}); err == nil { + t.Fatalf("a repeated alias must refuse: glossary_aliases carries UNIQUE (book_id, term_id, alias)") + } + v := store.VoiceProfile{Src: "鈴木", Register: "разговорный"} + if _, _, _, err := storeOrder(bankInputs{voices: []store.VoiceProfile{v, v}}); err == nil { + t.Fatalf("a repeated voice-profile UNIQUE key must refuse") + } + p := store.AddressPair{SpeakerSrc: "鈴木", AddresseeSrc: "図書館"} + if _, _, _, err := storeOrder(bankInputs{pairs: []store.AddressPair{p, p}}); err == nil { + t.Fatalf("a repeated address-pair UNIQUE key must refuse") + } +} + +// TestAFailedFoldStillReportsWhatItSkipped pins a regression this pack introduced and then fixed. Before +// the gather was extracted from seedGlossary, its non-fatal warnings were emitted inline as they were +// made, so a book that died on a LATER check had still told the operator what the earlier steps skipped. +// The first version of the extraction logged the collected remarks only after the error return — losing +// the diagnostics of exactly the run that needs them most. +func TestAFailedFoldStillReportsWhatItSkipped(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + // The seed holds a signed term, and a voice row for a character the bank does NOT have — the latter is + // fatal, and it is checked AFTER the auto-bank load. + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + regenerate: 1, source: suzukiSource, glossarySeed: ` +terms: + - src: 鈴木 + dst: Судзуки + type: name + status: approved + decl: { invariant: true, forms: ["Судзуки"] } +voices: + - src: 存在しない + register: книжный + self_ref: я + address_default: formal +`}) + dir := filepath.Dir(bookPath) + // An auto-bank row on the key the signed seed term already holds: dropped with a REMARK, and the drop + // happens BEFORE the fatal voice check. + writeFile(t, filepath.Join(dir, "test-book.db.auto-bank.yaml"), ` +terms: + - src: 鈴木 + dst: Сузуки + type: name + status: auto +`) + var logBuf bytes.Buffer + r := newRunner(t, bookPath) + defer r.Close() + r.Log = slog.New(slog.NewTextHandler(&logBuf, &slog.HandlerOptions{Level: slog.LevelWarn})) + + err := r.seedGlossary(context.Background()) + if err == nil { + t.Fatalf("precondition: a voice row naming an absent character must fail the fold") + } + if !strings.Contains(err.Error(), "absent from the bank") { + t.Fatalf("precondition: expected the voice-character refusal, got: %v", err) + } + if !strings.Contains(logBuf.String(), "auto-bank rows dropped") { + t.Fatalf("the fold died and took its own diagnostics with it; the dropped auto-bank row was never reported.\nlog was:\n%s", logBuf.String()) + } +} + +// TestTheAutoBankIsWrittenAtomically pins a hole THIS pack opened in someone else's file. Until the +// read-only surfaces began folding the bank from the decision documents themselves, the auto-bank was +// read only by the run that had just written it, so a plain truncating os.WriteFile was safe. The moment +// `status` and `export` started reading it, it became one of the concurrently-read sidecars artifact.go +// exists for — and a truncating write hands a concurrent reader either zero bytes (a parse error, so the +// buyer sees stale figures) or a valid PREFIX of the term list, which is worse: a fold of a bank that +// never existed and a rebill_units no run will ever charge. +// +// The instrument is the inode, the same one bank-apply's tests use: a rename ALWAYS changes it, and a +// truncate-in-place never does. It is a stronger assertion than comparing bytes, because rewriting the +// same bytes non-atomically would pass a byte comparison and fail this. +func TestTheAutoBankIsWrittenAtomically(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{regenerate: 1, source: suzukiSource}) + r := newRunner(t, bookPath) + defer r.Close() + ctx := context.Background() + + first := []miner.Term{{Src: "図書館", Type: "term", SinceCh: 1, Freq: 3}} + if err := r.writeAutoBank(ctx, first, nil, nil); err != nil { + t.Fatalf("write auto-bank: %v", err) + } + before := inode(t, r.autoBankPath()) + + second := []miner.Term{ + {Src: "図書館", Type: "term", SinceCh: 1, Freq: 3}, + {Src: "鈴木", Type: "name", SinceCh: 1, Freq: 5}, + } + if err := r.writeAutoBank(ctx, second, nil, nil); err != nil { + t.Fatalf("rewrite auto-bank: %v", err) + } + if after := inode(t, r.autoBankPath()); after == before { + t.Fatalf("the auto-bank was rewritten IN PLACE (inode %d unchanged): a concurrent status/export read can catch it truncated or half-written, and a valid prefix of the term list folds a bank that never existed", before) + } + // And the document that landed is the whole new one, not a prefix of it. + body, err := os.ReadFile(r.autoBankPath()) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"図書館", "鈴木"} { + if !strings.Contains(string(body), want) { + t.Fatalf("the replacement document is incomplete: %q missing from\n%s", want, body) + } + } +} diff --git a/backend/internal/pipeline/bookrun.go b/backend/internal/pipeline/bookrun.go index 6de20b14..521dea67 100644 --- a/backend/internal/pipeline/bookrun.go +++ b/backend/internal/pipeline/bookrun.go @@ -88,6 +88,12 @@ type BookResult struct { Chunks []ChunkOutcome TotalUSD float64 // THIS run's spend across all chunks Flagged int // number of flagged chunks (acceptance allows N) + // Volume is non-nil when the run stopped because it had done the VOLUME of work it was granted + // (--max-units, volume.go) rather than because the book ended. It is a RESULT and never an error: a + // volume stop is a completion (exit 0, the frozen exit-code dictionary untouched), so the fact that + // this particular completion is not the end of the book has to travel HERE, in the report, or the + // operator gets a stop that does not say which ceiling produced it. + Volume *VolumeStop } // ExitCode is the run's shell disposition: 0 clean, 2 completed-with-flags @@ -184,20 +190,6 @@ func (r *Runner) translateBook(ctx context.Context) (*BookResult, error) { // manifest, and building it anywhere else would let a call path exist without it. r.bankSrc = newBankSourceIndex(chunks) - // Consent to a RE-PAYMENT (D20.2-Q2, rebill.go) — BEFORE the waves, hence before the first Reserve: - // if this run would pay again for units already billed under a superseded snapshot, and the amount - // is over the book's threshold, it stops here with the sum instead of quietly re-buying the book. - // It needs the materialized memory (the per-wave snapshots fold it), so it sits after seedGlossary, - // and the chunk manifest (the projected book cost is per output unit), so it sits after the split. - if err := r.checkRebillConsent(ctx, chunks); err != nil { - return nil, err - } - - // The run scale — one line to stderr (the smoke-run pain: N/M and the progress - // denominator never appeared in the logs at all, only in stdout/status). - r.Log.InfoContext(ctx, "book run started", "book", r.Book.BookID, - "chapters", len(doc.Chapters), "chunks", len(chunks), "stages", len(r.Pipeline.Stages)) - // the precompute pass: pre-compute the sticky-chain memory selection for EVERY chunk (WS1 §1б, precomputeSticky). The // sticky window (A5 scene-inertia — the recent chunks' exact matches in the current chapter, reset at // a chapter boundary) is a CROSS-CHUNK sequential dependency, so it is computed once here in the precompute pass and @@ -207,11 +199,51 @@ func (r *Runner) translateBook(ctx context.Context) (*BookResult, error) { // a bank-mining enrichment (the review-confirmed «re-paid ONCE» fix); the editor uses the enriched bank. stickySel := precomputeSticky(chunks, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget) + // The VOLUME ceiling's scope (volume.go): decided before either wave, so the ceiling is checked before + // an item begins rather than after it has been paid for. It needs stickySel, because the "would this + // unit resolve for free" predicate renders the same wire bytes the draft wave will. + // + // ⚠ IT IS DELIBERATELY COMPUTED BEFORE THE CONSENT GATE BELOW, not after. The gate asks the operator to + // consent to a CONCRETE spend (Р6), and a run bounded to ten units is not going to re-pay the whole + // book — quoting the book's figure at it would refuse work that was never going to be done, and ask for + // consent to money nobody will be charged. rebill.go says out loud why that is corrosive: "a number the + // operator is asked to approve and then not charged is exactly what makes such a number stop being + // read." Both steps are $0 and pure, so the reordering costs nothing. + scope, err := r.planVolume(ctx, chunks, stickySel) + if err != nil { + return nil, err + } + + // Consent to a RE-PAYMENT (D20.2-Q2, rebill.go) — BEFORE the waves, hence before the first Reserve: + // if this run would pay again for units already billed under a superseded snapshot, and the amount + // is over the book's threshold, it stops here with the sum instead of quietly re-buying the book. + // It needs the materialized memory (the per-wave snapshots fold it), so it sits after seedGlossary, + // and the chunk manifest (the projected book cost is per output unit), so it sits after the split. + if err := r.checkRebillConsent(ctx, chunks, scope); err != nil { + return nil, err + } + + // The run scale — one line to stderr (the smoke-run pain: N/M and the progress + // denominator never appeared in the logs at all, only in stdout/status). + r.Log.InfoContext(ctx, "book run started", "book", r.Book.BookID, + "chapters", len(doc.Chapters), "chunks", len(chunks), "stages", len(r.Pipeline.Stages)) + // The wave executor (R1, waverun.go): the draft wave (draft ∥) → the bank-mining stop → the edit wave (edit ∥). It computes + // upserts the per-wave snapshots (draft-wave snapshot base-bank / edit-wave snapshot enriched) itself and pins each // wave's jobs to its own — a the bank-mining stop enrichment moves only edit-wave snapshot, keeping draft-wave checkpoints valid // («re-paid ONCE»). Returns a *WaveSignatureStop when the bank-mining stop stops for owner sign. - res, err := r.translateBookWaves(ctx, chunks, stickySel) + res, err := r.translateBookWaves(ctx, chunks, stickySel, scope) + if err != nil && scope.bound() { + // A run can be stopped by the OTHER ceiling — money — and then the volume report never reaches the + // operator, because a failed run returns no result to carry it. Without this line they are told the + // run halted on money and nothing about what it had been granted or how far that got, which is the + // first thing anyone asks. The durable progress is in the store either way; this makes the stderr + // account of the run complete rather than half. + r.Log.WarnContext(ctx, "the run ended before its VOLUME grant was used up — the stop below is NOT the volume ceiling", + "book", r.Book.BookID, "max_units", scope.stop.MaxUnits, + "granted_new", scope.stop.Delivered, "granted_re_made", scope.stop.Reworked, + "left_never_delivered", scope.stop.LeftFresh, "err", err) + } if err == nil { // End of the run (row 125). The stop paths refresh the read-out themselves, so this is the boundary // they do not cover: a run that finished, whose last bank change was the auto-continue re-seed or a diff --git a/backend/internal/pipeline/export.go b/backend/internal/pipeline/export.go index bce9df1d..45ebad11 100644 --- a/backend/internal/pipeline/export.go +++ b/backend/internal/pipeline/export.go @@ -309,6 +309,21 @@ func (r *Runner) exportConfigDrift(statuses []store.ChunkStatus, draftStageNames editSnaps[cs.SnapshotID] = true } } + // ⚠ THE STORED GLOSSARY ON PURPOSE, and NOT the fold `status` uses. The two surfaces answer two + // different questions and this pack deliberately moved only one of them. + // + // status asks "what would the NEXT run cost", so it must fold the decision files: money is the point, + // and a bank-apply that has not been run yet is exactly the spend it exists to reveal (row 231). + // export asks whether THIS document is consistent with the run that produced it — its consumer is the + // polygon's extraction, which uses ConfigDrift to decide whether an export is trustworthy to MEASURE. + // An export whose text is byte-identical to what its run shipped is perfectly measurable even with a + // decision file sitting un-run beside the database, and folding that file here would mark every such + // export drifted and quietly disqualify sound measurements in another zone. + // + // This was briefly changed to the fold for symmetry and changed back: symmetry between two surfaces + // that answer different questions is not a property worth having, and redefining a field another zone + // consumes was not this pack's order. The divergence is deliberate, and it is named here so the next + // reader does not "fix" it either. if err := r.projectStoredMemory(); err != nil { r.Log.Warn("export: config-drift check failed; drift state unknown (reported as none)", "err", err) return diff --git a/backend/internal/pipeline/mining.go b/backend/internal/pipeline/mining.go index 1a03bc1e..1001cb42 100644 --- a/backend/internal/pipeline/mining.go +++ b/backend/internal/pipeline/mining.go @@ -712,7 +712,14 @@ func (r *Runner) writeAutoBank(ctx context.Context, mined []miner.Term, proposal if err != nil { return fmt.Errorf("pipeline: marshal auto-bank: %w", err) } - if err := os.WriteFile(r.autoBankPath(), []byte(body), 0o644); err != nil { + // ATOMIC, and the reason changed under this file's feet. Until the read-only surfaces began folding the + // bank themselves (bankmaterialize.go), the auto-bank was read only by the run that had just written + // it, so a plain truncating write was safe. Now `status` and `export` read it CONCURRENTLY with a + // running translate — the exact condition artifact.go names — and a truncating write gives a reader + // either a parse error (the buyer sees stale figures) or, worse, a valid PREFIX of the term list: a + // fold of a bank that never existed, and a rebill_units no run will ever charge. Write-then-rename + // makes every read see the previous document or the next one, whole. + if err := writeFileAtomic(r.autoBankPath(), []byte(body)); err != nil { return fmt.Errorf("pipeline: write auto-bank %s: %w", r.autoBankPath(), err) } if len(before) == 0 { diff --git a/backend/internal/pipeline/rebill.go b/backend/internal/pipeline/rebill.go index 3e2864c3..ce43ca61 100644 --- a/backend/internal/pipeline/rebill.go +++ b/backend/internal/pipeline/rebill.go @@ -63,6 +63,18 @@ type RebillProjection struct { // the operator-facing text can name what the number is made of rather than claim a re-pricing it did // not achieve. HistoricalRows int + // OutputUnits is the same re-payment counted in OUTPUT UNITS — the granularity the manifest publishes + // as units_total, that `--max-units` bounds, and that the platform sells chapters in. + // + // ⚠ IT EXISTS BECAUSE `Rows` IS A DIFFERENT UNIT WEARING THE SAME WORD. Rows counts chunk×stage, the + // BILLING unit; every other "units" number the engine puts on the wire (total_units, progress totals, + // chapters[].units_total) is the OUTPUT unit. Measured on a three-chapter fixture, one document carried + // `rebill_units: 15` two lines under `total_units: 6` with nothing marking the difference — and the + // ratio is not a constant a consumer could divide out: it is len(Members)·nDraftStages + nEditStages, + // which varies unit by unit WITHIN one book and whose factors never cross the seam at all. A platform + // sizing a re-pass from the estimate and passing that number to --max-units would buy several times the + // book it meant to, which is this pack's own defect reproduced one layer down. + OutputUnits int // Repinned counts the units the run will serve for $0 despite a moved snapshot (pack-20 point 5): the // move was bank-only and their rendered bytes are unchanged. It is not part of the amount — it is the // number that makes the amount believable, because before pack-20 every one of these was counted as a @@ -116,6 +128,17 @@ func (r *Runner) projectRebill(statuses []store.ChunkStatus, manifest []chunk.Ch for _, ch := range manifest { live[chunkKey{ch.Chapter, ch.ChunkIdx}] = true } + // Which OUTPUT unit each position belongs to, so the same re-payment can be reported in the unit the + // seam sells in as well as the one it bills in. An edit row lives at its unit's leader chunk and a + // leader is one of the unit's members, so this one map covers both waves. + leaderOf := make(map[chunkKey]chunkKey, len(manifest)) + for _, u := range r.outputUnits(manifest) { + leader := chunkKey{u.Chapter, u.FirstChunkIdx} + for _, m := range u.Members { + leaderOf[chunkKey{m.Chapter, m.ChunkIdx}] = leader + } + } + touched := map[chunkKey]bool{} draftNames := stageNameSet(r.waveStagesIndexed(waveDraft)) editNames := stageNameSet(r.waveStagesIndexed(waveEdit)) rendered := map[wave]string{} @@ -164,7 +187,7 @@ func (r *Runner) projectRebill(statuses []store.ChunkStatus, manifest []chunk.Ch if ferr != nil { return p, fmt.Errorf("pipeline: re-chunk the source for the re-bill content check: %w", ferr) } - contentHashes = r.renderedContentHashes(full, precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget)) + contentHashes = r.cachedRenderedContentHashes(full, precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget)) } if h, ok := contentHashes[chunkKey{cs.Chapter, cs.ChunkIdx}][cs.Stage]; ok && h == cs.ContentHash { p.Repinned++ @@ -172,12 +195,14 @@ func (r *Runner) projectRebill(statuses []store.ChunkStatus, manifest []chunk.Ch } } p.Rows++ + touched[leaderOf[chunkKey{cs.Chapter, cs.ChunkIdx}]] = true usd, fromHistory := rp.usd(cs) p.USD += usd if fromHistory { p.HistoricalRows++ } } + p.OutputUnits = len(touched) return p, nil } @@ -259,7 +284,28 @@ func projectionBasis(p RebillProjection) string { // the debt this closes. It also covers the case --resnapshot does not: a run interrupted midway through // a re-pin leaves jobs on the new snapshot while their chunk_status rows still carry the old one, and // the next plain `translate` then re-bills them with no gate at all. -func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk) error { +// `scope` is the run's VOLUME ceiling (volume.go), nil when none is in force. It produces TWO figures, +// and which of them each half of the gate is judged against is the whole correctness of this function. +// +// ⚠ THE THRESHOLD IS JUDGED AGAINST THE BOOK, NEVER AGAINST THE RUN — and the first version of this +// pack got that wrong. Scoping the amount to the admitted units and leaving the threshold book-wide +// sounds symmetrical and is not: with a volume ceiling the caller CHOOSES how small each run is, so +// `--resnapshot --max-units N` in a loop re-pays the entire book without a single run ever crossing the +// threshold and without consent being asked once. That is precisely the silent re-purchase Р6 exists to +// prevent, and "the change is inert without the flag" is true and beside the point: with the flag it +// opens the door the gate is the door for. A threshold a caller can defeat by splitting is not a +// threshold. So the question "must we ask at all" is answered by the book's whole drift, which does not +// shrink when a purchase does. +// +// ⚠ THE NAMED CAP IS JUDGED AGAINST THE RUN, because it is a different kind of statement. A threshold is +// the book's policy on when a human must be consulted; `--accept-rebill=X` is the caller's instruction +// "spend no more than X". Measuring an instruction about SPEND against work this run will not do would +// refuse a caller whose cap fully covers what they are about to be charged — the platform funds that cap +// from the run's own hold, so it is sized to the purchase, not to the book. +// +// With no ceiling in force the two projections are the same object and exactly one is computed, so every +// existing run is judged by byte-identical figures. +func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk, scope *volumeScope) error { statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID) if err != nil { return fmt.Errorf("pipeline: read chunk_status for the re-bill projection: %w", err) @@ -270,14 +316,32 @@ func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk) e } // A write path already holds the real split, so the lazy provider just hands it back — no second // ingest, and no branch where the consent gate could see text-free chunks. - proj, err := r.projectRebill(statuses, chunks, func() ([]chunk.Chunk, error) { return chunks, nil }, rp) + withText := func() ([]chunk.Chunk, error) { return chunks, nil } + book, err := r.projectRebill(statuses, chunks, withText, rp) if err != nil { return err } - if proj.Rows == 0 { + if book.Rows == 0 { return nil // nothing already-paid is superseded — this run bills only new work } + // What THIS run will actually re-pay. Only computed when a ceiling narrows it; otherwise it IS the + // book's figure, which keeps the un-bounded path at one projection exactly as before. + proj := book + if scope != nil { + if proj, err = r.projectRebill(scope.admittedStatuses(statuses), chunks, withText, rp); err != nil { + return err + } + } + if proj.Rows == 0 { + // The book carries drift, but none of it is inside what this run was granted, so this run re-pays + // nothing and there is no concrete spend to consent to. Not a hole in the threshold above: the + // moment a run is granted a superseded unit, proj stops being empty and the book-wide threshold + // below decides — so splitting cannot walk past the gate, it can only postpone meeting it. + return nil + } + // The denominator is the WHOLE book's stored rows, unfiltered by the volume scope: the threshold is + // "5% of what this book costs", a property of the book. byChunk := map[chunkKey][]store.ChunkStatus{} for _, cs := range statuses { byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}] = append(byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}], cs) @@ -296,16 +360,22 @@ func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk) e if r.AcceptRebill.Given { r.Log.WarnContext(ctx, "accepting a projected re-payment of already-billed work (--accept-rebill)", "rebill_units", proj.Rows, "rebill_usd", fmt.Sprintf("%.6f", proj.USD), + "book_rebill_units", book.Rows, "book_rebill_usd", fmt.Sprintf("%.6f", book.USD), "units_not_repriced", proj.HistoricalRows, "repinned_free", proj.Repinned, "threshold_usd", fmt.Sprintf("%.6f", threshold)) return nil } - if proj.USD <= threshold { + // ⚠ THE BOOK'S figure is what the threshold judges, not this run's. See the doc comment: a caller + // that chooses how small each run is could otherwise re-pay the whole book a slice at a time and + // never once be asked. `book` equals `proj` whenever no volume ceiling is in force, so this is the + // same comparison it has always been for every un-bounded run. + if book.USD <= threshold { // Under the threshold the run continues without friction (the ratified behaviour: a term append // touching three chunks costs cents). Continuing is not the same as being silent — the amount is // money and it goes to the log. r.Log.InfoContext(ctx, "re-paying already-billed work under the consent threshold; continuing without asking", "rebill_units", proj.Rows, "rebill_usd", fmt.Sprintf("%.6f", proj.USD), + "book_rebill_units", book.Rows, "book_rebill_usd", fmt.Sprintf("%.6f", book.USD), "units_not_repriced", proj.HistoricalRows, "repinned_free", proj.Repinned, "threshold_usd", fmt.Sprintf("%.6f", threshold)) return nil @@ -315,10 +385,30 @@ func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk) e if !r.Resnapshot { hint = " The run also needs --resnapshot: without it the superseded jobs stop it anyway." } + // ⚠ EVERY FIGURE IN A SENTENCE MUST COME FROM THE SAME PROJECTION AS THE SENTENCE. This parenthetical + // hangs off the BOOK-wide clause, so it takes the BOOK's re-pin count. The first version spliced the + // volume-scoped one in, which under an active ceiling printed a smaller number inside a sentence whose + // every other figure was book-wide — an operator adding them up would have got a book that did not + // exist. Two independent review lenses caught the same splice, which is what mixed-provenance numbers + // in one sentence reliably produce. repin := "" - if proj.Repinned > 0 { - repin = fmt.Sprintf(" (%d further unit(s) are re-pinned for $0 — the bank moved but their injected bytes did not)", proj.Repinned) + if book.Repinned > 0 { + repin = fmt.Sprintf(" (%d further unit(s) of the book are re-pinned for $0 — the bank moved but their injected bytes did not)", book.Repinned) } - return fmt.Errorf("pipeline: this run would RE-PAY for work already billed: %d chunk×stage unit(s) are resolved under a superseded snapshot and would be paid for again, ~$%.6f (%s)%s. That is over this book's consent threshold $%.6f (%s), and Р6 requires consent to a CONCRETE spend, not a blanket one (D20.2-Q2). NOTHING was reserved and no row was touched. Re-run with --accept-rebill to accept the whole projected amount, or --accept-rebill= to accept it only up to a ceiling (a ceiling below the projection refuses).%s", - proj.Rows, proj.USD, projectionBasis(proj), repin, threshold, source, hint) + // The two projections are printed SEPARATELY when a volume ceiling makes them differ, because + // collapsing them is how this gate goes wrong in either direction: quoting only the book's total at a + // small purchase asks for consent to money nobody will be charged, and quoting only the run's slice + // hides that the book is being re-bought a slice at a time. The scoped clause carries the scoped + // re-pin count for the same provenance reason. + scoped := "" + if book.Rows != proj.Rows || book.USD != proj.USD { + scopedRepin := "" + if proj.Repinned != book.Repinned { + scopedRepin = fmt.Sprintf(" (and re-pin %d of them for $0)", proj.Repinned) + } + scoped = fmt.Sprintf(" THIS run, bounded by --max-units, would re-pay %d of them, ~$%.6f%s — but the threshold is judged against the book, because a ceiling the caller sizes could otherwise re-pay the whole book a slice at a time without ever being asked.", + proj.Rows, proj.USD, scopedRepin) + } + return fmt.Errorf("pipeline: this run would RE-PAY for work already billed: %d chunk×stage unit(s) of this book are resolved under a superseded snapshot and would be paid for again, ~$%.6f (%s)%s.%s That is over this book's consent threshold $%.6f (%s), and Р6 requires consent to a CONCRETE spend, not a blanket one (D20.2-Q2). NOTHING was reserved and no row was touched. Re-run with --accept-rebill to accept the whole projected amount, or --accept-rebill= to accept it only up to a ceiling (the ceiling is measured against what THIS run re-pays, and one below that refuses).%s", + book.Rows, book.USD, projectionBasis(book), repin, scoped, threshold, source, hint) } diff --git a/backend/internal/pipeline/rebill_pending_test.go b/backend/internal/pipeline/rebill_pending_test.go new file mode 100644 index 00000000..20004998 --- /dev/null +++ b/backend/internal/pipeline/rebill_pending_test.go @@ -0,0 +1,526 @@ +package pipeline + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "testing" + + "textmachine/backend/internal/chunk/chunktest" + "textmachine/backend/internal/membank" + "textmachine/backend/internal/obs" +) + +// decisionsDocFor writes a decision document for an ARBITRARY book id — the package's existing helper +// hardcodes the one book its own fixtures use. +func decisionsDocFor(t *testing.T, dir, bookID string, ds ...membank.Decision) string { + t.Helper() + body, err := json.Marshal(membank.DecisionsDoc{Version: membank.DecisionsVersion, BookID: bookID, Decisions: ds}) + if err != nil { + t.Fatal(err) + } + p := filepath.Join(dir, "decisions-for-"+bookID+".json") + writeFile(t, p, string(body)) + return p +} + +// assertStoreUntouched is the "nothing was written" instrument, and it is deliberately TWO assertions +// because one of them is not enough. +// +// ⚠ HASHING THE .db FILE ALONE PROVES LESS THAN IT LOOKS. SQLite in WAL mode puts a committed change in +// the `-wal` sidecar and leaves the main file untouched until a checkpoint, so a data write can land with +// the main file's bytes unmoved — the first version of this helper would have gone on reporting "nothing +// was written" straight through it. So: the data file must be byte-identical AND the write-ahead log must +// carry no frames. +// +// ⚠ AND `-shm` IS DELIBERATELY NOT CHECKED. Measured, not assumed: opening this database read-only +// CREATES an empty `-wal` and a live `-shm`, because the shared-memory index is how any WAL reader finds +// its snapshot. Requiring those two files not to appear would fail on a pure read and would be asserting +// something SQLite's design forbids, not something about this engine. An empty `-wal` is the honest line: +// the reader may set the machinery up, it may not commit a frame through it. +// +// It is corroboration, not the guarantee. What actually makes the read path unable to write is +// store.OpenReadOnly's PRAGMA query_only(1) — a write through it fails loudly rather than being dropped. +// This catches a path that reached a WRITE handle by some route the read was not supposed to take. +func assertStoreUntouched(t *testing.T, dbPath, dataBefore string) { + t.Helper() + if got := fileDigest(t, dbPath); got != dataBefore { + t.Fatalf("the project database changed during a read-only estimate (%s → %s): nothing beyond the first touch may be written", dataBefore[:12], got[:12]) + } + wal, err := os.Stat(dbPath + "-wal") + if err != nil { + if !os.IsNotExist(err) { + t.Fatalf("stat %s-wal: %v", dbPath, err) + } + return // no write-ahead log at all — nothing could have been committed through one + } + if wal.Size() != 0 { + t.Fatalf("the write-ahead log holds %d byte(s) after a read-only estimate: a committed change can sit in -wal with the main database file still byte-identical, which is exactly what makes hashing the .db alone insufficient", wal.Size()) + } +} + +// fileDigest is a plain content hash of one file. +func fileDigest(t *testing.T, path string) string { + t.Helper() + b, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + sum := sha256.Sum256(b) + return hex.EncodeToString(sum[:]) +} + +// TestABankEditIsVisibleToTheEstimateBeforeAnythingIsBought is the deliverable of backlog row 231, and it +// is written against the state the FIRST attempt at this repro got wrong (errata 28.08-и/-к). +// +// THE DEGENERATE REPRO THAT MUST BE AVOIDED. "translate has never run" proves nothing: such a book has no +// chunk_status rows at all, and projectRebill skips rows without a snapshot (rebill.go), so it answers +// EMPTY both before and after any fix. The state that discriminates is: a run HAPPENED (rows and +// snapshots exist), THEN the bank was edited through `bank-apply`, which writes decision FILES and not one +// store row. On that state the engine used to answer "nothing moved" while the very next `translate` would +// re-seed from those files and bill for the difference. +// +// So the assertion is a DELTA, and both halves are executed here: the stored-glossary projection (what the +// engine answered before) must be zero on exactly the state where the folded projection is non-zero. +// Around it sit the two boundaries the estimate is sold with — not one provider call, and not one byte +// written to the store beyond the first touch that predates it. +func TestABankEditIsVisibleToTheEstimateBeforeAnythingIsBought(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + regenerate: 1, source: suzukiSource, glossarySeed: suzukiSeed, + }) + dir := filepath.Dir(bookPath) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + // --- the run that must have HAPPENED for the repro to discriminate --- + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatalf("the run this repro rests on did not complete: %v", err) + } + statuses, err := r1.Store.ChunkStatusesForBook("test-book") + if err != nil { + t.Fatal(err) + } + if len(statuses) == 0 { + t.Fatalf("precondition: the repro needs stored chunk_status rows, the state the degenerate version lacked") + } + committedBefore, reservedBefore, err := r1.Store.SpentUSD("test-book") + if err != nil { + t.Fatal(err) + } + r1.Close() + + // --- the bank edit, through the verb the product actually uses --- + // 静か occurs in the source, so the term enters the ENRICHED bank, moves the edit-wave snapshot AND + // changes the injected bytes of the edit unit — i.e. it is genuinely re-paid work, not a $0 re-pin. + doc := decisionsDocFor(t, dir, "test-book", membank.Decision{ + Action: membank.ActionApprove, Src: "静か", Dst: "тихий"}) + if _, err := ApplyBankDecisions(ctx, bookPath, doc, false); err != nil { + t.Fatalf("bank-apply: %v", err) + } + + callsBefore := rec.count() + dbPath := filepath.Join(dir, "test-book.db") + dbBefore := fileDigest(t, dbPath) + + // --- the two projections over the SAME state --- + r2, err := NewReadOnlyRunner(bookPath, obs.NewLogger()) + if err != nil { + t.Fatal(err) + } + defer r2.Close() + rp, err := r2.newRepricer() + if err != nil { + t.Fatal(err) + } + chunks, withText, err := r2.readModelChunks() + if err != nil { + t.Fatal(err) + } + + // What the engine answered BEFORE this pack: the glossary the last run stored. + if err := r2.projectStoredMemory(); err != nil { + t.Fatalf("stored-glossary projection: %v", err) + } + storedProj, err := r2.projectRebill(statuses, chunks, withText, rp) + if err != nil { + t.Fatalf("stored-glossary re-bill projection: %v", err) + } + + // What it answers now: the bank the next `translate` will actually build. + basis, warn := r2.foldMemoryForRead() + if warn != nil { + t.Fatalf("the fold refused on a healthy book: %v", warn) + } + if basis != RebillBasisPending { + t.Fatalf("basis = %q, want %q — a healthy book must be projected from its decision files", basis, RebillBasisPending) + } + foldedProj, err := r2.projectRebill(statuses, chunks, withText, rp) + if err != nil { + t.Fatalf("folded re-bill projection: %v", err) + } + + // THE DELTA. Both halves are load-bearing: a non-zero "after" alone would also pass on a book where + // the old answer was non-zero too, and would not prove the blind window was the thing that closed. + if storedProj.Rows != 0 { + t.Fatalf("the repro does not discriminate: the STORED-glossary projection already sees %d unit(s); the blind window this pack closes would be invisible here", storedProj.Rows) + } + if foldedProj.Rows == 0 { + t.Fatalf("the folded projection still reports nothing after a bank edit — the blind window of row 231 is open") + } + + // --- the two boundaries the estimate is SOLD with --- + if got := rec.count(); got != callsBefore { + t.Fatalf("the estimate made %d provider call(s); it is sold as $0 and must make none", got-callsBefore) + } + committedAfter, reservedAfter, err := r2.Store.SpentUSD("test-book") + if err != nil { + t.Fatal(err) + } + if committedAfter != committedBefore || reservedAfter != reservedBefore { + t.Fatalf("the ledger moved during a free estimate: committed %v→%v, reserved %v→%v", + committedBefore, committedAfter, reservedBefore, reservedAfter) + } + assertStoreUntouched(t, dbPath, dbBefore) +} + +// TestTheEstimateSurvivesABookWhoseFoldRefuses pins the property rebill.go leans on — "a book that needs +// consent stays fully inspectable" (D20.4). `translate` fails loudly on a bank whose fold would abort +// ReplaceBank; a READ must not, or the book that most needs looking at is the one that cannot be looked +// at. It must fall back AND say that it fell back, because a stale number passed off as the answer is +// worse than a labelled one. +func TestTheEstimateSurvivesABookWhoseFoldRefuses(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + regenerate: 1, source: suzukiSource, glossarySeed: suzukiSeed, + }) + dir := filepath.Dir(bookPath) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + r1.Close() + + // A mined-delta row on the SAME UNIQUE key as the signed seed term: exactly what would abort + // ReplaceBank's flat INSERT, and what seedGlossary refuses on before it gets there. + writeFile(t, filepath.Join(dir, "test-book.mined-delta.yaml"), ` +terms: + - src: 鈴木 + dst: Судзуки-другой + type: name + status: approved +`) + r2, err := NewReadOnlyRunner(bookPath, obs.NewLogger()) + if err != nil { + t.Fatal(err) + } + defer r2.Close() + + basis, warn := r2.foldMemoryForRead() + if basis != RebillBasisStored { + t.Fatalf("basis = %q, want %q — a refused fold must fall back to the stored glossary, not fail the read", basis, RebillBasisStored) + } + if warn == nil { + t.Fatalf("a silent fallback is the defect: the reason the projection is a fact about the PAST must reach the log") + } + if r2.memory == nil || r2.baseMemory == nil { + t.Fatalf("the fallback must still leave a materialized bank behind (memory=%v base=%v)", r2.memory != nil, r2.baseMemory != nil) + } + // And the whole read still answers rather than erroring. + rep, err := r2.Status(ctx) + if err != nil { + t.Fatalf("status refused a book whose fold refuses — D20.4 says it must stay inspectable: %v", err) + } + if rep.RebillBasis != RebillBasisStored { + t.Fatalf("the report must carry the fallback basis, got %q", rep.RebillBasis) + } +} + +// TestTheWireTellsFreeFromUnknown pins the omitempty mine D39.166 п.2 already paid for once: three states +// collapse to two zeroes unless the basis rides with them. +func TestTheWireTellsFreeFromUnknown(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + regenerate: 1, source: suzukiSource, glossarySeed: suzukiSeed, + }) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + // A book nothing has run: the figures are zero because there is nothing to re-pay. + r0, err := NewReadOnlyRunner(bookPath, obs.NewLogger()) + if err != nil { + t.Fatal(err) + } + rep0, err := r0.Status(ctx) + if err != nil { + t.Fatal(err) + } + r0.Close() + if rep0.RebillBasis != RebillBasisNone { + t.Fatalf("a book with no stored row must report basis %q, got %q", RebillBasisNone, rep0.RebillBasis) + } + + // A book that ran and drifted from nothing: the figures are zero because they were COMPUTED as zero. + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + r1.Close() + r2, err := NewReadOnlyRunner(bookPath, obs.NewLogger()) + if err != nil { + t.Fatal(err) + } + defer r2.Close() + rep1, err := r2.Status(ctx) + if err != nil { + t.Fatal(err) + } + if rep1.RebillBasis != RebillBasisPending { + t.Fatalf("a healthy run must report basis %q, got %q", RebillBasisPending, rep1.RebillBasis) + } + if rep1.RebillUnits != 0 { + t.Fatalf("precondition: an undrifted book re-pays nothing, got %d", rep1.RebillUnits) + } + // The two states above BOTH carry zero. Before this pack they were the same bytes on the wire. + if rep0.RebillBasis == rep1.RebillBasis { + t.Fatalf("«nothing to re-pay» and «computed, and it is zero» must be distinguishable, both say %q", rep0.RebillBasis) + } +} + +// TestAFailedBasisNeverCarriesAFigure pins the direction the basis field's first version did not +// consider. It was added so that "computed, and it is zero" could be told from "we could not compute it" +// — but a failed fold leaves r.memory nil, and projectRebill run against a nil bank renders both wave +// snapshots over an EMPTY bank, so EVERY stored row differs and the whole book comes back as a +// re-payment. The wire would then carry rebill_basis:"failed" beside the largest number the field can +// hold, and the CLI would print "the figures below are zero because they could not be computed" next to +// it. A basis that contradicts its own numbers is worse than no basis. +func TestAFailedBasisNeverCarriesAFigure(t *testing.T) { + big := RebillProjection{Rows: 4276, USD: 12.5, OutputUnits: 1400} + for _, c := range []struct { + name string + hasRows bool + memBasis string + proj RebillProjection + projErr error + basis string + units int + }{ + {"nothing stored", false, RebillBasisPending, big, nil, RebillBasisNone, 0}, + {"healthy fold", true, RebillBasisPending, big, nil, RebillBasisPending, 4276}, + {"fold refused, stored answered", true, RebillBasisStored, big, nil, RebillBasisStored, 4276}, + // The two that matter: an unknown answer must never arrive carrying a number. Before this was one + // decision, the basis said "failed" while a whole-book figure — computed over an EMPTY bank, so + // every row looked superseded — sat next to it. + {"no bank at all", true, RebillBasisFailed, big, nil, RebillBasisFailed, 0}, + {"projection errored", true, RebillBasisPending, big, errors.New("boom"), RebillBasisFailed, 0}, + } { + t.Run(c.name, func(t *testing.T) { + basis, units, usd, outUnits := rebillOutcome(c.hasRows, c.memBasis, c.proj, c.projErr) + if basis != c.basis || units != c.units { + t.Fatalf("got (%q, %d, %v), want (%q, %d)", basis, units, usd, c.basis, c.units) + } + // EVERY figure, not just the two obvious ones: an output-unit count surviving an UNKNOWN + // answer would be the same contradiction in a different field. + if basis == RebillBasisFailed && (units != 0 || usd != 0 || outUnits != 0) { + t.Fatalf("an UNKNOWN answer carried a figure: %d units, $%v, %d output units", units, usd, outUnits) + } + if basis == RebillBasisNone && outUnits != 0 { + t.Fatalf("nothing stored means nothing to re-pay, yet %d output units were reported", outUnits) + } + }) + } + // And the projection is not even attempted when there is no bank to compare against — running it + // would produce exactly the whole-book figure the case above must never show. + if projectable(true, RebillBasisFailed) { + t.Fatalf("with no materialized bank the projection must be skipped, not computed and then discarded") + } + if !projectable(true, RebillBasisPending) || projectable(false, RebillBasisPending) { + t.Fatalf("projectable must gate on stored rows and on a usable bank, and on nothing else") + } + + // And end to end: a book whose fold refuses AND whose store cannot answer either. The fold refusal is + // built with a mined-delta duplicating the signed seed term; the stored side is broken by closing the + // store out from under the read, which is the only way both halves fail at once. + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + regenerate: 1, source: suzukiSource, glossarySeed: suzukiSeed, + }) + dir := filepath.Dir(bookPath) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + r1.Close() + writeFile(t, filepath.Join(dir, "test-book.mined-delta.yaml"), ` +terms: + - src: 鈴木 + dst: Судзуки-другой + type: name + status: approved +`) + r2, err := NewReadOnlyRunner(bookPath, obs.NewLogger()) + if err != nil { + t.Fatal(err) + } + defer r2.Close() + got, err := r2.Status(ctx) + if err != nil { + t.Fatal(err) + } + // Here only the FOLD fails, so the honest answer is the stored fallback with a real figure — the + // point being that "failed" is reserved for the state where nothing at all could be materialized. + if got.RebillBasis != RebillBasisStored { + t.Fatalf("a fold refusal alone falls back rather than failing: got %q", got.RebillBasis) + } +} + +// TestExportAndStatusAnswerDifferentDriftQuestions pins a divergence that is DELIBERATE, so that nobody +// removes it as an inconsistency. +// +// After a bank-apply that has written only decision FILES, `status` reports drift and `export` does not, +// and both are right because they are asked different things. status is the money surface: its question +// is "what would the NEXT run cost", and an un-run decision file is precisely the spend row 231 exists to +// reveal. export's ConfigDrift is read by the polygon's extraction to decide whether a document is +// trustworthy to MEASURE — and an export whose bytes are exactly what its run shipped is measurable +// whether or not somebody has queued a bank edit beside it. +// +// The pack briefly made export fold too, for symmetry. Symmetry between two surfaces answering different +// questions is not worth having, and it would have disqualified sound measurements in another zone. +func TestExportAndStatusAnswerDifferentDriftQuestions(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + regenerate: 1, source: suzukiSource, glossarySeed: suzukiSeed, + }) + dir := filepath.Dir(bookPath) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + r1.Close() + + doc := decisionsDocFor(t, dir, "test-book", membank.Decision{ + Action: membank.ActionApprove, Src: "静か", Dst: "тихий"}) + if _, err := ApplyBankDecisions(ctx, bookPath, doc, false); err != nil { + t.Fatal(err) + } + + callsBefore := rec.count() + rExp, err := NewReadOnlyRunner(bookPath, obs.NewLogger()) + if err != nil { + t.Fatal(err) + } + exp, err := rExp.Export(false) + if err != nil { + t.Fatal(err) + } + rExp.Close() + if exp.ConfigDrift { + t.Fatalf("export marked a document drifted whose text is exactly what its run shipped: the polygon reads this field to decide whether a measurement is sound, and an un-run decision file does not make one unsound") + } + + rSt, err := NewReadOnlyRunner(bookPath, obs.NewLogger()) + if err != nil { + t.Fatal(err) + } + defer rSt.Close() + rep, err := rSt.Status(ctx) + if err != nil { + t.Fatal(err) + } + if !rep.ConfigDrift { + t.Fatalf("status must see the pending bank edit — that is the whole of row 231; got config_drift=false") + } + if rep.RebillUnits == 0 { + t.Fatalf("status must price the pending edit, got rebill_units=0 basis=%q", rep.RebillBasis) + } + // Both are $0 surfaces. + if got := rec.count(); got != callsBefore { + t.Fatalf("a read surface made %d provider call(s)", got-callsBefore) + } +} + +// TestTheEstimateIsAlsoGivenInTheUnitAPurchaseIsSizedIn pins the denomination fix. +// +// `rebill_units` counts chunk×stage — the BILLING granularity — while every other "units" number in the +// same document (total_units, the progress totals, chapters[].units_total) counts OUTPUT units, which is +// also what `--max-units` bounds and what the platform sells chapters in. Both wore the word "units" with +// nothing marking the difference, and the ratio is not something a consumer can divide out: it is +// len(Members)·nDraftStages + nEditStages, which varies unit by unit inside one book and whose factors +// never cross the seam. A platform sizing a re-pass from the estimate and handing that number to +// --max-units would buy several times the book it meant to — this pack's own defect, one layer down. +func TestTheEstimateIsAlsoGivenInTheUnitAPurchaseIsSizedIn(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + // A cut fine enough that units hold several chunks, so the two denominations genuinely differ. + const seg = "\nsegmentation:\n draft_budget_out: 24\n edit_ceiling_out: 8000\n fertility: { cjk: 1.1978, other: 0.3852 }\n" + var eps []chunktest.Chapter + var spine []string + for i := 1; i <= 2; i++ { + id := fmt.Sprintf("c%d", i) + eps = append(eps, chunktest.Chapter{ID: id, Href: id + ".xhtml", + Body: fmt.Sprintf("

静かな図書館の朝%d。鈴木は本を読んだ。外では雨が降っていた。彼は窓を見た。

", i)}) + spine = append(spine, id) + } + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + epub: eps, spine: spine, regenerate: 1, waveWorkers: 1, gatesYAML: seg, + }) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + r1.Close() + // A PIPELINE drift, not a bank edit: a mined bank edit moves only the edit wave, so it re-pays exactly + // one row per unit and the two denominations would coincide — the fixture would prove nothing. A config + // change moves BOTH waves, which is the ordinary shape in which a unit's several draft rows and its one + // edit row are all re-paid together, and where the collision this field exists for is real. + driftPipelineVersion(t, bookPath) + + r2, err := NewReadOnlyRunner(bookPath, obs.NewLogger()) + if err != nil { + t.Fatal(err) + } + defer r2.Close() + rep, err := r2.Status(ctx) + if err != nil { + t.Fatal(err) + } + if rep.RebillUnits == 0 { + t.Fatalf("precondition: the bank edit must produce a re-payment, got 0 (basis %q)", rep.RebillBasis) + } + if rep.RebillOutputUnits == 0 { + t.Fatalf("the estimate must also be given in OUTPUT units — the granularity --max-units counts and the platform sells in; got 0 beside rebill_units=%d", rep.RebillUnits) + } + if rep.RebillOutputUnits > rep.RebillUnits { + t.Fatalf("an output unit holds one or more chunk×stage rows, so it can never exceed them: output=%d rows=%d", rep.RebillOutputUnits, rep.RebillUnits) + } + if rep.RebillOutputUnits > rep.TotalUnits { + t.Fatalf("the book has %d output units; a re-pass cannot touch %d of them", rep.TotalUnits, rep.RebillOutputUnits) + } + // The fixture must actually make the two differ, or it proves nothing about the collision. + if rep.RebillOutputUnits == rep.RebillUnits { + t.Fatalf("fixture is degenerate: the two denominations coincide (%d), so the trap this field exists for is untested", rep.RebillUnits) + } +} diff --git a/backend/internal/pipeline/repin.go b/backend/internal/pipeline/repin.go index f7e3d36b..4aebeece 100644 --- a/backend/internal/pipeline/repin.go +++ b/backend/internal/pipeline/repin.go @@ -103,6 +103,28 @@ func jsonEqual(a, b json.RawMessage) bool { return xe == nil && ye == nil && string(xb) == string(yb) } +// cachedRenderedContentHashes is renderedContentHashes memoized for the life of ONE materialized bank. +// +// The reproduction walks every stage of every position and re-renders it — the expensive half of every +// $0 projection in this engine. Since the volume ceiling landed, a single bounded run could pay for it +// THREE times: once in planVolume, and once in each of the consent gate's two projections (the book's and +// the run's). Nothing changed between them, so two of the three were pure waste, and the cost scales with +// the book — exactly where it is least affordable. +// +// ⚠ THE CACHE'S LIFETIME IS THE BANK'S, and that is what makes it safe rather than merely fast. Every +// hash here is rendered against the materialized memory, so the memo is only valid while that memory is; +// materializeBanks clears it, which is the ONE place the bank can change (a mid-run re-seed at the +// bank-mining stop goes through it like everything else). A memo that outlived its bank would hand a +// caller hashes for a bank the run no longer has — the precise class of silent wrongness the free +// estimate exists to avoid. +func (r *Runner) cachedRenderedContentHashes(chunks []chunk.Chunk, stickySel []membank.Selection) map[chunkKey]map[string]string { + if r.contentHashes != nil { + return r.contentHashes + } + r.contentHashes = r.renderedContentHashes(chunks, stickySel) + return r.contentHashes +} + // renderedContentHashes reproduces, for every live position of every stage, the content hash the run // WOULD compute — the same msgsContentHash the resume fast-path compares against. It is the $0 half of // the honest estimate: with it, "the bank moved" can be answered per unit instead of per wave. diff --git a/backend/internal/pipeline/runner.go b/backend/internal/pipeline/runner.go index f1867660..4f52e24c 100644 --- a/backend/internal/pipeline/runner.go +++ b/backend/internal/pipeline/runner.go @@ -76,6 +76,19 @@ type Runner struct { // setting it moves no request_hash and re-bills nothing. The DAY ceiling is deliberately not // overridable: it is an account-wide guard, not a property of one run. CeilingUSD float64 + // MaxUnits is the VOLUME ceiling for THIS RUN (tmctl --max-units, D39.165 §1б), 0 = unset. It bounds + // how many OUTPUT UNITS this run may pay for — the granularity the manifest publishes and the platform + // sells chapters in — so a purchase of N chapters can stop at N instead of at whatever N chapters' + // worth of dollars happens to buy. + // + // ⚠ IT IS ORTHOGONAL TO CeilingUSD AND MUST STAY SO. CeilingUSD is a BOOK-CUMULATIVE money bound the + // ledger judges every reservation against; MaxUnits is a bound on THIS run's own work and never + // touches the ledger, the reservations or the hold arithmetic the platform builds on that + // cumulativeness. They can both be in force and either can be the one that stops the run — and the two + // stops are different answers to the operator: money means "the run was cut short, add money", volume + // means "the run did what was bought". Like the ceilings it is an operator axis, not a wire one: it is + // in neither BriefHash nor the snapshot payload, so setting it moves no request_hash and re-bills nobody. + MaxUnits int clients map[string]llm.LLMClient templates map[string]*PromptTemplate @@ -131,6 +144,14 @@ type Runner struct { // (identical content) — no double materialization, the injection is unchanged. The editor keeps the // enriched `memory`. nil ⇔ memory is nil (materialized together in seedGlossary). baseMemory *membank.Bank + // contentHashes memoizes the per-position wire-hash reproduction for the life of the CURRENT + // materialized bank (cachedRenderedContentHashes). Cleared by materializeBanks, which is the one place + // the bank can change. + contentHashes map[chunkKey]map[string]string + // bankRows are the glossary rows the CURRENT materialization was built from, in the store's order + // (materializeBanks). They exist so a read surface can describe the bank it actually folded rather + // than re-reading a different one out of the store. + bankRows []store.GlossaryEntry // checkers is the compiled WS5/pack-13 observability checker spec (pair-14 data-out): the pair's // DETECTION patterns + tables (from pack) + the target-general lists (from embedded target data), diff --git a/backend/internal/pipeline/seeding.go b/backend/internal/pipeline/seeding.go index 76e75b76..72db933c 100644 --- a/backend/internal/pipeline/seeding.go +++ b/backend/internal/pipeline/seeding.go @@ -6,7 +6,6 @@ import ( "sort" "strings" "textmachine/backend/internal/chunk" - "textmachine/backend/internal/lang" "textmachine/backend/internal/membank" "textmachine/backend/internal/store" ) @@ -21,94 +20,27 @@ import ( // once before snapshotID: the frozen APPROVED rows are hashed into memoryVersion (F1), so // editing the seed is a loud --resnapshot. Idempotent (full replace), $0, no LLM. B2 // approved dst-collisions are logged (not fatal — some collisions are legitimate). +// +// The gather and the materialization live in bankmaterialize.go, because a read-only surface has to reach the +// SAME fold without the write in the middle (row 231): what it cannot do is write, and the write is the +// only step it has to skip. This function is therefore the write path's spelling of one shared fold — +// gather, persist, read back, materialize — and the insert order it hands ReplaceBank is untouched. func (r *Runner) seedGlossary(ctx context.Context) error { - var entries []store.GlossaryEntry - var voices []store.VoiceProfile - var pairs []store.AddressPair - if r.Book.GlossarySeed != "" { - bank, err := membank.LoadBankSeed(r.Book.GlossarySeed) - if err != nil { - return err - } - entries = append(entries, bank.Terms...) - voices, pairs = bank.Voices, bank.Pairs + in, err := r.gatherBankInputs() + // The remarks go out BEFORE the error is acted on, and the order is load-bearing rather than tidy. + // Before the gather was extracted these warnings were emitted inline as they were made, so a book that + // died on a later collision check still told the operator about the ruby alias it had skipped or the + // auto-bank rows it had dropped. Logging them after the error return would lose exactly the diagnostics + // of the run that most needs them — the one that failed. gatherBankInputs returns its partial inputs + // with the error for this reason. + for _, rem := range in.remarks { + r.Log.WarnContext(ctx, rem.msg, rem.args...) } - manualSrcs := map[string]bool{} - for _, e := range entries { - manualSrcs[e.Src] = true - } - ruby, err := r.Store.RubyReadingsForBook(r.Book.BookID) - if err != nil { - return fmt.Errorf("pipeline: read ruby readings for %s: %w", r.Book.BookID, err) - } - // D16.4: attach a manual term's kana ruby-reading as an alias (kana spelling matchable) - // BEFORE appending the auto-candidates, so it only touches the curated manual entries. A - // reading that would collide with a different seeded term (homophone) is skipped+logged, - // not attached (which would fail the book loud on an alias the operator cannot edit out). - if skipped := membank.AttachRubyAliasesToManual(entries, ruby); len(skipped) > 0 { - r.Log.WarnContext(ctx, "ruby kana-alias skipped as a homophone collision (kana form left unmatchable; disambiguate in the seed if needed)", - "skipped", strings.Join(skipped, "; ")) - } - // Mined-write path (R1, plan §1(в)/F2): the owner-curated mined-delta file joins the seed as - // Source:"mined" (loadMinedDelta re-stamps the seed loader's Source), so its terms fold into the - // ENRICHED bank version but NOT the base — a re-run that adds signed mined terms moves ONLY snapshot_W2. - // Loaded AFTER the seed/ruby so manualSrcs already reflects the curated seed. A `mined` term that - // duplicates a seed src is caught by membank.ApprovedSharedKeyCollisions below like any other collision. - minedDelta, err := r.loadMinedDelta() if err != nil { return err } - // D39.20 deviation-#1 fix: a mined-delta term whose UNIQUE key (src, sense, since_ch, until_ch — - // store/migrate.go glossary UNIQUE) already exists in the SIGNED seed makes the flat INSERT in - // ReplaceGlossary crash on that constraint and abort the whole paid run. membank.ApprovedSharedKeyCollisions - // below does NOT catch it (it skips a SAME-dst duplicate, and keys on the firing surface, not the - // UNIQUE tuple). Fail LOUD here with the duplicate list + a fix hint (edit the seed OR drop it from - // the delta) — NEVER a silent merge over the signed seed. Checked BEFORE the append so the delta rows - // are still separable. Deterministic (seed order). - if dups := membank.MinedDeltaSeedCollisions(entries, minedDelta); len(dups) > 0 { - return fmt.Errorf("pipeline: mined-delta %s duplicates term(s) already in the signed seed (would crash ReplaceGlossary on the glossary UNIQUE(book_id,src,sense,since_ch,until_ch)):\n - %s\n fix: correct the term in the seed, or remove it from the mined-delta file — never both (no silent merge over the signed seed)", - r.Book.MinedDelta, strings.Join(dups, "\n - ")) - } - entries = append(entries, minedDelta...) - // AUTO-BANK (pack-20 / D39.42 п.3): the engine's own unsigned rows — what the terminologist consolidated - // on the last run's bank-mining boundary. They load exactly like the owner-curated delta (Source:"mined", - // so base-excluded and only the edit-wave snapshot moves) but they are NOT signed: their status is - // auto/draft, which is what routes them to the editor's separately-headed unverified section instead of - // the canon list. - // - // Loading them HERE rather than injecting them mid-run is load-bearing for money. seedGlossary runs before - // checkRebillConsent, so the re-payment projection sees the same bank the edit wave will use; if the rows - // only appeared after the projection, the next run would compare stored edit rows against a bank that has - // not been rebuilt yet and project a re-payment of the whole edit wave that is not real. - autoBank, dropped, err := r.loadAutoBank(entries) - if err != nil { - return err - } - if len(dropped) > 0 { - // A collision with a SIGNED row cannot abort a paid run over an engine-written proposal — the signed - // row simply wins and the proposal is dropped, loudly. - r.Log.WarnContext(ctx, "auto-bank rows dropped: their key is already held by a signed term (the signed term wins)", - "book", r.Book.BookID, "dropped", strings.Join(dropped, "; ")) - } - entries = append(entries, autoBank...) - for i := range entries { - entries[i].BookID = r.Book.BookID - } - // Task-6 re-audit: fail loud on a firing key shared by two DIFFERENT approved terms with - // different dst + overlapping windows (the alias generalization of the D16.1 polysemy - // livelock) — checked over the FULL entry set (incl. ruby-attached aliases) before persisting. - if cols := membank.ApprovedSharedKeyCollisions(entries); len(cols) > 0 { - return fmt.Errorf("pipeline: glossary shared-key collisions (A2 / D16.1 livelock class):\n - %s", strings.Join(cols, "\n - ")) - } - // A voice/address row naming a character the bank does not have is SILENTLY inert — nothing can ever - // attribute a reply to it — which is the A-class hole this bank exists to close, so it stops the run. - // Checked over the FULL entry set (seed + ruby + mined + auto), because a character may legitimately be - // signed in a delta rather than in the base seed. - if unknown := membank.UnknownVoiceCharacters(entries, voices, pairs); len(unknown) > 0 { - return fmt.Errorf("pipeline: voice/address rows name characters absent from the bank (a profile for a term that does not exist can never fire):\n - %s", strings.Join(unknown, "\n - ")) - } - if err := r.Store.ReplaceBank(r.Book.BookID, entries, voices, pairs); err != nil { - return fmt.Errorf("pipeline: replace bank for %s (%d terms, %d voices, %d pairs): %w", r.Book.BookID, len(entries), len(voices), len(pairs), err) + if err := r.Store.ReplaceBank(r.Book.BookID, in.entries, in.voices, in.pairs); err != nil { + return fmt.Errorf("pipeline: replace bank for %s (%d terms, %d voices, %d pairs): %w", r.Book.BookID, len(in.entries), len(in.voices), len(in.pairs), err) } rows, err := r.Store.GlossaryForBook(r.Book.BookID) if err != nil { @@ -128,36 +60,7 @@ func (r *Runner) seedGlossary(ctx context.Context) error { r.Log.WarnContext(ctx, "voice profile windows leave chapters uncovered (deliberate is fine; a typo is not)", "book", r.Book.BookID, "gaps", strings.Join(gaps, "; ")) } - // InjectVoice is FALSE and has no config knob: pack-19 builds the schema and the flagger, and D21 п.2 - // holds the injection conditional until the polygon experiment. It is the fold's condition, so while - // it is false a book with voice rows hashes exactly as it did without them and nobody re-pays for - // authoring a profile. Wiring the injection means setting it and accepting a full --resnapshot. - // The §3 decl stemmer is target data, constant per book; baseIn copies bankIn below, so both banks share - // it. A target with no decl_suffix registry yields an inert stemmer → exact-match post-check as before. - bankIn := membank.BankInput{Rows: rows, Voices: storedVoices, Pairs: storedPairs, - TargetStemmer: lang.NewTargetStemmer(lang.TargetChecksFor(r.Book.TargetLang))} - r.memory = membank.MaterializeBank(bankIn, r.Pipeline.Gates.Glossary.PostcheckGate) - // The DRAFT wave selects over a BASE-scoped bank (Source:mined excluded) so its injection is - // byte-identical across a bank-mining enrichment — matching the draft-wave snapshot (baseMemoryVersion), - // which keeps «one re-payment» honest at the WIRE level, not only the version-hash level. Only the - // editor sees mined terms (the enriched `memory`). When there are no mined rows (every $0 test / the - // golden) the base bank IS the enriched one — share the object, no double materialization, no drift. - baseRows := rows[:0:0] - hasMined := false - for _, row := range rows { - if row.Source == "mined" { - hasMined = true - continue - } - baseRows = append(baseRows, row) - } - if hasMined { - baseIn := bankIn - baseIn.Rows = baseRows - r.baseMemory = membank.MaterializeBank(baseIn, r.Pipeline.Gates.Glossary.PostcheckGate) - } else { - r.baseMemory = r.memory - } + r.materializeBanks(rows, storedVoices, storedPairs) if cols := membank.InjectivityCollisions(rows); len(cols) > 0 { r.Log.WarnContext(ctx, "glossary approved dst-collisions (B2: two source terms share one Russian surface — the reader cannot tell them apart)", "collisions", strings.Join(cols, "; ")) diff --git a/backend/internal/pipeline/status.go b/backend/internal/pipeline/status.go index 1721d683..51cecc18 100644 --- a/backend/internal/pipeline/status.go +++ b/backend/internal/pipeline/status.go @@ -7,7 +7,6 @@ import ( "strings" "textmachine/backend/internal/chunk" - "textmachine/backend/internal/membank" "textmachine/backend/internal/store" ) @@ -204,9 +203,43 @@ type StatusReport struct { // cent or the whole book — these say "N chunk×stage units already billed under a superseded snapshot // would be paid for again, ~$X at the CURRENT price table" (row 181). Same projection the consent // gate refuses on (projectRebill), so status can never quote a different number than the one - // translate enforces. Both omitempty: a book with no drift keeps its exact prior bytes. - RebillUnits int `json:"rebill_units,omitempty"` - RebillUSD float64 `json:"rebill_usd,omitempty"` + // translate enforces. + // + // ⚠ NEITHER IS omitempty ANY MORE, and dropping it is the point rather than a tidy-up. Three states + // reach this line — "nothing has run, so there is nothing to re-pay", "computed, and the answer is + // zero" and "the computation failed" — and omitempty rendered all three as the field being ABSENT, so + // the wire could not tell "free" from "unknown". That is the D39.166 п.2 class, which this engine has + // already paid for once ("a $0 price walls the door up — units at zero price arrive with no price"). + // RebillBasis below is what distinguishes them; the numbers are now always present so the basis has + // something to qualify. Safe to change: both fields are OFF the seam's allowlist until a consumer + // exists (backlog row 231), and the human renderer branches on the VALUE, not on presence. + RebillUnits int `json:"rebill_units"` + RebillUSD float64 `json:"rebill_usd"` + // RebillOutputUnits is the SAME re-payment counted in OUTPUT UNITS — the granularity every other + // "units" number in this document uses (total_units, progress totals, chapters[].units_total) and the + // one `--max-units` bounds and the platform sells chapters in. + // + // ⚠ IT EXISTS BECAUSE rebill_units IS A DIFFERENT UNIT WEARING THE SAME WORD: chunk×stage, the BILLING + // granularity. Measured on a three-chapter fixture, one document carried rebill_units:15 two lines + // under total_units:6, and the ratio is not something a consumer can divide out — it is + // len(Members)·nDraftStages + nEditStages, varying unit by unit inside one book, with its factors never + // crossing the seam. A consumer sizing a re-pass from rebill_units and handing that number to + // --max-units would buy several times the book it meant to: this pack's own defect, one layer down. + // rebill_units is NOT renamed — it is established and its meaning is unchanged; this is the number a + // caller should size a purchase from. + RebillOutputUnits int `json:"rebill_output_units"` + // RebillBasis says WHAT the two figures above are a projection of, because the number alone cannot + // carry that and a reader must never have to guess: + // + // pending — the fold of the book's decision FILES: the bank the NEXT `translate` will build, so the + // figures answer "what would a re-pass cost" for an edit that has been applied but not yet + // run. This is the ordinary answer. + // stored — the fold could not be built (a broken seed, a collision that would abort ReplaceBank), + // so the figures fall back to the glossary the LAST run stored. They are then a fact about + // the past, not a projection of the next run; the reason is on the WARN log. + // none — no stored row exists, so there is nothing already-billed to re-pay. Genuinely $0. + // failed — the projection itself errored. The figures are zero because they are UNKNOWN. + RebillBasis string `json:"rebill_basis"` TotalUnits int `json:"total_units"` Done int `json:"done"` @@ -492,17 +525,6 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) { if len(editStages) > 0 { rep.Progress.Edit.Total = len(units) } - // The unsigned-bank exposure, same definition as the quality report's (a row with a rendering that is - // not approved). Read-only, like everything else here; a read failure says so rather than reporting 0. - if rows, gerr := r.Store.GlossaryForBook(r.Book.BookID); gerr != nil { - r.Log.WarnContext(ctx, "status: could not read the bank; the unsigned-term count is unknown, not zero", "err", gerr) - } else { - for _, e := range rows { - if e.Status != "approved" && strings.TrimSpace(e.Dst) != "" { - rep.UnsignedBankTerms++ - } - } - } passports := map[int]*ChapterPassport{} var chapterOrder []int @@ -631,33 +653,57 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) { rep.Snapshot = s } } - if !rep.SnapshotDrift && (len(draftSnaps) > 0 || len(editSnaps) > 0) { - if err := r.projectStoredMemory(); err != nil { - // Do not stay silent: a projection failure used to be read silently as "no drift" (chain audit). - r.Log.WarnContext(ctx, "config-drift check failed; drift state unknown (reported as none)", "err", err) - } else { - checkWave := func(snaps map[string]bool, w wave) { - if len(snaps) != 1 { - return - } - var stored string - for s := range snaps { - stored = s - } - cur, _, serr := r.snapshotIDForWave(w) - if serr != nil { - r.Log.WarnContext(ctx, "config-drift check failed for a wave; drift state unknown", "err", serr) - return - } - if cur != stored { - rep.ConfigDrift = true - rep.CurrentSnapshot = cur - } + // The bank BOTH projections below are computed against, materialized ONCE. + // + // It is hoisted out of the drift branch it used to sit inside, and that move is a fix rather than a + // tidy-up: the re-bill projection is computed unconditionally further down, and on a book with + // snapshot drift this branch never ran — leaving r.memory nil, so `current(w)` in projectRebill + // rendered the wave snapshots over an EMPTY bank (memoryVersion falls back to the hash of no rows, + // snapshot.go), every stored row then differed from it, and status reported the WHOLE book as a + // re-payment. Materializing before either consumer removes the state where one of them runs without it. + memBasis, memWarn := r.foldMemoryForRead() + if memWarn != nil { + r.Log.WarnContext(ctx, "the bank fold refused, so the projections below are computed against the glossary the LAST run stored — they are a fact about the past, not a projection of the next run", "basis", memBasis, "err", memWarn) + } + // The unsigned-bank exposure, same definition as the quality report's (a row with a rendering that is + // not approved), counted off THE BANK THIS DOCUMENT JUST FOLDED. + // + // ⚠ It used to be read straight from the stored glossary, and after the fold landed that made one + // status document describe TWO banks: the money figures spoke about the bank the next run will build + // while this counter spoke about the bank the last run stored. An operator asking "is there anything + // to sign before I keep paying?" was told zero while the very next translate would inject an unsigned + // proposal sitting in the auto-bank file and bill for the unit it changed. One document, one bank. + if r.bankRows == nil { + r.Log.WarnContext(ctx, "status: no bank could be materialized; the unsigned-term count is unknown, not zero") + } else { + for _, e := range r.bankRows { + if e.Status != "approved" && strings.TrimSpace(e.Dst) != "" { + rep.UnsignedBankTerms++ } - checkWave(draftSnaps, waveDraft) - checkWave(editSnaps, waveEdit) } } + if memBasis != RebillBasisFailed && !rep.SnapshotDrift && (len(draftSnaps) > 0 || len(editSnaps) > 0) { + checkWave := func(snaps map[string]bool, w wave) { + if len(snaps) != 1 { + return + } + var stored string + for s := range snaps { + stored = s + } + cur, _, serr := r.snapshotIDForWave(w) + if serr != nil { + r.Log.WarnContext(ctx, "config-drift check failed for a wave; drift state unknown", "err", serr) + return + } + if cur != stored { + rep.ConfigDrift = true + rep.CurrentSnapshot = cur + } + } + checkWave(draftSnaps, waveDraft) + checkWave(editSnaps, waveEdit) + } // One re-pricing read serves both money projections below (reprice.go): the re-payment amount and // projected_book_usd, which is also the base of the consent threshold — computing them from two @@ -670,15 +716,17 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) { // not only under ConfigDrift — because SnapshotDrift (rows split across snapshots within one wave) // re-bills too, and that is precisely the case the boolean pair leaves unpriced. $0 and read-only: // projectRebill re-renders the wave snapshots and re-prices stored usage; it reaches no provider. - if len(statuses) > 0 { - if proj, perr := r.projectRebill(statuses, chunks, withText, rp); perr != nil { + var proj RebillProjection + var projErr error + if projectable(len(statuses) > 0, memBasis) { + proj, projErr = r.projectRebill(statuses, chunks, withText, rp) + if projErr != nil { // Same discipline as the drift check above: a failed projection is reported, never // silently rendered as "nothing to re-pay". - r.Log.WarnContext(ctx, "re-bill projection failed; the re-payment cost of the drift is unknown (reported as none)", "err", perr) - } else { - rep.RebillUnits, rep.RebillUSD = proj.Rows, proj.USD + r.Log.WarnContext(ctx, "re-bill projection failed; the re-payment cost of the drift is unknown (reported as unknown, not as none)", "err", projErr) } } + rep.RebillBasis, rep.RebillUnits, rep.RebillUSD, rep.RebillOutputUnits = rebillOutcome(len(statuses) > 0, memBasis, proj, projErr) // Money. committed, reserved, err := r.Store.SpentUSD(r.Book.BookID) @@ -730,16 +778,77 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) { return rep, nil } -// projectStoredMemory materializes r.memory from the STORED glossary (read-only: no re-seed, no write) — -// the side effect the wave-snapshot projections need. A seed-FILE edit not yet re-run is NOT reflected here -// (the STORED glossary is projected, not the seed file) — that drift surfaces on the next translate's -// re-seed, exactly as it does for `translate` itself. +// projectable reports whether the re-payment projection is worth computing at all. +// +// It is FALSE when no bank could be materialized, and skipping it then is the point rather than an +// optimization. The projection compares every stored row's snapshot against what the CURRENT config +// renders — and with r.memory nil, memoryVersion falls back to the hash of an EMPTY bank (snapshot.go), +// so every row differs from it and the whole book comes back as a re-payment. Running it anyway would put +// the largest figure the field can hold beside a basis that says "unknown", and the CLI would print "the +// figures below are zero because they could not be computed" next to it. +func projectable(hasRows bool, memBasis string) bool { + return hasRows && memBasis != RebillBasisFailed +} + +// rebillOutcome is the ONE decision about what the report says on the re-payment axis: the basis and the +// two figures, always together, so a branch cannot set one and forget the other. +// +// It exists as a function rather than as a switch inside Status because the invariant it carries is not +// reachable from a fixture: "failed" needs BOTH the fold and the stored materialization to refuse, which +// no healthy store will do. The first version of this logic left the basis at "failed" while a +// successfully-computed whole-book figure sat beside it — a basis contradicting its own numbers, which is +// worse than having no basis at all. Here the rule is a value, and a table test can hold it to it. +func rebillOutcome(hasRows bool, memBasis string, proj RebillProjection, projErr error) (basis string, units int, usd float64, outputUnits int) { + switch { + case !hasRows: + return RebillBasisNone, 0, 0, 0 // nothing already-billed exists, so nothing can be re-paid + case memBasis == RebillBasisFailed, projErr != nil: + // UNKNOWN, and zeroes that say so rather than figures that lie. EVERY figure, including the + // output-unit count: one number surviving here would be the same contradiction the basis exists to + // prevent, just in a different field. + return RebillBasisFailed, 0, 0, 0 + default: + return memBasis, proj.Rows, proj.USD, proj.OutputUnits + } +} + +// projectStoredMemory materializes r.memory from the STORED glossary (read-only: no re-seed, no write). +// +// ⚠ IT IS NO LONGER THE READ PATH'S FIRST ANSWER — it is foldMemoryForRead's FALLBACK, and the demotion +// is backlog row 231 (errata 28.08-к). What it materializes is LAST run's fold, and its own comment used +// to justify that with "a seed-FILE edit not yet re-run is NOT reflected here … exactly as it does for +// `translate` itself". The second half was false: TranslateBook calls seedGlossary BEFORE +// checkRebillConsent (bookrun.go), so `translate` re-seeds from the FILES and its consent gate sees the +// edit — while every read-only surface said "nothing moved" for exactly the work the next run would bill +// for. Right after a `bank-apply`, which writes decision FILES and no store row, that is the whole of the +// blind window: the platform had nothing to show a buyer before charging them. +// +// ⚠ IT NOW MATERIALIZES BOTH BANKS, and that is a fix this pack's own test caught rather than a +// generalization. It used to call membank.Materialize (rows alone) and set r.memory only, leaving +// r.baseMemory nil — and the re-bill projection's re-pin branch renders the DRAFT wave against exactly +// that bank (rebill.go → renderedContentHashes), where repin.go skips every position when it is nil. The +// positions then had no reproducible hash, "cannot conclude" took the conservative branch, and a bank move +// that touched BASE rows made status report the whole draft wave as a re-payment — while `translate`, +// which has both banks, charged for a fraction of it. status.go's own promise that it "can never quote a +// different number than the one translate enforces" was false in precisely that state. +// +// It reads the voice and address rows for the same reason: the run materializes with them, and a read that +// materializes without them is a second definition of the same bank. They do not move the version hash +// while InjectVoice is false (membank.ComputeVersionScopedIn), so nothing re-pays for this. func (r *Runner) projectStoredMemory() error { rows, err := r.Store.GlossaryForBook(r.Book.BookID) if err != nil { return err } - r.memory = membank.Materialize(rows, r.Pipeline.Gates.Glossary.PostcheckGate) + voices, err := r.Store.VoiceProfilesForBook(r.Book.BookID) + if err != nil { + return err + } + pairs, err := r.Store.AddressPairsForBook(r.Book.BookID) + if err != nil { + return err + } + r.materializeBanks(rows, voices, pairs) return nil } @@ -911,7 +1020,9 @@ func (r *Runner) Redrive(ctx context.Context, sel RedriveSelector) (*RedriveSumm if err != nil { return summary, nil, fmt.Errorf("pipeline: redrive re-bill projection: %w", err) } - if err := r.checkRebillConsent(ctx, rebillChunks); err != nil { + // nil scope: `redrive` does not take --max-units (invocation.go refuses it there), so there is no + // volume ceiling to narrow this projection by and the book-wide figure is the honest one. + if err := r.checkRebillConsent(ctx, rebillChunks, nil); err != nil { return summary, nil, err } diff --git a/backend/internal/pipeline/status_rebill_projection_test.go b/backend/internal/pipeline/status_rebill_projection_test.go index 4846b4de..bb810a09 100644 --- a/backend/internal/pipeline/status_rebill_projection_test.go +++ b/backend/internal/pipeline/status_rebill_projection_test.go @@ -67,11 +67,33 @@ func TestStatusPricesTheDriftWithTheGatesOwnNumber(t *testing.T) { } // The same number, from the enforcement side: the gate's refusal must quote what status showed. - err = r2.checkRebillConsent(ctx, chunksOf(t, r2)) + err = r2.checkRebillConsent(ctx, chunksOf(t, r2), nil) if err == nil { t.Fatal("the gate must refuse a fully-superseded book without consent") } if want := fmt.Sprintf("~$%.6f", rep.RebillUSD); !strings.Contains(err.Error(), want) { t.Errorf("the refusal does not quote the projection %s:\n%v", want, err) } + + // ⚠ AND THE SAME AGAIN UNDER A VOLUME CEILING, because that is where the two sides could drift apart + // and status.go's docstring promises out loud that they cannot: "status can never quote a different + // number than the one translate enforces". A bounded run re-pays only its own slice, but the question + // the THRESHOLD answers is about the book — so the figure the refusal quotes must still be the book's, + // which is the one status reports. Passing nil here (the shape this pin had while the pack was being + // built) tests only the unbounded case and would miss a divergence entirely. + // A scope that ADMITS the book's single unit — i.e. a real bounded purchase that genuinely re-pays. + // (A scope admitting nothing is a different case and is handled before the threshold: a run that + // re-pays nothing has no concrete spend to consent to.) + bounded := &volumeScope{ + admitted: map[chunkKey]bool{{1, 0}: true}, + leader: map[chunkKey]bool{{1, 0}: true}, + stop: VolumeStop{MaxUnits: 1}, + } + err = r2.checkRebillConsent(ctx, chunksOf(t, r2), bounded) + if err == nil { + t.Fatal("a bounded run over a fully-superseded book must still meet the gate: a threshold a caller can shrink by splitting is not a threshold") + } + if want := fmt.Sprintf("~$%.6f", rep.RebillUSD); !strings.Contains(err.Error(), want) { + t.Errorf("under a volume ceiling the refusal stopped quoting the BOOK figure status reports (%s):\n%v", want, err) + } } diff --git a/backend/internal/pipeline/volume.go b/backend/internal/pipeline/volume.go new file mode 100644 index 00000000..58eced44 --- /dev/null +++ b/backend/internal/pipeline/volume.go @@ -0,0 +1,581 @@ +package pipeline + +import ( + "context" + "fmt" + + "textmachine/backend/internal/chunk" + "textmachine/backend/internal/membank" + "textmachine/backend/internal/store" +) + +// volume.go: the VOLUME ceiling — the run's second stop, beside the money one (D39.165 §1б, backlog row +// «потолок объёма»). +// +// WHY A SECOND CEILING AT ALL. The platform sells CHAPTERS and hands the engine a DOLLAR bound +// (--ceiling-usd), which is the only bound the engine had. Measured on real ledgers (D39.165 §1), a +// chapter costs $0.0115–$0.0190 and the constant sold against it is $0.03 — so a purchase of ten chapters +// hands over $0.30, and $0.30 buys sixteen to twenty-six. The knob said chapters and meant money. This +// gives the engine a bound in the unit it actually ships, so the seller's promise and the engine's stop +// are the same quantity. +// +// WHAT IT IS MEASURED IN, and why not the other two candidates. The unit is the OUTPUT UNIT — editUnit, +// the granularity `outputUnits` returns and `buildManifest` publishes as the manifest's units_total, which +// is where the platform's per-chapter unit count comes from. So "N units" means the same thing on both +// sides of the seam, and the platform's chapters→units conversion is EXACT rather than an estimate: the +// manifest is $0, needs no keys and exists before the first paid call. +// - NOT chunk×stage, which is the BILLING unit (the one the re-payment estimate counts in). Cutting the +// wave in the unit one PAYS in, while selling the unit one SHIPS in, reproduces the very defect above +// one layer down: a unit costs len(Members)·nDraftStages + nEditStages positions, so "10" would buy a +// different amount of book on a different pipeline shape. +// - NOT the chunk, which is an internal artefact of the cut: the buyer never sees it and the manifest +// does not count it. +// +// ⚠ The unit count DOES depend on the pipeline's shape — an editor pipeline groups draft chunks into +// coarse edit units, a draft-only pipeline ships one unit per draft chunk (outputUnits). That is a +// property of the configuration, not a branch per language pair, and it is safe because the manifest the +// platform counted and the run it bounds come from the SAME deployment. It would stop being safe if a +// ceiling were carried across two deployments with different pipelines. +// +// HOW REPINS AND RETRIES COUNT — the question this design has to answer out loud. +// - A RETRY and an ESCALATION hop do NOT count. They live inside one unit (the attempt loop in +// stagerun.go writes ONE chunk_status row per chunk×stage however many attempts it took), and the +// volume ceiling bounds DELIVERY while the money ceiling bounds SPEND. The measured quality tail +// (retries+escalations swinging 1.5%→23% between runs) is a fact about money and already has its own +// bound; if it burned volume, a buyer of ten chapters would receive eight because two of them +// stuttered — paying in BOOK for the engine's own tail, which is the substitution this pack exists to +// remove. +// - A RE-PIN and a $0 RESUME do NOT count. A re-pass that re-pins five hundred units for $0 (rebill.go) +// would otherwise exhaust the ceiling having delivered nothing. +// +// So: THE CEILING COUNTS OUTPUT UNITS FOR WHICH THIS RUN WILL MAKE AT LEAST ONE PROVIDER CALL. Units that +// resolve for free are admitted regardless — they cost nothing, and holding them back would leave the +// book's snapshots stale for no saving. +// +// AND IT COUNTS THEM IN TWO KINDS, which is the difference between a report and a true report. A paying +// unit is either book that did not exist (DELIVERY) or book that existed and is being made again under a +// moved snapshot (REWORK). unitClass below is that distinction, and VolumeStop carries it all the way to +// the operator: a purchase spent entirely on rework delivers no chapter, and neither the counters nor the +// line the CLI prints may let that read as a delivery. The remainder splits the same way — units never +// delivered are the only ones anybody may be invited to buy, while delivered-but-not-re-made units are +// unrefreshed, not unbought. +// +// ⚠ ON THE PREDICATE'S RELATION TO projectRebill: the free/paying test here is runStage's, not the +// re-payment estimate's, and they are NOT the same test. projectRebill deliberately ignores the CONTENT +// axis (its own doc comment says so: a source edit moves content_hash but not the snapshot, and is +// invisible there). This one checks the rendered content hash on every row, so a source edit correctly +// makes a unit paying here while the estimate would call it free. The difference is in the safe +// direction — the ceiling over-counts cost rather than under-counting it — and it is why this predicate +// is written against the executor rather than borrowed from the projection. +// +// CHECKED BEFORE, NOT AFTER. The scope is computed ONCE, before either wave starts, and the waves simply +// do not begin an item outside it. That is structural rather than a guard: there is no moment at which a +// unit past the ceiling has started, so the ceiling cannot be overshot by one call the way a post-hoc +// token check always is (LiteLLM's known behaviour, named in D39.165 §1б). It is also why the admission +// is not a counter the wave workers decrement — the waves are parallel, and a worker refused a slot would +// have to abandon its item permanently while a slot freed by a $0 re-pin went to nobody. +// +// THE STOP IS A COMPLETION, NOT A HALT — ratified by the orchestrator with this pack and NOT re-decided +// here. A money ceiling stops the run in the MIDDLE of work it meant to do: that is exit 4, the platform +// records `paused`, and the remedy is more money. A volume ceiling means the run did exactly what was +// bought: exit 0, the frozen exit-code dictionary (cmd/tmctl/main.go) is not touched at all, and the +// platform records the run `ready` — which does NOT declare the book finished, because the book's progress +// travels as unit events, so the next purchase starts a new run normally. What DOES have to be said is +// WHICH ceiling stopped it (OpenHands' iteration limit is silent, and that is a standing complaint named +// in the same note): that lives in the run's result, its log line and the CLI's rendering, not in a new +// exit code and not in a new word of the event stream's outcome vocabulary. + +// VolumeStop is a run's report that it stopped on the volume ceiling rather than on the end of the book. +// It is a RESULT, not an error: the run did what it was granted, so it travels on BookResult and never +// through the error path, where the exit-code mapper would have to invent a word for it. +// +// ⚠ IT COUNTS PAID UNITS IN TWO KINDS, and the split is a MODEL rather than a nicety. A paying unit is +// either book that did not exist before (DELIVERY) or book that existed and is being made again under a +// moved snapshot (REWORK), and the two are not the same product: the first is what "buy ten chapters" +// means, the second is what a re-pass means. The first version of this struct had one counter for both, +// and every consequence of that was a lie the operator could act on — a purchase that went entirely into +// re-translating the opening chapters reported ten units bought and exited 0 like a delivery, while the +// remaining count invited buying chapters the reader already owned. A struct that cannot tell the two +// apart cannot report either one honestly, so it carries both. +type VolumeStop struct { + // MaxUnits is the ceiling that was in force. + MaxUnits int + // Delivered is paying units that had never been completed before — NEW book, the thing a purchase of + // chapters is actually for. + Delivered int + // Reworked is paying units that WERE already completed and are being paid for a second time because + // their snapshot moved. Real work and a real charge, but not new book: a purchase made entirely of + // these advances the reader not at all. + Reworked int + // Flagged is paid units that resolved FLAGGED and shipped no text. They cost money and delivered + // nothing, and calling them delivered is the lie this counter exists to stop: a purchase of two units + // one of which flagged used to print "2 NEW unit(s) delivered" over a single readable unit. + // + // ⚠ It is filled AFTER the waves, from what actually happened, not from the plan. Everything else on + // this struct is a plan-time decision about admission — correct for bounding money, and unable on its + // own to know whether the work it authorised produced a chapter. + Flagged int + // Free is units that rode along at $0 — resumed or re-pinned. Reported because otherwise the operator + // sees a run that touched far more units than it was granted and cannot tell why. + Free int + // LeftFresh is undelivered units still in the book. THIS is what "there is more to buy" honestly + // means, and it is the only one of the two remainders a buyer should ever be invited to purchase. + LeftFresh int + // LeftRework is completed units still carrying a superseded snapshot. They are "unrefreshed", not + // "unbought", and presenting them as stock for sale is how re-payment becomes a product. + LeftRework int +} + +// Paid is every unit this run was charged for, of either kind. +func (v VolumeStop) Paid() int { return v.Delivered + v.Reworked } + +// reconcile corrects the plan-time counters against what the run actually produced. +// +// Admission is a decision made BEFORE the work, and it has to be — that is what bounds the money. But a +// decision to pay for a unit is not a fact about a chapter existing: a unit can be paid for and come back +// flagged with nothing shippable. Reporting the plan as though it were the outcome is how "2 NEW unit(s) +// delivered" ended up printed over one readable unit. So the counters are trued up here, from the +// outcomes, before anyone reads them. +func (s *volumeScope) reconcile(outcomes []ChunkOutcome) { + if s == nil { + return + } + for _, oc := range outcomes { + if oc.Disposition != DispFlagged || oc.FinalText != "" { + continue // it shipped something readable; the plan's word for it stands + } + switch { + case s.stop.Delivered > 0 && s.class[chunkKey{oc.Chapter, oc.ChunkIdx}] == unitFresh: + s.stop.Delivered-- + case s.stop.Reworked > 0: + s.stop.Reworked-- + case s.stop.Delivered > 0: + s.stop.Delivered-- + default: + continue // it was never counted as paid (a free unit that flagged on resume); nothing to move + } + s.stop.Flagged++ + } +} + +// Left is every unit the ceiling held back, of either kind. +func (v VolumeStop) Left() int { return v.LeftFresh + v.LeftRework } + +func (v VolumeStop) String() string { + // ⚠ EVERY COUNT HERE IS IN OUTPUT UNITS, AND THE WORDS MUST SAY SO. An earlier version of this line + // read "%d NEW chapter(s) delivered" over a counter that increments per editUnit — and a chapter is + // more than one unit whenever the cut closes a unit at the edit ceiling, and is many units in a + // draft-only pipeline. That is the unit↔chapter substitution this whole ceiling exists to remove, + // committed by the ceiling's own report. Chapters are the PLATFORM's word; it converts them through + // the manifest. The engine speaks units and nothing else. + s := fmt.Sprintf("stopped on the VOLUME ceiling (--max-units %d), not on money and not at the end of the book: %d paying output unit(s) — %d NEW unit(s) delivered, %d already-delivered unit(s) re-made under a moved snapshot", + v.MaxUnits, v.Paid()+v.Flagged, v.Delivered, v.Reworked) + if v.Flagged > 0 { + s += fmt.Sprintf(", and %d PAID FOR BUT FLAGGED — money spent, no readable text produced (buying more will not fix them; `tmctl redrive` re-attacks a flag)", v.Flagged) + } + if v.Free > 0 { + s += fmt.Sprintf("; %d rode along at $0 (resumed or re-pinned)", v.Free) + } + s += fmt.Sprintf(". Still in the book: %d unit(s) NEVER delivered", v.LeftFresh) + if v.LeftRework > 0 { + s += fmt.Sprintf(" and %d already delivered but not yet re-made (those are unrefreshed, NOT unbought)", v.LeftRework) + } + return s +} + +// volumeScope is the run's admitted set of output units, decided before any work begins. A nil scope +// means no volume ceiling is in force and nothing anywhere changes behaviour. +type volumeScope struct { + admitted map[chunkKey]bool // unit leader key → admitted + leader map[chunkKey]bool // member chunk key → its unit is admitted (flattened for the draft wave) + stop VolumeStop + // class is what each unit was judged to be at PLAN time, kept because the edit wave has to re-judge + // the free ones (see rescopeEditWave). + class map[chunkKey]unitClass + // editSnapshot is the edit-wave snapshot the classification above was made AGAINST. The run can move + // it after planning — the bank-mining stop re-seeds mid-run — so this is what rescopeEditWave compares + // with to know whether the plan is still standing on the ground it was made on. + editSnapshot string + // editBlocked are units whose EDIT must not run even though they were admitted: they were judged free, + // the bank then moved under them, and no grant was left to pay for what they turned out to cost. + editBlocked map[chunkKey]bool +} + +// allows reports whether a DRAFT chunk may be started: it may when the unit it belongs to was admitted. +func (s *volumeScope) allows(ch chunk.Chunk) bool { + if s == nil { + return true + } + return s.leader[chunkKey{ch.Chapter, ch.ChunkIdx}] +} + +// allowsUnit reports whether an EDIT unit may be started. +func (s *volumeScope) allowsUnit(u editUnit) bool { + if s == nil { + return true + } + key := chunkKey{u.Chapter, u.FirstChunkIdx} + return s.admitted[key] && !s.editBlocked[key] +} + +// admittedStatuses narrows stored rows to the units this run is actually going to work on, so a money +// projection over them describes THIS run rather than the whole book. A nil scope is the identity, which +// is what keeps every un-bounded run's figures byte-identical to what they were before the ceiling +// existed. An edit row lives at its unit's LEADER chunk, and a leader is one of the unit's members, so +// the member map covers both waves' rows. +func (s *volumeScope) admittedStatuses(in []store.ChunkStatus) []store.ChunkStatus { + if s == nil { + return in + } + out := in[:0:0] + for _, cs := range in { + if s.leader[chunkKey{cs.Chapter, cs.ChunkIdx}] { + out = append(out, cs) + } + } + return out +} + +// bound reports whether the ceiling actually held work back. A run granted more than the book had left is +// an ordinary complete run and must not be reported as a volume stop — otherwise every generously-bounded +// run would end claiming it had been cut short. +func (s *volumeScope) bound() bool { return s != nil && s.stop.Left() > 0 } + +// planVolume decides, before any wave starts, which output units this run may work on. +// +// It is $0 by construction: store reads, string rendering and hashing, no provider anywhere. It is also +// skipped entirely when no ceiling is set, so a run without --max-units pays nothing for this existing — +// not even the content-hash reproduction, which is the expensive half. +func (r *Runner) planVolume(ctx context.Context, chunks []chunk.Chunk, stickySel []membank.Selection) (*volumeScope, error) { + if r.MaxUnits <= 0 { + return nil, nil + } + // ⚠ REFUSED on a pipeline whose SHIPPING stage is a translator while editor stages also exist, because + // in that one shape the two granularities this ceiling straddles stop being the same object. outputUnits + // keys off finalStageWave and returns a singleton per DRAFT CHUNK there, while the edit wave still + // groups those chunks into buildEditUnits — so a ceiling granted in singletons would admit one member's + // chunk and then let the edit wave run the whole multi-member unit over drafts that were never made, + // paying full price for half-empty input. config validates stage ROLES but imposes no order, so the + // shape loads; refusing loudly is the only answer that neither miscounts a purchase nor silently + // changes what a unit means. (It is also the shape in which the platform's chapters→units conversion + // would stop being exact, which is the whole basis of measuring in units at all.) + if len(r.waveStagesIndexed(waveEdit)) > 0 && r.finalStageWave() == waveDraft { + return nil, fmt.Errorf("pipeline: --max-units cannot bound this pipeline: its last stage is a translator while editor stages exist, so the shipping unit is the draft CHUNK while the edit wave still works in grouped edit UNITS — a ceiling counted in one would admit work measured in the other. Put the shipping stage last (an editor/other role), or run this pipeline without a volume ceiling") + } + // ⚠ THE ONE COMPOSITION THIS CEILING MAKES WORSE, said out loud where an operator will see it. + // + // A bounded purchase drafts only part of the book, and the bank-mining stop consolidates over the + // chunks drafted SO FAR — so the auto-bank it writes grows with every purchase. The next purchase folds + // that larger auto-bank into the ENRICHED bank, which moves the edit-wave snapshot, and the edit jobs + // the PREVIOUS purchase created are then pinned to a superseded one: without --resnapshot the run stops + // loudly, and with it the units whose injected bytes the new terms changed are re-translated and re-paid. + // + // None of that is new machinery — it is the mine D39.165 §3 named and errata 28.08-и narrowed to "an + // edit made AFTER edit jobs exist". What IS new is the frequency: without a volume ceiling that state + // needs an interrupted run (which usually has no edit jobs yet), while WITH one it is the ordinary + // shape of selling a book a few chapters at a time. Warned rather than refused, because the run is + // still correct — it re-pays through the consent gate like any other re-payment — and because refusing + // would take the ceiling away from exactly the books that most need selling in parts. + if r.pack != nil && r.Pipeline.Mining.ContrastPath != "" { + r.Log.WarnContext(ctx, "a VOLUME ceiling on a book that MINES its bank: each purchase drafts more, so the bank-mining stop writes a larger auto-bank, and the next purchase moves the edit-wave snapshot the previous purchase's edit jobs are pinned to. Expect that run to need --resnapshot and to re-pay the units the new terms actually touch (the re-payment consent gate still bounds it)", + "book", r.Book.BookID, "max_units", r.MaxUnits) + } + units := r.outputUnits(chunks) + statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID) + if err != nil { + return nil, fmt.Errorf("pipeline: read chunk_status for the volume ceiling: %w", err) + } + class, curEditAtPlan, err := r.classifyUnits(units, chunks, statuses, stickySel) + if err != nil { + return nil, err + } + s := &volumeScope{ + admitted: make(map[chunkKey]bool, len(units)), + leader: make(map[chunkKey]bool, len(chunks)), + stop: VolumeStop{MaxUnits: r.MaxUnits}, + class: class, + editSnapshot: curEditAtPlan, + editBlocked: map[chunkKey]bool{}, + } + admit := func(u editUnit) { + key := chunkKey{u.Chapter, u.FirstChunkIdx} + s.admitted[key] = true + for _, m := range u.Members { + s.leader[chunkKey{m.Chapter, m.ChunkIdx}] = true + } + } + // ⚠ A MISSING KEY READS AS unitFresh, and that is deliberate rather than incidental: classifyUnits + // returns an EMPTY map for a book with no stored rows, so the zero value has to be the class such a + // book's units actually are. unitFresh is iota 0 for exactly this reason — do not reorder the enum. + classOf := func(u editUnit) unitClass { return class[chunkKey{u.Chapter, u.FirstChunkIdx}] } + + // Free units first and unconditionally: they spend nothing, so holding them back would only leave + // their rows pinned to a superseded snapshot for no saving at all. + for _, u := range units { + if classOf(u) == unitFree { + s.stop.Free++ + admit(u) + } + } + + // ⚠ DELIVERY BEFORE REWORK, and this ORDER is the answer to "what did the buyer actually get". + // + // Both kinds are paying work, so a single pass in book order admits whichever comes first — and after + // any bank edit the already-delivered units ARE the early ones. Measured consequence, on a fixture + // where one chapter is one unit: a purchase of two units on a five-unit book with two delivered + // re-made units 1 and 2 and never started unit 3, reporting Delivered=0, Reworked=2, LeftFresh=3. The + // buyer paid for two units, received none, and three units that had never been translated at all sat + // untouched. A ceiling sold as "the seller's promise and the engine's stop are the same quantity" + // cannot behave that way. + // + // So the grant goes to NEW book while any is left, and only then to re-making. The reason it is safe to + // serve units out of book order is that nothing in a unit's rendered bytes depends on which other units + // ran: the sticky-window selection is precomputed over the WHOLE book before either wave + // (precomputeSticky), so running unit 3 before unit 1 renders exactly what running them in order would. + // + // ⚠ THE TRADE IS NAMED, not hidden: a purchase whose PURPOSE was a re-pass will now spend its grant on + // undelivered units if the book still has any. That is the right default — an undelivered unit is + // worth more to a reader than a re-made one, and the re-payment consent gate bounds rework by money in + // its own right — but it IS a default, and a caller who needs "re-make only" needs a flag that says so. + // Reported to the orchestrator as a chosen default rather than assumed. + for _, pass := range []unitClass{unitFresh, unitRework} { + for _, u := range units { + if classOf(u) != pass { + continue + } + if s.stop.Paid() >= r.MaxUnits { + if pass == unitFresh { + s.stop.LeftFresh++ + } else { + s.stop.LeftRework++ + } + continue + } + if pass == unitFresh { + s.stop.Delivered++ + } else { + s.stop.Reworked++ + } + admit(u) + } + } + return s, nil +} + +// rescopeEditWave re-judges the FREE units against the snapshot the edit wave is actually about to run +// under, and makes them pay for a grant slot if they turn out to cost money. +// +// ⛔ WITHOUT THIS THE CEILING DOES NOT HOLD, and the report lies while it fails. planVolume classifies +// before the draft wave, but the bank-mining stop sits between the two waves and RE-SEEDS the bank +// mid-run (mining.go, the auto-continue branch), after which waverun recomputes the edit-wave snapshot. +// Every unit admitted as FREE — and free units are admitted OUTSIDE the grant, because free work costs +// nothing — was judged against a snapshot the run itself then replaced. Any of them whose injected bytes +// the new bank changes gets a fresh PAID editor call that no grant ever authorised, and the stop line +// calls it "rode along at $0". Found by the acceptance hunter, who measured four provider calls under a +// grant of one and $0.007280 of spend reported as free. +// +// The fix is not a guard but a re-plan: the free/paying question is asked again, against the snapshot +// that is now real, at the last moment before the edit wave begins — so "check before the unit" still +// holds. A unit that has become paying takes a slot if one is left; if none is, its edit does not run at +// all and it is reported as an already-delivered unit still awaiting its re-make, which is what it is. +// +// It is a no-op on the ordinary path: with no mid-run re-seed the snapshot is unchanged and every free +// unit stays free, so a book that does not mine pays nothing for this existing. +func (r *Runner) rescopeEditWave(ctx context.Context, s *volumeScope, units []editUnit, chunks []chunk.Chunk, stickySel []membank.Selection, editSnapshot string) error { + if s == nil || editSnapshot == s.editSnapshot { + return nil // the ground the plan was made on is still the ground the wave runs on + } + statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID) + if err != nil { + return fmt.Errorf("pipeline: re-read chunk_status for the volume ceiling's edit-wave re-plan: %w", err) + } + draftStages, editStages := r.waveStagesIndexed(waveDraft), r.waveStagesIndexed(waveEdit) + draftNames, editNames := stageNameSet(draftStages), stageNameSet(editStages) + curDraft, _, err := r.snapshotIDForWave(waveDraft) + if err != nil { + return fmt.Errorf("pipeline: render the draft-wave snapshot for the edit-wave re-plan: %w", err) + } + // The memo was dropped by the re-seed that moved the snapshot (materializeBanks clears it), so this + // renders against the bank the edit wave will actually use. + hashes := r.cachedRenderedContentHashes(chunks, stickySel) + byChunk := map[chunkKey][]store.ChunkStatus{} + for _, cs := range statuses { + byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}] = append(byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}], cs) + } + + moved, paid, blocked := 0, 0, 0 + for _, u := range units { + key := chunkKey{u.Chapter, u.FirstChunkIdx} + if !s.admitted[key] || s.class[key] != unitFree { + continue // only the units admitted WITHOUT a slot can be wrong about being free + } + rows := unitRows(u, byChunk) + if r.rowsResumeFree(rows, draftNames, editNames, curDraft, editSnapshot, hashes) { + continue // still free under the snapshot that is now real + } + moved++ + s.stop.Free-- + if s.stop.Paid() < s.stop.MaxUnits { + // It costs money now, and there is grant left to pay for it: it becomes what it is — an + // already-delivered unit being re-made. + s.stop.Reworked++ + s.class[key] = unitRework + paid++ + continue + } + // No grant left. Its edit does not run; it stays delivered and stale, which is LeftRework. + s.editBlocked[key] = true + s.stop.LeftRework++ + s.class[key] = unitRework + blocked++ + } + if moved > 0 { + r.Log.WarnContext(ctx, "the bank moved between planning and the edit wave, so units judged FREE are no longer free: they have been re-judged against the snapshot the edit wave actually uses", + "book", r.Book.BookID, "no_longer_free", moved, "took_a_grant_slot", paid, "held_back_for_want_of_grant", blocked, + "planned_against", s.editSnapshot[:12], "running_under", editSnapshot[:12]) + } + s.editSnapshot = editSnapshot + return nil +} + +// unitClass is what this run would DO to an output unit, which is the distinction the ceiling both +// admits and reports on. +type unitClass int + +const ( + // unitFresh — never completed. Paying for it DELIVERS a chapter that did not exist. + unitFresh unitClass = iota + // unitFree — fully recorded and every row resumes or re-pins at $0. Costs nothing, so it rides along + // regardless of the ceiling. + unitFree + // unitRework — fully recorded, but at least one row will be paid for again under a moved snapshot. + // Real work and a real charge, and NOT new book: it replaces a chapter the reader already has. + unitRework +) + +// classifyUnits sorts every output unit into the three things this run can do to it. +// +// The free/paying predicate is deliberately the one the run itself applies, read off runStage's resume +// fast-path: a stored row is served for free when its rendered content hash is unchanged AND its snapshot +// either matches the current one or is a re-pinnable bank-only move. A unit is free when every one of its +// rows is, and when every position it will execute already HAS a row — a half-done unit resumes what it +// has and pays for the rest. +// +// The fresh/rework split falls straight out of the same completeness test, which is why it costs nothing +// to make: a unit that was never fully recorded has never been delivered, so paying for it is delivery; a +// unit that WAS fully recorded has been delivered once already, so paying for it again is rework. +// +// Every uncertainty resolves to "this unit will cost", never to "this unit is free". That direction is +// forced: mistaking a paying unit for a free one lets the run spend past the ceiling, which is the defect +// the ceiling exists to prevent, while the opposite merely makes a run stop one unit early. +func (r *Runner) classifyUnits(units []editUnit, chunks []chunk.Chunk, statuses []store.ChunkStatus, stickySel []membank.Selection) (map[chunkKey]unitClass, string, error) { + class := map[chunkKey]unitClass{} + if len(statuses) == 0 { + return class, "", nil // a book nothing has run is entirely fresh; skip the rendering entirely + } + draftStages, editStages := r.waveStagesIndexed(waveDraft), r.waveStagesIndexed(waveEdit) + draftNames, editNames := stageNameSet(draftStages), stageNameSet(editStages) + + curDraft, _, err := r.snapshotIDForWave(waveDraft) + if err != nil { + return nil, "", fmt.Errorf("pipeline: render the draft-wave snapshot for the volume ceiling: %w", err) + } + curEdit := curDraft + if len(editStages) > 0 { + if curEdit, _, err = r.snapshotIDForWave(waveEdit); err != nil { + return nil, "", fmt.Errorf("pipeline: render the edit-wave snapshot for the volume ceiling: %w", err) + } + } + // The same reproduction the re-payment estimate uses (repin.go): what the run WOULD render for every + // live position. A position whose inputs cannot be reproduced is simply absent, and an absent hash is + // read below as "cannot conclude", i.e. the paying answer. + hashes := r.cachedRenderedContentHashes(chunks, stickySel) + + byChunk := map[chunkKey][]store.ChunkStatus{} + for _, cs := range statuses { + byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}] = append(byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}], cs) + } + for _, u := range units { + rows := unitRows(u, byChunk) + key := chunkKey{u.Chapter, u.FirstChunkIdx} + if !unitFullyRecorded(u, rows, draftStages, editStages) { + class[key] = unitFresh // a position with no row is a position this run will call for + continue + } + if r.rowsResumeFree(rows, draftNames, editNames, curDraft, curEdit, hashes) { + class[key] = unitFree + continue + } + class[key] = unitRework + } + return class, curEdit, nil +} + +// unitFullyRecorded reports whether EVERY position this run would execute for the unit already has a +// stored row — the completeness half of "this unit costs nothing". +// +// ⚠ IT DELIBERATELY DOES NOT USE resolveChunkState, and the difference is money. That resolver answers +// "what does a reader call this unit", and it returns ChunkFlagged the moment ANY row is flagged, before +// it ever tests whether the expected positions are all present. That is right for a progress projection +// and wrong here: a unit whose draft flagged for one member and whose EDIT row does not exist yet — the +// state every book is left in by a bank-signing stop, a Ctrl-C, or any abort in the edit wave — would be +// read as terminal, judged free, admitted without consuming a grant, and then charged for a full editor +// call. The run would pay for more units than were bought and, with Deferred still 0, would not even +// report a volume stop. Found by this pack's own adversarial pass, reproduced before it was fixed. +// +// Positions are counted, not dispositions: a completed unit always leaves a row for every position it +// has — ok, flagged, or the `skipped` rows recordSkippedStages writes downstream of a flag, including the +// all-members-flagged case where the editor never runs. So full coverage IS terminality, and it is the +// property that actually predicts "no provider call". +func unitFullyRecorded(u editUnit, rows []store.ChunkStatus, draftStages, editStages []wavedStage) bool { + type pos struct { + chapter, chunkIdx int + stage string + } + have := make(map[pos]bool, len(rows)) + for _, cs := range rows { + have[pos{cs.Chapter, cs.ChunkIdx, cs.Stage}] = true + } + for _, m := range u.Members { + for _, ws := range draftStages { + if !have[pos{m.Chapter, m.ChunkIdx, ws.st.Name}] { + return false + } + } + } + // The unit's edit rows all live at its LEADER chunk — that is where the read-models look for them. + for _, ws := range editStages { + if !have[pos{u.Chapter, u.FirstChunkIdx, ws.st.Name}] { + return false + } + } + return true +} + +// rowsResumeFree is the per-row half of the predicate above. +func (r *Runner) rowsResumeFree(rows []store.ChunkStatus, draftNames, editNames map[string]bool, + curDraft, curEdit string, hashes map[chunkKey]map[string]string) bool { + for _, cs := range rows { + if cs.Disposition == string(DispSkipped) { + continue // a skipped stage is re-derived from the flag above it; it never reaches a provider + } + var cur string + var w wave + switch { + case draftNames[cs.Stage]: + cur, w = curDraft, waveDraft + case editNames[cs.Stage]: + cur, w = curEdit, waveEdit + default: + continue // a stage this pipeline no longer runs cannot cost anything (rebill.go's rule) + } + h, ok := hashes[chunkKey{cs.Chapter, cs.ChunkIdx}][cs.Stage] + if !ok || h != cs.ContentHash { + return false // the wire bytes moved, or could not be reproduced — either way, a fresh call + } + if cs.SnapshotID != cur && !r.repinnable(cs.SnapshotID, cur, w) { + return false + } + } + return true +} diff --git a/backend/internal/pipeline/volume_midrun_bankmove_test.go b/backend/internal/pipeline/volume_midrun_bankmove_test.go new file mode 100644 index 00000000..f8aa4fee --- /dev/null +++ b/backend/internal/pipeline/volume_midrun_bankmove_test.go @@ -0,0 +1,149 @@ +package pipeline + +import ( + "context" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "textmachine/backend/internal/chunk/chunktest" + "textmachine/backend/internal/obs" +) + +// volume_midrun_bankmove_test.go: the SCENARIO behind rescopeEditWave — a purchaser is charged twice for +// a chapter they already own, because the run moved the bank after deciding what was free. +// +// It is deliberately not the same assertion as the invariant test. That one pins "the re-plan runs before +// the wave"; this one pins "the buyer is not billed for a unit no grant paid for", which is the thing the +// buyer's money is actually exposed to. The two can come apart: a re-plan that ran but judged wrongly, a +// third path that moves the bank, or a later refactor that re-orders the seed — each keeps the invariant +// and breaks this. +// +// THE FIXTURE'S ONE TRICK, and the reason a homogeneous book will not show the defect. The mined delta has +// two sides. The WHICH side (the term list) is mined from the SOURCE of the whole book, so it is complete +// after the first purchase and never moves again. The WHAT side (each term's dst) is drafted-side: it is +// folded from the banknote blocks the translator emitted, so it grows with every purchase. So the source +// here is IDENTICAL in every chapter — the asymmetry is in what the drafts PROPOSE: +// +// - chapters 1-2 emit a banknote naming only 青茅山; +// - chapters 3-4 also propose a dst for 方源, which occurs in every chapter's source, chapters 1-2 +// included. +// +// Purchase 1 buys chapters 1-2, so the auto-bank it writes holds 方源 with NO rendering. Purchase 2 drafts +// chapter 3, whose banknote supplies one — and the re-seed at the bank-mining stop folds it into the +// ENRICHED bank the editor selects over. Chapters 1-2 were admitted as free against the bank as it was at +// planning; their editor injection now carries a term it did not carry, so their content hash moves and +// they cost a fresh paid call. That is production's ordinary shape (mining.go says so where it re-seeds: +// "a term whose drafts disagreed can carry a different proposal than it did before ... that is bank +// CONTENT"), and it needs no heterogeneous source at all. +func TestAMidRunBankMoveNeverBillsBeyondTheGrant(t *testing.T) { + const chapters, grant2 = 4, 1 + rec := &reqRec{} + srv := newJSONProvider(rec, func(body string) (string, string) { + if isEditBody(body) { + return "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД", "stop" + } + // The early chapters' drafts propose no rendering for 方源; the later ones do. + note := bankSeparator + "\n青茅山\tгора Цинмао\tplace" + if !strings.Contains(body, "MARK1") && !strings.Contains(body, "MARK2") { + note = bankSeparator + "\n方源\tФан Юань\tname\n青茅山\tгора Цинмао\tplace" + } + return "Фан Юань пришёл к горе Цинмао.\n" + note, "stop" + }) + defer srv.Close() + bookPath := miningBookOfNChapters(t, srv.URL, chapters) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + // PURCHASE 1: two units. They are delivered, and the auto-bank written at the mining stop holds 方源 + // with no rendering yet. + r1 := newRunner(t, bookPath) + r1.MaxUnits = 2 + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatalf("purchase 1: %v", err) + } + r1.Close() + + // PURCHASE 2: ONE unit. --resnapshot because the previous purchase's edit jobs are pinned to the + // snapshot this run's own re-seed will move — the state planVolume warns about by name. + before := rec.count() + r2 := newRunner(t, bookPath) + defer r2.Close() + r2.MaxUnits = grant2 + r2.Resnapshot = true + res2, err := r2.TranslateBook(ctx) + if err != nil { + t.Fatalf("purchase 2: %v", err) + } + newBodies := rec.all()[before:] + + // (1) THE MONEY. A grant of N units buys at most N drafts and N edits. Anything more is a unit the + // buyer did not pay for being billed to them. + if want := grant2 * 2; len(newBodies) != want { + t.Errorf("a grant of %d output unit(s) made %d provider call(s), want exactly %d (draft+edit each): the surplus is work no grant authorised", + grant2, len(newBodies), want) + } + if maxUSD := float64(grant2*2) * fakeCallUSD; res2.TotalUSD > maxUSD+1e-9 { + t.Errorf("the run spent $%.6f on a grant of %d unit(s); at most $%.6f can be owed", res2.TotalUSD, grant2, maxUSD) + } + + // (2) WHICH unit was billed, named. The already-delivered chapters must not be re-edited at all. + for _, b := range newBodies { + if !isEditBody(b) { + continue + } + for _, owned := range []string{"MARK1", "MARK2"} { + if strings.Contains(b, owned) { + t.Errorf("chapter %s was delivered by the previous purchase and was EDITED again on a grant that never covered it — the buyer paid twice for a chapter they already owned", owned) + } + } + } + + // (3) THE REPORT. A unit that was billed must never be counted among the ones that rode along at $0 — + // that is the line an operator reads to decide the run was cheap. + if res2.Volume == nil { + t.Fatalf("the run stopped short of the book and said nothing about which ceiling did it") + } + if res2.Volume.Free != 0 { + t.Errorf("VolumeStop reports %d unit(s) as riding along at $0, but the bank moved under every one of them: %+v", + res2.Volume.Free, *res2.Volume) + } + if res2.Volume.Paid() > res2.Volume.MaxUnits { + t.Errorf("VolumeStop admits paying for %d unit(s) under a ceiling of %d: %+v", + res2.Volume.Paid(), res2.Volume.MaxUnits, *res2.Volume) + } + t.Logf("purchase 2 (grant %d): %d call(s), $%.6f — %v", grant2, len(newBodies), res2.TotalUSD, res2.Volume) +} + +// miningBookOfNChapters is the mining-stop fixture over N EPUB chapters (setupMiningStopProject is +// single-source, and a volume ceiling needs a book with more than one output unit to bound). The chapter +// bodies are byte-identical apart from a MARK tag the provider stub keys off — see the file comment: +// the defect needs asymmetric BANKNOTES, not an asymmetric source. +func miningBookOfNChapters(t *testing.T, providerURL string, n int) string { + t.Helper() + body := strings.Repeat("方源来到青茅山。方源很强。花家很大。花家的人。", 6) + var eps []chunktest.Chapter + var spine []string + for i := 1; i <= n; i++ { + id := fmt.Sprintf("c%d", i) + eps = append(eps, chunktest.Chapter{ID: id, Href: id + ".xhtml", Body: fmt.Sprintf("

MARK%d %s

", i, body)}) + spine = append(spine, id) + } + bookPath := setupProjectOpts(t, providerURL, projectOpts{ + epub: eps, spine: spine, regenerate: 1, + gatesYAML: "\nmining:\n contrast_path: mining-contrast.txt\ngates:\n banknote:\n enabled: true\n", + }) + dir := filepath.Dir(bookPath) + writeFile(t, filepath.Join(dir, "mining-contrast.txt"), miningContrastData) + packRoot, err := filepath.Abs("../../configs/langpacks") + if err != nil { + t.Fatal(err) + } + raw, err := os.ReadFile(bookPath) + if err != nil { + t.Fatal(err) + } + writeFile(t, bookPath, strings.Replace(string(raw), "source_lang: ja", "source_lang: zh\nlangpack_root: "+packRoot, 1)) + return bookPath +} diff --git a/backend/internal/pipeline/volume_test.go b/backend/internal/pipeline/volume_test.go new file mode 100644 index 00000000..0c25ffa3 --- /dev/null +++ b/backend/internal/pipeline/volume_test.go @@ -0,0 +1,1516 @@ +package pipeline + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "path/filepath" + "reflect" + "strings" + "sync" + "testing" + + "textmachine/backend/internal/chunk/chunktest" + "textmachine/backend/internal/membank" + "textmachine/backend/internal/obs" + "textmachine/backend/internal/store" +) + +// writeFakeCompletion emits the same completion body newJSONProvider does, for the tests that need to +// control the HTTP status per request and therefore cannot use that helper. +func writeFakeCompletion(w http.ResponseWriter, text, finish string) { + tb, _ := json.Marshal(text) + fmt.Fprintf(w, `{"id":"fake","model":"fake-model","choices":[{"message":{"content":%s},"finish_reason":%q}], + "usage":{"prompt_tokens":1000,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":200}}}`, + tb, finish) +} + +// volumeBook builds an n-chapter epub. With the default two-stage pipeline each chapter is one draft +// chunk and therefore exactly one OUTPUT UNIT, so "units" and "chapters" coincide and the assertions +// below read as the purchase does. +func volumeBook(t *testing.T, providerURL string, n int) string { + t.Helper() + var eps []chunktest.Chapter + var spine []string + for i := 1; i <= n; i++ { + id := fmt.Sprintf("c%d", i) + eps = append(eps, chunktest.Chapter{ID: id, Href: id + ".xhtml", Body: fmt.Sprintf("

静かな図書館の朝%d。

", i)}) + spine = append(spine, id) + } + // waveWorkers > 1 on purpose: the admission must hold under the parallel fan-out, not only in a + // sequential walk. A design that counted slots inside the workers would be racy exactly here. + return setupProjectOpts(t, providerURL, projectOpts{epub: eps, spine: spine, regenerate: 1, waveWorkers: 4}) +} + +// chaptersWithRows reports which chapters left any chunk_status row behind — i.e. which units the run +// actually STARTED. It is the instrument for "stopped before N+1 rather than after it": a unit that began +// and was cut off would still have written a row. +func chaptersWithRows(t *testing.T, s *store.Store, bookID string) map[int]bool { + t.Helper() + rows, err := s.ChunkStatusesForBook(bookID) + if err != nil { + t.Fatal(err) + } + out := map[int]bool{} + for _, cs := range rows { + out[cs.Chapter] = true + } + return out +} + +// TestTheVolumeCeilingStopsAtWhatWasBought is the §4.2 post-hoc invariant, and all three of its clauses +// are asserted separately because each can fail on its own: the run stops AT N, it says it stopped on +// VOLUME (in a way a money stop could not be confused with), and it stops BEFORE beginning N+1 rather +// than after paying for it. +// +// The third clause is the one that matters most and the one external systems get wrong: LiteLLM's +// post-hoc token check always overshoots its ceiling by one call, because it looks after the call rather +// than before it. Here the evidence is direct — unit N+1 has no chunk_status row at all, and the provider +// call count is exactly what N units cost and not one call more. +func TestTheVolumeCeilingStopsAtWhatWasBought(t *testing.T) { + const chapters, granted = 5, 2 + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := volumeBook(t, srv.URL, chapters) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r := newRunner(t, bookPath) + defer r.Close() + r.MaxUnits = granted + res, err := r.TranslateBook(ctx) + if err != nil { + t.Fatalf("a volume stop must not be an error — it is a completion: %v", err) + } + + // (1) It stopped AT N. + if len(res.Chunks) != granted { + t.Fatalf("the run shipped %d unit(s), want exactly the %d that were granted", len(res.Chunks), granted) + } + if res.ExitCode() != 0 { + t.Fatalf("exit code %d: a volume stop is a COMPLETION, and the frozen exit dictionary must not gain a word", res.ExitCode()) + } + + // (2) It NAMES the ceiling, distinguishably. + if res.Volume == nil { + t.Fatalf("the run stopped short and said nothing about why — the silent-limit defect this pack exists to avoid") + } + if res.Volume.MaxUnits != granted || res.Volume.Delivered != granted || res.Volume.LeftFresh != chapters-granted { + t.Fatalf("the stop misreports itself: %+v (want max=%d delivered=%d left_fresh=%d)", + *res.Volume, granted, granted, chapters-granted) + } + // Nothing had ever been delivered, so nothing can be REWORK — and saying so is the point of the + // split: a purchase that delivers nothing new must not be reportable as though it did. + if res.Volume.Reworked != 0 || res.Volume.LeftRework != 0 { + t.Fatalf("a book nothing has run has nothing to re-make: %+v", *res.Volume) + } + if res.Volume.Free != 0 { + t.Fatalf("nothing had been run before, so nothing could ride free; got %d", res.Volume.Free) + } + + // (3) It stopped BEFORE beginning N+1, not after. + started := chaptersWithRows(t, r.Store, "test-book") + for ch := 1; ch <= granted; ch++ { + if !started[ch] { + t.Fatalf("chapter %d was granted but never ran: %v", ch, started) + } + } + for ch := granted + 1; ch <= chapters; ch++ { + if started[ch] { + t.Fatalf("chapter %d is past the ceiling and still left a row — the ceiling was checked AFTER the unit began, which is the overshoot-by-one this design exists to prevent", ch) + } + } + // The direct money evidence: two stages per unit, so N units cost exactly 2N calls. One more would + // mean a unit past the ceiling reached a provider. + if want := granted * 2; rec.count() != want { + t.Fatalf("%d provider call(s) for %d granted unit(s); want exactly %d (draft+edit each) — anything more is work past the ceiling", rec.count(), granted, want) + } +} + +// TestAVolumeStopIsNotAMoneyStop pins the distinction the two ceilings owe the operator. They are +// different answers with different remedies: money means the run was cut short mid-work and is resumable +// once there is more of it; volume means the run did precisely what was bought. Reporting them alike +// would either strand a paid-for run as "paused" or show a satisfied buyer a service error. +func TestAVolumeStopIsNotAMoneyStop(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + // The MONEY ceiling: an amount too small for the book's first call. + moneyBook := volumeBook(t, srv.URL, 3) + rm := newRunner(t, moneyBook) + rm.CeilingUSD = 0.0000001 + _, moneyErr := rm.TranslateBook(ctx) + rm.Close() + var halt *CeilingHalt + if !errors.As(moneyErr, &halt) { + t.Fatalf("a money ceiling must still halt through its own typed sentinel, got %v", moneyErr) + } + + // The VOLUME ceiling on an equivalent book: no error at all, and a report that says which ceiling. + volBook := volumeBook(t, srv.URL, 3) + rv := newRunner(t, volBook) + defer rv.Close() + rv.MaxUnits = 1 + res, err := rv.TranslateBook(ctx) + if err != nil { + t.Fatalf("a volume stop must not travel as an error: %v", err) + } + if res.Volume == nil { + t.Fatalf("the volume stop lost its name") + } + // And the money ceiling stayed out of it: nothing about the volume stop touched the ledger's semantics. + if rv.CeilingUSD != 0 { + t.Fatalf("the volume ceiling must not set a money ceiling; CeilingUSD = %v", rv.CeilingUSD) + } +} + +// TestAResumeDoesNotSpendItsVolumeOnFreeUnits is the repin/resume half of the counting rule, executed. +// A second purchase of two units on a book whose first two are already done must deliver units THREE and +// FOUR — not re-walk the first two and deliver nothing. If free work counted, the buyer would pay again +// for what they already own, which is the whole defect the ceiling is meant to close. +func TestAResumeDoesNotSpendItsVolumeOnFreeUnits(t *testing.T) { + const chapters = 5 + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := volumeBook(t, srv.URL, chapters) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r1 := newRunner(t, bookPath) + r1.MaxUnits = 2 + res1, err := r1.TranslateBook(ctx) + if err != nil { + t.Fatal(err) + } + r1.Close() + if len(res1.Chunks) != 2 { + t.Fatalf("first purchase: want 2 units, got %d", len(res1.Chunks)) + } + callsAfterFirst := rec.count() + + r2 := newRunner(t, bookPath) + defer r2.Close() + r2.MaxUnits = 2 + res2, err := r2.TranslateBook(ctx) + if err != nil { + t.Fatal(err) + } + if res2.Volume == nil { + t.Fatalf("the second purchase also stops on the ceiling and must say so") + } + // The two already-owned units rode along for free and did NOT consume the new grant. + if res2.Volume.Free != 2 { + t.Fatalf("%d unit(s) rode free, want the 2 already paid for — a $0 resume must not consume the ceiling", res2.Volume.Free) + } + if res2.Volume.Delivered != 2 || res2.Volume.Reworked != 0 { + t.Fatalf("the second purchase must DELIVER 2 new units and re-make none: %+v", *res2.Volume) + } + if res2.Volume.LeftFresh != 1 || res2.Volume.LeftRework != 0 { + t.Fatalf("1 of the 5 was never delivered and nothing is awaiting a re-make: %+v", *res2.Volume) + } + started := chaptersWithRows(t, r2.Store, "test-book") + for ch := 1; ch <= 4; ch++ { + if !started[ch] { + t.Fatalf("chapter %d should exist after two purchases of two: %v", ch, started) + } + } + if started[5] { + t.Fatalf("chapter 5 was never bought and must not have run") + } + // The second run paid for exactly two units and replayed the first two for nothing. + if fresh := rec.count() - callsAfterFirst; fresh != 4 { + t.Fatalf("the second purchase made %d provider call(s), want 4 (two units × two stages) — the free units must cost none", fresh) + } +} + +// TestRetriesDoNotBurnTheVolumeCeiling is the other half of the counting rule. A unit that stutters and +// is retried is still ONE unit of book. If a retry consumed the ceiling, the buyer of two chapters would +// receive one because the engine's own quality tail — measured swinging between 1.5% and 23% of calls — +// happened to land on their purchase. That tail is a fact about MONEY and is bounded by the money ceiling. +func TestRetriesDoNotBurnTheVolumeCeiling(t *testing.T) { + var mu sync.Mutex + stuttered := false + rec := &reqRec{} + srv := newJSONProvider(rec, func(body string) (string, string) { + if !isEditBody(body) { + mu.Lock() + first := !stuttered + stuttered = true + mu.Unlock() + if first { + // A truncated draft: `length` is in the retryable set, so runStage retries the same + // chunk×stage with a bigger budget and still writes ONE row for it. + return "ЧЕРНОВИК", "length" + } + } + return draftEdit(body) + }) + defer srv.Close() + bookPath := volumeBook(t, srv.URL, 4) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r := newRunner(t, bookPath) + defer r.Close() + r.MaxUnits = 2 + res, err := r.TranslateBook(ctx) + if err != nil { + t.Fatal(err) + } + if !stuttered { + t.Fatalf("the fixture never produced a retry, so it proves nothing") + } + if len(res.Chunks) != 2 { + t.Fatalf("a retried unit is still one unit: want 2 shipped, got %d", len(res.Chunks)) + } + if res.Volume == nil || res.Volume.Delivered != 2 { + t.Fatalf("the retry must not have consumed a unit of the grant: %+v", res.Volume) + } + // The extra provider call is real and paid for — it is simply not a unit of BOOK. + if rec.count() <= 4 { + t.Fatalf("precondition: the retry should have cost an extra call, got %d", rec.count()) + } +} + +// TestNoCeilingChangesNothing pins the default. A run without --max-units must behave exactly as it did +// before this existed — including not paying for the scope computation, which is why planVolume returns a +// nil scope rather than an all-admitting one. +func TestNoCeilingChangesNothing(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := volumeBook(t, srv.URL, 3) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r := newRunner(t, bookPath) + defer r.Close() + scope, err := r.planVolume(context.Background(), nil, nil) + if err != nil || scope != nil { + t.Fatalf("an unset ceiling must plan nothing at all, got scope=%v err=%v", scope, err) + } + res, err := r.TranslateBook(ctx) + if err != nil { + t.Fatal(err) + } + if len(res.Chunks) != 3 { + t.Fatalf("an unbounded run must translate the whole book, got %d units", len(res.Chunks)) + } + if res.Volume != nil { + t.Fatalf("an unbounded run must not claim it was cut short: %+v", res.Volume) + } +} + +// TestAGrantLargerThanTheBookIsAnOrdinaryCompletion pins the boundary the `bound()` guard exists for: a +// run granted more than there is left has not been cut short, and must not report as though it had. +func TestAGrantLargerThanTheBookIsAnOrdinaryCompletion(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := volumeBook(t, srv.URL, 2) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r := newRunner(t, bookPath) + defer r.Close() + r.MaxUnits = 50 + res, err := r.TranslateBook(ctx) + if err != nil { + t.Fatal(err) + } + if len(res.Chunks) != 2 { + t.Fatalf("the whole book should have run, got %d units", len(res.Chunks)) + } + if res.Volume != nil { + t.Fatalf("a grant larger than the book is not a volume stop: %+v", res.Volume) + } +} + +// TestABoundedRunCannotOutrunTheConsentThreshold pins the correction to a defect this pack shipped: the +// consent threshold must not be defeatable by splitting the run. +// +// The shipped version scoped the re-payment AMOUNT to the admitted units and left the THRESHOLD +// book-wide. That reads as symmetrical and is not, because with a volume ceiling the CALLER chooses how +// small each run is: `--resnapshot --max-units 1` in a loop re-pays the whole book while every single run +// stays under the threshold, and consent is never asked once. That is exactly the silent re-purchase Р6 +// exists to prevent, and "the change is inert without the flag" — the argument the narrowing was accepted +// on, mine — is true and beside the point, since with the flag it opens the door the gate IS. +// +// The correction splits the two questions by what each one is ABOUT. "Must a human be asked at all" is +// the book's policy and is judged against the book's whole drift, which a small purchase cannot shrink. +// "--accept-rebill=X", by contrast, is the caller's instruction about SPEND, so it is measured against +// what this run will actually be charged — the platform funds that cap from the run's own hold, and +// measuring it against work the run will not do would refuse a caller whose money fully covers the bill. +func TestABoundedRunCannotOutrunTheConsentThreshold(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + var eps []chunktest.Chapter + var spine []string + for i := 1; i <= 4; i++ { + id := fmt.Sprintf("c%d", i) + // 静か occurs in every chapter, so the signed term changes EVERY unit's injected bytes: all four + // units genuinely re-pay rather than re-pin, which is what makes the arithmetic below discriminate. + eps = append(eps, chunktest.Chapter{ID: id, Href: id + ".xhtml", Body: fmt.Sprintf("

静かな図書館の朝%d。

", i)}) + spine = append(spine, id) + } + // One unit re-pays one edit row ≈ $0.00182; four units ≈ $0.00728. The threshold sits between them, + // which is precisely the gap a splitting caller would walk through. + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + epub: eps, spine: spine, regenerate: 1, waveWorkers: 4, rebillConsentUSD: 0.005, + }) + dir := filepath.Dir(bookPath) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatalf("the baseline run must complete: %v", err) + } + r1.Close() + doc := decisionsDocFor(t, dir, "test-book", membank.Decision{ + Action: membank.ActionApprove, Src: "静か", Dst: "тихий"}) + if _, err := ApplyBankDecisions(ctx, bookPath, doc, false); err != nil { + t.Fatalf("bank-apply: %v", err) + } + + // THE ATTACK: a purchase small enough that its own slice is far under the threshold. It must still be + // stopped, because the BOOK's drift is over it. + callsBefore := rec.count() + rA := newRunner(t, bookPath) + rA.Resnapshot = true + rA.MaxUnits = 1 + _, errA := rA.TranslateBook(ctx) + rA.Close() + if errA == nil { + t.Fatalf("a run bounded to one unit walked past the consent gate; repeated four times it re-buys the whole book without ever asking") + } + if !strings.Contains(errA.Error(), "RE-PAY") { + t.Fatalf("want the re-payment gate's refusal, got: %v", errA) + } + // The refusal has to show BOTH numbers, or it teaches the operator the wrong thing: the book's total + // is what the threshold judged, and the run's slice is what they would actually be charged. + if !strings.Contains(errA.Error(), "THIS run, bounded by --max-units") { + t.Fatalf("the refusal must separate the book's drift from this run's slice, got: %v", errA) + } + if fresh := rec.count() - callsBefore; fresh != 0 { + t.Fatalf("the gate refuses BEFORE any reservation; %d call(s) were made", fresh) + } + + // AND THE OTHER DIRECTION, which is why the narrowing was attempted at all: a NAMED cap is an + // instruction about spend, so it is measured against what THIS run re-pays (~$0.00182) and not against + // the book's $0.00728. A caller whose money covers their own bill is not refused. + rB := newRunner(t, bookPath) + defer rB.Close() + rB.Resnapshot = true + rB.MaxUnits = 1 + rB.AcceptRebill = RebillConsent{Given: true, Capped: true, CapUSD: 0.002} + resB, errB := rB.TranslateBook(ctx) + if errB != nil { + t.Fatalf("a cap of $0.002 covers this run's own re-payment of ~$0.00182 and must be honoured; measuring it against the whole book would refuse a caller who can pay: %v", errB) + } + // ⚠ AND THE SPLIT EARNS ITSELF HERE. Every unit of this book was delivered by the baseline run, so + // this purchase delivers NOTHING new — it re-makes one chapter the reader already has. Reported as a + // single "granted" count it would have read exactly like a delivery of one chapter. + if resB.Volume == nil || resB.Volume.Reworked != 1 || resB.Volume.Delivered != 0 { + t.Fatalf("this purchase re-makes an already-delivered unit and delivers nothing new: %+v", resB.Volume) + } + if resB.Volume.LeftFresh != 0 || resB.Volume.LeftRework != 3 { + t.Fatalf("nothing is left UNDELIVERED here; 3 delivered units await a re-make: %+v", resB.Volume) + } + if fresh := rec.count() - callsBefore; fresh != 1 { + t.Fatalf("the consented run re-pays one unit = one edit call; it made %d", fresh) + } +} + +// TestTheVolumeCeilingHoldsOnADraftOnlyPipeline pins the ceiling on the OTHER pipeline shape. It matters +// because the output unit is not the same object in the two: an editor pipeline groups draft chunks into +// coarse edit units, while a draft-only pipeline ships one unit per draft chunk (outputUnits). A ceiling +// that only held for the shape the fixtures happen to use would silently sell a different amount of book +// on a deployment configured the other way. +func TestTheVolumeCeilingHoldsOnADraftOnlyPipeline(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + var eps []chunktest.Chapter + var spine []string + for i := 1; i <= 4; i++ { + id := fmt.Sprintf("c%d", i) + eps = append(eps, chunktest.Chapter{ID: id, Href: id + ".xhtml", Body: fmt.Sprintf("

静かな図書館の朝%d。

", i)}) + spine = append(spine, id) + } + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + epub: eps, spine: spine, regenerate: 1, waveWorkers: 4, draftOnly: true, + }) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r := newRunner(t, bookPath) + defer r.Close() + r.MaxUnits = 2 + res, err := r.TranslateBook(ctx) + if err != nil { + t.Fatal(err) + } + if len(res.Chunks) != 2 { + t.Fatalf("a draft-only run granted 2 units must ship 2, got %d", len(res.Chunks)) + } + if res.Volume == nil || res.Volume.Delivered != 2 || res.Volume.LeftFresh != 2 { + t.Fatalf("the stop misreports itself on a draft-only pipeline: %+v", res.Volume) + } + // One stage per unit here, so two units cost exactly two calls — and nothing past the ceiling ran. + if rec.count() != 2 { + t.Fatalf("%d provider call(s); a draft-only unit is one call, so 2 granted units cost 2", rec.count()) + } + started := chaptersWithRows(t, r.Store, "test-book") + if started[3] || started[4] { + t.Fatalf("chapters past the ceiling must not have started: %v", started) + } +} + +// TestAnUnreproducibleHashIsNeverFree pins the DIRECTIONAL invariant volume.go calls load-bearing: +// "every uncertainty resolves to «this unit will cost», never to «this unit is free»". The asymmetry is +// the whole safety of the ceiling — judging a paying unit free lets the run spend PAST what was bought, +// while the opposite merely stops a run one unit early. +// +// The dangerous branch is the one where the wire bytes cannot be REPRODUCED at all (an absent hash), +// which is a different state from "reproduced and different". renderedContentHashes omits a position +// whose inputs it cannot rebuild — a member's stored draft text gone, a chunk boundary moved by a +// re-split — and an absent entry must read as "cannot conclude", i.e. paying. A condition written +// `ok && h != cs.ContentHash` passes the whole battery while inverting exactly this case. +func TestAnUnreproducibleHashIsNeverFree(t *testing.T) { + r := &Runner{} + draftNames := map[string]bool{"draft": true} + editNames := map[string]bool{"edit": true} + const snap = "SNAP" + row := store.ChunkStatus{Chapter: 1, ChunkIdx: 0, Stage: "draft", + Disposition: string(DispOK), ContentHash: "h1", SnapshotID: snap} + rows := []store.ChunkStatus{row} + present := map[chunkKey]map[string]string{{1, 0}: {"draft": "h1"}} + + // Baseline: reproduced AND equal, on the current snapshot — genuinely free. + if !r.rowsResumeFree(rows, draftNames, editNames, snap, snap, present) { + t.Fatalf("a row whose bytes reproduce identically on the current snapshot resumes for $0") + } + // Reproduced and DIFFERENT — pays. + differs := map[chunkKey]map[string]string{{1, 0}: {"draft": "OTHER"}} + if r.rowsResumeFree(rows, draftNames, editNames, snap, snap, differs) { + t.Fatalf("the wire bytes moved; the unit will be called for and must not be judged free") + } + // NOT REPRODUCED AT ALL — the branch this test exists for. The stage key is missing. + noStage := map[chunkKey]map[string]string{{1, 0}: {}} + if r.rowsResumeFree(rows, draftNames, editNames, snap, snap, noStage) { + t.Fatalf("the bytes could not be reproduced, so nothing is known — judging the unit FREE lets the run spend past the ceiling") + } + // NOT REPRODUCED AT ALL — the position itself is absent. + if r.rowsResumeFree(rows, draftNames, editNames, snap, snap, map[chunkKey]map[string]string{}) { + t.Fatalf("an absent position must read as «cannot conclude», i.e. paying") + } + // A `skipped` row never reached a provider and never will: it is re-derived from the flag above it. + skipped := []store.ChunkStatus{{Chapter: 1, ChunkIdx: 0, Stage: "edit", + Disposition: string(DispSkipped), ContentHash: "whatever", SnapshotID: "STALE"}} + if !r.rowsResumeFree(skipped, draftNames, editNames, snap, snap, map[chunkKey]map[string]string{}) { + t.Fatalf("a skipped stage costs nothing even with no hash and a stale snapshot") + } + // A stage this pipeline no longer runs cannot cost anything either. + retired := []store.ChunkStatus{{Chapter: 1, ChunkIdx: 0, Stage: "gone", + Disposition: string(DispOK), ContentHash: "x", SnapshotID: "STALE"}} + if !r.rowsResumeFree(retired, draftNames, editNames, snap, snap, map[chunkKey]map[string]string{}) { + t.Fatalf("a retired stage is never run, so it is never a cost") + } +} + +// TestTheCeilingAdmitsEveryMemberOfAUnit closes the coverage hole every other fixture in this package +// leaves open: they all build one-sentence chapters, so every editUnit has exactly ONE member and the +// per-member loop in planVolume's admit() is never exercised. On a real book a chapter is longer than one +// draft chunk, and that is the shape volume.go's own comment describes — "an editor pipeline groups draft +// chunks into coarse edit units". +// +// What the hole hides is not a cosmetic slip. Admission is asked TWICE with two different keys: the draft +// wave asks per member chunk (scope.allows) and the edit wave asks per unit (scope.allowsUnit). If admit() +// marked only the unit's leader, the non-leader members would never be drafted while the unit still read +// as admitted — so the edit wave would run over a unit whose drafts are missing, and the buyer would +// receive a truncated chapter they had paid for in full, silently. +func TestTheCeilingAdmitsEveryMemberOfAUnit(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + // A fine DRAFT cut under a coarse EDIT ceiling is what produces multi-member units: each chapter + // splits into several draft chunks that regroup into one edit unit. + const seg = "\nsegmentation:\n draft_budget_out: 24\n edit_ceiling_out: 8000\n fertility: { cjk: 1.1978, other: 0.3852 }\n" + body := "

静かな図書館の朝。鈴木は本を読んだ。外では雨が降っていた。彼は窓を見た。時間は静かに過ぎた。

" + var eps []chunktest.Chapter + var spine []string + for i := 1; i <= 3; i++ { + id := fmt.Sprintf("c%d", i) + eps = append(eps, chunktest.Chapter{ID: id, Href: id + ".xhtml", Body: body}) + spine = append(spine, id) + } + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + epub: eps, spine: spine, regenerate: 1, waveWorkers: 4, gatesYAML: seg, + }) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r := newRunner(t, bookPath) + defer r.Close() + r.MaxUnits = 1 + res, err := r.TranslateBook(ctx) + if err != nil { + t.Fatal(err) + } + if res.Volume == nil || res.Volume.Delivered != 1 { + t.Fatalf("one unit was granted and delivered: %+v", res.Volume) + } + + // The fixture must actually produce a MULTI-member unit, or this test proves nothing about the loop. + rows, err := r.Store.ChunkStatusesForBook("test-book") + if err != nil { + t.Fatal(err) + } + drafted := map[int]map[int]bool{} + for _, cs := range rows { + if cs.Stage != "draft" { + continue + } + if drafted[cs.Chapter] == nil { + drafted[cs.Chapter] = map[int]bool{} + } + drafted[cs.Chapter][cs.ChunkIdx] = true + } + // How many chunks the CUT produced for chapter 1 — read from the manifest, which is persisted before + // the waves and is therefore unaffected by the ceiling. Counting drafted rows instead would let a + // defect that skips members disguise itself as a thin fixture. + manifestChunks, _, err := r.readModelChunks() + if err != nil { + t.Fatal(err) + } + var ch1 []int + for _, c := range manifestChunks { + if c.Chapter == 1 { + ch1 = append(ch1, c.ChunkIdx) + } + } + if len(ch1) < 2 { + t.Fatalf("fixture is degenerate: the cut gave chapter 1 %d draft chunk(s), so the multi-member path is untested", len(ch1)) + } + // EVERY member of the admitted unit was drafted — not just its leader. + for _, idx := range ch1 { + if !drafted[1][idx] { + t.Fatalf("chapter 1 has %d draft chunks in the manifest but member chunk %d was never drafted: the unit was admitted for the edit wave while part of its draft was skipped, so the buyer gets a truncated chapter they paid for in full", len(ch1), idx) + } + } + // And the unit's edit really ran over the whole thing: one shipped unit, no dropped members. + if len(res.Chunks) != 1 { + t.Fatalf("one admitted unit ships one output unit, got %d", len(res.Chunks)) + } + if res.Chunks[0].DroppedMembers != 0 { + t.Fatalf("the edit lost %d member(s) of a unit that was paid for in full: %+v", res.Chunks[0].DroppedMembers, res.Chunks[0]) + } + // Nothing outside the granted unit began. + if len(drafted[2]) != 0 || len(drafted[3]) != 0 { + t.Fatalf("chapters past the ceiling started anyway: ch2=%d ch3=%d chunks", len(drafted[2]), len(drafted[3])) + } +} + +// TestAFlaggedMemberWithNoEditRowIsNotFree reproduces an OVERSPEND this pack shipped and then fixed, and +// it is the most dangerous class the ceiling can have: a unit wrongly judged free is admitted WITHOUT +// consuming a grant and then charged for anyway, so the run pays for more units than were bought — and +// because nothing was deferred, it does not even report a volume stop to say so. +// +// The state that produces it is ordinary, not exotic: a unit whose draft flagged for one member and whose +// EDIT row does not exist yet. Every book left by a bank-signing stop, a Ctrl-C, or any abort inside the +// edit wave is in it. The first version of unitsServedFree asked resolveChunkState whether the unit was +// terminal, and that resolver answers ChunkFlagged as soon as ANY row is flagged — before it ever checks +// that the expected positions are present. +func TestAFlaggedMemberWithNoEditRowIsNotFree(t *testing.T) { + var mu sync.Mutex + editWorks := false + rec := &reqRec{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { + body, _ := io.ReadAll(req.Body) + rec.record(string(body)) + s := string(body) + if isEditBody(s) { + mu.Lock() + ok := editWorks + mu.Unlock() + if !ok { + // Run 1's editor never completes, so no edit chunk_status row is ever written — exactly + // the store state an interrupted edit wave leaves. + w.WriteHeader(http.StatusInternalServerError) + return + } + writeFakeCompletion(w, "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД", "stop") + return + } + if strings.Contains(s, "壊れた時計") { + writeFakeCompletion(w, "", "stop") // an empty draft flags the member + return + } + writeFakeCompletion(w, "ЧЕРНОВИК ПЕРЕВОДА", "stop") + })) + defer srv.Close() + + const seg = "\nsegmentation:\n draft_budget_out: 24\n edit_ceiling_out: 8000\n fertility: { cjk: 1.1978, other: 0.3852 }\n" + body := "

静かな図書館の朝。鈴木は本を読んだ。壊れた時計があった。外では雨が降っていた。彼は窓を見た。

" + clean := "

静かな図書館の朝。鈴木は本を読んだ。外では雨が降っていた。彼は窓を見た。時間は過ぎた。

" + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + epub: []chunktest.Chapter{{ID: "c1", Href: "c1.xhtml", Body: body}, {ID: "c2", Href: "c2.xhtml", Body: clean}}, + spine: []string{"c1", "c2"}, regenerate: 0, waveWorkers: 1, gatesYAML: seg, + }) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + // Run 1: drafts land (one member flagged empty), the editor dies, so NO edit row is written anywhere. + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err == nil { + t.Fatalf("precondition: the editor was supposed to fail this run") + } + rows, err := r1.Store.ChunkStatusesForBook("test-book") + if err != nil { + t.Fatal(err) + } + r1.Close() + flagged, edits := 0, 0 + for _, cs := range rows { + if cs.Stage == "edit" { + edits++ + } + if cs.Disposition == string(DispFlagged) { + flagged++ + } + } + if flagged == 0 || edits != 0 { + t.Fatalf("precondition: want a flagged draft member and NO edit row, got flagged=%d edit rows=%d", flagged, edits) + } + + // Run 2, granted ONE unit. The unit above is not free — its edit has never run — so it must consume + // the grant, and the second unit must be deferred rather than also paid for. + mu.Lock() + editWorks = true + mu.Unlock() + callsBefore := rec.count() + r2 := newRunner(t, bookPath) + defer r2.Close() + r2.MaxUnits = 1 + res, err := r2.TranslateBook(ctx) + if err != nil { + t.Fatal(err) + } + if res.Volume == nil { + t.Fatalf("one of the two units had to be deferred, so the run must report a volume stop; it reported none — the other unit was judged free and paid for anyway") + } + // The unit was never fully recorded, so it has never been DELIVERED — paying for it is delivery, not + // rework, and the split must say so. + if res.Volume.Delivered != 1 || res.Volume.Reworked != 0 || res.Volume.Free != 0 || res.Volume.LeftFresh != 1 { + t.Fatalf("a unit with no edit row is neither free nor rework — it has never been delivered: %+v", *res.Volume) + } + if len(res.Chunks) != 1 { + t.Fatalf("OVERSPEND: %d output unit(s) were paid for under --max-units 1", len(res.Chunks)) + } + // One editor call for the one granted unit, and nothing for the deferred one. + if fresh := rec.count() - callsBefore; fresh != 1 { + t.Fatalf("the granted unit needed one edit call; the run made %d", fresh) + } +} + +// TestTheCeilingRefusesAPipelineWhoseUnitsAreTwoThings pins the refusal for the one pipeline shape in +// which "output unit" stops naming a single object. +// +// config validates stage ROLES but imposes no ORDER, so [draft(translator), edit(editor), +// polish(translator)] loads. finalStageWave() then answers waveDraft — the last role is a translator — +// so outputUnits returns one singleton per DRAFT CHUNK, while the edit wave still groups those chunks +// through buildEditUnits. A ceiling granted in singletons would admit one member's chunk and then let the +// edit wave run the whole multi-member unit anyway, feeding the editor a unit whose source is complete +// and whose draft is missing a member — a full-price call over half-made input, charged as the one unit +// that was bought. It would also make the platform's chapters→units conversion stop being exact, which is +// the entire reason for measuring in units. Refused loudly rather than silently miscounted. +func TestTheCeilingRefusesAPipelineWhoseUnitsAreTwoThings(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + // gatesYAML is appended immediately after the stage list, so this continues it with a THIRD stage + // whose role is a translator — putting the shipping stage back in the draft wave. + const polish = " - { name: polish, role: translator, model: fake-model, prompt_override: prompts/translator.md, prompt_version: v-test, temperature: 0.3, reasoning: \"off\" }\n" + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + regenerate: 1, source: suzukiSource, gatesYAML: polish, + }) + r := newRunner(t, bookPath) + defer r.Close() + + // Precondition: the fixture really is the incoherent shape, or the test proves nothing. + if r.finalStageWave() != waveDraft || len(r.waveStagesIndexed(waveEdit)) == 0 { + t.Fatalf("fixture drifted: want a translator-last pipeline that still has editor stages, got finalWave=%v editStages=%d", + r.finalStageWave(), len(r.waveStagesIndexed(waveEdit))) + } + + // With no ceiling the shape is none of this pack's business and must be left exactly as it was. + r.MaxUnits = 0 + if scope, err := r.planVolume(context.Background(), nil, nil); err != nil || scope != nil { + t.Fatalf("an unbounded run must be untouched by this refusal: scope=%v err=%v", scope, err) + } + + // With one, the ceiling cannot mean one thing, so it refuses. + r.MaxUnits = 1 + _, err := r.planVolume(context.Background(), nil, nil) + if err == nil { + t.Fatalf("a ceiling counted in draft chunks while the edit wave works in grouped units must refuse, not guess") + } + if !strings.Contains(err.Error(), "cannot bound this pipeline") { + t.Fatalf("the refusal must say which shape it is turning away, got: %v", err) + } +} + +// TestAFullyDeliveredBookOffersNothingLeftToBuy pins the defect that made re-payment look like a product. +// +// The trajectory: a book translated to 100%, then a bank edit. Every unit is now "not free", so the old +// single counter admitted them, called them granted, and reported the rest as "remain in the book" — which +// the CLI presented as what is left to BUY. Purchase one said 29 remained, purchase two said 28, and not +// one new chapter existed at any point. exit 0 and `ready` both agreed it had gone well. +// +// The counters must now say the true thing: nothing was DELIVERED, N units were RE-MADE, and the number +// of never-delivered units is ZERO — so there is nothing to invite anyone to buy. +func TestAFullyDeliveredBookOffersNothingLeftToBuy(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := volumeBook(t, srv.URL, 4) + dir := filepath.Dir(bookPath) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + r1.Close() + // Every chapter contains 静か, so signing it moves every unit's injected bytes: the whole book becomes + // re-payable, and none of it becomes new. + doc := decisionsDocFor(t, dir, "test-book", membank.Decision{ + Action: membank.ActionApprove, Src: "静か", Dst: "тихий"}) + if _, err := ApplyBankDecisions(ctx, bookPath, doc, false); err != nil { + t.Fatal(err) + } + + r2 := newRunner(t, bookPath) + defer r2.Close() + r2.Resnapshot = true + r2.MaxUnits = 1 + r2.AcceptRebill = RebillConsent{Given: true} // the operator consented; that is not what is under test + res, err := r2.TranslateBook(ctx) + if err != nil { + t.Fatal(err) + } + v := res.Volume + if v == nil { + t.Fatalf("three units were held back, so the run must report a volume stop") + } + if v.Delivered != 0 { + t.Fatalf("this book was already complete: no purchase over it can DELIVER anything, yet %d was reported", v.Delivered) + } + if v.Reworked != 1 { + t.Fatalf("one already-delivered unit was re-made; got %d", v.Reworked) + } + if v.LeftFresh != 0 { + t.Fatalf("⚠ %d unit(s) reported as never-delivered on a fully translated book — this is the number the CLI turns into «buy more», and it would be selling chapters the reader already owns", v.LeftFresh) + } + if v.LeftRework != 3 { + t.Fatalf("three delivered units still await a re-make; got %d", v.LeftRework) + } + // And the sentence the stop carries has to say it in words, not only in fields. (The operator-facing + // invitation is the CLI's own line and is pinned in cmd/tmctl/render_test.go, where it lives.) + line := v.String() + if !strings.Contains(line, "0 NEW unit(s) delivered") { + t.Fatalf("the stop must state that nothing new was delivered: %s", line) + } + if !strings.Contains(line, "0 unit(s) NEVER delivered") { + t.Fatalf("the stop must state that nothing is left undelivered: %s", line) + } + if !strings.Contains(line, "unrefreshed, NOT unbought") { + t.Fatalf("the remainder must be named for what it is: %s", line) + } +} + +// TestARunThatRePaysNothingIsNotAskedForConsent pins the early return the consent gate needs in order not +// to refuse legitimate work — a branch a mutation showed nothing was holding. +// +// The gate judges its THRESHOLD against the whole book, precisely so a caller cannot shrink it by +// splitting the run. That is right, and it has an edge: a bounded run whose granted units are all FRESH +// re-pays nothing at all, and demanding consent from it would be asking a caller to approve a spend that +// is not going to happen — the other half of Р6, and the failure this pack was already caught committing +// once in the opposite direction. +// +// The state is built rather than waited for: the book is translated whole, the first two units' rows are +// then reset (which is exactly what a redrive does to a unit), and a term that occurs only in the LATER +// chapters is signed. That leaves units 1–2 never-delivered and units 3–4 delivered-and-superseded, so a +// purchase of two units is granted only fresh work while the book's drift sits over the threshold. +func TestARunThatRePaysNothingIsNotAskedForConsent(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + var eps []chunktest.Chapter + var spine []string + for i := 1; i <= 4; i++ { + id := fmt.Sprintf("c%d", i) + body := fmt.Sprintf("

朝%d。鈴木は歩いた。

", i) + if i >= 3 { + // 静か occurs ONLY in the later chapters, so signing it supersedes those units and leaves the + // earlier ones alone. + body = fmt.Sprintf("

静かな朝%d。鈴木は歩いた。

", i) + } + eps = append(eps, chunktest.Chapter{ID: id, Href: id + ".xhtml", Body: body}) + spine = append(spine, id) + } + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + epub: eps, spine: spine, regenerate: 1, waveWorkers: 1, rebillConsentUSD: 0.002, + }) + dir := filepath.Dir(bookPath) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + // Un-deliver the first two units, the way a redrive does. + for ch := 1; ch <= 2; ch++ { + if err := r1.Store.ResetChunkStages("test-book", ch, 0, []string{"draft", "edit"}); err != nil { + t.Fatal(err) + } + } + r1.Close() + doc := decisionsDocFor(t, dir, "test-book", membank.Decision{ + Action: membank.ActionApprove, Src: "静か", Dst: "тихий"}) + if _, err := ApplyBankDecisions(ctx, bookPath, doc, false); err != nil { + t.Fatal(err) + } + + // Precondition: the book's drift really is over the threshold, so an UNBOUNDED run would be refused. + rGate := newRunner(t, bookPath) + rGate.Resnapshot = true + _, gateErr := rGate.TranslateBook(ctx) + rGate.Close() + if gateErr == nil || !strings.Contains(gateErr.Error(), "RE-PAY") { + t.Fatalf("precondition: the book's drift must exceed the threshold, got %v", gateErr) + } + + // The bounded run is granted the two FRESH units only, so it re-pays nothing and must not be stopped. + r2 := newRunner(t, bookPath) + defer r2.Close() + r2.Resnapshot = true + r2.MaxUnits = 2 + res, err := r2.TranslateBook(ctx) + if err != nil { + t.Fatalf("this run re-pays nothing — every unit it was granted is new work — so there is no concrete spend to consent to, and refusing it denies legitimate work: %v", err) + } + if res.Volume == nil || res.Volume.Delivered != 2 || res.Volume.Reworked != 0 { + t.Fatalf("the grant should have gone entirely to fresh delivery: %+v", res.Volume) + } + if res.Volume.LeftRework != 2 { + t.Fatalf("the two superseded units are still waiting to be re-made: %+v", res.Volume) + } +} + +// TestTheWireHashMemoDiesWithItsBank pins the SAFETY half of the reproduction memo. The memo itself is +// only speed — a bounded run used to redo the full-book re-render up to three times (once in planVolume, +// once per projection in the consent gate) with nothing changed between them. But every hash in it is +// rendered against the materialized bank, so a memo that outlived its bank would hand a caller hashes for +// a bank the run no longer has, and the free estimate would quote a number the paid run does not charge — +// the exact class of silent wrongness this pack exists to remove. Its lifetime must therefore be the +// bank's, and materializeBanks is the one place a bank changes (a mid-run re-seed included). +func TestTheWireHashMemoDiesWithItsBank(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + regenerate: 1, source: suzukiSource, glossarySeed: suzukiSeed, + }) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + r := newRunner(t, bookPath) + defer r.Close() + if _, err := r.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + + chunks, withText, err := r.readModelChunks() + if err != nil { + t.Fatal(err) + } + _ = chunks + full, err := withText() + if err != nil { + t.Fatal(err) + } + sel := precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget) + + first := r.cachedRenderedContentHashes(full, sel) + if len(first) == 0 { + t.Fatalf("the fixture produced no positions, so the memo is untested") + } + // Same bank → the very SAME map, not merely an equal one. Identity is the assertion that separates a + // memo from a function that happens to be deterministic: an equal-value check would pass on code that + // re-renders the whole book every call, which is exactly the cost this exists to remove. + again := r.cachedRenderedContentHashes(full, sel) + if reflect.ValueOf(again).Pointer() != reflect.ValueOf(first).Pointer() { + t.Fatalf("the second call built a NEW map: the reproduction is being redone, not reused") + } + if !sameMap(again, first) { + t.Fatalf("the memo changed under the same bank") + } + + // A new materialization must throw it away. Anything else serves hashes belonging to a bank that is + // no longer the run's. + if err := r.projectFoldedMemory(); err != nil { + t.Fatal(err) + } + if r.contentHashes != nil { + t.Fatalf("the memo survived a re-materialization of the bank: its hashes are now about a bank this run does not have") + } +} + +// sameMap reports whether two hash maps hold the same positions and values. +func sameMap(a, b map[chunkKey]map[string]string) bool { + if len(a) != len(b) { + return false + } + for k, av := range a { + bv, ok := b[k] + if !ok || len(av) != len(bv) { + return false + } + for stage, h := range av { + if bv[stage] != h { + return false + } + } + } + return true +} + +// TestAVolumeCeilingOnAMiningBookWarnsAboutTheNextPurchase pins the one warning this pack owes an +// operator, on the one composition it makes worse. +// +// A bounded purchase drafts only part of the book, so the bank-mining stop consolidates over less than +// the whole and writes a SMALLER auto-bank than the next purchase will. The next purchase folds the +// larger one, which moves the edit-wave snapshot the previous purchase's edit jobs are pinned to — and +// the job guard treats that exactly like a config change, so the run stops loudly without --resnapshot. +// None of that is new machinery; what IS new is that a volume ceiling turns a rare interrupted-run corner +// into the ordinary shape of selling a book in parts. The engine cannot fix the guard (ratified Р6 +// contour, another pack's subject), so the least it owes is to say so before it happens — and a warning +// nothing tests is a warning that can silently stop being emitted. +func TestAVolumeCeilingOnAMiningBookWarnsAboutTheNextPurchase(t *testing.T) { + rec := &reqRec{} + // A plain provider, not the mining-stop one: this test is about the PLANNING warning, which is + // emitted before either wave, so the stop behaviour is beside the point and its assertions would only + // add an unrelated way to fail. + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupMiningStopProject(t, srv.URL, miningStopOpts{}) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + var logBuf bytes.Buffer + r := newRunner(t, bookPath) + defer r.Close() + r.Log = slog.New(slog.NewTextHandler(&logBuf, &slog.HandlerOptions{Level: slog.LevelWarn})) + r.MaxUnits = 1 + + // The warning is emitted while PLANNING, before any wave, so the run's own outcome is beside the point + // here — what matters is that the operator was told before the money moved. + if _, err := r.TranslateBook(ctx); err != nil { + t.Logf("run ended with %v (not what this test is about)", err) + } + if !strings.Contains(logBuf.String(), "book that MINES its bank") { + t.Fatalf("a volume ceiling on a mining-configured book must warn that the NEXT purchase moves the edit snapshot and will need --resnapshot; log was:\n%s", logBuf.String()) + } + if !strings.Contains(logBuf.String(), "--resnapshot") { + t.Fatalf("the warning must name the remedy, not just the problem:\n%s", logBuf.String()) + } + + // And it must NOT fire on a mining book with no ceiling — otherwise every ordinary run of every + // mining book carries a warning about a situation it is not in. + var quiet bytes.Buffer + r2 := newRunner(t, setupMiningStopProject(t, srv.URL, miningStopOpts{})) + defer r2.Close() + r2.Log = slog.New(slog.NewTextHandler(&quiet, &slog.HandlerOptions{Level: slog.LevelWarn})) + if _, err := r2.TranslateBook(ctx); err != nil { + t.Logf("unbounded run ended with %v", err) + } + if strings.Contains(quiet.String(), "book that MINES its bank") { + t.Fatalf("an unbounded run is not in this situation and must not be warned about it:\n%s", quiet.String()) + } +} + +// TestAConsentedSliceIsAlwaysToldWhatTheBookCarries pins the property that makes the capped-consent path +// consent rather than a hole — a distinction settled by ratification rather than by argument. +// +// A caller CAN re-pay a whole book a slice at a time with `--resnapshot --max-units 1 --accept-rebill=X` +// looped: every run stays under its own named cap and the book-wide threshold never refuses. That is not +// a bypass — it reaches no total a single bare `--accept-rebill` would not reach in one invocation, and +// each run names an amount and is charged exactly it. What makes it consent rather than a bypass is that +// the caller is TOLD, on every run, that the book carries drift beyond its threshold and is being bought +// down a slice at a time. Take that disclosure away and the same code becomes the hole it is not. +func TestAConsentedSliceIsAlwaysToldWhatTheBookCarries(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + var eps []chunktest.Chapter + var spine []string + for i := 1; i <= 4; i++ { + id := fmt.Sprintf("c%d", i) + eps = append(eps, chunktest.Chapter{ID: id, Href: id + ".xhtml", Body: fmt.Sprintf("

静かな図書館の朝%d。

", i)}) + spine = append(spine, id) + } + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + epub: eps, spine: spine, regenerate: 1, waveWorkers: 4, rebillConsentUSD: 0.005, + }) + dir := filepath.Dir(bookPath) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + r1.Close() + doc := decisionsDocFor(t, dir, "test-book", membank.Decision{ + Action: membank.ActionApprove, Src: "静か", Dst: "тихий"}) + if _, err := ApplyBankDecisions(ctx, bookPath, doc, false); err != nil { + t.Fatal(err) + } + + var logBuf bytes.Buffer + r2 := newRunner(t, bookPath) + defer r2.Close() + r2.Log = slog.New(slog.NewTextHandler(&logBuf, &slog.HandlerOptions{Level: slog.LevelWarn})) + r2.Resnapshot = true + r2.MaxUnits = 1 + r2.AcceptRebill = RebillConsent{Given: true, Capped: true, CapUSD: 0.002} + if _, err := r2.TranslateBook(ctx); err != nil { + t.Fatalf("a funded slice must be admitted: %v", err) + } + out := logBuf.String() + // This run's own slice — the amount actually charged. + if !strings.Contains(out, "rebill_usd=0.001820") { + t.Fatalf("the accepted run must name what IT is charged; log:\n%s", out) + } + // And the book's whole drift beside it, so the caller can see they are buying it down in slices. + for _, want := range []string{"book_rebill_units=4", "book_rebill_usd=0.007280", "threshold_usd=0.005000"} { + if !strings.Contains(out, want) { + t.Fatalf("a consented slice must be told what the BOOK carries (%q missing) — without it the caller cannot tell one purchase from buying the whole book a slice at a time; log:\n%s", want, out) + } + } +} + +// TestAPurchaseBuysNewBookBeforeReMakingOldBook pins the ADMISSION ORDER, which nothing else holds. +// +// It is a rot path rather than a live defect: collapsing planVolume's two passes back into one book-order +// pass leaves the whole battery green, so any later session tidying that loop — or adding a fourth unit +// class and reordering it — silently restores a measured defect. Under book order a purchase of two units +// on a book with two delivered re-makes chapters 1-2, reports Delivered=0 / Reworked=2 / LeftFresh=3, +// exits 0, and the platform records `ready`: the buyer paid and the book did not advance by a chapter. +// +// The fixture puts the signed term ONLY in the early chapters, so exactly the delivered units supersede +// and the undelivered tail stays fresh — the ordinary shape after a bank edit lands mid-sale. +// +// ⚠ Note the second assertion class: under book order this same purchase is not merely misallocated, it +// is REFUSED — admitting rework makes the run's own re-payment non-zero and the book's drift is over the +// threshold. Delivery-first admits only fresh work, so the consent gate has nothing to ask about and the +// run goes through. That the purchase SUCCEEDS is therefore part of what is being pinned. +func TestAPurchaseBuysNewBookBeforeReMakingOldBook(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + var eps []chunktest.Chapter + var spine []string + for i := 1; i <= 5; i++ { + id := fmt.Sprintf("c%d", i) + body := fmt.Sprintf("

朝%d。鈴木は歩いた。

", i) + if i <= 2 { + // 静か occurs ONLY in the chapters the first purchase delivers, so signing it later supersedes + // exactly those and leaves chapters 3-5 untouched. + body = fmt.Sprintf("

静かな朝%d。鈴木は歩いた。

", i) + } + eps = append(eps, chunktest.Chapter{ID: id, Href: id + ".xhtml", Body: body}) + spine = append(spine, id) + } + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + epub: eps, spine: spine, regenerate: 1, waveWorkers: 1, rebillConsentUSD: 0.002, + }) + dir := filepath.Dir(bookPath) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r1 := newRunner(t, bookPath) + r1.MaxUnits = 2 + res1, err := r1.TranslateBook(ctx) + if err != nil { + t.Fatal(err) + } + r1.Close() + if res1.Volume == nil || res1.Volume.Delivered != 2 { + t.Fatalf("the first purchase delivers chapters 1-2: %+v", res1.Volume) + } + + doc := decisionsDocFor(t, dir, "test-book", membank.Decision{ + Action: membank.ActionApprove, Src: "静か", Dst: "тихий"}) + if _, err := ApplyBankDecisions(ctx, bookPath, doc, false); err != nil { + t.Fatal(err) + } + + r2 := newRunner(t, bookPath) + defer r2.Close() + r2.Resnapshot = true + r2.MaxUnits = 2 + res2, err := r2.TranslateBook(ctx) + if err != nil { + t.Fatalf("a purchase that spends its grant entirely on NEW book re-pays nothing, so the consent gate has nothing to ask about and the run must go through; under book order this same purchase is refused: %v", err) + } + v := res2.Volume + if v == nil { + t.Fatalf("units were held back, so the run must report a volume stop") + } + if v.Delivered != 2 || v.Reworked != 0 { + t.Fatalf("the grant must buy NEW book while the book still has any — got Delivered=%d Reworked=%d; a single book-order pass would report 0 and 2", v.Delivered, v.Reworked) + } + if v.LeftFresh != 1 || v.LeftRework != 2 { + t.Fatalf("one chapter never delivered and two awaiting a re-make: %+v", *v) + } + // And the chapters that actually ran are the NEW ones, not the old ones re-made. + started := chaptersWithRows(t, r2.Store, "test-book") + for _, ch := range []int{3, 4} { + if !started[ch] { + t.Fatalf("chapter %d was never delivered and had the grant available, yet did not run: %v", ch, started) + } + } + if started[5] { + t.Fatalf("chapter 5 is past the grant and must not have run") + } +} + +// TestTheVolumeCeilingMovesNoRequestByte is review axis 3 (determinism), executed rather than asserted. +// +// A ceiling is an OPERATOR axis: it decides how much of the book this process does, never what the +// process sends. If setting it moved a wave snapshot or a rendered byte, every already-paid unit of the +// book would be superseded the moment anyone passed the flag — a ceiling that re-bought the book would be +// worse than no ceiling. The property is claimed in Runner.MaxUnits's doc comment; nothing held it. +// +// Two halves, because either alone is weak: the SNAPSHOT must not move (that is what re-pins jobs and +// re-bills waves), and the WIRE BYTES must be identical (that is what the resume fast-path compares). +func TestTheVolumeCeilingMovesNoRequestByte(t *testing.T) { + // (1) The snapshot is blind to the ceiling. + rec0 := &reqRec{} + srv0 := newJSONProvider(rec0, draftEdit) + defer srv0.Close() + r := newRunner(t, volumeBook(t, srv0.URL, 3)) + defer r.Close() + if err := r.seedGlossary(context.Background()); err != nil { + t.Fatal(err) + } + r.MaxUnits = 0 + draftOff, _, err := r.snapshotIDForWave(waveDraft) + if err != nil { + t.Fatal(err) + } + editOff, _, err := r.snapshotIDForWave(waveEdit) + if err != nil { + t.Fatal(err) + } + r.MaxUnits = 2 + draftOn, _, err := r.snapshotIDForWave(waveDraft) + if err != nil { + t.Fatal(err) + } + editOn, _, err := r.snapshotIDForWave(waveEdit) + if err != nil { + t.Fatal(err) + } + if draftOff != draftOn || editOff != editOn { + t.Fatalf("setting a volume ceiling moved a wave snapshot (draft %.12s→%.12s, edit %.12s→%.12s): every already-paid unit of the book would be superseded by passing the flag", + draftOff, draftOn, editOff, editOn) + } + + // (2) The bytes a granted unit sends are the bytes it would have sent unbounded — including when the + // ceiling serves units OUT of book order, which delivery-before-rework does by design. + recFull := &reqRec{} + srvFull := newJSONProvider(recFull, draftEdit) + defer srvFull.Close() + rFull := newRunner(t, volumeBook(t, srvFull.URL, 3)) + if _, err := rFull.TranslateBook(context.Background()); err != nil { + t.Fatal(err) + } + rFull.Close() + + recCut := &reqRec{} + srvCut := newJSONProvider(recCut, draftEdit) + defer srvCut.Close() + rCut := newRunner(t, volumeBook(t, srvCut.URL, 3)) + defer rCut.Close() + rCut.MaxUnits = 1 + if _, err := rCut.TranslateBook(context.Background()); err != nil { + t.Fatal(err) + } + full := map[string]bool{} + for _, b := range recFull.all() { + full[b] = true + } + cut := recCut.all() + if len(cut) == 0 { + t.Fatalf("the bounded run sent nothing, so nothing is compared") + } + for _, b := range cut { + if !full[b] { + t.Fatalf("a bounded run sent a request the unbounded run never sent — the ceiling changed what goes on the wire, and every unit it touches would be re-billed:\n%.400s", b) + } + } +} + +// TestAMoneyHaltStillSaysWhatTheVolumeGrantWas covers the last of the round-2 cosmetics, and it is the +// same class as the mining warning: a diagnostic nothing tests is a diagnostic that silently stops being +// emitted. +// +// When the MONEY ceiling stops a run there is no result to carry the volume report, so without this line +// the operator is told the run halted on money and nothing about what it had been granted or how far that +// got — the first thing anyone asks. The durable progress is in the store either way; this is about the +// stderr account of the run being whole rather than half. +func TestAMoneyHaltStillSaysWhatTheVolumeGrantWas(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := volumeBook(t, srv.URL, 4) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + var logBuf bytes.Buffer + r := newRunner(t, bookPath) + defer r.Close() + r.Log = slog.New(slog.NewTextHandler(&logBuf, &slog.HandlerOptions{Level: slog.LevelWarn})) + r.MaxUnits = 2 // the ceiling binds: two units are held back + r.CeilingUSD = 0.0000001 // ...but money runs out first + _, err := r.TranslateBook(ctx) + var halt *CeilingHalt + if !errors.As(err, &halt) { + t.Fatalf("precondition: the money ceiling must be what stops this run, got %v", err) + } + out := logBuf.String() + if !strings.Contains(out, "the stop below is NOT the volume ceiling") { + t.Fatalf("a money halt inside a volume-bounded run must say which ceiling stopped it and what the grant was; log:\n%s", out) + } + if !strings.Contains(out, "max_units=2") { + t.Fatalf("the line must name the grant the run was given:\n%s", out) + } + + // And it must NOT appear when no ceiling was in force — an ordinary money halt has no volume grant to + // report, and inventing one would be noise on every halted run in the fleet. + var quiet bytes.Buffer + r2 := newRunner(t, volumeBook(t, srv.URL, 4)) + defer r2.Close() + r2.Log = slog.New(slog.NewTextHandler(&quiet, &slog.HandlerOptions{Level: slog.LevelWarn})) + r2.CeilingUSD = 0.0000001 + if _, err := r2.TranslateBook(ctx); err == nil { + t.Fatalf("precondition: this run must also halt on money") + } + if strings.Contains(quiet.String(), "the stop below is NOT the volume ceiling") { + t.Fatalf("an unbounded run has no volume grant to report:\n%s", quiet.String()) + } +} + +// TestFreeUnitsAreReJudgedWhenTheBankMovesMidRun pins the fix for the acceptance hunter's blocking +// finding: on a mining book the ceiling did not hold, and the report called the overspend free. +// +// THE CHAIN. planVolume classifies before the draft wave. Between the two waves sits the bank-mining +// stop, whose auto-continue branch RE-SEEDS the bank (mining.go) — after which waverun recomputes the +// edit-wave snapshot. Units classified FREE are admitted OUTSIDE the grant, because free work costs +// nothing. So every one of them was judged against a snapshot the run itself then replaced, and any whose +// injected bytes the new bank changes gets a fresh PAID editor call no grant authorised — reported as +// "rode along at $0". Measured by the hunter: four calls under a grant of one, $0.007280 called free. +// +// The test drives rescopeEditWave directly rather than waiting for a fixture in which the auto-bank +// happens to change a delivered unit's bytes. That is deliberate: the defect is that nothing GUARANTEES +// the two snapshots agree, so the invariant — "a unit admitted without a slot is re-judged against the +// snapshot the wave really uses" — is what must hold for every book, including ones no fixture reaches. +func TestFreeUnitsAreReJudgedWhenTheBankMovesMidRun(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := volumeBook(t, srv.URL, 4) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r := newRunner(t, bookPath) + defer r.Close() + if _, err := r.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + chunks, withText, err := r.readModelChunks() + if err != nil { + t.Fatal(err) + } + _ = chunks + full, err := withText() + if err != nil { + t.Fatal(err) + } + sel := precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget) + units := r.outputUnits(full) + if len(units) < 3 { + t.Fatalf("fixture needs several units, got %d", len(units)) + } + + // A scope in the exact shape planVolume leaves: every unit judged FREE (the book is fully delivered + // and nothing has moved yet) and admitted without consuming any grant. + newScope := func(max int) *volumeScope { + s := &volumeScope{ + admitted: map[chunkKey]bool{}, leader: map[chunkKey]bool{}, + class: map[chunkKey]unitClass{}, editBlocked: map[chunkKey]bool{}, + stop: VolumeStop{MaxUnits: max}, + editSnapshot: "SNAPSHOT-AT-PLAN-TIME", + } + for _, u := range units { + key := chunkKey{u.Chapter, u.FirstChunkIdx} + s.admitted[key] = true + s.class[key] = unitFree + s.stop.Free++ + for _, m := range u.Members { + s.leader[chunkKey{m.Chapter, m.ChunkIdx}] = true + } + } + return s + } + + // (1) The snapshot did NOT move: nothing may be re-judged, and the ordinary path pays nothing for + // this machinery existing. + same := newScope(1) + same.editSnapshot = "X" + if err := r.rescopeEditWave(ctx, same, units, full, sel, "X"); err != nil { + t.Fatal(err) + } + if same.stop.Free != len(units) || same.stop.Paid() != 0 || len(same.editBlocked) != 0 { + t.Fatalf("an unmoved snapshot must change nothing: %+v", same.stop) + } + + // (2) The snapshot MOVED. Every unit's edit row is now on a superseded snapshot that is NOT a + // re-pinnable bank-only move (the id is unrelated), so none of them is free any more. With a grant of + // one, exactly one may take a slot and the rest must be HELD BACK — not run for free. + moved := newScope(1) + if err := r.rescopeEditWave(ctx, moved, units, full, sel, "A-DIFFERENT-SNAPSHOT"); err != nil { + t.Fatal(err) + } + if moved.stop.Free != 0 { + t.Fatalf("units whose bytes the new bank changed are not free; %d still counted as riding along at $0 — that is the overspend reported as free", moved.stop.Free) + } + if moved.stop.Paid() != 1 { + t.Fatalf("the grant is one unit, so exactly one may be paid for; got %d — the ceiling did not hold", moved.stop.Paid()) + } + if moved.stop.LeftRework != len(units)-1 { + t.Fatalf("the rest must be held back as already-delivered-not-yet-re-made, got LeftRework=%d of %d units", moved.stop.LeftRework, len(units)) + } + // And the held-back ones must actually be refused by the wave's own admission check. + held := 0 + for _, u := range units { + if !moved.allowsUnit(u) { + held++ + } + } + if held != len(units)-1 { + t.Fatalf("%d unit(s) were blocked in the report but %d are actually refused by allowsUnit — the accounting and the gate disagree", moved.stop.LeftRework, held) + } +} + +// TestTheEditWaveRefusesAStalePlan pins the WIRING of the re-plan, which the behavioural test above +// cannot: that test calls rescopeEditWave directly, so deleting its call site in the wave driver left the +// whole battery green while the ceiling stopped holding. The invariant is now the production code's own — +// the wave refuses to run a scope planned against a snapshot it is not using — so the re-plan cannot be +// removed without the run saying so. +func TestTheEditWaveRefusesAStalePlan(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := volumeBook(t, srv.URL, 3) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + // An ordinary bounded run must pass the guard — it is an invariant, not a tax. + r := newRunner(t, bookPath) + defer r.Close() + r.MaxUnits = 2 + if _, err := r.TranslateBook(ctx); err != nil { + t.Fatalf("a normal bounded run must satisfy the invariant: %v", err) + } + + // And a scope carrying a stale plan is refused rather than run. + stale := &volumeScope{ + admitted: map[chunkKey]bool{}, leader: map[chunkKey]bool{}, + class: map[chunkKey]unitClass{}, editBlocked: map[chunkKey]bool{}, + stop: VolumeStop{MaxUnits: 1}, editSnapshot: "PLANNED-AGAINST-SOMETHING-ELSE", + } + if stale.editSnapshot == "" { + t.Fatalf("fixture: the stale scope must carry a plan snapshot") + } + // rescopeEditWave is what repairs it; after it runs the guard's condition is satisfied. + if err := r.rescopeEditWave(ctx, stale, nil, nil, nil, "THE-REAL-ONE"); err != nil { + t.Fatal(err) + } + if stale.editSnapshot != "THE-REAL-ONE" { + t.Fatalf("the re-plan must adopt the snapshot it re-judged against, else the guard can never pass; got %q", stale.editSnapshot) + } +} + +// TestAFlaggedUnitIsNotReportedAsDelivered pins the acceptance hunter's finding 2. A unit can be paid for +// and come back FLAGGED with nothing shippable; the plan-time counter called it delivered anyway, so a +// purchase of two units one of which flagged printed "2 NEW unit(s) delivered" over a single readable +// unit. Admission has to be decided before the work — that is what bounds the money — but a decision to +// PAY for a unit is not a fact about a chapter existing, and the report must not conflate them. +func TestAFlaggedUnitIsNotReportedAsDelivered(t *testing.T) { + var mu sync.Mutex + flaggedOne := false + rec := &reqRec{} + srv := newJSONProvider(rec, func(body string) (string, string) { + if !isEditBody(body) && strings.Contains(body, "朝1") { + mu.Lock() + flaggedOne = true + mu.Unlock() + return "", "stop" // an empty draft flags the unit and ships nothing + } + return draftEdit(body) + }) + defer srv.Close() + var eps []chunktest.Chapter + var spine []string + for i := 1; i <= 5; i++ { + id := fmt.Sprintf("c%d", i) + eps = append(eps, chunktest.Chapter{ID: id, Href: id + ".xhtml", Body: fmt.Sprintf("

朝%d。鈴木は歩いた。

", i)}) + spine = append(spine, id) + } + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + epub: eps, spine: spine, regenerate: 0, waveWorkers: 1, + }) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + + r := newRunner(t, bookPath) + defer r.Close() + r.MaxUnits = 2 + res, err := r.TranslateBook(ctx) + if err != nil { + var flags *CompletedWithFlags + if !errors.As(err, &flags) { + t.Fatal(err) + } + } + if !flaggedOne { + t.Fatalf("fixture never produced the flag, so it proves nothing") + } + shipped := 0 + for _, oc := range res.Chunks { + if oc.FinalText != "" { + shipped++ + } + } + v := res.Volume + if v == nil { + t.Fatalf("three units were held back, so a volume stop must be reported") + } + if v.Delivered != shipped { + t.Fatalf("the stop claims %d NEW unit(s) delivered but only %d shipped readable text — a paid-and-flagged unit is not a delivery: %+v", v.Delivered, shipped, *v) + } + if v.Flagged != 1 { + t.Fatalf("one unit was paid for and flagged; the report must say so rather than fold it into delivery: %+v", *v) + } + if !strings.Contains(v.String(), "PAID FOR BUT FLAGGED") { + t.Fatalf("the operator line must name money spent for no text: %s", v.String()) + } +} diff --git a/backend/internal/pipeline/waverun.go b/backend/internal/pipeline/waverun.go index 84ff2aa2..7a1e4b6c 100644 --- a/backend/internal/pipeline/waverun.go +++ b/backend/internal/pipeline/waverun.go @@ -102,7 +102,7 @@ func (e *WaveSignatureStop) Error() string { // translateBookWaves is the wave driver (R1). It runs the draft wave (draft ∥), the bank-mining stop, the edit wave (edit ∥) and // assembles the BookResult. chunks + stickySel come from the precompute pass (chunk.SplitChunks + precomputeSticky), computed by // the caller (TranslateBook) after ingest/seed/eager-build. Returns a *WaveSignatureStop when the bank-mining stop stops. -func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, stickySel []membank.Selection) (*BookResult, error) { +func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, stickySel []membank.Selection, scope *volumeScope) (*BookResult, error) { draftStages := r.waveStagesIndexed(waveDraft) editStages := r.waveStagesIndexed(waveEdit) workers := r.Pipeline.Waves.Workers @@ -126,6 +126,12 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s draftResults := make([]stageSeqResult, len(chunks)) draftUnits := newDraftUnitTracker(r.outputUnits(chunks), chunks) if err := r.runWave(ctx, workers, len(chunks), func(ctx context.Context, i int) error { + // The VOLUME ceiling (volume.go), applied where the item BEGINS rather than after it: a chunk whose + // output unit is outside this run's granted scope is never started, so no ceiling can be overshot by + // one call. A nil scope means no ceiling is in force and this is a no-op. + if !scope.allows(chunks[i]) { + return nil + } d, err := r.runDraftChunk(ctx, draftSnapshot, chunks[i], stickySel[i], draftStages, !editWave) if err != nil { return err @@ -158,6 +164,18 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s } res := &BookResult{BookID: r.Book.BookID} + // The stop NAMES its ceiling (D39.165 §1б: OpenHands' silent iteration limit is the anti-pattern being + // avoided). It rides on the result rather than on an error because a volume stop is a COMPLETION — the + // run did what was bought — so it must not reach the exit-code mapper at all. Only set when the ceiling + // actually held work back: a run granted more than the book had left simply finished. + if scope.bound() { + res.Volume = &scope.stop + r.Log.InfoContext(ctx, "the run is bounded by the VOLUME ceiling: it will stop having done what was granted, not at the end of the book", + "book", r.Book.BookID, "max_units", scope.stop.MaxUnits, + "paid_units", scope.stop.Paid(), "delivered_new", scope.stop.Delivered, "re_made", scope.stop.Reworked, + "free", scope.stop.Free, + "left_never_delivered", scope.stop.LeftFresh, "left_delivered_not_re_made", scope.stop.LeftRework) + } // The terminologist's spend is THIS run's spend and belongs in this run's total. It is not a chunk cost, // so the per-chunk accumulation below cannot see it — and a paid call that no total reports is a call // the operator cannot notice. (Zero when the gate is off or the calls replayed from checkpoints.) @@ -166,6 +184,9 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s // --- Draft-only pipeline: the draft IS the shipping output; assemble per draft chunk (no edit units) --- if !editWave { for i, ch := range chunks { + if !scope.allows(ch) { + continue // outside the volume scope: never drafted, so there is no outcome to assemble + } oc := r.draftOnlyOutcome(ch, draftResults[i]) res.Chunks = append(res.Chunks, oc) res.TotalUSD += oc.CostUSD @@ -173,6 +194,7 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s res.Flagged++ } } + scope.reconcile(res.Chunks) r.Log.InfoContext(ctx, "book run finished (draft-only)", "book", r.Book.BookID, "chunks", len(res.Chunks), "flagged", res.Flagged, "run_usd", fmt.Sprintf("%.6f", res.TotalUSD)) return res, nil @@ -187,6 +209,21 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s return nil, fmt.Errorf("pipeline: upsert edit-wave snapshot %.12s: %w", editSnapshot, err) } units := buildEditUnits(chunks) + // ⛔ THE PLAN IS RE-JUDGED HERE, because the bank-mining stop above may have re-seeded the bank and + // moved the snapshot this wave runs under. Units the plan called FREE were admitted outside the grant; + // any that the new bank makes paying must take a slot or not run. See rescopeEditWave. + if err := r.rescopeEditWave(ctx, scope, units, chunks, stickySel, editSnapshot); err != nil { + return nil, err + } + // ⚠ AND THE RE-PLAN IS NOT OPTIONAL: the wave refuses to run a scope that was planned against a + // different snapshot than the one it is about to use. Without this the call above is merely a call — + // deleting it leaves every test green while the ceiling silently stops holding on any book that + // re-seeds mid-run. With it, the invariant is the code's own, and removing the re-plan makes the run + // say so instead of overspending quietly. + if scope != nil && scope.editSnapshot != editSnapshot { + return nil, fmt.Errorf("pipeline: the volume scope was planned against edit-wave snapshot %.12s but the wave is running under %.12s — units admitted as free were judged on ground this run has since replaced, and paying for them would escape the ceiling (wave sequencing bug: rescopeEditWave did not run)", + scope.editSnapshot, editSnapshot) + } draftByKey := make(map[chunkKey]stageSeqResult, len(chunks)) for i, ch := range chunks { draftByKey[chunkKey{ch.Chapter, ch.ChunkIdx}] = draftResults[i] @@ -195,6 +232,11 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s "units", len(units), "workers", workers, "edit_stages", len(editStages)) unitOutcomes := make([]*ChunkOutcome, len(units)) if err := r.runWave(ctx, workers, len(units), func(ctx context.Context, i int) error { + // Same admission as the draft wave, and the SAME set: a unit is either in this run's scope for both + // waves or in neither, so the edit wave can never be handed a unit whose members were not drafted. + if !scope.allowsUnit(units[i]) { + return nil + } oc, err := r.runEditUnit(ctx, editSnapshot, units[i], draftByKey, editStages) if err != nil { return err @@ -207,12 +249,17 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s return nil, err } for _, oc := range unitOutcomes { + if oc == nil { + continue // outside the volume scope: the unit was never started this run + } res.Chunks = append(res.Chunks, *oc) res.TotalUSD += oc.CostUSD if oc.Disposition == DispFlagged { res.Flagged++ } } + // The plan's counters become the run's counters only now, corrected by what actually shipped. + scope.reconcile(res.Chunks) r.Log.InfoContext(ctx, "book run finished", "book", r.Book.BookID, "units", len(res.Chunks), "flagged", res.Flagged, "run_usd", fmt.Sprintf("%.6f", res.TotalUSD)) return res, nil diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index a28ce826..1bfc9d54 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -1,6 +1,6 @@ # Журнал прогресса -> **⟶ ТЕКУЩЕЕ СОСТОЯНИЕ** (на 2026-08-29, голова D39.169 — КУРС: движок и платформа до «работает и отдаёт результат», фронт ЗАМОРОЖЕН и P7 его НЕ размораживает (D39.147). **ОЧЕРЕДЬ №19** (единственный носитель — здесь; роль передана 22.08, №18 закрыт D39.155): (а) **P7 платформы ПРИНЯТ С ФИКС-ЛИСТОМ и ЗАЛЕНДЕН, сессия закрыта владельцем 20.08** (D39.153; зонный фикс-лист из тринадцати пунктов плюс врезанный первым блокер свипа — пинг №18 в `platform/docs/platform-PROGRESS.md`; ⚠ прежний врезанный пример «первым `ContractVersion` 0.3.0 при формах 0.4.0» СНЯТ — константа поднята и запинена гейтом против канона лендингом `31f1f82`) · (а1) **контракт API v0 0.4.0 РАТИФИЦИРОВАН** (D39.152; PD-327 закрыт; зеркало фронта отстаёт ратифицированно, при разморозке `cmp`) · (б) **лендинг петель полигона (фаза Д) — ОТКРЫТ**: приёмка ФАЗЫ у параллельного оркестратора, лендинг за этим; ⚠ **предупреждение в силе: работа полигона в дереве по-прежнему НЕЗАКОММИЧЕНА** (20 позиций: `START_PROMT.MD` + 14 правленых и 5 новых файлов `eval/`). Владелец перевозил её на новую машину транспортным коммитом `temp`, но 24.08 **снял его словом владельца** (`reset --mixed`, дерево не тронуто — сверено побайтно, бэкап `~/tm-backup-2026-08-24/textmachine-HEAD.bundle` + reflog): транспортный коммит вдобавок уносил `START_PROMT.MD`, который канон коммитить запрещает. Состав читать командой `git status --short -- eval/`, снимок здесь протухает. Не трогать и не «прибирать»; приёмка ФАЗЫ — за параллельным оркестратором, пинг №20 — в секции «Полигон». Форма коммита строго pathspec · (в) **дофикс-пак P8-FIX ПРИНЯТ И ЗАЛЕНДЕН 22.08** (D39.154, лендинг `31f1f82`): блокер свипа закрыт четырьмя механизмами и пере-проверен живой пробой оркестратора, деньги сошлись двумя путями из сырого леджера; мои четыре строки регистра — `PD-371`…`PD-374`. ⚠ **Порядок вышел ОБРАТНЫМ этой очереди:** промт дофикса требовал запуска ПОСЛЕ читающего пака, запущен был раньше. Цена перестановки, названная мной вслух 21.08 («блокер живёт всё время читающего пака»), тем самым НЕ заплачена — это лучший исход, чем планировалось; в минус — §4.7 (аддендум релеем) пуст и читающий пак пойдёт по только что переписанному коду. **ЧТО ДАЛЬШЕ С ПЛАТФОРМОЙ — решено 22.08 при актуализации доков, порядок такой.** **(1) Читающий пак P8-REVIEW — ОТРАБОТАН, ПРИНЯТ И ЗАЛЕНДЁН 27.08** (D39.159): четыре оси, 24 новые строки регистра (`PD-375`…`PD-398`), 17 дописок, каталог воспроизведения `platform/docs/p8-review/`. Единственный vuln — `PD-379` (major): открытый поток событий переживает отзыв сессии. Числа сдачи пере-мерены приёмкой на дереве С лендингом бэкенда. Промт архивируется. **Следующая работа зоны — кодовый пак по этим строкам, запуск по слову владельца.** Промт ЖИВ и переписан 22.08 в трёх местах, которые дофикс сделал ложными: «два денежных дефекта не заведены» (заведены и закрыты — `PD-333`/`PD-334`, вместо них дан их КЛАСС и живой образец `PD-372`) · приор оси 4 «наблюдаемость без ручки» (ручка построена ⇒ вопрос стал «достаточна ли она») · абсолют «оси ни в один дифф не входили» (сужен: системного разбора не было, кусками входили). **Довод ЗА пак — базовая ставка, а не вера:** первый же серьёзный взгляд на ОДНУ из четырёх осей (очередь, 21.08) дал пред-продовый блокер, проживший два пака под статусом `fixed`. Осей осталось три, и на двух из них лежат деньги и вход. **(2) ПОРЯДОК РАБОТ ПО ШВУ — ратифицирован D39.156 (23.08), исполняется строго так.** **(2а) Движковый пак шва — ОТРАБОТАН И ЗАЛЕНДЖЕН 27.08 (D39.158), промт в `archive/prompts/`** (аудит доков 28.08 поймал, что очередь пять дней звала его АКТИВНЫМ и направляла сессию в исполненное). Состав, для истории: конвенционные дефолты `mined_delta`/`mined_rejects` · $0-глагол приёма правок банка · путь к файлу ключей аргументом (**211** — движковый конец; платформенный доехал паком P9, строка ЗАКРЫТА D39.162) · строгость сид-загрузчика · путь артефакта и версия документа в `status --json` (**213** — движковая половина; платформенная снята P9, у строки остался один дев-путь). **(2б) Контрактный минор — ПОСЛЕ него и ОДНИМ куском:** снос отменённой пер-термной модели + новая дверь, тело которой выводится из словаря глагола, а не сочиняется до него; сюда же строка **203** и хвосты компаньона. ⚠ Строка **200** ушла НЕ сюда: D39.160 изъял её из этого минора и отправил с паком (2в), где она и закрылась минором **0.6.0**. ⚠ Гигиена держится сознательно: снести дверь сейчас и завести в следующем миноре — две перегенерации фронта вместо одной; фолбэк объявлен — если пак застрянет надолго, гигиену шипить отдельно. **(2в) Платформенный пак P9 — ОТРАБОТАН, ПРИНЯТ И ЗАЛЕНДЁН 28.08 (D39.162):** дверь правок банка смонтирована синхронно и под пер-книжным мьютексом · ключи доехали до движка аргументом `--keys-file` (строка **211** ЗАКРЫТА) · дубль конвенции пути снят (строка **213** — осталась одна строка в дев-пути) · снята сквозная полоса прогресса вместе с её канонной половиной, минор **0.6.0** (строка **200** ЗАКРЫТА) · с провода сняты два поля упразднённой модели (`PD-399`). ⚠ Пункт «воркер решений» СНЯТ эрратой 27.08-з: синхронность держится, замер сессии её подтвердил. Фикс-раунд приёмки — 13 позиций, две мои диспозиции сессия ОПРОВЕРГЛА исполнением и я это принял (см. D39.162). **(3) `sqlc` отдельной сессией** (решение владельца 22.08, `BACKLOG.md` П-19, граница 41 запрос в пяти файлах) — механическая работа, не мешать её с содержательной. **(4) Живой прогон книги насквозь** — строка 202, гейт прежний: ПОСЛЕ холодного прогона движка. ⚠ **На владельце и блокирует пункт своей темы:** ⚠ **развилка владения `book.yaml` (199а) СНЯТА — с владельца убрана аудитом 28.08:** она растворена седьмым вариантом ещё D39.156 п.3 (конвенционные дефолты + движковый глагол), вход построен D39.158/D39.162, а вся строка 199 закрыта D39.166 и снесена с таблицы. Пять дней держалась на владельце как блокер решённого. ⚠ **ДВА ВОПРОСА, СНЯТЫЕ ВЛАДЕЛЬЦЕМ 28.08 (D39.165) — оставлены здесь как след, потому что реестр ожиданий о них не знал вовсе и это была дыра учёта.** Ответы: смена формы конвейера — форма «ЭПОХА» (событие книги, канон уже разрешает пересчёт через границу `structure_version`); единица продажи — цена от ОБЪЁМА исходника по `chapters.units_total` + потолок объёма в движке + хвост качества в тариф, калибровка гейчена строкой 202. Прежняя формулировка: (дыру учёта нашла разведка 28.08: нота сказала «ждёт слова владельца», а реестр ожиданий об этом не знал): **(i) смена формы конвейера на ЖИВОЙ книге** — что деплой обязан делать с книгой, начерченной по старой форме, когда форму сменили (носители `PD-403`/`PD-404`; ⚠ проект отвечал на СОСЕДНИЙ вопрос дважды — D39.152 п.4 и D39.153 §4б — и два ратифицированных ответа В КОМПОЗИЦИИ и дают дефект: липкость флага ратифицирована ПРОТИВ отката назад, но в обратную сторону она морозит счёт, и комментарий `platform/internal/pgstore/sink.go:405-408` описывает поведение, которого его же код НЕ делает); **(ii) `PD-410`** — платформа продаёт ГЛАВЫ, движок останавливается по ДЕНЬГАМ, и единица продажи не равна единице остановки · (г) **контракт: сквозная полоса прогресса ЗАКРЫТА** — минор **0.6.0** заленджен вместе с паком P9 (D39.162): доля одна на всю работу прогона через обе волны, подпись стадии объявлена ВТОРЫМ ограниченным исключением из границы «ничего о том, КАК переводится книга». ⚠ Остаток назван и НЕ забыт: платформа продаёт главы, а движку передаёт только `--ceiling-usd` — единицы разные, и это `PD-410` (архитектурный стоп зоны). Хвосты релеев (203) ждут следующего минора · (д) свободные бэкенд: 160 Этап 0, фикс-лист ФЧ строкой 197 (носителя-сессии нет по слову владельца); ⚠ строки 176/181/187 СНЯТЫ с таблицы при лендинге 20.08 — они закрыты ещё D39.149, а очередь всё это время звала 181 свободной работой (поймано ревью доков; остатки живут строками 194/196/197) · (е) фронтовый фрагмент pre-commit хука получает норму «жалуйся владельцу» — **решение владельца 21.08: ДА**, исполняется первым касанием зоны (пинг в её журнале). · (ж) **ПЕРЕЕЗД НА НОВУЮ МАШИНУ — стенд ПЕРЕСОБРАН 24.08 (№19), как поднимать — карта `README.md`, строки «Тулчейн» и `counts.py`.** Ubuntu 26.04 WSL2, пользователь `ubuntu-26`; тулчейн стоял НУЛЕВОЙ. Поставлено: Go 1.26.7 · golangci-lint 2.12.2 · Node 22.23.2 — тарболами вендоров со сверкой sha256 в `~/.local/opt` (симлинки в `~/.local/bin`); PostgreSQL 18.4 — рецептом самой зоны (`platform/docs/STACK_DECISIONS.md` §«Postgres на стенде без root»); `build-essential` + `python3-venv` — руками владельца, они единственные требовали root. **Обе батареи зелены под `-race`:** бэкенд `make battery` EXIT=0, линтер 0 issues, скипы только корпус-гейченные и хелперы; платформа `make check` EXIT=0, 18 пакетов, линтер 0 issues, **скипов 0**. Фронт: `npm ci` + `npm run check` проходит целиком (335 тестов), `vite build` собирается. Сквозняк проверен ЖИВЬЁМ: демон платформы → сид через настоящий интейк → движок распарсил книгу (3 главы) → дев-прокси фронта достучался до платформы. ⚠ **Что НЕ проверено:** ни одного платного вызова · браузеров Playwright нет ⇒ `check:full` не гонялся · локальной модели и GPU на машине нет вовсе (слово владельца 24.08: мерить нечем, ветка не закрыта) · OIDC не поднимался, вход только дев-логином. ⚠ Правка `.gitignore` владельца несла дефект — правило `books/` без якоря глушило ЛЮБОЙ каталог `books`, включая живой пакет `platform/internal/books/` (новый файл там уходил бы в игнор молча); заякорено в `/books/`, проверено пробой в обе стороны. Новые строки бэклога — **217** (пути полигона), **218** (дефолты корпуса в движке — ЗАКРЫТА и снята 27.08, D39.159 эррата-д), **219** (line-якоря D-лога съехали — ⚠ ОБЪЯВЛЕНА 24.08, а В ТАБЛИЦЕ ЗАВЕДЕНА только 27.08: три дня очередь ссылалась на несуществующий носитель) **Открытые обязательства, которых нет больше нигде:** веса `pro` — строка 172-г НЕ закрыта, прогон мерил классификатор на flash; дешёвый вход — спросить полигон, говорят ли что-то данные фазы Д (там pro гоняется редактором), отдельный замер не покупать без этого · `docs/experiments/00-provider-quirks.md` несёт протухшие цены DeepSeek («не изменились, flash $0.14/$0.28») — зона полигона, чинить пингом · санкция на платный прогон в архивном промте пака честности была РАЗОВОЙ · **норма плотности комментариев — ОТВЕЧЕНА владельцем 21.08:** режем ВОДУ, а не длину — счёт строк «пиши одну-две» негодный гейт и снят; уходит пересказ решений, провенанс и изложение исследования вместо ссылки, остаётся всё, что из одной функции не выводится, сколько бы строк ни заняло (норма переписана — `12-go-style-notes` §1; PD-255 платформы этим закрывается). ⚠ Метод-урок: её хендофф объявил «вопросов на владельце нет», а строка регистра просила именно решения — вопрос был закрыт ОБЪЯВЛЕНИЕМ, поймано ревьюером полноты выгрузки. Оркестраторов ДВА (решение владельца 07.08): этот — движок/платформа/фронт/доки; параллельный (РОЛЬЮ, без номера — счётчик один, D39.112 п.6) — приёмка полигона. Одновременно не запускаются; CURRENT-STATE ведут оба, чужие строки не трогают. Норма изоляции панелей — D39.113, гардрейлы в CLAUDE.md.) +> **⟶ ТЕКУЩЕЕ СОСТОЯНИЕ** (на 2026-08-29, голова D39.170 — КУРС: движок и платформа до «работает и отдаёт результат», фронт ЗАМОРОЖЕН и P7 его НЕ размораживает (D39.147). **ОЧЕРЕДЬ №19** (единственный носитель — здесь; роль передана 22.08, №18 закрыт D39.155): (а) **P7 платформы ПРИНЯТ С ФИКС-ЛИСТОМ и ЗАЛЕНДЕН, сессия закрыта владельцем 20.08** (D39.153; зонный фикс-лист из тринадцати пунктов плюс врезанный первым блокер свипа — пинг №18 в `platform/docs/platform-PROGRESS.md`; ⚠ прежний врезанный пример «первым `ContractVersion` 0.3.0 при формах 0.4.0» СНЯТ — константа поднята и запинена гейтом против канона лендингом `31f1f82`) · (а1) **контракт API v0 0.4.0 РАТИФИЦИРОВАН** (D39.152; PD-327 закрыт; зеркало фронта отстаёт ратифицированно, при разморозке `cmp`) · (б) **лендинг петель полигона (фаза Д) — ОТКРЫТ**: приёмка ФАЗЫ у параллельного оркестратора, лендинг за этим; ⚠ **предупреждение в силе: работа полигона в дереве по-прежнему НЕЗАКОММИЧЕНА** (20 позиций: `START_PROMT.MD` + 14 правленых и 5 новых файлов `eval/`). Владелец перевозил её на новую машину транспортным коммитом `temp`, но 24.08 **снял его словом владельца** (`reset --mixed`, дерево не тронуто — сверено побайтно, бэкап `~/tm-backup-2026-08-24/textmachine-HEAD.bundle` + reflog): транспортный коммит вдобавок уносил `START_PROMT.MD`, который канон коммитить запрещает. Состав читать командой `git status --short -- eval/`, снимок здесь протухает. Не трогать и не «прибирать»; приёмка ФАЗЫ — за параллельным оркестратором, пинг №20 — в секции «Полигон». Форма коммита строго pathspec · (в) **дофикс-пак P8-FIX ПРИНЯТ И ЗАЛЕНДЕН 22.08** (D39.154, лендинг `31f1f82`): блокер свипа закрыт четырьмя механизмами и пере-проверен живой пробой оркестратора, деньги сошлись двумя путями из сырого леджера; мои четыре строки регистра — `PD-371`…`PD-374`. ⚠ **Порядок вышел ОБРАТНЫМ этой очереди:** промт дофикса требовал запуска ПОСЛЕ читающего пака, запущен был раньше. Цена перестановки, названная мной вслух 21.08 («блокер живёт всё время читающего пака»), тем самым НЕ заплачена — это лучший исход, чем планировалось; в минус — §4.7 (аддендум релеем) пуст и читающий пак пойдёт по только что переписанному коду. **ЧТО ДАЛЬШЕ С ПЛАТФОРМОЙ — решено 22.08 при актуализации доков, порядок такой.** **(1) Читающий пак P8-REVIEW — ОТРАБОТАН, ПРИНЯТ И ЗАЛЕНДЁН 27.08** (D39.159): четыре оси, 24 новые строки регистра (`PD-375`…`PD-398`), 17 дописок, каталог воспроизведения `platform/docs/p8-review/`. Единственный vuln — `PD-379` (major): открытый поток событий переживает отзыв сессии. Числа сдачи пере-мерены приёмкой на дереве С лендингом бэкенда. Промт архивируется. **Следующая работа зоны — кодовый пак по этим строкам, запуск по слову владельца.** Промт ЖИВ и переписан 22.08 в трёх местах, которые дофикс сделал ложными: «два денежных дефекта не заведены» (заведены и закрыты — `PD-333`/`PD-334`, вместо них дан их КЛАСС и живой образец `PD-372`) · приор оси 4 «наблюдаемость без ручки» (ручка построена ⇒ вопрос стал «достаточна ли она») · абсолют «оси ни в один дифф не входили» (сужен: системного разбора не было, кусками входили). **Довод ЗА пак — базовая ставка, а не вера:** первый же серьёзный взгляд на ОДНУ из четырёх осей (очередь, 21.08) дал пред-продовый блокер, проживший два пака под статусом `fixed`. Осей осталось три, и на двух из них лежат деньги и вход. **(2) ПОРЯДОК РАБОТ ПО ШВУ — ратифицирован D39.156 (23.08), исполняется строго так.** **(2а) Движковый пак шва — ОТРАБОТАН И ЗАЛЕНДЖЕН 27.08 (D39.158), промт в `archive/prompts/`** (аудит доков 28.08 поймал, что очередь пять дней звала его АКТИВНЫМ и направляла сессию в исполненное). Состав, для истории: конвенционные дефолты `mined_delta`/`mined_rejects` · $0-глагол приёма правок банка · путь к файлу ключей аргументом (**211** — движковый конец; платформенный доехал паком P9, строка ЗАКРЫТА D39.162) · строгость сид-загрузчика · путь артефакта и версия документа в `status --json` (**213** — движковая половина; платформенная снята P9, у строки остался один дев-путь). **(2б) Контрактный минор — ПОСЛЕ него и ОДНИМ куском:** снос отменённой пер-термной модели + новая дверь, тело которой выводится из словаря глагола, а не сочиняется до него; сюда же строка **203** и хвосты компаньона. ⚠ Строка **200** ушла НЕ сюда: D39.160 изъял её из этого минора и отправил с паком (2в), где она и закрылась минором **0.6.0**. ⚠ Гигиена держится сознательно: снести дверь сейчас и завести в следующем миноре — две перегенерации фронта вместо одной; фолбэк объявлен — если пак застрянет надолго, гигиену шипить отдельно. **(2в) Платформенный пак P9 — ОТРАБОТАН, ПРИНЯТ И ЗАЛЕНДЁН 28.08 (D39.162):** дверь правок банка смонтирована синхронно и под пер-книжным мьютексом · ключи доехали до движка аргументом `--keys-file` (строка **211** ЗАКРЫТА) · дубль конвенции пути снят (строка **213** — осталась одна строка в дев-пути) · снята сквозная полоса прогресса вместе с её канонной половиной, минор **0.6.0** (строка **200** ЗАКРЫТА) · с провода сняты два поля упразднённой модели (`PD-399`). ⚠ Пункт «воркер решений» СНЯТ эрратой 27.08-з: синхронность держится, замер сессии её подтвердил. Фикс-раунд приёмки — 13 позиций, две мои диспозиции сессия ОПРОВЕРГЛА исполнением и я это принял (см. D39.162). **(3) `sqlc` отдельной сессией** (решение владельца 22.08, `BACKLOG.md` П-19, граница 41 запрос в пяти файлах) — механическая работа, не мешать её с содержательной. **(4) Живой прогон книги насквозь** — строка 202, гейт прежний: ПОСЛЕ холодного прогона движка. ⚠ **На владельце и блокирует пункт своей темы:** ⚠ **развилка владения `book.yaml` (199а) СНЯТА — с владельца убрана аудитом 28.08:** она растворена седьмым вариантом ещё D39.156 п.3 (конвенционные дефолты + движковый глагол), вход построен D39.158/D39.162, а вся строка 199 закрыта D39.166 и снесена с таблицы. Пять дней держалась на владельце как блокер решённого. ⚠ **ДВА ВОПРОСА, СНЯТЫЕ ВЛАДЕЛЬЦЕМ 28.08 (D39.165) — оставлены здесь как след, потому что реестр ожиданий о них не знал вовсе и это была дыра учёта.** Ответы: смена формы конвейера — форма «ЭПОХА» (событие книги, канон уже разрешает пересчёт через границу `structure_version`); единица продажи — цена от ОБЪЁМА исходника по `chapters.units_total` + потолок объёма в движке + хвост качества в тариф, калибровка гейчена строкой 202. Прежняя формулировка: (дыру учёта нашла разведка 28.08: нота сказала «ждёт слова владельца», а реестр ожиданий об этом не знал): **(i) смена формы конвейера на ЖИВОЙ книге** — что деплой обязан делать с книгой, начерченной по старой форме, когда форму сменили (носители `PD-403`/`PD-404`; ⚠ проект отвечал на СОСЕДНИЙ вопрос дважды — D39.152 п.4 и D39.153 §4б — и два ратифицированных ответа В КОМПОЗИЦИИ и дают дефект: липкость флага ратифицирована ПРОТИВ отката назад, но в обратную сторону она морозит счёт, и комментарий `platform/internal/pgstore/sink.go:405-408` описывает поведение, которого его же код НЕ делает); **(ii) `PD-410`** — платформа продаёт ГЛАВЫ, движок останавливается по ДЕНЬГАМ, и единица продажи не равна единице остановки · (г) **контракт: сквозная полоса прогресса ЗАКРЫТА** — минор **0.6.0** заленджен вместе с паком P9 (D39.162): доля одна на всю работу прогона через обе волны, подпись стадии объявлена ВТОРЫМ ограниченным исключением из границы «ничего о том, КАК переводится книга». ⚠ Остаток назван и НЕ забыт: платформа продаёт главы, а движку передаёт только `--ceiling-usd` — единицы разные, и это `PD-410` (архитектурный стоп зоны). Хвосты релеев (203) ждут следующего минора · (д) свободные бэкенд: 160 Этап 0, фикс-лист ФЧ строкой 197 (носителя-сессии нет по слову владельца); ⚠ строки 176/181/187 СНЯТЫ с таблицы при лендинге 20.08 — они закрыты ещё D39.149, а очередь всё это время звала 181 свободной работой (поймано ревью доков; остатки живут строками 194/196/197) · (е) фронтовый фрагмент pre-commit хука получает норму «жалуйся владельцу» — **решение владельца 21.08: ДА**, исполняется первым касанием зоны (пинг в её журнале). · (ж) **ПЕРЕЕЗД НА НОВУЮ МАШИНУ — стенд ПЕРЕСОБРАН 24.08 (№19), как поднимать — карта `README.md`, строки «Тулчейн» и `counts.py`.** Ubuntu 26.04 WSL2, пользователь `ubuntu-26`; тулчейн стоял НУЛЕВОЙ. Поставлено: Go 1.26.7 · golangci-lint 2.12.2 · Node 22.23.2 — тарболами вендоров со сверкой sha256 в `~/.local/opt` (симлинки в `~/.local/bin`); PostgreSQL 18.4 — рецептом самой зоны (`platform/docs/STACK_DECISIONS.md` §«Postgres на стенде без root»); `build-essential` + `python3-venv` — руками владельца, они единственные требовали root. **Обе батареи зелены под `-race`:** бэкенд `make battery` EXIT=0, линтер 0 issues, скипы только корпус-гейченные и хелперы; платформа `make check` EXIT=0, 18 пакетов, линтер 0 issues, **скипов 0**. Фронт: `npm ci` + `npm run check` проходит целиком (335 тестов), `vite build` собирается. Сквозняк проверен ЖИВЬЁМ: демон платформы → сид через настоящий интейк → движок распарсил книгу (3 главы) → дев-прокси фронта достучался до платформы. ⚠ **Что НЕ проверено:** ни одного платного вызова · браузеров Playwright нет ⇒ `check:full` не гонялся · локальной модели и GPU на машине нет вовсе (слово владельца 24.08: мерить нечем, ветка не закрыта) · OIDC не поднимался, вход только дев-логином. ⚠ Правка `.gitignore` владельца несла дефект — правило `books/` без якоря глушило ЛЮБОЙ каталог `books`, включая живой пакет `platform/internal/books/` (новый файл там уходил бы в игнор молча); заякорено в `/books/`, проверено пробой в обе стороны. Новые строки бэклога — **217** (пути полигона), **218** (дефолты корпуса в движке — ЗАКРЫТА и снята 27.08, D39.159 эррата-д), **219** (line-якоря D-лога съехали — ⚠ ОБЪЯВЛЕНА 24.08, а В ТАБЛИЦЕ ЗАВЕДЕНА только 27.08: три дня очередь ссылалась на несуществующий носитель) **Открытые обязательства, которых нет больше нигде:** веса `pro` — строка 172-г НЕ закрыта, прогон мерил классификатор на flash; дешёвый вход — спросить полигон, говорят ли что-то данные фазы Д (там pro гоняется редактором), отдельный замер не покупать без этого · `docs/experiments/00-provider-quirks.md` несёт протухшие цены DeepSeek («не изменились, flash $0.14/$0.28») — зона полигона, чинить пингом · санкция на платный прогон в архивном промте пака честности была РАЗОВОЙ · **норма плотности комментариев — ОТВЕЧЕНА владельцем 21.08:** режем ВОДУ, а не длину — счёт строк «пиши одну-две» негодный гейт и снят; уходит пересказ решений, провенанс и изложение исследования вместо ссылки, остаётся всё, что из одной функции не выводится, сколько бы строк ни заняло (норма переписана — `12-go-style-notes` §1; PD-255 платформы этим закрывается). ⚠ Метод-урок: её хендофф объявил «вопросов на владельце нет», а строка регистра просила именно решения — вопрос был закрыт ОБЪЯВЛЕНИЕМ, поймано ревьюером полноты выгрузки. Оркестраторов ДВА (решение владельца 07.08): этот — движок/платформа/фронт/доки; параллельный (РОЛЬЮ, без номера — счётчик один, D39.112 п.6) — приёмка полигона. Одновременно не запускаются; CURRENT-STATE ведут оба, чужие строки не трогают. Норма изоляции панелей — D39.113, гардрейлы в CLAUDE.md.) > - ⚠ **ОБЯЗАТЕЛЬСТВА ЛЕНДИНГА P9, часть 2 (28.08):** перевести половину 2 строки `PD-400` (внутрипроцессный мьютекс) в статус **`accepted-risk`** — граница v1 названа с условием («перестаёт держать в день второй реплики; лечение тогда арбитр в хранилище, не больший мьютекс») и ценой (ограничена убираемым шумом: холостая попытка у translate, «повтори позже» у глагола — ни ложного слова, ни денег); однорепличность — сегодняшний допуск всей зоны. Смена статуса — акт лендинга, строка сессией под это уже сформулирована. ⚠ Там же закрыть `PD-401` — лечение в дереве и зелёное. > - ⚠ **ОБЯЗАТЕЛЬСТВО ЛЕНДИНГА P9, записано 27.08, чтобы не потерялось второй раз:** при взятии дерева закрыть **`PD-166`** — её тело уже объявляет механизм НЕДОСТИЖИМЫМ (запись версии переехала из `Begin` в `effect`, одна транзакция с курсором; `Begin` — legacy), диспозиция «закрыть как построенное» написана, пин `TestTheChunkerVersionOfTheStreamReachesTheBook` (`pgstore/sink_test.go:618`) на месте, согласие получено ещё до рестарта — **не исполнена только смена статуса**. Проверка одной командой: `awk -F'|' '/^\| PD-166 \|/ {print $7}' platform/docs/DEFECT_REGISTER.md` → ` open `. Закрытие — акт лендинга, не работа пака (прецедент P6). ⚠ Сейчас регистр правит сессия P9 — параллельно не лезть. > - ⚠ **ЛЕНТА КОНТРАКТА за 27–28.08, свёрнуто аудитом 28.08 (D39.167):** **0.5.0** — D39.161 (снос отменённой пер-термной модели подписи + дверь правок банка) · **0.6.0** — D39.162/D39.163 (сквозная полоса прогресса; `Progress.stage` ВТОРЫМ и последним исключением границы) · **0.7.0** — D39.166 (покупка пере-прохода членом `RunRequest.re_pass`). Прожитые дампы приёмки P9 и P10 вынесены в `archive/PROGRESS-2026-08-backend.md` и в тела нот; здесь остаётся только лента версий. @@ -14,7 +14,7 @@ > - **Курс:** ОБЩНОСТЬ ✅ → КАЧЕСТВО БАНКА ✅ → ПАКЕТ-ЧЕКЕРОВ ✅ (D39.59–78) → **ФРОНТ-ЭРА** (D39.81–100: зоны живые, контракт API ратифицирован) → **шов/платформа/движковые блокеры построены** (D39.106–123). Хвосты курса живут строками: 16 (полная цена холодного старта не измерена: у `coldrun-a`/`coldrun-b` нет НИ ОДНОГО редакторского вызова, хотя сама edit-волна отработала и оплачена — 58 вызовов волновым драйвером 23–25.07 на малых прогонах, плюс 648 до драйвера; ⚠ прежняя редакция писала «не гонялась НИ РАЗУ» — абсолют ложен, испр. оркестратором №19 по леджерам `~/books/gu-zhenren/**/*.db`; держит и оси голоса 13б/24) · 46 (дизайн заморожен D39.92/93, промт ждёт выдачи) · coldrun-b фаза C заморожена чекпойнтом легитимно (D39.86; эталон денег/поведения — coldrun-a, read-only); развилка 0731 решена и исполнена (D39.87/91, код `553f1a3`). > - **Горизонт (D39.62/67, освежён D39.95):** **ДОБОР ИДЕАЛА** (первым прогоном: оси голоса 24 · авто-режим · цена 16; жильцы ролей решаются ДО прогона эксп-22 — строка 149 · веса K1–K12 13а · вне-претрейн чекпоинт 55; остаток арбитража банка = рецензент спорных кластеров при ре-пробе 74 — D39.102) → ВТОРАЯ ПАРА живьём (ja→ru; преп 81) → МАСШТАБ → пилот Ф2.5 (гейт резюме-строки 80; строки 62–68, 85) → Ф3 ридер-IDE (69–71). **Стоячие:** ToS-триггер 25.10 · Ш-2 до go1.27 (⚠ + x/text Unicode 17 тем же тулчейном — строка 119, реестр §Б-108 справочника якорей) · платные прогоны разблокированы (проба провода — D39.97, конфиги 112 залендены). > - **Стек (полная карта роль→модель→конфиг→квирки — [STACK.md](STACK.md), D39.126):** draft deepseek-v4-flash thinking-ON `low` **⚠0731** → терминолог (та же модель) → editor deepseek-v4-pro БИЛИНГВ ИНТЕРИМ (топология ПОДТВЕРЖДЕНА при неразличимости жильцов — D39.117; закон-блок ОБЯЗАТЕЛЕН — строка 134; glm-5 резерв; вахта маппинга эффорта pro — §Б-108) → судья gemini (Ф2, в движке НЕ построен — строка 33); канал B Mistral+grok; ~$0.85/ранобэ (D30.4, пере-калибровка при следующем платном прогоне). ⚠ **Вендор-факты 13–15.08 (пере-пин ИСПОЛНЕН и ПРИНЯТ, D39.137):** таблица цен запинена ПИКОМ (flash 0.44/1.32 · pro 1.32/3.96/кэш-хит 0.044 за 1M; счёт шиппинг-c1 = 100% DeepSeek ⇒ ×4.2–4.4 в пике / ×2.1–2.2 в долине — замер по трём прогонам); ⚠ про эффорт pro НОСИТЕЛИ ПРОТИВОРЕЧАТ и это НЕ РЕШЕНО (испр. 23.08 №19: прежняя редакция утверждала здесь одну сторону — «эффорт стал настраиваемым low/high/max, квирк 3а устарел»): квирк-канон и `STACK.md` говорят «ручки у pro НЕТ», замера поведением после 13.08 нет ни у одной стороны, и по гардрейлу владельца такое закрывается ВЕНДОР-СВЕРКОЙ, а не выбором стороны; до неё в силе канон, разбор — `STACK.md` строка редактора; ⚠ ВЕСА pro сменились под тем же слагом (V4-Pro-0813, класс D39.61) — вахта-риг готов (остаток 172); ⚠ посылка интерим-редактора «dspro дешевле glm» в пике ПЕРЕВЁРНУТА (×1.26 дороже — вход ратификации фазы Д, D39.137 п.4); покупки фазы Д на deepseek до 16.08 16:00 UTC — по старым ценам. ⚠ **ВЕСЬ банковый контур (банкнота+терминолог+классификатор) в shipping-c1 НЕ включён** — жив ран-локальным конфигом книги (строка 140; сверка STACK.md 09.08 — факт шире прежней декларации «одна банкнота»); эскалация в shipping за `budget_usd: 0` (STACK.md §примечания). -> - **ЕДИНЫЙ БЭКЛОГ — секция «Бэклог» ниже** (одна таблица, единственный трекер; каждая петля обязана иметь диспозицию: решено / отложено-с-записью / отклонено; ведёт оркестратор). **СЧЁТ ОЧЕРЕДИ на 28.08 (скриптом по таблице — `python3 docs/scripts/counts.py`; обновлять при каждом лендинге):** всего **175** строк · зона бэкенд **83** строго / **120** широко (175 пере-скоуплена D39.134; 176/177 заведены 15.08; 178 — D39.136; 179 ЗАКРЫТА D39.138; 180/181 — D39.137; 182 — wire-батч, аудит 15.08; 183–187 — контракт-ревью 28, D39.138; 188–190 — аудит бэклога, D39.140; 191/192 — модель подписи банка и пост-ридинговый цикл, D39.144; 193 — молчащие дыры выдачи, D39.147; 194–196 — приёмка пака честности, D39.149; 197 — фикс-лист ФЧ, D39.150; **198–204 — приёмка P7 и слова владельца 20.08, D39.153; 205 — слепота гейта якорей, аудит 21.08; 206–207 — приёмка P8-FIX, D39.154; 208–209 — аудит доков 22.08 (потеря глоссария на Gemini, риг живых проб); 210 — трассировка цепи банка 22.08 (род не производится авто-путём); **211–213 — консилиум шва 22–23.08: ключи провайдеров не доезжают до движка на SaaS (блокер 202), нестрогий загрузчик сида, дубль конвенции пути у платформы — 211 ЗАКРЫТА и 213 сужена лендингом P9 (D39.162); 214–216 — разбор журнала трассировки банка: подпись не оставляет следа, $0-репин недостижим с платформы, три несущих сценария не проверены живым движком; 217–218, 220 — аудит переезда машины 24.08: абсолютные пути полигона, дефолты корпуса в движке, бутстрап гейта доков чужой замороженной зоной; 219 (съехавшие line-якоря D-лога) ЗАКРЫТА тем же заходом — D39.157 п.6**) (⚠ колонки счётчик читает С КОНЦА — испр. D39.116) · **блокеров очереди 0**, платные прогоны разблокированы · «скоро» **49** (перечень — грепом по таблице, рукописный список снят D39.126) · гейт-строки эксп-22: 55·153 (плюс 150 — руки владельца); строка 5 — остаток гейчен ре-пробой 74, носитель события теперь строка 188 (аудит D39.140); остальное «когда-нибудь». ⚠ Счёт — НИЖНЯЯ граница долга, не потолок (разбор — легенда таблицы ниже). +> - **ЕДИНЫЙ БЭКЛОГ — секция «Бэклог» ниже** (одна таблица, единственный трекер; каждая петля обязана иметь диспозицию: решено / отложено-с-записью / отклонено; ведёт оркестратор). **СЧЁТ ОЧЕРЕДИ на 28.08 (скриптом по таблице — `python3 docs/scripts/counts.py`; обновлять при каждом лендинге):** всего **177** строк · зона бэкенд **85** строго / **121** широко (175 пере-скоуплена D39.134; 176/177 заведены 15.08; 178 — D39.136; 179 ЗАКРЫТА D39.138; 180/181 — D39.137; 182 — wire-батч, аудит 15.08; 183–187 — контракт-ревью 28, D39.138; 188–190 — аудит бэклога, D39.140; 191/192 — модель подписи банка и пост-ридинговый цикл, D39.144; 193 — молчащие дыры выдачи, D39.147; 194–196 — приёмка пака честности, D39.149; 197 — фикс-лист ФЧ, D39.150; **198–204 — приёмка P7 и слова владельца 20.08, D39.153; 205 — слепота гейта якорей, аудит 21.08; 206–207 — приёмка P8-FIX, D39.154; 208–209 — аудит доков 22.08 (потеря глоссария на Gemini, риг живых проб); 210 — трассировка цепи банка 22.08 (род не производится авто-путём); **211–213 — консилиум шва 22–23.08: ключи провайдеров не доезжают до движка на SaaS (блокер 202), нестрогий загрузчик сида, дубль конвенции пути у платформы — 211 ЗАКРЫТА и 213 сужена лендингом P9 (D39.162); 214–216 — разбор журнала трассировки банка: подпись не оставляет следа, $0-репин недостижим с платформы, три несущих сценария не проверены живым движком; 217–218, 220 — аудит переезда машины 24.08: абсолютные пути полигона, дефолты корпуса в движке, бутстрап гейта доков чужой замороженной зоной; 219 (съехавшие line-якоря D-лога) ЗАКРЫТА тем же заходом — D39.157 п.6**) (⚠ колонки счётчик читает С КОНЦА — испр. D39.116) · **блокеров очереди 0**, платные прогоны разблокированы · «скоро» **48** (перечень — грепом по таблице, рукописный список снят D39.126) · гейт-строки эксп-22: 55·153 (плюс 150 — руки владельца); строка 5 — остаток гейчен ре-пробой 74, носитель события теперь строка 188 (аудит D39.140); остальное «когда-нибудь». ⚠ Счёт — НИЖНЯЯ граница долга, не потолок (разбор — легенда таблицы ниже). > - Архивы хроники: `archive/PROGRESS-2026-07-04-10.md` (D31) · `-10-13` (D39.6-гигиена) · `-13-25` (стройка паков 11–16, rerun2) · **`-25-31` (паки 17–20 · мини-прогон · полигон-пакеты 5–8 · ToS · холодный прогон; D39.26–58)** · **`-08-01-02` (сессии №9/№10: общность · качество банка · coldrun-b · открытие фронта/платформы; D39.59–90, срез D39.105)** · **`-08-02-04` (сессии №11–№13: ручки эффорта · стандарты · контракт API · платформа P0 · банк-арбитраж; D39.91–105 + снимок шапки эры №15)** · `-08-04-09` (пинги закрытых паков эры №15) · **`-08-14-15` (закрытые бэкенд-записи №16–17: эмиттер шва · migrate · пере-пин DeepSeek; вынесено D39.139)**. Записи ниже — живой хвост (№16+, эра D39.124+; подрезка D39.139). ## 📌 СОСТОЯНИЕ ПАКОВ (оркестратор №19, 29.08) @@ -25,15 +25,12 @@ → переподключение `401`, номер кадром не потреблён); деньги сведены ТРЕМЯ путями. Единственный FAIL — `TestARunIsBoundedByItsOwnCgroup`, причина в хосте (`PD-423`), пакет диффом не тронут. -**Движковый пак «деньги» — НЕ ПРИНЯТ, на доработке.** Батарея пере-прогнана мной: 21 пакет из 21, -0 FAIL, полнота проверена списком. Блокирует денежный охотник: **потолок объёма на майнящей книге не -держит** — `planVolume` считает edit-снапшот один раз до волн (`volume.go:353-355`), free-юниты -допускаются вне гранта (`:259-265`), а bank-mining пере-сидит банк посреди прогона (`mining.go:242`), -после чего снапшот берётся заново (`waverun.go:203`); замер: `--max-units 1` дал 4 вызова вместо 2, и -отчёт назвал переоплаченные юниты «проехавшими за $0». Плюс четыре подтверждённых прогоном: флагнутый -юнит съедает слот и исчезает из остатка навсегда · добавление стадии превращает дочитанную книгу в -«никогда не доставляли» · прерванный между волнами юнит тратит слот дважды · терминолог -переплачивается на каждой покупке (~1.5× стоимости юнита при покупке по одному). +**Движковый пак «деньги» — ПРИНЯТ И ЗАЛЕНДЖЕН** (D39.170). Приёмка исполнением: моя батарея — 21 +пакет из 21, 0 FAIL, полнота сверена списком `go list` против вердиктов; линтер `0 issues`; четыре +ключевых пина зелёные поимённо. **Две мои мутационные посадки на сценарном тесте:** снятие вызова +пере-плана → структурный отказ адресным сообщением; снятие вызова И гарда → дефект целиком, всеми +четырьмя денежными утверждениями (4 вызова при гранте 1, $0.007280 вместо $0.003640, две доставленные +главы оплачены дважды и названы бесплатными). Значит тест ловит ПЕРЕОПЛАТУ, а не факт вызова. ⛔ **Проводка `--max-units` в платформу остаётся ГЕЙЧЕНОЙ** — основание усилилось: дело не только в падении второй покупки без `--resnapshot`, а в том, что С `--resnapshot` потолок пробивается. @@ -194,7 +191,9 @@ | 228 | **Отклонённая поверхность возвращается АЛИАСОМ уцелевшей строки — движок не держит того, что канон уже обещает** (находка воркфлоу-ревью P9 в форме Д1, УЗКО пере-сформулирована бэкенд-сессией 28.08 и принята приёмкой): канон говорит дословно «declining a surface removes EVERY window of that surface» (`docs/architecture/14-api-contract/openapi.yaml:2051`), а эмиссия майнера энтити-широка (`backend/internal/pipeline/miner_emit.go`, `clusterTouches`), тогда как фильтр авто-банка ключуется только по собственному `src` строки (`backend/internal/pipeline/mining.go:662`). ⚠ **Правильная форма — снять АЛИАС со строки, а не снести строку:** расширение `decline` до энтити противоречило бы ратифицированному контракту, и именно поэтому бэкенд-сессия применила право §9 и НЕ чинила это попутно. Предмет — банковая онтология (`18-bank-ontology.md`), не тихая порча | бэкенд | скоро (гейт: заказ по читающей стороне банка) | отдельный заказ узкой формы | воркфлоу-ревью P9; форма — бэкенд-сессия 28.08, D39.164 | | 229 | **Снапшот не фолдит модель ВНУТРЕННИХ гейтов — флип провода под неизменным `request_hash`** (самонаходка бэкенд-сессии 28.08, подтверждена приёмкой): снапшот фолдит `Capability` СТАДИЙНЫХ моделей и их эскалации (`backend/internal/pipeline/snapshot.go:316-340`), но модель `gates.terminology.model` / `gates.repair.model` (`backend/internal/config/internal_call.go:72`) не фолдится сознательно — а терминолог шлёт ДВА системных сообщения, так что смена оси `capabilities.system_messages` у провайдера, которым пользуется только гейт, меняет байты запроса при неизменном хеше: тихий false-hit класса D5.2. ⚠ **Сегодня ЛАТЕНТНА и денег не стоит — проверено приёмкой: гейта `terminology` нет НИ В ОДНОМ конфиге репозитория** (`grep -c terminology configs/pipeline-c1.yaml` = 0). Триггер починки — день, когда гейт включат с провайдером, объявляющим НЕдефолтную возможность. ⚠ Цена лечения — денежная: фолд гейт-моделей сдвигает хеши и обесценивает чекпойнты; дешёвая форма — фолдить ТОЛЬКО недефолтное (приём `omitempty`, прецедент `MinMaxTokens`), тогда сегодняшние снапшоты остаются байт-равными | бэкенд | когда-нибудь (гейт: включение внутреннего гейта либо следующее касание снапшот-контракта) | правка снапшот-контракта | самонаходка бэкенд-сессии, D39.164 | | 230 | **Инертный `decline` подписанного сид-терма отвечает `already_applied` вместо единственной работающей инструкции** (названный размен пака «тихая порча», D39.164): сузив отказ по поверхности ради СХОДИМОСТИ повтора, движок потерял поучение в одном углу — когда отказ и записан, и по-прежнему инертен против `glossary_seed`, пользователь получает «уже применено» вместо «убери терм из сида». Сходимость сочтена более тяжёлой обязанностью (на ней стоит вся раскладка класса 15 и синхронная дверь платформы), но размен РЕАЛЕН. **Форма закрытия — поле отчёта со стоячим фактом**, то есть аддитивная правка формы шва: платформенный `BankReport` — аллоулист, лишнее поле на провод не уедет само | бэкенд + контракт | скоро (с ближайшим касанием отчёта двери) | аддитивное поле отчёта | размен пака «тихая порча», D39.164 | -| 231 | **Смета пере-прохода не доезжает до покупателя: движковой оценки ВНЕ прогона не существует** (названная цена минора 0.7.0, D39.166). `bank-apply` пишет только ФАЙЛЫ решений, а `status --json` считает ре-билл от СОХРАНЁННОГО глоссария (`backend/internal/pipeline/status.go:815-826`=`projectStoredMemory materializes r.memory from the STORED glossary`) — свёртка происходит внутри СЛЕДУЮЩЕГО `translate`, поэтому сразу после правки движок честно отвечает «ничего не двигалось». Следствие: показать «затронуто N юнитов» ДО покупки нечем, и согласие сегодня даётся ДЕНЬГАМИ (потолок пере-прохода = холд). **Форма лечения — движковый глагол либо флаг «свернуть банк и оценить, ничего не покупая»**; родня строки **124(в)** (`resnapshot --dry-run`, поименован там же). ⚠ Поля `rebill_units`/`rebill_usd` СНЯТЫ с аллоулиста шва до появления потребителя — вернутся с этим глаголом | бэкенд (+платформа вторым концом) | скоро | движковый глагол + минор контракта | D39.166; посылка D39.165 §3 исправлена эрратой 28.08-к | +| 232 | **Ось «свежий/пере-делка» выведена из ПОЛНОТЫ СТРОК, а не из факта отгрузки** (D39.170, находки охотника 3 и 4). Следствия ДЕНЕЖНЫЕ на слух покупателя: добавление стадии в пайплайн превращает ДОЧИТАННУЮ книгу в «3 unit(s) NEVER delivered» и приглашает купить её снова; юнит, прерванный между волнами (signature stop, денежный потолок, Ctrl-C), второй раз считается свежим и тратит слот гранта повторно — замерено 4 купленных юнита → 2 главы. Носитель у движка УЖЕ есть: реестр анонсов `events_outbox.once_key` (`backend/internal/pipeline/events.go:396`=`unitOnceKey is the identity of one announcement`), ключ `unit:<книга>:<волна>:<глава>:<юнит>`, монотонный на всю жизнь книги и переживающий и добавление стадии, и обрыв между волнами. Не хватает ЧИТАЮЩЕГО метода поверх готовой константы (`backend/internal/store/outbox.go:31`=`SELECT 1 FROM events_outbox WHERE once_key = ? AND once_key <> ''`, с уже написанным объяснением, почему хвост `<> ''` синтаксически обязателен). ⚠ РАЗВИЛКА для промта, которую сессия не назвала: ключ несёт ВОЛНУ, значит «юнит отгружен» — факт per-wave, и заказ обязан сказать, какая волна считается отгрузкой, иначе исполнитель решит это молча | бэкенд | K | Читающий метод стора + перевод оси на факт отгрузки; отдельный пак | приёмка D39.170 | +| 233 | **Трата терминолога вне объёмного потолка масштабируется КНИГОЙ, а не грантом** (D39.170). Замер: три последовательные покупки по одному юниту на четырёхглавной книге дали три полнокнижных консолидации по $0.005460 каждая — покупка одного юнита обходится в ~1.5× стоимости самого юнита. Книга на 500 юнитов, проданная по одному, оплатит 500 полнокнижных проходов. Место траты — в ЦЕНЕ, а не в потолке (решение подтверждено), но при мелкой нарезке продажи она перестаёт быть накладной и становится основной статьёй: это ВХОД В КАЛИБРОВКУ ЦЕНЫ, а не сноска | бэкенд | K | Учесть в модели цены при следующей калибровке; либо чекпойнт консолидации, переживающий покупку | приёмка D39.170 | +| 234 | **Основание платформы «не брать `rebill_*` через шов» УСТАРЕЛО этим же лендингом** (D39.170). `platform/internal/ingest/resync.go:37-43` объясняет отказ ТАЙМИНГОМ: «status проецирует СОХРАНЁННУЮ память, и сразу после `bank-apply` он честно читает ноль». Движковый пак это починил: `foldMemoryForRead` стал ПЕРВЫМ ответом читающего пути, `projectStoredMemory` понижена до фолбэка (`backend/internal/pipeline/status.go:817-824`=`IT IS NO LONGER THE READ PATH'S FIRST ANSWER`). Комментарий чужой зоны теперь несёт снятую посылку и будет прочитан следующей сессией как действующий довод. ⚠ Проводка полей при этом НЕ разблокирована: она гейчена вместе с `--max-units` | платформа | K | Пинг зоне платформы + строка её регистра; проводка — после снятия гейта `--max-units` | приёмка D39.170 | | 217 | **Скрипты полигона захардкодили АБСОЛЮТНЫЕ пути двух корней и не работают на этой машине вовсе.** ДВА РАЗНЫХ слома. **Корень репо** — смена пользователя: литерал в 121 файле, из них 104 дословно `REPO = Path("/home/ubuntu/projects/textmachine")`, портабельных форм всего 5. **Корень книг** — оба события порознь: литерал `/home/ubuntu/books` 86 файлов (пользователь), портабельный `Path.home()/"books"` 89 файлов (переезд каталога; на прежних машинах работал). Объединение 173 файла. Пользователь теперь `ubuntu-26`, книги переехали в `<репозиторий>/books` ⇒ ломается и то и другое, включая ЖИВОЙ гейт фазы Д `eval/conformance.py:48`=`REPO = Path("/home/ubuntu/projects/textmachine")`. Живых носителей 26 по корню книг (среди них портабельная форма преобладает 20:6) плюс корневые гейты и сборщики корпусов; остальное — скрипты закрытых эксп-12–16, их владелец велел ВЫБРОСИТЬ, а не чинить (D39.157 п.4а). Форма правки — за зоной; приор оркестратора: один модуль путей, корень резолвится МАРКЕРОМ вверх по дереву (движок так и делает — `backend/internal/miner/miner_parity_test.go:29`=`derived from the repository MARKER`), книги от него, обе ручки перекрываются env | полигон | скоро (блокирует любой прогон зоны) | пак путей полигона | переезд машины 24.08, №19 | | 219 | **Line-якоря в тела D-лога обречены съезжать, и механизм именно в дисциплине эррат.** Эрраты вписываются в КАРТУ ШАПКИ (append-only, D23.3), то есть в начало файла, — значит каждая эррата сдвигает номера строк ВСЕХ тел ниже, и любой якорь вида `05-decisions-log.md:NNN` умирает молча. ⚠ Замерено на себе 27.08: три эрраты за сессию убили якоря в `17-seam-inbound-law.md:120` и `research/25-seam-cold-review.md:5` (оба целили в `:232`, тело уехало на `:239`). Лечение — не пере-нацеливание (оно повторится через эррату), а СМЕНА ФОРМЫ якоря: в тела D-лога целиться номером ноты (`^## D39.106`), который стабилен навсегда, а не строкой. ⚠ Строка ОБЪЯВЛЕНА в CURRENT-STATE 24.08 и до 27.08 в таблице НЕ СУЩЕСТВОВАЛА — очередь три дня ссылалась на носитель, которого нет; поймано свипом якорей при выдаче контрактного минора | оркестратор | скоро (растёт с каждой эрратой) | `--lint` учит форму `файл:^## D<номер>`, живые якоря в D-лог переводятся на неё | ревью доков 24.08, механизм и отсутствие носителя — 27.08, D39.160 | | 220 | **Репо-широкий гейт доков бутстрапится пакетным менеджером ОДНОЙ зоны, и та заморожена.** Диспетчер `.git/hooks/pre-commit` зоно-нейтрален по построению (`for hook in */scripts/githooks/pre-commit`, `#!/bin/sh`, node не нужен) и судит `CLAUDE.md` + `docs/**` + `platform/docs/**` + `frontend/docs/**` (`docs/scripts/counts.py:500`=`ROOT / "CLAUDE.md"`), но единственный его УСТАНОВЩИК — `frontend/scripts/githooks/install.mjs`, вызываемый ключом `prepare` из `frontend/package.json`. Охват и бутстрап не совпадают ⇒ инструмент решает свою задачу не в полной мере. Проверено исполнением 24.08: на этой машине `.git/hooks/` содержал только `*.sample`, гейт МОЛЧАЛ, ничего не проверив, и поймал первые четыре дефекта только после ручной установки; сессия бэкенда или доков `npm install` не делает никогда. Приор: зоно-нейтральный установщик в `docs/scripts/githooks/` как ЕДИНСТВЕННЫЙ писатель диспетчера, `prepare` фронта зовёт его — фронтовая половина выписана пингом №19 в зонный журнал фронта. ⚠ Ручная установка — временная мера, пока строка открыта; записывать её как порядок работы ЗАПРЕЩЕНО (это ровно тот обход, цену которого называет шапка `CLAUDE.md`) | доки + фронт | скоро | зоно-нейтральный бутстрап + правка `prepare` при разморозке | аудит переезда 24.08, №19 | @@ -271,7 +270,7 @@ > пере-прохода) · **229** (снапшот не фолдит модель внутренних гейтов, латентна) · **230** (размен > «сходимость против поучения») · **141**-остаток · **131**. Плюс потолок объёма из `D39.165` §1б. -### 📝 ЗАПИСКА-ПЛАН — пак «деньги» (промт `docs/BACKEND_MONEY_PACK_SESSION_PROMPT.md`, D39.165 §1б + строка 231), сессия `textmachine-e4`, 28.08 +### 📝 ЗАПИСКА-ПЛАН — пак «деньги» (промт `docs/archive/prompts/BACKEND_MONEY_PACK_SESSION_PROMPT_2026-08-29.md`, D39.165 §1б + строка 231), сессия `textmachine-e4`, 28.08 **Скоуп:** `backend/` — §3.1 потолок ОБЪЁМА рядом с денежным · §3.2 смета пере-прохода без покупки. `platform/` не трогаю (проводка `CeilingChapters` — не мой заказ, честная граница §0 промта принята). @@ -318,7 +317,7 @@ **НЕ делаю:** калибровку $0.03 (строка 202) · цену от объёма исходника · строки 228/230/141/131 · словарь кодов выхода · словарь потока событий · `platform/` · коммиты. Незакоммиченную работу полигона не касаюсь. -### ✅ ИТОГ — пак «деньги» (промт `docs/BACKEND_MONEY_PACK_SESSION_PROMPT.md`, D39.165 §1б + строка 231), сессия `textmachine-e4`, 29.08 +### ✅ ИТОГ — пак «деньги» (промт `docs/archive/prompts/BACKEND_MONEY_PACK_SESSION_PROMPT_2026-08-29.md`, D39.165 §1б + строка 231), сессия `textmachine-e4`, 29.08 **Батарея зоны:** `make battery` — **21 пакет из 21, 0 FAIL**, линтер **0 issues**, `test -race` зелёный целиком. ⚠ **Проверяю ПОЛНОТОЙ СПИСКА, а не отсутствием `FAIL`, и это правило стоило мне двух ложных выводов.** Первый «зелёный» прогон был на 20 пакетов из 21 с `FAIL` в хвосте, который обёртка подала как `exit code 0`. Сверка: `comm -23 <(go list ./...) <(grep -E "^(ok|\?)" <лог>)` — пусто. @@ -454,6 +453,32 @@ — **тройное повторение самого дорогого расчёта.** Обрезанный прогон пере-рендеривал всю книгу до трёх раз: в `planVolume` и в каждой из двух проекций гейта, при том что между ними ничего не менялось. Заведён мемо, **срок жизни которого — жизнь БАНКА**: `materializeBanks` его сбрасывает, потому что переживший свой банк мемо отдал бы хеши банка, которого у прогона больше нет, — ровно тот класс тихой неверности, ради устранения которого пак и существует. Мемо — скорость, а его инвалидация — корректность, поэтому запинена именно она (`TestTheWireHashMemoDiesWithItsBank`). — **предупреждение про майнящие книги не было покрыто ничем** — а незакрытое тестом предупреждение молча перестаёт эмититься. Закрыто, вместе с обратной стороной: на книге БЕЗ потолка оно не должно возникать. +#### Четвёртый круг — денежный охотник приёмки: 6 находок, 3 вылечены, 3 переданы + +**1. ⛔ БЛОКИРУЮЩАЯ, и она пробивала саму цель пака — ВЫЛЕЧЕНА.** `planVolume` классифицирует ДО черновой волны, но между волнами стоит стоп майнинга, который **пере-сеивает банк прямо посреди прогона** (`mining.go`, ветка авто-продолжения), после чего `waverun` пере-считывает edit-снапшот. А free-юниты допускаются ВНЕ гранта — потому что бесплатная работа ничего не стоит. Значит каждый из них судился по снапшоту, который прогон сам же и заменил, и любой, чьи инжектируемые байты новый банк изменил, получал свежий ПЛАТНЫЙ вызов редактора, не разрешённый никаким грантом, — а строка отчёта называла его «rode along at $0». Охотник замерил: 4 вызова при гранте 1, $0.007280 объявлены бесплатными. +**Лечение — не гард, а ПЕРЕ-ПЛАН:** вопрос «бесплатен ли юнит» задаётся заново, против снапшота, который стал реальным, в последний момент перед редакторской волной — то есть «проверять перед единицей» по-прежнему держится. Ставший платным берёт слот, если он есть; если нет — его редактура НЕ идёт, и он честно отчитывается как доставленный-но-не-пере-сделанный. +⚠ **И сделано СТРУКТУРНО, а не вызовом:** волна ОТКАЗЫВАЕТСЯ работать со скоупом, спланированным против другого снапшота. Без этого удаление вызова пере-плана оставляло всю батарею зелёной, а потолок молча переставал держать — я это проверила мутацией. Теперь удаление вызова роняет прогон с адресным сообщением. +**Доказано:** `TestFreeUnitsAreReJudgedWhenTheBankMovesMidRun` (снапшот не двигался → ничего не меняется; двигался → ни одного «free», ровно один слот занят, остальные удержаны и отказаны `allowsUnit`) + `TestTheEditWaveRefusesAStalePlan`. Две мутации. +⚠ **Своей фикстурой конечное следствие я не воспроизвела — ПЯТЬ попыток, — и разбор этого провала оказался ценнее самих попыток.** Две причины, обе названы приёмкой и обе проверены мной на своём дереве. +**Причина 1: я искала не на той стороне дельты.** У mined-дельты две стороны. **WHICH** (список термов) майнится из исходника ВСЕЙ книги — он полон после первой покупки и больше не двигается; отсюда мой неверный вывод «однородный источник ничего не даст». А байты банка двигает **WHAT** — `dst` каждого терма, и она **draft-side**: складывается из банкнот, которые модель уже выдала (`mining.go:109`), **то есть растёт с каждой покупкой ДАЖЕ при полностью однородном источнике**. Я меняла ТЕКСТ глав, а надо было менять то, что ПРЕДЛАГАЮТ ЧЕРНОВИКИ. +**Причина 2, из-за которой попытка (5) не могла сработать в принципе, — спойлер-гейт `since_ch`** (`membank/memory.go:640-647`, жёсткое `chapter < since_ch`). Вводя терм поздними главами, я выталкивала его `since_ch` за пределы уже доставленных — и он не попал бы в их инъекцию НИКОГДА, какой бы `dst` потом ни получил. ⚠ **Условия тянут в РАЗНЫЕ стороны:** естественный способ заставить дельту расти (поздние главы вводят новый терм) ровно этим и убивает достижимость. Дефекту нужна ОБРАТНАЯ асимметрия: терм присутствует в РАННЕМ исходнике, а его рендеринг приходит поздно — то есть асимметрия ПОВЕДЕНИЯ МОДЕЛИ, а не текста. +**Пере-проверила зондом приёмки у себя:** `方源 dst="" since_ch=3` → `dst=Фан Юань since_ch=3`, банк двинулся, а `content_hash` главы 1 `MOVED=false`, вызовов ровно два. То есть мой прогон был верен, а гипотеза о причине — нет. +⚠ **Диагностика на будущее, две строки:** дампить авто-банк между покупками и смотреть у терма ДВА поля — что `dst` перешёл из пустого в непустой **и** что `since_ch` ≤ номера уже доставленной главы. Второго я не проверяла. + +**Сценарный тест теперь есть** — `TestAMidRunBankMoveNeverBillsBeyondTheGrant` (рецепт приёмки: четыре главы, побайтово одинаковые, различаются только банкнотами). **Я пере-проверила его двумя своими посадками, а не приняла на слово:** снятие вызова пере-плана падает структурным отказом; снятие вызова И гарда воспроизводит дефект целиком — **4 вызова вместо 2, $0.007280 вместо $0.003640, обе уже купленные главы пере-отредактированы, и отчёт зовёт их «2 rode along at $0»**. +⚠ **Оба пина стоят рядом и не заменяют друг друга** (различение оркестратора, точнее моего): инвариантный утверждает «пере-план вызывается», сценарный — «покупатель не платит второй раз», а платит покупатель именно за второе. +⚠ **Жёсткое равенство `== 2×грант` оставлено намеренно:** гейт терминолога в фикстуре выключен явно, счёт детерминирован, и равенство ловит не только перерасход, но и молчаливую ПОТЕРЮ оплаченной работы. +⚠ **Условие достижимости УЗКОЕ, и это записано с обеих сторон:** нужна конъюнкция трёх условий (терм на WHICH-списке · его `dst` приходит поздно · `since_ch` покрывает уже доставленную главу). В фикстуре приёмки два терма из трёх её не выполняют и остаются инертными. Оценку «покупка №50 переоплачивает 490 юнитов» автор находки признал ПОТОЛКОМ ТЯЖЕСТИ, когда сработало, а не ожидаемым случаем; частоту на реальной книге никто не мерил. На диспозицию не влияет — лечение структурное и покрывает класс независимо от узости входа. + +**2. ⛔ Флагнутый юнит объявлялся доставленным — ВЫЛЕЧЕНА.** Допуск решается ДО работы (этим и ограничиваются деньги), но решение ЗАПЛАТИТЬ за юнит — не факт существования главы: юнит может вернуться флагнутым и не отгрузить ничего. Отчёт печатал «2 NEW unit(s) delivered» поверх одного читаемого юнита. **Лечение: счётчики выравниваются ПОСЛЕ волн, по фактическим исходам** (`reconcile`), и заведён отдельный счётчик «оплачено, но флагнуто» с прямой формулировкой: деньги потрачены, текста нет, покупкой не чинится — это `redrive`. Тест + мутация. + +**6. Подсказка обещала то, чего грант не сделает — ВЫЛЕЧЕНА.** `rebill_output_units` подавался как «та единица, в которой считает `--max-units`» — правда, и в одиночку вводит в заблуждение: грант идёт СПЕРВА в недоставленное, поэтому на книге, где оно есть, `--max-units N` доставит новое и не пере-сделает ничего. Оговорка была в коде, но не там, где читает оператор. Теперь в подсказке. Тест. + +**3 и 4 — ПОДТВЕРЖДАЮ, НЕ вылечила, и говорю почему.** Обе об одном: ось «свежий/пере-делка» выведена из ПОЛНОТЫ СТРОК, а не из факта отгрузки. Добавление стадии делает доставленную книгу «никогда не доставлявшейся» (#3); юнит, прерванный между волнами, второй раз считается свежим и тратит слот повторно (#4). **Правильный носитель у движка уже есть — реестр анонсов (`events_outbox.once_key`, `unitOnceKey` = книга+волна+юнит), то есть собственное определение «этот юнит был отгружен читателю», монотонное на всю жизнь книги.** Но читающего метода у стора нет (есть только `EnqueueOnce`, который ПИШЕТ), а `internal/store` я не трогала ни строкой за весь пак и не считаю правильным открывать его в хвосте четвёртого круга. **Это отдельный заказ на одну функцию чтения + перевод оси на неё.** +⛔ **ПОПРАВКА к моей же формулировке, внесена оркестратором при приёмке и принята.** Я написала «деньги целы: ошибается слово, а не потолок». Слишком мягко — **слово тут ПРОДАЁТ.** Отчёт, называющий дочитанную книгу «3 unit(s) NEVER delivered», через строку CLI приглашает их купить, и покупатель платит за уже полученное. Деньги целы В ДВИЖКЕ — потолок держит, счёт вызовов верен, — но не в том, что увидит платформа и предъявит покупателю. Диспозиция не меняется (предмет отдельный, `internal/store` в хвосте пака не открываю), но цена ошибки — покупательская, а не редакторская, и в строке заказа стоит именно так. + +**5 — цифру принимаю, договорённость в силе, но она теперь важнее.** Терминолог переигрывается каждой покупкой: три покупки по 1 юниту дали три полнокнижных консолидации по `usd=0.005460` каждая, при том что сам юнит дешевле. **Накладные масштабируются КНИГОЙ, а не грантом** — книга на 500 юнитов, проданная по одному, оплатит 500 полнокнижных проходов. Мы согласились, что место этой траты — в цене; при мелкой нарезке продажи она перестаёт быть накладной и становится основной статьёй, и цена обязана это знать. + #### Сверка по буллетам промта — что закрыто и чем **§3.2, «родня 124(в)» — заказано сказать ПРЯМО, говорю:** да, моя форма закрывает и `resnapshot --dry-run`. `status` считает ре-билл при ЛЮБОМ дрейфе, не только банковом (`status.go`: проекция идёт при наличии строк, а не под `ConfigDrift`), поэтому после починки свёртки он И ЕСТЬ «оценить пере-проход, ничего не покупая» — ровно та способность, ради которой строка 124(в) поименована будущей дверью. **Отдельного глагола `resnapshot --dry-run` заводить не нужно.** Это удешевляет проект на один глагол и один документ. diff --git a/docs/README.md b/docs/README.md index f77cc49e..9259b0d8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -22,7 +22,7 @@ | Роль | Активный промт | Статус | |---|---|---| | Оркестратор | [ORCHESTRATOR_SESSION_PROMPT.md](ORCHESTRATOR_SESSION_PROMPT.md) | роль и нормы; счётчик роли — CURRENT-STATE | - | Бэкенд | [BACKEND_MONEY_PACK_SESSION_PROMPT.md](BACKEND_MONEY_PACK_SESSION_PROMPT.md) | **НАПИСАН, ждёт запуска владельцем.** Пак «деньги»: потолок ОБЪЁМА в движке рядом с денежным (D39.165 §1б) + $0-глагол сметы «оценить, ничего не покупая» (строка **231**). Оба рубежа пройдены: механический + опровергатель, **9 находок применены**, включая три фатальные — форма ответа через шов (словарь кодов ЗАМОРОЖЕН; решено: стоп по объёму = ЗАВЕРШЕНИЕ, код 0), выдуманный мной якорь, и вырожденное репро, повторявшее эррату 28.08-и. ⚠ Честная граница вписана в §0: проводка числа глав до движка паком НЕ заказана | + | Бэкенд | активного НЕТ | пак «деньги» ПРИНЯТ И ЗАЛЕНДЖЕН 29.08 (**D39.170**): потолок ОБЪЁМА оплаченной работы + читающий путь `status` сворачивает банк, поэтому смета доезжает до покупателя ДО покупки. Промт отработан — `archive/prompts/`. ⚠ Следующий заказ уже назван строкой **232** (ось отгрузки на `once_key`), проводка `--max-units` в платформу ГЕЙЧЕНА | | Платформа | активного НЕТ | пак **P11** ПРИНЯТ И ЗАЛЕНДЖЕН 29.08 (**D39.169**): `PD-379` закрыт живой пробой оркестратора (отзыв → поток гаснет за секунду кадром `session_ended` → переподключение `401`), видимость застрявших денег, денежная группа. Промт отработан — `archive/prompts/`. ⚠ Открытым остаётся денежный `major` `PD-425` (дверь коррекций теряет пост-verb факт при обрыве клиента), отсрочка на оркестраторе | | Полигон | [POLYGON_EXP2223_REDO_SESSION_PROMPT.md](POLYGON_EXP2223_REDO_SESSION_PROMPT.md) (отложенный — [POLYGON_PACKAGE4_SESSION_PROMPT.md](POLYGON_PACKAGE4_SESSION_PROMPT.md), строка 85) | фаза Д ИДЁТ; ⚠ живой носитель курса — в `eval/dovodka/`, какой именно называет зона (⚠ [POLYGON_PHASE_D_HANDOFF.md](POLYGON_PHASE_D_HANDOFF.md) — перекрытый снимок, читать не как курс) | | Фронт | активного НЕТ | **ЗОНА ЗАМОРОЖЕНА** (D39.136 п.2 + D39.147: разморозка отдельным словом владельца, не привязана к P7); перечень первого касания — в зонном журнале | diff --git a/docs/architecture/05-decisions-index.md b/docs/architecture/05-decisions-index.md index 816863c6..b7eafdab 100644 --- a/docs/architecture/05-decisions-index.md +++ b/docs/architecture/05-decisions-index.md @@ -1,4 +1,4 @@ -# Реестр D-нот — карта актуальности v2 (D1–D39.169; +# Реестр D-нот — карта актуальности v2 (D1–D39.170; > ⚠ **СЛАБОЕ МЕСТО, КОТОРОЕ БЫЛО ЗДЕСЬ (вписано 22.08, ЗАКРЫТО 24.08 — D39.157 п.6).** Колонка ТЕЛА > у нот D39.107…D39.123 говорила «жив», хотя тела уехали в слайс подрезкой D39.139; семнадцать строк @@ -230,3 +230,4 @@ | D39.167 | 28.08 | **Аудит документации (хребет)**: очередь пять дней звала активным исполненный промт и держала на владельце решённую развилку; регистр платформы расходился с актами лендинга ЧЕТЫРЬМЯ строками, одна major — класс «open, а лекарство в дереве» без гейта (строка 225); реестр D-нот потерял две колонки на восьми моих строках; справочники отстали от двух суток миноров. Журнал ужат 943 → 347 строк, прожитая проза бэкенда — в слайс. | жив | ЖИВОЕ: строка 225 (гейт класса); три участка обхода не пройдены | доки процесс регистр аудит | | D39.168 | 28.08 | **Аудит доков, часть 2 — архитектура и ресёрчи.** Две НОРМЫ требовали построенного (закон шва: «`status --json` версию не несёт — закрыть», а несёт); доки врали про `accepts_labels` («пусты у всех» при трёх заполненных эндпоинтах — ось аккаунта); денежный вход цитировал грант $5 при коде 0. Вынесены в `archive/research/` четыре мёртвых отчёта (290 КБ) по графу входящих ссылок; двум ресёрчам без шапки вовсе поставлены ревю-шапки. | жив | ЖИВОЕ: системный класс line-якорей в архитектуре; часть 3 не пройдена | доки ресёрчи архив аудит | | D39.169 | 29.08 | **Платформенный пак P11 принят и заленджен + контрактный минор 0.8.0.** Отзыв сессии гасит открытый SSE-поток (`PD-379`, единственная уязвимость `major`) и объявляет причину кадром `session_ended`, а не молчаливым обрывом в `401`; застрявший расчёт видим и закрываем; денежная группа `PD-384/391/394/397/376`. Второй рубеж сессии поймал СЕМЬ ложных подтверждений в её же отчёте (включая выдуманную мутационную посадку) — код цел, достоверность рассказа нет. Охотник вне карты нашёл ЧЕТЫРЕ регресса, введённых самим паком, все закрыты до сдачи. Две мои ошибки в каноне: схема нового кадра была сиротой вне союза `anyOf`, перечень кадров соединения противоречил телу схемы. Линтер якорей нашёл 21 указатель на исчезнувший код — норма: гонять его ПОСЛЕДНИМ шагом, после того как код замер. | жив | ЖИВОЕ: `PD-425` — открытый денежный major, отсрочка на оркестраторе; `PD-424`/`PD-426` отложены с доводом; `PD-423` — условие батареи, которого не знал рецепт | платформа контракт деньги сессии приёмка | +| D39.170 | 29.08 | **Движковый пак «деньги» принят и заленджен**: потолок ОБЪЁМА оплаченной работы (`--max-units`) + читающий путь `status` теперь СВОРАЧИВАЕТ банк, поэтому смета пере-прохода впервые доезжает до покупателя ДО покупки, оставаясь $0. ⚠ Пак СНЯЛ ПОСЫЛКУ чужой зоны: платформа не берёт `rebill_*` по доводу «status читает ноль сразу после apply» (`ingest/resync.go:37-43`) — довод устарел. Четыре круга приёмки; блокирующая находка охотника: потолок ПРОБИВАЛСЯ пере-сидом банка посреди прогона (грант 1 → 4 вызова, $0.0073 вместо $0.0036, две доставленные главы оплачены дважды и названы бесплатными). Лечение структурное — пере-план + отказ волны работать с планом чужого снапшота. Сессия не воспроизвела сценарий ПЯТЬ раз при верных прогонах: дельту двигает не текст, а предложения черновиков, а `spoilerBlocked` (`membank/memory.go:640-647`) режет термин с поздним `since_ch` навсегда. | жив | ЖИВОЕ: ось отгрузки на `once_key` — отдельный заказ; терминолог вне потолка — вход в калибровку цены; проводка `--max-units` ГЕЙЧЕНА | движок деньги потолок приёмка шов | diff --git a/docs/architecture/05-decisions-log.md b/docs/architecture/05-decisions-log.md index 457ff6eb..a00617f1 100644 --- a/docs/architecture/05-decisions-log.md +++ b/docs/architecture/05-decisions-log.md @@ -1,4 +1,4 @@ -# Журнал решений оркестратора — контракт D1–D39.169 (живой файл: карта · эрраты · живые тела · голова D39.124+ (подрезка D39.139); тела закрытых эр — в слайсах `docs/archive/architecture/`, указатель ниже; реестр всех нот — `05-decisions-index.md`) +# Журнал решений оркестратора — контракт D1–D39.170 (живой файл: карта · эрраты · живые тела · голова D39.124+ (подрезка D39.139); тела закрытых эр — в слайсах `docs/archive/architecture/`, указатель ниже; реестр всех нот — `05-decisions-index.md`) > **⟶ КАРТА АКТУАЛЬНОСТИ (ревизия D31, продлена до D38.2 [12.07]; исторические записи ниже НЕ переписываются — дисциплина D23.3).** Работая с контрактом (греп номера: живой файл → слайсы, целиком НЕ читать — D39.125), держи под рукой, что чем перекрыто: > ⚠ **Эррата 09.08 (D39.125):** D39.111 п.1 предписывал промту S3 «максимум = баланс МИНУС открытые холды» — формула ОШИБОЧНА (вычитание дважды), исправлена D39.115 п.2(а): максимум = Balance КАК ЕСТЬ; тело D39.111 живёт ниже в этом файле (голова D39.106+). @@ -1235,3 +1235,100 @@ D39.165 «смета уже публикуется в `status`» верна то **Строки.** Регистр платформы: 426 строк, 105 open, 7 major. Семь заказанных строк закрыты с телами; девять новых заведены тем же деревом. + +## D39.170 — ДВИЖКОВЫЙ ПАК «ДЕНЬГИ» ПРИНЯТ И ЗАЛЕНДЖЕН: потолок ОБЪЁМА оплаченной работы + смета пере-прохода, которая наконец доезжает до покупателя (29.08, оркестратор №19). ✅ + +**Что заленджено.** Движок останавливается по ОБЪЁМУ оплаченной работы, а не только по деньгам +(`--max-units`, `VolumeStop`) — покупатель платит за N юнитов и получает ровно N · читающий путь +`status` СВОРАЧИВАЕТ банк, а не проецирует сохранённый глоссарий, и потому впервые отвечает на вопрос +«сколько будет стоить пере-проход» ДО покупки, оставаясь $0-глаголом без записи · денежный отчёт +разведён на `Delivered`/`Reworked`/`LeftFresh`/`LeftRework`, и приглашение купить произносится только +про никогда-не-доставленное · авто-банк пишется атомарно. + +⚠ **ГЛАВНОЕ, ЧЕГО НЕ НАЗВАЛА НИ ОДНА СЕССИЯ: этот пак СНЯЛ ПОСЫЛКУ, на которой стоит решение ЧУЖОЙ +зоны.** Платформа сознательно не берёт `rebill_units`/`rebill_usd` через шов, и её основание записано в +коде (`platform/internal/ingest/resync.go:37-43`): «status проецирует СОХРАНЁННУЮ память, и сразу после +`bank-apply` — в единственный момент, когда согласие хотело бы цифру, — он честно читает ноль». Это было +верно и ратифицировано эрратой 28.08-к. **Теперь неверно:** `foldMemoryForRead` стал ПЕРВЫМ ответом +читающего пути, а `projectStoredMemory` понижена до фолбэка (`status.go:817-824`, комментарий самого +пака это и объявляет). Слепое окно закрыто. Следствия проведены этим же лендингом: строка бэклога +**231** закрывается, комментарий платформы получает строку в её регистр, а проводка полей через шов +становится ВОЗМОЖНОЙ — но не выполняется, потому что гейчена вместе с `--max-units` (ниже). + +**Приёмка — четыре круга, и первые три сдачи были неверны.** Мой первый проход дал 4 денежных дефекта +(обход гейта согласия дроблением покупок; потолок, тративший покупку на пере-делку вперёд доставки; +расхождение `status`/`translate`; неатомарный авто-банк). Второй круг сессии — ещё шесть её собственных, +включая то, что **её же строка стопа сообщала ЮНИТЫ как ГЛАВЫ**, то есть подмену, ради устранения которой +пак и заведён, совершал его собственный отчёт. Четвёртый круг — денежный охотник вне карты, пять +подтверждённых прогоном. + +⛔ **БЛОКИРУЮЩАЯ НАХОДКА ЧЕТВЁРТОГО КРУГА: потолок объёма ПРОБИВАЛСЯ, и пробивал его сам прогон.** +`planVolume` классифицирует юниты ДО волн и по снапшоту, который берёт один раз (`volume.go:353-355`); +free-юниты допускаются БЕЗУСЛОВНО, вне гранта (`:259-265`) — бесплатная работа ничего не стоит. Но между +планированием и редакторской волной стоит стоп майнинга, который пере-сеивает банк ПОСРЕДИ прогона +(`mining.go:242`), после чего edit-снапшот берётся заново (`waverun.go:203`). Значит каждый «бесплатный» +юнит судился по снапшоту, который прогон сам же и заменил. Замер: грант 1 → **4 вызова вместо 2**, +$0.007280 вместо максимум $0.003640, две уже доставленные главы отредактированы повторно — и строка +отчёта назвала их «rode along at $0». +**Лечение — не гард, а ПЕРЕ-ПЛАН:** вопрос «бесплатен ли юнит» задаётся заново против снапшота, ставшего +реальным, перед самой волной; ставший платным берёт слот или не идёт. **И оно структурно:** волна +ОТКАЗЫВАЕТСЯ работать со скоупом, спланированным против другого снапшота — без этого удаление вызова +оставляло батарею зелёной при молча переставшем держать потолке. + +⚠ **ЧЕТЫРЕ КРУГА ПОТРЕБОВАЛИСЬ НЕ ПОТОМУ, ЧТО СЕССИЯ ПЛОХО РАБОТАЛА — А ПОТОМУ ЧТО СЦЕНАРИЙ +КОНТРИНТУИТИВЕН, И ЭТО САМОЕ ЦЕННОЕ ЗНАНИЕ ПАКА.** Сессия пыталась воспроизвести дефект ПЯТЬ раз и не +смогла ни разу; её прогоны были ВЕРНЫ, ошибочна была гипотеза о причине. Две ловушки, обе измерены: +1. **Не та сторона дельты.** Список терминов майнится из ИСХОДНИКА всей книги (`mining.go:80-98`) — он + полон после первой покупки и не растёт. Байты банка двигает другая сторона: `dst` термина, который + складывается из banknote-блоков ЧЕРНОВИКОВ (`mining.go:109`) и растёт с каждой покупкой **даже при + побайтово однородном источнике**. Асимметрию надо строить в том, что предлагает МОДЕЛЬ, а не в тексте. +2. **Условия тянут в РАЗНЫЕ стороны.** Естественный способ заставить дельту расти — дать поздним главам + новый термин — ровно этим выталкивает `since_ch` за пределы уже доставленного, а + `spoilerBlocked` (`membank/memory.go:640-647`) — жёсткий гейт `chapter < since_ch`: такой термин не + попадёт в инъекцию ранней главы НИКОГДА, какой бы `dst` он ни получил. Нужна ОБРАТНАЯ асимметрия: + термин в раннем исходнике, рендеринг поздно. +**Урок, годный за пределами этого пака:** пять верных прогонов при неверной гипотезе неотличимы от +«дефекта нет». Различил их только тот, у кого сценарий уже работал. + +**Доказательства, которые я снял САМ, а не принял.** Батарея: 21 пакет из 21, 0 FAIL, полнота сверена +списком `go list` против вердиктов. Линтер `0 issues`. Четыре ключевых пина зелёные поимённо. **Две мои +мутационные посадки на сценарном тесте:** снятие вызова пере-плана → структурный отказ адресным +сообщением; снятие вызова И гарда → дефект целиком, всеми четырьмя денежными утверждениями. То есть +находка настоящая, тест ловит ПЕРЕОПЛАТУ, а не факт вызова, и фикс её закрывает. + +**Статус находки уточнён ПРОТИВ автора, обеими сторонами.** Возражение сессии («сдвиг `memory_version` +ничего не доказывает — мерить надо `content_hash` юнита») принято, и охотник показал, что мерил именно +его. Но он же признал, что **завысил срочность**: дефект требует конъюнкции трёх условий, в его +собственной фикстуре два термина из трёх её не выполняют и остаются инертными; «покупка №50 +переоплачивает 490 юнитов» — потолок тяжести, когда сработало, а не ожидаемый случай, и частоту на +реальной книге он не мерил и назвать не может. Лечение от узости входа не зависит — оно структурно. + +⛔ **ПРОВОДКА `--max-units` В ПЛАТФОРМУ ОСТАЁТСЯ ГЕЙЧЕНОЙ, и основание усилилось.** Я ставил гейт на +доводе «вторая покупка на майнящей книге падает без `--resnapshot`». Настоящее основание оказалось +сильнее — **с `--resnapshot` потолок не держал**, — и нашёл его охотник, не я. Записываю как есть: +везение, а не прозорливость. + +**ЧТО ОТЛОЖЕНО СТРОКАМИ, диспозиции мои.** (1) **Ось «свежий/пере-делка» выведена из ПОЛНОТЫ СТРОК, а +не из факта отгрузки** — поэтому добавление стадии превращает дочитанную книгу в «никогда не +доставлявшуюся», а юнит, прерванный между волнами, тратит слот дважды. Носитель у движка уже есть — +реестр анонсов `events_outbox.once_key` (`unitOnceKey` = книга+волна+глава+юнит), монотонный на всю +жизнь книги; не хватает читающего метода поверх готовой константы `onceKeyLookup`. Отдельный заказ: +сессия законно не полезла в чужой пакет в хвосте четвёртого круга. ⚠ Формулировка строки — «ошибается +не только слово»: отчёт, зовущий дочитанную книгу недоставленной, ПРИГЛАШАЕТ купить её снова. +(2) **Терминолог вне объёмного потолка** — место траты в ЦЕНЕ, как договорено, но цифра меняет вес +договорённости: накладные масштабируются КНИГОЙ, а не грантом (три покупки по одному юниту дали три +полнокнижных консолидации по $0.005460 каждая, при том что юнит дешевле). Книга на 500 юнитов, +проданная по одному, оплатит 500 полнокнижных проходов — это вход в калибровку цены, а не сноска. +(3) **Майнящие книги под потолком** — корень в джоб-гарде ратифицированного Р6-контура, чужой предмет. + +**Названная граница, которую сессия записала, а не умолчала:** флагнутый юнит следующей покупкой едет +как `Free` и в остатке стопа не виден. Я проверил цепь до конца и границу принимаю: движок отчитывает +его в `status` (`flagged`, пер-главные паспорта), платформа берёт это аллоулистом (`Flagged`, +`UnitsFlagged` — `ingest/resync.go`), а читателю он доезжает как `withheld` (`ingest/export.go:47`, +канон §UnitState). Покупателю не показывают фальшивое «доставлено» — цепь цела, дублировать её в стопе +покупки незачем. + +**Сужение D20.2-Q2 — подтверждена ДЕЙСТВУЮЩАЯ редакция:** порог судится по ВСЕЙ книге (дробить +бесполезно), именованный кап — по тому, что заплатит ЭТОТ прогон (законная работа не отклоняется), +прогон без пере-оплаты гейта не встречает. Первая редакция отозвана самой сессией и помечена отозванной, +а не переписана молча. diff --git a/docs/archive/prompts/BACKEND_MONEY_PACK_SESSION_PROMPT_2026-08-29.md b/docs/archive/prompts/BACKEND_MONEY_PACK_SESSION_PROMPT_2026-08-29.md new file mode 100644 index 00000000..d337aff3 --- /dev/null +++ b/docs/archive/prompts/BACKEND_MONEY_PACK_SESSION_PROMPT_2026-08-29.md @@ -0,0 +1,240 @@ +> ⚠ **ОТРАБОТАН И ЗАКРЫТ 29.08 — D39.170.** Пак принят и заленджен. Исход: потолок ОБЪЁМА оплаченной +> работы построен; читающий путь `status` сворачивает банк, поэтому смета пере-прохода впервые доезжает +> до покупателя ДО покупки, оставаясь $0-глаголом. Приёмка шла ЧЕТЫРЕ круга: первые три сдачи были +> неверны, и блокирующую находку — потолок пробивался пере-сидом банка посреди прогона — дал охотник вне +> карты, а не отчёт. Отложены строками **232** (ось отгрузки на `once_key`) и **233** (терминолог вне +> потолка); проводка `--max-units` в платформу ГЕЙЧЕНА. **Инструкции отсюда НЕ исполнять** — файл +> историчен. + +# Промт: бэкенд, пак «деньги» — продажа обретает настоящий стоп, а смета становится доступной до покупки + +> **Выдан оркестратором №19, 28.08.2026.** Основание — **D39.165** §1 (три продуктовых развилки сняты +> словом владельца) и строка бэклога **231**. Пак идёт ПАРАЛЛЕЛЬНО платформенному. +> ⚠ **ФАЙЛЫ вы не делите** — модули отдельные, платформа кода движка НЕ компилирует и его тестов НЕ +> запускает (проверено). **Но ШОВ общий и ведётся РУКАМИ в трёх точках:** таблица кодов выхода · +> argv из строковых литералов · аллоулист полей `status --json`. **Твой §3.1 трогает первые две** — +> форму ответа я решил за тебя (§3.1), а не оставил на согласование. + +## §0. Какая проблема и что решит твой результат + +**1. Ручка «купить N глав» ничего не ограничивает.** Платформа продаёт ГЛАВЫ, а движку передаёт +только денежный потолок: `--ceiling-usd` (`backend/cmd/tmctl/invocation.go:129`), и тот каппит +**КУМУЛЯТИВНУЮ** трату по книге (`backend/internal/store/ledger.go:67` — +`bookTotal+estimate > c.BookUSD`). **Про число глав движок не знает вовсе**, флага такого нет. + +**Измерено приёмкой 28.08 (леджеры реальных прогонов, $0):** настоящая цена главы — средняя +**$0.0115–0.0190**, максимум одной главы **$0.0375**; константа продажи платформы — **$0.03**. То +есть покупка «10 глав» отдаёт движку $0.30, а $0.30 на этом материале покупает **16–26 глав**. +Пользователь просит одно, получает другое, и полоса каппится на купленном. + +⚠ **И хвост цены порождает КАЧЕСТВО, а не объём** (наблюдение владельца, подтверждено данными): +вызов может отработать успешно и вернуть мусор — деньги списаны, юнит уезжает на ретрай, иногда на +эскалационный хоп к дорогой модели. В леджере это видно с именами причин (`empty`, `length`, +`cjk_artifact`, `sanitizer_defect`), и доля ретраев+эскалаций гуляет **1.5% → 23%** между прогонами. + +**2. Смету пере-прохода показать НЕЧЕМ — и это названная цена вчерашнего минора 0.7.0.** Движок уже +считает её (`projectRebill`, `backend/internal/pipeline/rebill.go:111`) и публикует в `status --json`. +Но `status` материализует банк из СОХРАНЁННОГО глоссария +(`backend/internal/pipeline/status.go:733-744`, `projectStoredMemory` — его собственный комментарий: +«A seed-FILE edit not yet re-run is NOT reflected here»), а `bank-apply` пишет только ФАЙЛЫ решений. +Свёртка происходит внутри СЛЕДУЮЩЕГО `translate`. **Итог: сразу после правки банка движок честно +отвечает «ничего не двигалось», и платформе нечего показать покупателю.** Носитель — строка **231**. + +**Что решит результат:** движок получает СПОСОБНОСТЬ остановиться по объёму, а пользователь получает +возможность увидеть, что затронет пере-проход, ДО оплаты. + +⛔ **ЧЕСТНАЯ ГРАНИЦА, которую первая редакция промта скрывала (нашёл опровергатель).** «Купил N глав — +получил N глав» этим паком НЕ становится правдой, потому что **число глав до движка не доезжает**: +`CeilingChapters` живёт на платформе (`platform/internal/runs/runs.go:232`) до самого спавна, а +`runner.TranslateArgs` собирает argv литералами и канала для него не имеет. **Проводка — платформенная +половина, и она в этом паке НЕ заказана.** Ты строишь механизм; труба к нему — отдельный заказ, и я +его завожу строкой. Не считай пак провалившимся оттого, что после него обещание §0 ещё не выполнено. + +## §1. Зона записи и git + +**Твоя зона — `backend/`.** Итоги и находки — своя секция журнала `docs/PROGRESS.md` («Бэкенд»). + +- **Ты НЕ коммитишь.** Лендит оркестратор после адверсариальной приёмки. +- ⚠ **`platform/` НЕ ТРОГАТЬ.** Там параллельно работает платформенный пак. Нужна правка на той + стороне — это ПИНГ, а не правка. +- ⚠ **`docs/` — не твоя зона**, кроме своей секции журнала. ⚠ Перед записью в неё **перечитай файл**: + журнал правят две сессии, и 28.08 оркестратор уже унёс чужую секцию своим коммитом из-за общего + окна записи. +- ⚠ **Стендовый `tmctl` собирай из ЗАФИКСИРОВАННОЙ копии дерева, а не из рабочего**, если он тебе + понадобится: рабочее дерево грязное твоими же правками, и бинарь из него мерит непонятно что. +- В дереве незакоммиченная работа полигона (20 позиций) — не касаться. + +## §2. Карта чтения — ≤5 позиций, ЗАКОН + +1. **`backend/cmd/tmctl/main.go:252-262`** — три денежных флага и почему они ОРТОГОНАЛЬНЫ; там же + предупреждение «`--ceiling-usd` — НЕ бюджет прогона» · **`invocation.go:120-210`** (парсинг и + валидация флагов). +2. **`backend/internal/store/ledger.go:30-80`** — где потолок реально судит трату (`Reserve`). +3. **`backend/internal/pipeline/rebill.go`** — `RebillProjection`, `projectRebill` (`:111`), порог + согласия и его текст отказа (`:318-324`). +4. **`backend/internal/pipeline/status.go:733-744`** — `projectStoredMemory`; **читать ВНИМАТЕЛЬНО: + это и есть слепое окно из §0.2**. +5. **`docs/architecture/05-decisions-log.md`, тело D39.165** (греп `^## D39.165`) — основание пака; + **и эррата 28.08-к в шапке того же файла** — там названа моя ошибка про смету, не повтори её. + +## §3. Состав пака + +### §3.1. Потолок ОБЪЁМА рядом с денежным — ЗАКАЗ; форма свободна + +**Заказано:** движок умеет остановиться по объёму работы, а не только по деньгам, и **говорит, по +какому потолку встал**. + +**Ратифицировано D39.165 §1б и обсуждению не подлежит:** потолок живёт в ДВИЖКЕ. Резать волну на +стороне платформы нельзя — она волной не владеет и пер-юнитной цены заранее не знает. + +**Свободен и обязан обосновать — В ЧЁМ мерить. ⚠ Кандидатов ТРИ, а не два (поправка опровергателя):** +(а) **выходной юнит** — `ManifestUnit`, та самая гранулярность, которую движок отгружает, и она же +платформенный `chapters.units_total` с интейка; (б) **чанк**; (в) **`chunk×stage` — БИЛЛИНГОВАЯ +единица движка**, ровно та, в которой считает смета (`rebill.go:322`: «%d chunk×stage unit(s)»). +⚠ **Ловушка ровно здесь:** режущий волну естественно считает то, за что ПЛАТЯТ, то есть (в), а +платформа продаёт (а) — и купивший десять глав получит их долю. **Дефект пака воспроизведётся слоем +ниже, в той же форме.** Выбор объясни и назови, как он ложится на платформенную единицу продажи. + +⚠ **И ещё одно, чего первая редакция не спрашивала: как против объёмного потолка считаются РЕПИНЫ и +РЕТРАИ.** Пере-проход, пере-привязывающий пятьсот юнитов за $0 (`rebill.go:315-320`), сожжёт объёмный +потолок, не потратив цента. Ответь на это явно. + +⛔ **ФОРМА ОТВЕТА ЧЕРЕЗ ШОВ РЕШЕНА МНОЙ, и это НЕ предмет твоего выбора — потому что выбор здесь +ломает деньги в обе стороны (нашёл опровергатель).** Словарь кодов выхода объявлен ЗАМОРОЖЕННЫМ самим +движком (`backend/cmd/tmctl/main.go:125-127`: «both bands are frozen seams and a fresh code would be a +word added to a ratified dictionary»), а платформа незнакомый код читает как ОТКАЗ +(`platform/internal/ingest/exit.go:168-169`, `default: OutcomeFailed`). Отсюда: +- переиспользуешь код денежного потолка → платформа объявит прогон ПРИОСТАНОВЛЕННЫМ и возобновит его + за уже купленный объём — дефект, ради которого пак заведён, воспроизведётся слоем ниже; +- заведёшь новый код → пользователь, получивший РОВНО купленное, увидит «ошибка сервиса». + +**РЕШЕНИЕ: остановка по объёму — это НЕ остановка, а ЗАВЕРШЕНИЕ.** По деньгам прогон встал ПОСРЕДИ +работы, там честно «приостановлен» и возобновление. По объёму он сделал ровно то, что куплено, — это +УСПЕХ, код **0**, замороженный словарь не трогается вовсе, платформа читает штатное «готово». +Различение, которого требует пункт ниже, живёт тогда не в КОДЕ ВЫХОДА, а в отчёте/логе. Если найдёшь, +что это ломает что-то, чего я не вижу, — **пинг, и я пере-решаю**: на этой развилке уже стоял выбор, +ломающий деньги в обе стороны. + +⚠ **Две грабли, забранные у внешних систем — исполнить обе:** +- **остановка обязана НАЗЫВАТЬ свой потолок.** У OpenHands (`max_budget_per_task` + `max_iterations`) + агент упирается в лимит итераций МОЛЧА — известная жалоба. У нас «встал по деньгам» и «встал по + объёму» — разные ответы пользователю и разные ремеди; +- **проверять ПЕРЕД началом единицы работы, а не после.** У LiteLLM пост-фактум-проверка токенного + потолка всегда пробивает его на один вызов. Денежный потолок у нас проверяется в `Reserve` до + вызова — объёмный обязан вести себя так же. + +⚠ **Не сломай ортогональность.** `--ceiling-usd` — кумулятивный КНИЖНЫЙ кап, не бюджет прогона +(`main.go:256-260`). Новый потолок — про работу ЭТОГО прогона. Смешаешь семантику — сломаешь +платформенный расчёт холдов, который на кумулятивности и стоит. + +### §3.2. Смета БЕЗ покупки — ЗАКАЗ; форма свободна + +**Заказано:** платформа может узнать «сколько юнитов затронет пере-проход», НЕ запуская `translate` и +ничего не покупая. + +**Что знать** (проверено мной, пере-проверь): смета уже считается (`projectRebill`) и уже публикуется +в `status --json` полями `rebill_units`/`rebill_usd`. Мешает ровно одно — **тайминг свёртки**: +`status` читает СОХРАНЁННЫЙ глоссарий, а решения лежат в ФАЙЛАХ до следующего `translate`. + +**Форма свободна.** Очевидные кандидаты, оба со своей ценой: +- **флаг у `status`** («сверни решения в память для расчёта и не пиши») — дёшево, но `status` + ратифицирован как ЧИСТО ЧИТАЮЩИЙ ремонтный глагол, и свёртка внутри него это свойство ломает; +- **новый $0-глагол** — честнее по разделению, но новая поверхность CLI и новый документ. +⚠ **Родня, которую надо знать: `resnapshot --dry-run`** — строка **124(в)**, поименована в законе шва +как будущая дверь. Она просит ТУ ЖЕ способность «оценить, ничего не покупая». **Если твоя форма +закрывает обе — скажи это прямо, это удешевляет проект.** + +⛔ **Границы, обе жёсткие, но сформулированы ТОЧНО (первая редакция обе переоценила — нашёл +опровергатель):** +- **$0 — абсолютно:** ни одного провайдерского вызова. +- **Read-only — с ОДНОЙ названной оговоркой:** «ни одной записи в стор» абсолютом быть НЕ МОЖЕТ — + первое касание проекта создаёт и мигрирует базу движка по построению (`backend/internal/pipeline/runner.go:193-197`; + ратифицировано D39.122, у `status` тот же побочный эффект). Требование верное: **никаких записей + СВЕРХ этого first-touch**, и ни одной записи в банк/глоссарий/чекпойнты. +- **Ключей НЕ требует.** ⚠ Якорь первой редакции (`dotenv.go:53-55`) я ВЫДУМАЛ — там середина + докблока про другое. Настоящий носитель: `backend/cmd/tmctl/invocation.go:152-155` (текст отказа, + перечисляющий читающие глаголы) и пин `backend/cmd/tmctl/keysfile_test.go`. + +⚠ **ТРЕТИЙ КАНДИДАТ, ДЕШЕВЛЕ ОБОИХ МОИХ — его нашёл опровергатель, и я его не видел.** У `bank-apply` +УЖЕ ЕСТЬ режим проекции (`invocation.go:126`: «print the projection of the decisions and write +NOTHING»), он берёт тот же арбитр-флок, ничего не пишет и печатает JSON-отчёт. **И у него уже на +руках ровно недостающее состояние**: `readBookState` поднимает bank + seed + delta + rejects ДО +свёртки. Не хватает только statuses и manifest для `projectRebill`. Рассмотри этот путь ПЕРВЫМ: он не +трогает «`status` — чисто читающий» вовсе. + +⛔ **ЛОВУШКА, КОТОРАЯ УБЬЁТ ВЕСЬ СМЫСЛ, если её не увидеть.** Свернуть решения В СТОР ты не сможешь +(`store.go:117-128`, `OpenReadOnly` ставит `PRAGMA query_only=1`), значит фолд будет В ПАМЯТИ. Но +канонический фолд идёт ЧЕРЕЗ стор: `seeding.go` пишет банк, читает его обратно `ORDER BY src, sense, +…` и материализует. Порядок НЕ безразличен — `membank/memory.go:610-617` сортирует стабильно «в +порядке замороженных entries», а дальше токенный бюджет РЕЖЕТ хвост. **Материализуешь «будущий банк» +в порядке добавления — получишь другой набор строк глоссария, другие отрендеренные байты, другой +content-hash: бесплатная смета назовёт одно число, а платный прогон выставит другое.** Это ровно тот +дефект, который пак продаёт как решённый. Докажи тождество, а не предположи его. + +⚠ **И мина формы:** оба поля сметы объявлены `omitempty` (`status.go:208-209`). D39.166 п.2 уже +заплатил за это однажды: «$0-цена мурует дверь — юниты по нулевой цене приходят БЕЗ цены». Если +переиспользуешь форму, «ноль юнитов» и «не считалось» на проводе станут неотличимы. +⚠ **Факт из строки 231, который тебе полезен:** поля `rebill_units`/`rebill_usd` СНЯТЫ с аллоулиста +шва до появления потребителя — вернутся вместе с твоим глаголом. + +### §3.3. Чего в паке НЕТ — пропуски подписаны + +- **Калибровка константы $0.03 — НЕ твоя и НЕ сейчас.** Она платформенная и гейчена строкой **202** + (живой прогон текущего стека). Ты даёшь МЕХАНИЗМ, цену считает не движок. +- **Цена от объёма исходника** (D39.165 §1а) — платформенная половина, носитель `chapters.units_total`. +- **Строки 228 / 230 / 141-остаток / 131** — соседи, не заказ. Увидишь — назови строкой, не чини. + +## §4. Самопроверка ИСПОЛНЕНИЕМ — «перечитал сам» её не удовлетворяет + +1. **Батарея зоны** под `-race`, линтер 0 issues; числа — с командами. +2. **Потолок объёма доказывается ПОСТ-ФИКСНЫМ инвариантом:** прогон с потолком в N единиц + останавливается на N, отвечает признаком «встал по ОБЪЁМУ» (отличимым от денежного), и + останавливается ПЕРЕД началом N+1, а не после. +3. ⛔ **Смета доказывается $0 — и РЕПРО ПЕРВОЙ РЕДАКЦИИ БЫЛО ВЫРОЖДЕННЫМ, я повторил ошибку, о + которой сам же написал эрратy 28.08-и.** Состояние «`translate` не запускался» негодно: у такой + книги строк `chunk_status` нет вовсе, и `projectRebill` отдаёт ПУСТО и до фикса, и после + (`rebill.go:132-134` пропускает строки без снапшота). **Верное состояние: прогон СОСТОЯЛСЯ** + (строки `chunk_status` есть, снапшоты записаны) → **ПОТОМ применена правка банка** → и твой + механизм отдаёт НЕнулевой счёт затронутых юнитов, тогда как сегодняшний `status` на том же + состоянии отвечает нулём. Разница «сегодня ноль / после фикса N» и есть деливерабл. + При этом леджер не двинулся ни на цент, а в стор не ушло ничего сверх first-touch. +4. **Свои посадки мутаций:** минимум по одной на пункт; вердикт — по ДЕЛЬТЕ против чистой базовой + линии и по ТОПИЧНОСТИ упавшего теста, не по цвету батареи. +5. **Адверсариальный проход по своей готовой работе — заказан; глубину и веер выбираешь сама.** + ⚠ Куда смотреть в ЭТОМ паке: деньги (перерасход либо отказ законной работе) · тождество бесплатной сметы и платного прогона · шов (форма ответа, argv). + Дефекты, которые внёс сам пак, называй первыми — приёмке они не видны. + ⚠ **Fable 5** — обычно хватает одного-двух агентов, но это РЕКОМЕНДАЦИЯ, не потолок: веер под предмет выбирает сессия. Модель задавай ЯВНО и знай, сколько их работает. + +## §5. Оси ревью — 1–3, вправе заменить с аргументом + +1. **Ось денег:** может ли новый потолок дать перерасход или, наоборот, отказать законной работе; + не сломана ли кумулятивная семантика денежного капа. +2. **Ось «пара, которой в репо нет»:** ветвления по паре/книге в Go быть не должно. +3. **Ось детерминизма:** трогаешь сборку запроса или снапшот — не перекупается ли прогон. + +## §6. Записка-план, комплектность + +**До правок** — записка-план в своей секции журнала. **В конце** — таблица комплектности против §3: +пункт → что сделано → каким ИСПОЛНЕНИЕМ подтверждено. Слово вместо команды = пункт НЕ сделан. + +## §7. Эхо-протокол старта + +ДО работы — ≤10 строк: **скоуп · инварианты · не-делать**. Первое действие — вписать свой блок в файл +канала (§10), сразу второе — послать мне эхо. + +## §8. Obstacle — обязательная секция + +**«Что НЕ удалось и что НЕ проверено»** отдельной секцией, не россыпью. + +## §9. Канал вопросов и твоё право отказаться + +Конфликт промта с кодом или доками — **пинг, не интерпретация в свою пользу**. +⚠ **У тебя есть право сказать «этого делать не надо» — с аргументом.** За последние сутки исполнители +опровергли ЧЕТЫРЕ моих заказа и все четыре раза были правы; два я пере-проверил своей посадкой и +отозвал. Отказ с разбором дороже послушного исполнения. +⚠ **Тесты и гейты не подгонять под зелень** (D39.121). + +## §10. Как со мной связаться + +**Адрес — в файле `/tmp/textmachine-channel`.** Впиши свой блок первым делом, чужие не трогай. +⚠ **Канала нет ⇒ НЕ искать:** вопрос секцией в отчёт, работа продолжается.