Land the engine money pack: the run now stops on the VOLUME of work a buyer paid for, and the read path folds the bank so a re-pass can be priced before it is bought
This commit is contained in:
parent
e14561b258
commit
6ec9f8a66a
31 changed files with 4773 additions and 225 deletions
|
|
@ -8,7 +8,7 @@
|
||||||
|
|
||||||
| Пакет | Что делает | Ключевые файлы |
|
| Пакет | Что делает | Ключевые файлы |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) / `manifest` (**$0-производитель персиста манифеста глав/чанков**, строка 100: нужен ДО первого прогона — дерево глав разобранной, но не запущенной книги; `--json` печатает сам документ) / `migrate` (**$0-write-open без прогона**, строка 174: read-only команды требуют ТОЧНОГО совпадения схемы и не мигрируют, поэтому апгрейд бинаря запирал все существующие книги — платформа зовёт `status --json` перед каждым спавном, а write-команда, которая мигрировала бы, не наступала никогда. Грузит ТОЛЬКО `book.yaml` — ни цен, ни ключей на $0-пути (строка 146); restore point берёт ПОД ЛОКОМ и только когда шаг реально применяется (имя `<метка>-pre-migrate.db` — вне секундного неймспейса платного пути, строка 173); несовпадение схемы у read-only путей — типизированный отказ **exit 13** с машинным токеном `schema_mismatch found=N expected=M`). Денежный аргумент прогона: `--ceiling-usd` (строка 145) перекрывает книжный `ceilings.book_usd` ТОЛЬКО на этот прогон, в book.yaml не пишется, ноль/отрицательное = отказ запуска. main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` |
|
| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) / `manifest` (**$0-производитель персиста манифеста глав/чанков**, строка 100: нужен ДО первого прогона — дерево глав разобранной, но не запущенной книги; `--json` печатает сам документ) / `migrate` (**$0-write-open без прогона**, строка 174: read-only команды требуют ТОЧНОГО совпадения схемы и не мигрируют, поэтому апгрейд бинаря запирал все существующие книги — платформа зовёт `status --json` перед каждым спавном, а write-команда, которая мигрировала бы, не наступала никогда. Грузит ТОЛЬКО `book.yaml` — ни цен, ни ключей на $0-пути (строка 146); restore point берёт ПОД ЛОКОМ и только когда шаг реально применяется (имя `<метка>-pre-migrate.db` — вне секундного неймспейса платного пути, строка 173); несовпадение схемы у read-only путей — типизированный отказ **exit 13** с машинным токеном `schema_mismatch found=N expected=M`). **ДВА потолка прогона, и они ортогональны.** `--ceiling-usd` (строка 145) — ДЕНЬГИ: перекрывает книжный `ceilings.book_usd` ТОЛЬКО на этот прогон, в book.yaml не пишется, ноль/отрицательное = отказ запуска; каппит КУМУЛЯТИВНУЮ трату книги, а не приращение прогона. `--max-units` (D39.165 §1б) — ОБЪЁМ: не больше N ВЫХОДНЫХ ЮНИТОВ (гранулярность `units_total` манифеста, та самая, в которой платформа продаёт главы) будет ОПЛАЧЕНО этим прогоном. Юниты, которые прогон отдаёт за $0 (резюм, ре-пин), едут бесплатно и потолок не тратят; ретраи и эскалации — тоже нет, они внутри юнита. Принимает только `translate`. **Остановка по объёму — ЗАВЕРШЕНИЕ (exit 0), а не пауза:** словарь кодов выхода не расширялся, различение живёт в отчёте прогона и в логе. Отчёт различает ДОСТАВКУ (юнит, которого не было) и ПЕРЕ-ДЕЛКУ (уже доставленный юнит под сдвинутым снапшотом) — покупка, целиком ушедшая в переделку, обязана читаться как переделка. ⚠ На книге, которая МАЙНИТ банк, вторая покупка требует `--resnapshot` (авто-банк растёт между покупками и двигает edit-снапшот) — прогон предупреждает об этом в логе. main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` |
|
||||||
| `internal/llm` | OpenAI-совместимый транспорт + retry/backoff, **capability-слой** (budget_field/temperature/reasoning per-модель), провайдеры openai/local (no-proxy)/anthropic (DEPRECATED-референс); `failover.go` удалён паком-17 — маршрутизация лейблами живёт в config/pipeline (канал B, один хоп `chain[0]`, fail-closed) | `httpllm.go`, `capability.go`, `provider_*.go` |
|
| `internal/llm` | OpenAI-совместимый транспорт + retry/backoff, **capability-слой** (budget_field/temperature/reasoning per-модель), провайдеры openai/local (no-proxy)/anthropic (DEPRECATED-референс); `failover.go` удалён паком-17 — маршрутизация лейблами живёт в config/pipeline (канал B, один хоп `chain[0]`, fail-closed) | `httpllm.go`, `capability.go`, `provider_*.go` |
|
||||||
| `internal/ledger` | Цены по usage (вкл. reasoning/cache-поля), `PriceForResponse` по фактической модели | `pricing.go` |
|
| `internal/ledger` | Цены по usage (вкл. reasoning/cache-поля), `PriceForResponse` по фактической модели | `pricing.go` |
|
||||||
| `internal/store` | SQLite (modernc, CGO-free), цепочка миграций `schema_version` (⚠ ALTER-шаги v8+ не идемпотентны вопреки шапке — бэклог-строка 49а), reserve/settle+checkpoint, chunk_status, глоссарий (+подписной цикл терминолога), ruby, retrieval_state, request_log; `OpenReadOnly` — без flock/миграций/recovery для status/report/export (схема не совпала — типизированный `*store.SchemaMismatchError{Found,Expected}`, обе стороны); `Migrate` (+шов `beforeApply` под локом: restore point берётся там) — поверхность деплой-шага строки 174, `SchemaHead` — та самая `Expected`; write-open БД новее бинаря теперь ОТКАЗ, а не тихое открытие | `ledger.go`, `migrate.go`, `glossary.go`, `store.go` |
|
| `internal/store` | SQLite (modernc, CGO-free), цепочка миграций `schema_version` (⚠ ALTER-шаги v8+ не идемпотентны вопреки шапке — бэклог-строка 49а), reserve/settle+checkpoint, chunk_status, глоссарий (+подписной цикл терминолога), ruby, retrieval_state, request_log; `OpenReadOnly` — без flock/миграций/recovery для status/report/export (схема не совпала — типизированный `*store.SchemaMismatchError{Found,Expected}`, обе стороны); `Migrate` (+шов `beforeApply` под локом: restore point берётся там) — поверхность деплой-шага строки 174, `SchemaHead` — та самая `Expected`; write-open БД новее бинаря теперь ОТКАЗ, а не тихое открытие | `ledger.go`, `migrate.go`, `glossary.go`, `store.go` |
|
||||||
|
|
@ -66,7 +66,8 @@ go run ./cmd/tmctl report --config example/book.yaml # $0, quality-repor
|
||||||
go run ./cmd/tmctl export --config example/book.yaml # $0, экспорт-JSON для полигона (--plaintext для человека)
|
go run ./cmd/tmctl export --config example/book.yaml # $0, экспорт-JSON для полигона (--plaintext для человека)
|
||||||
go run ./cmd/tmctl manifest --config example/book.yaml # $0, пере-строить персист манифеста глав/чанков (--json — сам документ)
|
go run ./cmd/tmctl manifest --config example/book.yaml # $0, пере-строить персист манифеста глав/чанков (--json — сам документ)
|
||||||
go run ./cmd/tmctl migrate --config example/book.yaml # $0, довести схему проекта до головы бинаря (деплой-шаг: стоп прогонов → НОВЫЙ бинарь на место → migrate ИМ по каждой книге без открытых попыток → прогоны в работу)
|
go run ./cmd/tmctl migrate --config example/book.yaml # $0, довести схему проекта до головы бинаря (деплой-шаг: стоп прогонов → НОВЫЙ бинарь на место → migrate ИМ по каждой книге без открытых попыток → прогоны в работу)
|
||||||
go run ./cmd/tmctl translate --config example/book.yaml --ceiling-usd 0.5 # потолок ТОЛЬКО на этот прогон, book.yaml не пишется
|
go run ./cmd/tmctl translate --config example/book.yaml --ceiling-usd 0.5 # ДЕНЕЖНЫЙ потолок ТОЛЬКО на этот прогон, book.yaml не пишется
|
||||||
|
go run ./cmd/tmctl translate --config example/book.yaml --max-units 10 # ОБЪЁМНЫЙ потолок: оплатить не больше 10 выходных юнитов; стоп = завершение (exit 0)
|
||||||
# live-conformance (реальные провайдеры, платно, вне CI):
|
# live-conformance (реальные провайдеры, платно, вне CI):
|
||||||
set -a; . ./.env; set +a; TM_LIVE=1 go test -tags live -run TestLive -v ./internal/pipeline/
|
set -a; . ./.env; set +a; TM_LIVE=1 go test -tags live -run TestLive -v ./internal/pipeline/
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -65,7 +65,7 @@ func TestFakeTranslatePathCreatesNoBackup(t *testing.T) {
|
||||||
defer srv.Close()
|
defer srv.Close()
|
||||||
|
|
||||||
bookPath := setupCLIProject(t, srv.URL)
|
bookPath := setupCLIProject(t, srv.URL)
|
||||||
if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false, 0); err != nil {
|
if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false, 0, 0); err != nil {
|
||||||
t.Fatalf("fake translate: %v", err)
|
t.Fatalf("fake translate: %v", err)
|
||||||
}
|
}
|
||||||
if _, err := os.Stat(filepath.Join(filepath.Dir(bookPath), "backups")); err == nil {
|
if _, err := os.Stat(filepath.Join(filepath.Dir(bookPath), "backups")); err == nil {
|
||||||
|
|
|
||||||
|
|
@ -45,7 +45,11 @@ type invocation struct {
|
||||||
// spend on one run belongs to that run, not to the book's data — writing it into book.yaml would make
|
// spend on one run belongs to that run, not to the book's data — writing it into book.yaml would make
|
||||||
// a caller's number a permanent record in the engine's config and mix the zones (D39.81/D39.85).
|
// a caller's number a permanent record in the engine's config and mix the zones (D39.81/D39.85).
|
||||||
ceilingUSD float64
|
ceilingUSD float64
|
||||||
sel pipeline.RedriveSelector
|
// maxUnits is the VOLUME ceiling for THIS RUN (D39.165 §1б), 0 when the flag was absent. It is an
|
||||||
|
// invocation property for exactly the reason ceilingUSD is — how much of the book a caller is buying
|
||||||
|
// right now belongs to the call, not to the book's data.
|
||||||
|
maxUnits int
|
||||||
|
sel pipeline.RedriveSelector
|
||||||
}
|
}
|
||||||
|
|
||||||
// rebillConsentValue parses `--accept-rebill[=usd]` (D20.2-Q2): the OPTIONAL-VALUE form of Р6, where a
|
// rebillConsentValue parses `--accept-rebill[=usd]` (D20.2-Q2): the OPTIONAL-VALUE form of Р6, where a
|
||||||
|
|
@ -127,6 +131,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
|
||||||
dryRun := fs.Bool("dry-run", false, "redrive: report what would be re-attacked without touching anything; bank-apply: print the projection of the decisions and write NOTHING")
|
dryRun := fs.Bool("dry-run", false, "redrive: report what would be re-attacked without touching anything; bank-apply: print the projection of the decisions and write NOTHING")
|
||||||
verifyBank := fs.Bool("verify-bank", false, "translate/redrive: STOP at the bank-mining boundary and print the bank table when the delta holds terms no earlier stop has shown (D39.144: the flag trips on novelty; a resumed run goes on and undecided terms ride to the editor marked). Default: never stop")
|
verifyBank := fs.Bool("verify-bank", false, "translate/redrive: STOP at the bank-mining boundary and print the bank table when the delta holds terms no earlier stop has shown (D39.144: the flag trips on novelty; a resumed run goes on and undecided terms ride to the editor marked). Default: never stop")
|
||||||
ceilingUSD := fs.Float64("ceiling-usd", 0, "translate/redrive: the book USD ceiling in force for THIS RUN ONLY — it OVERRIDES book.yaml `ceilings.book_usd` and is never written back. It caps the book's CUMULATIVE committed+reserved spend, not this run's increment, and must be > 0")
|
ceilingUSD := fs.Float64("ceiling-usd", 0, "translate/redrive: the book USD ceiling in force for THIS RUN ONLY — it OVERRIDES book.yaml `ceilings.book_usd` and is never written back. It caps the book's CUMULATIVE committed+reserved spend, not this run's increment, and must be > 0")
|
||||||
|
maxUnits := fs.Int("max-units", 0, "translate: the VOLUME ceiling for THIS RUN — at most N OUTPUT UNITS (the manifest's units_total granularity) may be PAID for. Units this run resolves for free (resumed or re-pinned) ride along and do not count, and neither do retries or escalation hops inside a unit. ORTHOGONAL to --ceiling-usd, which caps the book's CUMULATIVE money: this caps THIS run's work. Stopping on it is a COMPLETION (exit 0), not a pause")
|
||||||
seed := fs.String("seed", "", "seed-lint: path to the glossary seed YAML to validate ($0, no --config)")
|
seed := fs.String("seed", "", "seed-lint: path to the glossary seed YAML to validate ($0, no --config)")
|
||||||
keysFile := fs.String("keys-file", "", "translate: path to the DEPLOYMENT's provider-key file (KEY=VALUE lines). Loaded FIRST, so it wins over the .env beside book.yaml; a named file that cannot be read is a refusal, never a silent skip")
|
keysFile := fs.String("keys-file", "", "translate: path to the DEPLOYMENT's provider-key file (KEY=VALUE lines). Loaded FIRST, so it wins over the .env beside book.yaml; a named file that cannot be read is a refusal, never a silent skip")
|
||||||
decisions := fs.String("decisions", "", "bank-apply: path to the JSON decision document to apply to the book's memory bank ($0)")
|
decisions := fs.String("decisions", "", "bank-apply: path to the JSON decision document to apply to the book's memory bank ($0)")
|
||||||
|
|
@ -181,6 +186,31 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
|
||||||
if given["dry-run"] && cmd != "bank-apply" && cmd != "redrive" {
|
if given["dry-run"] && cmd != "bank-apply" && cmd != "redrive" {
|
||||||
return invocation{}, fmt.Errorf("--dry-run is accepted by `bank-apply` and `redrive` only, not by %q: %q has no projection mode, and running it with this flag would spend real money on a call its caller believes is free", cmd, cmd)
|
return invocation{}, fmt.Errorf("--dry-run is accepted by `bank-apply` and `redrive` only, not by %q: %q has no projection mode, and running it with this flag would spend real money on a call its caller believes is free", cmd, cmd)
|
||||||
}
|
}
|
||||||
|
// The volume ceiling follows the same rule as every other flag above: a command that does not act on it
|
||||||
|
// REFUSES it rather than ignoring it. Silently accepting it here would be the expensive spelling —
|
||||||
|
// `redrive --max-units 10` re-attacks flagged chunks and then translates the WHOLE book through
|
||||||
|
// TranslateBook, so a caller who believed the flag bounded the work would be billed for everything
|
||||||
|
// left. Scoped to `translate`, which is the one verb whose volume a purchase actually buys.
|
||||||
|
if given["max-units"] {
|
||||||
|
if cmd != "translate" {
|
||||||
|
// The reason has to be TRUE of the command it addresses, which is the discipline the
|
||||||
|
// --keys-file refusal above already carries a test for. `redrive` is the case that makes it
|
||||||
|
// matter: it DOES drive a book wave — it calls TranslateBook after its reset — so telling it
|
||||||
|
// "you have no book wave to bound" would teach the reader a false fact about the engine, and
|
||||||
|
// the next session wiring this flag would believe it.
|
||||||
|
if cmd == "redrive" {
|
||||||
|
return invocation{}, fmt.Errorf("--max-units is accepted by `translate` only, not by `redrive`: redrive DOES drive a book wave (it re-attacks the flagged chunks and then translates through TranslateBook), but the destructive reset it performs first is bounded by its own selector and by nothing else — a units cap here would bound the second half of the command while reading as a cap on the whole. Bound the re-attack with --chapter/--chunk/--reason, and the translation that follows with a separate `translate --max-units`")
|
||||||
|
}
|
||||||
|
return invocation{}, fmt.Errorf("--max-units is accepted by `translate` only, not by %q: it caps the OUTPUT UNITS a run pays for, and %q neither runs nor pays for them — accepting the flag would tell a caller their work was capped when it was not", cmd, cmd)
|
||||||
|
}
|
||||||
|
// Refused rather than defaulted when present-but-not-a-volume, the same rule --ceiling-usd follows
|
||||||
|
// (D39.110: "off" is not a state). Zero is the ABSENT value, so admitting `--max-units 0` would turn
|
||||||
|
// an unset variable in a deployment's unit file into an unbounded run that the caller believes is
|
||||||
|
// bounded — and a negative is not a quantity of book at all.
|
||||||
|
if *maxUnits <= 0 {
|
||||||
|
return invocation{}, fmt.Errorf("--max-units must be a positive number of output units (it is the volume ceiling for THIS run); got %d — omit the flag to run the whole book", *maxUnits)
|
||||||
|
}
|
||||||
|
}
|
||||||
// seed-lint validates a standalone seed YAML — it takes --seed, not --config (no book/store/keys).
|
// seed-lint validates a standalone seed YAML — it takes --seed, not --config (no book/store/keys).
|
||||||
if cmd == "seed-lint" {
|
if cmd == "seed-lint" {
|
||||||
if *seed == "" {
|
if *seed == "" {
|
||||||
|
|
@ -218,7 +248,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
|
||||||
return invocation{
|
return invocation{
|
||||||
cmd: cmd, cfgPath: *cfgPath, resnapshot: *resnapshot, acceptRebill: acceptRebill.c,
|
cmd: cmd, cfgPath: *cfgPath, resnapshot: *resnapshot, acceptRebill: acceptRebill.c,
|
||||||
asJSON: *asJSON, asPlaintext: *asPlaintext, asPairs: *asPairs, verifyBank: *verifyBank,
|
asJSON: *asJSON, asPlaintext: *asPlaintext, asPairs: *asPairs, verifyBank: *verifyBank,
|
||||||
ceilingUSD: *ceilingUSD, keysFile: *keysFile, decisionsPath: *decisions, dryRun: *dryRun,
|
ceilingUSD: *ceilingUSD, maxUnits: *maxUnits, keysFile: *keysFile, decisionsPath: *decisions, dryRun: *dryRun,
|
||||||
sel: pipeline.RedriveSelector{
|
sel: pipeline.RedriveSelector{
|
||||||
Chapter: *chapter, ChunkIdx: *chunk, Reason: *reason, DryRun: *dryRun,
|
Chapter: *chapter, ChunkIdx: *chunk, Reason: *reason, DryRun: *dryRun,
|
||||||
},
|
},
|
||||||
|
|
|
||||||
|
|
@ -299,3 +299,64 @@ func TestDispatchCommandsCoversTheSwitch(t *testing.T) {
|
||||||
t.Errorf("the dispatch switch has %d commands, dispatchCommands lists %d — the two must be one list", found, len(dispatchCommands))
|
t.Errorf("the dispatch switch has %d commands, dispatchCommands lists %d — the two must be one list", found, len(dispatchCommands))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestMaxUnitsIsRefusedWhereItWouldDoNothing pins the volume ceiling onto the rule the other four flags
|
||||||
|
// already follow: a command that does not act on a flag REFUSES it rather than ignoring it.
|
||||||
|
//
|
||||||
|
// The silent version of this one costs book rather than dollars, and `redrive` is the expensive case. It
|
||||||
|
// re-attacks the flagged chunks and then runs TranslateBook over the WHOLE book, so `redrive --max-units
|
||||||
|
// 10` accepted-and-ignored would bill a caller for every unit the book has left while their own logs said
|
||||||
|
// they had capped the work at ten.
|
||||||
|
func TestMaxUnitsIsRefusedWhereItWouldDoNothing(t *testing.T) {
|
||||||
|
for _, cmd := range dispatchCommands {
|
||||||
|
if cmd == "translate" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
args := []string{cmd, "--config", "book.yaml", "--max-units", "10"}
|
||||||
|
if cmd == "bank-apply" {
|
||||||
|
args = append(args, "--decisions", "d.json")
|
||||||
|
}
|
||||||
|
_, err := parseInvocation(args, &bytes.Buffer{})
|
||||||
|
if err == nil {
|
||||||
|
t.Errorf("%s accepted --max-units; a flag a command does not act on must be refused, never ignored", cmd)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "--max-units is accepted by `translate` only") {
|
||||||
|
t.Errorf("%s: the refusal must name the flag and the one verb that takes it, got: %v", cmd, err)
|
||||||
|
}
|
||||||
|
// And the REASON must be true of the command it addresses — the discipline the --keys-file
|
||||||
|
// refusal already carries a test for. The first draft of this refusal told every command it
|
||||||
|
// "does not run a book wave whose volume this could bound", which is false of `redrive`: it
|
||||||
|
// calls TranslateBook after its reset. A refusal that misdescribes its reader teaches them the
|
||||||
|
// wrong thing about the engine, and the next session to wire this flag inherits it.
|
||||||
|
if strings.Contains(err.Error(), "neither runs nor pays for them") && cmd == "redrive" {
|
||||||
|
t.Errorf("redrive DOES drive a book wave (it calls TranslateBook); its refusal must not claim otherwise, got: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// redrive gets its own reason, and that reason has to say the true thing.
|
||||||
|
_, err := parseInvocation([]string{"redrive", "--config", "book.yaml", "--max-units", "10"}, &bytes.Buffer{})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "redrive DOES drive a book wave") {
|
||||||
|
t.Fatalf("redrive's refusal must name the real reason (the reset is bounded by its selector, not by units), got: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestMaxUnitsRefusesANonVolume pins the same "«off» is not a state" rule --ceiling-usd has needed since
|
||||||
|
// D39.110: a PRESENT flag with a meaningless value is a refusal, because the live trigger is an unset
|
||||||
|
// variable in a deployment's unit file, and defaulting it to «unbounded» would hand a caller a whole book
|
||||||
|
// while their own configuration said ten units.
|
||||||
|
func TestMaxUnitsRefusesANonVolume(t *testing.T) {
|
||||||
|
for _, v := range []string{"0", "-1"} {
|
||||||
|
_, err := parseInvocation([]string{"translate", "--config", "book.yaml", "--max-units", v}, &bytes.Buffer{})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "--max-units must be a positive number of output units") {
|
||||||
|
t.Fatalf("--max-units %s: want a refusal naming the rule, got %v", v, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Absent is the ONE spelling that means "the whole book", and it must stay silent.
|
||||||
|
inv, err := parseInvocation([]string{"translate", "--config", "book.yaml"}, &bytes.Buffer{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("an absent --max-units must parse cleanly: %v", err)
|
||||||
|
}
|
||||||
|
if inv.maxUnits != 0 {
|
||||||
|
t.Fatalf("an absent --max-units must leave the run unbounded, got %d", inv.maxUnits)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -219,7 +219,7 @@ func run() error {
|
||||||
if err := preflightBackup(inv.cfgPath, os.Stdout); err != nil {
|
if err := preflightBackup(inv.cfgPath, os.Stdout); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
return translate(ctx, inv.cfgPath, inv.resnapshot, inv.acceptRebill, inv.verifyBank, inv.ceilingUSD)
|
return translate(ctx, inv.cfgPath, inv.resnapshot, inv.acceptRebill, inv.verifyBank, inv.ceilingUSD, inv.maxUnits)
|
||||||
case "report":
|
case "report":
|
||||||
return report(inv.cfgPath)
|
return report(inv.cfgPath)
|
||||||
case "status":
|
case "status":
|
||||||
|
|
@ -261,7 +261,17 @@ func run() error {
|
||||||
//
|
//
|
||||||
// Over the book's consent threshold the run stops with the sum before reserving anything — a flag that
|
// Over the book's consent threshold the run stops with the sum before reserving anything — a flag that
|
||||||
// names no money cannot carry a Р6 consent to a spend.
|
// names no money cannot carry a Р6 consent to a spend.
|
||||||
func translate(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill pipeline.RebillConsent, verifyBank bool, ceilingUSD float64) error {
|
//
|
||||||
|
// --max-units is the FOURTH flag and the second CEILING, and it is orthogonal to --ceiling-usd in the way
|
||||||
|
// that matters most (D39.165 §1б): that one bounds the book's CUMULATIVE money, this one bounds THIS
|
||||||
|
// run's WORK, in output units — the granularity the manifest publishes and a purchase of chapters
|
||||||
|
// converts into exactly. Both can be in force; whichever binds first stops the run, and the two stops are
|
||||||
|
// different answers. Money stops the run in the middle of what it meant to do — exit 4, resumable, "add
|
||||||
|
// money". Volume means the run did precisely what was bought — exit 0, an ordinary completion, and the
|
||||||
|
// exit-code dictionary above is not touched. Which one stopped it is said in the run's report and its log
|
||||||
|
// (pipeline.VolumeStop), never by a new code: the platform reads an unknown exit code as a FAILURE, so a
|
||||||
|
// user who received exactly what they paid for would have been shown a service error.
|
||||||
|
func translate(ctx context.Context, cfgPath string, resnapshot bool, acceptRebill pipeline.RebillConsent, verifyBank bool, ceilingUSD float64, maxUnits int) error {
|
||||||
r, err := pipeline.NewRunner(cfgPath, obs.NewLogger())
|
r, err := pipeline.NewRunner(cfgPath, obs.NewLogger())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
|
|
@ -271,6 +281,7 @@ func translate(ctx context.Context, cfgPath string, resnapshot bool, acceptRebil
|
||||||
r.AcceptRebill = acceptRebill
|
r.AcceptRebill = acceptRebill
|
||||||
r.VerifyBank = verifyBank
|
r.VerifyBank = verifyBank
|
||||||
r.CeilingUSD = ceilingUSD
|
r.CeilingUSD = ceilingUSD
|
||||||
|
r.MaxUnits = maxUnits
|
||||||
|
|
||||||
res, err := r.TranslateBook(ctx)
|
res, err := r.TranslateBook(ctx)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|
|
||||||
|
|
@ -253,7 +253,7 @@ func TestMigrateNeedsNeitherPricesNorModels(t *testing.T) {
|
||||||
func TestMigrateKeepsTheMoneyAndRepeatsAsANoOp(t *testing.T) {
|
func TestMigrateKeepsTheMoneyAndRepeatsAsANoOp(t *testing.T) {
|
||||||
bookPath := setupCLIProject(t, mockProvider(t).URL)
|
bookPath := setupCLIProject(t, mockProvider(t).URL)
|
||||||
dbPath := projectDBOf(bookPath)
|
dbPath := projectDBOf(bookPath)
|
||||||
if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false, 0); err != nil {
|
if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false, 0, 0); err != nil {
|
||||||
t.Fatalf("setup run: %v", err)
|
t.Fatalf("setup run: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -10,6 +10,7 @@ import (
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
|
@ -98,7 +99,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) {
|
||||||
bookPath := setupCLIProject(t, srv.URL)
|
bookPath := setupCLIProject(t, srv.URL)
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
|
|
||||||
if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0); err != nil {
|
if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0, 0); err != nil {
|
||||||
t.Fatalf("first run: %v", err)
|
t.Fatalf("first run: %v", err)
|
||||||
}
|
}
|
||||||
if calls != 1 {
|
if calls != 1 {
|
||||||
|
|
@ -114,7 +115,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) {
|
||||||
writeCLIFile(t, pipePath, strings.Replace(string(raw), "prompt_version: v-cli", "prompt_version: v-cli-drift", 1))
|
writeCLIFile(t, pipePath, strings.Replace(string(raw), "prompt_version: v-cli", "prompt_version: v-cli-drift", 1))
|
||||||
|
|
||||||
// --resnapshot WITHOUT consent → refused, nothing called.
|
// --resnapshot WITHOUT consent → refused, nothing called.
|
||||||
err = translate(ctx, bookPath, true, pipeline.RebillConsent{}, false, 0)
|
err = translate(ctx, bookPath, true, pipeline.RebillConsent{}, false, 0, 0)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("`translate --resnapshot` without --accept-rebill must be refused over the threshold")
|
t.Fatal("`translate --resnapshot` without --accept-rebill must be refused over the threshold")
|
||||||
}
|
}
|
||||||
|
|
@ -127,7 +128,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) {
|
||||||
|
|
||||||
// A ceiling BELOW the projection is still a refusal — proving the CAPPED value is wired, not just
|
// A ceiling BELOW the projection is still a refusal — proving the CAPPED value is wired, not just
|
||||||
// the boolean.
|
// the boolean.
|
||||||
if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true, Capped: true, CapUSD: 0.0001}, false, 0); err == nil {
|
if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true, Capped: true, CapUSD: 0.0001}, false, 0, 0); err == nil {
|
||||||
t.Fatal("a ceiling below the projection must be refused through the CLI entry point too")
|
t.Fatal("a ceiling below the projection must be refused through the CLI entry point too")
|
||||||
}
|
}
|
||||||
if calls != 1 {
|
if calls != 1 {
|
||||||
|
|
@ -135,7 +136,7 @@ func TestTranslateWiresAcceptRebill(t *testing.T) {
|
||||||
}
|
}
|
||||||
|
|
||||||
// Consent given → the run proceeds and really re-pays.
|
// Consent given → the run proceeds and really re-pays.
|
||||||
if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true}, false, 0); err != nil {
|
if err := translate(ctx, bookPath, true, pipeline.RebillConsent{Given: true}, false, 0, 0); err != nil {
|
||||||
t.Fatalf("`translate --resnapshot --accept-rebill` must proceed: %v", err)
|
t.Fatalf("`translate --resnapshot --accept-rebill` must proceed: %v", err)
|
||||||
}
|
}
|
||||||
if calls != 2 {
|
if calls != 2 {
|
||||||
|
|
@ -160,7 +161,7 @@ func TestRedriveWiresAcceptRebill(t *testing.T) {
|
||||||
ctx := context.Background()
|
ctx := context.Background()
|
||||||
|
|
||||||
// The refusal text flags the only chunk (exit 2 is a sentinel, not an infra failure).
|
// The refusal text flags the only chunk (exit 2 is a sentinel, not an infra failure).
|
||||||
if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0); exitCode(err) != 2 {
|
if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0, 0); exitCode(err) != 2 {
|
||||||
t.Fatalf("setup: the refusal fixture must complete with flags (exit 2), got %v", err)
|
t.Fatalf("setup: the refusal fixture must complete with flags (exit 2), got %v", err)
|
||||||
}
|
}
|
||||||
callsAfterRun1 := calls
|
callsAfterRun1 := calls
|
||||||
|
|
@ -211,7 +212,7 @@ func TestTranslateAndRedriveWireTheRunCeiling(t *testing.T) {
|
||||||
// A ceiling far below anything the book can cost. The book's own ceiling is comfortable, so ONLY the
|
// A ceiling far below anything the book can cost. The book's own ceiling is comfortable, so ONLY the
|
||||||
// argument can produce this stop — and only if the command actually hands it to the runner.
|
// argument can produce this stop — and only if the command actually hands it to the runner.
|
||||||
const tiny = 0.000001
|
const tiny = 0.000001
|
||||||
err := translate(ctx, setupCLIProject(t, srv.URL), false, pipeline.RebillConsent{}, false, tiny)
|
err := translate(ctx, setupCLIProject(t, srv.URL), false, pipeline.RebillConsent{}, false, tiny, 0)
|
||||||
if err == nil || !strings.Contains(err.Error(), "ceiling reached") {
|
if err == nil || !strings.Contains(err.Error(), "ceiling reached") {
|
||||||
t.Fatalf("translate must carry --ceiling-usd into the run; got %v", err)
|
t.Fatalf("translate must carry --ceiling-usd into the run; got %v", err)
|
||||||
}
|
}
|
||||||
|
|
@ -236,7 +237,7 @@ func TestTranslateAndRedriveWireTheRunCeiling(t *testing.T) {
|
||||||
}))
|
}))
|
||||||
defer srv2.Close()
|
defer srv2.Close()
|
||||||
bookPath := setupCLIProject(t, srv2.URL)
|
bookPath := setupCLIProject(t, srv2.URL)
|
||||||
if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0); exitCode(err) != 2 {
|
if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0, 0); exitCode(err) != 2 {
|
||||||
t.Fatalf("setup: the run must finish WITH a flagged chunk (exit 2), got %v", err)
|
t.Fatalf("setup: the run must finish WITH a flagged chunk (exit 2), got %v", err)
|
||||||
}
|
}
|
||||||
refuse = false
|
refuse = false
|
||||||
|
|
@ -246,3 +247,76 @@ func TestTranslateAndRedriveWireTheRunCeiling(t *testing.T) {
|
||||||
t.Fatalf("redrive must carry --ceiling-usd into the re-attack; got %v", err)
|
t.Fatalf("redrive must carry --ceiling-usd into the re-attack; got %v", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// setupCLIProjectMulti is setupCLIProject with a source long enough — and a draft cut fine enough — to
|
||||||
|
// produce SEVERAL output units, which is what a volume ceiling needs in order to bite at all.
|
||||||
|
func setupCLIProjectMulti(t *testing.T, providerURL string) string {
|
||||||
|
t.Helper()
|
||||||
|
bookPath := setupCLIProject(t, providerURL)
|
||||||
|
dir := filepath.Dir(bookPath)
|
||||||
|
writeCLIFile(t, filepath.Join(dir, "source.txt"),
|
||||||
|
"静かな図書館の朝。鈴木は本を読んだ。外では雨が降っていた。彼は窓を見た。時間は静かに過ぎた。夜になった。")
|
||||||
|
writeCLIFile(t, filepath.Join(dir, "pipeline.yaml"), `
|
||||||
|
core: C1
|
||||||
|
version: 1
|
||||||
|
defaults: { max_output_ratio: 2.0, min_max_tokens: 512 }
|
||||||
|
retries: { regenerate_before_escalate: 0 }
|
||||||
|
context: { glossary_injection: selective, glossary_token_budget: 800 }
|
||||||
|
segmentation:
|
||||||
|
draft_budget_out: 24
|
||||||
|
edit_ceiling_out: 48
|
||||||
|
fertility: { cjk: 1.1978, other: 0.3852 }
|
||||||
|
stages:
|
||||||
|
- { name: draft, role: translator, model: fake-model, prompt_override: prompts/translator.md, prompt_version: v-cli, temperature: 0.3, reasoning: "off" }
|
||||||
|
`)
|
||||||
|
return bookPath
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTranslateWiresMaxUnits is the same kind of pin as TestTranslateWiresAcceptRebill above, for the
|
||||||
|
// other run ceiling: it proves the flag changes what the RUN DOES, not merely what the parser returns.
|
||||||
|
//
|
||||||
|
// Every other proof of the volume ceiling sets Runner.MaxUnits directly, which leaves exactly one line
|
||||||
|
// unverified — `r.MaxUnits = maxUnits` in main.go's translate(). That line is the entire seam between the
|
||||||
|
// flag a platform passes and the engine that honours it; dropping it would leave every volume test in the
|
||||||
|
// pipeline package green while `tmctl translate --max-units 2` translated the whole book.
|
||||||
|
func TestTranslateWiresMaxUnits(t *testing.T) {
|
||||||
|
var mu sync.Mutex
|
||||||
|
calls := 0
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
_, _ = io.ReadAll(r.Body)
|
||||||
|
mu.Lock()
|
||||||
|
calls++
|
||||||
|
mu.Unlock()
|
||||||
|
tb, _ := json.Marshal("ЧЕРНОВИК ПЕРЕВОДА")
|
||||||
|
fmt.Fprintf(w, `{"id":"fake","model":"fake-model","choices":[{"message":{"content":%s},"finish_reason":"stop"}],
|
||||||
|
"usage":{"prompt_tokens":1000,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":200}}}`, tb)
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
bookPath := setupCLIProjectMulti(t, srv.URL)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
// Unbounded first, on a throwaway copy of the project, to learn how many units this book really has —
|
||||||
|
// so the bounded assertion below is against a measured number rather than a guessed one.
|
||||||
|
fullPath := setupCLIProjectMulti(t, srv.URL)
|
||||||
|
if err := translate(ctx, fullPath, false, pipeline.RebillConsent{}, false, 0, 0); err != nil {
|
||||||
|
t.Fatalf("the unbounded baseline must complete: %v", err)
|
||||||
|
}
|
||||||
|
mu.Lock()
|
||||||
|
total := calls
|
||||||
|
calls = 0
|
||||||
|
mu.Unlock()
|
||||||
|
if total < 3 {
|
||||||
|
t.Fatalf("fixture is too thin to bound: the whole book took %d call(s), so a ceiling of 2 proves nothing", total)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Now the same book through the CLI entry point WITH the flag.
|
||||||
|
if err := translate(ctx, bookPath, false, pipeline.RebillConsent{}, false, 0, 2); err != nil {
|
||||||
|
t.Fatalf("a volume-bounded run is a completion, not an error: %v", err)
|
||||||
|
}
|
||||||
|
mu.Lock()
|
||||||
|
bounded := calls
|
||||||
|
mu.Unlock()
|
||||||
|
if bounded != 2 {
|
||||||
|
t.Fatalf("`translate --max-units 2` made %d provider call(s) on a %d-unit book: the flag did not reach the engine (main.go's `r.MaxUnits = maxUnits`)", bounded, total)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -121,6 +121,36 @@ func renderTranslate(w io.Writer, res *pipeline.BookResult, ledger func() (commi
|
||||||
}
|
}
|
||||||
fmt.Fprintf(w, "Book ledger: committed=$%.6f reserved=$%.6f\n", committed, reserved)
|
fmt.Fprintf(w, "Book ledger: committed=$%.6f reserved=$%.6f\n", committed, reserved)
|
||||||
|
|
||||||
|
// The stop NAMES its ceiling. A run bounded by --max-units ends with exit 0 like any other completed
|
||||||
|
// run, so without this line the operator sees "TOTAL … chunks 3" on a hundred-unit book and has no way
|
||||||
|
// to tell "the volume I bought is done" from "something went quietly wrong" — the silent-limit
|
||||||
|
// complaint D39.165 §1б took from OpenHands. It names the ceiling, what was delivered as against
|
||||||
|
// re-made, and what remains of each kind, so the next action is readable off the line itself.
|
||||||
|
if v := res.Volume; v != nil {
|
||||||
|
fmt.Fprintf(w, "VOLUME CEILING: %s\n", v)
|
||||||
|
fmt.Fprintf(w, " (this is a COMPLETION, not a pause — the run did what it was granted; the money ceiling is a separate stop with its own exit code.)\n")
|
||||||
|
// ⚠ The invitation to buy more is spoken ONLY about units that were never delivered. The first
|
||||||
|
// version said "run again to buy more" unconditionally, which on a fully-translated book with a
|
||||||
|
// moved bank offered the reader chapters they already own — re-payment presented as stock.
|
||||||
|
//
|
||||||
|
// Both remainders are reported when both exist. An if/else here would have printed the fresh half
|
||||||
|
// and silently dropped the other, leaving an operator who has BOTH kinds pending believing the
|
||||||
|
// first number was all of it.
|
||||||
|
if v.LeftFresh > 0 {
|
||||||
|
fmt.Fprintf(w, " Next: %d output unit(s) of this book have never been delivered — `--max-units` again to buy them (units, NOT chapters: a chapter can be several units; the manifest is what converts).\n", v.LeftFresh)
|
||||||
|
}
|
||||||
|
if v.LeftRework > 0 {
|
||||||
|
if v.LeftFresh > 0 {
|
||||||
|
fmt.Fprintf(w, " Also pending, and NOT a purchase: %d already-delivered unit(s) carry a superseded snapshot and would be RE-MADE. Buying them delivers no new chapter.\n", v.LeftRework)
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(w, " Next: every unit of this book HAS been delivered; the %d left carry a superseded snapshot and would be RE-MADE, not delivered. That is a re-pass, not a purchase of new book.\n", v.LeftRework)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if v.Delivered == 0 && v.Reworked > 0 {
|
||||||
|
fmt.Fprintf(w, " ⚠ This run delivered NO new unit: all %d paid unit(s) were re-made under a moved snapshot.\n", v.Reworked)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Exit code 2 is a typed sentinel, not an infra error: the report above is
|
// Exit code 2 is a typed sentinel, not an infra error: the report above is
|
||||||
// already on stdout; main() maps this to a non-zero exit for the operator.
|
// already on stdout; main() maps this to a non-zero exit for the operator.
|
||||||
if res.Flagged > 0 {
|
if res.Flagged > 0 {
|
||||||
|
|
@ -663,7 +693,21 @@ func renderStatusHuman(w io.Writer, rep *pipeline.StatusReport, cfgPath string)
|
||||||
// It is the same projection `translate` refuses on, so the operator reads the decision number here
|
// It is the same projection `translate` refuses on, so the operator reads the decision number here
|
||||||
// instead of discovering it in a refusal.
|
// instead of discovering it in a refusal.
|
||||||
if rep.RebillUnits > 0 {
|
if rep.RebillUnits > 0 {
|
||||||
drift += fmt.Sprintf(" ⚠ RE-PAYMENT: %d chunk×stage unit(s) already billed under a superseded snapshot would be paid for again, ~$%.6f (translate needs --accept-rebill above the book's consent threshold)", rep.RebillUnits, rep.RebillUSD)
|
// BOTH denominations, always, and labelled. The chunk×stage count is what the money is billed in;
|
||||||
|
// the output-unit count is what a purchase is sized in. Printing only the first is how a reader
|
||||||
|
// hands `--max-units` a number several times too large.
|
||||||
|
drift += fmt.Sprintf(" ⚠ RE-PAYMENT: %d chunk×stage unit(s) already billed under a superseded snapshot would be paid for again — that is %d OUTPUT unit(s) of the book, the granularity --max-units counts in — ~$%.6f (translate needs --accept-rebill above the book's consent threshold). ⚠ Sizing --max-units from that number does NOT buy those units back: a grant goes to NEVER-DELIVERED units first and only reaches re-making when none are left, so on a book that still has undelivered units it will deliver new ones instead",
|
||||||
|
rep.RebillUnits, rep.RebillOutputUnits, rep.RebillUSD)
|
||||||
|
}
|
||||||
|
// A zero is not self-explanatory, and printing it as though it were is the mistake the `rebill_basis`
|
||||||
|
// field exists to stop: "nothing to re-pay" and "we could not work out what a re-pass would cost" are
|
||||||
|
// the same two zeroes. Only the two states that are NOT an answer speak here — a computed figure needs
|
||||||
|
// no caveat, and a book with nothing stored has nothing to say about re-payment at all.
|
||||||
|
switch rep.RebillBasis {
|
||||||
|
case pipeline.RebillBasisFailed:
|
||||||
|
drift += " ⚠ RE-PAYMENT UNKNOWN (the projection failed — the figures below are zero because they could not be computed, NOT because continuing is free; see the log)"
|
||||||
|
case pipeline.RebillBasisStored:
|
||||||
|
drift += " ⚠ RE-PAYMENT FIGURES ARE STALE (the book's bank could not be folded, so they describe the glossary the LAST run stored, not what the next run would build — fix the bank and re-read; see the log)"
|
||||||
}
|
}
|
||||||
fmt.Fprintf(w, "=== STATUS: %s (snapshot %s)%s ===\n", rep.BookID, snap, drift)
|
fmt.Fprintf(w, "=== STATUS: %s (snapshot %s)%s ===\n", rep.BookID, snap, drift)
|
||||||
fmt.Fprintf(w, "Progress: %d/%d units (%.1f%%) — done=%d in_progress=%d flagged=%d pending=%d\n",
|
fmt.Fprintf(w, "Progress: %d/%d units (%.1f%%) — done=%d in_progress=%d flagged=%d pending=%d\n",
|
||||||
|
|
|
||||||
|
|
@ -321,3 +321,106 @@ func TestRenderSignatureStopOrdersTheReviewLeastSureFirst(t *testing.T) {
|
||||||
t.Fatalf("the banner must say what its order means, or a sorted table reads as arbitrary:\n%s", out)
|
t.Fatalf("the banner must say what its order means, or a sorted table reads as arbitrary:\n%s", out)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestRenderTranslateNamesTheVolumeCeiling pins the operator-facing half of the "the stop must NAME its
|
||||||
|
// ceiling" rule. A volume-bounded run exits 0 like any completed run, so without this line an operator
|
||||||
|
// looking at "chunks 2" on a hundred-unit book cannot tell "the volume I bought is finished" from
|
||||||
|
// "something went quietly wrong" — the silent-limit complaint the design took from OpenHands. The line
|
||||||
|
// has to carry all three actionable numbers and to say it is a completion, not a pause.
|
||||||
|
func TestRenderTranslateNamesTheVolumeCeiling(t *testing.T) {
|
||||||
|
var b strings.Builder
|
||||||
|
res := &pipeline.BookResult{BookID: "b1", TotalUSD: 0.01,
|
||||||
|
Volume: &pipeline.VolumeStop{MaxUnits: 2, Delivered: 1, Reworked: 1, Free: 3, LeftFresh: 5, LeftRework: 2}}
|
||||||
|
if err := renderTranslate(&b, res, okLedger); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
out := b.String()
|
||||||
|
for _, want := range []string{"VOLUME CEILING", "--max-units 2", "2 paying output unit(s)",
|
||||||
|
"1 NEW unit(s) delivered", "1 already-delivered unit(s) re-made",
|
||||||
|
"3 rode along at $0", "5 unit(s) NEVER delivered", "2 already delivered but not yet re-made",
|
||||||
|
"COMPLETION, not a pause",
|
||||||
|
// The invitation must be about the units that were never delivered, and only those.
|
||||||
|
"5 output unit(s) of this book have never been delivered"} {
|
||||||
|
if !strings.Contains(out, want) {
|
||||||
|
t.Errorf("the volume stop must say %q; got:\n%s", want, out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// A run that was NOT bounded must say nothing at all — otherwise every ordinary run reads as cut short.
|
||||||
|
var b2 strings.Builder
|
||||||
|
if err := renderTranslate(&b2, &pipeline.BookResult{BookID: "b1"}, okLedger); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if strings.Contains(b2.String(), "VOLUME CEILING") {
|
||||||
|
t.Errorf("an unbounded run must not mention a ceiling:\n%s", b2.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRenderTranslateNeverSellsWhatIsAlreadyOwned pins the operator-facing half of the same defect. On a
|
||||||
|
// book every unit of which has been delivered, the stop must not invite a purchase: the units it held
|
||||||
|
// back are unrefreshed, not unbought, and the first version of this line offered them as "run again with
|
||||||
|
// --max-units to buy more".
|
||||||
|
func TestRenderTranslateNeverSellsWhatIsAlreadyOwned(t *testing.T) {
|
||||||
|
var b strings.Builder
|
||||||
|
res := &pipeline.BookResult{BookID: "b1",
|
||||||
|
Volume: &pipeline.VolumeStop{MaxUnits: 1, Delivered: 0, Reworked: 1, LeftFresh: 0, LeftRework: 3}}
|
||||||
|
if err := renderTranslate(&b, res, okLedger); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
out := b.String()
|
||||||
|
if strings.Contains(out, "have never been delivered") {
|
||||||
|
t.Fatalf("the stop offered chapters the reader already owns:\n%s", out)
|
||||||
|
}
|
||||||
|
for _, want := range []string{
|
||||||
|
"every unit of this book HAS been delivered",
|
||||||
|
"That is a re-pass, not a purchase of new book",
|
||||||
|
"delivered NO new unit",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(out, want) {
|
||||||
|
t.Errorf("want %q in:\n%s", want, out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRenderTranslateReportsBothRemaindersWhenBothExist pins the case an if/else would have swallowed:
|
||||||
|
// a book with BOTH never-delivered units and delivered-but-superseded ones. Printing only the first
|
||||||
|
// leaves an operator who has both believing the one number was the whole picture.
|
||||||
|
func TestRenderTranslateReportsBothRemaindersWhenBothExist(t *testing.T) {
|
||||||
|
var b strings.Builder
|
||||||
|
res := &pipeline.BookResult{BookID: "b1",
|
||||||
|
Volume: &pipeline.VolumeStop{MaxUnits: 2, Delivered: 2, LeftFresh: 4, LeftRework: 6}}
|
||||||
|
if err := renderTranslate(&b, res, okLedger); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
out := b.String()
|
||||||
|
if !strings.Contains(out, "4 output unit(s) of this book have never been delivered") {
|
||||||
|
t.Errorf("the buyable remainder is missing:\n%s", out)
|
||||||
|
}
|
||||||
|
if !strings.Contains(out, "6 already-delivered unit(s) carry a superseded snapshot") {
|
||||||
|
t.Errorf("the re-make remainder was dropped — an if/else over the two would do exactly this:\n%s", out)
|
||||||
|
}
|
||||||
|
// A run that DID deliver must not carry the "delivered NO new unit" warning.
|
||||||
|
if strings.Contains(out, "delivered NO new unit") {
|
||||||
|
t.Errorf("this run delivered 2 units; it must not claim otherwise:\n%s", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTheRePaymentHintDoesNotPromiseWhatTheGrantWillNotDo pins the acceptance hunter's finding 6. The
|
||||||
|
// hint offers rebill_output_units as "the granularity --max-units counts in", which is true and, on its
|
||||||
|
// own, misleading: the grant goes to NEVER-DELIVERED units first, so on a book that still has any, sizing
|
||||||
|
// --max-units from the re-payment figure delivers new units and re-makes nothing. The caveat lives in the
|
||||||
|
// code; it has to live where the operator reads.
|
||||||
|
func TestTheRePaymentHintDoesNotPromiseWhatTheGrantWillNotDo(t *testing.T) {
|
||||||
|
var b strings.Builder
|
||||||
|
rep := &pipeline.StatusReport{BookID: "b1", ConfigDrift: true,
|
||||||
|
RebillUnits: 9, RebillOutputUnits: 3, RebillUSD: 0.02, RebillBasis: pipeline.RebillBasisPending}
|
||||||
|
if err := renderStatusHuman(&b, rep, ""); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
out := b.String()
|
||||||
|
if !strings.Contains(out, "3 OUTPUT unit(s)") {
|
||||||
|
t.Fatalf("the sale-sized figure must be shown:\n%s", out)
|
||||||
|
}
|
||||||
|
if !strings.Contains(out, "does NOT buy those units back") {
|
||||||
|
t.Fatalf("the hint must say the grant goes to never-delivered units first, or it promises a re-pass it will not perform:\n%s", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,9 @@ import (
|
||||||
|
|
||||||
// artifact.go: the shared write discipline of the engine's READ-OUT FILES — the sidecars beside the
|
// artifact.go: the shared write discipline of the engine's READ-OUT FILES — the sidecars beside the
|
||||||
// project DB that another process (the platform, D39.81/D39.85) reads while this one runs: the
|
// project DB that another process (the platform, D39.81/D39.85) reads while this one runs: the
|
||||||
// chapter/chunk manifest (row 100), the machine bank-stop table (row 101), the bank export (row 125).
|
// chapter/chunk manifest (row 100), the machine bank-stop table (row 101), the bank export (row 125) and
|
||||||
|
// the engine's auto-bank (row 231 — it joined this list when the read-only surfaces started folding the
|
||||||
|
// bank from the decision files themselves, which is what made it a concurrently-read document).
|
||||||
//
|
//
|
||||||
// They have one property in common that the older sidecars did not have to care about: they are read
|
// They have one property in common that the older sidecars did not have to care about: they are read
|
||||||
// CONCURRENTLY with the run that writes them. A plain os.WriteFile truncates first, so a reader that
|
// CONCURRENTLY with the run that writes them. A plain os.WriteFile truncates first, so a reader that
|
||||||
|
|
|
||||||
403
backend/internal/pipeline/bankmaterialize.go
Normal file
403
backend/internal/pipeline/bankmaterialize.go
Normal file
|
|
@ -0,0 +1,403 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"sort"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/lang"
|
||||||
|
"textmachine/backend/internal/membank"
|
||||||
|
"textmachine/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// bankmaterialize.go: the book's bank fold, split so that the SAME fold can be performed with or without the
|
||||||
|
// store round-trip that sits in the middle of it (row 231 / D39.165 §3, errata 28.08-к).
|
||||||
|
//
|
||||||
|
// THE PROBLEM IT EXISTS FOR. `seedGlossary` folds the bank from FILES (the curated seed, the ruby
|
||||||
|
// readings, the owner's mined-delta, the engine's auto-bank), writes the result with ReplaceBank, reads
|
||||||
|
// it straight back with GlossaryForBook and materializes THAT. Every read-only surface, however, had no
|
||||||
|
// way to reach the same answer: it cannot write, so `projectStoredMemory` materialized the glossary the
|
||||||
|
// LAST run stored — i.e. the previous fold. Right after a `bank-apply` the decision files hold rows the
|
||||||
|
// stored glossary does not, so a read-only projection answered "nothing moved" while the very next
|
||||||
|
// `translate` would re-seed, see the rows and bill for them. The projection and the run disagreed about
|
||||||
|
// money, which is the one thing they may never do.
|
||||||
|
//
|
||||||
|
// WHY THE ROUND-TRIP IS NOT A FORMALITY, and what makes skipping it PROVABLE rather than plausible.
|
||||||
|
// The store contributes exactly one thing to the fold: an ORDER. `GlossaryForBook` returns
|
||||||
|
// `ORDER BY src, sense, since_ch, until_ch, status, dst`, and that order is load-bearing twice over —
|
||||||
|
// `MaterializeBank` iterates the rows as given (it never re-sorts), so the order decides both the bank's
|
||||||
|
// content hash (ComputeVersionScopedIn streams a SHA-256 over the rows in iteration order) and, through
|
||||||
|
// Select's stable budget sort, the literal LINE ORDER of the rendered glossary block, whose bytes are
|
||||||
|
// hashed into chunk_status.content_hash. Reproduce the set but not the order and the free estimate names
|
||||||
|
// one number while the paid run charges another.
|
||||||
|
//
|
||||||
|
// The reproduction is exact, and the reason is a constraint rather than an observation: `glossary`
|
||||||
|
// carries UNIQUE (book_id, src, sense, since_ch, until_ch) (store/migrate.go), so the ORDER BY's first
|
||||||
|
// four columns CANNOT tie within one book — the order is total and fully determined by values, with no
|
||||||
|
// dependence on rowid or insertion order (`glossary.id` is deliberately excluded from the fold, see
|
||||||
|
// membank.ComputeVersion). The schema declares no COLLATE anywhere, so SQLite compares TEXT with its
|
||||||
|
// default BINARY collation — byte-wise, which is exactly what Go's `<` on a string does. Aliases come
|
||||||
|
// back `ORDER BY term_id, alias`, i.e. sorted by alias within an entry. Voice profiles and address pairs
|
||||||
|
// each have their own ORDER BY matching their own UNIQUE key, so the same argument covers them.
|
||||||
|
//
|
||||||
|
// storeOrder below therefore reproduces the round-trip's ONE contribution, and bankmaterialize_identity_test.go
|
||||||
|
// proves the equality by running both paths over the same inputs rather than by asserting this comment.
|
||||||
|
|
||||||
|
// bankInputs is everything the deterministic $0 prelude gathers about a book's bank BEFORE anything is
|
||||||
|
// written: the glossary rows in the order seedGlossary builds them (which is the order ReplaceBank
|
||||||
|
// inserts them in), the two pack-19 record types, and the non-fatal remarks the gather produced.
|
||||||
|
//
|
||||||
|
// It is deliberately the PRE-store form. Turning it into what a run actually materializes is storeOrder's
|
||||||
|
// job, and it is one function so a caller cannot accidentally materialize the unsorted build order.
|
||||||
|
type bankInputs struct {
|
||||||
|
entries []store.GlossaryEntry
|
||||||
|
voices []store.VoiceProfile
|
||||||
|
pairs []store.AddressPair
|
||||||
|
// remarks are the gather's non-fatal findings, in the order they were made. seedGlossary logs them;
|
||||||
|
// a read-only projection drops them, because a $0 read must not narrate a run it is not making.
|
||||||
|
remarks []bankRemark
|
||||||
|
}
|
||||||
|
|
||||||
|
// bankRemark is one non-fatal finding of the gather, carried rather than logged so the two callers can
|
||||||
|
// decide for themselves whether their surface is one that speaks.
|
||||||
|
type bankRemark struct {
|
||||||
|
msg string
|
||||||
|
args []any
|
||||||
|
}
|
||||||
|
|
||||||
|
// gatherBankInputs performs the pure, $0, WRITE-FREE half of the bank fold: it reads the book's
|
||||||
|
// deterministic inputs and assembles the exact row set seedGlossary hands to ReplaceBank.
|
||||||
|
//
|
||||||
|
// Everything fatal here is fatal for the same reason it was fatal inside seedGlossary — a collision that
|
||||||
|
// would crash ReplaceBank mid-run, or a voice row naming a character the bank does not have, is a broken
|
||||||
|
// book and not a projection detail. What the two callers differ in is what they DO with that error:
|
||||||
|
// `translate` fails the run, a read-only projection falls back and says so.
|
||||||
|
func (r *Runner) gatherBankInputs() (bankInputs, error) {
|
||||||
|
var in bankInputs
|
||||||
|
if r.Book.GlossarySeed != "" {
|
||||||
|
bank, err := membank.LoadBankSeed(r.Book.GlossarySeed)
|
||||||
|
if err != nil {
|
||||||
|
return in, err
|
||||||
|
}
|
||||||
|
in.entries = append(in.entries, bank.Terms...)
|
||||||
|
in.voices, in.pairs = bank.Voices, bank.Pairs
|
||||||
|
}
|
||||||
|
ruby, err := r.Store.RubyReadingsForBook(r.Book.BookID)
|
||||||
|
if err != nil {
|
||||||
|
return in, fmt.Errorf("pipeline: read ruby readings for %s: %w", r.Book.BookID, err)
|
||||||
|
}
|
||||||
|
// D16.4: attach a manual term's kana ruby-reading as an alias (kana spelling matchable)
|
||||||
|
// BEFORE appending the auto-candidates, so it only touches the curated manual entries. A
|
||||||
|
// reading that would collide with a different seeded term (homophone) is skipped+logged,
|
||||||
|
// not attached (which would fail the book loud on an alias the operator cannot edit out).
|
||||||
|
if skipped := membank.AttachRubyAliasesToManual(in.entries, ruby); len(skipped) > 0 {
|
||||||
|
in.remark("ruby kana-alias skipped as a homophone collision (kana form left unmatchable; disambiguate in the seed if needed)",
|
||||||
|
"skipped", joinSemi(skipped))
|
||||||
|
}
|
||||||
|
// Mined-write path (R1, plan §1(в)/F2): the owner-curated mined-delta file joins the seed as
|
||||||
|
// Source:"mined" (loadMinedDelta re-stamps the seed loader's Source), so its terms fold into the
|
||||||
|
// ENRICHED bank version but NOT the base — a re-run that adds signed mined terms moves ONLY snapshot_W2.
|
||||||
|
minedDelta, err := r.loadMinedDelta()
|
||||||
|
if err != nil {
|
||||||
|
return in, err
|
||||||
|
}
|
||||||
|
// D39.20 deviation-#1 fix: a mined-delta term whose UNIQUE key (src, sense, since_ch, until_ch —
|
||||||
|
// store/migrate.go glossary UNIQUE) already exists in the SIGNED seed makes the flat INSERT in
|
||||||
|
// ReplaceGlossary crash on that constraint and abort the whole paid run. membank.ApprovedSharedKeyCollisions
|
||||||
|
// below does NOT catch it (it skips a SAME-dst duplicate, and keys on the firing surface, not the
|
||||||
|
// UNIQUE tuple). Fail LOUD here with the duplicate list + a fix hint (edit the seed OR drop it from
|
||||||
|
// the delta) — NEVER a silent merge over the signed seed. Checked BEFORE the append so the delta rows
|
||||||
|
// are still separable. Deterministic (seed order).
|
||||||
|
if dups := membank.MinedDeltaSeedCollisions(in.entries, minedDelta); len(dups) > 0 {
|
||||||
|
return in, fmt.Errorf("pipeline: mined-delta %s duplicates term(s) already in the signed seed (would crash ReplaceGlossary on the glossary UNIQUE(book_id,src,sense,since_ch,until_ch)):\n - %s\n fix: correct the term in the seed, or remove it from the mined-delta file — never both (no silent merge over the signed seed)",
|
||||||
|
r.Book.MinedDelta, joinLines(dups))
|
||||||
|
}
|
||||||
|
in.entries = append(in.entries, minedDelta...)
|
||||||
|
// AUTO-BANK (pack-20 / D39.42 п.3): the engine's own unsigned rows — what the terminologist consolidated
|
||||||
|
// on the last run's bank-mining boundary. They load exactly like the owner-curated delta (Source:"mined",
|
||||||
|
// so base-excluded and only the edit-wave snapshot moves) but they are NOT signed: their status is
|
||||||
|
// auto/draft, which is what routes them to the editor's separately-headed unverified section instead of
|
||||||
|
// the canon list.
|
||||||
|
//
|
||||||
|
// Loading them HERE rather than injecting them mid-run is load-bearing for money. The gather runs before
|
||||||
|
// checkRebillConsent, so the re-payment projection sees the same bank the edit wave will use; if the rows
|
||||||
|
// only appeared after the projection, the next run would compare stored edit rows against a bank that has
|
||||||
|
// not been rebuilt yet and project a re-payment of the whole edit wave that is not real.
|
||||||
|
autoBank, dropped, err := r.loadAutoBank(in.entries)
|
||||||
|
if err != nil {
|
||||||
|
return in, err
|
||||||
|
}
|
||||||
|
if len(dropped) > 0 {
|
||||||
|
// A collision with a SIGNED row cannot abort a paid run over an engine-written proposal — the signed
|
||||||
|
// row simply wins and the proposal is dropped, loudly.
|
||||||
|
in.remark("auto-bank rows dropped: their key is already held by a signed term (the signed term wins)",
|
||||||
|
"book", r.Book.BookID, "dropped", joinSemi(dropped))
|
||||||
|
}
|
||||||
|
in.entries = append(in.entries, autoBank...)
|
||||||
|
for i := range in.entries {
|
||||||
|
in.entries[i].BookID = r.Book.BookID
|
||||||
|
}
|
||||||
|
// Task-6 re-audit: fail loud on a firing key shared by two DIFFERENT approved terms with
|
||||||
|
// different dst + overlapping windows (the alias generalization of the D16.1 polysemy
|
||||||
|
// livelock) — checked over the FULL entry set (incl. ruby-attached aliases) before persisting.
|
||||||
|
if cols := membank.ApprovedSharedKeyCollisions(in.entries); len(cols) > 0 {
|
||||||
|
return in, fmt.Errorf("pipeline: glossary shared-key collisions (A2 / D16.1 livelock class):\n - %s", joinLines(cols))
|
||||||
|
}
|
||||||
|
// A voice/address row naming a character the bank does not have is SILENTLY inert — nothing can ever
|
||||||
|
// attribute a reply to it — which is the A-class hole this bank exists to close, so it stops the run.
|
||||||
|
// Checked over the FULL entry set (seed + ruby + mined + auto), because a character may legitimately be
|
||||||
|
// signed in a delta rather than in the base seed.
|
||||||
|
if unknown := membank.UnknownVoiceCharacters(in.entries, in.voices, in.pairs); len(unknown) > 0 {
|
||||||
|
return in, fmt.Errorf("pipeline: voice/address rows name characters absent from the bank (a profile for a term that does not exist can never fire):\n - %s", joinLines(unknown))
|
||||||
|
}
|
||||||
|
return in, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (in *bankInputs) remark(msg string, args ...any) {
|
||||||
|
in.remarks = append(in.remarks, bankRemark{msg: msg, args: args})
|
||||||
|
}
|
||||||
|
|
||||||
|
// storeOrder reproduces, WITHOUT a store, exactly what a ReplaceBank+read-back round-trip returns: the
|
||||||
|
// same rows in the store's ORDER BY, each entry's aliases in the store's alias order, and the same for
|
||||||
|
// voice profiles and address pairs. The inputs are left untouched — a caller may still hand the ORIGINAL
|
||||||
|
// slice to ReplaceBank and get the insert order it has always had.
|
||||||
|
//
|
||||||
|
// It REFUSES the states the store itself would refuse rather than papering over them, because a
|
||||||
|
// projection of a book whose fold would crash the next run is not a projection of anything: a duplicate
|
||||||
|
// UNIQUE key would abort ReplaceBank's flat INSERT mid-transaction, and reporting a tidy number for it
|
||||||
|
// would promise a run that cannot start. The three refusals correspond one-to-one to the three UNIQUE
|
||||||
|
// constraints the round-trip passes through.
|
||||||
|
func storeOrder(in bankInputs) (rows []store.GlossaryEntry, voices []store.VoiceProfile, pairs []store.AddressPair, err error) {
|
||||||
|
type gkey struct {
|
||||||
|
src, sense string
|
||||||
|
since, until int
|
||||||
|
}
|
||||||
|
rows = append(rows, in.entries...)
|
||||||
|
seen := make(map[gkey]bool, len(rows))
|
||||||
|
for i := range rows {
|
||||||
|
k := gkey{rows[i].Src, rows[i].Sense, rows[i].SinceCh, rows[i].UntilCh}
|
||||||
|
if seen[k] {
|
||||||
|
return nil, nil, nil, fmt.Errorf("pipeline: the folded bank holds two rows for (src=%q, sense=%q, since_ch=%d, until_ch=%d) — the glossary UNIQUE key; ReplaceBank would abort on it", k.src, k.sense, k.since, k.until)
|
||||||
|
}
|
||||||
|
seen[k] = true
|
||||||
|
// The store returns a term's aliases ORDER BY alias (within its term_id), and refuses a repeated
|
||||||
|
// one on UNIQUE (book_id, term_id, alias). The aliases are copied before sorting so an input slice
|
||||||
|
// shared with the write path keeps the order ReplaceBank inserts in.
|
||||||
|
if len(rows[i].Aliases) > 1 {
|
||||||
|
al := append([]store.GlossaryAlias(nil), rows[i].Aliases...)
|
||||||
|
sort.SliceStable(al, func(a, b int) bool { return al[a].Alias < al[b].Alias })
|
||||||
|
for j := 1; j < len(al); j++ {
|
||||||
|
if al[j].Alias == al[j-1].Alias {
|
||||||
|
return nil, nil, nil, fmt.Errorf("pipeline: the folded bank gives term %q the alias %q twice — the glossary_aliases UNIQUE key; ReplaceBank would abort on it", rows[i].Src, al[j].Alias)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rows[i].Aliases = al
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// GlossaryForBook: ORDER BY src, sense, since_ch, until_ch, status, dst. The first four columns are the
|
||||||
|
// table's UNIQUE key, so status/dst can never actually decide anything — they are kept so this reads as
|
||||||
|
// the SQL it reproduces and would still hold if that constraint were ever widened.
|
||||||
|
sort.SliceStable(rows, func(a, b int) bool { return glossaryLess(rows[a], rows[b]) })
|
||||||
|
|
||||||
|
voices = append(voices, in.voices...)
|
||||||
|
vseen := make(map[gkey]bool, len(voices))
|
||||||
|
for _, v := range voices {
|
||||||
|
k := gkey{v.Src, v.Sense, v.SinceCh, v.UntilCh}
|
||||||
|
if vseen[k] {
|
||||||
|
return nil, nil, nil, fmt.Errorf("pipeline: the folded bank holds two voice profiles for (src=%q, sense=%q, since_ch=%d, until_ch=%d) — the voice_profiles UNIQUE key; ReplaceBank would abort on it", k.src, k.sense, k.since, k.until)
|
||||||
|
}
|
||||||
|
vseen[k] = true
|
||||||
|
}
|
||||||
|
// VoiceProfilesForBook: ORDER BY src, sense, since_ch, until_ch.
|
||||||
|
sort.SliceStable(voices, func(a, b int) bool {
|
||||||
|
x, y := voices[a], voices[b]
|
||||||
|
return lessKey4(x.Src, y.Src, x.Sense, y.Sense, x.SinceCh, y.SinceCh, x.UntilCh, y.UntilCh)
|
||||||
|
})
|
||||||
|
|
||||||
|
type pkey struct {
|
||||||
|
spSrc, spSense, adSrc, adSense string
|
||||||
|
since, until int
|
||||||
|
}
|
||||||
|
pairs = append(pairs, in.pairs...)
|
||||||
|
pseen := make(map[pkey]bool, len(pairs))
|
||||||
|
for _, p := range pairs {
|
||||||
|
k := pkey{p.SpeakerSrc, p.SpeakerSense, p.AddresseeSrc, p.AddresseeSense, p.SinceCh, p.UntilCh}
|
||||||
|
if pseen[k] {
|
||||||
|
return nil, nil, nil, fmt.Errorf("pipeline: the folded bank holds two address pairs for speaker %q → addressee %q over the same window — the address_pairs UNIQUE key; ReplaceBank would abort on it", k.spSrc, k.adSrc)
|
||||||
|
}
|
||||||
|
pseen[k] = true
|
||||||
|
}
|
||||||
|
// AddressPairsForBook: ORDER BY speaker_src, speaker_sense, addressee_src, addressee_sense, since_ch, until_ch.
|
||||||
|
sort.SliceStable(pairs, func(a, b int) bool {
|
||||||
|
x, y := pairs[a], pairs[b]
|
||||||
|
if x.SpeakerSrc != y.SpeakerSrc {
|
||||||
|
return x.SpeakerSrc < y.SpeakerSrc
|
||||||
|
}
|
||||||
|
if x.SpeakerSense != y.SpeakerSense {
|
||||||
|
return x.SpeakerSense < y.SpeakerSense
|
||||||
|
}
|
||||||
|
if x.AddresseeSrc != y.AddresseeSrc {
|
||||||
|
return x.AddresseeSrc < y.AddresseeSrc
|
||||||
|
}
|
||||||
|
if x.AddresseeSense != y.AddresseeSense {
|
||||||
|
return x.AddresseeSense < y.AddresseeSense
|
||||||
|
}
|
||||||
|
if x.SinceCh != y.SinceCh {
|
||||||
|
return x.SinceCh < y.SinceCh
|
||||||
|
}
|
||||||
|
return x.UntilCh < y.UntilCh
|
||||||
|
})
|
||||||
|
return rows, voices, pairs, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// glossaryLess is GlossaryForBook's ORDER BY as a Go comparison. Both sides are byte-wise on TEXT (the
|
||||||
|
// schema declares no COLLATE, so SQLite uses BINARY) and numeric on the two INTEGER columns.
|
||||||
|
func glossaryLess(x, y store.GlossaryEntry) bool {
|
||||||
|
if x.Src != y.Src {
|
||||||
|
return x.Src < y.Src
|
||||||
|
}
|
||||||
|
if x.Sense != y.Sense {
|
||||||
|
return x.Sense < y.Sense
|
||||||
|
}
|
||||||
|
if x.SinceCh != y.SinceCh {
|
||||||
|
return x.SinceCh < y.SinceCh
|
||||||
|
}
|
||||||
|
if x.UntilCh != y.UntilCh {
|
||||||
|
return x.UntilCh < y.UntilCh
|
||||||
|
}
|
||||||
|
if x.Status != y.Status {
|
||||||
|
return x.Status < y.Status
|
||||||
|
}
|
||||||
|
return x.Dst < y.Dst
|
||||||
|
}
|
||||||
|
|
||||||
|
func lessKey4(s1, s2, e1, e2 string, a1, a2, b1, b2 int) bool {
|
||||||
|
if s1 != s2 {
|
||||||
|
return s1 < s2
|
||||||
|
}
|
||||||
|
if e1 != e2 {
|
||||||
|
return e1 < e2
|
||||||
|
}
|
||||||
|
if a1 != a2 {
|
||||||
|
return a1 < a2
|
||||||
|
}
|
||||||
|
return b1 < b2
|
||||||
|
}
|
||||||
|
|
||||||
|
// materializeBanks builds the two banks a run works against — the ENRICHED one the editor sees and the
|
||||||
|
// BASE one (Source:"mined" excluded) the draft wave selects over — from bank content already in the
|
||||||
|
// store's order. It is the shared tail of both folds, so the write path and the read-only projection can
|
||||||
|
// never materialize the same rows two different ways.
|
||||||
|
func (r *Runner) materializeBanks(rows []store.GlossaryEntry, voices []store.VoiceProfile, pairs []store.AddressPair) {
|
||||||
|
// InjectVoice is FALSE and has no config knob: pack-19 builds the schema and the flagger, and D21 п.2
|
||||||
|
// holds the injection conditional until the polygon experiment. It is the fold's condition, so while
|
||||||
|
// it is false a book with voice rows hashes exactly as it did without them and nobody re-pays for
|
||||||
|
// authoring a profile. Wiring the injection means setting it and accepting a full --resnapshot.
|
||||||
|
// The §3 decl stemmer is target data, constant per book; baseIn copies bankIn below, so both banks share
|
||||||
|
// it. A target with no decl_suffix registry yields an inert stemmer → exact-match post-check as before.
|
||||||
|
// The rows are retained because a caller that has just folded the bank must be able to say things
|
||||||
|
// ABOUT it — how many of its terms are unsigned, for one — without going back to the store and
|
||||||
|
// getting a DIFFERENT bank than the one it just materialized. A status document that folds the
|
||||||
|
// decision files for its money figures and counts unsigned terms off the stored glossary is one
|
||||||
|
// document describing two banks.
|
||||||
|
r.bankRows = rows
|
||||||
|
// The wire-hash memo belongs to the bank that produced it: a new materialization invalidates every
|
||||||
|
// hash rendered against the old one (cachedRenderedContentHashes).
|
||||||
|
r.contentHashes = nil
|
||||||
|
bankIn := membank.BankInput{Rows: rows, Voices: voices, Pairs: pairs,
|
||||||
|
TargetStemmer: lang.NewTargetStemmer(lang.TargetChecksFor(r.Book.TargetLang))}
|
||||||
|
r.memory = membank.MaterializeBank(bankIn, r.Pipeline.Gates.Glossary.PostcheckGate)
|
||||||
|
// The DRAFT wave selects over a BASE-scoped bank (Source:mined excluded) so its injection is
|
||||||
|
// byte-identical across a bank-mining enrichment — matching the draft-wave snapshot (baseMemoryVersion),
|
||||||
|
// which keeps «one re-payment» honest at the WIRE level, not only the version-hash level. Only the
|
||||||
|
// editor sees mined terms (the enriched `memory`). When there are no mined rows (every $0 test / the
|
||||||
|
// golden) the base bank IS the enriched one — share the object, no double materialization, no drift.
|
||||||
|
baseRows := rows[:0:0]
|
||||||
|
hasMined := false
|
||||||
|
for _, row := range rows {
|
||||||
|
if row.Source == "mined" {
|
||||||
|
hasMined = true
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
baseRows = append(baseRows, row)
|
||||||
|
}
|
||||||
|
if hasMined {
|
||||||
|
baseIn := bankIn
|
||||||
|
baseIn.Rows = baseRows
|
||||||
|
r.baseMemory = membank.MaterializeBank(baseIn, r.Pipeline.Gates.Glossary.PostcheckGate)
|
||||||
|
} else {
|
||||||
|
r.baseMemory = r.memory
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// projectFoldedMemory materializes BOTH banks from the book's deterministic inputs WITHOUT writing
|
||||||
|
// anything — the read-only twin of seedGlossary, and the answer to backlog row 231.
|
||||||
|
//
|
||||||
|
// What it fixes is a disagreement about money. `translate` re-seeds from the FILES and does it BEFORE
|
||||||
|
// checkRebillConsent, so its consent gate sees a `bank-apply` edit the moment the files change; the
|
||||||
|
// read-only surfaces materialized the STORED glossary instead — last run's fold — and therefore reported
|
||||||
|
// "nothing moved" for exactly the edit the next run would bill for. Reading the files rather than the
|
||||||
|
// store keeps every read-only guarantee intact: this writes nothing at all, so a store opened
|
||||||
|
// query_only(1) is untouched, and `status` remains the purely-reading verb it is ratified as.
|
||||||
|
//
|
||||||
|
// It also sets r.baseMemory, which projectStoredMemory never did — see the note on projectStoredMemory.
|
||||||
|
func (r *Runner) projectFoldedMemory() error {
|
||||||
|
in, err := r.gatherBankInputs()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
rows, voices, pairs, err := storeOrder(in)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
r.materializeBanks(rows, voices, pairs)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// The values of StatusReport.RebillBasis — see its doc comment for what each one promises a reader.
|
||||||
|
// Exported because they are a WIRE vocabulary: the CLI renders them and a consumer branches on them, and
|
||||||
|
// a second hand-typed copy of an enum in the renderer is how the two drift apart.
|
||||||
|
const (
|
||||||
|
RebillBasisPending = "pending"
|
||||||
|
RebillBasisStored = "stored"
|
||||||
|
RebillBasisNone = "none"
|
||||||
|
RebillBasisFailed = "failed"
|
||||||
|
)
|
||||||
|
|
||||||
|
// foldMemoryForRead materializes the bank a READ-ONLY surface must judge against, and reports WHICH bank
|
||||||
|
// it managed to get. It is the one place the fallback lives, so status and export can never end up
|
||||||
|
// judging drift against two different banks.
|
||||||
|
//
|
||||||
|
// The preferred answer is the FOLD of the book's current inputs, because that is the bank the next
|
||||||
|
// `translate` will build (it re-seeds before its consent gate) — projecting anything else is projecting a
|
||||||
|
// run nobody is going to make. The fallback exists because the fold can legitimately refuse: a seed with a
|
||||||
|
// shared-key collision, a mined-delta duplicating a signed term, a voice row naming an absent character.
|
||||||
|
// Those are broken books, and `translate` fails loudly on them — but a READ command must not, or the book
|
||||||
|
// that most needs inspecting becomes the one that cannot be inspected (the D20.4 property rebill.go leans
|
||||||
|
// on: "a book that needs consent stays fully inspectable"). So the read falls back to the glossary the
|
||||||
|
// last run stored, and says so through the basis rather than passing the older number off as the answer.
|
||||||
|
//
|
||||||
|
// The returned error is the fold's refusal, for the caller to LOG in its own idiom; it is never fatal.
|
||||||
|
func (r *Runner) foldMemoryForRead() (basis string, warn error) {
|
||||||
|
foldErr := r.projectFoldedMemory()
|
||||||
|
if foldErr == nil {
|
||||||
|
return RebillBasisPending, nil
|
||||||
|
}
|
||||||
|
if storedErr := r.projectStoredMemory(); storedErr != nil {
|
||||||
|
return RebillBasisFailed, fmt.Errorf("the bank fold refused (%w) and the stored glossary could not be materialized either: %w", foldErr, storedErr)
|
||||||
|
}
|
||||||
|
return RebillBasisStored, foldErr
|
||||||
|
}
|
||||||
|
|
||||||
|
func joinSemi(v []string) string { return joinWith(v, "; ") }
|
||||||
|
func joinLines(v []string) string { return joinWith(v, "\n - ") }
|
||||||
|
|
||||||
|
func joinWith(v []string, sep string) string {
|
||||||
|
out := ""
|
||||||
|
for i, s := range v {
|
||||||
|
if i > 0 {
|
||||||
|
out += sep
|
||||||
|
}
|
||||||
|
out += s
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
414
backend/internal/pipeline/bankmaterialize_identity_test.go
Normal file
414
backend/internal/pipeline/bankmaterialize_identity_test.go
Normal file
|
|
@ -0,0 +1,414 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"log/slog"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/miner"
|
||||||
|
"textmachine/backend/internal/obs"
|
||||||
|
"textmachine/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// identitySeed is deliberately hostile to the fold: every ordered axis is presented in the WRONG order.
|
||||||
|
// The terms descend by src where the store ascends, one term carries two senses whose order is inverted,
|
||||||
|
// two terms share a src and differ only by spoiler window (presented later-window-first), the aliases of
|
||||||
|
// one term are listed in reverse, and the voice/address rows are scrambled against their own ORDER BY.
|
||||||
|
// A fold that "keeps the order it was given" therefore cannot accidentally pass this fixture.
|
||||||
|
const identitySeed = `
|
||||||
|
terms:
|
||||||
|
- src: 鈴木
|
||||||
|
dst: Судзуки
|
||||||
|
type: name
|
||||||
|
status: approved
|
||||||
|
since_ch: 5
|
||||||
|
decl: { invariant: true, forms: ["Судзуки"] }
|
||||||
|
aliases:
|
||||||
|
- { alias: すずき, type: name }
|
||||||
|
- { alias: SUZUKI, type: name }
|
||||||
|
- { alias: Suzuki, type: name }
|
||||||
|
- src: 鈴木
|
||||||
|
dst: Судзуки-старший
|
||||||
|
type: name
|
||||||
|
status: approved
|
||||||
|
since_ch: 1
|
||||||
|
until_ch: 4
|
||||||
|
decl: { invariant: true, forms: ["Судзуки-старший"] }
|
||||||
|
- src: 図書館
|
||||||
|
dst: библиотека
|
||||||
|
type: term
|
||||||
|
status: approved
|
||||||
|
sense: здание
|
||||||
|
since_ch: 3
|
||||||
|
decl: { invariant: false, forms: ["библиотека", "библиотеки"] }
|
||||||
|
- src: 図書館
|
||||||
|
dst: книгохранилище
|
||||||
|
type: term
|
||||||
|
status: draft
|
||||||
|
sense: а-архаизм
|
||||||
|
since_ch: 1
|
||||||
|
until_ch: 2
|
||||||
|
- src: 朝
|
||||||
|
dst: утро
|
||||||
|
type: term
|
||||||
|
status: approved
|
||||||
|
voices:
|
||||||
|
- src: 図書館
|
||||||
|
sense: здание
|
||||||
|
register: нейтральный
|
||||||
|
self_ref: я
|
||||||
|
address_default: formal
|
||||||
|
since_ch: 3
|
||||||
|
- src: 鈴木
|
||||||
|
register: разговорный
|
||||||
|
self_ref: я
|
||||||
|
address_default: informal
|
||||||
|
since_ch: 1
|
||||||
|
until_ch: 4
|
||||||
|
- src: 鈴木
|
||||||
|
register: книжный
|
||||||
|
self_ref: я
|
||||||
|
address_default: formal
|
||||||
|
since_ch: 5
|
||||||
|
addresses:
|
||||||
|
- speaker: 鈴木
|
||||||
|
addressee: 図書館
|
||||||
|
addressee_sense: здание
|
||||||
|
register: formal
|
||||||
|
form: вы
|
||||||
|
closeness: далеко
|
||||||
|
since_ch: 5
|
||||||
|
- speaker: 図書館
|
||||||
|
speaker_sense: здание
|
||||||
|
addressee: 鈴木
|
||||||
|
register: informal
|
||||||
|
form: ты
|
||||||
|
closeness: близко
|
||||||
|
since_ch: 1
|
||||||
|
`
|
||||||
|
|
||||||
|
// identityDelta is the owner's decision file — the state that only exists AFTER a `bank-apply` and that
|
||||||
|
// the stored glossary therefore does not hold. Its rows sort INTO the middle of the seed's, so a fold
|
||||||
|
// that appended them instead of merging them by value would be caught.
|
||||||
|
const identityDelta = `
|
||||||
|
terms:
|
||||||
|
- src: 静か
|
||||||
|
dst: тихий
|
||||||
|
type: term
|
||||||
|
status: approved
|
||||||
|
- src: 元海
|
||||||
|
dst: Изначальное море
|
||||||
|
type: term
|
||||||
|
status: approved
|
||||||
|
`
|
||||||
|
|
||||||
|
// identityAutoBank is the engine's own unsigned proposal file (Source:"mined", base-excluded), included
|
||||||
|
// so the fixture exercises the BASE/ENRICHED split as well as the plain ordering.
|
||||||
|
const identityAutoBank = `
|
||||||
|
terms:
|
||||||
|
- src: 行った
|
||||||
|
dst: пошёл
|
||||||
|
type: term
|
||||||
|
status: auto
|
||||||
|
`
|
||||||
|
|
||||||
|
// bankFoldFixture builds a project whose bank is folded from all four sources at once — curated seed,
|
||||||
|
// ruby readings, the owner's mined-delta and the engine's auto-bank — and returns the book path.
|
||||||
|
func bankFoldFixture(t *testing.T, providerURL string) string {
|
||||||
|
t.Helper()
|
||||||
|
bookPath := setupProjectOpts(t, providerURL, projectOpts{
|
||||||
|
regenerate: 1, source: suzukiSource, glossarySeed: identitySeed,
|
||||||
|
})
|
||||||
|
dir := filepath.Dir(bookPath)
|
||||||
|
writeFile(t, filepath.Join(dir, "test-book.mined-delta.yaml"), identityDelta)
|
||||||
|
writeFile(t, filepath.Join(dir, "test-book.db.auto-bank.yaml"), identityAutoBank)
|
||||||
|
return bookPath
|
||||||
|
}
|
||||||
|
|
||||||
|
// stripIDs zeroes the store's autoincrement id, the ONE field the round-trip adds and the fold never
|
||||||
|
// hashes (membank.ComputeVersion excludes it by construction). Everything else must match exactly.
|
||||||
|
func stripIDs(rows []store.GlossaryEntry) []store.GlossaryEntry {
|
||||||
|
out := append([]store.GlossaryEntry(nil), rows...)
|
||||||
|
for i := range out {
|
||||||
|
out[i].ID = 0
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTheInMemoryFoldIsIdenticalToTheStoreRoundTrip is the proof the free estimate rests on, and it is a
|
||||||
|
// proof by EXECUTION rather than by argument: both folds are run over the same inputs and every quantity
|
||||||
|
// that money depends on is compared.
|
||||||
|
//
|
||||||
|
// What is at stake. The canonical fold goes THROUGH the store — ReplaceBank writes, GlossaryForBook reads
|
||||||
|
// back ORDER BY src, sense, since_ch, until_ch, status, dst — and that order decides the bank's content
|
||||||
|
// hash and, through Select's stable budget sort, the literal bytes of the rendered glossary block, which
|
||||||
|
// are hashed into chunk_status.content_hash. A read-only surface cannot write, so it must reproduce the
|
||||||
|
// order without the store. If it reproduced it only approximately, the free projection would name one
|
||||||
|
// number and the paid run would charge another — which is precisely the defect the projection is sold as
|
||||||
|
// fixing.
|
||||||
|
//
|
||||||
|
// The comparison is made at three depths on purpose: the rows themselves (so a divergence is localised),
|
||||||
|
// the two bank versions (the snapshot component), and the per-position rendered content hashes (the wire
|
||||||
|
// bytes the resume fast-path and the re-bill projection actually compare).
|
||||||
|
func TestTheInMemoryFoldIsIdenticalToTheStoreRoundTrip(t *testing.T) {
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := newJSONProvider(rec, draftEdit)
|
||||||
|
defer srv.Close()
|
||||||
|
bookPath := bankFoldFixture(t, srv.URL)
|
||||||
|
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
|
||||||
|
|
||||||
|
// A RUN MUST HAVE HAPPENED before either fold is measured, and it is load-bearing rather than
|
||||||
|
// scene-setting. renderedContentHashes reproduces an EDIT position only when every member's stored
|
||||||
|
// draft is retrievable (repin.go), so on a book with no chunk_status rows the hash map comes back
|
||||||
|
// draft-only — and the edit wave is exactly the half the mined-delta and auto-bank rows reach, since
|
||||||
|
// Source:"mined" folds into the ENRICHED bank alone. Without the run this comparison would be blind to
|
||||||
|
// every possible defect in the half it was written to prove.
|
||||||
|
r0 := newRunner(t, bookPath)
|
||||||
|
if err := r0.Store.ReplaceRubyReadings("test-book", []store.RubyReading{
|
||||||
|
{BookID: "test-book", Base: "鈴木", Reading: "すずき", FirstChapter: 1, Occurrences: 3},
|
||||||
|
{BookID: "test-book", Base: "図書館", Reading: "としょかん", FirstChapter: 1, Occurrences: 2},
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := r0.TranslateBook(ctx); err != nil {
|
||||||
|
t.Fatalf("the run this comparison rests on did not complete: %v", err)
|
||||||
|
}
|
||||||
|
r0.Close()
|
||||||
|
|
||||||
|
// --- Path A: the canonical fold, through the store ---
|
||||||
|
ra := newRunner(t, bookPath)
|
||||||
|
if err := ra.seedGlossary(ctx); err != nil {
|
||||||
|
t.Fatalf("canonical fold: %v", err)
|
||||||
|
}
|
||||||
|
rowsA, err := ra.Store.GlossaryForBook("test-book")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
voicesA, err := ra.Store.VoiceProfilesForBook("test-book")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
pairsA, err := ra.Store.AddressPairsForBook("test-book")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
verA, baseA := ra.memory.Version(), ra.baseMemory.Version()
|
||||||
|
hashesA := foldRenderedHashes(t, ra)
|
||||||
|
// The project flock is exclusive, so the second runner cannot open until the first lets go.
|
||||||
|
ra.Close()
|
||||||
|
|
||||||
|
// --- Path B: the same fold with the write removed ---
|
||||||
|
rb := newRunner(t, bookPath)
|
||||||
|
defer rb.Close()
|
||||||
|
if err := rb.projectFoldedMemory(); err != nil {
|
||||||
|
t.Fatalf("in-memory fold: %v", err)
|
||||||
|
}
|
||||||
|
in, err := rb.gatherBankInputs()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rowsB, voicesB, pairsB, err := storeOrder(in)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
verB, baseB := rb.memory.Version(), rb.baseMemory.Version()
|
||||||
|
hashesB := foldRenderedHashes(t, rb)
|
||||||
|
|
||||||
|
// The fixture must actually exercise the ordering — a fold that never had to sort anything would
|
||||||
|
// pass this test while proving nothing.
|
||||||
|
if len(rowsA) < 8 {
|
||||||
|
t.Fatalf("fixture too thin to prove an ordering: %d rows", len(rowsA))
|
||||||
|
}
|
||||||
|
if reflect.DeepEqual(stripIDs(rowsA), stripIDs(in.entries)) {
|
||||||
|
t.Fatalf("fixture is degenerate: the BUILD order already equals the store order, so sorting is untested")
|
||||||
|
}
|
||||||
|
|
||||||
|
if !reflect.DeepEqual(stripIDs(rowsA), stripIDs(rowsB)) {
|
||||||
|
for i := range rowsA {
|
||||||
|
if i >= len(rowsB) {
|
||||||
|
t.Fatalf("row %d: store has %+v, memory ran out (%d rows)", i, rowsA[i], len(rowsB))
|
||||||
|
}
|
||||||
|
a, b := rowsA[i], rowsB[i]
|
||||||
|
a.ID, b.ID = 0, 0
|
||||||
|
if !reflect.DeepEqual(a, b) {
|
||||||
|
t.Fatalf("row %d differs:\n store = %+v\n memory = %+v", i, a, b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t.Fatalf("row counts differ: store %d, memory %d", len(rowsA), len(rowsB))
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(voicesA, voicesB) {
|
||||||
|
t.Fatalf("voice profiles differ:\n store = %+v\n memory = %+v", voicesA, voicesB)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(pairsA, pairsB) {
|
||||||
|
t.Fatalf("address pairs differ:\n store = %+v\n memory = %+v", pairsA, pairsB)
|
||||||
|
}
|
||||||
|
if verA != verB {
|
||||||
|
t.Fatalf("enriched bank version differs: store %s, memory %s", verA, verB)
|
||||||
|
}
|
||||||
|
if baseA != baseB {
|
||||||
|
t.Fatalf("base bank version differs: store %s, memory %s", baseA, baseB)
|
||||||
|
}
|
||||||
|
if baseA == verA {
|
||||||
|
t.Fatalf("fixture is degenerate: the auto-bank row must make base ≠ enriched, both are %s", verA)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(hashesA, hashesB) {
|
||||||
|
t.Fatalf("rendered content hashes differ — the free estimate would name a number the paid run does not charge:\n store = %+v\n memory = %+v", hashesA, hashesB)
|
||||||
|
}
|
||||||
|
// ⚠ NOT merely "non-empty". Two empty maps compare equal, and so do two DRAFT-ONLY maps — and the
|
||||||
|
// draft wave is the half the fixture's mined rows do not even reach (Source:"mined" folds into the
|
||||||
|
// ENRICHED bank only). The comparison is only a proof if it actually covers the edit wave, so the
|
||||||
|
// coverage is asserted rather than assumed.
|
||||||
|
waves := map[string]int{}
|
||||||
|
for _, byStage := range hashesA {
|
||||||
|
for stage := range byStage {
|
||||||
|
waves[stage]++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if waves["draft"] == 0 || waves["edit"] == 0 {
|
||||||
|
t.Fatalf("the byte-level comparison must cover BOTH waves — the mined rows this fixture carries reach only the edit one; got %v", waves)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// foldRenderedHashes reproduces the per-position wire hashes exactly as the re-bill projection does
|
||||||
|
// (rebill.go), so the comparison is over the quantity money is decided by rather than over a proxy.
|
||||||
|
func foldRenderedHashes(t *testing.T, r *Runner) map[chunkKey]map[string]string {
|
||||||
|
t.Helper()
|
||||||
|
chunks, withText, err := r.readModelChunks()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read-model chunks: %v", err)
|
||||||
|
}
|
||||||
|
_ = chunks
|
||||||
|
full, err := withText()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("re-chunk with text: %v", err)
|
||||||
|
}
|
||||||
|
return r.renderedContentHashes(full, precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget))
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTheInMemoryFoldRefusesWhatTheStoreWouldRefuse pins the other half of the identity: where the
|
||||||
|
// round-trip would ABORT, the projection must abort too. A fold that quietly de-duplicated a repeated
|
||||||
|
// UNIQUE key would answer with a tidy number for a book whose next run cannot even start.
|
||||||
|
func TestTheInMemoryFoldRefusesWhatTheStoreWouldRefuse(t *testing.T) {
|
||||||
|
dup := store.GlossaryEntry{BookID: "b", Src: "鈴木", Dst: "Судзуки", Status: "approved"}
|
||||||
|
if _, _, _, err := storeOrder(bankInputs{entries: []store.GlossaryEntry{dup, dup}}); err == nil {
|
||||||
|
t.Fatalf("a repeated glossary UNIQUE key must refuse, not de-duplicate")
|
||||||
|
}
|
||||||
|
aliased := store.GlossaryEntry{BookID: "b", Src: "図書館", Dst: "библиотека", Status: "approved",
|
||||||
|
Aliases: []store.GlossaryAlias{{Alias: "としょかん"}, {Alias: "としょかん"}}}
|
||||||
|
if _, _, _, err := storeOrder(bankInputs{entries: []store.GlossaryEntry{aliased}}); err == nil {
|
||||||
|
t.Fatalf("a repeated alias must refuse: glossary_aliases carries UNIQUE (book_id, term_id, alias)")
|
||||||
|
}
|
||||||
|
v := store.VoiceProfile{Src: "鈴木", Register: "разговорный"}
|
||||||
|
if _, _, _, err := storeOrder(bankInputs{voices: []store.VoiceProfile{v, v}}); err == nil {
|
||||||
|
t.Fatalf("a repeated voice-profile UNIQUE key must refuse")
|
||||||
|
}
|
||||||
|
p := store.AddressPair{SpeakerSrc: "鈴木", AddresseeSrc: "図書館"}
|
||||||
|
if _, _, _, err := storeOrder(bankInputs{pairs: []store.AddressPair{p, p}}); err == nil {
|
||||||
|
t.Fatalf("a repeated address-pair UNIQUE key must refuse")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestAFailedFoldStillReportsWhatItSkipped pins a regression this pack introduced and then fixed. Before
|
||||||
|
// the gather was extracted from seedGlossary, its non-fatal warnings were emitted inline as they were
|
||||||
|
// made, so a book that died on a LATER check had still told the operator what the earlier steps skipped.
|
||||||
|
// The first version of the extraction logged the collected remarks only after the error return — losing
|
||||||
|
// the diagnostics of exactly the run that needs them most.
|
||||||
|
func TestAFailedFoldStillReportsWhatItSkipped(t *testing.T) {
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := newJSONProvider(rec, draftEdit)
|
||||||
|
defer srv.Close()
|
||||||
|
// The seed holds a signed term, and a voice row for a character the bank does NOT have — the latter is
|
||||||
|
// fatal, and it is checked AFTER the auto-bank load.
|
||||||
|
bookPath := setupProjectOpts(t, srv.URL, projectOpts{
|
||||||
|
regenerate: 1, source: suzukiSource, glossarySeed: `
|
||||||
|
terms:
|
||||||
|
- src: 鈴木
|
||||||
|
dst: Судзуки
|
||||||
|
type: name
|
||||||
|
status: approved
|
||||||
|
decl: { invariant: true, forms: ["Судзуки"] }
|
||||||
|
voices:
|
||||||
|
- src: 存在しない
|
||||||
|
register: книжный
|
||||||
|
self_ref: я
|
||||||
|
address_default: formal
|
||||||
|
`})
|
||||||
|
dir := filepath.Dir(bookPath)
|
||||||
|
// An auto-bank row on the key the signed seed term already holds: dropped with a REMARK, and the drop
|
||||||
|
// happens BEFORE the fatal voice check.
|
||||||
|
writeFile(t, filepath.Join(dir, "test-book.db.auto-bank.yaml"), `
|
||||||
|
terms:
|
||||||
|
- src: 鈴木
|
||||||
|
dst: Сузуки
|
||||||
|
type: name
|
||||||
|
status: auto
|
||||||
|
`)
|
||||||
|
var logBuf bytes.Buffer
|
||||||
|
r := newRunner(t, bookPath)
|
||||||
|
defer r.Close()
|
||||||
|
r.Log = slog.New(slog.NewTextHandler(&logBuf, &slog.HandlerOptions{Level: slog.LevelWarn}))
|
||||||
|
|
||||||
|
err := r.seedGlossary(context.Background())
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("precondition: a voice row naming an absent character must fail the fold")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "absent from the bank") {
|
||||||
|
t.Fatalf("precondition: expected the voice-character refusal, got: %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(logBuf.String(), "auto-bank rows dropped") {
|
||||||
|
t.Fatalf("the fold died and took its own diagnostics with it; the dropped auto-bank row was never reported.\nlog was:\n%s", logBuf.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTheAutoBankIsWrittenAtomically pins a hole THIS pack opened in someone else's file. Until the
|
||||||
|
// read-only surfaces began folding the bank from the decision documents themselves, the auto-bank was
|
||||||
|
// read only by the run that had just written it, so a plain truncating os.WriteFile was safe. The moment
|
||||||
|
// `status` and `export` started reading it, it became one of the concurrently-read sidecars artifact.go
|
||||||
|
// exists for — and a truncating write hands a concurrent reader either zero bytes (a parse error, so the
|
||||||
|
// buyer sees stale figures) or a valid PREFIX of the term list, which is worse: a fold of a bank that
|
||||||
|
// never existed and a rebill_units no run will ever charge.
|
||||||
|
//
|
||||||
|
// The instrument is the inode, the same one bank-apply's tests use: a rename ALWAYS changes it, and a
|
||||||
|
// truncate-in-place never does. It is a stronger assertion than comparing bytes, because rewriting the
|
||||||
|
// same bytes non-atomically would pass a byte comparison and fail this.
|
||||||
|
func TestTheAutoBankIsWrittenAtomically(t *testing.T) {
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := newJSONProvider(rec, draftEdit)
|
||||||
|
defer srv.Close()
|
||||||
|
bookPath := setupProjectOpts(t, srv.URL, projectOpts{regenerate: 1, source: suzukiSource})
|
||||||
|
r := newRunner(t, bookPath)
|
||||||
|
defer r.Close()
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
first := []miner.Term{{Src: "図書館", Type: "term", SinceCh: 1, Freq: 3}}
|
||||||
|
if err := r.writeAutoBank(ctx, first, nil, nil); err != nil {
|
||||||
|
t.Fatalf("write auto-bank: %v", err)
|
||||||
|
}
|
||||||
|
before := inode(t, r.autoBankPath())
|
||||||
|
|
||||||
|
second := []miner.Term{
|
||||||
|
{Src: "図書館", Type: "term", SinceCh: 1, Freq: 3},
|
||||||
|
{Src: "鈴木", Type: "name", SinceCh: 1, Freq: 5},
|
||||||
|
}
|
||||||
|
if err := r.writeAutoBank(ctx, second, nil, nil); err != nil {
|
||||||
|
t.Fatalf("rewrite auto-bank: %v", err)
|
||||||
|
}
|
||||||
|
if after := inode(t, r.autoBankPath()); after == before {
|
||||||
|
t.Fatalf("the auto-bank was rewritten IN PLACE (inode %d unchanged): a concurrent status/export read can catch it truncated or half-written, and a valid prefix of the term list folds a bank that never existed", before)
|
||||||
|
}
|
||||||
|
// And the document that landed is the whole new one, not a prefix of it.
|
||||||
|
body, err := os.ReadFile(r.autoBankPath())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"図書館", "鈴木"} {
|
||||||
|
if !strings.Contains(string(body), want) {
|
||||||
|
t.Fatalf("the replacement document is incomplete: %q missing from\n%s", want, body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -88,6 +88,12 @@ type BookResult struct {
|
||||||
Chunks []ChunkOutcome
|
Chunks []ChunkOutcome
|
||||||
TotalUSD float64 // THIS run's spend across all chunks
|
TotalUSD float64 // THIS run's spend across all chunks
|
||||||
Flagged int // number of flagged chunks (acceptance allows N)
|
Flagged int // number of flagged chunks (acceptance allows N)
|
||||||
|
// Volume is non-nil when the run stopped because it had done the VOLUME of work it was granted
|
||||||
|
// (--max-units, volume.go) rather than because the book ended. It is a RESULT and never an error: a
|
||||||
|
// volume stop is a completion (exit 0, the frozen exit-code dictionary untouched), so the fact that
|
||||||
|
// this particular completion is not the end of the book has to travel HERE, in the report, or the
|
||||||
|
// operator gets a stop that does not say which ceiling produced it.
|
||||||
|
Volume *VolumeStop
|
||||||
}
|
}
|
||||||
|
|
||||||
// ExitCode is the run's shell disposition: 0 clean, 2 completed-with-flags
|
// ExitCode is the run's shell disposition: 0 clean, 2 completed-with-flags
|
||||||
|
|
@ -184,20 +190,6 @@ func (r *Runner) translateBook(ctx context.Context) (*BookResult, error) {
|
||||||
// manifest, and building it anywhere else would let a call path exist without it.
|
// manifest, and building it anywhere else would let a call path exist without it.
|
||||||
r.bankSrc = newBankSourceIndex(chunks)
|
r.bankSrc = newBankSourceIndex(chunks)
|
||||||
|
|
||||||
// Consent to a RE-PAYMENT (D20.2-Q2, rebill.go) — BEFORE the waves, hence before the first Reserve:
|
|
||||||
// if this run would pay again for units already billed under a superseded snapshot, and the amount
|
|
||||||
// is over the book's threshold, it stops here with the sum instead of quietly re-buying the book.
|
|
||||||
// It needs the materialized memory (the per-wave snapshots fold it), so it sits after seedGlossary,
|
|
||||||
// and the chunk manifest (the projected book cost is per output unit), so it sits after the split.
|
|
||||||
if err := r.checkRebillConsent(ctx, chunks); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
// The run scale — one line to stderr (the smoke-run pain: N/M and the progress
|
|
||||||
// denominator never appeared in the logs at all, only in stdout/status).
|
|
||||||
r.Log.InfoContext(ctx, "book run started", "book", r.Book.BookID,
|
|
||||||
"chapters", len(doc.Chapters), "chunks", len(chunks), "stages", len(r.Pipeline.Stages))
|
|
||||||
|
|
||||||
// the precompute pass: pre-compute the sticky-chain memory selection for EVERY chunk (WS1 §1б, precomputeSticky). The
|
// the precompute pass: pre-compute the sticky-chain memory selection for EVERY chunk (WS1 §1б, precomputeSticky). The
|
||||||
// sticky window (A5 scene-inertia — the recent chunks' exact matches in the current chapter, reset at
|
// sticky window (A5 scene-inertia — the recent chunks' exact matches in the current chapter, reset at
|
||||||
// a chapter boundary) is a CROSS-CHUNK sequential dependency, so it is computed once here in the precompute pass and
|
// a chapter boundary) is a CROSS-CHUNK sequential dependency, so it is computed once here in the precompute pass and
|
||||||
|
|
@ -207,11 +199,51 @@ func (r *Runner) translateBook(ctx context.Context) (*BookResult, error) {
|
||||||
// a bank-mining enrichment (the review-confirmed «re-paid ONCE» fix); the editor uses the enriched bank.
|
// a bank-mining enrichment (the review-confirmed «re-paid ONCE» fix); the editor uses the enriched bank.
|
||||||
stickySel := precomputeSticky(chunks, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget)
|
stickySel := precomputeSticky(chunks, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget)
|
||||||
|
|
||||||
|
// The VOLUME ceiling's scope (volume.go): decided before either wave, so the ceiling is checked before
|
||||||
|
// an item begins rather than after it has been paid for. It needs stickySel, because the "would this
|
||||||
|
// unit resolve for free" predicate renders the same wire bytes the draft wave will.
|
||||||
|
//
|
||||||
|
// ⚠ IT IS DELIBERATELY COMPUTED BEFORE THE CONSENT GATE BELOW, not after. The gate asks the operator to
|
||||||
|
// consent to a CONCRETE spend (Р6), and a run bounded to ten units is not going to re-pay the whole
|
||||||
|
// book — quoting the book's figure at it would refuse work that was never going to be done, and ask for
|
||||||
|
// consent to money nobody will be charged. rebill.go says out loud why that is corrosive: "a number the
|
||||||
|
// operator is asked to approve and then not charged is exactly what makes such a number stop being
|
||||||
|
// read." Both steps are $0 and pure, so the reordering costs nothing.
|
||||||
|
scope, err := r.planVolume(ctx, chunks, stickySel)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Consent to a RE-PAYMENT (D20.2-Q2, rebill.go) — BEFORE the waves, hence before the first Reserve:
|
||||||
|
// if this run would pay again for units already billed under a superseded snapshot, and the amount
|
||||||
|
// is over the book's threshold, it stops here with the sum instead of quietly re-buying the book.
|
||||||
|
// It needs the materialized memory (the per-wave snapshots fold it), so it sits after seedGlossary,
|
||||||
|
// and the chunk manifest (the projected book cost is per output unit), so it sits after the split.
|
||||||
|
if err := r.checkRebillConsent(ctx, chunks, scope); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// The run scale — one line to stderr (the smoke-run pain: N/M and the progress
|
||||||
|
// denominator never appeared in the logs at all, only in stdout/status).
|
||||||
|
r.Log.InfoContext(ctx, "book run started", "book", r.Book.BookID,
|
||||||
|
"chapters", len(doc.Chapters), "chunks", len(chunks), "stages", len(r.Pipeline.Stages))
|
||||||
|
|
||||||
// The wave executor (R1, waverun.go): the draft wave (draft ∥) → the bank-mining stop → the edit wave (edit ∥). It computes +
|
// The wave executor (R1, waverun.go): the draft wave (draft ∥) → the bank-mining stop → the edit wave (edit ∥). It computes +
|
||||||
// upserts the per-wave snapshots (draft-wave snapshot base-bank / edit-wave snapshot enriched) itself and pins each
|
// upserts the per-wave snapshots (draft-wave snapshot base-bank / edit-wave snapshot enriched) itself and pins each
|
||||||
// wave's jobs to its own — a the bank-mining stop enrichment moves only edit-wave snapshot, keeping draft-wave checkpoints valid
|
// wave's jobs to its own — a the bank-mining stop enrichment moves only edit-wave snapshot, keeping draft-wave checkpoints valid
|
||||||
// («re-paid ONCE»). Returns a *WaveSignatureStop when the bank-mining stop stops for owner sign.
|
// («re-paid ONCE»). Returns a *WaveSignatureStop when the bank-mining stop stops for owner sign.
|
||||||
res, err := r.translateBookWaves(ctx, chunks, stickySel)
|
res, err := r.translateBookWaves(ctx, chunks, stickySel, scope)
|
||||||
|
if err != nil && scope.bound() {
|
||||||
|
// A run can be stopped by the OTHER ceiling — money — and then the volume report never reaches the
|
||||||
|
// operator, because a failed run returns no result to carry it. Without this line they are told the
|
||||||
|
// run halted on money and nothing about what it had been granted or how far that got, which is the
|
||||||
|
// first thing anyone asks. The durable progress is in the store either way; this makes the stderr
|
||||||
|
// account of the run complete rather than half.
|
||||||
|
r.Log.WarnContext(ctx, "the run ended before its VOLUME grant was used up — the stop below is NOT the volume ceiling",
|
||||||
|
"book", r.Book.BookID, "max_units", scope.stop.MaxUnits,
|
||||||
|
"granted_new", scope.stop.Delivered, "granted_re_made", scope.stop.Reworked,
|
||||||
|
"left_never_delivered", scope.stop.LeftFresh, "err", err)
|
||||||
|
}
|
||||||
if err == nil {
|
if err == nil {
|
||||||
// End of the run (row 125). The stop paths refresh the read-out themselves, so this is the boundary
|
// End of the run (row 125). The stop paths refresh the read-out themselves, so this is the boundary
|
||||||
// they do not cover: a run that finished, whose last bank change was the auto-continue re-seed or a
|
// they do not cover: a run that finished, whose last bank change was the auto-continue re-seed or a
|
||||||
|
|
|
||||||
|
|
@ -309,6 +309,21 @@ func (r *Runner) exportConfigDrift(statuses []store.ChunkStatus, draftStageNames
|
||||||
editSnaps[cs.SnapshotID] = true
|
editSnaps[cs.SnapshotID] = true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// ⚠ THE STORED GLOSSARY ON PURPOSE, and NOT the fold `status` uses. The two surfaces answer two
|
||||||
|
// different questions and this pack deliberately moved only one of them.
|
||||||
|
//
|
||||||
|
// status asks "what would the NEXT run cost", so it must fold the decision files: money is the point,
|
||||||
|
// and a bank-apply that has not been run yet is exactly the spend it exists to reveal (row 231).
|
||||||
|
// export asks whether THIS document is consistent with the run that produced it — its consumer is the
|
||||||
|
// polygon's extraction, which uses ConfigDrift to decide whether an export is trustworthy to MEASURE.
|
||||||
|
// An export whose text is byte-identical to what its run shipped is perfectly measurable even with a
|
||||||
|
// decision file sitting un-run beside the database, and folding that file here would mark every such
|
||||||
|
// export drifted and quietly disqualify sound measurements in another zone.
|
||||||
|
//
|
||||||
|
// This was briefly changed to the fold for symmetry and changed back: symmetry between two surfaces
|
||||||
|
// that answer different questions is not a property worth having, and redefining a field another zone
|
||||||
|
// consumes was not this pack's order. The divergence is deliberate, and it is named here so the next
|
||||||
|
// reader does not "fix" it either.
|
||||||
if err := r.projectStoredMemory(); err != nil {
|
if err := r.projectStoredMemory(); err != nil {
|
||||||
r.Log.Warn("export: config-drift check failed; drift state unknown (reported as none)", "err", err)
|
r.Log.Warn("export: config-drift check failed; drift state unknown (reported as none)", "err", err)
|
||||||
return
|
return
|
||||||
|
|
|
||||||
|
|
@ -712,7 +712,14 @@ func (r *Runner) writeAutoBank(ctx context.Context, mined []miner.Term, proposal
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("pipeline: marshal auto-bank: %w", err)
|
return fmt.Errorf("pipeline: marshal auto-bank: %w", err)
|
||||||
}
|
}
|
||||||
if err := os.WriteFile(r.autoBankPath(), []byte(body), 0o644); err != nil {
|
// ATOMIC, and the reason changed under this file's feet. Until the read-only surfaces began folding the
|
||||||
|
// bank themselves (bankmaterialize.go), the auto-bank was read only by the run that had just written
|
||||||
|
// it, so a plain truncating write was safe. Now `status` and `export` read it CONCURRENTLY with a
|
||||||
|
// running translate — the exact condition artifact.go names — and a truncating write gives a reader
|
||||||
|
// either a parse error (the buyer sees stale figures) or, worse, a valid PREFIX of the term list: a
|
||||||
|
// fold of a bank that never existed, and a rebill_units no run will ever charge. Write-then-rename
|
||||||
|
// makes every read see the previous document or the next one, whole.
|
||||||
|
if err := writeFileAtomic(r.autoBankPath(), []byte(body)); err != nil {
|
||||||
return fmt.Errorf("pipeline: write auto-bank %s: %w", r.autoBankPath(), err)
|
return fmt.Errorf("pipeline: write auto-bank %s: %w", r.autoBankPath(), err)
|
||||||
}
|
}
|
||||||
if len(before) == 0 {
|
if len(before) == 0 {
|
||||||
|
|
|
||||||
|
|
@ -63,6 +63,18 @@ type RebillProjection struct {
|
||||||
// the operator-facing text can name what the number is made of rather than claim a re-pricing it did
|
// the operator-facing text can name what the number is made of rather than claim a re-pricing it did
|
||||||
// not achieve.
|
// not achieve.
|
||||||
HistoricalRows int
|
HistoricalRows int
|
||||||
|
// OutputUnits is the same re-payment counted in OUTPUT UNITS — the granularity the manifest publishes
|
||||||
|
// as units_total, that `--max-units` bounds, and that the platform sells chapters in.
|
||||||
|
//
|
||||||
|
// ⚠ IT EXISTS BECAUSE `Rows` IS A DIFFERENT UNIT WEARING THE SAME WORD. Rows counts chunk×stage, the
|
||||||
|
// BILLING unit; every other "units" number the engine puts on the wire (total_units, progress totals,
|
||||||
|
// chapters[].units_total) is the OUTPUT unit. Measured on a three-chapter fixture, one document carried
|
||||||
|
// `rebill_units: 15` two lines under `total_units: 6` with nothing marking the difference — and the
|
||||||
|
// ratio is not a constant a consumer could divide out: it is len(Members)·nDraftStages + nEditStages,
|
||||||
|
// which varies unit by unit WITHIN one book and whose factors never cross the seam at all. A platform
|
||||||
|
// sizing a re-pass from the estimate and passing that number to --max-units would buy several times the
|
||||||
|
// book it meant to, which is this pack's own defect reproduced one layer down.
|
||||||
|
OutputUnits int
|
||||||
// Repinned counts the units the run will serve for $0 despite a moved snapshot (pack-20 point 5): the
|
// Repinned counts the units the run will serve for $0 despite a moved snapshot (pack-20 point 5): the
|
||||||
// move was bank-only and their rendered bytes are unchanged. It is not part of the amount — it is the
|
// move was bank-only and their rendered bytes are unchanged. It is not part of the amount — it is the
|
||||||
// number that makes the amount believable, because before pack-20 every one of these was counted as a
|
// number that makes the amount believable, because before pack-20 every one of these was counted as a
|
||||||
|
|
@ -116,6 +128,17 @@ func (r *Runner) projectRebill(statuses []store.ChunkStatus, manifest []chunk.Ch
|
||||||
for _, ch := range manifest {
|
for _, ch := range manifest {
|
||||||
live[chunkKey{ch.Chapter, ch.ChunkIdx}] = true
|
live[chunkKey{ch.Chapter, ch.ChunkIdx}] = true
|
||||||
}
|
}
|
||||||
|
// Which OUTPUT unit each position belongs to, so the same re-payment can be reported in the unit the
|
||||||
|
// seam sells in as well as the one it bills in. An edit row lives at its unit's leader chunk and a
|
||||||
|
// leader is one of the unit's members, so this one map covers both waves.
|
||||||
|
leaderOf := make(map[chunkKey]chunkKey, len(manifest))
|
||||||
|
for _, u := range r.outputUnits(manifest) {
|
||||||
|
leader := chunkKey{u.Chapter, u.FirstChunkIdx}
|
||||||
|
for _, m := range u.Members {
|
||||||
|
leaderOf[chunkKey{m.Chapter, m.ChunkIdx}] = leader
|
||||||
|
}
|
||||||
|
}
|
||||||
|
touched := map[chunkKey]bool{}
|
||||||
draftNames := stageNameSet(r.waveStagesIndexed(waveDraft))
|
draftNames := stageNameSet(r.waveStagesIndexed(waveDraft))
|
||||||
editNames := stageNameSet(r.waveStagesIndexed(waveEdit))
|
editNames := stageNameSet(r.waveStagesIndexed(waveEdit))
|
||||||
rendered := map[wave]string{}
|
rendered := map[wave]string{}
|
||||||
|
|
@ -164,7 +187,7 @@ func (r *Runner) projectRebill(statuses []store.ChunkStatus, manifest []chunk.Ch
|
||||||
if ferr != nil {
|
if ferr != nil {
|
||||||
return p, fmt.Errorf("pipeline: re-chunk the source for the re-bill content check: %w", ferr)
|
return p, fmt.Errorf("pipeline: re-chunk the source for the re-bill content check: %w", ferr)
|
||||||
}
|
}
|
||||||
contentHashes = r.renderedContentHashes(full, precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget))
|
contentHashes = r.cachedRenderedContentHashes(full, precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget))
|
||||||
}
|
}
|
||||||
if h, ok := contentHashes[chunkKey{cs.Chapter, cs.ChunkIdx}][cs.Stage]; ok && h == cs.ContentHash {
|
if h, ok := contentHashes[chunkKey{cs.Chapter, cs.ChunkIdx}][cs.Stage]; ok && h == cs.ContentHash {
|
||||||
p.Repinned++
|
p.Repinned++
|
||||||
|
|
@ -172,12 +195,14 @@ func (r *Runner) projectRebill(statuses []store.ChunkStatus, manifest []chunk.Ch
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
p.Rows++
|
p.Rows++
|
||||||
|
touched[leaderOf[chunkKey{cs.Chapter, cs.ChunkIdx}]] = true
|
||||||
usd, fromHistory := rp.usd(cs)
|
usd, fromHistory := rp.usd(cs)
|
||||||
p.USD += usd
|
p.USD += usd
|
||||||
if fromHistory {
|
if fromHistory {
|
||||||
p.HistoricalRows++
|
p.HistoricalRows++
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
p.OutputUnits = len(touched)
|
||||||
return p, nil
|
return p, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -259,7 +284,28 @@ func projectionBasis(p RebillProjection) string {
|
||||||
// the debt this closes. It also covers the case --resnapshot does not: a run interrupted midway through
|
// the debt this closes. It also covers the case --resnapshot does not: a run interrupted midway through
|
||||||
// a re-pin leaves jobs on the new snapshot while their chunk_status rows still carry the old one, and
|
// a re-pin leaves jobs on the new snapshot while their chunk_status rows still carry the old one, and
|
||||||
// the next plain `translate` then re-bills them with no gate at all.
|
// the next plain `translate` then re-bills them with no gate at all.
|
||||||
func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk) error {
|
// `scope` is the run's VOLUME ceiling (volume.go), nil when none is in force. It produces TWO figures,
|
||||||
|
// and which of them each half of the gate is judged against is the whole correctness of this function.
|
||||||
|
//
|
||||||
|
// ⚠ THE THRESHOLD IS JUDGED AGAINST THE BOOK, NEVER AGAINST THE RUN — and the first version of this
|
||||||
|
// pack got that wrong. Scoping the amount to the admitted units and leaving the threshold book-wide
|
||||||
|
// sounds symmetrical and is not: with a volume ceiling the caller CHOOSES how small each run is, so
|
||||||
|
// `--resnapshot --max-units N` in a loop re-pays the entire book without a single run ever crossing the
|
||||||
|
// threshold and without consent being asked once. That is precisely the silent re-purchase Р6 exists to
|
||||||
|
// prevent, and "the change is inert without the flag" is true and beside the point: with the flag it
|
||||||
|
// opens the door the gate is the door for. A threshold a caller can defeat by splitting is not a
|
||||||
|
// threshold. So the question "must we ask at all" is answered by the book's whole drift, which does not
|
||||||
|
// shrink when a purchase does.
|
||||||
|
//
|
||||||
|
// ⚠ THE NAMED CAP IS JUDGED AGAINST THE RUN, because it is a different kind of statement. A threshold is
|
||||||
|
// the book's policy on when a human must be consulted; `--accept-rebill=X` is the caller's instruction
|
||||||
|
// "spend no more than X". Measuring an instruction about SPEND against work this run will not do would
|
||||||
|
// refuse a caller whose cap fully covers what they are about to be charged — the platform funds that cap
|
||||||
|
// from the run's own hold, so it is sized to the purchase, not to the book.
|
||||||
|
//
|
||||||
|
// With no ceiling in force the two projections are the same object and exactly one is computed, so every
|
||||||
|
// existing run is judged by byte-identical figures.
|
||||||
|
func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk, scope *volumeScope) error {
|
||||||
statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID)
|
statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("pipeline: read chunk_status for the re-bill projection: %w", err)
|
return fmt.Errorf("pipeline: read chunk_status for the re-bill projection: %w", err)
|
||||||
|
|
@ -270,14 +316,32 @@ func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk) e
|
||||||
}
|
}
|
||||||
// A write path already holds the real split, so the lazy provider just hands it back — no second
|
// A write path already holds the real split, so the lazy provider just hands it back — no second
|
||||||
// ingest, and no branch where the consent gate could see text-free chunks.
|
// ingest, and no branch where the consent gate could see text-free chunks.
|
||||||
proj, err := r.projectRebill(statuses, chunks, func() ([]chunk.Chunk, error) { return chunks, nil }, rp)
|
withText := func() ([]chunk.Chunk, error) { return chunks, nil }
|
||||||
|
book, err := r.projectRebill(statuses, chunks, withText, rp)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if proj.Rows == 0 {
|
if book.Rows == 0 {
|
||||||
return nil // nothing already-paid is superseded — this run bills only new work
|
return nil // nothing already-paid is superseded — this run bills only new work
|
||||||
}
|
}
|
||||||
|
// What THIS run will actually re-pay. Only computed when a ceiling narrows it; otherwise it IS the
|
||||||
|
// book's figure, which keeps the un-bounded path at one projection exactly as before.
|
||||||
|
proj := book
|
||||||
|
if scope != nil {
|
||||||
|
if proj, err = r.projectRebill(scope.admittedStatuses(statuses), chunks, withText, rp); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if proj.Rows == 0 {
|
||||||
|
// The book carries drift, but none of it is inside what this run was granted, so this run re-pays
|
||||||
|
// nothing and there is no concrete spend to consent to. Not a hole in the threshold above: the
|
||||||
|
// moment a run is granted a superseded unit, proj stops being empty and the book-wide threshold
|
||||||
|
// below decides — so splitting cannot walk past the gate, it can only postpone meeting it.
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// The denominator is the WHOLE book's stored rows, unfiltered by the volume scope: the threshold is
|
||||||
|
// "5% of what this book costs", a property of the book.
|
||||||
byChunk := map[chunkKey][]store.ChunkStatus{}
|
byChunk := map[chunkKey][]store.ChunkStatus{}
|
||||||
for _, cs := range statuses {
|
for _, cs := range statuses {
|
||||||
byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}] = append(byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}], cs)
|
byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}] = append(byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}], cs)
|
||||||
|
|
@ -296,16 +360,22 @@ func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk) e
|
||||||
if r.AcceptRebill.Given {
|
if r.AcceptRebill.Given {
|
||||||
r.Log.WarnContext(ctx, "accepting a projected re-payment of already-billed work (--accept-rebill)",
|
r.Log.WarnContext(ctx, "accepting a projected re-payment of already-billed work (--accept-rebill)",
|
||||||
"rebill_units", proj.Rows, "rebill_usd", fmt.Sprintf("%.6f", proj.USD),
|
"rebill_units", proj.Rows, "rebill_usd", fmt.Sprintf("%.6f", proj.USD),
|
||||||
|
"book_rebill_units", book.Rows, "book_rebill_usd", fmt.Sprintf("%.6f", book.USD),
|
||||||
"units_not_repriced", proj.HistoricalRows,
|
"units_not_repriced", proj.HistoricalRows,
|
||||||
"repinned_free", proj.Repinned, "threshold_usd", fmt.Sprintf("%.6f", threshold))
|
"repinned_free", proj.Repinned, "threshold_usd", fmt.Sprintf("%.6f", threshold))
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
if proj.USD <= threshold {
|
// ⚠ THE BOOK'S figure is what the threshold judges, not this run's. See the doc comment: a caller
|
||||||
|
// that chooses how small each run is could otherwise re-pay the whole book a slice at a time and
|
||||||
|
// never once be asked. `book` equals `proj` whenever no volume ceiling is in force, so this is the
|
||||||
|
// same comparison it has always been for every un-bounded run.
|
||||||
|
if book.USD <= threshold {
|
||||||
// Under the threshold the run continues without friction (the ratified behaviour: a term append
|
// Under the threshold the run continues without friction (the ratified behaviour: a term append
|
||||||
// touching three chunks costs cents). Continuing is not the same as being silent — the amount is
|
// touching three chunks costs cents). Continuing is not the same as being silent — the amount is
|
||||||
// money and it goes to the log.
|
// money and it goes to the log.
|
||||||
r.Log.InfoContext(ctx, "re-paying already-billed work under the consent threshold; continuing without asking",
|
r.Log.InfoContext(ctx, "re-paying already-billed work under the consent threshold; continuing without asking",
|
||||||
"rebill_units", proj.Rows, "rebill_usd", fmt.Sprintf("%.6f", proj.USD),
|
"rebill_units", proj.Rows, "rebill_usd", fmt.Sprintf("%.6f", proj.USD),
|
||||||
|
"book_rebill_units", book.Rows, "book_rebill_usd", fmt.Sprintf("%.6f", book.USD),
|
||||||
"units_not_repriced", proj.HistoricalRows,
|
"units_not_repriced", proj.HistoricalRows,
|
||||||
"repinned_free", proj.Repinned, "threshold_usd", fmt.Sprintf("%.6f", threshold))
|
"repinned_free", proj.Repinned, "threshold_usd", fmt.Sprintf("%.6f", threshold))
|
||||||
return nil
|
return nil
|
||||||
|
|
@ -315,10 +385,30 @@ func (r *Runner) checkRebillConsent(ctx context.Context, chunks []chunk.Chunk) e
|
||||||
if !r.Resnapshot {
|
if !r.Resnapshot {
|
||||||
hint = " The run also needs --resnapshot: without it the superseded jobs stop it anyway."
|
hint = " The run also needs --resnapshot: without it the superseded jobs stop it anyway."
|
||||||
}
|
}
|
||||||
|
// ⚠ EVERY FIGURE IN A SENTENCE MUST COME FROM THE SAME PROJECTION AS THE SENTENCE. This parenthetical
|
||||||
|
// hangs off the BOOK-wide clause, so it takes the BOOK's re-pin count. The first version spliced the
|
||||||
|
// volume-scoped one in, which under an active ceiling printed a smaller number inside a sentence whose
|
||||||
|
// every other figure was book-wide — an operator adding them up would have got a book that did not
|
||||||
|
// exist. Two independent review lenses caught the same splice, which is what mixed-provenance numbers
|
||||||
|
// in one sentence reliably produce.
|
||||||
repin := ""
|
repin := ""
|
||||||
if proj.Repinned > 0 {
|
if book.Repinned > 0 {
|
||||||
repin = fmt.Sprintf(" (%d further unit(s) are re-pinned for $0 — the bank moved but their injected bytes did not)", proj.Repinned)
|
repin = fmt.Sprintf(" (%d further unit(s) of the book are re-pinned for $0 — the bank moved but their injected bytes did not)", book.Repinned)
|
||||||
}
|
}
|
||||||
return fmt.Errorf("pipeline: this run would RE-PAY for work already billed: %d chunk×stage unit(s) are resolved under a superseded snapshot and would be paid for again, ~$%.6f (%s)%s. That is over this book's consent threshold $%.6f (%s), and Р6 requires consent to a CONCRETE spend, not a blanket one (D20.2-Q2). NOTHING was reserved and no row was touched. Re-run with --accept-rebill to accept the whole projected amount, or --accept-rebill=<usd> to accept it only up to a ceiling (a ceiling below the projection refuses).%s",
|
// The two projections are printed SEPARATELY when a volume ceiling makes them differ, because
|
||||||
proj.Rows, proj.USD, projectionBasis(proj), repin, threshold, source, hint)
|
// collapsing them is how this gate goes wrong in either direction: quoting only the book's total at a
|
||||||
|
// small purchase asks for consent to money nobody will be charged, and quoting only the run's slice
|
||||||
|
// hides that the book is being re-bought a slice at a time. The scoped clause carries the scoped
|
||||||
|
// re-pin count for the same provenance reason.
|
||||||
|
scoped := ""
|
||||||
|
if book.Rows != proj.Rows || book.USD != proj.USD {
|
||||||
|
scopedRepin := ""
|
||||||
|
if proj.Repinned != book.Repinned {
|
||||||
|
scopedRepin = fmt.Sprintf(" (and re-pin %d of them for $0)", proj.Repinned)
|
||||||
|
}
|
||||||
|
scoped = fmt.Sprintf(" THIS run, bounded by --max-units, would re-pay %d of them, ~$%.6f%s — but the threshold is judged against the book, because a ceiling the caller sizes could otherwise re-pay the whole book a slice at a time without ever being asked.",
|
||||||
|
proj.Rows, proj.USD, scopedRepin)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("pipeline: this run would RE-PAY for work already billed: %d chunk×stage unit(s) of this book are resolved under a superseded snapshot and would be paid for again, ~$%.6f (%s)%s.%s That is over this book's consent threshold $%.6f (%s), and Р6 requires consent to a CONCRETE spend, not a blanket one (D20.2-Q2). NOTHING was reserved and no row was touched. Re-run with --accept-rebill to accept the whole projected amount, or --accept-rebill=<usd> to accept it only up to a ceiling (the ceiling is measured against what THIS run re-pays, and one below that refuses).%s",
|
||||||
|
book.Rows, book.USD, projectionBasis(book), repin, scoped, threshold, source, hint)
|
||||||
}
|
}
|
||||||
|
|
|
||||||
526
backend/internal/pipeline/rebill_pending_test.go
Normal file
526
backend/internal/pipeline/rebill_pending_test.go
Normal file
|
|
@ -0,0 +1,526 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/chunk/chunktest"
|
||||||
|
"textmachine/backend/internal/membank"
|
||||||
|
"textmachine/backend/internal/obs"
|
||||||
|
)
|
||||||
|
|
||||||
|
// decisionsDocFor writes a decision document for an ARBITRARY book id — the package's existing helper
|
||||||
|
// hardcodes the one book its own fixtures use.
|
||||||
|
func decisionsDocFor(t *testing.T, dir, bookID string, ds ...membank.Decision) string {
|
||||||
|
t.Helper()
|
||||||
|
body, err := json.Marshal(membank.DecisionsDoc{Version: membank.DecisionsVersion, BookID: bookID, Decisions: ds})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
p := filepath.Join(dir, "decisions-for-"+bookID+".json")
|
||||||
|
writeFile(t, p, string(body))
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertStoreUntouched is the "nothing was written" instrument, and it is deliberately TWO assertions
|
||||||
|
// because one of them is not enough.
|
||||||
|
//
|
||||||
|
// ⚠ HASHING THE .db FILE ALONE PROVES LESS THAN IT LOOKS. SQLite in WAL mode puts a committed change in
|
||||||
|
// the `-wal` sidecar and leaves the main file untouched until a checkpoint, so a data write can land with
|
||||||
|
// the main file's bytes unmoved — the first version of this helper would have gone on reporting "nothing
|
||||||
|
// was written" straight through it. So: the data file must be byte-identical AND the write-ahead log must
|
||||||
|
// carry no frames.
|
||||||
|
//
|
||||||
|
// ⚠ AND `-shm` IS DELIBERATELY NOT CHECKED. Measured, not assumed: opening this database read-only
|
||||||
|
// CREATES an empty `-wal` and a live `-shm`, because the shared-memory index is how any WAL reader finds
|
||||||
|
// its snapshot. Requiring those two files not to appear would fail on a pure read and would be asserting
|
||||||
|
// something SQLite's design forbids, not something about this engine. An empty `-wal` is the honest line:
|
||||||
|
// the reader may set the machinery up, it may not commit a frame through it.
|
||||||
|
//
|
||||||
|
// It is corroboration, not the guarantee. What actually makes the read path unable to write is
|
||||||
|
// store.OpenReadOnly's PRAGMA query_only(1) — a write through it fails loudly rather than being dropped.
|
||||||
|
// This catches a path that reached a WRITE handle by some route the read was not supposed to take.
|
||||||
|
func assertStoreUntouched(t *testing.T, dbPath, dataBefore string) {
|
||||||
|
t.Helper()
|
||||||
|
if got := fileDigest(t, dbPath); got != dataBefore {
|
||||||
|
t.Fatalf("the project database changed during a read-only estimate (%s → %s): nothing beyond the first touch may be written", dataBefore[:12], got[:12])
|
||||||
|
}
|
||||||
|
wal, err := os.Stat(dbPath + "-wal")
|
||||||
|
if err != nil {
|
||||||
|
if !os.IsNotExist(err) {
|
||||||
|
t.Fatalf("stat %s-wal: %v", dbPath, err)
|
||||||
|
}
|
||||||
|
return // no write-ahead log at all — nothing could have been committed through one
|
||||||
|
}
|
||||||
|
if wal.Size() != 0 {
|
||||||
|
t.Fatalf("the write-ahead log holds %d byte(s) after a read-only estimate: a committed change can sit in -wal with the main database file still byte-identical, which is exactly what makes hashing the .db alone insufficient", wal.Size())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// fileDigest is a plain content hash of one file.
|
||||||
|
func fileDigest(t *testing.T, path string) string {
|
||||||
|
t.Helper()
|
||||||
|
b, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read %s: %v", path, err)
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256(b)
|
||||||
|
return hex.EncodeToString(sum[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestABankEditIsVisibleToTheEstimateBeforeAnythingIsBought is the deliverable of backlog row 231, and it
|
||||||
|
// is written against the state the FIRST attempt at this repro got wrong (errata 28.08-и/-к).
|
||||||
|
//
|
||||||
|
// THE DEGENERATE REPRO THAT MUST BE AVOIDED. "translate has never run" proves nothing: such a book has no
|
||||||
|
// chunk_status rows at all, and projectRebill skips rows without a snapshot (rebill.go), so it answers
|
||||||
|
// EMPTY both before and after any fix. The state that discriminates is: a run HAPPENED (rows and
|
||||||
|
// snapshots exist), THEN the bank was edited through `bank-apply`, which writes decision FILES and not one
|
||||||
|
// store row. On that state the engine used to answer "nothing moved" while the very next `translate` would
|
||||||
|
// re-seed from those files and bill for the difference.
|
||||||
|
//
|
||||||
|
// So the assertion is a DELTA, and both halves are executed here: the stored-glossary projection (what the
|
||||||
|
// engine answered before) must be zero on exactly the state where the folded projection is non-zero.
|
||||||
|
// Around it sit the two boundaries the estimate is sold with — not one provider call, and not one byte
|
||||||
|
// written to the store beyond the first touch that predates it.
|
||||||
|
func TestABankEditIsVisibleToTheEstimateBeforeAnythingIsBought(t *testing.T) {
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := newJSONProvider(rec, draftEdit)
|
||||||
|
defer srv.Close()
|
||||||
|
bookPath := setupProjectOpts(t, srv.URL, projectOpts{
|
||||||
|
regenerate: 1, source: suzukiSource, glossarySeed: suzukiSeed,
|
||||||
|
})
|
||||||
|
dir := filepath.Dir(bookPath)
|
||||||
|
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
|
||||||
|
|
||||||
|
// --- the run that must have HAPPENED for the repro to discriminate ---
|
||||||
|
r1 := newRunner(t, bookPath)
|
||||||
|
if _, err := r1.TranslateBook(ctx); err != nil {
|
||||||
|
t.Fatalf("the run this repro rests on did not complete: %v", err)
|
||||||
|
}
|
||||||
|
statuses, err := r1.Store.ChunkStatusesForBook("test-book")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(statuses) == 0 {
|
||||||
|
t.Fatalf("precondition: the repro needs stored chunk_status rows, the state the degenerate version lacked")
|
||||||
|
}
|
||||||
|
committedBefore, reservedBefore, err := r1.Store.SpentUSD("test-book")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
r1.Close()
|
||||||
|
|
||||||
|
// --- the bank edit, through the verb the product actually uses ---
|
||||||
|
// 静か occurs in the source, so the term enters the ENRICHED bank, moves the edit-wave snapshot AND
|
||||||
|
// changes the injected bytes of the edit unit — i.e. it is genuinely re-paid work, not a $0 re-pin.
|
||||||
|
doc := decisionsDocFor(t, dir, "test-book", membank.Decision{
|
||||||
|
Action: membank.ActionApprove, Src: "静か", Dst: "тихий"})
|
||||||
|
if _, err := ApplyBankDecisions(ctx, bookPath, doc, false); err != nil {
|
||||||
|
t.Fatalf("bank-apply: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
callsBefore := rec.count()
|
||||||
|
dbPath := filepath.Join(dir, "test-book.db")
|
||||||
|
dbBefore := fileDigest(t, dbPath)
|
||||||
|
|
||||||
|
// --- the two projections over the SAME state ---
|
||||||
|
r2, err := NewReadOnlyRunner(bookPath, obs.NewLogger())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer r2.Close()
|
||||||
|
rp, err := r2.newRepricer()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
chunks, withText, err := r2.readModelChunks()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// What the engine answered BEFORE this pack: the glossary the last run stored.
|
||||||
|
if err := r2.projectStoredMemory(); err != nil {
|
||||||
|
t.Fatalf("stored-glossary projection: %v", err)
|
||||||
|
}
|
||||||
|
storedProj, err := r2.projectRebill(statuses, chunks, withText, rp)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("stored-glossary re-bill projection: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// What it answers now: the bank the next `translate` will actually build.
|
||||||
|
basis, warn := r2.foldMemoryForRead()
|
||||||
|
if warn != nil {
|
||||||
|
t.Fatalf("the fold refused on a healthy book: %v", warn)
|
||||||
|
}
|
||||||
|
if basis != RebillBasisPending {
|
||||||
|
t.Fatalf("basis = %q, want %q — a healthy book must be projected from its decision files", basis, RebillBasisPending)
|
||||||
|
}
|
||||||
|
foldedProj, err := r2.projectRebill(statuses, chunks, withText, rp)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("folded re-bill projection: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// THE DELTA. Both halves are load-bearing: a non-zero "after" alone would also pass on a book where
|
||||||
|
// the old answer was non-zero too, and would not prove the blind window was the thing that closed.
|
||||||
|
if storedProj.Rows != 0 {
|
||||||
|
t.Fatalf("the repro does not discriminate: the STORED-glossary projection already sees %d unit(s); the blind window this pack closes would be invisible here", storedProj.Rows)
|
||||||
|
}
|
||||||
|
if foldedProj.Rows == 0 {
|
||||||
|
t.Fatalf("the folded projection still reports nothing after a bank edit — the blind window of row 231 is open")
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- the two boundaries the estimate is SOLD with ---
|
||||||
|
if got := rec.count(); got != callsBefore {
|
||||||
|
t.Fatalf("the estimate made %d provider call(s); it is sold as $0 and must make none", got-callsBefore)
|
||||||
|
}
|
||||||
|
committedAfter, reservedAfter, err := r2.Store.SpentUSD("test-book")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if committedAfter != committedBefore || reservedAfter != reservedBefore {
|
||||||
|
t.Fatalf("the ledger moved during a free estimate: committed %v→%v, reserved %v→%v",
|
||||||
|
committedBefore, committedAfter, reservedBefore, reservedAfter)
|
||||||
|
}
|
||||||
|
assertStoreUntouched(t, dbPath, dbBefore)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTheEstimateSurvivesABookWhoseFoldRefuses pins the property rebill.go leans on — "a book that needs
|
||||||
|
// consent stays fully inspectable" (D20.4). `translate` fails loudly on a bank whose fold would abort
|
||||||
|
// ReplaceBank; a READ must not, or the book that most needs looking at is the one that cannot be looked
|
||||||
|
// at. It must fall back AND say that it fell back, because a stale number passed off as the answer is
|
||||||
|
// worse than a labelled one.
|
||||||
|
func TestTheEstimateSurvivesABookWhoseFoldRefuses(t *testing.T) {
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := newJSONProvider(rec, draftEdit)
|
||||||
|
defer srv.Close()
|
||||||
|
bookPath := setupProjectOpts(t, srv.URL, projectOpts{
|
||||||
|
regenerate: 1, source: suzukiSource, glossarySeed: suzukiSeed,
|
||||||
|
})
|
||||||
|
dir := filepath.Dir(bookPath)
|
||||||
|
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
|
||||||
|
|
||||||
|
r1 := newRunner(t, bookPath)
|
||||||
|
if _, err := r1.TranslateBook(ctx); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
r1.Close()
|
||||||
|
|
||||||
|
// A mined-delta row on the SAME UNIQUE key as the signed seed term: exactly what would abort
|
||||||
|
// ReplaceBank's flat INSERT, and what seedGlossary refuses on before it gets there.
|
||||||
|
writeFile(t, filepath.Join(dir, "test-book.mined-delta.yaml"), `
|
||||||
|
terms:
|
||||||
|
- src: 鈴木
|
||||||
|
dst: Судзуки-другой
|
||||||
|
type: name
|
||||||
|
status: approved
|
||||||
|
`)
|
||||||
|
r2, err := NewReadOnlyRunner(bookPath, obs.NewLogger())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer r2.Close()
|
||||||
|
|
||||||
|
basis, warn := r2.foldMemoryForRead()
|
||||||
|
if basis != RebillBasisStored {
|
||||||
|
t.Fatalf("basis = %q, want %q — a refused fold must fall back to the stored glossary, not fail the read", basis, RebillBasisStored)
|
||||||
|
}
|
||||||
|
if warn == nil {
|
||||||
|
t.Fatalf("a silent fallback is the defect: the reason the projection is a fact about the PAST must reach the log")
|
||||||
|
}
|
||||||
|
if r2.memory == nil || r2.baseMemory == nil {
|
||||||
|
t.Fatalf("the fallback must still leave a materialized bank behind (memory=%v base=%v)", r2.memory != nil, r2.baseMemory != nil)
|
||||||
|
}
|
||||||
|
// And the whole read still answers rather than erroring.
|
||||||
|
rep, err := r2.Status(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("status refused a book whose fold refuses — D20.4 says it must stay inspectable: %v", err)
|
||||||
|
}
|
||||||
|
if rep.RebillBasis != RebillBasisStored {
|
||||||
|
t.Fatalf("the report must carry the fallback basis, got %q", rep.RebillBasis)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTheWireTellsFreeFromUnknown pins the omitempty mine D39.166 п.2 already paid for once: three states
|
||||||
|
// collapse to two zeroes unless the basis rides with them.
|
||||||
|
func TestTheWireTellsFreeFromUnknown(t *testing.T) {
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := newJSONProvider(rec, draftEdit)
|
||||||
|
defer srv.Close()
|
||||||
|
bookPath := setupProjectOpts(t, srv.URL, projectOpts{
|
||||||
|
regenerate: 1, source: suzukiSource, glossarySeed: suzukiSeed,
|
||||||
|
})
|
||||||
|
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
|
||||||
|
|
||||||
|
// A book nothing has run: the figures are zero because there is nothing to re-pay.
|
||||||
|
r0, err := NewReadOnlyRunner(bookPath, obs.NewLogger())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rep0, err := r0.Status(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
r0.Close()
|
||||||
|
if rep0.RebillBasis != RebillBasisNone {
|
||||||
|
t.Fatalf("a book with no stored row must report basis %q, got %q", RebillBasisNone, rep0.RebillBasis)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A book that ran and drifted from nothing: the figures are zero because they were COMPUTED as zero.
|
||||||
|
r1 := newRunner(t, bookPath)
|
||||||
|
if _, err := r1.TranslateBook(ctx); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
r1.Close()
|
||||||
|
r2, err := NewReadOnlyRunner(bookPath, obs.NewLogger())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer r2.Close()
|
||||||
|
rep1, err := r2.Status(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if rep1.RebillBasis != RebillBasisPending {
|
||||||
|
t.Fatalf("a healthy run must report basis %q, got %q", RebillBasisPending, rep1.RebillBasis)
|
||||||
|
}
|
||||||
|
if rep1.RebillUnits != 0 {
|
||||||
|
t.Fatalf("precondition: an undrifted book re-pays nothing, got %d", rep1.RebillUnits)
|
||||||
|
}
|
||||||
|
// The two states above BOTH carry zero. Before this pack they were the same bytes on the wire.
|
||||||
|
if rep0.RebillBasis == rep1.RebillBasis {
|
||||||
|
t.Fatalf("«nothing to re-pay» and «computed, and it is zero» must be distinguishable, both say %q", rep0.RebillBasis)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestAFailedBasisNeverCarriesAFigure pins the direction the basis field's first version did not
|
||||||
|
// consider. It was added so that "computed, and it is zero" could be told from "we could not compute it"
|
||||||
|
// — but a failed fold leaves r.memory nil, and projectRebill run against a nil bank renders both wave
|
||||||
|
// snapshots over an EMPTY bank, so EVERY stored row differs and the whole book comes back as a
|
||||||
|
// re-payment. The wire would then carry rebill_basis:"failed" beside the largest number the field can
|
||||||
|
// hold, and the CLI would print "the figures below are zero because they could not be computed" next to
|
||||||
|
// it. A basis that contradicts its own numbers is worse than no basis.
|
||||||
|
func TestAFailedBasisNeverCarriesAFigure(t *testing.T) {
|
||||||
|
big := RebillProjection{Rows: 4276, USD: 12.5, OutputUnits: 1400}
|
||||||
|
for _, c := range []struct {
|
||||||
|
name string
|
||||||
|
hasRows bool
|
||||||
|
memBasis string
|
||||||
|
proj RebillProjection
|
||||||
|
projErr error
|
||||||
|
basis string
|
||||||
|
units int
|
||||||
|
}{
|
||||||
|
{"nothing stored", false, RebillBasisPending, big, nil, RebillBasisNone, 0},
|
||||||
|
{"healthy fold", true, RebillBasisPending, big, nil, RebillBasisPending, 4276},
|
||||||
|
{"fold refused, stored answered", true, RebillBasisStored, big, nil, RebillBasisStored, 4276},
|
||||||
|
// The two that matter: an unknown answer must never arrive carrying a number. Before this was one
|
||||||
|
// decision, the basis said "failed" while a whole-book figure — computed over an EMPTY bank, so
|
||||||
|
// every row looked superseded — sat next to it.
|
||||||
|
{"no bank at all", true, RebillBasisFailed, big, nil, RebillBasisFailed, 0},
|
||||||
|
{"projection errored", true, RebillBasisPending, big, errors.New("boom"), RebillBasisFailed, 0},
|
||||||
|
} {
|
||||||
|
t.Run(c.name, func(t *testing.T) {
|
||||||
|
basis, units, usd, outUnits := rebillOutcome(c.hasRows, c.memBasis, c.proj, c.projErr)
|
||||||
|
if basis != c.basis || units != c.units {
|
||||||
|
t.Fatalf("got (%q, %d, %v), want (%q, %d)", basis, units, usd, c.basis, c.units)
|
||||||
|
}
|
||||||
|
// EVERY figure, not just the two obvious ones: an output-unit count surviving an UNKNOWN
|
||||||
|
// answer would be the same contradiction in a different field.
|
||||||
|
if basis == RebillBasisFailed && (units != 0 || usd != 0 || outUnits != 0) {
|
||||||
|
t.Fatalf("an UNKNOWN answer carried a figure: %d units, $%v, %d output units", units, usd, outUnits)
|
||||||
|
}
|
||||||
|
if basis == RebillBasisNone && outUnits != 0 {
|
||||||
|
t.Fatalf("nothing stored means nothing to re-pay, yet %d output units were reported", outUnits)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
// And the projection is not even attempted when there is no bank to compare against — running it
|
||||||
|
// would produce exactly the whole-book figure the case above must never show.
|
||||||
|
if projectable(true, RebillBasisFailed) {
|
||||||
|
t.Fatalf("with no materialized bank the projection must be skipped, not computed and then discarded")
|
||||||
|
}
|
||||||
|
if !projectable(true, RebillBasisPending) || projectable(false, RebillBasisPending) {
|
||||||
|
t.Fatalf("projectable must gate on stored rows and on a usable bank, and on nothing else")
|
||||||
|
}
|
||||||
|
|
||||||
|
// And end to end: a book whose fold refuses AND whose store cannot answer either. The fold refusal is
|
||||||
|
// built with a mined-delta duplicating the signed seed term; the stored side is broken by closing the
|
||||||
|
// store out from under the read, which is the only way both halves fail at once.
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := newJSONProvider(rec, draftEdit)
|
||||||
|
defer srv.Close()
|
||||||
|
bookPath := setupProjectOpts(t, srv.URL, projectOpts{
|
||||||
|
regenerate: 1, source: suzukiSource, glossarySeed: suzukiSeed,
|
||||||
|
})
|
||||||
|
dir := filepath.Dir(bookPath)
|
||||||
|
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
|
||||||
|
|
||||||
|
r1 := newRunner(t, bookPath)
|
||||||
|
if _, err := r1.TranslateBook(ctx); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
r1.Close()
|
||||||
|
writeFile(t, filepath.Join(dir, "test-book.mined-delta.yaml"), `
|
||||||
|
terms:
|
||||||
|
- src: 鈴木
|
||||||
|
dst: Судзуки-другой
|
||||||
|
type: name
|
||||||
|
status: approved
|
||||||
|
`)
|
||||||
|
r2, err := NewReadOnlyRunner(bookPath, obs.NewLogger())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer r2.Close()
|
||||||
|
got, err := r2.Status(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// Here only the FOLD fails, so the honest answer is the stored fallback with a real figure — the
|
||||||
|
// point being that "failed" is reserved for the state where nothing at all could be materialized.
|
||||||
|
if got.RebillBasis != RebillBasisStored {
|
||||||
|
t.Fatalf("a fold refusal alone falls back rather than failing: got %q", got.RebillBasis)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestExportAndStatusAnswerDifferentDriftQuestions pins a divergence that is DELIBERATE, so that nobody
|
||||||
|
// removes it as an inconsistency.
|
||||||
|
//
|
||||||
|
// After a bank-apply that has written only decision FILES, `status` reports drift and `export` does not,
|
||||||
|
// and both are right because they are asked different things. status is the money surface: its question
|
||||||
|
// is "what would the NEXT run cost", and an un-run decision file is precisely the spend row 231 exists to
|
||||||
|
// reveal. export's ConfigDrift is read by the polygon's extraction to decide whether a document is
|
||||||
|
// trustworthy to MEASURE — and an export whose bytes are exactly what its run shipped is measurable
|
||||||
|
// whether or not somebody has queued a bank edit beside it.
|
||||||
|
//
|
||||||
|
// The pack briefly made export fold too, for symmetry. Symmetry between two surfaces answering different
|
||||||
|
// questions is not worth having, and it would have disqualified sound measurements in another zone.
|
||||||
|
func TestExportAndStatusAnswerDifferentDriftQuestions(t *testing.T) {
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := newJSONProvider(rec, draftEdit)
|
||||||
|
defer srv.Close()
|
||||||
|
bookPath := setupProjectOpts(t, srv.URL, projectOpts{
|
||||||
|
regenerate: 1, source: suzukiSource, glossarySeed: suzukiSeed,
|
||||||
|
})
|
||||||
|
dir := filepath.Dir(bookPath)
|
||||||
|
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
|
||||||
|
|
||||||
|
r1 := newRunner(t, bookPath)
|
||||||
|
if _, err := r1.TranslateBook(ctx); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
r1.Close()
|
||||||
|
|
||||||
|
doc := decisionsDocFor(t, dir, "test-book", membank.Decision{
|
||||||
|
Action: membank.ActionApprove, Src: "静か", Dst: "тихий"})
|
||||||
|
if _, err := ApplyBankDecisions(ctx, bookPath, doc, false); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
callsBefore := rec.count()
|
||||||
|
rExp, err := NewReadOnlyRunner(bookPath, obs.NewLogger())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
exp, err := rExp.Export(false)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rExp.Close()
|
||||||
|
if exp.ConfigDrift {
|
||||||
|
t.Fatalf("export marked a document drifted whose text is exactly what its run shipped: the polygon reads this field to decide whether a measurement is sound, and an un-run decision file does not make one unsound")
|
||||||
|
}
|
||||||
|
|
||||||
|
rSt, err := NewReadOnlyRunner(bookPath, obs.NewLogger())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer rSt.Close()
|
||||||
|
rep, err := rSt.Status(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !rep.ConfigDrift {
|
||||||
|
t.Fatalf("status must see the pending bank edit — that is the whole of row 231; got config_drift=false")
|
||||||
|
}
|
||||||
|
if rep.RebillUnits == 0 {
|
||||||
|
t.Fatalf("status must price the pending edit, got rebill_units=0 basis=%q", rep.RebillBasis)
|
||||||
|
}
|
||||||
|
// Both are $0 surfaces.
|
||||||
|
if got := rec.count(); got != callsBefore {
|
||||||
|
t.Fatalf("a read surface made %d provider call(s)", got-callsBefore)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTheEstimateIsAlsoGivenInTheUnitAPurchaseIsSizedIn pins the denomination fix.
|
||||||
|
//
|
||||||
|
// `rebill_units` counts chunk×stage — the BILLING granularity — while every other "units" number in the
|
||||||
|
// same document (total_units, the progress totals, chapters[].units_total) counts OUTPUT units, which is
|
||||||
|
// also what `--max-units` bounds and what the platform sells chapters in. Both wore the word "units" with
|
||||||
|
// nothing marking the difference, and the ratio is not something a consumer can divide out: it is
|
||||||
|
// len(Members)·nDraftStages + nEditStages, which varies unit by unit inside one book and whose factors
|
||||||
|
// never cross the seam. A platform sizing a re-pass from the estimate and handing that number to
|
||||||
|
// --max-units would buy several times the book it meant to — this pack's own defect, one layer down.
|
||||||
|
func TestTheEstimateIsAlsoGivenInTheUnitAPurchaseIsSizedIn(t *testing.T) {
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := newJSONProvider(rec, draftEdit)
|
||||||
|
defer srv.Close()
|
||||||
|
// A cut fine enough that units hold several chunks, so the two denominations genuinely differ.
|
||||||
|
const seg = "\nsegmentation:\n draft_budget_out: 24\n edit_ceiling_out: 8000\n fertility: { cjk: 1.1978, other: 0.3852 }\n"
|
||||||
|
var eps []chunktest.Chapter
|
||||||
|
var spine []string
|
||||||
|
for i := 1; i <= 2; i++ {
|
||||||
|
id := fmt.Sprintf("c%d", i)
|
||||||
|
eps = append(eps, chunktest.Chapter{ID: id, Href: id + ".xhtml",
|
||||||
|
Body: fmt.Sprintf("<p>静かな図書館の朝%d。鈴木は本を読んだ。外では雨が降っていた。彼は窓を見た。</p>", i)})
|
||||||
|
spine = append(spine, id)
|
||||||
|
}
|
||||||
|
bookPath := setupProjectOpts(t, srv.URL, projectOpts{
|
||||||
|
epub: eps, spine: spine, regenerate: 1, waveWorkers: 1, gatesYAML: seg,
|
||||||
|
})
|
||||||
|
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
|
||||||
|
|
||||||
|
r1 := newRunner(t, bookPath)
|
||||||
|
if _, err := r1.TranslateBook(ctx); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
r1.Close()
|
||||||
|
// A PIPELINE drift, not a bank edit: a mined bank edit moves only the edit wave, so it re-pays exactly
|
||||||
|
// one row per unit and the two denominations would coincide — the fixture would prove nothing. A config
|
||||||
|
// change moves BOTH waves, which is the ordinary shape in which a unit's several draft rows and its one
|
||||||
|
// edit row are all re-paid together, and where the collision this field exists for is real.
|
||||||
|
driftPipelineVersion(t, bookPath)
|
||||||
|
|
||||||
|
r2, err := NewReadOnlyRunner(bookPath, obs.NewLogger())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer r2.Close()
|
||||||
|
rep, err := r2.Status(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if rep.RebillUnits == 0 {
|
||||||
|
t.Fatalf("precondition: the bank edit must produce a re-payment, got 0 (basis %q)", rep.RebillBasis)
|
||||||
|
}
|
||||||
|
if rep.RebillOutputUnits == 0 {
|
||||||
|
t.Fatalf("the estimate must also be given in OUTPUT units — the granularity --max-units counts and the platform sells in; got 0 beside rebill_units=%d", rep.RebillUnits)
|
||||||
|
}
|
||||||
|
if rep.RebillOutputUnits > rep.RebillUnits {
|
||||||
|
t.Fatalf("an output unit holds one or more chunk×stage rows, so it can never exceed them: output=%d rows=%d", rep.RebillOutputUnits, rep.RebillUnits)
|
||||||
|
}
|
||||||
|
if rep.RebillOutputUnits > rep.TotalUnits {
|
||||||
|
t.Fatalf("the book has %d output units; a re-pass cannot touch %d of them", rep.TotalUnits, rep.RebillOutputUnits)
|
||||||
|
}
|
||||||
|
// The fixture must actually make the two differ, or it proves nothing about the collision.
|
||||||
|
if rep.RebillOutputUnits == rep.RebillUnits {
|
||||||
|
t.Fatalf("fixture is degenerate: the two denominations coincide (%d), so the trap this field exists for is untested", rep.RebillUnits)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -103,6 +103,28 @@ func jsonEqual(a, b json.RawMessage) bool {
|
||||||
return xe == nil && ye == nil && string(xb) == string(yb)
|
return xe == nil && ye == nil && string(xb) == string(yb)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// cachedRenderedContentHashes is renderedContentHashes memoized for the life of ONE materialized bank.
|
||||||
|
//
|
||||||
|
// The reproduction walks every stage of every position and re-renders it — the expensive half of every
|
||||||
|
// $0 projection in this engine. Since the volume ceiling landed, a single bounded run could pay for it
|
||||||
|
// THREE times: once in planVolume, and once in each of the consent gate's two projections (the book's and
|
||||||
|
// the run's). Nothing changed between them, so two of the three were pure waste, and the cost scales with
|
||||||
|
// the book — exactly where it is least affordable.
|
||||||
|
//
|
||||||
|
// ⚠ THE CACHE'S LIFETIME IS THE BANK'S, and that is what makes it safe rather than merely fast. Every
|
||||||
|
// hash here is rendered against the materialized memory, so the memo is only valid while that memory is;
|
||||||
|
// materializeBanks clears it, which is the ONE place the bank can change (a mid-run re-seed at the
|
||||||
|
// bank-mining stop goes through it like everything else). A memo that outlived its bank would hand a
|
||||||
|
// caller hashes for a bank the run no longer has — the precise class of silent wrongness the free
|
||||||
|
// estimate exists to avoid.
|
||||||
|
func (r *Runner) cachedRenderedContentHashes(chunks []chunk.Chunk, stickySel []membank.Selection) map[chunkKey]map[string]string {
|
||||||
|
if r.contentHashes != nil {
|
||||||
|
return r.contentHashes
|
||||||
|
}
|
||||||
|
r.contentHashes = r.renderedContentHashes(chunks, stickySel)
|
||||||
|
return r.contentHashes
|
||||||
|
}
|
||||||
|
|
||||||
// renderedContentHashes reproduces, for every live position of every stage, the content hash the run
|
// renderedContentHashes reproduces, for every live position of every stage, the content hash the run
|
||||||
// WOULD compute — the same msgsContentHash the resume fast-path compares against. It is the $0 half of
|
// WOULD compute — the same msgsContentHash the resume fast-path compares against. It is the $0 half of
|
||||||
// the honest estimate: with it, "the bank moved" can be answered per unit instead of per wave.
|
// the honest estimate: with it, "the bank moved" can be answered per unit instead of per wave.
|
||||||
|
|
|
||||||
|
|
@ -76,6 +76,19 @@ type Runner struct {
|
||||||
// setting it moves no request_hash and re-bills nothing. The DAY ceiling is deliberately not
|
// setting it moves no request_hash and re-bills nothing. The DAY ceiling is deliberately not
|
||||||
// overridable: it is an account-wide guard, not a property of one run.
|
// overridable: it is an account-wide guard, not a property of one run.
|
||||||
CeilingUSD float64
|
CeilingUSD float64
|
||||||
|
// MaxUnits is the VOLUME ceiling for THIS RUN (tmctl --max-units, D39.165 §1б), 0 = unset. It bounds
|
||||||
|
// how many OUTPUT UNITS this run may pay for — the granularity the manifest publishes and the platform
|
||||||
|
// sells chapters in — so a purchase of N chapters can stop at N instead of at whatever N chapters'
|
||||||
|
// worth of dollars happens to buy.
|
||||||
|
//
|
||||||
|
// ⚠ IT IS ORTHOGONAL TO CeilingUSD AND MUST STAY SO. CeilingUSD is a BOOK-CUMULATIVE money bound the
|
||||||
|
// ledger judges every reservation against; MaxUnits is a bound on THIS run's own work and never
|
||||||
|
// touches the ledger, the reservations or the hold arithmetic the platform builds on that
|
||||||
|
// cumulativeness. They can both be in force and either can be the one that stops the run — and the two
|
||||||
|
// stops are different answers to the operator: money means "the run was cut short, add money", volume
|
||||||
|
// means "the run did what was bought". Like the ceilings it is an operator axis, not a wire one: it is
|
||||||
|
// in neither BriefHash nor the snapshot payload, so setting it moves no request_hash and re-bills nobody.
|
||||||
|
MaxUnits int
|
||||||
|
|
||||||
clients map[string]llm.LLMClient
|
clients map[string]llm.LLMClient
|
||||||
templates map[string]*PromptTemplate
|
templates map[string]*PromptTemplate
|
||||||
|
|
@ -131,6 +144,14 @@ type Runner struct {
|
||||||
// (identical content) — no double materialization, the injection is unchanged. The editor keeps the
|
// (identical content) — no double materialization, the injection is unchanged. The editor keeps the
|
||||||
// enriched `memory`. nil ⇔ memory is nil (materialized together in seedGlossary).
|
// enriched `memory`. nil ⇔ memory is nil (materialized together in seedGlossary).
|
||||||
baseMemory *membank.Bank
|
baseMemory *membank.Bank
|
||||||
|
// contentHashes memoizes the per-position wire-hash reproduction for the life of the CURRENT
|
||||||
|
// materialized bank (cachedRenderedContentHashes). Cleared by materializeBanks, which is the one place
|
||||||
|
// the bank can change.
|
||||||
|
contentHashes map[chunkKey]map[string]string
|
||||||
|
// bankRows are the glossary rows the CURRENT materialization was built from, in the store's order
|
||||||
|
// (materializeBanks). They exist so a read surface can describe the bank it actually folded rather
|
||||||
|
// than re-reading a different one out of the store.
|
||||||
|
bankRows []store.GlossaryEntry
|
||||||
|
|
||||||
// checkers is the compiled WS5/pack-13 observability checker spec (pair-14 data-out): the pair's
|
// checkers is the compiled WS5/pack-13 observability checker spec (pair-14 data-out): the pair's
|
||||||
// DETECTION patterns + tables (from pack) + the target-general lists (from embedded target data),
|
// DETECTION patterns + tables (from pack) + the target-general lists (from embedded target data),
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,6 @@ import (
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"textmachine/backend/internal/chunk"
|
"textmachine/backend/internal/chunk"
|
||||||
"textmachine/backend/internal/lang"
|
|
||||||
"textmachine/backend/internal/membank"
|
"textmachine/backend/internal/membank"
|
||||||
"textmachine/backend/internal/store"
|
"textmachine/backend/internal/store"
|
||||||
)
|
)
|
||||||
|
|
@ -21,94 +20,27 @@ import (
|
||||||
// once before snapshotID: the frozen APPROVED rows are hashed into memoryVersion (F1), so
|
// once before snapshotID: the frozen APPROVED rows are hashed into memoryVersion (F1), so
|
||||||
// editing the seed is a loud --resnapshot. Idempotent (full replace), $0, no LLM. B2
|
// editing the seed is a loud --resnapshot. Idempotent (full replace), $0, no LLM. B2
|
||||||
// approved dst-collisions are logged (not fatal — some collisions are legitimate).
|
// approved dst-collisions are logged (not fatal — some collisions are legitimate).
|
||||||
|
//
|
||||||
|
// The gather and the materialization live in bankmaterialize.go, because a read-only surface has to reach the
|
||||||
|
// SAME fold without the write in the middle (row 231): what it cannot do is write, and the write is the
|
||||||
|
// only step it has to skip. This function is therefore the write path's spelling of one shared fold —
|
||||||
|
// gather, persist, read back, materialize — and the insert order it hands ReplaceBank is untouched.
|
||||||
func (r *Runner) seedGlossary(ctx context.Context) error {
|
func (r *Runner) seedGlossary(ctx context.Context) error {
|
||||||
var entries []store.GlossaryEntry
|
in, err := r.gatherBankInputs()
|
||||||
var voices []store.VoiceProfile
|
// The remarks go out BEFORE the error is acted on, and the order is load-bearing rather than tidy.
|
||||||
var pairs []store.AddressPair
|
// Before the gather was extracted these warnings were emitted inline as they were made, so a book that
|
||||||
if r.Book.GlossarySeed != "" {
|
// died on a later collision check still told the operator about the ruby alias it had skipped or the
|
||||||
bank, err := membank.LoadBankSeed(r.Book.GlossarySeed)
|
// auto-bank rows it had dropped. Logging them after the error return would lose exactly the diagnostics
|
||||||
if err != nil {
|
// of the run that most needs them — the one that failed. gatherBankInputs returns its partial inputs
|
||||||
return err
|
// with the error for this reason.
|
||||||
}
|
for _, rem := range in.remarks {
|
||||||
entries = append(entries, bank.Terms...)
|
r.Log.WarnContext(ctx, rem.msg, rem.args...)
|
||||||
voices, pairs = bank.Voices, bank.Pairs
|
|
||||||
}
|
}
|
||||||
manualSrcs := map[string]bool{}
|
|
||||||
for _, e := range entries {
|
|
||||||
manualSrcs[e.Src] = true
|
|
||||||
}
|
|
||||||
ruby, err := r.Store.RubyReadingsForBook(r.Book.BookID)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("pipeline: read ruby readings for %s: %w", r.Book.BookID, err)
|
|
||||||
}
|
|
||||||
// D16.4: attach a manual term's kana ruby-reading as an alias (kana spelling matchable)
|
|
||||||
// BEFORE appending the auto-candidates, so it only touches the curated manual entries. A
|
|
||||||
// reading that would collide with a different seeded term (homophone) is skipped+logged,
|
|
||||||
// not attached (which would fail the book loud on an alias the operator cannot edit out).
|
|
||||||
if skipped := membank.AttachRubyAliasesToManual(entries, ruby); len(skipped) > 0 {
|
|
||||||
r.Log.WarnContext(ctx, "ruby kana-alias skipped as a homophone collision (kana form left unmatchable; disambiguate in the seed if needed)",
|
|
||||||
"skipped", strings.Join(skipped, "; "))
|
|
||||||
}
|
|
||||||
// Mined-write path (R1, plan §1(в)/F2): the owner-curated mined-delta file joins the seed as
|
|
||||||
// Source:"mined" (loadMinedDelta re-stamps the seed loader's Source), so its terms fold into the
|
|
||||||
// ENRICHED bank version but NOT the base — a re-run that adds signed mined terms moves ONLY snapshot_W2.
|
|
||||||
// Loaded AFTER the seed/ruby so manualSrcs already reflects the curated seed. A `mined` term that
|
|
||||||
// duplicates a seed src is caught by membank.ApprovedSharedKeyCollisions below like any other collision.
|
|
||||||
minedDelta, err := r.loadMinedDelta()
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
// D39.20 deviation-#1 fix: a mined-delta term whose UNIQUE key (src, sense, since_ch, until_ch —
|
if err := r.Store.ReplaceBank(r.Book.BookID, in.entries, in.voices, in.pairs); err != nil {
|
||||||
// store/migrate.go glossary UNIQUE) already exists in the SIGNED seed makes the flat INSERT in
|
return fmt.Errorf("pipeline: replace bank for %s (%d terms, %d voices, %d pairs): %w", r.Book.BookID, len(in.entries), len(in.voices), len(in.pairs), err)
|
||||||
// ReplaceGlossary crash on that constraint and abort the whole paid run. membank.ApprovedSharedKeyCollisions
|
|
||||||
// below does NOT catch it (it skips a SAME-dst duplicate, and keys on the firing surface, not the
|
|
||||||
// UNIQUE tuple). Fail LOUD here with the duplicate list + a fix hint (edit the seed OR drop it from
|
|
||||||
// the delta) — NEVER a silent merge over the signed seed. Checked BEFORE the append so the delta rows
|
|
||||||
// are still separable. Deterministic (seed order).
|
|
||||||
if dups := membank.MinedDeltaSeedCollisions(entries, minedDelta); len(dups) > 0 {
|
|
||||||
return fmt.Errorf("pipeline: mined-delta %s duplicates term(s) already in the signed seed (would crash ReplaceGlossary on the glossary UNIQUE(book_id,src,sense,since_ch,until_ch)):\n - %s\n fix: correct the term in the seed, or remove it from the mined-delta file — never both (no silent merge over the signed seed)",
|
|
||||||
r.Book.MinedDelta, strings.Join(dups, "\n - "))
|
|
||||||
}
|
|
||||||
entries = append(entries, minedDelta...)
|
|
||||||
// AUTO-BANK (pack-20 / D39.42 п.3): the engine's own unsigned rows — what the terminologist consolidated
|
|
||||||
// on the last run's bank-mining boundary. They load exactly like the owner-curated delta (Source:"mined",
|
|
||||||
// so base-excluded and only the edit-wave snapshot moves) but they are NOT signed: their status is
|
|
||||||
// auto/draft, which is what routes them to the editor's separately-headed unverified section instead of
|
|
||||||
// the canon list.
|
|
||||||
//
|
|
||||||
// Loading them HERE rather than injecting them mid-run is load-bearing for money. seedGlossary runs before
|
|
||||||
// checkRebillConsent, so the re-payment projection sees the same bank the edit wave will use; if the rows
|
|
||||||
// only appeared after the projection, the next run would compare stored edit rows against a bank that has
|
|
||||||
// not been rebuilt yet and project a re-payment of the whole edit wave that is not real.
|
|
||||||
autoBank, dropped, err := r.loadAutoBank(entries)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if len(dropped) > 0 {
|
|
||||||
// A collision with a SIGNED row cannot abort a paid run over an engine-written proposal — the signed
|
|
||||||
// row simply wins and the proposal is dropped, loudly.
|
|
||||||
r.Log.WarnContext(ctx, "auto-bank rows dropped: their key is already held by a signed term (the signed term wins)",
|
|
||||||
"book", r.Book.BookID, "dropped", strings.Join(dropped, "; "))
|
|
||||||
}
|
|
||||||
entries = append(entries, autoBank...)
|
|
||||||
for i := range entries {
|
|
||||||
entries[i].BookID = r.Book.BookID
|
|
||||||
}
|
|
||||||
// Task-6 re-audit: fail loud on a firing key shared by two DIFFERENT approved terms with
|
|
||||||
// different dst + overlapping windows (the alias generalization of the D16.1 polysemy
|
|
||||||
// livelock) — checked over the FULL entry set (incl. ruby-attached aliases) before persisting.
|
|
||||||
if cols := membank.ApprovedSharedKeyCollisions(entries); len(cols) > 0 {
|
|
||||||
return fmt.Errorf("pipeline: glossary shared-key collisions (A2 / D16.1 livelock class):\n - %s", strings.Join(cols, "\n - "))
|
|
||||||
}
|
|
||||||
// A voice/address row naming a character the bank does not have is SILENTLY inert — nothing can ever
|
|
||||||
// attribute a reply to it — which is the A-class hole this bank exists to close, so it stops the run.
|
|
||||||
// Checked over the FULL entry set (seed + ruby + mined + auto), because a character may legitimately be
|
|
||||||
// signed in a delta rather than in the base seed.
|
|
||||||
if unknown := membank.UnknownVoiceCharacters(entries, voices, pairs); len(unknown) > 0 {
|
|
||||||
return fmt.Errorf("pipeline: voice/address rows name characters absent from the bank (a profile for a term that does not exist can never fire):\n - %s", strings.Join(unknown, "\n - "))
|
|
||||||
}
|
|
||||||
if err := r.Store.ReplaceBank(r.Book.BookID, entries, voices, pairs); err != nil {
|
|
||||||
return fmt.Errorf("pipeline: replace bank for %s (%d terms, %d voices, %d pairs): %w", r.Book.BookID, len(entries), len(voices), len(pairs), err)
|
|
||||||
}
|
}
|
||||||
rows, err := r.Store.GlossaryForBook(r.Book.BookID)
|
rows, err := r.Store.GlossaryForBook(r.Book.BookID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|
@ -128,36 +60,7 @@ func (r *Runner) seedGlossary(ctx context.Context) error {
|
||||||
r.Log.WarnContext(ctx, "voice profile windows leave chapters uncovered (deliberate is fine; a typo is not)",
|
r.Log.WarnContext(ctx, "voice profile windows leave chapters uncovered (deliberate is fine; a typo is not)",
|
||||||
"book", r.Book.BookID, "gaps", strings.Join(gaps, "; "))
|
"book", r.Book.BookID, "gaps", strings.Join(gaps, "; "))
|
||||||
}
|
}
|
||||||
// InjectVoice is FALSE and has no config knob: pack-19 builds the schema and the flagger, and D21 п.2
|
r.materializeBanks(rows, storedVoices, storedPairs)
|
||||||
// holds the injection conditional until the polygon experiment. It is the fold's condition, so while
|
|
||||||
// it is false a book with voice rows hashes exactly as it did without them and nobody re-pays for
|
|
||||||
// authoring a profile. Wiring the injection means setting it and accepting a full --resnapshot.
|
|
||||||
// The §3 decl stemmer is target data, constant per book; baseIn copies bankIn below, so both banks share
|
|
||||||
// it. A target with no decl_suffix registry yields an inert stemmer → exact-match post-check as before.
|
|
||||||
bankIn := membank.BankInput{Rows: rows, Voices: storedVoices, Pairs: storedPairs,
|
|
||||||
TargetStemmer: lang.NewTargetStemmer(lang.TargetChecksFor(r.Book.TargetLang))}
|
|
||||||
r.memory = membank.MaterializeBank(bankIn, r.Pipeline.Gates.Glossary.PostcheckGate)
|
|
||||||
// The DRAFT wave selects over a BASE-scoped bank (Source:mined excluded) so its injection is
|
|
||||||
// byte-identical across a bank-mining enrichment — matching the draft-wave snapshot (baseMemoryVersion),
|
|
||||||
// which keeps «one re-payment» honest at the WIRE level, not only the version-hash level. Only the
|
|
||||||
// editor sees mined terms (the enriched `memory`). When there are no mined rows (every $0 test / the
|
|
||||||
// golden) the base bank IS the enriched one — share the object, no double materialization, no drift.
|
|
||||||
baseRows := rows[:0:0]
|
|
||||||
hasMined := false
|
|
||||||
for _, row := range rows {
|
|
||||||
if row.Source == "mined" {
|
|
||||||
hasMined = true
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
baseRows = append(baseRows, row)
|
|
||||||
}
|
|
||||||
if hasMined {
|
|
||||||
baseIn := bankIn
|
|
||||||
baseIn.Rows = baseRows
|
|
||||||
r.baseMemory = membank.MaterializeBank(baseIn, r.Pipeline.Gates.Glossary.PostcheckGate)
|
|
||||||
} else {
|
|
||||||
r.baseMemory = r.memory
|
|
||||||
}
|
|
||||||
if cols := membank.InjectivityCollisions(rows); len(cols) > 0 {
|
if cols := membank.InjectivityCollisions(rows); len(cols) > 0 {
|
||||||
r.Log.WarnContext(ctx, "glossary approved dst-collisions (B2: two source terms share one Russian surface — the reader cannot tell them apart)",
|
r.Log.WarnContext(ctx, "glossary approved dst-collisions (B2: two source terms share one Russian surface — the reader cannot tell them apart)",
|
||||||
"collisions", strings.Join(cols, "; "))
|
"collisions", strings.Join(cols, "; "))
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,6 @@ import (
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"textmachine/backend/internal/chunk"
|
"textmachine/backend/internal/chunk"
|
||||||
"textmachine/backend/internal/membank"
|
|
||||||
"textmachine/backend/internal/store"
|
"textmachine/backend/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
@ -204,9 +203,43 @@ type StatusReport struct {
|
||||||
// cent or the whole book — these say "N chunk×stage units already billed under a superseded snapshot
|
// cent or the whole book — these say "N chunk×stage units already billed under a superseded snapshot
|
||||||
// would be paid for again, ~$X at the CURRENT price table" (row 181). Same projection the consent
|
// would be paid for again, ~$X at the CURRENT price table" (row 181). Same projection the consent
|
||||||
// gate refuses on (projectRebill), so status can never quote a different number than the one
|
// gate refuses on (projectRebill), so status can never quote a different number than the one
|
||||||
// translate enforces. Both omitempty: a book with no drift keeps its exact prior bytes.
|
// translate enforces.
|
||||||
RebillUnits int `json:"rebill_units,omitempty"`
|
//
|
||||||
RebillUSD float64 `json:"rebill_usd,omitempty"`
|
// ⚠ NEITHER IS omitempty ANY MORE, and dropping it is the point rather than a tidy-up. Three states
|
||||||
|
// reach this line — "nothing has run, so there is nothing to re-pay", "computed, and the answer is
|
||||||
|
// zero" and "the computation failed" — and omitempty rendered all three as the field being ABSENT, so
|
||||||
|
// the wire could not tell "free" from "unknown". That is the D39.166 п.2 class, which this engine has
|
||||||
|
// already paid for once ("a $0 price walls the door up — units at zero price arrive with no price").
|
||||||
|
// RebillBasis below is what distinguishes them; the numbers are now always present so the basis has
|
||||||
|
// something to qualify. Safe to change: both fields are OFF the seam's allowlist until a consumer
|
||||||
|
// exists (backlog row 231), and the human renderer branches on the VALUE, not on presence.
|
||||||
|
RebillUnits int `json:"rebill_units"`
|
||||||
|
RebillUSD float64 `json:"rebill_usd"`
|
||||||
|
// RebillOutputUnits is the SAME re-payment counted in OUTPUT UNITS — the granularity every other
|
||||||
|
// "units" number in this document uses (total_units, progress totals, chapters[].units_total) and the
|
||||||
|
// one `--max-units` bounds and the platform sells chapters in.
|
||||||
|
//
|
||||||
|
// ⚠ IT EXISTS BECAUSE rebill_units IS A DIFFERENT UNIT WEARING THE SAME WORD: chunk×stage, the BILLING
|
||||||
|
// granularity. Measured on a three-chapter fixture, one document carried rebill_units:15 two lines
|
||||||
|
// under total_units:6, and the ratio is not something a consumer can divide out — it is
|
||||||
|
// len(Members)·nDraftStages + nEditStages, varying unit by unit inside one book, with its factors never
|
||||||
|
// crossing the seam. A consumer sizing a re-pass from rebill_units and handing that number to
|
||||||
|
// --max-units would buy several times the book it meant to: this pack's own defect, one layer down.
|
||||||
|
// rebill_units is NOT renamed — it is established and its meaning is unchanged; this is the number a
|
||||||
|
// caller should size a purchase from.
|
||||||
|
RebillOutputUnits int `json:"rebill_output_units"`
|
||||||
|
// RebillBasis says WHAT the two figures above are a projection of, because the number alone cannot
|
||||||
|
// carry that and a reader must never have to guess:
|
||||||
|
//
|
||||||
|
// pending — the fold of the book's decision FILES: the bank the NEXT `translate` will build, so the
|
||||||
|
// figures answer "what would a re-pass cost" for an edit that has been applied but not yet
|
||||||
|
// run. This is the ordinary answer.
|
||||||
|
// stored — the fold could not be built (a broken seed, a collision that would abort ReplaceBank),
|
||||||
|
// so the figures fall back to the glossary the LAST run stored. They are then a fact about
|
||||||
|
// the past, not a projection of the next run; the reason is on the WARN log.
|
||||||
|
// none — no stored row exists, so there is nothing already-billed to re-pay. Genuinely $0.
|
||||||
|
// failed — the projection itself errored. The figures are zero because they are UNKNOWN.
|
||||||
|
RebillBasis string `json:"rebill_basis"`
|
||||||
|
|
||||||
TotalUnits int `json:"total_units"`
|
TotalUnits int `json:"total_units"`
|
||||||
Done int `json:"done"`
|
Done int `json:"done"`
|
||||||
|
|
@ -492,17 +525,6 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
|
||||||
if len(editStages) > 0 {
|
if len(editStages) > 0 {
|
||||||
rep.Progress.Edit.Total = len(units)
|
rep.Progress.Edit.Total = len(units)
|
||||||
}
|
}
|
||||||
// The unsigned-bank exposure, same definition as the quality report's (a row with a rendering that is
|
|
||||||
// not approved). Read-only, like everything else here; a read failure says so rather than reporting 0.
|
|
||||||
if rows, gerr := r.Store.GlossaryForBook(r.Book.BookID); gerr != nil {
|
|
||||||
r.Log.WarnContext(ctx, "status: could not read the bank; the unsigned-term count is unknown, not zero", "err", gerr)
|
|
||||||
} else {
|
|
||||||
for _, e := range rows {
|
|
||||||
if e.Status != "approved" && strings.TrimSpace(e.Dst) != "" {
|
|
||||||
rep.UnsignedBankTerms++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
passports := map[int]*ChapterPassport{}
|
passports := map[int]*ChapterPassport{}
|
||||||
var chapterOrder []int
|
var chapterOrder []int
|
||||||
|
|
||||||
|
|
@ -631,33 +653,57 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
|
||||||
rep.Snapshot = s
|
rep.Snapshot = s
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !rep.SnapshotDrift && (len(draftSnaps) > 0 || len(editSnaps) > 0) {
|
// The bank BOTH projections below are computed against, materialized ONCE.
|
||||||
if err := r.projectStoredMemory(); err != nil {
|
//
|
||||||
// Do not stay silent: a projection failure used to be read silently as "no drift" (chain audit).
|
// It is hoisted out of the drift branch it used to sit inside, and that move is a fix rather than a
|
||||||
r.Log.WarnContext(ctx, "config-drift check failed; drift state unknown (reported as none)", "err", err)
|
// tidy-up: the re-bill projection is computed unconditionally further down, and on a book with
|
||||||
} else {
|
// snapshot drift this branch never ran — leaving r.memory nil, so `current(w)` in projectRebill
|
||||||
checkWave := func(snaps map[string]bool, w wave) {
|
// rendered the wave snapshots over an EMPTY bank (memoryVersion falls back to the hash of no rows,
|
||||||
if len(snaps) != 1 {
|
// snapshot.go), every stored row then differed from it, and status reported the WHOLE book as a
|
||||||
return
|
// re-payment. Materializing before either consumer removes the state where one of them runs without it.
|
||||||
}
|
memBasis, memWarn := r.foldMemoryForRead()
|
||||||
var stored string
|
if memWarn != nil {
|
||||||
for s := range snaps {
|
r.Log.WarnContext(ctx, "the bank fold refused, so the projections below are computed against the glossary the LAST run stored — they are a fact about the past, not a projection of the next run", "basis", memBasis, "err", memWarn)
|
||||||
stored = s
|
}
|
||||||
}
|
// The unsigned-bank exposure, same definition as the quality report's (a row with a rendering that is
|
||||||
cur, _, serr := r.snapshotIDForWave(w)
|
// not approved), counted off THE BANK THIS DOCUMENT JUST FOLDED.
|
||||||
if serr != nil {
|
//
|
||||||
r.Log.WarnContext(ctx, "config-drift check failed for a wave; drift state unknown", "err", serr)
|
// ⚠ It used to be read straight from the stored glossary, and after the fold landed that made one
|
||||||
return
|
// status document describe TWO banks: the money figures spoke about the bank the next run will build
|
||||||
}
|
// while this counter spoke about the bank the last run stored. An operator asking "is there anything
|
||||||
if cur != stored {
|
// to sign before I keep paying?" was told zero while the very next translate would inject an unsigned
|
||||||
rep.ConfigDrift = true
|
// proposal sitting in the auto-bank file and bill for the unit it changed. One document, one bank.
|
||||||
rep.CurrentSnapshot = cur
|
if r.bankRows == nil {
|
||||||
}
|
r.Log.WarnContext(ctx, "status: no bank could be materialized; the unsigned-term count is unknown, not zero")
|
||||||
|
} else {
|
||||||
|
for _, e := range r.bankRows {
|
||||||
|
if e.Status != "approved" && strings.TrimSpace(e.Dst) != "" {
|
||||||
|
rep.UnsignedBankTerms++
|
||||||
}
|
}
|
||||||
checkWave(draftSnaps, waveDraft)
|
|
||||||
checkWave(editSnaps, waveEdit)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if memBasis != RebillBasisFailed && !rep.SnapshotDrift && (len(draftSnaps) > 0 || len(editSnaps) > 0) {
|
||||||
|
checkWave := func(snaps map[string]bool, w wave) {
|
||||||
|
if len(snaps) != 1 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var stored string
|
||||||
|
for s := range snaps {
|
||||||
|
stored = s
|
||||||
|
}
|
||||||
|
cur, _, serr := r.snapshotIDForWave(w)
|
||||||
|
if serr != nil {
|
||||||
|
r.Log.WarnContext(ctx, "config-drift check failed for a wave; drift state unknown", "err", serr)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if cur != stored {
|
||||||
|
rep.ConfigDrift = true
|
||||||
|
rep.CurrentSnapshot = cur
|
||||||
|
}
|
||||||
|
}
|
||||||
|
checkWave(draftSnaps, waveDraft)
|
||||||
|
checkWave(editSnaps, waveEdit)
|
||||||
|
}
|
||||||
|
|
||||||
// One re-pricing read serves both money projections below (reprice.go): the re-payment amount and
|
// One re-pricing read serves both money projections below (reprice.go): the re-payment amount and
|
||||||
// projected_book_usd, which is also the base of the consent threshold — computing them from two
|
// projected_book_usd, which is also the base of the consent threshold — computing them from two
|
||||||
|
|
@ -670,15 +716,17 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
|
||||||
// not only under ConfigDrift — because SnapshotDrift (rows split across snapshots within one wave)
|
// not only under ConfigDrift — because SnapshotDrift (rows split across snapshots within one wave)
|
||||||
// re-bills too, and that is precisely the case the boolean pair leaves unpriced. $0 and read-only:
|
// re-bills too, and that is precisely the case the boolean pair leaves unpriced. $0 and read-only:
|
||||||
// projectRebill re-renders the wave snapshots and re-prices stored usage; it reaches no provider.
|
// projectRebill re-renders the wave snapshots and re-prices stored usage; it reaches no provider.
|
||||||
if len(statuses) > 0 {
|
var proj RebillProjection
|
||||||
if proj, perr := r.projectRebill(statuses, chunks, withText, rp); perr != nil {
|
var projErr error
|
||||||
|
if projectable(len(statuses) > 0, memBasis) {
|
||||||
|
proj, projErr = r.projectRebill(statuses, chunks, withText, rp)
|
||||||
|
if projErr != nil {
|
||||||
// Same discipline as the drift check above: a failed projection is reported, never
|
// Same discipline as the drift check above: a failed projection is reported, never
|
||||||
// silently rendered as "nothing to re-pay".
|
// silently rendered as "nothing to re-pay".
|
||||||
r.Log.WarnContext(ctx, "re-bill projection failed; the re-payment cost of the drift is unknown (reported as none)", "err", perr)
|
r.Log.WarnContext(ctx, "re-bill projection failed; the re-payment cost of the drift is unknown (reported as unknown, not as none)", "err", projErr)
|
||||||
} else {
|
|
||||||
rep.RebillUnits, rep.RebillUSD = proj.Rows, proj.USD
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
rep.RebillBasis, rep.RebillUnits, rep.RebillUSD, rep.RebillOutputUnits = rebillOutcome(len(statuses) > 0, memBasis, proj, projErr)
|
||||||
|
|
||||||
// Money.
|
// Money.
|
||||||
committed, reserved, err := r.Store.SpentUSD(r.Book.BookID)
|
committed, reserved, err := r.Store.SpentUSD(r.Book.BookID)
|
||||||
|
|
@ -730,16 +778,77 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
|
||||||
return rep, nil
|
return rep, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// projectStoredMemory materializes r.memory from the STORED glossary (read-only: no re-seed, no write) —
|
// projectable reports whether the re-payment projection is worth computing at all.
|
||||||
// the side effect the wave-snapshot projections need. A seed-FILE edit not yet re-run is NOT reflected here
|
//
|
||||||
// (the STORED glossary is projected, not the seed file) — that drift surfaces on the next translate's
|
// It is FALSE when no bank could be materialized, and skipping it then is the point rather than an
|
||||||
// re-seed, exactly as it does for `translate` itself.
|
// optimization. The projection compares every stored row's snapshot against what the CURRENT config
|
||||||
|
// renders — and with r.memory nil, memoryVersion falls back to the hash of an EMPTY bank (snapshot.go),
|
||||||
|
// so every row differs from it and the whole book comes back as a re-payment. Running it anyway would put
|
||||||
|
// the largest figure the field can hold beside a basis that says "unknown", and the CLI would print "the
|
||||||
|
// figures below are zero because they could not be computed" next to it.
|
||||||
|
func projectable(hasRows bool, memBasis string) bool {
|
||||||
|
return hasRows && memBasis != RebillBasisFailed
|
||||||
|
}
|
||||||
|
|
||||||
|
// rebillOutcome is the ONE decision about what the report says on the re-payment axis: the basis and the
|
||||||
|
// two figures, always together, so a branch cannot set one and forget the other.
|
||||||
|
//
|
||||||
|
// It exists as a function rather than as a switch inside Status because the invariant it carries is not
|
||||||
|
// reachable from a fixture: "failed" needs BOTH the fold and the stored materialization to refuse, which
|
||||||
|
// no healthy store will do. The first version of this logic left the basis at "failed" while a
|
||||||
|
// successfully-computed whole-book figure sat beside it — a basis contradicting its own numbers, which is
|
||||||
|
// worse than having no basis at all. Here the rule is a value, and a table test can hold it to it.
|
||||||
|
func rebillOutcome(hasRows bool, memBasis string, proj RebillProjection, projErr error) (basis string, units int, usd float64, outputUnits int) {
|
||||||
|
switch {
|
||||||
|
case !hasRows:
|
||||||
|
return RebillBasisNone, 0, 0, 0 // nothing already-billed exists, so nothing can be re-paid
|
||||||
|
case memBasis == RebillBasisFailed, projErr != nil:
|
||||||
|
// UNKNOWN, and zeroes that say so rather than figures that lie. EVERY figure, including the
|
||||||
|
// output-unit count: one number surviving here would be the same contradiction the basis exists to
|
||||||
|
// prevent, just in a different field.
|
||||||
|
return RebillBasisFailed, 0, 0, 0
|
||||||
|
default:
|
||||||
|
return memBasis, proj.Rows, proj.USD, proj.OutputUnits
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// projectStoredMemory materializes r.memory from the STORED glossary (read-only: no re-seed, no write).
|
||||||
|
//
|
||||||
|
// ⚠ IT IS NO LONGER THE READ PATH'S FIRST ANSWER — it is foldMemoryForRead's FALLBACK, and the demotion
|
||||||
|
// is backlog row 231 (errata 28.08-к). What it materializes is LAST run's fold, and its own comment used
|
||||||
|
// to justify that with "a seed-FILE edit not yet re-run is NOT reflected here … exactly as it does for
|
||||||
|
// `translate` itself". The second half was false: TranslateBook calls seedGlossary BEFORE
|
||||||
|
// checkRebillConsent (bookrun.go), so `translate` re-seeds from the FILES and its consent gate sees the
|
||||||
|
// edit — while every read-only surface said "nothing moved" for exactly the work the next run would bill
|
||||||
|
// for. Right after a `bank-apply`, which writes decision FILES and no store row, that is the whole of the
|
||||||
|
// blind window: the platform had nothing to show a buyer before charging them.
|
||||||
|
//
|
||||||
|
// ⚠ IT NOW MATERIALIZES BOTH BANKS, and that is a fix this pack's own test caught rather than a
|
||||||
|
// generalization. It used to call membank.Materialize (rows alone) and set r.memory only, leaving
|
||||||
|
// r.baseMemory nil — and the re-bill projection's re-pin branch renders the DRAFT wave against exactly
|
||||||
|
// that bank (rebill.go → renderedContentHashes), where repin.go skips every position when it is nil. The
|
||||||
|
// positions then had no reproducible hash, "cannot conclude" took the conservative branch, and a bank move
|
||||||
|
// that touched BASE rows made status report the whole draft wave as a re-payment — while `translate`,
|
||||||
|
// which has both banks, charged for a fraction of it. status.go's own promise that it "can never quote a
|
||||||
|
// different number than the one translate enforces" was false in precisely that state.
|
||||||
|
//
|
||||||
|
// It reads the voice and address rows for the same reason: the run materializes with them, and a read that
|
||||||
|
// materializes without them is a second definition of the same bank. They do not move the version hash
|
||||||
|
// while InjectVoice is false (membank.ComputeVersionScopedIn), so nothing re-pays for this.
|
||||||
func (r *Runner) projectStoredMemory() error {
|
func (r *Runner) projectStoredMemory() error {
|
||||||
rows, err := r.Store.GlossaryForBook(r.Book.BookID)
|
rows, err := r.Store.GlossaryForBook(r.Book.BookID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
r.memory = membank.Materialize(rows, r.Pipeline.Gates.Glossary.PostcheckGate)
|
voices, err := r.Store.VoiceProfilesForBook(r.Book.BookID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
pairs, err := r.Store.AddressPairsForBook(r.Book.BookID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
r.materializeBanks(rows, voices, pairs)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -911,7 +1020,9 @@ func (r *Runner) Redrive(ctx context.Context, sel RedriveSelector) (*RedriveSumm
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return summary, nil, fmt.Errorf("pipeline: redrive re-bill projection: %w", err)
|
return summary, nil, fmt.Errorf("pipeline: redrive re-bill projection: %w", err)
|
||||||
}
|
}
|
||||||
if err := r.checkRebillConsent(ctx, rebillChunks); err != nil {
|
// nil scope: `redrive` does not take --max-units (invocation.go refuses it there), so there is no
|
||||||
|
// volume ceiling to narrow this projection by and the book-wide figure is the honest one.
|
||||||
|
if err := r.checkRebillConsent(ctx, rebillChunks, nil); err != nil {
|
||||||
return summary, nil, err
|
return summary, nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -67,11 +67,33 @@ func TestStatusPricesTheDriftWithTheGatesOwnNumber(t *testing.T) {
|
||||||
}
|
}
|
||||||
|
|
||||||
// The same number, from the enforcement side: the gate's refusal must quote what status showed.
|
// The same number, from the enforcement side: the gate's refusal must quote what status showed.
|
||||||
err = r2.checkRebillConsent(ctx, chunksOf(t, r2))
|
err = r2.checkRebillConsent(ctx, chunksOf(t, r2), nil)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("the gate must refuse a fully-superseded book without consent")
|
t.Fatal("the gate must refuse a fully-superseded book without consent")
|
||||||
}
|
}
|
||||||
if want := fmt.Sprintf("~$%.6f", rep.RebillUSD); !strings.Contains(err.Error(), want) {
|
if want := fmt.Sprintf("~$%.6f", rep.RebillUSD); !strings.Contains(err.Error(), want) {
|
||||||
t.Errorf("the refusal does not quote the projection %s:\n%v", want, err)
|
t.Errorf("the refusal does not quote the projection %s:\n%v", want, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ⚠ AND THE SAME AGAIN UNDER A VOLUME CEILING, because that is where the two sides could drift apart
|
||||||
|
// and status.go's docstring promises out loud that they cannot: "status can never quote a different
|
||||||
|
// number than the one translate enforces". A bounded run re-pays only its own slice, but the question
|
||||||
|
// the THRESHOLD answers is about the book — so the figure the refusal quotes must still be the book's,
|
||||||
|
// which is the one status reports. Passing nil here (the shape this pin had while the pack was being
|
||||||
|
// built) tests only the unbounded case and would miss a divergence entirely.
|
||||||
|
// A scope that ADMITS the book's single unit — i.e. a real bounded purchase that genuinely re-pays.
|
||||||
|
// (A scope admitting nothing is a different case and is handled before the threshold: a run that
|
||||||
|
// re-pays nothing has no concrete spend to consent to.)
|
||||||
|
bounded := &volumeScope{
|
||||||
|
admitted: map[chunkKey]bool{{1, 0}: true},
|
||||||
|
leader: map[chunkKey]bool{{1, 0}: true},
|
||||||
|
stop: VolumeStop{MaxUnits: 1},
|
||||||
|
}
|
||||||
|
err = r2.checkRebillConsent(ctx, chunksOf(t, r2), bounded)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("a bounded run over a fully-superseded book must still meet the gate: a threshold a caller can shrink by splitting is not a threshold")
|
||||||
|
}
|
||||||
|
if want := fmt.Sprintf("~$%.6f", rep.RebillUSD); !strings.Contains(err.Error(), want) {
|
||||||
|
t.Errorf("under a volume ceiling the refusal stopped quoting the BOOK figure status reports (%s):\n%v", want, err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
581
backend/internal/pipeline/volume.go
Normal file
581
backend/internal/pipeline/volume.go
Normal file
|
|
@ -0,0 +1,581 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/chunk"
|
||||||
|
"textmachine/backend/internal/membank"
|
||||||
|
"textmachine/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// volume.go: the VOLUME ceiling — the run's second stop, beside the money one (D39.165 §1б, backlog row
|
||||||
|
// «потолок объёма»).
|
||||||
|
//
|
||||||
|
// WHY A SECOND CEILING AT ALL. The platform sells CHAPTERS and hands the engine a DOLLAR bound
|
||||||
|
// (--ceiling-usd), which is the only bound the engine had. Measured on real ledgers (D39.165 §1), a
|
||||||
|
// chapter costs $0.0115–$0.0190 and the constant sold against it is $0.03 — so a purchase of ten chapters
|
||||||
|
// hands over $0.30, and $0.30 buys sixteen to twenty-six. The knob said chapters and meant money. This
|
||||||
|
// gives the engine a bound in the unit it actually ships, so the seller's promise and the engine's stop
|
||||||
|
// are the same quantity.
|
||||||
|
//
|
||||||
|
// WHAT IT IS MEASURED IN, and why not the other two candidates. The unit is the OUTPUT UNIT — editUnit,
|
||||||
|
// the granularity `outputUnits` returns and `buildManifest` publishes as the manifest's units_total, which
|
||||||
|
// is where the platform's per-chapter unit count comes from. So "N units" means the same thing on both
|
||||||
|
// sides of the seam, and the platform's chapters→units conversion is EXACT rather than an estimate: the
|
||||||
|
// manifest is $0, needs no keys and exists before the first paid call.
|
||||||
|
// - NOT chunk×stage, which is the BILLING unit (the one the re-payment estimate counts in). Cutting the
|
||||||
|
// wave in the unit one PAYS in, while selling the unit one SHIPS in, reproduces the very defect above
|
||||||
|
// one layer down: a unit costs len(Members)·nDraftStages + nEditStages positions, so "10" would buy a
|
||||||
|
// different amount of book on a different pipeline shape.
|
||||||
|
// - NOT the chunk, which is an internal artefact of the cut: the buyer never sees it and the manifest
|
||||||
|
// does not count it.
|
||||||
|
//
|
||||||
|
// ⚠ The unit count DOES depend on the pipeline's shape — an editor pipeline groups draft chunks into
|
||||||
|
// coarse edit units, a draft-only pipeline ships one unit per draft chunk (outputUnits). That is a
|
||||||
|
// property of the configuration, not a branch per language pair, and it is safe because the manifest the
|
||||||
|
// platform counted and the run it bounds come from the SAME deployment. It would stop being safe if a
|
||||||
|
// ceiling were carried across two deployments with different pipelines.
|
||||||
|
//
|
||||||
|
// HOW REPINS AND RETRIES COUNT — the question this design has to answer out loud.
|
||||||
|
// - A RETRY and an ESCALATION hop do NOT count. They live inside one unit (the attempt loop in
|
||||||
|
// stagerun.go writes ONE chunk_status row per chunk×stage however many attempts it took), and the
|
||||||
|
// volume ceiling bounds DELIVERY while the money ceiling bounds SPEND. The measured quality tail
|
||||||
|
// (retries+escalations swinging 1.5%→23% between runs) is a fact about money and already has its own
|
||||||
|
// bound; if it burned volume, a buyer of ten chapters would receive eight because two of them
|
||||||
|
// stuttered — paying in BOOK for the engine's own tail, which is the substitution this pack exists to
|
||||||
|
// remove.
|
||||||
|
// - A RE-PIN and a $0 RESUME do NOT count. A re-pass that re-pins five hundred units for $0 (rebill.go)
|
||||||
|
// would otherwise exhaust the ceiling having delivered nothing.
|
||||||
|
//
|
||||||
|
// So: THE CEILING COUNTS OUTPUT UNITS FOR WHICH THIS RUN WILL MAKE AT LEAST ONE PROVIDER CALL. Units that
|
||||||
|
// resolve for free are admitted regardless — they cost nothing, and holding them back would leave the
|
||||||
|
// book's snapshots stale for no saving.
|
||||||
|
//
|
||||||
|
// AND IT COUNTS THEM IN TWO KINDS, which is the difference between a report and a true report. A paying
|
||||||
|
// unit is either book that did not exist (DELIVERY) or book that existed and is being made again under a
|
||||||
|
// moved snapshot (REWORK). unitClass below is that distinction, and VolumeStop carries it all the way to
|
||||||
|
// the operator: a purchase spent entirely on rework delivers no chapter, and neither the counters nor the
|
||||||
|
// line the CLI prints may let that read as a delivery. The remainder splits the same way — units never
|
||||||
|
// delivered are the only ones anybody may be invited to buy, while delivered-but-not-re-made units are
|
||||||
|
// unrefreshed, not unbought.
|
||||||
|
//
|
||||||
|
// ⚠ ON THE PREDICATE'S RELATION TO projectRebill: the free/paying test here is runStage's, not the
|
||||||
|
// re-payment estimate's, and they are NOT the same test. projectRebill deliberately ignores the CONTENT
|
||||||
|
// axis (its own doc comment says so: a source edit moves content_hash but not the snapshot, and is
|
||||||
|
// invisible there). This one checks the rendered content hash on every row, so a source edit correctly
|
||||||
|
// makes a unit paying here while the estimate would call it free. The difference is in the safe
|
||||||
|
// direction — the ceiling over-counts cost rather than under-counting it — and it is why this predicate
|
||||||
|
// is written against the executor rather than borrowed from the projection.
|
||||||
|
//
|
||||||
|
// CHECKED BEFORE, NOT AFTER. The scope is computed ONCE, before either wave starts, and the waves simply
|
||||||
|
// do not begin an item outside it. That is structural rather than a guard: there is no moment at which a
|
||||||
|
// unit past the ceiling has started, so the ceiling cannot be overshot by one call the way a post-hoc
|
||||||
|
// token check always is (LiteLLM's known behaviour, named in D39.165 §1б). It is also why the admission
|
||||||
|
// is not a counter the wave workers decrement — the waves are parallel, and a worker refused a slot would
|
||||||
|
// have to abandon its item permanently while a slot freed by a $0 re-pin went to nobody.
|
||||||
|
//
|
||||||
|
// THE STOP IS A COMPLETION, NOT A HALT — ratified by the orchestrator with this pack and NOT re-decided
|
||||||
|
// here. A money ceiling stops the run in the MIDDLE of work it meant to do: that is exit 4, the platform
|
||||||
|
// records `paused`, and the remedy is more money. A volume ceiling means the run did exactly what was
|
||||||
|
// bought: exit 0, the frozen exit-code dictionary (cmd/tmctl/main.go) is not touched at all, and the
|
||||||
|
// platform records the run `ready` — which does NOT declare the book finished, because the book's progress
|
||||||
|
// travels as unit events, so the next purchase starts a new run normally. What DOES have to be said is
|
||||||
|
// WHICH ceiling stopped it (OpenHands' iteration limit is silent, and that is a standing complaint named
|
||||||
|
// in the same note): that lives in the run's result, its log line and the CLI's rendering, not in a new
|
||||||
|
// exit code and not in a new word of the event stream's outcome vocabulary.
|
||||||
|
|
||||||
|
// VolumeStop is a run's report that it stopped on the volume ceiling rather than on the end of the book.
|
||||||
|
// It is a RESULT, not an error: the run did what it was granted, so it travels on BookResult and never
|
||||||
|
// through the error path, where the exit-code mapper would have to invent a word for it.
|
||||||
|
//
|
||||||
|
// ⚠ IT COUNTS PAID UNITS IN TWO KINDS, and the split is a MODEL rather than a nicety. A paying unit is
|
||||||
|
// either book that did not exist before (DELIVERY) or book that existed and is being made again under a
|
||||||
|
// moved snapshot (REWORK), and the two are not the same product: the first is what "buy ten chapters"
|
||||||
|
// means, the second is what a re-pass means. The first version of this struct had one counter for both,
|
||||||
|
// and every consequence of that was a lie the operator could act on — a purchase that went entirely into
|
||||||
|
// re-translating the opening chapters reported ten units bought and exited 0 like a delivery, while the
|
||||||
|
// remaining count invited buying chapters the reader already owned. A struct that cannot tell the two
|
||||||
|
// apart cannot report either one honestly, so it carries both.
|
||||||
|
type VolumeStop struct {
|
||||||
|
// MaxUnits is the ceiling that was in force.
|
||||||
|
MaxUnits int
|
||||||
|
// Delivered is paying units that had never been completed before — NEW book, the thing a purchase of
|
||||||
|
// chapters is actually for.
|
||||||
|
Delivered int
|
||||||
|
// Reworked is paying units that WERE already completed and are being paid for a second time because
|
||||||
|
// their snapshot moved. Real work and a real charge, but not new book: a purchase made entirely of
|
||||||
|
// these advances the reader not at all.
|
||||||
|
Reworked int
|
||||||
|
// Flagged is paid units that resolved FLAGGED and shipped no text. They cost money and delivered
|
||||||
|
// nothing, and calling them delivered is the lie this counter exists to stop: a purchase of two units
|
||||||
|
// one of which flagged used to print "2 NEW unit(s) delivered" over a single readable unit.
|
||||||
|
//
|
||||||
|
// ⚠ It is filled AFTER the waves, from what actually happened, not from the plan. Everything else on
|
||||||
|
// this struct is a plan-time decision about admission — correct for bounding money, and unable on its
|
||||||
|
// own to know whether the work it authorised produced a chapter.
|
||||||
|
Flagged int
|
||||||
|
// Free is units that rode along at $0 — resumed or re-pinned. Reported because otherwise the operator
|
||||||
|
// sees a run that touched far more units than it was granted and cannot tell why.
|
||||||
|
Free int
|
||||||
|
// LeftFresh is undelivered units still in the book. THIS is what "there is more to buy" honestly
|
||||||
|
// means, and it is the only one of the two remainders a buyer should ever be invited to purchase.
|
||||||
|
LeftFresh int
|
||||||
|
// LeftRework is completed units still carrying a superseded snapshot. They are "unrefreshed", not
|
||||||
|
// "unbought", and presenting them as stock for sale is how re-payment becomes a product.
|
||||||
|
LeftRework int
|
||||||
|
}
|
||||||
|
|
||||||
|
// Paid is every unit this run was charged for, of either kind.
|
||||||
|
func (v VolumeStop) Paid() int { return v.Delivered + v.Reworked }
|
||||||
|
|
||||||
|
// reconcile corrects the plan-time counters against what the run actually produced.
|
||||||
|
//
|
||||||
|
// Admission is a decision made BEFORE the work, and it has to be — that is what bounds the money. But a
|
||||||
|
// decision to pay for a unit is not a fact about a chapter existing: a unit can be paid for and come back
|
||||||
|
// flagged with nothing shippable. Reporting the plan as though it were the outcome is how "2 NEW unit(s)
|
||||||
|
// delivered" ended up printed over one readable unit. So the counters are trued up here, from the
|
||||||
|
// outcomes, before anyone reads them.
|
||||||
|
func (s *volumeScope) reconcile(outcomes []ChunkOutcome) {
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, oc := range outcomes {
|
||||||
|
if oc.Disposition != DispFlagged || oc.FinalText != "" {
|
||||||
|
continue // it shipped something readable; the plan's word for it stands
|
||||||
|
}
|
||||||
|
switch {
|
||||||
|
case s.stop.Delivered > 0 && s.class[chunkKey{oc.Chapter, oc.ChunkIdx}] == unitFresh:
|
||||||
|
s.stop.Delivered--
|
||||||
|
case s.stop.Reworked > 0:
|
||||||
|
s.stop.Reworked--
|
||||||
|
case s.stop.Delivered > 0:
|
||||||
|
s.stop.Delivered--
|
||||||
|
default:
|
||||||
|
continue // it was never counted as paid (a free unit that flagged on resume); nothing to move
|
||||||
|
}
|
||||||
|
s.stop.Flagged++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Left is every unit the ceiling held back, of either kind.
|
||||||
|
func (v VolumeStop) Left() int { return v.LeftFresh + v.LeftRework }
|
||||||
|
|
||||||
|
func (v VolumeStop) String() string {
|
||||||
|
// ⚠ EVERY COUNT HERE IS IN OUTPUT UNITS, AND THE WORDS MUST SAY SO. An earlier version of this line
|
||||||
|
// read "%d NEW chapter(s) delivered" over a counter that increments per editUnit — and a chapter is
|
||||||
|
// more than one unit whenever the cut closes a unit at the edit ceiling, and is many units in a
|
||||||
|
// draft-only pipeline. That is the unit↔chapter substitution this whole ceiling exists to remove,
|
||||||
|
// committed by the ceiling's own report. Chapters are the PLATFORM's word; it converts them through
|
||||||
|
// the manifest. The engine speaks units and nothing else.
|
||||||
|
s := fmt.Sprintf("stopped on the VOLUME ceiling (--max-units %d), not on money and not at the end of the book: %d paying output unit(s) — %d NEW unit(s) delivered, %d already-delivered unit(s) re-made under a moved snapshot",
|
||||||
|
v.MaxUnits, v.Paid()+v.Flagged, v.Delivered, v.Reworked)
|
||||||
|
if v.Flagged > 0 {
|
||||||
|
s += fmt.Sprintf(", and %d PAID FOR BUT FLAGGED — money spent, no readable text produced (buying more will not fix them; `tmctl redrive` re-attacks a flag)", v.Flagged)
|
||||||
|
}
|
||||||
|
if v.Free > 0 {
|
||||||
|
s += fmt.Sprintf("; %d rode along at $0 (resumed or re-pinned)", v.Free)
|
||||||
|
}
|
||||||
|
s += fmt.Sprintf(". Still in the book: %d unit(s) NEVER delivered", v.LeftFresh)
|
||||||
|
if v.LeftRework > 0 {
|
||||||
|
s += fmt.Sprintf(" and %d already delivered but not yet re-made (those are unrefreshed, NOT unbought)", v.LeftRework)
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// volumeScope is the run's admitted set of output units, decided before any work begins. A nil scope
|
||||||
|
// means no volume ceiling is in force and nothing anywhere changes behaviour.
|
||||||
|
type volumeScope struct {
|
||||||
|
admitted map[chunkKey]bool // unit leader key → admitted
|
||||||
|
leader map[chunkKey]bool // member chunk key → its unit is admitted (flattened for the draft wave)
|
||||||
|
stop VolumeStop
|
||||||
|
// class is what each unit was judged to be at PLAN time, kept because the edit wave has to re-judge
|
||||||
|
// the free ones (see rescopeEditWave).
|
||||||
|
class map[chunkKey]unitClass
|
||||||
|
// editSnapshot is the edit-wave snapshot the classification above was made AGAINST. The run can move
|
||||||
|
// it after planning — the bank-mining stop re-seeds mid-run — so this is what rescopeEditWave compares
|
||||||
|
// with to know whether the plan is still standing on the ground it was made on.
|
||||||
|
editSnapshot string
|
||||||
|
// editBlocked are units whose EDIT must not run even though they were admitted: they were judged free,
|
||||||
|
// the bank then moved under them, and no grant was left to pay for what they turned out to cost.
|
||||||
|
editBlocked map[chunkKey]bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// allows reports whether a DRAFT chunk may be started: it may when the unit it belongs to was admitted.
|
||||||
|
func (s *volumeScope) allows(ch chunk.Chunk) bool {
|
||||||
|
if s == nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return s.leader[chunkKey{ch.Chapter, ch.ChunkIdx}]
|
||||||
|
}
|
||||||
|
|
||||||
|
// allowsUnit reports whether an EDIT unit may be started.
|
||||||
|
func (s *volumeScope) allowsUnit(u editUnit) bool {
|
||||||
|
if s == nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
key := chunkKey{u.Chapter, u.FirstChunkIdx}
|
||||||
|
return s.admitted[key] && !s.editBlocked[key]
|
||||||
|
}
|
||||||
|
|
||||||
|
// admittedStatuses narrows stored rows to the units this run is actually going to work on, so a money
|
||||||
|
// projection over them describes THIS run rather than the whole book. A nil scope is the identity, which
|
||||||
|
// is what keeps every un-bounded run's figures byte-identical to what they were before the ceiling
|
||||||
|
// existed. An edit row lives at its unit's LEADER chunk, and a leader is one of the unit's members, so
|
||||||
|
// the member map covers both waves' rows.
|
||||||
|
func (s *volumeScope) admittedStatuses(in []store.ChunkStatus) []store.ChunkStatus {
|
||||||
|
if s == nil {
|
||||||
|
return in
|
||||||
|
}
|
||||||
|
out := in[:0:0]
|
||||||
|
for _, cs := range in {
|
||||||
|
if s.leader[chunkKey{cs.Chapter, cs.ChunkIdx}] {
|
||||||
|
out = append(out, cs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// bound reports whether the ceiling actually held work back. A run granted more than the book had left is
|
||||||
|
// an ordinary complete run and must not be reported as a volume stop — otherwise every generously-bounded
|
||||||
|
// run would end claiming it had been cut short.
|
||||||
|
func (s *volumeScope) bound() bool { return s != nil && s.stop.Left() > 0 }
|
||||||
|
|
||||||
|
// planVolume decides, before any wave starts, which output units this run may work on.
|
||||||
|
//
|
||||||
|
// It is $0 by construction: store reads, string rendering and hashing, no provider anywhere. It is also
|
||||||
|
// skipped entirely when no ceiling is set, so a run without --max-units pays nothing for this existing —
|
||||||
|
// not even the content-hash reproduction, which is the expensive half.
|
||||||
|
func (r *Runner) planVolume(ctx context.Context, chunks []chunk.Chunk, stickySel []membank.Selection) (*volumeScope, error) {
|
||||||
|
if r.MaxUnits <= 0 {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
// ⚠ REFUSED on a pipeline whose SHIPPING stage is a translator while editor stages also exist, because
|
||||||
|
// in that one shape the two granularities this ceiling straddles stop being the same object. outputUnits
|
||||||
|
// keys off finalStageWave and returns a singleton per DRAFT CHUNK there, while the edit wave still
|
||||||
|
// groups those chunks into buildEditUnits — so a ceiling granted in singletons would admit one member's
|
||||||
|
// chunk and then let the edit wave run the whole multi-member unit over drafts that were never made,
|
||||||
|
// paying full price for half-empty input. config validates stage ROLES but imposes no order, so the
|
||||||
|
// shape loads; refusing loudly is the only answer that neither miscounts a purchase nor silently
|
||||||
|
// changes what a unit means. (It is also the shape in which the platform's chapters→units conversion
|
||||||
|
// would stop being exact, which is the whole basis of measuring in units at all.)
|
||||||
|
if len(r.waveStagesIndexed(waveEdit)) > 0 && r.finalStageWave() == waveDraft {
|
||||||
|
return nil, fmt.Errorf("pipeline: --max-units cannot bound this pipeline: its last stage is a translator while editor stages exist, so the shipping unit is the draft CHUNK while the edit wave still works in grouped edit UNITS — a ceiling counted in one would admit work measured in the other. Put the shipping stage last (an editor/other role), or run this pipeline without a volume ceiling")
|
||||||
|
}
|
||||||
|
// ⚠ THE ONE COMPOSITION THIS CEILING MAKES WORSE, said out loud where an operator will see it.
|
||||||
|
//
|
||||||
|
// A bounded purchase drafts only part of the book, and the bank-mining stop consolidates over the
|
||||||
|
// chunks drafted SO FAR — so the auto-bank it writes grows with every purchase. The next purchase folds
|
||||||
|
// that larger auto-bank into the ENRICHED bank, which moves the edit-wave snapshot, and the edit jobs
|
||||||
|
// the PREVIOUS purchase created are then pinned to a superseded one: without --resnapshot the run stops
|
||||||
|
// loudly, and with it the units whose injected bytes the new terms changed are re-translated and re-paid.
|
||||||
|
//
|
||||||
|
// None of that is new machinery — it is the mine D39.165 §3 named and errata 28.08-и narrowed to "an
|
||||||
|
// edit made AFTER edit jobs exist". What IS new is the frequency: without a volume ceiling that state
|
||||||
|
// needs an interrupted run (which usually has no edit jobs yet), while WITH one it is the ordinary
|
||||||
|
// shape of selling a book a few chapters at a time. Warned rather than refused, because the run is
|
||||||
|
// still correct — it re-pays through the consent gate like any other re-payment — and because refusing
|
||||||
|
// would take the ceiling away from exactly the books that most need selling in parts.
|
||||||
|
if r.pack != nil && r.Pipeline.Mining.ContrastPath != "" {
|
||||||
|
r.Log.WarnContext(ctx, "a VOLUME ceiling on a book that MINES its bank: each purchase drafts more, so the bank-mining stop writes a larger auto-bank, and the next purchase moves the edit-wave snapshot the previous purchase's edit jobs are pinned to. Expect that run to need --resnapshot and to re-pay the units the new terms actually touch (the re-payment consent gate still bounds it)",
|
||||||
|
"book", r.Book.BookID, "max_units", r.MaxUnits)
|
||||||
|
}
|
||||||
|
units := r.outputUnits(chunks)
|
||||||
|
statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("pipeline: read chunk_status for the volume ceiling: %w", err)
|
||||||
|
}
|
||||||
|
class, curEditAtPlan, err := r.classifyUnits(units, chunks, statuses, stickySel)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
s := &volumeScope{
|
||||||
|
admitted: make(map[chunkKey]bool, len(units)),
|
||||||
|
leader: make(map[chunkKey]bool, len(chunks)),
|
||||||
|
stop: VolumeStop{MaxUnits: r.MaxUnits},
|
||||||
|
class: class,
|
||||||
|
editSnapshot: curEditAtPlan,
|
||||||
|
editBlocked: map[chunkKey]bool{},
|
||||||
|
}
|
||||||
|
admit := func(u editUnit) {
|
||||||
|
key := chunkKey{u.Chapter, u.FirstChunkIdx}
|
||||||
|
s.admitted[key] = true
|
||||||
|
for _, m := range u.Members {
|
||||||
|
s.leader[chunkKey{m.Chapter, m.ChunkIdx}] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// ⚠ A MISSING KEY READS AS unitFresh, and that is deliberate rather than incidental: classifyUnits
|
||||||
|
// returns an EMPTY map for a book with no stored rows, so the zero value has to be the class such a
|
||||||
|
// book's units actually are. unitFresh is iota 0 for exactly this reason — do not reorder the enum.
|
||||||
|
classOf := func(u editUnit) unitClass { return class[chunkKey{u.Chapter, u.FirstChunkIdx}] }
|
||||||
|
|
||||||
|
// Free units first and unconditionally: they spend nothing, so holding them back would only leave
|
||||||
|
// their rows pinned to a superseded snapshot for no saving at all.
|
||||||
|
for _, u := range units {
|
||||||
|
if classOf(u) == unitFree {
|
||||||
|
s.stop.Free++
|
||||||
|
admit(u)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ⚠ DELIVERY BEFORE REWORK, and this ORDER is the answer to "what did the buyer actually get".
|
||||||
|
//
|
||||||
|
// Both kinds are paying work, so a single pass in book order admits whichever comes first — and after
|
||||||
|
// any bank edit the already-delivered units ARE the early ones. Measured consequence, on a fixture
|
||||||
|
// where one chapter is one unit: a purchase of two units on a five-unit book with two delivered
|
||||||
|
// re-made units 1 and 2 and never started unit 3, reporting Delivered=0, Reworked=2, LeftFresh=3. The
|
||||||
|
// buyer paid for two units, received none, and three units that had never been translated at all sat
|
||||||
|
// untouched. A ceiling sold as "the seller's promise and the engine's stop are the same quantity"
|
||||||
|
// cannot behave that way.
|
||||||
|
//
|
||||||
|
// So the grant goes to NEW book while any is left, and only then to re-making. The reason it is safe to
|
||||||
|
// serve units out of book order is that nothing in a unit's rendered bytes depends on which other units
|
||||||
|
// ran: the sticky-window selection is precomputed over the WHOLE book before either wave
|
||||||
|
// (precomputeSticky), so running unit 3 before unit 1 renders exactly what running them in order would.
|
||||||
|
//
|
||||||
|
// ⚠ THE TRADE IS NAMED, not hidden: a purchase whose PURPOSE was a re-pass will now spend its grant on
|
||||||
|
// undelivered units if the book still has any. That is the right default — an undelivered unit is
|
||||||
|
// worth more to a reader than a re-made one, and the re-payment consent gate bounds rework by money in
|
||||||
|
// its own right — but it IS a default, and a caller who needs "re-make only" needs a flag that says so.
|
||||||
|
// Reported to the orchestrator as a chosen default rather than assumed.
|
||||||
|
for _, pass := range []unitClass{unitFresh, unitRework} {
|
||||||
|
for _, u := range units {
|
||||||
|
if classOf(u) != pass {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if s.stop.Paid() >= r.MaxUnits {
|
||||||
|
if pass == unitFresh {
|
||||||
|
s.stop.LeftFresh++
|
||||||
|
} else {
|
||||||
|
s.stop.LeftRework++
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if pass == unitFresh {
|
||||||
|
s.stop.Delivered++
|
||||||
|
} else {
|
||||||
|
s.stop.Reworked++
|
||||||
|
}
|
||||||
|
admit(u)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return s, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// rescopeEditWave re-judges the FREE units against the snapshot the edit wave is actually about to run
|
||||||
|
// under, and makes them pay for a grant slot if they turn out to cost money.
|
||||||
|
//
|
||||||
|
// ⛔ WITHOUT THIS THE CEILING DOES NOT HOLD, and the report lies while it fails. planVolume classifies
|
||||||
|
// before the draft wave, but the bank-mining stop sits between the two waves and RE-SEEDS the bank
|
||||||
|
// mid-run (mining.go, the auto-continue branch), after which waverun recomputes the edit-wave snapshot.
|
||||||
|
// Every unit admitted as FREE — and free units are admitted OUTSIDE the grant, because free work costs
|
||||||
|
// nothing — was judged against a snapshot the run itself then replaced. Any of them whose injected bytes
|
||||||
|
// the new bank changes gets a fresh PAID editor call that no grant ever authorised, and the stop line
|
||||||
|
// calls it "rode along at $0". Found by the acceptance hunter, who measured four provider calls under a
|
||||||
|
// grant of one and $0.007280 of spend reported as free.
|
||||||
|
//
|
||||||
|
// The fix is not a guard but a re-plan: the free/paying question is asked again, against the snapshot
|
||||||
|
// that is now real, at the last moment before the edit wave begins — so "check before the unit" still
|
||||||
|
// holds. A unit that has become paying takes a slot if one is left; if none is, its edit does not run at
|
||||||
|
// all and it is reported as an already-delivered unit still awaiting its re-make, which is what it is.
|
||||||
|
//
|
||||||
|
// It is a no-op on the ordinary path: with no mid-run re-seed the snapshot is unchanged and every free
|
||||||
|
// unit stays free, so a book that does not mine pays nothing for this existing.
|
||||||
|
func (r *Runner) rescopeEditWave(ctx context.Context, s *volumeScope, units []editUnit, chunks []chunk.Chunk, stickySel []membank.Selection, editSnapshot string) error {
|
||||||
|
if s == nil || editSnapshot == s.editSnapshot {
|
||||||
|
return nil // the ground the plan was made on is still the ground the wave runs on
|
||||||
|
}
|
||||||
|
statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("pipeline: re-read chunk_status for the volume ceiling's edit-wave re-plan: %w", err)
|
||||||
|
}
|
||||||
|
draftStages, editStages := r.waveStagesIndexed(waveDraft), r.waveStagesIndexed(waveEdit)
|
||||||
|
draftNames, editNames := stageNameSet(draftStages), stageNameSet(editStages)
|
||||||
|
curDraft, _, err := r.snapshotIDForWave(waveDraft)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("pipeline: render the draft-wave snapshot for the edit-wave re-plan: %w", err)
|
||||||
|
}
|
||||||
|
// The memo was dropped by the re-seed that moved the snapshot (materializeBanks clears it), so this
|
||||||
|
// renders against the bank the edit wave will actually use.
|
||||||
|
hashes := r.cachedRenderedContentHashes(chunks, stickySel)
|
||||||
|
byChunk := map[chunkKey][]store.ChunkStatus{}
|
||||||
|
for _, cs := range statuses {
|
||||||
|
byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}] = append(byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}], cs)
|
||||||
|
}
|
||||||
|
|
||||||
|
moved, paid, blocked := 0, 0, 0
|
||||||
|
for _, u := range units {
|
||||||
|
key := chunkKey{u.Chapter, u.FirstChunkIdx}
|
||||||
|
if !s.admitted[key] || s.class[key] != unitFree {
|
||||||
|
continue // only the units admitted WITHOUT a slot can be wrong about being free
|
||||||
|
}
|
||||||
|
rows := unitRows(u, byChunk)
|
||||||
|
if r.rowsResumeFree(rows, draftNames, editNames, curDraft, editSnapshot, hashes) {
|
||||||
|
continue // still free under the snapshot that is now real
|
||||||
|
}
|
||||||
|
moved++
|
||||||
|
s.stop.Free--
|
||||||
|
if s.stop.Paid() < s.stop.MaxUnits {
|
||||||
|
// It costs money now, and there is grant left to pay for it: it becomes what it is — an
|
||||||
|
// already-delivered unit being re-made.
|
||||||
|
s.stop.Reworked++
|
||||||
|
s.class[key] = unitRework
|
||||||
|
paid++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
// No grant left. Its edit does not run; it stays delivered and stale, which is LeftRework.
|
||||||
|
s.editBlocked[key] = true
|
||||||
|
s.stop.LeftRework++
|
||||||
|
s.class[key] = unitRework
|
||||||
|
blocked++
|
||||||
|
}
|
||||||
|
if moved > 0 {
|
||||||
|
r.Log.WarnContext(ctx, "the bank moved between planning and the edit wave, so units judged FREE are no longer free: they have been re-judged against the snapshot the edit wave actually uses",
|
||||||
|
"book", r.Book.BookID, "no_longer_free", moved, "took_a_grant_slot", paid, "held_back_for_want_of_grant", blocked,
|
||||||
|
"planned_against", s.editSnapshot[:12], "running_under", editSnapshot[:12])
|
||||||
|
}
|
||||||
|
s.editSnapshot = editSnapshot
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// unitClass is what this run would DO to an output unit, which is the distinction the ceiling both
|
||||||
|
// admits and reports on.
|
||||||
|
type unitClass int
|
||||||
|
|
||||||
|
const (
|
||||||
|
// unitFresh — never completed. Paying for it DELIVERS a chapter that did not exist.
|
||||||
|
unitFresh unitClass = iota
|
||||||
|
// unitFree — fully recorded and every row resumes or re-pins at $0. Costs nothing, so it rides along
|
||||||
|
// regardless of the ceiling.
|
||||||
|
unitFree
|
||||||
|
// unitRework — fully recorded, but at least one row will be paid for again under a moved snapshot.
|
||||||
|
// Real work and a real charge, and NOT new book: it replaces a chapter the reader already has.
|
||||||
|
unitRework
|
||||||
|
)
|
||||||
|
|
||||||
|
// classifyUnits sorts every output unit into the three things this run can do to it.
|
||||||
|
//
|
||||||
|
// The free/paying predicate is deliberately the one the run itself applies, read off runStage's resume
|
||||||
|
// fast-path: a stored row is served for free when its rendered content hash is unchanged AND its snapshot
|
||||||
|
// either matches the current one or is a re-pinnable bank-only move. A unit is free when every one of its
|
||||||
|
// rows is, and when every position it will execute already HAS a row — a half-done unit resumes what it
|
||||||
|
// has and pays for the rest.
|
||||||
|
//
|
||||||
|
// The fresh/rework split falls straight out of the same completeness test, which is why it costs nothing
|
||||||
|
// to make: a unit that was never fully recorded has never been delivered, so paying for it is delivery; a
|
||||||
|
// unit that WAS fully recorded has been delivered once already, so paying for it again is rework.
|
||||||
|
//
|
||||||
|
// Every uncertainty resolves to "this unit will cost", never to "this unit is free". That direction is
|
||||||
|
// forced: mistaking a paying unit for a free one lets the run spend past the ceiling, which is the defect
|
||||||
|
// the ceiling exists to prevent, while the opposite merely makes a run stop one unit early.
|
||||||
|
func (r *Runner) classifyUnits(units []editUnit, chunks []chunk.Chunk, statuses []store.ChunkStatus, stickySel []membank.Selection) (map[chunkKey]unitClass, string, error) {
|
||||||
|
class := map[chunkKey]unitClass{}
|
||||||
|
if len(statuses) == 0 {
|
||||||
|
return class, "", nil // a book nothing has run is entirely fresh; skip the rendering entirely
|
||||||
|
}
|
||||||
|
draftStages, editStages := r.waveStagesIndexed(waveDraft), r.waveStagesIndexed(waveEdit)
|
||||||
|
draftNames, editNames := stageNameSet(draftStages), stageNameSet(editStages)
|
||||||
|
|
||||||
|
curDraft, _, err := r.snapshotIDForWave(waveDraft)
|
||||||
|
if err != nil {
|
||||||
|
return nil, "", fmt.Errorf("pipeline: render the draft-wave snapshot for the volume ceiling: %w", err)
|
||||||
|
}
|
||||||
|
curEdit := curDraft
|
||||||
|
if len(editStages) > 0 {
|
||||||
|
if curEdit, _, err = r.snapshotIDForWave(waveEdit); err != nil {
|
||||||
|
return nil, "", fmt.Errorf("pipeline: render the edit-wave snapshot for the volume ceiling: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// The same reproduction the re-payment estimate uses (repin.go): what the run WOULD render for every
|
||||||
|
// live position. A position whose inputs cannot be reproduced is simply absent, and an absent hash is
|
||||||
|
// read below as "cannot conclude", i.e. the paying answer.
|
||||||
|
hashes := r.cachedRenderedContentHashes(chunks, stickySel)
|
||||||
|
|
||||||
|
byChunk := map[chunkKey][]store.ChunkStatus{}
|
||||||
|
for _, cs := range statuses {
|
||||||
|
byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}] = append(byChunk[chunkKey{cs.Chapter, cs.ChunkIdx}], cs)
|
||||||
|
}
|
||||||
|
for _, u := range units {
|
||||||
|
rows := unitRows(u, byChunk)
|
||||||
|
key := chunkKey{u.Chapter, u.FirstChunkIdx}
|
||||||
|
if !unitFullyRecorded(u, rows, draftStages, editStages) {
|
||||||
|
class[key] = unitFresh // a position with no row is a position this run will call for
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if r.rowsResumeFree(rows, draftNames, editNames, curDraft, curEdit, hashes) {
|
||||||
|
class[key] = unitFree
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
class[key] = unitRework
|
||||||
|
}
|
||||||
|
return class, curEdit, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// unitFullyRecorded reports whether EVERY position this run would execute for the unit already has a
|
||||||
|
// stored row — the completeness half of "this unit costs nothing".
|
||||||
|
//
|
||||||
|
// ⚠ IT DELIBERATELY DOES NOT USE resolveChunkState, and the difference is money. That resolver answers
|
||||||
|
// "what does a reader call this unit", and it returns ChunkFlagged the moment ANY row is flagged, before
|
||||||
|
// it ever tests whether the expected positions are all present. That is right for a progress projection
|
||||||
|
// and wrong here: a unit whose draft flagged for one member and whose EDIT row does not exist yet — the
|
||||||
|
// state every book is left in by a bank-signing stop, a Ctrl-C, or any abort in the edit wave — would be
|
||||||
|
// read as terminal, judged free, admitted without consuming a grant, and then charged for a full editor
|
||||||
|
// call. The run would pay for more units than were bought and, with Deferred still 0, would not even
|
||||||
|
// report a volume stop. Found by this pack's own adversarial pass, reproduced before it was fixed.
|
||||||
|
//
|
||||||
|
// Positions are counted, not dispositions: a completed unit always leaves a row for every position it
|
||||||
|
// has — ok, flagged, or the `skipped` rows recordSkippedStages writes downstream of a flag, including the
|
||||||
|
// all-members-flagged case where the editor never runs. So full coverage IS terminality, and it is the
|
||||||
|
// property that actually predicts "no provider call".
|
||||||
|
func unitFullyRecorded(u editUnit, rows []store.ChunkStatus, draftStages, editStages []wavedStage) bool {
|
||||||
|
type pos struct {
|
||||||
|
chapter, chunkIdx int
|
||||||
|
stage string
|
||||||
|
}
|
||||||
|
have := make(map[pos]bool, len(rows))
|
||||||
|
for _, cs := range rows {
|
||||||
|
have[pos{cs.Chapter, cs.ChunkIdx, cs.Stage}] = true
|
||||||
|
}
|
||||||
|
for _, m := range u.Members {
|
||||||
|
for _, ws := range draftStages {
|
||||||
|
if !have[pos{m.Chapter, m.ChunkIdx, ws.st.Name}] {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// The unit's edit rows all live at its LEADER chunk — that is where the read-models look for them.
|
||||||
|
for _, ws := range editStages {
|
||||||
|
if !have[pos{u.Chapter, u.FirstChunkIdx, ws.st.Name}] {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// rowsResumeFree is the per-row half of the predicate above.
|
||||||
|
func (r *Runner) rowsResumeFree(rows []store.ChunkStatus, draftNames, editNames map[string]bool,
|
||||||
|
curDraft, curEdit string, hashes map[chunkKey]map[string]string) bool {
|
||||||
|
for _, cs := range rows {
|
||||||
|
if cs.Disposition == string(DispSkipped) {
|
||||||
|
continue // a skipped stage is re-derived from the flag above it; it never reaches a provider
|
||||||
|
}
|
||||||
|
var cur string
|
||||||
|
var w wave
|
||||||
|
switch {
|
||||||
|
case draftNames[cs.Stage]:
|
||||||
|
cur, w = curDraft, waveDraft
|
||||||
|
case editNames[cs.Stage]:
|
||||||
|
cur, w = curEdit, waveEdit
|
||||||
|
default:
|
||||||
|
continue // a stage this pipeline no longer runs cannot cost anything (rebill.go's rule)
|
||||||
|
}
|
||||||
|
h, ok := hashes[chunkKey{cs.Chapter, cs.ChunkIdx}][cs.Stage]
|
||||||
|
if !ok || h != cs.ContentHash {
|
||||||
|
return false // the wire bytes moved, or could not be reproduced — either way, a fresh call
|
||||||
|
}
|
||||||
|
if cs.SnapshotID != cur && !r.repinnable(cs.SnapshotID, cur, w) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
149
backend/internal/pipeline/volume_midrun_bankmove_test.go
Normal file
149
backend/internal/pipeline/volume_midrun_bankmove_test.go
Normal file
|
|
@ -0,0 +1,149 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/chunk/chunktest"
|
||||||
|
"textmachine/backend/internal/obs"
|
||||||
|
)
|
||||||
|
|
||||||
|
// volume_midrun_bankmove_test.go: the SCENARIO behind rescopeEditWave — a purchaser is charged twice for
|
||||||
|
// a chapter they already own, because the run moved the bank after deciding what was free.
|
||||||
|
//
|
||||||
|
// It is deliberately not the same assertion as the invariant test. That one pins "the re-plan runs before
|
||||||
|
// the wave"; this one pins "the buyer is not billed for a unit no grant paid for", which is the thing the
|
||||||
|
// buyer's money is actually exposed to. The two can come apart: a re-plan that ran but judged wrongly, a
|
||||||
|
// third path that moves the bank, or a later refactor that re-orders the seed — each keeps the invariant
|
||||||
|
// and breaks this.
|
||||||
|
//
|
||||||
|
// THE FIXTURE'S ONE TRICK, and the reason a homogeneous book will not show the defect. The mined delta has
|
||||||
|
// two sides. The WHICH side (the term list) is mined from the SOURCE of the whole book, so it is complete
|
||||||
|
// after the first purchase and never moves again. The WHAT side (each term's dst) is drafted-side: it is
|
||||||
|
// folded from the banknote blocks the translator emitted, so it grows with every purchase. So the source
|
||||||
|
// here is IDENTICAL in every chapter — the asymmetry is in what the drafts PROPOSE:
|
||||||
|
//
|
||||||
|
// - chapters 1-2 emit a banknote naming only 青茅山;
|
||||||
|
// - chapters 3-4 also propose a dst for 方源, which occurs in every chapter's source, chapters 1-2
|
||||||
|
// included.
|
||||||
|
//
|
||||||
|
// Purchase 1 buys chapters 1-2, so the auto-bank it writes holds 方源 with NO rendering. Purchase 2 drafts
|
||||||
|
// chapter 3, whose banknote supplies one — and the re-seed at the bank-mining stop folds it into the
|
||||||
|
// ENRICHED bank the editor selects over. Chapters 1-2 were admitted as free against the bank as it was at
|
||||||
|
// planning; their editor injection now carries a term it did not carry, so their content hash moves and
|
||||||
|
// they cost a fresh paid call. That is production's ordinary shape (mining.go says so where it re-seeds:
|
||||||
|
// "a term whose drafts disagreed can carry a different proposal than it did before ... that is bank
|
||||||
|
// CONTENT"), and it needs no heterogeneous source at all.
|
||||||
|
func TestAMidRunBankMoveNeverBillsBeyondTheGrant(t *testing.T) {
|
||||||
|
const chapters, grant2 = 4, 1
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := newJSONProvider(rec, func(body string) (string, string) {
|
||||||
|
if isEditBody(body) {
|
||||||
|
return "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД", "stop"
|
||||||
|
}
|
||||||
|
// The early chapters' drafts propose no rendering for 方源; the later ones do.
|
||||||
|
note := bankSeparator + "\n青茅山\tгора Цинмао\tplace"
|
||||||
|
if !strings.Contains(body, "MARK1") && !strings.Contains(body, "MARK2") {
|
||||||
|
note = bankSeparator + "\n方源\tФан Юань\tname\n青茅山\tгора Цинмао\tplace"
|
||||||
|
}
|
||||||
|
return "Фан Юань пришёл к горе Цинмао.\n" + note, "stop"
|
||||||
|
})
|
||||||
|
defer srv.Close()
|
||||||
|
bookPath := miningBookOfNChapters(t, srv.URL, chapters)
|
||||||
|
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
|
||||||
|
|
||||||
|
// PURCHASE 1: two units. They are delivered, and the auto-bank written at the mining stop holds 方源
|
||||||
|
// with no rendering yet.
|
||||||
|
r1 := newRunner(t, bookPath)
|
||||||
|
r1.MaxUnits = 2
|
||||||
|
if _, err := r1.TranslateBook(ctx); err != nil {
|
||||||
|
t.Fatalf("purchase 1: %v", err)
|
||||||
|
}
|
||||||
|
r1.Close()
|
||||||
|
|
||||||
|
// PURCHASE 2: ONE unit. --resnapshot because the previous purchase's edit jobs are pinned to the
|
||||||
|
// snapshot this run's own re-seed will move — the state planVolume warns about by name.
|
||||||
|
before := rec.count()
|
||||||
|
r2 := newRunner(t, bookPath)
|
||||||
|
defer r2.Close()
|
||||||
|
r2.MaxUnits = grant2
|
||||||
|
r2.Resnapshot = true
|
||||||
|
res2, err := r2.TranslateBook(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("purchase 2: %v", err)
|
||||||
|
}
|
||||||
|
newBodies := rec.all()[before:]
|
||||||
|
|
||||||
|
// (1) THE MONEY. A grant of N units buys at most N drafts and N edits. Anything more is a unit the
|
||||||
|
// buyer did not pay for being billed to them.
|
||||||
|
if want := grant2 * 2; len(newBodies) != want {
|
||||||
|
t.Errorf("a grant of %d output unit(s) made %d provider call(s), want exactly %d (draft+edit each): the surplus is work no grant authorised",
|
||||||
|
grant2, len(newBodies), want)
|
||||||
|
}
|
||||||
|
if maxUSD := float64(grant2*2) * fakeCallUSD; res2.TotalUSD > maxUSD+1e-9 {
|
||||||
|
t.Errorf("the run spent $%.6f on a grant of %d unit(s); at most $%.6f can be owed", res2.TotalUSD, grant2, maxUSD)
|
||||||
|
}
|
||||||
|
|
||||||
|
// (2) WHICH unit was billed, named. The already-delivered chapters must not be re-edited at all.
|
||||||
|
for _, b := range newBodies {
|
||||||
|
if !isEditBody(b) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, owned := range []string{"MARK1", "MARK2"} {
|
||||||
|
if strings.Contains(b, owned) {
|
||||||
|
t.Errorf("chapter %s was delivered by the previous purchase and was EDITED again on a grant that never covered it — the buyer paid twice for a chapter they already owned", owned)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// (3) THE REPORT. A unit that was billed must never be counted among the ones that rode along at $0 —
|
||||||
|
// that is the line an operator reads to decide the run was cheap.
|
||||||
|
if res2.Volume == nil {
|
||||||
|
t.Fatalf("the run stopped short of the book and said nothing about which ceiling did it")
|
||||||
|
}
|
||||||
|
if res2.Volume.Free != 0 {
|
||||||
|
t.Errorf("VolumeStop reports %d unit(s) as riding along at $0, but the bank moved under every one of them: %+v",
|
||||||
|
res2.Volume.Free, *res2.Volume)
|
||||||
|
}
|
||||||
|
if res2.Volume.Paid() > res2.Volume.MaxUnits {
|
||||||
|
t.Errorf("VolumeStop admits paying for %d unit(s) under a ceiling of %d: %+v",
|
||||||
|
res2.Volume.Paid(), res2.Volume.MaxUnits, *res2.Volume)
|
||||||
|
}
|
||||||
|
t.Logf("purchase 2 (grant %d): %d call(s), $%.6f — %v", grant2, len(newBodies), res2.TotalUSD, res2.Volume)
|
||||||
|
}
|
||||||
|
|
||||||
|
// miningBookOfNChapters is the mining-stop fixture over N EPUB chapters (setupMiningStopProject is
|
||||||
|
// single-source, and a volume ceiling needs a book with more than one output unit to bound). The chapter
|
||||||
|
// bodies are byte-identical apart from a MARK<n> tag the provider stub keys off — see the file comment:
|
||||||
|
// the defect needs asymmetric BANKNOTES, not an asymmetric source.
|
||||||
|
func miningBookOfNChapters(t *testing.T, providerURL string, n int) string {
|
||||||
|
t.Helper()
|
||||||
|
body := strings.Repeat("方源来到青茅山。方源很强。花家很大。花家的人。", 6)
|
||||||
|
var eps []chunktest.Chapter
|
||||||
|
var spine []string
|
||||||
|
for i := 1; i <= n; i++ {
|
||||||
|
id := fmt.Sprintf("c%d", i)
|
||||||
|
eps = append(eps, chunktest.Chapter{ID: id, Href: id + ".xhtml", Body: fmt.Sprintf("<p>MARK%d %s</p>", i, body)})
|
||||||
|
spine = append(spine, id)
|
||||||
|
}
|
||||||
|
bookPath := setupProjectOpts(t, providerURL, projectOpts{
|
||||||
|
epub: eps, spine: spine, regenerate: 1,
|
||||||
|
gatesYAML: "\nmining:\n contrast_path: mining-contrast.txt\ngates:\n banknote:\n enabled: true\n",
|
||||||
|
})
|
||||||
|
dir := filepath.Dir(bookPath)
|
||||||
|
writeFile(t, filepath.Join(dir, "mining-contrast.txt"), miningContrastData)
|
||||||
|
packRoot, err := filepath.Abs("../../configs/langpacks")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
raw, err := os.ReadFile(bookPath)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writeFile(t, bookPath, strings.Replace(string(raw), "source_lang: ja", "source_lang: zh\nlangpack_root: "+packRoot, 1))
|
||||||
|
return bookPath
|
||||||
|
}
|
||||||
1516
backend/internal/pipeline/volume_test.go
Normal file
1516
backend/internal/pipeline/volume_test.go
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -102,7 +102,7 @@ func (e *WaveSignatureStop) Error() string {
|
||||||
// translateBookWaves is the wave driver (R1). It runs the draft wave (draft ∥), the bank-mining stop, the edit wave (edit ∥) and
|
// translateBookWaves is the wave driver (R1). It runs the draft wave (draft ∥), the bank-mining stop, the edit wave (edit ∥) and
|
||||||
// assembles the BookResult. chunks + stickySel come from the precompute pass (chunk.SplitChunks + precomputeSticky), computed by
|
// assembles the BookResult. chunks + stickySel come from the precompute pass (chunk.SplitChunks + precomputeSticky), computed by
|
||||||
// the caller (TranslateBook) after ingest/seed/eager-build. Returns a *WaveSignatureStop when the bank-mining stop stops.
|
// the caller (TranslateBook) after ingest/seed/eager-build. Returns a *WaveSignatureStop when the bank-mining stop stops.
|
||||||
func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, stickySel []membank.Selection) (*BookResult, error) {
|
func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, stickySel []membank.Selection, scope *volumeScope) (*BookResult, error) {
|
||||||
draftStages := r.waveStagesIndexed(waveDraft)
|
draftStages := r.waveStagesIndexed(waveDraft)
|
||||||
editStages := r.waveStagesIndexed(waveEdit)
|
editStages := r.waveStagesIndexed(waveEdit)
|
||||||
workers := r.Pipeline.Waves.Workers
|
workers := r.Pipeline.Waves.Workers
|
||||||
|
|
@ -126,6 +126,12 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s
|
||||||
draftResults := make([]stageSeqResult, len(chunks))
|
draftResults := make([]stageSeqResult, len(chunks))
|
||||||
draftUnits := newDraftUnitTracker(r.outputUnits(chunks), chunks)
|
draftUnits := newDraftUnitTracker(r.outputUnits(chunks), chunks)
|
||||||
if err := r.runWave(ctx, workers, len(chunks), func(ctx context.Context, i int) error {
|
if err := r.runWave(ctx, workers, len(chunks), func(ctx context.Context, i int) error {
|
||||||
|
// The VOLUME ceiling (volume.go), applied where the item BEGINS rather than after it: a chunk whose
|
||||||
|
// output unit is outside this run's granted scope is never started, so no ceiling can be overshot by
|
||||||
|
// one call. A nil scope means no ceiling is in force and this is a no-op.
|
||||||
|
if !scope.allows(chunks[i]) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
d, err := r.runDraftChunk(ctx, draftSnapshot, chunks[i], stickySel[i], draftStages, !editWave)
|
d, err := r.runDraftChunk(ctx, draftSnapshot, chunks[i], stickySel[i], draftStages, !editWave)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
|
|
@ -158,6 +164,18 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s
|
||||||
}
|
}
|
||||||
|
|
||||||
res := &BookResult{BookID: r.Book.BookID}
|
res := &BookResult{BookID: r.Book.BookID}
|
||||||
|
// The stop NAMES its ceiling (D39.165 §1б: OpenHands' silent iteration limit is the anti-pattern being
|
||||||
|
// avoided). It rides on the result rather than on an error because a volume stop is a COMPLETION — the
|
||||||
|
// run did what was bought — so it must not reach the exit-code mapper at all. Only set when the ceiling
|
||||||
|
// actually held work back: a run granted more than the book had left simply finished.
|
||||||
|
if scope.bound() {
|
||||||
|
res.Volume = &scope.stop
|
||||||
|
r.Log.InfoContext(ctx, "the run is bounded by the VOLUME ceiling: it will stop having done what was granted, not at the end of the book",
|
||||||
|
"book", r.Book.BookID, "max_units", scope.stop.MaxUnits,
|
||||||
|
"paid_units", scope.stop.Paid(), "delivered_new", scope.stop.Delivered, "re_made", scope.stop.Reworked,
|
||||||
|
"free", scope.stop.Free,
|
||||||
|
"left_never_delivered", scope.stop.LeftFresh, "left_delivered_not_re_made", scope.stop.LeftRework)
|
||||||
|
}
|
||||||
// The terminologist's spend is THIS run's spend and belongs in this run's total. It is not a chunk cost,
|
// The terminologist's spend is THIS run's spend and belongs in this run's total. It is not a chunk cost,
|
||||||
// so the per-chunk accumulation below cannot see it — and a paid call that no total reports is a call
|
// so the per-chunk accumulation below cannot see it — and a paid call that no total reports is a call
|
||||||
// the operator cannot notice. (Zero when the gate is off or the calls replayed from checkpoints.)
|
// the operator cannot notice. (Zero when the gate is off or the calls replayed from checkpoints.)
|
||||||
|
|
@ -166,6 +184,9 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s
|
||||||
// --- Draft-only pipeline: the draft IS the shipping output; assemble per draft chunk (no edit units) ---
|
// --- Draft-only pipeline: the draft IS the shipping output; assemble per draft chunk (no edit units) ---
|
||||||
if !editWave {
|
if !editWave {
|
||||||
for i, ch := range chunks {
|
for i, ch := range chunks {
|
||||||
|
if !scope.allows(ch) {
|
||||||
|
continue // outside the volume scope: never drafted, so there is no outcome to assemble
|
||||||
|
}
|
||||||
oc := r.draftOnlyOutcome(ch, draftResults[i])
|
oc := r.draftOnlyOutcome(ch, draftResults[i])
|
||||||
res.Chunks = append(res.Chunks, oc)
|
res.Chunks = append(res.Chunks, oc)
|
||||||
res.TotalUSD += oc.CostUSD
|
res.TotalUSD += oc.CostUSD
|
||||||
|
|
@ -173,6 +194,7 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s
|
||||||
res.Flagged++
|
res.Flagged++
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
scope.reconcile(res.Chunks)
|
||||||
r.Log.InfoContext(ctx, "book run finished (draft-only)", "book", r.Book.BookID,
|
r.Log.InfoContext(ctx, "book run finished (draft-only)", "book", r.Book.BookID,
|
||||||
"chunks", len(res.Chunks), "flagged", res.Flagged, "run_usd", fmt.Sprintf("%.6f", res.TotalUSD))
|
"chunks", len(res.Chunks), "flagged", res.Flagged, "run_usd", fmt.Sprintf("%.6f", res.TotalUSD))
|
||||||
return res, nil
|
return res, nil
|
||||||
|
|
@ -187,6 +209,21 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s
|
||||||
return nil, fmt.Errorf("pipeline: upsert edit-wave snapshot %.12s: %w", editSnapshot, err)
|
return nil, fmt.Errorf("pipeline: upsert edit-wave snapshot %.12s: %w", editSnapshot, err)
|
||||||
}
|
}
|
||||||
units := buildEditUnits(chunks)
|
units := buildEditUnits(chunks)
|
||||||
|
// ⛔ THE PLAN IS RE-JUDGED HERE, because the bank-mining stop above may have re-seeded the bank and
|
||||||
|
// moved the snapshot this wave runs under. Units the plan called FREE were admitted outside the grant;
|
||||||
|
// any that the new bank makes paying must take a slot or not run. See rescopeEditWave.
|
||||||
|
if err := r.rescopeEditWave(ctx, scope, units, chunks, stickySel, editSnapshot); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
// ⚠ AND THE RE-PLAN IS NOT OPTIONAL: the wave refuses to run a scope that was planned against a
|
||||||
|
// different snapshot than the one it is about to use. Without this the call above is merely a call —
|
||||||
|
// deleting it leaves every test green while the ceiling silently stops holding on any book that
|
||||||
|
// re-seeds mid-run. With it, the invariant is the code's own, and removing the re-plan makes the run
|
||||||
|
// say so instead of overspending quietly.
|
||||||
|
if scope != nil && scope.editSnapshot != editSnapshot {
|
||||||
|
return nil, fmt.Errorf("pipeline: the volume scope was planned against edit-wave snapshot %.12s but the wave is running under %.12s — units admitted as free were judged on ground this run has since replaced, and paying for them would escape the ceiling (wave sequencing bug: rescopeEditWave did not run)",
|
||||||
|
scope.editSnapshot, editSnapshot)
|
||||||
|
}
|
||||||
draftByKey := make(map[chunkKey]stageSeqResult, len(chunks))
|
draftByKey := make(map[chunkKey]stageSeqResult, len(chunks))
|
||||||
for i, ch := range chunks {
|
for i, ch := range chunks {
|
||||||
draftByKey[chunkKey{ch.Chapter, ch.ChunkIdx}] = draftResults[i]
|
draftByKey[chunkKey{ch.Chapter, ch.ChunkIdx}] = draftResults[i]
|
||||||
|
|
@ -195,6 +232,11 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s
|
||||||
"units", len(units), "workers", workers, "edit_stages", len(editStages))
|
"units", len(units), "workers", workers, "edit_stages", len(editStages))
|
||||||
unitOutcomes := make([]*ChunkOutcome, len(units))
|
unitOutcomes := make([]*ChunkOutcome, len(units))
|
||||||
if err := r.runWave(ctx, workers, len(units), func(ctx context.Context, i int) error {
|
if err := r.runWave(ctx, workers, len(units), func(ctx context.Context, i int) error {
|
||||||
|
// Same admission as the draft wave, and the SAME set: a unit is either in this run's scope for both
|
||||||
|
// waves or in neither, so the edit wave can never be handed a unit whose members were not drafted.
|
||||||
|
if !scope.allowsUnit(units[i]) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
oc, err := r.runEditUnit(ctx, editSnapshot, units[i], draftByKey, editStages)
|
oc, err := r.runEditUnit(ctx, editSnapshot, units[i], draftByKey, editStages)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
|
|
@ -207,12 +249,17 @@ func (r *Runner) translateBookWaves(ctx context.Context, chunks []chunk.Chunk, s
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
for _, oc := range unitOutcomes {
|
for _, oc := range unitOutcomes {
|
||||||
|
if oc == nil {
|
||||||
|
continue // outside the volume scope: the unit was never started this run
|
||||||
|
}
|
||||||
res.Chunks = append(res.Chunks, *oc)
|
res.Chunks = append(res.Chunks, *oc)
|
||||||
res.TotalUSD += oc.CostUSD
|
res.TotalUSD += oc.CostUSD
|
||||||
if oc.Disposition == DispFlagged {
|
if oc.Disposition == DispFlagged {
|
||||||
res.Flagged++
|
res.Flagged++
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// The plan's counters become the run's counters only now, corrected by what actually shipped.
|
||||||
|
scope.reconcile(res.Chunks)
|
||||||
r.Log.InfoContext(ctx, "book run finished", "book", r.Book.BookID,
|
r.Log.InfoContext(ctx, "book run finished", "book", r.Book.BookID,
|
||||||
"units", len(res.Chunks), "flagged", res.Flagged, "run_usd", fmt.Sprintf("%.6f", res.TotalUSD))
|
"units", len(res.Chunks), "flagged", res.Flagged, "run_usd", fmt.Sprintf("%.6f", res.TotalUSD))
|
||||||
return res, nil
|
return res, nil
|
||||||
|
|
|
||||||
File diff suppressed because one or more lines are too long
|
|
@ -22,7 +22,7 @@
|
||||||
| Роль | Активный промт | Статус |
|
| Роль | Активный промт | Статус |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Оркестратор | [ORCHESTRATOR_SESSION_PROMPT.md](ORCHESTRATOR_SESSION_PROMPT.md) | роль и нормы; счётчик роли — CURRENT-STATE |
|
| Оркестратор | [ORCHESTRATOR_SESSION_PROMPT.md](ORCHESTRATOR_SESSION_PROMPT.md) | роль и нормы; счётчик роли — CURRENT-STATE |
|
||||||
| Бэкенд | [BACKEND_MONEY_PACK_SESSION_PROMPT.md](BACKEND_MONEY_PACK_SESSION_PROMPT.md) | **НАПИСАН, ждёт запуска владельцем.** Пак «деньги»: потолок ОБЪЁМА в движке рядом с денежным (D39.165 §1б) + $0-глагол сметы «оценить, ничего не покупая» (строка **231**). Оба рубежа пройдены: механический + опровергатель, **9 находок применены**, включая три фатальные — форма ответа через шов (словарь кодов ЗАМОРОЖЕН; решено: стоп по объёму = ЗАВЕРШЕНИЕ, код 0), выдуманный мной якорь, и вырожденное репро, повторявшее эррату 28.08-и. ⚠ Честная граница вписана в §0: проводка числа глав до движка паком НЕ заказана |
|
| Бэкенд | активного НЕТ | пак «деньги» ПРИНЯТ И ЗАЛЕНДЖЕН 29.08 (**D39.170**): потолок ОБЪЁМА оплаченной работы + читающий путь `status` сворачивает банк, поэтому смета доезжает до покупателя ДО покупки. Промт отработан — `archive/prompts/`. ⚠ Следующий заказ уже назван строкой **232** (ось отгрузки на `once_key`), проводка `--max-units` в платформу ГЕЙЧЕНА |
|
||||||
| Платформа | активного НЕТ | пак **P11** ПРИНЯТ И ЗАЛЕНДЖЕН 29.08 (**D39.169**): `PD-379` закрыт живой пробой оркестратора (отзыв → поток гаснет за секунду кадром `session_ended` → переподключение `401`), видимость застрявших денег, денежная группа. Промт отработан — `archive/prompts/`. ⚠ Открытым остаётся денежный `major` `PD-425` (дверь коррекций теряет пост-verb факт при обрыве клиента), отсрочка на оркестраторе |
|
| Платформа | активного НЕТ | пак **P11** ПРИНЯТ И ЗАЛЕНДЖЕН 29.08 (**D39.169**): `PD-379` закрыт живой пробой оркестратора (отзыв → поток гаснет за секунду кадром `session_ended` → переподключение `401`), видимость застрявших денег, денежная группа. Промт отработан — `archive/prompts/`. ⚠ Открытым остаётся денежный `major` `PD-425` (дверь коррекций теряет пост-verb факт при обрыве клиента), отсрочка на оркестраторе |
|
||||||
| Полигон | [POLYGON_EXP2223_REDO_SESSION_PROMPT.md](POLYGON_EXP2223_REDO_SESSION_PROMPT.md) (отложенный — [POLYGON_PACKAGE4_SESSION_PROMPT.md](POLYGON_PACKAGE4_SESSION_PROMPT.md), строка 85) | фаза Д ИДЁТ; ⚠ живой носитель курса — в `eval/dovodka/`, какой именно называет зона (⚠ [POLYGON_PHASE_D_HANDOFF.md](POLYGON_PHASE_D_HANDOFF.md) — перекрытый снимок, читать не как курс) |
|
| Полигон | [POLYGON_EXP2223_REDO_SESSION_PROMPT.md](POLYGON_EXP2223_REDO_SESSION_PROMPT.md) (отложенный — [POLYGON_PACKAGE4_SESSION_PROMPT.md](POLYGON_PACKAGE4_SESSION_PROMPT.md), строка 85) | фаза Д ИДЁТ; ⚠ живой носитель курса — в `eval/dovodka/`, какой именно называет зона (⚠ [POLYGON_PHASE_D_HANDOFF.md](POLYGON_PHASE_D_HANDOFF.md) — перекрытый снимок, читать не как курс) |
|
||||||
| Фронт | активного НЕТ | **ЗОНА ЗАМОРОЖЕНА** (D39.136 п.2 + D39.147: разморозка отдельным словом владельца, не привязана к P7); перечень первого касания — в зонном журнале |
|
| Фронт | активного НЕТ | **ЗОНА ЗАМОРОЖЕНА** (D39.136 п.2 + D39.147: разморозка отдельным словом владельца, не привязана к P7); перечень первого касания — в зонном журнале |
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
# Реестр D-нот — карта актуальности v2 (D1–D39.169;
|
# Реестр D-нот — карта актуальности v2 (D1–D39.170;
|
||||||
|
|
||||||
> ⚠ **СЛАБОЕ МЕСТО, КОТОРОЕ БЫЛО ЗДЕСЬ (вписано 22.08, ЗАКРЫТО 24.08 — D39.157 п.6).** Колонка ТЕЛА
|
> ⚠ **СЛАБОЕ МЕСТО, КОТОРОЕ БЫЛО ЗДЕСЬ (вписано 22.08, ЗАКРЫТО 24.08 — D39.157 п.6).** Колонка ТЕЛА
|
||||||
> у нот D39.107…D39.123 говорила «жив», хотя тела уехали в слайс подрезкой D39.139; семнадцать строк
|
> у нот D39.107…D39.123 говорила «жив», хотя тела уехали в слайс подрезкой D39.139; семнадцать строк
|
||||||
|
|
@ -230,3 +230,4 @@
|
||||||
| D39.167 | 28.08 | **Аудит документации (хребет)**: очередь пять дней звала активным исполненный промт и держала на владельце решённую развилку; регистр платформы расходился с актами лендинга ЧЕТЫРЬМЯ строками, одна major — класс «open, а лекарство в дереве» без гейта (строка 225); реестр D-нот потерял две колонки на восьми моих строках; справочники отстали от двух суток миноров. Журнал ужат 943 → 347 строк, прожитая проза бэкенда — в слайс. | жив | ЖИВОЕ: строка 225 (гейт класса); три участка обхода не пройдены | доки процесс регистр аудит |
|
| D39.167 | 28.08 | **Аудит документации (хребет)**: очередь пять дней звала активным исполненный промт и держала на владельце решённую развилку; регистр платформы расходился с актами лендинга ЧЕТЫРЬМЯ строками, одна major — класс «open, а лекарство в дереве» без гейта (строка 225); реестр D-нот потерял две колонки на восьми моих строках; справочники отстали от двух суток миноров. Журнал ужат 943 → 347 строк, прожитая проза бэкенда — в слайс. | жив | ЖИВОЕ: строка 225 (гейт класса); три участка обхода не пройдены | доки процесс регистр аудит |
|
||||||
| D39.168 | 28.08 | **Аудит доков, часть 2 — архитектура и ресёрчи.** Две НОРМЫ требовали построенного (закон шва: «`status --json` версию не несёт — закрыть», а несёт); доки врали про `accepts_labels` («пусты у всех» при трёх заполненных эндпоинтах — ось аккаунта); денежный вход цитировал грант $5 при коде 0. Вынесены в `archive/research/` четыре мёртвых отчёта (290 КБ) по графу входящих ссылок; двум ресёрчам без шапки вовсе поставлены ревю-шапки. | жив | ЖИВОЕ: системный класс line-якорей в архитектуре; часть 3 не пройдена | доки ресёрчи архив аудит |
|
| D39.168 | 28.08 | **Аудит доков, часть 2 — архитектура и ресёрчи.** Две НОРМЫ требовали построенного (закон шва: «`status --json` версию не несёт — закрыть», а несёт); доки врали про `accepts_labels` («пусты у всех» при трёх заполненных эндпоинтах — ось аккаунта); денежный вход цитировал грант $5 при коде 0. Вынесены в `archive/research/` четыре мёртвых отчёта (290 КБ) по графу входящих ссылок; двум ресёрчам без шапки вовсе поставлены ревю-шапки. | жив | ЖИВОЕ: системный класс line-якорей в архитектуре; часть 3 не пройдена | доки ресёрчи архив аудит |
|
||||||
| D39.169 | 29.08 | **Платформенный пак P11 принят и заленджен + контрактный минор 0.8.0.** Отзыв сессии гасит открытый SSE-поток (`PD-379`, единственная уязвимость `major`) и объявляет причину кадром `session_ended`, а не молчаливым обрывом в `401`; застрявший расчёт видим и закрываем; денежная группа `PD-384/391/394/397/376`. Второй рубеж сессии поймал СЕМЬ ложных подтверждений в её же отчёте (включая выдуманную мутационную посадку) — код цел, достоверность рассказа нет. Охотник вне карты нашёл ЧЕТЫРЕ регресса, введённых самим паком, все закрыты до сдачи. Две мои ошибки в каноне: схема нового кадра была сиротой вне союза `anyOf`, перечень кадров соединения противоречил телу схемы. Линтер якорей нашёл 21 указатель на исчезнувший код — норма: гонять его ПОСЛЕДНИМ шагом, после того как код замер. | жив | ЖИВОЕ: `PD-425` — открытый денежный major, отсрочка на оркестраторе; `PD-424`/`PD-426` отложены с доводом; `PD-423` — условие батареи, которого не знал рецепт | платформа контракт деньги сессии приёмка |
|
| D39.169 | 29.08 | **Платформенный пак P11 принят и заленджен + контрактный минор 0.8.0.** Отзыв сессии гасит открытый SSE-поток (`PD-379`, единственная уязвимость `major`) и объявляет причину кадром `session_ended`, а не молчаливым обрывом в `401`; застрявший расчёт видим и закрываем; денежная группа `PD-384/391/394/397/376`. Второй рубеж сессии поймал СЕМЬ ложных подтверждений в её же отчёте (включая выдуманную мутационную посадку) — код цел, достоверность рассказа нет. Охотник вне карты нашёл ЧЕТЫРЕ регресса, введённых самим паком, все закрыты до сдачи. Две мои ошибки в каноне: схема нового кадра была сиротой вне союза `anyOf`, перечень кадров соединения противоречил телу схемы. Линтер якорей нашёл 21 указатель на исчезнувший код — норма: гонять его ПОСЛЕДНИМ шагом, после того как код замер. | жив | ЖИВОЕ: `PD-425` — открытый денежный major, отсрочка на оркестраторе; `PD-424`/`PD-426` отложены с доводом; `PD-423` — условие батареи, которого не знал рецепт | платформа контракт деньги сессии приёмка |
|
||||||
|
| D39.170 | 29.08 | **Движковый пак «деньги» принят и заленджен**: потолок ОБЪЁМА оплаченной работы (`--max-units`) + читающий путь `status` теперь СВОРАЧИВАЕТ банк, поэтому смета пере-прохода впервые доезжает до покупателя ДО покупки, оставаясь $0. ⚠ Пак СНЯЛ ПОСЫЛКУ чужой зоны: платформа не берёт `rebill_*` по доводу «status читает ноль сразу после apply» (`ingest/resync.go:37-43`) — довод устарел. Четыре круга приёмки; блокирующая находка охотника: потолок ПРОБИВАЛСЯ пере-сидом банка посреди прогона (грант 1 → 4 вызова, $0.0073 вместо $0.0036, две доставленные главы оплачены дважды и названы бесплатными). Лечение структурное — пере-план + отказ волны работать с планом чужого снапшота. Сессия не воспроизвела сценарий ПЯТЬ раз при верных прогонах: дельту двигает не текст, а предложения черновиков, а `spoilerBlocked` (`membank/memory.go:640-647`) режет термин с поздним `since_ch` навсегда. | жив | ЖИВОЕ: ось отгрузки на `once_key` — отдельный заказ; терминолог вне потолка — вход в калибровку цены; проводка `--max-units` ГЕЙЧЕНА | движок деньги потолок приёмка шов |
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
# Журнал решений оркестратора — контракт D1–D39.169 (живой файл: карта · эрраты · живые тела · голова D39.124+ (подрезка D39.139); тела закрытых эр — в слайсах `docs/archive/architecture/`, указатель ниже; реестр всех нот — `05-decisions-index.md`)
|
# Журнал решений оркестратора — контракт D1–D39.170 (живой файл: карта · эрраты · живые тела · голова D39.124+ (подрезка D39.139); тела закрытых эр — в слайсах `docs/archive/architecture/`, указатель ниже; реестр всех нот — `05-decisions-index.md`)
|
||||||
|
|
||||||
> **⟶ КАРТА АКТУАЛЬНОСТИ (ревизия D31, продлена до D38.2 [12.07]; исторические записи ниже НЕ переписываются — дисциплина D23.3).** Работая с контрактом (греп номера: живой файл → слайсы, целиком НЕ читать — D39.125), держи под рукой, что чем перекрыто:
|
> **⟶ КАРТА АКТУАЛЬНОСТИ (ревизия D31, продлена до D38.2 [12.07]; исторические записи ниже НЕ переписываются — дисциплина D23.3).** Работая с контрактом (греп номера: живой файл → слайсы, целиком НЕ читать — D39.125), держи под рукой, что чем перекрыто:
|
||||||
> ⚠ **Эррата 09.08 (D39.125):** D39.111 п.1 предписывал промту S3 «максимум = баланс МИНУС открытые холды» — формула ОШИБОЧНА (вычитание дважды), исправлена D39.115 п.2(а): максимум = Balance КАК ЕСТЬ; тело D39.111 живёт ниже в этом файле (голова D39.106+).
|
> ⚠ **Эррата 09.08 (D39.125):** D39.111 п.1 предписывал промту S3 «максимум = баланс МИНУС открытые холды» — формула ОШИБОЧНА (вычитание дважды), исправлена D39.115 п.2(а): максимум = Balance КАК ЕСТЬ; тело D39.111 живёт ниже в этом файле (голова D39.106+).
|
||||||
|
|
@ -1235,3 +1235,100 @@ D39.165 «смета уже публикуется в `status`» верна то
|
||||||
|
|
||||||
**Строки.** Регистр платформы: 426 строк, 105 open, 7 major. Семь заказанных строк закрыты с телами;
|
**Строки.** Регистр платформы: 426 строк, 105 open, 7 major. Семь заказанных строк закрыты с телами;
|
||||||
девять новых заведены тем же деревом.
|
девять новых заведены тем же деревом.
|
||||||
|
|
||||||
|
## D39.170 — ДВИЖКОВЫЙ ПАК «ДЕНЬГИ» ПРИНЯТ И ЗАЛЕНДЖЕН: потолок ОБЪЁМА оплаченной работы + смета пере-прохода, которая наконец доезжает до покупателя (29.08, оркестратор №19). ✅
|
||||||
|
|
||||||
|
**Что заленджено.** Движок останавливается по ОБЪЁМУ оплаченной работы, а не только по деньгам
|
||||||
|
(`--max-units`, `VolumeStop`) — покупатель платит за N юнитов и получает ровно N · читающий путь
|
||||||
|
`status` СВОРАЧИВАЕТ банк, а не проецирует сохранённый глоссарий, и потому впервые отвечает на вопрос
|
||||||
|
«сколько будет стоить пере-проход» ДО покупки, оставаясь $0-глаголом без записи · денежный отчёт
|
||||||
|
разведён на `Delivered`/`Reworked`/`LeftFresh`/`LeftRework`, и приглашение купить произносится только
|
||||||
|
про никогда-не-доставленное · авто-банк пишется атомарно.
|
||||||
|
|
||||||
|
⚠ **ГЛАВНОЕ, ЧЕГО НЕ НАЗВАЛА НИ ОДНА СЕССИЯ: этот пак СНЯЛ ПОСЫЛКУ, на которой стоит решение ЧУЖОЙ
|
||||||
|
зоны.** Платформа сознательно не берёт `rebill_units`/`rebill_usd` через шов, и её основание записано в
|
||||||
|
коде (`platform/internal/ingest/resync.go:37-43`): «status проецирует СОХРАНЁННУЮ память, и сразу после
|
||||||
|
`bank-apply` — в единственный момент, когда согласие хотело бы цифру, — он честно читает ноль». Это было
|
||||||
|
верно и ратифицировано эрратой 28.08-к. **Теперь неверно:** `foldMemoryForRead` стал ПЕРВЫМ ответом
|
||||||
|
читающего пути, а `projectStoredMemory` понижена до фолбэка (`status.go:817-824`, комментарий самого
|
||||||
|
пака это и объявляет). Слепое окно закрыто. Следствия проведены этим же лендингом: строка бэклога
|
||||||
|
**231** закрывается, комментарий платформы получает строку в её регистр, а проводка полей через шов
|
||||||
|
становится ВОЗМОЖНОЙ — но не выполняется, потому что гейчена вместе с `--max-units` (ниже).
|
||||||
|
|
||||||
|
**Приёмка — четыре круга, и первые три сдачи были неверны.** Мой первый проход дал 4 денежных дефекта
|
||||||
|
(обход гейта согласия дроблением покупок; потолок, тративший покупку на пере-делку вперёд доставки;
|
||||||
|
расхождение `status`/`translate`; неатомарный авто-банк). Второй круг сессии — ещё шесть её собственных,
|
||||||
|
включая то, что **её же строка стопа сообщала ЮНИТЫ как ГЛАВЫ**, то есть подмену, ради устранения которой
|
||||||
|
пак и заведён, совершал его собственный отчёт. Четвёртый круг — денежный охотник вне карты, пять
|
||||||
|
подтверждённых прогоном.
|
||||||
|
|
||||||
|
⛔ **БЛОКИРУЮЩАЯ НАХОДКА ЧЕТВЁРТОГО КРУГА: потолок объёма ПРОБИВАЛСЯ, и пробивал его сам прогон.**
|
||||||
|
`planVolume` классифицирует юниты ДО волн и по снапшоту, который берёт один раз (`volume.go:353-355`);
|
||||||
|
free-юниты допускаются БЕЗУСЛОВНО, вне гранта (`:259-265`) — бесплатная работа ничего не стоит. Но между
|
||||||
|
планированием и редакторской волной стоит стоп майнинга, который пере-сеивает банк ПОСРЕДИ прогона
|
||||||
|
(`mining.go:242`), после чего edit-снапшот берётся заново (`waverun.go:203`). Значит каждый «бесплатный»
|
||||||
|
юнит судился по снапшоту, который прогон сам же и заменил. Замер: грант 1 → **4 вызова вместо 2**,
|
||||||
|
$0.007280 вместо максимум $0.003640, две уже доставленные главы отредактированы повторно — и строка
|
||||||
|
отчёта назвала их «rode along at $0».
|
||||||
|
**Лечение — не гард, а ПЕРЕ-ПЛАН:** вопрос «бесплатен ли юнит» задаётся заново против снапшота, ставшего
|
||||||
|
реальным, перед самой волной; ставший платным берёт слот или не идёт. **И оно структурно:** волна
|
||||||
|
ОТКАЗЫВАЕТСЯ работать со скоупом, спланированным против другого снапшота — без этого удаление вызова
|
||||||
|
оставляло батарею зелёной при молча переставшем держать потолке.
|
||||||
|
|
||||||
|
⚠ **ЧЕТЫРЕ КРУГА ПОТРЕБОВАЛИСЬ НЕ ПОТОМУ, ЧТО СЕССИЯ ПЛОХО РАБОТАЛА — А ПОТОМУ ЧТО СЦЕНАРИЙ
|
||||||
|
КОНТРИНТУИТИВЕН, И ЭТО САМОЕ ЦЕННОЕ ЗНАНИЕ ПАКА.** Сессия пыталась воспроизвести дефект ПЯТЬ раз и не
|
||||||
|
смогла ни разу; её прогоны были ВЕРНЫ, ошибочна была гипотеза о причине. Две ловушки, обе измерены:
|
||||||
|
1. **Не та сторона дельты.** Список терминов майнится из ИСХОДНИКА всей книги (`mining.go:80-98`) — он
|
||||||
|
полон после первой покупки и не растёт. Байты банка двигает другая сторона: `dst` термина, который
|
||||||
|
складывается из banknote-блоков ЧЕРНОВИКОВ (`mining.go:109`) и растёт с каждой покупкой **даже при
|
||||||
|
побайтово однородном источнике**. Асимметрию надо строить в том, что предлагает МОДЕЛЬ, а не в тексте.
|
||||||
|
2. **Условия тянут в РАЗНЫЕ стороны.** Естественный способ заставить дельту расти — дать поздним главам
|
||||||
|
новый термин — ровно этим выталкивает `since_ch` за пределы уже доставленного, а
|
||||||
|
`spoilerBlocked` (`membank/memory.go:640-647`) — жёсткий гейт `chapter < since_ch`: такой термин не
|
||||||
|
попадёт в инъекцию ранней главы НИКОГДА, какой бы `dst` он ни получил. Нужна ОБРАТНАЯ асимметрия:
|
||||||
|
термин в раннем исходнике, рендеринг поздно.
|
||||||
|
**Урок, годный за пределами этого пака:** пять верных прогонов при неверной гипотезе неотличимы от
|
||||||
|
«дефекта нет». Различил их только тот, у кого сценарий уже работал.
|
||||||
|
|
||||||
|
**Доказательства, которые я снял САМ, а не принял.** Батарея: 21 пакет из 21, 0 FAIL, полнота сверена
|
||||||
|
списком `go list` против вердиктов. Линтер `0 issues`. Четыре ключевых пина зелёные поимённо. **Две мои
|
||||||
|
мутационные посадки на сценарном тесте:** снятие вызова пере-плана → структурный отказ адресным
|
||||||
|
сообщением; снятие вызова И гарда → дефект целиком, всеми четырьмя денежными утверждениями. То есть
|
||||||
|
находка настоящая, тест ловит ПЕРЕОПЛАТУ, а не факт вызова, и фикс её закрывает.
|
||||||
|
|
||||||
|
**Статус находки уточнён ПРОТИВ автора, обеими сторонами.** Возражение сессии («сдвиг `memory_version`
|
||||||
|
ничего не доказывает — мерить надо `content_hash` юнита») принято, и охотник показал, что мерил именно
|
||||||
|
его. Но он же признал, что **завысил срочность**: дефект требует конъюнкции трёх условий, в его
|
||||||
|
собственной фикстуре два термина из трёх её не выполняют и остаются инертными; «покупка №50
|
||||||
|
переоплачивает 490 юнитов» — потолок тяжести, когда сработало, а не ожидаемый случай, и частоту на
|
||||||
|
реальной книге он не мерил и назвать не может. Лечение от узости входа не зависит — оно структурно.
|
||||||
|
|
||||||
|
⛔ **ПРОВОДКА `--max-units` В ПЛАТФОРМУ ОСТАЁТСЯ ГЕЙЧЕНОЙ, и основание усилилось.** Я ставил гейт на
|
||||||
|
доводе «вторая покупка на майнящей книге падает без `--resnapshot`». Настоящее основание оказалось
|
||||||
|
сильнее — **с `--resnapshot` потолок не держал**, — и нашёл его охотник, не я. Записываю как есть:
|
||||||
|
везение, а не прозорливость.
|
||||||
|
|
||||||
|
**ЧТО ОТЛОЖЕНО СТРОКАМИ, диспозиции мои.** (1) **Ось «свежий/пере-делка» выведена из ПОЛНОТЫ СТРОК, а
|
||||||
|
не из факта отгрузки** — поэтому добавление стадии превращает дочитанную книгу в «никогда не
|
||||||
|
доставлявшуюся», а юнит, прерванный между волнами, тратит слот дважды. Носитель у движка уже есть —
|
||||||
|
реестр анонсов `events_outbox.once_key` (`unitOnceKey` = книга+волна+глава+юнит), монотонный на всю
|
||||||
|
жизнь книги; не хватает читающего метода поверх готовой константы `onceKeyLookup`. Отдельный заказ:
|
||||||
|
сессия законно не полезла в чужой пакет в хвосте четвёртого круга. ⚠ Формулировка строки — «ошибается
|
||||||
|
не только слово»: отчёт, зовущий дочитанную книгу недоставленной, ПРИГЛАШАЕТ купить её снова.
|
||||||
|
(2) **Терминолог вне объёмного потолка** — место траты в ЦЕНЕ, как договорено, но цифра меняет вес
|
||||||
|
договорённости: накладные масштабируются КНИГОЙ, а не грантом (три покупки по одному юниту дали три
|
||||||
|
полнокнижных консолидации по $0.005460 каждая, при том что юнит дешевле). Книга на 500 юнитов,
|
||||||
|
проданная по одному, оплатит 500 полнокнижных проходов — это вход в калибровку цены, а не сноска.
|
||||||
|
(3) **Майнящие книги под потолком** — корень в джоб-гарде ратифицированного Р6-контура, чужой предмет.
|
||||||
|
|
||||||
|
**Названная граница, которую сессия записала, а не умолчала:** флагнутый юнит следующей покупкой едет
|
||||||
|
как `Free` и в остатке стопа не виден. Я проверил цепь до конца и границу принимаю: движок отчитывает
|
||||||
|
его в `status` (`flagged`, пер-главные паспорта), платформа берёт это аллоулистом (`Flagged`,
|
||||||
|
`UnitsFlagged` — `ingest/resync.go`), а читателю он доезжает как `withheld` (`ingest/export.go:47`,
|
||||||
|
канон §UnitState). Покупателю не показывают фальшивое «доставлено» — цепь цела, дублировать её в стопе
|
||||||
|
покупки незачем.
|
||||||
|
|
||||||
|
**Сужение D20.2-Q2 — подтверждена ДЕЙСТВУЮЩАЯ редакция:** порог судится по ВСЕЙ книге (дробить
|
||||||
|
бесполезно), именованный кап — по тому, что заплатит ЭТОТ прогон (законная работа не отклоняется),
|
||||||
|
прогон без пере-оплаты гейта не встречает. Первая редакция отозвана самой сессией и помечена отозванной,
|
||||||
|
а не переписана молча.
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,240 @@
|
||||||
|
> ⚠ **ОТРАБОТАН И ЗАКРЫТ 29.08 — D39.170.** Пак принят и заленджен. Исход: потолок ОБЪЁМА оплаченной
|
||||||
|
> работы построен; читающий путь `status` сворачивает банк, поэтому смета пере-прохода впервые доезжает
|
||||||
|
> до покупателя ДО покупки, оставаясь $0-глаголом. Приёмка шла ЧЕТЫРЕ круга: первые три сдачи были
|
||||||
|
> неверны, и блокирующую находку — потолок пробивался пере-сидом банка посреди прогона — дал охотник вне
|
||||||
|
> карты, а не отчёт. Отложены строками **232** (ось отгрузки на `once_key`) и **233** (терминолог вне
|
||||||
|
> потолка); проводка `--max-units` в платформу ГЕЙЧЕНА. **Инструкции отсюда НЕ исполнять** — файл
|
||||||
|
> историчен.
|
||||||
|
|
||||||
|
# Промт: бэкенд, пак «деньги» — продажа обретает настоящий стоп, а смета становится доступной до покупки
|
||||||
|
|
||||||
|
> **Выдан оркестратором №19, 28.08.2026.** Основание — **D39.165** §1 (три продуктовых развилки сняты
|
||||||
|
> словом владельца) и строка бэклога **231**. Пак идёт ПАРАЛЛЕЛЬНО платформенному.
|
||||||
|
> ⚠ **ФАЙЛЫ вы не делите** — модули отдельные, платформа кода движка НЕ компилирует и его тестов НЕ
|
||||||
|
> запускает (проверено). **Но ШОВ общий и ведётся РУКАМИ в трёх точках:** таблица кодов выхода ·
|
||||||
|
> argv из строковых литералов · аллоулист полей `status --json`. **Твой §3.1 трогает первые две** —
|
||||||
|
> форму ответа я решил за тебя (§3.1), а не оставил на согласование.
|
||||||
|
|
||||||
|
## §0. Какая проблема и что решит твой результат
|
||||||
|
|
||||||
|
**1. Ручка «купить N глав» ничего не ограничивает.** Платформа продаёт ГЛАВЫ, а движку передаёт
|
||||||
|
только денежный потолок: `--ceiling-usd` (`backend/cmd/tmctl/invocation.go:129`), и тот каппит
|
||||||
|
**КУМУЛЯТИВНУЮ** трату по книге (`backend/internal/store/ledger.go:67` —
|
||||||
|
`bookTotal+estimate > c.BookUSD`). **Про число глав движок не знает вовсе**, флага такого нет.
|
||||||
|
|
||||||
|
**Измерено приёмкой 28.08 (леджеры реальных прогонов, $0):** настоящая цена главы — средняя
|
||||||
|
**$0.0115–0.0190**, максимум одной главы **$0.0375**; константа продажи платформы — **$0.03**. То
|
||||||
|
есть покупка «10 глав» отдаёт движку $0.30, а $0.30 на этом материале покупает **16–26 глав**.
|
||||||
|
Пользователь просит одно, получает другое, и полоса каппится на купленном.
|
||||||
|
|
||||||
|
⚠ **И хвост цены порождает КАЧЕСТВО, а не объём** (наблюдение владельца, подтверждено данными):
|
||||||
|
вызов может отработать успешно и вернуть мусор — деньги списаны, юнит уезжает на ретрай, иногда на
|
||||||
|
эскалационный хоп к дорогой модели. В леджере это видно с именами причин (`empty`, `length`,
|
||||||
|
`cjk_artifact`, `sanitizer_defect`), и доля ретраев+эскалаций гуляет **1.5% → 23%** между прогонами.
|
||||||
|
|
||||||
|
**2. Смету пере-прохода показать НЕЧЕМ — и это названная цена вчерашнего минора 0.7.0.** Движок уже
|
||||||
|
считает её (`projectRebill`, `backend/internal/pipeline/rebill.go:111`) и публикует в `status --json`.
|
||||||
|
Но `status` материализует банк из СОХРАНЁННОГО глоссария
|
||||||
|
(`backend/internal/pipeline/status.go:733-744`, `projectStoredMemory` — его собственный комментарий:
|
||||||
|
«A seed-FILE edit not yet re-run is NOT reflected here»), а `bank-apply` пишет только ФАЙЛЫ решений.
|
||||||
|
Свёртка происходит внутри СЛЕДУЮЩЕГО `translate`. **Итог: сразу после правки банка движок честно
|
||||||
|
отвечает «ничего не двигалось», и платформе нечего показать покупателю.** Носитель — строка **231**.
|
||||||
|
|
||||||
|
**Что решит результат:** движок получает СПОСОБНОСТЬ остановиться по объёму, а пользователь получает
|
||||||
|
возможность увидеть, что затронет пере-проход, ДО оплаты.
|
||||||
|
|
||||||
|
⛔ **ЧЕСТНАЯ ГРАНИЦА, которую первая редакция промта скрывала (нашёл опровергатель).** «Купил N глав —
|
||||||
|
получил N глав» этим паком НЕ становится правдой, потому что **число глав до движка не доезжает**:
|
||||||
|
`CeilingChapters` живёт на платформе (`platform/internal/runs/runs.go:232`) до самого спавна, а
|
||||||
|
`runner.TranslateArgs` собирает argv литералами и канала для него не имеет. **Проводка — платформенная
|
||||||
|
половина, и она в этом паке НЕ заказана.** Ты строишь механизм; труба к нему — отдельный заказ, и я
|
||||||
|
его завожу строкой. Не считай пак провалившимся оттого, что после него обещание §0 ещё не выполнено.
|
||||||
|
|
||||||
|
## §1. Зона записи и git
|
||||||
|
|
||||||
|
**Твоя зона — `backend/`.** Итоги и находки — своя секция журнала `docs/PROGRESS.md` («Бэкенд»).
|
||||||
|
|
||||||
|
- **Ты НЕ коммитишь.** Лендит оркестратор после адверсариальной приёмки.
|
||||||
|
- ⚠ **`platform/` НЕ ТРОГАТЬ.** Там параллельно работает платформенный пак. Нужна правка на той
|
||||||
|
стороне — это ПИНГ, а не правка.
|
||||||
|
- ⚠ **`docs/` — не твоя зона**, кроме своей секции журнала. ⚠ Перед записью в неё **перечитай файл**:
|
||||||
|
журнал правят две сессии, и 28.08 оркестратор уже унёс чужую секцию своим коммитом из-за общего
|
||||||
|
окна записи.
|
||||||
|
- ⚠ **Стендовый `tmctl` собирай из ЗАФИКСИРОВАННОЙ копии дерева, а не из рабочего**, если он тебе
|
||||||
|
понадобится: рабочее дерево грязное твоими же правками, и бинарь из него мерит непонятно что.
|
||||||
|
- В дереве незакоммиченная работа полигона (20 позиций) — не касаться.
|
||||||
|
|
||||||
|
## §2. Карта чтения — ≤5 позиций, ЗАКОН
|
||||||
|
|
||||||
|
1. **`backend/cmd/tmctl/main.go:252-262`** — три денежных флага и почему они ОРТОГОНАЛЬНЫ; там же
|
||||||
|
предупреждение «`--ceiling-usd` — НЕ бюджет прогона» · **`invocation.go:120-210`** (парсинг и
|
||||||
|
валидация флагов).
|
||||||
|
2. **`backend/internal/store/ledger.go:30-80`** — где потолок реально судит трату (`Reserve`).
|
||||||
|
3. **`backend/internal/pipeline/rebill.go`** — `RebillProjection`, `projectRebill` (`:111`), порог
|
||||||
|
согласия и его текст отказа (`:318-324`).
|
||||||
|
4. **`backend/internal/pipeline/status.go:733-744`** — `projectStoredMemory`; **читать ВНИМАТЕЛЬНО:
|
||||||
|
это и есть слепое окно из §0.2**.
|
||||||
|
5. **`docs/architecture/05-decisions-log.md`, тело D39.165** (греп `^## D39.165`) — основание пака;
|
||||||
|
**и эррата 28.08-к в шапке того же файла** — там названа моя ошибка про смету, не повтори её.
|
||||||
|
|
||||||
|
## §3. Состав пака
|
||||||
|
|
||||||
|
### §3.1. Потолок ОБЪЁМА рядом с денежным — ЗАКАЗ; форма свободна
|
||||||
|
|
||||||
|
**Заказано:** движок умеет остановиться по объёму работы, а не только по деньгам, и **говорит, по
|
||||||
|
какому потолку встал**.
|
||||||
|
|
||||||
|
**Ратифицировано D39.165 §1б и обсуждению не подлежит:** потолок живёт в ДВИЖКЕ. Резать волну на
|
||||||
|
стороне платформы нельзя — она волной не владеет и пер-юнитной цены заранее не знает.
|
||||||
|
|
||||||
|
**Свободен и обязан обосновать — В ЧЁМ мерить. ⚠ Кандидатов ТРИ, а не два (поправка опровергателя):**
|
||||||
|
(а) **выходной юнит** — `ManifestUnit`, та самая гранулярность, которую движок отгружает, и она же
|
||||||
|
платформенный `chapters.units_total` с интейка; (б) **чанк**; (в) **`chunk×stage` — БИЛЛИНГОВАЯ
|
||||||
|
единица движка**, ровно та, в которой считает смета (`rebill.go:322`: «%d chunk×stage unit(s)»).
|
||||||
|
⚠ **Ловушка ровно здесь:** режущий волну естественно считает то, за что ПЛАТЯТ, то есть (в), а
|
||||||
|
платформа продаёт (а) — и купивший десять глав получит их долю. **Дефект пака воспроизведётся слоем
|
||||||
|
ниже, в той же форме.** Выбор объясни и назови, как он ложится на платформенную единицу продажи.
|
||||||
|
|
||||||
|
⚠ **И ещё одно, чего первая редакция не спрашивала: как против объёмного потолка считаются РЕПИНЫ и
|
||||||
|
РЕТРАИ.** Пере-проход, пере-привязывающий пятьсот юнитов за $0 (`rebill.go:315-320`), сожжёт объёмный
|
||||||
|
потолок, не потратив цента. Ответь на это явно.
|
||||||
|
|
||||||
|
⛔ **ФОРМА ОТВЕТА ЧЕРЕЗ ШОВ РЕШЕНА МНОЙ, и это НЕ предмет твоего выбора — потому что выбор здесь
|
||||||
|
ломает деньги в обе стороны (нашёл опровергатель).** Словарь кодов выхода объявлен ЗАМОРОЖЕННЫМ самим
|
||||||
|
движком (`backend/cmd/tmctl/main.go:125-127`: «both bands are frozen seams and a fresh code would be a
|
||||||
|
word added to a ratified dictionary»), а платформа незнакомый код читает как ОТКАЗ
|
||||||
|
(`platform/internal/ingest/exit.go:168-169`, `default: OutcomeFailed`). Отсюда:
|
||||||
|
- переиспользуешь код денежного потолка → платформа объявит прогон ПРИОСТАНОВЛЕННЫМ и возобновит его
|
||||||
|
за уже купленный объём — дефект, ради которого пак заведён, воспроизведётся слоем ниже;
|
||||||
|
- заведёшь новый код → пользователь, получивший РОВНО купленное, увидит «ошибка сервиса».
|
||||||
|
|
||||||
|
**РЕШЕНИЕ: остановка по объёму — это НЕ остановка, а ЗАВЕРШЕНИЕ.** По деньгам прогон встал ПОСРЕДИ
|
||||||
|
работы, там честно «приостановлен» и возобновление. По объёму он сделал ровно то, что куплено, — это
|
||||||
|
УСПЕХ, код **0**, замороженный словарь не трогается вовсе, платформа читает штатное «готово».
|
||||||
|
Различение, которого требует пункт ниже, живёт тогда не в КОДЕ ВЫХОДА, а в отчёте/логе. Если найдёшь,
|
||||||
|
что это ломает что-то, чего я не вижу, — **пинг, и я пере-решаю**: на этой развилке уже стоял выбор,
|
||||||
|
ломающий деньги в обе стороны.
|
||||||
|
|
||||||
|
⚠ **Две грабли, забранные у внешних систем — исполнить обе:**
|
||||||
|
- **остановка обязана НАЗЫВАТЬ свой потолок.** У OpenHands (`max_budget_per_task` + `max_iterations`)
|
||||||
|
агент упирается в лимит итераций МОЛЧА — известная жалоба. У нас «встал по деньгам» и «встал по
|
||||||
|
объёму» — разные ответы пользователю и разные ремеди;
|
||||||
|
- **проверять ПЕРЕД началом единицы работы, а не после.** У LiteLLM пост-фактум-проверка токенного
|
||||||
|
потолка всегда пробивает его на один вызов. Денежный потолок у нас проверяется в `Reserve` до
|
||||||
|
вызова — объёмный обязан вести себя так же.
|
||||||
|
|
||||||
|
⚠ **Не сломай ортогональность.** `--ceiling-usd` — кумулятивный КНИЖНЫЙ кап, не бюджет прогона
|
||||||
|
(`main.go:256-260`). Новый потолок — про работу ЭТОГО прогона. Смешаешь семантику — сломаешь
|
||||||
|
платформенный расчёт холдов, который на кумулятивности и стоит.
|
||||||
|
|
||||||
|
### §3.2. Смета БЕЗ покупки — ЗАКАЗ; форма свободна
|
||||||
|
|
||||||
|
**Заказано:** платформа может узнать «сколько юнитов затронет пере-проход», НЕ запуская `translate` и
|
||||||
|
ничего не покупая.
|
||||||
|
|
||||||
|
**Что знать** (проверено мной, пере-проверь): смета уже считается (`projectRebill`) и уже публикуется
|
||||||
|
в `status --json` полями `rebill_units`/`rebill_usd`. Мешает ровно одно — **тайминг свёртки**:
|
||||||
|
`status` читает СОХРАНЁННЫЙ глоссарий, а решения лежат в ФАЙЛАХ до следующего `translate`.
|
||||||
|
|
||||||
|
**Форма свободна.** Очевидные кандидаты, оба со своей ценой:
|
||||||
|
- **флаг у `status`** («сверни решения в память для расчёта и не пиши») — дёшево, но `status`
|
||||||
|
ратифицирован как ЧИСТО ЧИТАЮЩИЙ ремонтный глагол, и свёртка внутри него это свойство ломает;
|
||||||
|
- **новый $0-глагол** — честнее по разделению, но новая поверхность CLI и новый документ.
|
||||||
|
⚠ **Родня, которую надо знать: `resnapshot --dry-run`** — строка **124(в)**, поименована в законе шва
|
||||||
|
как будущая дверь. Она просит ТУ ЖЕ способность «оценить, ничего не покупая». **Если твоя форма
|
||||||
|
закрывает обе — скажи это прямо, это удешевляет проект.**
|
||||||
|
|
||||||
|
⛔ **Границы, обе жёсткие, но сформулированы ТОЧНО (первая редакция обе переоценила — нашёл
|
||||||
|
опровергатель):**
|
||||||
|
- **$0 — абсолютно:** ни одного провайдерского вызова.
|
||||||
|
- **Read-only — с ОДНОЙ названной оговоркой:** «ни одной записи в стор» абсолютом быть НЕ МОЖЕТ —
|
||||||
|
первое касание проекта создаёт и мигрирует базу движка по построению (`backend/internal/pipeline/runner.go:193-197`;
|
||||||
|
ратифицировано D39.122, у `status` тот же побочный эффект). Требование верное: **никаких записей
|
||||||
|
СВЕРХ этого first-touch**, и ни одной записи в банк/глоссарий/чекпойнты.
|
||||||
|
- **Ключей НЕ требует.** ⚠ Якорь первой редакции (`dotenv.go:53-55`) я ВЫДУМАЛ — там середина
|
||||||
|
докблока про другое. Настоящий носитель: `backend/cmd/tmctl/invocation.go:152-155` (текст отказа,
|
||||||
|
перечисляющий читающие глаголы) и пин `backend/cmd/tmctl/keysfile_test.go`.
|
||||||
|
|
||||||
|
⚠ **ТРЕТИЙ КАНДИДАТ, ДЕШЕВЛЕ ОБОИХ МОИХ — его нашёл опровергатель, и я его не видел.** У `bank-apply`
|
||||||
|
УЖЕ ЕСТЬ режим проекции (`invocation.go:126`: «print the projection of the decisions and write
|
||||||
|
NOTHING»), он берёт тот же арбитр-флок, ничего не пишет и печатает JSON-отчёт. **И у него уже на
|
||||||
|
руках ровно недостающее состояние**: `readBookState` поднимает bank + seed + delta + rejects ДО
|
||||||
|
свёртки. Не хватает только statuses и manifest для `projectRebill`. Рассмотри этот путь ПЕРВЫМ: он не
|
||||||
|
трогает «`status` — чисто читающий» вовсе.
|
||||||
|
|
||||||
|
⛔ **ЛОВУШКА, КОТОРАЯ УБЬЁТ ВЕСЬ СМЫСЛ, если её не увидеть.** Свернуть решения В СТОР ты не сможешь
|
||||||
|
(`store.go:117-128`, `OpenReadOnly` ставит `PRAGMA query_only=1`), значит фолд будет В ПАМЯТИ. Но
|
||||||
|
канонический фолд идёт ЧЕРЕЗ стор: `seeding.go` пишет банк, читает его обратно `ORDER BY src, sense,
|
||||||
|
…` и материализует. Порядок НЕ безразличен — `membank/memory.go:610-617` сортирует стабильно «в
|
||||||
|
порядке замороженных entries», а дальше токенный бюджет РЕЖЕТ хвост. **Материализуешь «будущий банк»
|
||||||
|
в порядке добавления — получишь другой набор строк глоссария, другие отрендеренные байты, другой
|
||||||
|
content-hash: бесплатная смета назовёт одно число, а платный прогон выставит другое.** Это ровно тот
|
||||||
|
дефект, который пак продаёт как решённый. Докажи тождество, а не предположи его.
|
||||||
|
|
||||||
|
⚠ **И мина формы:** оба поля сметы объявлены `omitempty` (`status.go:208-209`). D39.166 п.2 уже
|
||||||
|
заплатил за это однажды: «$0-цена мурует дверь — юниты по нулевой цене приходят БЕЗ цены». Если
|
||||||
|
переиспользуешь форму, «ноль юнитов» и «не считалось» на проводе станут неотличимы.
|
||||||
|
⚠ **Факт из строки 231, который тебе полезен:** поля `rebill_units`/`rebill_usd` СНЯТЫ с аллоулиста
|
||||||
|
шва до появления потребителя — вернутся вместе с твоим глаголом.
|
||||||
|
|
||||||
|
### §3.3. Чего в паке НЕТ — пропуски подписаны
|
||||||
|
|
||||||
|
- **Калибровка константы $0.03 — НЕ твоя и НЕ сейчас.** Она платформенная и гейчена строкой **202**
|
||||||
|
(живой прогон текущего стека). Ты даёшь МЕХАНИЗМ, цену считает не движок.
|
||||||
|
- **Цена от объёма исходника** (D39.165 §1а) — платформенная половина, носитель `chapters.units_total`.
|
||||||
|
- **Строки 228 / 230 / 141-остаток / 131** — соседи, не заказ. Увидишь — назови строкой, не чини.
|
||||||
|
|
||||||
|
## §4. Самопроверка ИСПОЛНЕНИЕМ — «перечитал сам» её не удовлетворяет
|
||||||
|
|
||||||
|
1. **Батарея зоны** под `-race`, линтер 0 issues; числа — с командами.
|
||||||
|
2. **Потолок объёма доказывается ПОСТ-ФИКСНЫМ инвариантом:** прогон с потолком в N единиц
|
||||||
|
останавливается на N, отвечает признаком «встал по ОБЪЁМУ» (отличимым от денежного), и
|
||||||
|
останавливается ПЕРЕД началом N+1, а не после.
|
||||||
|
3. ⛔ **Смета доказывается $0 — и РЕПРО ПЕРВОЙ РЕДАКЦИИ БЫЛО ВЫРОЖДЕННЫМ, я повторил ошибку, о
|
||||||
|
которой сам же написал эрратy 28.08-и.** Состояние «`translate` не запускался» негодно: у такой
|
||||||
|
книги строк `chunk_status` нет вовсе, и `projectRebill` отдаёт ПУСТО и до фикса, и после
|
||||||
|
(`rebill.go:132-134` пропускает строки без снапшота). **Верное состояние: прогон СОСТОЯЛСЯ**
|
||||||
|
(строки `chunk_status` есть, снапшоты записаны) → **ПОТОМ применена правка банка** → и твой
|
||||||
|
механизм отдаёт НЕнулевой счёт затронутых юнитов, тогда как сегодняшний `status` на том же
|
||||||
|
состоянии отвечает нулём. Разница «сегодня ноль / после фикса N» и есть деливерабл.
|
||||||
|
При этом леджер не двинулся ни на цент, а в стор не ушло ничего сверх first-touch.
|
||||||
|
4. **Свои посадки мутаций:** минимум по одной на пункт; вердикт — по ДЕЛЬТЕ против чистой базовой
|
||||||
|
линии и по ТОПИЧНОСТИ упавшего теста, не по цвету батареи.
|
||||||
|
5. **Адверсариальный проход по своей готовой работе — заказан; глубину и веер выбираешь сама.**
|
||||||
|
⚠ Куда смотреть в ЭТОМ паке: деньги (перерасход либо отказ законной работе) · тождество бесплатной сметы и платного прогона · шов (форма ответа, argv).
|
||||||
|
Дефекты, которые внёс сам пак, называй первыми — приёмке они не видны.
|
||||||
|
⚠ **Fable 5** — обычно хватает одного-двух агентов, но это РЕКОМЕНДАЦИЯ, не потолок: веер под предмет выбирает сессия. Модель задавай ЯВНО и знай, сколько их работает.
|
||||||
|
|
||||||
|
## §5. Оси ревью — 1–3, вправе заменить с аргументом
|
||||||
|
|
||||||
|
1. **Ось денег:** может ли новый потолок дать перерасход или, наоборот, отказать законной работе;
|
||||||
|
не сломана ли кумулятивная семантика денежного капа.
|
||||||
|
2. **Ось «пара, которой в репо нет»:** ветвления по паре/книге в Go быть не должно.
|
||||||
|
3. **Ось детерминизма:** трогаешь сборку запроса или снапшот — не перекупается ли прогон.
|
||||||
|
|
||||||
|
## §6. Записка-план, комплектность
|
||||||
|
|
||||||
|
**До правок** — записка-план в своей секции журнала. **В конце** — таблица комплектности против §3:
|
||||||
|
пункт → что сделано → каким ИСПОЛНЕНИЕМ подтверждено. Слово вместо команды = пункт НЕ сделан.
|
||||||
|
|
||||||
|
## §7. Эхо-протокол старта
|
||||||
|
|
||||||
|
ДО работы — ≤10 строк: **скоуп · инварианты · не-делать**. Первое действие — вписать свой блок в файл
|
||||||
|
канала (§10), сразу второе — послать мне эхо.
|
||||||
|
|
||||||
|
## §8. Obstacle — обязательная секция
|
||||||
|
|
||||||
|
**«Что НЕ удалось и что НЕ проверено»** отдельной секцией, не россыпью.
|
||||||
|
|
||||||
|
## §9. Канал вопросов и твоё право отказаться
|
||||||
|
|
||||||
|
Конфликт промта с кодом или доками — **пинг, не интерпретация в свою пользу**.
|
||||||
|
⚠ **У тебя есть право сказать «этого делать не надо» — с аргументом.** За последние сутки исполнители
|
||||||
|
опровергли ЧЕТЫРЕ моих заказа и все четыре раза были правы; два я пере-проверил своей посадкой и
|
||||||
|
отозвал. Отказ с разбором дороже послушного исполнения.
|
||||||
|
⚠ **Тесты и гейты не подгонять под зелень** (D39.121).
|
||||||
|
|
||||||
|
## §10. Как со мной связаться
|
||||||
|
|
||||||
|
**Адрес — в файле `/tmp/textmachine-channel`.** Впиши свой блок первым делом, чужие не трогай.
|
||||||
|
⚠ **Канала нет ⇒ НЕ искать:** вопрос секцией в отчёт, работа продолжается.
|
||||||
Loading…
Add table
Reference in a new issue