heaven/textmachine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Land the book-writer pack: tmctl build renders EPUB 3 and clean text from BookExport with six honesty states, refusing or marking every hole, paths published in StatusArtifacts

Browse source
This commit is contained in:
heaven 2026-08-30 20:14:11 +03:00
parent 67c05bedef
commit 8adcb8611b
23 changed files with 2941 additions and 47 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
backend/README.md
View file

@ -8,7 +8,7 @@
| Пакет | Что делает | Ключевые файлы |
|---|---|---|
| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) / `manifest` (**$0-производитель персиста манифеста глав/чанков**, строка 100: нужен ДО первого прогона — дерево глав разобранной, но не запущенной книги; `--json` печатает сам документ) / `migrate` (**$0-write-open без прогона**, строка 174: read-only команды требуют ТОЧНОГО совпадения схемы и не мигрируют, поэтому апгрейд бинаря запирал все существующие книги — платформа зовёт `status --json` перед каждым спавном, а write-команда, которая мигрировала бы, не наступала никогда. Грузит ТОЛЬКО `book.yaml` — ни цен, ни ключей на $0-пути (строка 146); restore point берёт ПОД ЛОКОМ и только когда шаг реально применяется (имя `<метка>-pre-migrate.db` — вне секундного неймспейса платного пути, строка 173); несовпадение схемы у read-only путей — типизированный отказ **exit 13** с машинным токеном `schema_mismatch found=N expected=M`). **ДВА потолка прогона, и они ортогональны.** `--ceiling-usd` (строка 145) — ДЕНЬГИ: перекрывает книжный `ceilings.book_usd` ТОЛЬКО на этот прогон, в book.yaml не пишется, ноль/отрицательное = отказ запуска; каппит КУМУЛЯТИВНУЮ трату книги, а не приращение прогона. `--max-units` (D39.165 §1б) — ОБЪЁМ: не больше N ВЫХОДНЫХ ЮНИТОВ (гранулярность `units_total` манифеста, та самая, в которой платформа продаёт главы) будет ОПЛАЧЕНО этим прогоном. Юниты, которые прогон отдаёт за $0 (резюм, ре-пин), едут бесплатно и потолок не тратят; ретраи и эскалации — тоже нет, они внутри юнита. Принимает только `translate`. **Остановка по объёму — ЗАВЕРШЕНИЕ (exit 0), а не пауза:** словарь кодов выхода не расширялся, различение живёт в отчёте прогона и в логе. Отчёт различает ДОСТАВКУ (юнит, которого не было) и ПЕРЕ-ДЕЛКУ (уже доставленный юнит под сдвинутым снапшотом) — покупка, целиком ушедшая в переделку, обязана читаться как переделка. ⚠ На книге, которая МАЙНИТ банк, вторая покупка требует `--resnapshot` (авто-банк растёт между покупками и двигает edit-снапшот) — прогон предупреждает об этом в логе. main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` |
| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `build` (**писатель книги**, строка 236: EPUB 3 + чистый txt из той же экспорт-проекции в `<project_db>.book.<fmt>` рядом с БД, пути — в `artifacts.book_files`; книга с ЛЮБОЙ дырой — pending/withheld/incomplete/ghost — по умолчанию отказ **exit 16** с перечнем, `--partial` пишет с пометкой на первой странице и маркером на каждой дыре; слова читателя — данные `configs/langpacks/<цель>/reader.txt`, вне снапшота; `--format epub,txt`, `--out`) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) / `manifest` (**$0-производитель персиста манифеста глав/чанков**, строка 100: нужен ДО первого прогона — дерево глав разобранной, но не запущенной книги; `--json` печатает сам документ) / `migrate` (**$0-write-open без прогона**, строка 174: read-only команды требуют ТОЧНОГО совпадения схемы и не мигрируют, поэтому апгрейд бинаря запирал все существующие книги — платформа зовёт `status --json` перед каждым спавном, а write-команда, которая мигрировала бы, не наступала никогда. Грузит ТОЛЬКО `book.yaml` — ни цен, ни ключей на $0-пути (строка 146); restore point берёт ПОД ЛОКОМ и только когда шаг реально применяется (имя `<метка>-pre-migrate.db` — вне секундного неймспейса платного пути, строка 173); несовпадение схемы у read-only путей — типизированный отказ **exit 13** с машинным токеном `schema_mismatch found=N expected=M`). **ДВА потолка прогона, и они ортогональны.** `--ceiling-usd` (строка 145) — ДЕНЬГИ: перекрывает книжный `ceilings.book_usd` ТОЛЬКО на этот прогон, в book.yaml не пишется, ноль/отрицательное = отказ запуска; каппит КУМУЛЯТИВНУЮ трату книги, а не приращение прогона. `--max-units` (D39.165 §1б) — ОБЪЁМ: не больше N ВЫХОДНЫХ ЮНИТОВ (гранулярность `units_total` манифеста, та самая, в которой платформа продаёт главы) будет ОПЛАЧЕНО этим прогоном. Юниты, которые прогон отдаёт за $0 (резюм, ре-пин), едут бесплатно и потолок не тратят; ретраи и эскалации — тоже нет, они внутри юнита. Принимает только `translate`. **Остановка по объёму — ЗАВЕРШЕНИЕ (exit 0), а не пауза:** словарь кодов выхода не расширялся, различение живёт в отчёте прогона и в логе. Отчёт различает ДОСТАВКУ (юнит, которого не было) и ПЕРЕ-ДЕЛКУ (уже доставленный юнит под сдвинутым снапшотом) — покупка, целиком ушедшая в переделку, обязана читаться как переделка. ⚠ На книге, которая МАЙНИТ банк, вторая покупка требует `--resnapshot` (авто-банк растёт между покупками и двигает edit-снапшот) — прогон предупреждает об этом в логе. main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` |
| `internal/llm` | OpenAI-совместимый транспорт + retry/backoff, **capability-слой** (budget_field/temperature/reasoning per-модель), провайдеры openai/local (no-proxy)/anthropic (DEPRECATED-референс); `failover.go` удалён паком-17 — маршрутизация лейблами живёт в config/pipeline (канал B, один хоп `chain[0]`, fail-closed) | `httpllm.go`, `capability.go`, `provider_*.go` |
| `internal/ledger` | Цены по usage (вкл. reasoning/cache-поля), `PriceForResponse` по фактической модели | `pricing.go` |
| `internal/store` | SQLite (modernc, CGO-free), цепочка миграций `schema_version` (⚠ ALTER-шаги v8+ не идемпотентны вопреки шапке — бэклог-строка 49а), reserve/settle+checkpoint, chunk_status, глоссарий (+подписной цикл терминолога), ruby, retrieval_state, request_log; `OpenReadOnly` — без flock/миграций/recovery для status/report/export (схема не совпала — типизированный `*store.SchemaMismatchError{Found,Expected}`, обе стороны); `Migrate` (+шов `beforeApply` под локом: restore point берётся там) — поверхность деплой-шага строки 174, `SchemaHead` — та самая `Expected`; write-open БД новее бинаря теперь ОТКАЗ, а не тихое открытие | `ledger.go`, `migrate.go`, `glossary.go`, `store.go` |

167
backend/cmd/tmctl/build_cli_test.go Normal file
View file

@ -0,0 +1,167 @@
package main
import (
"bytes"
"encoding/json"
"errors"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
// build_cli_test.go: `tmctl build` at the shell — the flag contract, and the incomplete state driven
// through the REAL binary on the c-lite harness (setupGapProject), because the claim is about the bytes
// a reader gets and the exit code a supervisor gets.
func TestParseBuildFlags(t *testing.T) {
inv, err := parseInvocation([]string{"build", "--config", "b.yaml", "--format", "epub", "--out", "x.epub", "--partial"}, &bytes.Buffer{})
if err != nil {
t.Fatal(err)
}
if inv.cmd != "build" || len(inv.buildFormats) != 1 || inv.buildFormats[0] != "epub" || inv.buildOut != "x.epub" || !inv.buildPartial {
t.Fatalf("parsed: %+v", inv)
}
inv, err = parseInvocation([]string{"build", "--config", "b.yaml"}, &bytes.Buffer{})
if err != nil || inv.buildFormats != nil || inv.buildOut != "" || inv.buildPartial {
t.Fatalf("bare build: %+v %v", inv, err)
}
inv, err = parseInvocation([]string{"build", "--config", "b.yaml", "--format", "txt, epub"}, &bytes.Buffer{})
if err != nil || strings.Join(inv.buildFormats, ",") != "txt,epub" {
t.Fatalf("comma list: %+v %v", inv.buildFormats, err)
}
for name, args := range map[string][]string{
"export's --plaintext": {"build", "--config", "b.yaml", "--plaintext"},
"export's --pairs": {"build", "--config", "b.yaml", "--pairs"},
"status's --json": {"build", "--config", "b.yaml", "--json"},
"unknown format": {"build", "--config", "b.yaml", "--format", "pdf"},
"empty format": {"build", "--config", "b.yaml", "--format", ""},
"out without a format": {"build", "--config", "b.yaml", "--out", "x"},
"out with two formats": {"build", "--config", "b.yaml", "--format", "epub,txt", "--out", "x"},
"empty out": {"build", "--config", "b.yaml", "--format", "epub", "--out", ""},
} {
if _, err := parseInvocation(args, &bytes.Buffer{}); err == nil {
t.Errorf("%s: must be refused", name)
}
}
}
// TestBuildFlagsAreRefusedElsewhere: a command that does not act on a flag refuses it — the rule the
// other flags follow, enumerated from dispatchCommands.
func TestBuildFlagsAreRefusedElsewhere(t *testing.T) {
for _, cmd := range dispatchCommands {
if cmd == "build" {
continue
}
for _, flag := range [][]string{{"--partial"}, {"--format", "epub"}, {"--out", "x"}} {
args := append([]string{cmd, "--config", "b.yaml"}, flag...)
if cmd == "bank-apply" {
args = append(args, "--decisions", "d.json")
}
if cmd == "seed-lint" {
args = append([]string{cmd, "--seed", "s.yaml"}, flag...)
}
_, err := parseInvocation(args, &bytes.Buffer{})
if err == nil {
t.Errorf("%s must refuse %v", cmd, flag)
continue
}
rest := strings.Replace(err.Error(), `"`+cmd+`"`, "", 1)
if !strings.Contains(rest, cmd) && !strings.Contains(rest, "flag provided but not defined") {
t.Errorf("the reason given to `%s` names it only to refuse it: %v", cmd, err)
}
}
}
}
// runTmctlCode is runTmctl without the exit-code assertion: it returns what the process said and how it
// ended, for the commands whose refusal IS the thing under test.
func runTmctlCode(t *testing.T, bookPath string, args ...string) (stdout, stderr string, code int) {
t.Helper()
cmd := exec.Command(buildTmctl(t), append(args, "--config", bookPath)...)
var out, errb bytes.Buffer
cmd.Stdout, cmd.Stderr = &out, &errb
err := cmd.Run()
return out.String(), errb.String(), exitCodeOf(t, err)
}
func TestBuildRefusesTheGapAndPartialMarksIt(t *testing.T) {
const echoMarker = "禁"
srv := gapProvider(t, echoMarker)
bookPath := setupGapProject(t, srv.URL, echoMarker)
dir := filepath.Dir(bookPath)
runTmctl(t, bookPath, "translate")
// The state really is the one under test: pending 0, one INCOMPLETE unit.
var exp map[string]any
if err := json.Unmarshal([]byte(runTmctl(t, bookPath, "export")), &exp); err != nil {
t.Fatal(err)
}
if exp["pending_units"].(float64) != 0 {
t.Fatalf("fixture: pending must be 0: %v", exp)
}
stdout, stderr, code := runTmctlCode(t, bookPath, "build")
if code != exitBookIncomplete {
t.Fatalf("a book with a hole must be refused with exit %d, got %d\nstdout:%s\nstderr:%s", exitBookIncomplete, code, stdout, stderr)
}
if !strings.Contains(stderr, "chapter 1 unit 0: incomplete") || !strings.Contains(stderr, "cjk_artifact") {
t.Errorf("the refusal must name the hole and its cause:\n%s", stderr)
}
for _, f := range []string{"gap-book.db.book.epub", "gap-book.db.book.txt"} {
if _, err := os.Stat(filepath.Join(dir, f)); !errors.Is(err, os.ErrNotExist) {
t.Errorf("a refusal writes nothing, but %s exists (%v)", f, err)
}
}
stdout, stderr, code = runTmctlCode(t, bookPath, "build", "--partial")
if code != 0 {
t.Fatalf("--partial must write: exit %d\nstderr:%s", code, stderr)
}
var rep struct {
Version string `json:"build_version"`
Files map[string]string `json:"files"`
IncompleteUnits int `json:"incomplete_units"`
Complete bool `json:"complete"`
}
if err := json.Unmarshal([]byte(stdout), &rep); err != nil {
t.Fatalf("stdout is not the report: %v\n%s", err, stdout)
}
if rep.Version != "tm-build-v1" || rep.IncompleteUnits != 1 || rep.Complete {
t.Errorf("report: %+v", rep)
}
txt, err := os.ReadFile(rep.Files["txt"])
if err != nil {
t.Fatal(err)
}
shipped := string(txt)
t.Logf("READER'S TEXT:\n%s", shipped)
// No langpack → the non-verbal mark ahead of the surviving text; the edited text is there; the audit
// vocabulary of `export --plaintext` is not.
if !strings.Contains(shipped, "⚠ 1.0 −1\n\nОтредактированный текст уцелевшей части.") {
t.Errorf("the mark must stand right before the surviving text:\n%s", shipped)
}
if !strings.HasPrefix(shipped, "Тест\n\n⚠ 1/1\n\n\n1\n\n") {
t.Errorf("the notice must open the book:\n%s", shipped)
}
for _, banned := range []string{"=== CHAPTER", "TEXT MISSING", "INCOMPLETE", "cjk_artifact", "flagged"} {
if strings.Contains(shipped, banned) {
t.Errorf("operator vocabulary %q reached the reader:\n%s", banned, shipped)
}
}
// status publishes the same places the build wrote to.
var st struct {
Artifacts struct {
BookFiles map[string]string `json:"book_files"`
} `json:"artifacts"`
}
if err := json.Unmarshal([]byte(runTmctl(t, bookPath, "status", "--json")), &st); err != nil {
t.Fatal(err)
}
for f, p := range rep.Files {
if st.Artifacts.BookFiles[f] != p {
t.Errorf("status artifacts.book_files[%s] = %q, build wrote %q", f, st.Artifacts.BookFiles[f], p)
}
}
}

64
backend/cmd/tmctl/invocation.go
View file

@ -6,7 +6,9 @@ import (
"io"
"math"
"strconv"
"strings"
"textmachine/backend/internal/bookfile"
"textmachine/backend/internal/pipeline"
)
@ -49,7 +51,14 @@ type invocation struct {
// invocation property for exactly the reason ceilingUSD is — how much of the book a caller is buying
// right now belongs to the call, not to the book's data.
maxUnits int
sel pipeline.RedriveSelector
// buildFormats / buildOut / buildPartial are `build`'s own (backlog row 236): which reader's copies
// to write, where the one file goes when the caller names a place, and whether a book with holes is
// written marked rather than refused. Invocation properties like the flags above — a caller's
// choice for one build, not a fact about the book.
buildFormats []string
buildOut string
buildPartial bool
sel pipeline.RedriveSelector
}
// rebillConsentValue parses `--accept-rebill[=usd]` (D20.2-Q2): the OPTIONAL-VALUE form of Р6, where a
@ -112,7 +121,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
// tells the operator they do not exist. seed-lint and bank-apply are on their own clauses because
// their arguments differ — seed-lint takes --seed and no --config, bank-apply needs --decisions —
// and folding them into the first clause would be the same class of lie.
return invocation{}, fmt.Errorf("usage: tmctl <translate|report|status|export|redrive|manifest|backup|migrate> --config book.yaml | tmctl bank-apply --config book.yaml --decisions decisions.json | tmctl seed-lint --seed glossary.yaml")
return invocation{}, fmt.Errorf("usage: tmctl <translate|report|status|export|build|redrive|manifest|backup|migrate> --config book.yaml | tmctl bank-apply --config book.yaml --decisions decisions.json | tmctl seed-lint --seed glossary.yaml")
}
cmd, rest := args[0], args[1:]
@ -135,6 +144,9 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
seed := fs.String("seed", "", "seed-lint: path to the glossary seed YAML to validate ($0, no --config)")
keysFile := fs.String("keys-file", "", "translate: path to the DEPLOYMENT's provider-key file (KEY=VALUE lines). Loaded FIRST, so it wins over the .env beside book.yaml; a named file that cannot be read is a refusal, never a silent skip")
decisions := fs.String("decisions", "", "bank-apply: path to the JSON decision document to apply to the book's memory bank ($0)")
buildFormat := fs.String("format", "", "build: the reader's copies to write, comma-separated ("+strings.Join(bookfile.Formats, ",")+"); default: all of them")
buildOut := fs.String("out", "", "build: write the ONE requested format to this path instead of beside the project database (needs --format with a single format)")
buildPartial := fs.Bool("partial", false, "build: write the book even when it is not whole — the notice is on its first page and every hole (a unit not translated, withheld, or missing a piece; text the current cut cannot place) is marked in the text. Without it a book with any hole is REFUSED (exit 16) and nothing is written")
if err := fs.Parse(rest); err != nil {
return invocation{}, err
}
@ -159,7 +171,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
// is taught something false about the system. It names them all, and the test that guards it
// enumerates dispatchCommands rather than a hand-written list — the hand list is what left
// `bank-apply`, the very verb this pack added, undescribed by its own refusal.
return invocation{}, fmt.Errorf("--keys-file is accepted by `translate` only, not by %q: `translate` is the one command that spends. The $0 read commands (report/status/export/manifest/seed-lint/bank-apply) must not demand provider keys at all (D20.4); `redrive` does re-attack and re-bill, but takes its keys from its own conventional .env; `backup` and `migrate` are deploy steps that call no provider", cmd)
return invocation{}, fmt.Errorf("--keys-file is accepted by `translate` only, not by %q: `translate` is the one command that spends. The $0 read commands (report/status/export/build/manifest/seed-lint/bank-apply) must not demand provider keys at all (D20.4); `redrive` does re-attack and re-bill, but takes its keys from its own conventional .env; `backup` and `migrate` are deploy steps that call no provider", cmd)
}
// A PRESENT flag with an empty value is refused rather than ignored, for the reason the flag
// exists: an unset variable in a deployment's unit file would otherwise fall back to the
@ -211,6 +223,51 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
return invocation{}, fmt.Errorf("--max-units must be a positive number of output units (it is the volume ceiling for THIS run); got %d — omit the flag to run the whole book", *maxUnits)
}
}
// `build`'s flags follow the same rule: a command that does not act on them refuses them. The silent
// version of --partial is the expensive one in the reader's currency rather than in dollars — a
// caller who passed it to `export` would believe they had asked for an honest, marked copy and get
// the audit projection, which marks nothing.
for _, name := range []string{"format", "out", "partial"} {
if given[name] && cmd != "build" {
return invocation{}, fmt.Errorf("--%s is accepted by `build` only, not by %q: it shapes the reader's copy of the book that `build` writes, and %q writes no such copy — accepting it would tell a caller a file was written the way they asked when nothing was", name, cmd, cmd)
}
}
// And the converse: `build` prints its report as JSON and writes files, so export's and status's
// output flags mean nothing to it — refused, not ignored (`build --plaintext` would otherwise read as
// «the text file, please» and quietly do something else).
if cmd == "build" {
for _, name := range []string{"plaintext", "pairs", "json"} {
if given[name] {
return invocation{}, fmt.Errorf("--%s is not a `build` flag: build writes the reader's copies (--format epub,txt) and prints its report as JSON regardless; accepting it would tell a caller it changed the output when it did not", name)
}
}
}
var buildFormats []string
if given["format"] {
seen := map[string]bool{}
for _, f := range strings.Split(*buildFormat, ",") {
f = strings.TrimSpace(f)
if f == "" || seen[f] {
continue // a format named twice is one file, not two writes of it
}
if !bookfile.KnownFormat(f) {
return invocation{}, fmt.Errorf("--format %q is not a format `build` writes (want one or more of %s, comma-separated)", f, strings.Join(bookfile.Formats, ","))
}
seen[f] = true
buildFormats = append(buildFormats, f)
}
if len(buildFormats) == 0 {
return invocation{}, fmt.Errorf("--format was given with no format: an unset variable must not read as «all formats» — omit the flag for that")
}
}
if given["out"] {
if *buildOut == "" {
return invocation{}, fmt.Errorf("--out was given with no path: an unset variable must not read as «the default place»")
}
if len(buildFormats) != 1 {
return invocation{}, fmt.Errorf("--out names ONE file, so it needs --format with exactly one format (got %d)", len(buildFormats))
}
}
// seed-lint validates a standalone seed YAML — it takes --seed, not --config (no book/store/keys).
if cmd == "seed-lint" {
if *seed == "" {
@ -249,6 +306,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
cmd: cmd, cfgPath: *cfgPath, resnapshot: *resnapshot, acceptRebill: acceptRebill.c,
asJSON: *asJSON, asPlaintext: *asPlaintext, asPairs: *asPairs, verifyBank: *verifyBank,
ceilingUSD: *ceilingUSD, maxUnits: *maxUnits, keysFile: *keysFile, decisionsPath: *decisions, dryRun: *dryRun,
buildFormats: buildFormats, buildOut: *buildOut, buildPartial: *buildPartial,
sel: pipeline.RedriveSelector{
Chapter: *chapter, ChunkIdx: *chunk, Reason: *reason, DryRun: *dryRun,
},

7
backend/cmd/tmctl/invocation_test.go
View file

@ -26,9 +26,10 @@ func TestParseNoArgsUsage(t *testing.T) {
// learned: `backup`/`migrate` (the deploy step, row 174) and `seed-lint`, which gets its own clause
// because it takes --seed rather than --config (row 176, sanctioned by D39.134 п.3). `bank-apply` is
// the sixth (D39.156 — the inbound door for the owner's bank decisions) and gets its own clause for
// the same reason seed-lint does: it needs --decisions on top of --config. Every one is a deliberate
// contract extension and the rest of the usage text stays frozen.
if err == nil || err.Error() != "usage: tmctl <translate|report|status|export|redrive|manifest|backup|migrate> --config book.yaml | tmctl bank-apply --config book.yaml --decisions decisions.json | tmctl seed-lint --seed glossary.yaml" {
// the same reason seed-lint does: it needs --decisions on top of --config. `build` is the seventh
// (backlog row 236 — the reader's copy of the book, EPUB/text, beside the database). Every one is a
// deliberate contract extension and the rest of the usage text stays frozen.
if err == nil || err.Error() != "usage: tmctl <translate|report|status|export|build|redrive|manifest|backup|migrate> --config book.yaml | tmctl bank-apply --config book.yaml --decisions decisions.json | tmctl seed-lint --seed glossary.yaml" {
t.Fatalf("usage error text is frozen, got: %v", err)
}
// The line and the dispatch switch are the same list: a command reachable in run() and missing here

30
backend/cmd/tmctl/main.go
View file

@ -1,5 +1,5 @@
// tmctl is the TextMachine CLI: translate / report / status / export / redrive / manifest / backup /
// migrate / seed-lint / bank-apply (dispatchCommands).
// tmctl is the TextMachine CLI: translate / report / status / export / build / redrive / manifest /
// backup / migrate / seed-lint / bank-apply (dispatchCommands).
// main.go — thin wiring (package №4): argument parsing — invocation.go,
// output renderers — render.go, .env — dotenv.go; here just the
// «parse → env → ctx → fetch → render» wiring and exit-code mapping.
@ -72,6 +72,7 @@ const (
exitSchemaMismatch = 13 // this project's schema is not this binary's: run `tmctl migrate` (row 174)
exitDecisionsRejected = 14 // the decision document was read and declined; the USER re-decides (D39.156)
exitWriteIncomplete = 15 // the document was accepted and the write did not complete; the report says which file landed — re-send the same document
exitBookIncomplete = 16 // `build` refused to write a book with holes; the message lists them — finish the book, or ask for the marked copy with --partial
exitRefusedOther = 19 // a refusal class this build of tmctl has no number for
)
@ -83,6 +84,7 @@ var refusalExit = map[pipeline.RefusalClass]int{
pipeline.RefusalSchemaMismatch: exitSchemaMismatch,
pipeline.RefusalDecisionsRejected: exitDecisionsRejected,
pipeline.RefusalWriteIncomplete: exitWriteIncomplete,
pipeline.RefusalBookIncomplete: exitBookIncomplete,
}
// exitCode maps a run() error onto the ratified shell contract (Milestone 2 / R1-FL-A):
@ -179,7 +181,7 @@ func traceID() string {
// because the usage line and the unknown-command hint were two hand-maintained copies of it, and the
// usage line fell three commands behind (row 176) while the hint stayed current.
var dispatchCommands = []string{
"translate", "report", "status", "export", "redrive", "manifest", "backup", "migrate", "seed-lint",
"translate", "report", "status", "export", "build", "redrive", "manifest", "backup", "migrate", "seed-lint",
"bank-apply",
}
@ -226,6 +228,8 @@ func run() error {
return status(ctx, inv.cfgPath, inv.asJSON)
case "export":
return export(inv.cfgPath, inv.asPlaintext, inv.asPairs)
case "build":
return build(inv.cfgPath, inv.buildFormats, inv.buildOut, inv.buildPartial)
case "redrive":
// A dry-run redrive spends nothing and mutates nothing; only a real redrive re-attacks and re-bills.
if !inv.sel.DryRun {
@ -360,6 +364,26 @@ func export(cfgPath string, asPlaintext, asPairs bool) error {
return renderExport(os.Stdout, exp, asPlaintext)
}
// build writes the READER's copy of the book (backlog row 236): EPUB and/or plain text from the same
// export projection `export` prints, beside the project database (pipeline.BuildBook). $0 and key-less
// like export — it reads the store and writes a file; no provider is called. A book with any hole is
// refused (exit 16, the holes listed) unless --partial asks for the marked copy. Its own verb rather than a
// flag on `export`: `export` is a pure read whose stdout IS its output, and a command that writes files
// beside the database is a different thing to script around (the `manifest` precedent).
func build(cfgPath string, formats []string, out string, partial bool) error {
r, err := pipeline.NewReadOnlyRunner(cfgPath, obs.NewLogger())
if err != nil {
return err
}
defer r.Close()
rep, err := r.BuildBook(pipeline.BuildOptions{Formats: formats, Out: out, Partial: partial})
if err != nil {
return err
}
return renderBuild(os.Stdout, rep)
}
// status prints the READ-ONLY progress projection (D15.3): 0 LLM, 0 replay. --json emits the
// StatusReport with stable disposition/flag_reason enums for CI/IDE; the default is a human
// dashboard (unit counts, per-chapter quality passports, money, secondary ETA). BOTH modes exit 2

23
backend/cmd/tmctl/render.go
View file

@ -570,16 +570,9 @@ func renderExport(w io.Writer, exp *pipeline.BookExport, asPlaintext bool) error
// (exported = total − pending), so a book that shipped nothing for two units still
// announced them as exported — the same lie as the mislabelled banner below, told in
// numbers: a reader who trusts the summary never learns to look.
withheld, incomplete := 0, 0
for _, ce := range exp.Chunks {
switch {
case ce.Disposition == "pending":
case ce.FinalText == "":
withheld++
case ce.DroppedMembers > 0:
incomplete++
}
}
// The predicate is pipeline.UnitHole — ONE definition shared with the book writer, so the
// numbers this header prints and the holes `tmctl build` marks can never disagree.
withheld, incomplete := pipeline.HoleCounts(exp)
// `incomplete` is a SUBSET of `exported`, not a fourth part of the total: those units DID ship
// text, with a piece of it missing. Spelled that way so a reader adding the numbers up is not
// misled into thinking they partition the book.
@ -636,6 +629,16 @@ func flagParen(r string) string {
return " (" + r + ")"
}
// renderBuild prints the build report as indented JSON: the versioned envelope every JSON output of the
// engine carries (17-seam-inbound-law п.3), and the surface the platform's export door will read the
// file paths from. JSON only: the consumer is a caller that opens the files, and the human reads paths
// out of JSON as well as out of prose.
func renderBuild(w io.Writer, rep *pipeline.BuildReport) error {
enc := json.NewEncoder(w)
enc.SetIndent("", " ")
return enc.Encode(rep)
}
// renderManifest prints the persisted chapter/chunk manifest (row 100). The default is a human summary —
// a 2283-chapter tree is not something anyone reads in a terminal — and --json emits the document itself,
// byte-identical to the sidecar, so a consumer can pipe it instead of knowing where the file lives.

21
backend/configs/langpacks/ru/reader.txt Normal file
View file

@ -0,0 +1,21 @@
# Reader-facing words for a RUSSIAN-language book file (internal/lang/reader.go). These are RENDER data —
# what the file itself says to its reader at a hole in the text and on the first page of a book that is not
# whole. They are keyed by the TARGET language (the language the reader reads), not by the pair.
#
# ⚠ NOT part of the language pack: this directory is read by lang.LoadReaderWords only. The pack loader
# (lang.Load) reads <source>/ and <source>-<target>/ and folds those bytes into Pack.Version() and thus the
# run snapshot; a target directory is outside it by construction, so an edit here moves no snapshot and
# re-bills no book. Keep it that way — a wording change must never cost a re-translation.
#
# `key<TAB>value` per line; every key is required. Placeholders in braces are filled by the engine:
# {chapter} {unit} — the hole's coordinates (dense chapter number, output unit index within it)
# {dropped} — how many source fragments of the unit are missing from its text
# {ghost} — how many translated fragments the current cut of the book cannot place
# {holes} {total} — units with a hole / units in the book
hole.pending ⚠ Этот фрагмент ещё не переведён.
hole.withheld ⚠ Перевод этого фрагмента не выпущен: он требует проверки человеком.
hole.incomplete ⚠ В этом фрагменте недостаёт части текста ({dropped}).
hole.stale ⚠ Оригинал этого фрагмента изменился после перевода; фрагмент ждёт нового перевода.
hole.ghost ⚠ Часть перевода этой главы ({ghost}) не вошла в файл: книга была перенарезана после перевода.
notice.holes ⚠ Перевод не завершён: фрагментов с пропусками — {holes} из {total}. Каждый пропуск отмечен в тексте знаком ⚠.
notice.ghost ⚠ Книга была перенарезана после перевода: фрагментов перевода вне текущей нарезки — {ghost}; они в файл не вошли.

344
backend/internal/bookfile/bookfile_test.go Normal file
View file

@ -0,0 +1,344 @@
package bookfile
import (
"archive/zip"
"bytes"
"encoding/xml"
"io"
"regexp"
"strings"
"testing"
"time"
)
// bookfile_test.go pins the writer's two load-bearing properties — determinism and a container the
// engine's own reader round-trips — and the EPUB 3 shape epubcheck was run against (the stand proof).
func sampleBook() *Book {
return &Book{
Identifier: "urn:textmachine:book:test",
Title: "Тест & <проба>",
Language: "ru",
Modified: time.Date(2026, 8, 24, 1, 15, 0, 0, time.UTC),
Chapters: []Chapter{
{Title: "Глава 1", Paragraphs: []string{"Первый абзац.", "Второй — с «кавычками» & <тегом>."}},
{Title: "2", Paragraphs: []string{"Одна строка."}},
},
}
}
func epubBytes(t *testing.T, b *Book) []byte {
t.Helper()
var buf bytes.Buffer
if err := WriteEPUB(&buf, b); err != nil {
t.Fatal(err)
}
return buf.Bytes()
}
func zipEntries(t *testing.T, data []byte) (*zip.Reader, map[string]string) {
t.Helper()
zr, err := zip.NewReader(bytes.NewReader(data), int64(len(data)))
if err != nil {
t.Fatal(err)
}
out := map[string]string{}
for _, f := range zr.File {
rc, err := f.Open()
if err != nil {
t.Fatal(err)
}
b, _ := io.ReadAll(rc)
rc.Close()
out[f.Name] = string(b)
}
return zr, out
}
func TestContainerIsOCFConformant(t *testing.T) {
data := epubBytes(t, sampleBook())
zr, entries := zipEntries(t, data)
// The mimetype's LOCAL header carries its CRC and sizes itself: general-purpose flag bit 3 (a data
// descriptor after the entry) is clear, and the extra-field length is zero — the plain form a reader
// that identifies the file by its first bytes expects. Bytes 6–7 are the flags, 28–29 the extra length.
if flags := uint16(data[6]) | uint16(data[7])<<8; flags&0x8 != 0 {
t.Errorf("mimetype local header has the data-descriptor flag set (flags=%#x)", flags)
}
if extra := uint16(data[28]) | uint16(data[29])<<8; extra != 0 {
t.Errorf("mimetype local header carries an extra field of %d bytes", extra)
}
if string(data[30:38]) != "mimetype" || string(data[38:58]) != "application/epub+zip" {
t.Errorf("the mimetype entry is not at the fixed OCF offsets: %q", data[30:58])
}
if zr.File[0].Name != "mimetype" || zr.File[0].Method != zip.Store {
t.Fatalf("mimetype must be the first, STORED entry; got %q method %d", zr.File[0].Name, zr.File[0].Method)
}
if entries["mimetype"] != "application/epub+zip" {
t.Fatalf("mimetype = %q", entries["mimetype"])
}
if !strings.Contains(entries["META-INF/container.xml"], `full-path="OEBPS/content.opf"`) {
t.Fatalf("container.xml must point at the OPF:\n%s", entries["META-INF/container.xml"])
}
for _, f := range zr.File {
if !f.Modified.IsZero() && f.Modified.Unix() != -2208988800 { // zero MS-DOS time decodes to 1980-00-00 → 1979/1980 epochs vary; a real clock reading is what is forbidden
if f.Modified.Year() > 1981 {
t.Errorf("entry %s carries a clock timestamp %v — the file would differ between builds", f.Name, f.Modified)
}
}
}
}
type opfDoc struct {
UniqueID string `xml:"unique-identifier,attr"`
Metadata struct {
Identifier struct {
ID string `xml:"id,attr"`
Value string `xml:",chardata"`
} `xml:"identifier"`
Title string `xml:"title"`
Language string `xml:"language"`
Description string `xml:"description"`
Meta []struct {
Property string `xml:"property,attr"`
Value string `xml:",chardata"`
} `xml:"meta"`
} `xml:"metadata"`
Manifest []struct {
ID string `xml:"id,attr"`
Href string `xml:"href,attr"`
MediaType string `xml:"media-type,attr"`
Properties string `xml:"properties,attr"`
} `xml:"manifest>item"`
Spine []struct {
IDRef string `xml:"idref,attr"`
} `xml:"spine>itemref"`
}
func parseOPF(t *testing.T, entries map[string]string) opfDoc {
t.Helper()
var doc opfDoc
if err := xml.Unmarshal([]byte(entries["OEBPS/content.opf"]), &doc); err != nil {
t.Fatalf("OPF does not parse: %v\n%s", err, entries["OEBPS/content.opf"])
}
return doc
}
func TestEPUBCarriesTheFiveRequiredThingsAndKeepsNavOutOfTheSpine(t *testing.T) {
b := sampleBook()
b.Description = "⚠ 1/2"
_, entries := zipEntries(t, epubBytes(t, b))
doc := parseOPF(t, entries)
if doc.Metadata.Identifier.Value != b.Identifier || doc.Metadata.Identifier.ID != doc.UniqueID {
t.Errorf("dc:identifier %+v must be the one unique-identifier=%q names", doc.Metadata.Identifier, doc.UniqueID)
}
if doc.Metadata.Title != b.Title {
t.Errorf("dc:title = %q, want %q (escaped in the file, intact when parsed)", doc.Metadata.Title, b.Title)
}
padded := sampleBook()
padded.Title = " T "
if _, e := zipEntries(t, epubBytes(t, padded)); !strings.Contains(e["OEBPS/content.opf"], "<dc:title>T</dc:title>") {
t.Errorf("dc:title must be trimmed like the text file's title line:\n%s", e["OEBPS/content.opf"])
}
if doc.Metadata.Language != "ru" {
t.Errorf("dc:language = %q", doc.Metadata.Language)
}
if doc.Metadata.Description != b.Description {
t.Errorf("dc:description = %q, want %q", doc.Metadata.Description, b.Description)
}
modified := ""
for _, m := range doc.Metadata.Meta {
if m.Property == "dcterms:modified" {
modified = m.Value
}
}
if modified != "2026-08-24T01:15:00Z" {
t.Errorf("dcterms:modified = %q, want the CCYY-MM-DDThh:mm:ssZ render of Book.Modified", modified)
}
navID := ""
for _, it := range doc.Manifest {
if it.Properties == "nav" {
navID = it.ID
if _, ok := entries["OEBPS/"+it.Href]; !ok {
t.Errorf("the nav item points at %q, which is not in the zip", it.Href)
}
}
}
if navID == "" {
t.Fatal("no manifest item carries properties=\"nav\" — EPUB 3 requires a navigation document")
}
if len(doc.Spine) != len(b.Chapters) {
t.Fatalf("spine has %d itemrefs, want one per chapter (%d) and nothing else", len(doc.Spine), len(b.Chapters))
}
for _, ref := range doc.Spine {
if ref.IDRef == navID {
t.Fatal("the nav document is in the spine — the engine's reader would read it back as a chapter")
}
}
nav := entries["OEBPS/nav.xhtml"]
for _, ch := range b.Chapters {
if !strings.Contains(nav, ">"+xmlText(ch.Title)+"</a>") {
t.Errorf("nav lacks an entry titled %q:\n%s", ch.Title, nav)
}
}
}
// blocksOf pulls the text of every <h1>/<p> of a chapter document, in order — what the engine's reader
// turns into paragraphs.
var blockRE = regexp.MustCompile(`(?s)<(h1|p)>(.*?)</(?:h1|p)>`)
func blocksOf(doc string) []string {
var out []string
for _, m := range blockRE.FindAllStringSubmatch(doc, -1) {
out = append(out, m[2])
}
return out
}
func TestChapterDocumentsCarryExactlyBlocks(t *testing.T) {
b := sampleBook()
b.Notice = []string{"⚠ 1/2", "⚠ +3"}
_, entries := zipEntries(t, epubBytes(t, b))
for i := range b.Chapters {
got := blocksOf(entries["OEBPS/"+chapterEntry(i)])
want := b.Blocks(i)
if len(got) != len(want) {
t.Fatalf("chapter %d: %d blocks in the document, Blocks() says %d\n%v\n%v", i+1, len(got), len(want), got, want)
}
for j := range want {
if got[j] != xmlText(want[j]) {
t.Errorf("chapter %d block %d = %q, want %q", i+1, j, got[j], xmlText(want[j]))
}
}
}
// The notice is in the FIRST document only, ahead of its title.
if first := blocksOf(entries["OEBPS/ch1.xhtml"]); first[0] != "⚠ 1/2" || first[2] != xmlText("Глава 1") {
t.Errorf("first document must open with the notice, then the title: %v", first)
}
if second := blocksOf(entries["OEBPS/ch2.xhtml"]); second[0] != "2" {
t.Errorf("second document must open with its title, no notice: %v", second)
}
}
func TestEPUBAndTXTAreDeterministic(t *testing.T) {
b := sampleBook()
first, second := epubBytes(t, b), epubBytes(t, b)
if !bytes.Equal(first, second) {
t.Fatal("two EPUB builds of one Book differ")
}
var t1, t2 bytes.Buffer
if err := WriteTXT(&t1, b); err != nil {
t.Fatal(err)
}
if err := WriteTXT(&t2, b); err != nil {
t.Fatal(err)
}
if !bytes.Equal(t1.Bytes(), t2.Bytes()) {
t.Fatal("two TXT builds of one Book differ")
}
}
func TestTextIsEscapedAndControlCharactersAreDropped(t *testing.T) {
b := sampleBook()
b.Chapters[0].Paragraphs = []string{"a < b & c > d \"q\"", "bad\x00\x0bchars\x7f kept"}
if got := CleanText("bad\x00\x0bchars\x7f kept\xff"); got != "badchars\x7f kept\uFFFD" {
t.Errorf("CleanText = %q", got)
}
_, entries := zipEntries(t, epubBytes(t, b))
doc := entries["OEBPS/ch1.xhtml"]
for _, want := range []string{"a &lt; b &amp; c &gt; d &quot;q&quot;", "badchars\x7f kept"} {
if !strings.Contains(doc, want) {
t.Errorf("document lacks %q:\n%s", want, doc)
}
}
if strings.ContainsAny(doc, "\x00\x0b") {
t.Error("an XML-illegal control character reached the document")
}
if !strings.Contains(entries["OEBPS/content.opf"], "<dc:title>Тест &amp; &lt;проба&gt;</dc:title>") {
t.Errorf("title not escaped:\n%s", entries["OEBPS/content.opf"])
}
}
func TestWritersRefuseAnInvalidBook(t *testing.T) {
cases := map[string]func(*Book){
"no chapters": func(b *Book) { b.Chapters = nil },
"empty title": func(b *Book) { b.Chapters[1].Title = " " },
// Judged after CleanText: a title of control characters only would render as an empty <h1> and
// an empty nav link — the invalid EPUB validate exists to refuse, not something to write.
"control-only chapter title": func(b *Book) { b.Chapters[1].Title = "\x01\x02 \x1f" },
"control-only book title": func(b *Book) { b.Title = "\x03\x0b" },
"zero modified": func(b *Book) { b.Modified = time.Time{} },
"no language": func(b *Book) { b.Language = "" },
"no identifier": func(b *Book) { b.Identifier = "" },
"empty book title": func(b *Book) { b.Title = "" },
}
for name, mutate := range cases {
b := sampleBook()
mutate(b)
if err := WriteEPUB(io.Discard, b); err == nil {
t.Errorf("%s: WriteEPUB must refuse", name)
}
if err := WriteTXT(io.Discard, b); err == nil {
t.Errorf("%s: WriteTXT must refuse", name)
}
}
}
func TestTXTShape(t *testing.T) {
b := sampleBook()
b.Notice = []string{"⚠ 1/2"}
var buf bytes.Buffer
if err := WriteTXT(&buf, b); err != nil {
t.Fatal(err)
}
want := "Тест & <проба>\n\n⚠ 1/2\n\n\nГлава 1\n\nПервый абзац.\n\nВторой — с «кавычками» & <тегом>.\n\n\n2\n\nОдна строка.\n"
if buf.String() != want {
t.Fatalf("txt =\n%q\nwant\n%q", buf.String(), want)
}
for _, banned := range []string{"=== CHAPTER", "TEXT MISSING", "pending", "flagged"} {
if strings.Contains(buf.String(), banned) {
t.Errorf("operator vocabulary %q in the reader's text", banned)
}
}
}
// TestTitlesAreCleanedTheSameInBothFormats pins format parity on the strings the assembler does NOT
// pre-clean — the book title, the chapter titles, the notice: the EPUB drops a control character through
// xmlText, and the text file must drop it the same way (acceptance finding: TXT wrote titles raw).
func TestTitlesAreCleanedTheSameInBothFormats(t *testing.T) {
b := sampleBook()
b.Title = "Заглавие\x01книги"
b.Notice = []string{"⚠\x03 1/2"}
b.Chapters[0].Title = "Гла\x02ва 1"
b.Chapters[0].Paragraphs = []string{"Абзац\x1fс контролем."}
_, entries := zipEntries(t, epubBytes(t, b))
doc := parseOPF(t, entries)
if doc.Metadata.Title != "Заглавиекниги" {
t.Errorf("EPUB dc:title = %q", doc.Metadata.Title)
}
if blocks := blocksOf(entries["OEBPS/ch1.xhtml"]); len(blocks) < 3 || blocks[0] != "⚠ 1/2" || blocks[1] != "Глава 1" || blocks[2] != "Абзацс контролем." {
t.Errorf("EPUB blocks = %q", blocks)
}
var txt bytes.Buffer
if err := WriteTXT(&txt, b); err != nil {
t.Fatal(err)
}
want := "Заглавиекниги\n\n⚠ 1/2\n\n\nГлава 1\n\nАбзацс контролем.\n\n\n2\n\nОдна строка.\n"
if txt.String() != want {
t.Errorf("TXT =\n%q\nwant\n%q", txt.String(), want)
}
for name, out := range map[string]string{"epub opf": entries["OEBPS/content.opf"], "epub nav": entries["OEBPS/nav.xhtml"], "epub ch1": entries["OEBPS/ch1.xhtml"], "txt": txt.String()} {
if strings.ContainsAny(out, "\x01\x02\x03\x1f") {
t.Errorf("%s carries a control character the other format drops", name)
}
}
}
func TestOneChapterBook(t *testing.T) {
b := &Book{Identifier: "urn:x", Title: "T", Language: "en", Modified: time.Unix(0, 0).UTC(),
Chapters: []Chapter{{Title: "1"}}} // a chapter with a title and no paragraphs is a legal, empty chapter
_, entries := zipEntries(t, epubBytes(t, b))
doc := parseOPF(t, entries)
if len(doc.Spine) != 1 || len(blocksOf(entries["OEBPS/ch1.xhtml"])) != 1 {
t.Fatalf("one-chapter book: spine %d, blocks %v", len(doc.Spine), blocksOf(entries["OEBPS/ch1.xhtml"]))
}
}

77
backend/internal/bookfile/container.go Normal file
View file

@ -0,0 +1,77 @@
package bookfile
import (
"archive/zip"
"fmt"
"hash/crc32"
"io"
)
// container.go: the OCF container — the zip layout every EPUB shares, independent of what the package
// inside says. It is the one place the layout is spelled, for the shipping writer (epub.go) and for the
// ingest test fixture (internal/chunk/chunktest) alike.
// mimetypeBody is the fixed content of the OCF `mimetype` entry.
const mimetypeBody = "application/epub+zip"
// Container is an OCF container under construction. Entries land in the order they are added, with
// ZERO timestamps (archive/zip leaves the MS-DOS date fields at zero and writes no extended-timestamp
// extra field when FileHeader.Modified is unset), so the bytes depend on the entries alone — never on the
// clock or on a filesystem — which is what makes a build reproducible across processes.
type Container struct {
zw *zip.Writer
}
// NewContainer starts an OCF container on w whose package document (the OPF) will be added at
// rootfile — a path inside the zip such as "OEBPS/content.opf". It writes the two entries OCF fixes:
// `mimetype` FIRST and STORED (uncompressed, so a reader can identify the file by its first bytes),
// then META-INF/container.xml pointing at rootfile. The caller adds rootfile itself with Add.
func NewContainer(w io.Writer, rootfile string) (*Container, error) {
zw := zip.NewWriter(w)
// CreateRaw with the CRC and sizes given up front: the local header then carries them itself and
// no data descriptor follows the entry. OCF's letter is met either way, but a reader that
// identifies the file by its first bytes wants the plain form, and the 20 bytes cost nothing.
body := []byte(mimetypeBody)
mw, err := zw.CreateRaw(&zip.FileHeader{
Name: "mimetype", Method: zip.Store, CRC32: crc32.ChecksumIEEE(body),
CompressedSize64: uint64(len(body)), UncompressedSize64: uint64(len(body)),
})
if err != nil {
return nil, fmt.Errorf("bookfile: container mimetype: %w", err)
}
if _, err := mw.Write(body); err != nil {
return nil, fmt.Errorf("bookfile: container mimetype: %w", err)
}
c := &Container{zw: zw}
container := `<?xml version="1.0" encoding="UTF-8"?>
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
<rootfiles>
<rootfile full-path="` + xmlText(rootfile) + `" media-type="application/oebps-package+xml"/>
</rootfiles>
</container>
`
if err := c.Add("META-INF/container.xml", []byte(container)); err != nil {
return nil, err
}
return c, nil
}
// Add writes one entry (deflated, zero timestamp) at name — a slash-separated path inside the zip.
func (c *Container) Add(name string, data []byte) error {
w, err := c.zw.CreateHeader(&zip.FileHeader{Name: name, Method: zip.Deflate})
if err != nil {
return fmt.Errorf("bookfile: container entry %s: %w", name, err)
}
if _, err := w.Write(data); err != nil {
return fmt.Errorf("bookfile: container entry %s: %w", name, err)
}
return nil
}
// Close finishes the zip (central directory). The container is complete only after it returns nil.
func (c *Container) Close() error {
if err := c.zw.Close(); err != nil {
return fmt.Errorf("bookfile: close container: %w", err)
}
return nil
}

139
backend/internal/bookfile/epub.go Normal file
View file

@ -0,0 +1,139 @@
package bookfile
import (
"fmt"
"io"
"strings"
"time"
)
// epub.go: the EPUB 3 package. What the format REQUIRES and this writer supplies, in order: a package
// identifier (dc:identifier, referenced by unique-identifier), dc:title, dc:language, a
// dcterms:modified in the CCYY-MM-DDThh:mm:ssZ form, and a navigation document (the manifest item with
// properties="nav"). Each chapter is one XHTML content document in the spine.
//
// ⚠ The nav document is NOT in the spine, and there is no title page document either. The engine's own
// reader (internal/chunk/ingest.go) turns EVERY spine document into a chapter, an empty one included, so
// a nav or a title page in the spine would come back as a chapter of its own and the loaded↔received
// circle would not close. The book-level notice therefore rides inside the FIRST chapter document, ahead
// of its title (Book.Notice / Blocks).
const (
opfPath = "OEBPS/content.opf"
navEntry = "nav.xhtml"
// modifiedLayout is the ONLY form EPUB 3 admits for dcterms:modified (a UTC timestamp to the second).
modifiedLayout = "2006-01-02T15:04:05Z"
)
// chapterEntry names chapter i's document (0-based) inside OEBPS/. The name carries the dense chapter
// number; reading order is the spine's, not the name's.
func chapterEntry(i int) string { return fmt.Sprintf("ch%d.xhtml", i+1) }
// chapterID is chapter i's manifest id.
func chapterID(i int) string { return fmt.Sprintf("ch%d", i+1) }
// WriteEPUB writes b as an EPUB 3 file to w. It is deterministic: the same Book yields the same bytes.
func WriteEPUB(w io.Writer, b *Book) error {
if err := b.validate(); err != nil {
return err
}
c, err := NewContainer(w, opfPath)
if err != nil {
return err
}
if err := c.Add(opfPath, []byte(packageDocument(b))); err != nil {
return err
}
if err := c.Add("OEBPS/"+navEntry, []byte(navDocument(b))); err != nil {
return err
}
for i := range b.Chapters {
if err := c.Add("OEBPS/"+chapterEntry(i), []byte(chapterDocument(b, i))); err != nil {
return err
}
}
return c.Close()
}
// packageDocument renders the OPF.
func packageDocument(b *Book) string {
lang := xmlText(b.Language)
var sb strings.Builder
sb.WriteString(`<?xml version="1.0" encoding="UTF-8"?>
<package xmlns="http://www.idpf.org/2007/opf" version="3.0" unique-identifier="pub-id" xml:lang="` + lang + `">
<metadata xmlns:dc="http://purl.org/dc/elements/1.1/">
<dc:identifier id="pub-id">` + xmlText(b.Identifier) + `</dc:identifier>
<dc:title>` + xmlText(strings.TrimSpace(b.Title)) + `</dc:title>
<dc:language>` + lang + `</dc:language>
<meta property="dcterms:modified">` + b.Modified.UTC().Format(modifiedLayout) + `</meta>
`)
if b.Description != "" {
sb.WriteString(` <dc:description>` + xmlText(b.Description) + `</dc:description>` + "\n")
}
sb.WriteString(` </metadata>
<manifest>
<item id="nav" href="` + navEntry + `" media-type="application/xhtml+xml" properties="nav"/>
`)
for i := range b.Chapters {
sb.WriteString(` <item id="` + chapterID(i) + `" href="` + chapterEntry(i) + `" media-type="application/xhtml+xml"/>` + "\n")
}
sb.WriteString(" </manifest>\n <spine>\n")
for i := range b.Chapters {
sb.WriteString(` <itemref idref="` + chapterID(i) + `"/>` + "\n")
}
sb.WriteString(" </spine>\n</package>\n")
return sb.String()
}
// navDocument renders the EPUB 3 navigation document: a toc nav whose entries are the chapters, in spine
// order, each linking to its document by the chapter's Title.
func navDocument(b *Book) string {
lang := xmlText(b.Language)
var sb strings.Builder
sb.WriteString(`<?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:epub="http://www.idpf.org/2007/ops" xml:lang="` + lang + `" lang="` + lang + `">
<head>
<title>` + xmlText(strings.TrimSpace(b.Title)) + `</title>
</head>
<body>
<nav epub:type="toc">
<h1>` + xmlText(strings.TrimSpace(b.Title)) + `</h1>
<ol>
`)
for i, ch := range b.Chapters {
sb.WriteString(` <li><a href="` + chapterEntry(i) + `">` + xmlText(ch.Title) + `</a></li>` + "\n")
}
sb.WriteString(" </ol>\n </nav>\n</body>\n</html>\n")
return sb.String()
}
// chapterDocument renders chapter i as an XHTML content document: the blocks of Blocks(i), the title as
// <h1> and every other block as <p>. One block = one block-level element, which is what the engine's
// reader turns back into one paragraph.
func chapterDocument(b *Book, i int) string {
lang := xmlText(b.Language)
ch := b.Chapters[i]
var sb strings.Builder
sb.WriteString(`<?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="` + lang + `" lang="` + lang + `">
<head>
<title>` + xmlText(ch.Title) + `</title>
</head>
<body>
`)
if i == 0 {
for _, p := range b.Notice {
sb.WriteString(" <p>" + xmlText(p) + "</p>\n")
}
}
sb.WriteString(" <h1>" + xmlText(ch.Title) + "</h1>\n")
for _, p := range ch.Paragraphs {
sb.WriteString(" <p>" + xmlText(p) + "</p>\n")
}
sb.WriteString("</body>\n</html>\n")
return sb.String()
}
// Modified renders t the way dcterms:modified is written, for a caller that wants to show the value it
// is about to put in the file.
func Modified(t time.Time) string { return t.UTC().Format(modifiedLayout) }

165
backend/internal/bookfile/model.go Normal file
View file

@ -0,0 +1,165 @@
// Package bookfile writes a translated book as a READER's file — the artifact the engine hands out — from
// a neutral document model: EPUB 3 for a reading app (epub.go) and plain text (txt.go), both from the same
// Book. It is the engine's only writer of a book, and it is deliberately language- and pair-blind: every
// word a reader sees (a chapter title, a hole marker, the not-whole notice) arrives in the model as DATA
// decided upstream; nothing here knows what a chapter is called in any language (12-go-style-notes §0).
//
// Two properties are load-bearing and proven by tests rather than promised:
// - DETERMINISM: the same Book yields byte-identical files across processes. Nothing reads a clock or
// a filesystem timestamp; the zip carries zero timestamps; entries are written in a fixed order.
// - ROUND TRIP: the engine's own ingest (internal/chunk) reads an EPUB written here back into exactly
// the blocks Blocks() lays down, chapter by chapter — which is what closes the "loaded → received"
// circle the reader path was built on.
//
// The OCF container half (container.go) is ALSO what the ingest test fixture builds on
// (internal/chunk/chunktest): one zip/OCF builder, two callers, so the fixture the reader is pinned
// against and the file the writer ships cannot drift apart in the container layer.
package bookfile
import (
"errors"
"fmt"
"strings"
"time"
)
// Formats are the file formats this package writes, in the order a build produces them. A format name
// is also the file extension.
var Formats = []string{"epub", "txt"}
// KnownFormat reports whether name is one of Formats.
func KnownFormat(name string) bool {
for _, f := range Formats {
if f == name {
return true
}
}
return false
}
// Book is the document model both writers render.
type Book struct {
// Identifier is the package identifier (dc:identifier, the OPF's unique-identifier). It is derived
// from the book's id by the caller; never minted from a clock or a random source.
Identifier string
// Title is dc:title, spelled exactly as the book config spells it. It may be the source-language
// display name — the writer has no translation of it and invents none.
Title string
// Language is dc:language and the documents' xml:lang: the TARGET language tag, from the book's data.
Language string
// Modified is dcterms:modified. It is a value from the run's data, never the wall clock, so that two
// builds of one store are byte-identical (a zero value is rejected: a missing fact must be decided
// upstream, not silently rendered as year one).
Modified time.Time
// Description is dc:description; "" omits the element. The assembler puts the not-whole notice here
// so a library shelf that shows descriptions shows it too.
Description string
// Notice are the book-level paragraphs a reader sees on OPENING the book — before the first chapter's
// title, inside the first spine document (a separate title document would be read by ingest as a
// chapter of its own), and at the head of the text file. Empty for a book with nothing to declare.
Notice []string
// Chapters in reading order; at least one (an EPUB with an empty spine is not an EPUB).
Chapters []Chapter
}
// Chapter is one spine document.
type Chapter struct {
// Title is the chapter's heading: the <h1> of its document, the text of its nav entry, its heading
// line in the text file. Required — an empty nav link is an invalid EPUB — and decided upstream
// (the engine's deterministic heading, or the bare chapter number when there is none).
Title string
// Paragraphs are the chapter's text blocks in order, one <p> each. A hole marker is an ordinary
// paragraph here: the model carries what the assembler decided to say, and nothing about why.
Paragraphs []string
}
// Blocks returns chapter i's text blocks in the order its document carries them: the book Notice (first
// chapter only), the Title, then the Paragraphs. It is the ONE definition the EPUB writer emits from and
// the round-trip proof compares against — what ingest reads back as paragraphs is exactly this list.
func (b *Book) Blocks(i int) []string {
ch := b.Chapters[i]
out := make([]string, 0, len(b.Notice)+1+len(ch.Paragraphs))
if i == 0 {
out = append(out, b.Notice...)
}
out = append(out, ch.Title)
out = append(out, ch.Paragraphs...)
return out
}
// validate refuses a Book the writers could only render into an invalid file. Every failure is the
// caller's to fix upstream; the writers never paper over one. Titles are judged AFTER CleanText — the
// form they reach the file in: a title of nothing but control characters renders as an empty dc:title
// or an empty nav link, and both are invalid EPUB.
func (b *Book) validate() error {
switch {
case b.Identifier == "":
return errors.New("bookfile: the book has no identifier")
case strings.TrimSpace(CleanText(b.Title)) == "":
return errors.New("bookfile: the book has no title")
case strings.TrimSpace(b.Language) == "":
return errors.New("bookfile: the book has no language")
case b.Modified.IsZero():
return errors.New("bookfile: the book has no modification time (it must come from the run's data, not be left unset)")
case len(b.Chapters) == 0:
return errors.New("bookfile: the book has no chapters (an empty spine is not an EPUB)")
}
for i, ch := range b.Chapters {
if strings.TrimSpace(CleanText(ch.Title)) == "" {
return fmt.Errorf("bookfile: chapter %d has no title (an empty nav entry is an invalid EPUB)", i+1)
}
}
return nil
}
// CleanText makes s carriable by BOTH formats: invalid UTF-8 replaced, and the code points XML 1.0
// forbids (C0 controls other than tab/newline/return, U+FFFE/U+FFFF) dropped — a control character that
// reached the export text would make an XHTML chapter unparseable to every reader, and is junk in a text
// file too. The assembler applies it to every paragraph once, so the EPUB and the text file say the same.
func CleanText(s string) string {
s = strings.ToValidUTF8(s, "\uFFFD")
if !strings.ContainsFunc(s, isXMLIllegal) {
return s
}
var b strings.Builder
b.Grow(len(s))
for _, r := range s {
if !isXMLIllegal(r) {
b.WriteRune(r)
}
}
return b.String()
}
func isXMLIllegal(r rune) bool {
switch {
case r == '\t' || r == '\n' || r == '\r':
return false
case r < 0x20, r == 0xFFFE, r == 0xFFFF:
return true
}
return false
}
// xmlText renders s as XML character data: CleanText, then the three markup characters escaped, plus the
// quote so the same function serves attribute values.
func xmlText(s string) string {
s = CleanText(s)
var b strings.Builder
b.Grow(len(s))
for _, r := range s {
switch r {
case '&':
b.WriteString("&amp;")
case '<':
b.WriteString("&lt;")
case '>':
b.WriteString("&gt;")
case '"':
b.WriteString("&quot;")
default:
b.WriteRune(r)
}
}
return b.String()
}

37
backend/internal/bookfile/txt.go Normal file
View file

@ -0,0 +1,37 @@
package bookfile
import (
"io"
"strings"
)
// txt.go: the plain-text book. No audit banners, no flags in brackets, no operator vocabulary — the file
// is prose plus what the model carries: the title first, the notice (when there is one), then every
// chapter as its title line followed by its paragraphs. Blocks are separated by ONE blank line, chapters
// by TWO, so a chapter boundary reads differently from a paragraph boundary even in a viewer that shows
// nothing but text. The file ends with a newline.
//
// The same block list the EPUB renders (Blocks) is what lands here, so the two formats say the same thing —
// cleaned the same way too: every string the EPUB passes through xmlText goes through CleanText here, so
// a control character in a title or a notice cannot reach the text reader when the EPUB reader is spared it.
// WriteTXT writes b as plain UTF-8 text to w. Deterministic.
func WriteTXT(w io.Writer, b *Book) error {
if err := b.validate(); err != nil {
return err
}
var sb strings.Builder
sb.WriteString(strings.TrimSpace(CleanText(b.Title)))
for _, p := range b.Notice {
sb.WriteString("\n\n" + CleanText(p))
}
for _, ch := range b.Chapters {
sb.WriteString("\n\n\n" + CleanText(ch.Title))
for _, p := range ch.Paragraphs {
sb.WriteString("\n\n" + CleanText(p))
}
}
sb.WriteString("\n")
_, err := io.WriteString(w, sb.String())
return err
}

40
backend/internal/chunk/chunktest/epub.go
View file

@ -2,14 +2,22 @@
// package because two packages need the SAME epub shape: the ingest unit tests and the runner's
// end-to-end tests. Duplicating a zip/OPF builder in both would let the two drift apart, and the
// whole point of the e2e fixture is that it is the file the unit tests already pin.
//
// The CONTAINER layer (mimetype first and stored, container.xml, the entry writes) is not built here
// any more: it is internal/bookfile's, the same code the engine's book writer ships an EPUB through, so
// the fixture the reader is pinned against and the file the writer produces cannot drift apart at the
// zip level. What stays here is deliberately the TEST half — the bare OPF stub, the decoy <title>/<style>
// in every chapter head that ingest_test proves are dropped, the MType/EntryName knobs for malformed
// manifests — none of which may ever reach a reader.
package chunktest
import (
"archive/zip"
"os"
"path/filepath"
"strings"
"testing"
"textmachine/backend/internal/bookfile"
)
type Chapter struct {
@ -49,30 +57,18 @@ func BuildEPUBAt(t *testing.T, path string, chapters []Chapter, spineIDs []strin
t.Fatal(err)
}
defer f.Close()
zw := zip.NewWriter(f)
add := func(name, content string) {
w, err := zw.Create(name)
if err != nil {
t.Fatal(err)
}
if _, err := w.Write([]byte(content)); err != nil {
t.Fatal(err)
}
}
// OCF requires "mimetype" first in the archive and STORED, not deflated. Nothing in ingest
// reads it today; the fixture matches the spec so it stays a real epub as the reader grows.
mw, err := zw.CreateHeader(&zip.FileHeader{Name: "mimetype", Method: zip.Store})
// OCF requires "mimetype" first in the archive and STORED, not deflated, and container.xml pointing
// at the OPF. The production container writes both (bookfile.NewContainer); the fixture rides it so
// it stays the real epub shape the reader will meet, by construction rather than by copy.
c, err := bookfile.NewContainer(f, "OEBPS/content.opf")
if err != nil {
t.Fatal(err)
}
if _, err := mw.Write([]byte("application/epub+zip")); err != nil {
t.Fatal(err)
add := func(name, content string) {
if err := c.Add(name, []byte(content)); err != nil {
t.Fatal(err)
}
}
add("META-INF/container.xml", `<?xml version="1.0"?>
<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container">
<rootfiles><rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/></rootfiles>
</container>`)
var manifest, spine strings.Builder
for _, c := range chapters {
@ -106,7 +102,7 @@ func BuildEPUBAt(t *testing.T, path string, chapters []Chapter, spineIDs []strin
add("OEBPS/"+entry, c.Body) // non-xhtml asset (e.g. image bytes stand-in)
}
}
if err := zw.Close(); err != nil {
if err := c.Close(); err != nil {
t.Fatal(err)
}
}

150
backend/internal/lang/reader.go Normal file
View file

@ -0,0 +1,150 @@
package lang
import (
"fmt"
"os"
"path/filepath"
"strings"
)
// reader.go: the READER-FACING words a target language supplies for what a book FILE has to say about
// itself — the mark a reader meets at a hole in the text and the notice that the book is not whole
// (the engine's book writer, internal/pipeline bookbuild.go). They are RENDER data (what the file says),
// not TRANSLATION data (what the model sees), and the distinction is load-bearing:
//
// - They live at <langpack_root>/<TARGET lang>/reader.txt — keyed by the language the READER reads,
// not by the pair, because «this fragment is not translated» is the same sentence for a zh→ru and
// a ja→ru book.
// - They are read by THIS loader and by nothing else. Load (langpack.go) reads only the source and
// the pair directories and folds THOSE bytes into Pack.Version(), the value the run snapshot carries;
// a target directory is outside its reach by construction, so adding or editing reader.txt moves no
// snapshot and re-bills no book. That is the property the file was placed here for: a rendering fact
// must never look like a translation fact to the money.
//
// A target language that ships no file gets DefaultReaderWords: a non-verbal form — a symbol and the
// numbers the engine knows — so a book with no language data still cannot look whole when it is not,
// and no Go source needs to know how to say «chapter» or «not translated» in any language.
// ReaderWords are the templates the book writer fills. Each is a reader-facing sentence or mark; the
// placeholders in braces are the only things the engine substitutes. A template may omit a placeholder
// it has no use for.
type ReaderWords struct {
// HolePending marks a unit the run has not reached: {chapter} {unit}.
HolePending string
// HoleWithheld marks a unit whose text was deliberately not shipped (a substantive flag upstream):
// {chapter} {unit}.
HoleWithheld string
// HoleIncomplete marks a unit whose text IS present but lost a piece (a member chunk the editor
// left out): {chapter} {unit} {dropped}.
HoleIncomplete string
// HoleStale marks a unit whose translation is of a source that has since changed (the engine re-buys
// it on the next run; the old text is not shown): {chapter} {unit}.
HoleStale string
// HoleGhost marks a chapter that has translated text the current cut of the book cannot place:
// {chapter} {ghost}.
HoleGhost string
// NoticeHoles is the book-level notice for a book with holes: {holes} {total}.
NoticeHoles string
// NoticeGhost is the book-level notice for text outside the current cut: {ghost}.
NoticeGhost string
}
// readerKeys maps reader.txt keys to the field they fill. Every key is REQUIRED when the file exists:
// a file that names four of six marks leaves the other two to the non-verbal form in the same book,
// which is a mixed register no author intended.
var readerKeys = map[string]func(*ReaderWords, string){
"hole.pending": func(w *ReaderWords, v string) { w.HolePending = v },
"hole.withheld": func(w *ReaderWords, v string) { w.HoleWithheld = v },
"hole.incomplete": func(w *ReaderWords, v string) { w.HoleIncomplete = v },
"hole.stale": func(w *ReaderWords, v string) { w.HoleStale = v },
"hole.ghost": func(w *ReaderWords, v string) { w.HoleGhost = v },
"notice.holes": func(w *ReaderWords, v string) { w.NoticeHoles = v },
"notice.ghost": func(w *ReaderWords, v string) { w.NoticeGhost = v },
}
// readerFile is the file name under the target-language directory.
const readerFile = "reader.txt"
// DefaultReaderWords is the NON-VERBAL form used when the target language ships no reader.txt: a warning
// sign and the numbers the engine knows. It contains no word of any language on purpose — a word here
// would be the engine deciding how a language says something (12-go-style-notes §0).
func DefaultReaderWords() ReaderWords {
return ReaderWords{
HolePending: "⚠ {chapter}.{unit}",
HoleWithheld: "⚠ {chapter}.{unit}",
HoleIncomplete: "⚠ {chapter}.{unit} −{dropped}",
HoleStale: "⚠ {chapter}.{unit}",
HoleGhost: "⚠ {chapter} +{ghost}",
NoticeHoles: "⚠ {holes}/{total}",
NoticeGhost: "⚠ +{ghost}",
}
}
// LoadReaderWords reads <root>/<targetLang>/reader.txt. An empty root (a book with no langpack) or an
// absent file yields DefaultReaderWords and present=false; a present file must be whole and well-formed
// (every key once, no unknown key, no empty value) or the load fails loud — a half-written vocabulary is
// a corrupt data file, not an intended mixture.
func LoadReaderWords(root, targetLang string) (words ReaderWords, present bool, err error) {
words = DefaultReaderWords()
if root == "" || strings.TrimSpace(targetLang) == "" {
return words, false, nil
}
b, err := os.ReadFile(filepath.Join(root, targetLang, readerFile))
if err != nil {
if os.IsNotExist(err) {
return words, false, nil
}
return words, false, fmt.Errorf("reader words %s/%s: %w", targetLang, readerFile, err)
}
parsed, err := parseReaderWords(b)
if err != nil {
return words, false, fmt.Errorf("reader words %s/%s: %w", targetLang, readerFile, err)
}
return parsed, true, nil
}
// parseReaderWords reads the `key<TAB>value` lines of reader.txt (comments and blank lines skipped).
func parseReaderWords(b []byte) (ReaderWords, error) {
var w ReaderWords
seen := map[string]bool{}
for i, raw := range strings.Split(string(b), "\n") {
t := strings.TrimSpace(strings.TrimRight(raw, "\r"))
if t == "" || strings.HasPrefix(t, "#") {
continue
}
f := strings.SplitN(t, "\t", 2)
if len(f) != 2 {
return w, fmt.Errorf("line %d: want `key<TAB>value`, got %q", i+1, t)
}
key, val := strings.TrimSpace(f[0]), strings.TrimSpace(f[1])
set, ok := readerKeys[key]
if !ok {
return w, fmt.Errorf("line %d: unknown key %q", i+1, key)
}
if seen[key] {
return w, fmt.Errorf("line %d: key %q given twice", i+1, key)
}
if val == "" {
return w, fmt.Errorf("line %d: key %q has an empty value", i+1, key)
}
seen[key] = true
set(&w, val)
}
for key := range readerKeys {
if !seen[key] {
return w, fmt.Errorf("missing key %q (every mark must be given; omit the file to use the non-verbal form)", key)
}
}
return w, nil
}
// FillReaderTemplate substitutes {name} placeholders in tmpl from pairs (name, value, name, value…).
// A placeholder the template does not use is simply not there; a name the caller did not pass stays as
// written, which is visible in the output rather than silently blank.
func FillReaderTemplate(tmpl string, pairs ...string) string {
oldnew := make([]string, 0, len(pairs))
for i := 0; i+1 < len(pairs); i += 2 {
oldnew = append(oldnew, "{"+pairs[i]+"}", pairs[i+1])
}
return strings.NewReplacer(oldnew...).Replace(tmpl)
}

122
backend/internal/lang/reader_test.go Normal file
View file

@ -0,0 +1,122 @@
package lang
import (
"os"
"path/filepath"
"strings"
"testing"
"unicode"
)
// reader_test.go: the reader words are DATA of the target language, read outside the pack loader and
// outside Pack.Version() — a wording edit must never re-bill a book.
const sampleReader = `# ru
hole.pending ⚠ Не переведено ({chapter}.{unit}).
hole.withheld ⚠ Не выпущено.
hole.incomplete ⚠ Недостаёт {dropped}.
hole.stale ⚠ Устарело ({chapter}.{unit}).
hole.ghost ⚠ Глава {chapter}: вне нарезки {ghost}.
notice.holes ⚠ Пропусков {holes} из {total}.
notice.ghost ⚠ Вне нарезки {ghost}.
`
func writeReader(t *testing.T, root, tgt, body string) {
t.Helper()
if err := os.MkdirAll(filepath.Join(root, tgt), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(root, tgt, readerFile), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
func TestReaderWordsLoadAndFill(t *testing.T) {
root := t.TempDir()
writeReader(t, root, "ru", sampleReader)
w, present, err := LoadReaderWords(root, "ru")
if err != nil || !present {
t.Fatalf("present=%v err=%v", present, err)
}
if got := FillReaderTemplate(w.HolePending, "chapter", "3", "unit", "0"); got != "⚠ Не переведено (3.0)." {
t.Errorf("filled = %q", got)
}
if got := FillReaderTemplate(w.NoticeHoles, "holes", "2", "total", "37"); got != "⚠ Пропусков 2 из 37." {
t.Errorf("filled = %q", got)
}
// A placeholder the caller did not pass stays visible rather than vanishing.
if got := FillReaderTemplate("{x} {y}", "x", "1"); got != "1 {y}" {
t.Errorf("unfilled placeholder must stay: %q", got)
}
}
func TestReaderWordsAbsentIsTheNonVerbalForm(t *testing.T) {
for _, root := range []string{"", t.TempDir()} {
w, present, err := LoadReaderWords(root, "ru")
if err != nil || present {
t.Fatalf("root %q: present=%v err=%v", root, present, err)
}
if w != DefaultReaderWords() {
t.Fatalf("root %q: absent file must yield the defaults", root)
}
}
// The non-verbal form contains no letter of any language: a symbol and the numbers only.
d := DefaultReaderWords()
for name, tmpl := range map[string]string{"pending": d.HolePending, "withheld": d.HoleWithheld, "incomplete": d.HoleIncomplete,
"stale": d.HoleStale, "ghost": d.HoleGhost, "notice": d.NoticeHoles, "notice-ghost": d.NoticeGhost} {
bare := tmpl
for _, ph := range []string{"{chapter}", "{unit}", "{dropped}", "{ghost}", "{holes}", "{total}"} {
bare = strings.ReplaceAll(bare, ph, "")
}
for _, r := range bare {
if unicode.IsLetter(r) {
t.Errorf("default %s template %q carries a letter — that is a word the engine decided", name, tmpl)
break
}
}
}
}
func TestReaderWordsFailLoud(t *testing.T) {
cases := map[string]string{
"unknown key": sampleReader + "hole.other\tx\n",
"missing key": strings.Replace(sampleReader, "notice.ghost\t⚠ Вне нарезки {ghost}.\n", "", 1),
"duplicate key": sampleReader + "hole.pending\tагаин\n",
"empty value": strings.Replace(sampleReader, "⚠ Не выпущено.", "", 1),
"no tab": strings.Replace(sampleReader, "hole.withheld\t", "hole.withheld ", 1),
}
for name, body := range cases {
root := t.TempDir()
writeReader(t, root, "ru", body)
if _, _, err := LoadReaderWords(root, "ru"); err == nil {
t.Errorf("%s: must fail loud", name)
}
}
}
// TestReaderWordsDoNotMoveThePackVersion is the money pin: reader.txt beside the pack changes NOTHING
// the snapshot folds. (The stand proof is `current_snapshot` of the minirun export before and after.)
func TestReaderWordsDoNotMoveThePackVersion(t *testing.T) {
root := t.TempDir()
writeSyntheticPack(t, root, "zh", "ru")
before, err := Load(root, "zh", "ru")
if err != nil {
t.Fatal(err)
}
writeReader(t, root, "ru", sampleReader)
after, err := Load(root, "zh", "ru")
if err != nil {
t.Fatal(err)
}
if before.Version() != after.Version() {
t.Fatalf("adding <root>/ru/reader.txt moved Pack.Version() %s → %s — a wording edit would re-bill every book of the pair", before.Version(), after.Version())
}
// And the real shipped file loads.
w, present, err := LoadReaderWords(filepath.Join("..", "..", "configs", "langpacks"), "ru")
if err != nil || !present {
t.Fatalf("configs/langpacks/ru/reader.txt: present=%v err=%v", present, err)
}
if !strings.Contains(w.HoleIncomplete, "{dropped}") || !strings.Contains(w.NoticeHoles, "{total}") {
t.Errorf("shipped ru words lost a placeholder: %+v", w)
}
}

598
backend/internal/pipeline/bookbuild.go Normal file
View file

@ -0,0 +1,598 @@
package pipeline
import (
"bytes"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"textmachine/backend/internal/bookfile"
"textmachine/backend/internal/lang"
)
// bookbuild.go: the translated book as a FILE — the `tmctl build` half of backlog row 236. It turns the
// export projection (export.go) into the reader's copy (internal/bookfile: EPUB 3 and plain text), and
// its whole job beyond that is honesty: a copy of an unfinished book must never look finished.
//
// SIX states the file may not lie about. Three are BookExport's counters (pending, ghost rows, config
// drift); two are per unit and read through UnitHole (withheld, incomplete — holes.go); the sixth is the
// writer's own, read from the store: STALE — a unit whose final row was made for a source that is no
// longer the book's (staleUnits). The rule:
//
// - pending · withheld · incomplete · stale · ghost are HOLES. By default a book with any hole is
// REFUSED (RefusalBookIncomplete, listing every hole) — a reader-facing copy is fail-closed
// (D29.1(б)). With --partial the same book is written with a NOTICE a reader meets on opening it (the
// first blocks of the first spine document, and dc:description) and a MARK at the place of every
// hole: instead of the text for a pending/withheld/stale unit, ahead of the text for an incomplete
// unit, at the end of a chapter for ghost rows (their place inside the chapter is not known). Both
// behaviours exist because the contract says an unfinished book may be exported and leaves what such
// a file contains to the engine (14 §createExport); which is the default is this file's decision.
// - ghost rows are a hole and not a footnote: when the cut coarsened since the run, the leader row the
// export joins carries the text of a SHORTER unit than the one it now stands for, and the rest of that
// source's translation sits in the ghost rows — and reaches no file. The acceptance stand book shows
// it exactly: 0 pending, 20 ghosts, a «complete» export that is not.
// - stale is a hole because the file would otherwise carry the translation of text that is no longer
// in the book, byte-identical to the previous build, under `complete: true` — reached by an ordinary
// sequence (build; fix a typo in the source; build). The engine already holds the disproving fact and
// acts on it at the next run (the resume fast-path re-buys the unit); the writer reads the same fact.
// Under config drift the fact is unreadable (see staleUnits) and the report says UNKNOWN, never none.
// - config drift is NOT a hole: the text in the file is byte-for-byte the text the run shipped (the
// drift is between the current config and that run — export.go). It is reported (BuildReport,
// the log), never written into the file: «config drift» is operator vocabulary a reader cannot use.
//
// WORDS. Every word a reader sees at a hole or in the notice is DATA of the TARGET language
// (lang.LoadReaderWords: <langpack_root>/<target>/reader.txt, outside the pack's version — a rendering
// fact must never move a snapshot). A book with no language data gets the non-verbal form: a sign and
// the numbers. No Go source here knows how any language says «chapter» or «not translated».
//
// HEADINGS. A chapter's title is the deterministic heading the engine ALREADY glued onto its first
// unit's export text (ChunkExport.Heading == chunk.ApplyHeading's literal): the writer strips that
// known prefix and gives the same literal to the chapter's <h1> and its nav entry. When the heading is ""
// — a legal state (D39.100 п.1) — the chapter is titled with its bare number, the one label that decides
// nothing (which label a reader should see there is the open K-3 question; it is not settled here). A
// title the MODEL wrote inside the prose is prose and stays prose (backlog row 160 owns that seam).
//
// PARAGRAPHS. A paragraph is a non-blank line of the export text (CR, LF or CRLF all end a line, as on
// ingest). The editor's output separates paragraphs with a blank line in one place and a bare newline in
// the next with no meaning behind the difference (measured on the minirun export: unit 1/0 has 14 of one
// and 7 of the other, unit 2/0 has 56 and 0), so a rule that tells them apart invents a distinction the
// text does not carry. The line is the unit of a paragraph in the web-novel source, and it is what
// ingest reads back (the round-trip proof). A unit whose text yields NO paragraph — control characters
// only — shipped nothing a reader can see and is a hole of the withheld kind, whatever its row says.
//
// FILES. Everything a build writes beside the database is from ONE build: every requested format is
// staged first and committed together, and a format that was NOT requested is removed, so `book_files`
// never points at an older copy next to a newer one — two files of different vintage beside one
// database is the «silently complete» outcome again, reached through a legitimate command sequence
// (build, translate more, build --format epub). A failure to PREPARE a file leaves the previous set
// whole; a failure between two commits (a rename in a directory that was just written to — rare, but
// not impossible) is RefusalWriteIncomplete naming what landed, and a rebuild converges. A copy is still
// a snapshot of the store at build time: a consumer that hands a file to a reader — the platform's export
// door — builds first and serves what it built; it does not serve whatever is lying beside the database.
//
// --out names a NEW file, and it is never replaced: an explicit path can be anything the caller typed —
// the project database, the source, the other format's copy — and the writer cannot tell a previous copy
// of the book from any of those, so it refuses to write over whatever exists there. The engine's own
// place beside the database IS replaced, because there the writer knows what it is replacing.
// buildVersion versions the SHAPE of the build report.
const buildVersion = "tm-build-v1"
// BuildOptions are the caller's choices for one build.
type BuildOptions struct {
// Formats to write, each one of bookfile.Formats; empty means all of them.
Formats []string
// Out is an explicit path for the ONE requested format; "" writes beside the project database. The
// path must not exist yet (see the file comment) and its directory must.
Out string
// Partial writes a book with holes, marked, instead of refusing it.
Partial bool
}
// BuildReport is what `tmctl build` prints: where the files landed and what the book is.
type BuildReport struct {
Version string `json:"build_version"`
BookID string `json:"book_id"`
// Files maps each written format to its absolute path.
Files map[string]string `json:"files"`
// The six states, so a caller does not have to open the file to learn what it wrote.
TotalUnits int `json:"total_units"`
PendingUnits int `json:"pending_units"`
WithheldUnits int `json:"withheld_units"`
IncompleteUnits int `json:"incomplete_units"`
StaleUnits int `json:"stale_units"`
// StaleUnknown is true when the stale check could not be made for at least one shipped unit: under
// config drift (a moved config moves every rendered hash, so a source edit cannot be told from it),
// or for a row with no content hash (written before the engine kept one). Reported, never folded
// into «none».
StaleUnknown bool `json:"stale_unknown"`
GhostRows int `json:"ghost_rows"`
ConfigDrift bool `json:"config_drift"`
// Complete is true when the file has no hole of any kind — pending, withheld, incomplete, stale,
// ghost. A file written under --partial with Complete=false carries the notice and the marks.
Complete bool `json:"complete"`
// TextModified is the file's dcterms:modified (BookExport.TextModified, or the epoch for a book that
// shipped nothing), shown so the determinism of the stamp is visible without unzipping the file.
TextModified string `json:"text_modified"`
}
// hole is one place the file cannot be whole, in reading order.
type hole struct {
Kind HoleKind
Chapter int
Unit int
Reason string // the unit's flag reason (withheld) or the dropped member's (incomplete)
Dropped int // incomplete: how many members are missing
}
// bookIdentifierPrefix is the URN namespace of dc:identifier: the book id under it is deterministic —
// the same book yields the same identifier in every build, which is what a reader's library keys on.
const bookIdentifierPrefix = "urn:textmachine:book:"
// bookFilePath is where format's copy of the book lives: beside the project DB, like every sidecar.
func (r *Runner) bookFilePath(format string) string { return r.Book.ProjectDB + ".book." + format }
// bookFilePaths is the complete map StatusArtifacts publishes — every format, whether written or not.
func (r *Runner) bookFilePaths() map[string]string {
out := make(map[string]string, len(bookfile.Formats))
for _, f := range bookfile.Formats {
out[f] = absPath(r.bookFilePath(f))
}
return out
}
// BuildBook writes the reader's copy of the book (see the file comment). It is a $0 read of the store
// plus a file write: no job, no reservation, no provider. Safe whenever `export` is.
//
// It writes through the atomic staging of artifact.go: a reader — the platform's export door — sees
// either the previous copy or the new one, whole. An existing copy beside the database is REPLACED: the
// file is a deterministic function of the store and re-made at will, unlike a backup, which refuses to
// overwrite because it is a restore point nothing can re-make.
func (r *Runner) BuildBook(opts BuildOptions) (*BuildReport, error) {
formats := opts.Formats
if len(formats) == 0 {
formats = bookfile.Formats
}
for _, f := range formats {
if !bookfile.KnownFormat(f) {
return nil, fmt.Errorf("pipeline: build: unknown format %q (want one of %s)", f, strings.Join(bookfile.Formats, ", "))
}
}
if opts.Out != "" {
if len(formats) != 1 {
return nil, fmt.Errorf("pipeline: build: --out names ONE file, but %d formats were asked for (%s) — pass --format with a single format", len(formats), strings.Join(formats, ","))
}
if err := r.checkOutPath(opts.Out); err != nil {
return nil, err
}
}
exp, err := r.Export(false)
if err != nil {
return nil, err
}
stale, staleUnknown := r.staleUnits(exp)
return r.buildFromExport(exp, stale, staleUnknown, formats, opts)
}
// checkOutPath is the --out rule of the file comment: a new file, in a directory that exists. Refused
// with the config class — the operator's argument is the thing to fix, and a plain error would put the
// typo on exit 1 among the crashes.
func (r *Runner) checkOutPath(out string) error {
if _, err := os.Lstat(out); err == nil {
return refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: --out %s already exists and an explicit path is never replaced (it could be anything the caller typed — the project database included); name a new file, or omit --out for the engine's own place beside the database, which is replaced", r.Book.BookID, out))
} else if !errors.Is(err, fs.ErrNotExist) {
return refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: --out %s: %w", r.Book.BookID, out, err))
}
if fi, err := os.Stat(filepath.Dir(out)); err != nil || !fi.IsDir() {
return refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: --out %s: its directory does not exist", r.Book.BookID, out))
}
return nil
}
// staleUnits finds the shipped units whose FINAL row was made for a source that is no longer the book's:
// the row's content hash — the signature of the rendered messages, source text included, which the
// resume fast-path compares before serving a row for $0 (stagerun.go) — differs from the hash the run
// would render now (repin.go renderedContentHashes, the same reproduction the money projection uses).
//
// It answers only when the config has NOT drifted: a prompt bump moves every rendered hash, and a
// source edit cannot be told apart from it — under drift the answer is UNKNOWN, reported as such. A unit
// whose hash cannot be reproduced (an old row with no content hash, a template the runner has not
// loaded, a member draft missing) is unknown too, never «not stale». It never fails the build: a check
// that cannot run says so in the report and the log.
func (r *Runner) staleUnits(exp *BookExport) (stale map[UnitRef]bool, unknown bool) {
if exp.ConfigDrift {
return nil, true
}
shipped := 0
for _, ce := range exp.Chunks {
if ce.Disposition != exportPending {
shipped++
}
}
if shipped == 0 {
return nil, false
}
cannot := func(why string, err error) (map[UnitRef]bool, bool) {
r.Log.Warn("build: the stale check could not run; whether the source moved under the shipped rows is UNKNOWN (reported as unknown, not as none)",
"book", r.Book.BookID, "step", why, "err", err)
return nil, true
}
statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID)
if err != nil {
return cannot("read chunk_status", err)
}
if err := r.projectStoredMemory(); err != nil {
return cannot("materialize the stored bank", err)
}
full, err := r.bookChunks()
if err != nil {
return cannot("re-chunk the source", err)
}
hashes := r.cachedRenderedContentHashes(full, precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget))
finalRow := r.finalStageRows(statuses)
stale = map[UnitRef]bool{}
for _, ce := range exp.Chunks {
if ce.Disposition == exportPending {
continue
}
key := chunkKey{ce.Chapter, ce.ChunkIdx}
cs, ok := finalRow[key]
h, hok := hashes[key][cs.Stage]
if !ok || cs.ContentHash == "" || !hok {
unknown = true
continue
}
if h != cs.ContentHash {
stale[UnitRef{Chapter: ce.Chapter, ChunkIdx: ce.ChunkIdx}] = true
}
}
if unknown {
r.Log.Warn("build: the stale check could not be made for every shipped unit (a row without a content hash, or a position the run cannot re-render); those units are reported as unknown", "book", r.Book.BookID)
}
return stale, unknown
}
// buildFromExport is BuildBook after the projection: the decision (refuse or write), the assembly and
// the writes. Split from the export read so the states the harness cannot reach through a run — a book
// of no units — are testable on a synthetic projection.
func (r *Runner) buildFromExport(exp *BookExport, stale map[UnitRef]bool, staleUnknown bool, formats []string, opts BuildOptions) (*BuildReport, error) {
// The two metadata the file cannot do without and the config does not require. Refused with the
// config class rather than failed: book.yaml is the thing to fix, and a plain error would put the
// operator's typo on exit 1 among the crashes.
if strings.TrimSpace(r.Book.Title) == "" {
return nil, refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: book.yaml has no `title` — the reader's copy carries the title as dc:title and the writer invents none", r.Book.BookID))
}
if !looksLikeLanguageTag(r.Book.TargetLang) {
return nil, refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: book.yaml target_lang %q is not a language tag (dc:language wants BCP 47: `ru`, `zh`, `pt-BR`)", r.Book.BookID, r.Book.TargetLang))
}
if exp.TotalUnits == 0 {
// A spine with no document is not an EPUB, and a text file of nothing is not a book. The class is
// the one the ingest path raises for the same fact: the source was read and cut and there is no
// book in it (refusal.go sourceHasNoContent).
return nil, sourceHasNoContent(fmt.Errorf("pipeline: build %s: the book has no output units — nothing to write", r.Book.BookID))
}
words, present, err := lang.LoadReaderWords(r.Book.LangpackRoot, r.Book.TargetLang)
if err != nil {
// A data file the operator ships and this engine cannot read: the operator's file is the thing to
// fix, nothing about the book is wrong — the config class.
return nil, refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: %w", r.Book.BookID, err))
}
book, holes, err := assembleBook(exp, stale, r.Book.Title, r.Book.TargetLang, words)
if err != nil {
return nil, err
}
complete := len(holes) == 0 && exp.GhostRows == 0
if !complete && !opts.Partial {
return nil, refuse(RefusalBookIncomplete, describeHoles(r.Book.BookID, exp, holes, staleUnknown))
}
if !complete && !present {
r.Log.Warn("build: the target language ships no reader words; holes and the notice are marked in the NON-VERBAL form (a sign and numbers)",
"book", r.Book.BookID, "target_lang", r.Book.TargetLang, "langpack_root", r.Book.LangpackRoot)
}
rep := &BuildReport{
Version: buildVersion, BookID: exp.BookID, Files: map[string]string{},
TotalUnits: exp.TotalUnits, GhostRows: exp.GhostRows, ConfigDrift: exp.ConfigDrift, StaleUnknown: staleUnknown,
Complete: complete, TextModified: bookfile.Modified(book.Modified),
}
for _, h := range holes {
switch h.Kind {
case HolePending:
rep.PendingUnits++
case HoleWithheld:
rep.WithheldUnits++
case HoleIncomplete:
rep.IncompleteUnits++
case HoleStale:
rep.StaleUnits++
}
}
// Stage every format, then commit every format (the writeDecisionFiles discipline, artifact.go): an
// environment failure while PREPARING the second file must not leave a new epub beside the previous txt.
var staged []*stagedFile
abort := func() {
for _, s := range staged {
s.abort()
}
}
for _, f := range formats {
var buf bytes.Buffer
switch f {
case "epub":
err = bookfile.WriteEPUB(&buf, book)
case "txt":
err = bookfile.WriteTXT(&buf, book)
}
if err != nil {
abort()
return nil, err
}
path := r.bookFilePath(f)
if opts.Out != "" {
path = opts.Out
}
s, err := stageFileAtomic(path, buf.Bytes())
if err != nil {
abort()
if opts.Out != "" {
// The caller named the place and the place cannot take a file: their argument, the
// config class — not an engine failure on exit 1.
return nil, refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: --out %s cannot be written: %w", r.Book.BookID, opts.Out, err))
}
return nil, err
}
staged = append(staged, s)
rep.Files[f] = absPath(path)
}
for i, s := range staged {
if err := s.commit(); err != nil {
for _, rest := range staged[i+1:] {
rest.abort()
}
landed := make([]string, 0, i)
for _, done := range staged[:i] {
landed = append(landed, done.path)
}
return nil, refuse(RefusalWriteIncomplete, fmt.Errorf("pipeline: build %s: the copies were prepared and the write did not complete — %w; landed: [%s]; not landed: %s and the rest — rebuild to converge",
r.Book.BookID, err, strings.Join(landed, " "), s.path))
}
}
if opts.Out == "" {
// The set beside the database is from THIS build: a format not asked for is removed rather than
// left as an older copy the envelope would present as current (see the file comment).
for _, f := range bookfile.Formats {
if _, wanted := rep.Files[f]; wanted {
continue
}
if err := os.Remove(r.bookFilePath(f)); err != nil && !errors.Is(err, fs.ErrNotExist) {
return nil, fmt.Errorf("pipeline: build %s: remove the previous %s copy: %w", r.Book.BookID, f, err)
}
}
}
if exp.ConfigDrift {
r.Log.Warn("build: CONFIG-DRIFT — the file carries the text the run shipped; the current config would render a different snapshot (not written into the file)",
"book", r.Book.BookID, "current", exp.CurrentSnapshot)
}
r.Log.Info("build: book written", "book", r.Book.BookID, "files", rep.Files, "complete", complete,
"pending", rep.PendingUnits, "withheld", rep.WithheldUnits, "incomplete", rep.IncompleteUnits, "stale", rep.StaleUnits,
"stale_unknown", staleUnknown, "ghost_rows", exp.GhostRows)
return rep, nil
}
// describeListMax bounds the per-hole lines of a refusal: a never-run 2000-chapter book has thousands of
// pending units, and the refusal is a diagnostic, not the manifest.
const describeListMax = 50
// describeHoles is the refusal's text: the counts, then every hole by place and reason.
func describeHoles(bookID string, exp *BookExport, holes []hole, staleUnknown bool) error {
counts := map[HoleKind]int{}
for _, h := range holes {
counts[h.Kind]++
}
var sb strings.Builder
fmt.Fprintf(&sb, "pipeline: build %s: the book is not whole and --partial was not given — nothing written: pending=%d withheld=%d incomplete=%d stale=%d ghost_rows=%d of total_units=%d",
bookID, counts[HolePending], counts[HoleWithheld], counts[HoleIncomplete], counts[HoleStale], exp.GhostRows, exp.TotalUnits)
if staleUnknown {
sb.WriteString(" (stale: UNKNOWN for some or all units — the config drifted, or a row carries no content hash)")
}
for i, h := range holes {
if i == describeListMax {
fmt.Fprintf(&sb, "\n … and %d more", len(holes)-describeListMax)
break
}
switch h.Kind {
case HolePending:
fmt.Fprintf(&sb, "\n chapter %d unit %d: pending (not yet translated)", h.Chapter, h.Unit)
case HoleWithheld:
fmt.Fprintf(&sb, "\n chapter %d unit %d: withheld%s", h.Chapter, h.Unit, parenReason(h.Reason))
case HoleIncomplete:
fmt.Fprintf(&sb, "\n chapter %d unit %d: incomplete — %d member(s) of the unit missing from its text%s", h.Chapter, h.Unit, h.Dropped, parenReason(h.Reason))
case HoleStale:
fmt.Fprintf(&sb, "\n chapter %d unit %d: stale (the source changed under this unit since the run; `translate` re-buys it)", h.Chapter, h.Unit)
}
}
if len(exp.GhostUnits) > 0 {
sb.WriteString("\n ghost rows (translated text the current cut of the book cannot place):")
for i, g := range exp.GhostUnits {
if i == describeListMax {
fmt.Fprintf(&sb, " … and %d more", len(exp.GhostUnits)-describeListMax)
break
}
fmt.Fprintf(&sb, " %d/%d", g.Chapter, g.ChunkIdx)
}
}
return fmt.Errorf("%s", sb.String())
}
func parenReason(reason string) string {
if reason == "" {
return ""
}
return " (" + reason + ")"
}
// assembleBook turns the export into the document model: chapters from the units in manifest order,
// the known heading prefix stripped and re-used as the title, a mark at every hole, the notice when
// there is one — and returns the holes it marked, in reading order, which is what the refusal and the
// report count (the writer's own reading of the units, not the counters'). Pure: the same export, stale
// set and words yield the same Book.
func assembleBook(exp *BookExport, stale map[UnitRef]bool, title, language string, words lang.ReaderWords) (*bookfile.Book, []hole, error) {
modified, err := bookModified(exp.TextModified)
if err != nil {
return nil, nil, err
}
book := &bookfile.Book{
Identifier: bookIdentifierPrefix + exp.BookID,
Title: strings.TrimSpace(title),
Language: language,
Modified: modified,
}
// Ghost rows by the chapter number they were STORED under — the old cut's numbering. Chapter numbers
// are dense, so a source edit that empties a chapter shifts every later number, and a ghost's mark can
// then sit in a neighbouring chapter's text; the book is still marked not-whole, only the mark's
// coordinate is the old cut's. A ghost whose chapter the current cut no longer has is in the notice's
// count and nowhere else.
ghostsByChapter := map[int]int{}
for _, g := range exp.GhostUnits {
ghostsByChapter[g.Chapter]++
}
var holes []hole
chapterNo := 0 // dense chapter number of the chapter being assembled; 0 = none yet (numbers are 1-based)
closeChapter := func() {
if chapterNo == 0 {
return
}
if n := ghostsByChapter[chapterNo]; n > 0 {
i := len(book.Chapters) - 1
book.Chapters[i].Paragraphs = append(book.Chapters[i].Paragraphs,
lang.FillReaderTemplate(words.HoleGhost, "chapter", strconv.Itoa(chapterNo), "ghost", strconv.Itoa(n)))
delete(ghostsByChapter, chapterNo)
}
}
for _, ce := range exp.Chunks {
if ce.Chapter != chapterNo {
closeChapter()
chapterNo = ce.Chapter
t := ce.Heading
if t == "" {
t = strconv.Itoa(ce.Chapter)
}
book.Chapters = append(book.Chapters, bookfile.Chapter{Title: t})
}
i := len(book.Chapters) - 1
chapter, unit := strconv.Itoa(ce.Chapter), strconv.Itoa(ce.ChunkIdx)
text := ce.FinalText
if ce.Heading != "" {
// The literal chunk.ApplyHeading glued on: `heading + "\n\n" + text`. A KNOWN prefix, stripped
// only when it is there; never a guess at the text.
text = strings.TrimPrefix(text, ce.Heading+"\n\n")
}
kind := UnitHole(ce)
var paragraphs []string
switch kind {
case HoleNone, HoleIncomplete:
if stale[UnitRef{Chapter: ce.Chapter, ChunkIdx: ce.ChunkIdx}] {
// The text is the translation of a source that is no longer the book's: not shown.
kind = HoleStale
break
}
paragraphs = textParagraphs(text)
if len(paragraphs) == 0 {
// A row that says «text» over characters no reader's file can carry shipped nothing a
// reader can see; the file says so rather than carrying an empty chapter.
kind = HoleWithheld
}
}
mark := func(tmpl string, extra ...string) {
args := append([]string{"chapter", chapter, "unit", unit}, extra...)
book.Chapters[i].Paragraphs = append(book.Chapters[i].Paragraphs, lang.FillReaderTemplate(tmpl, args...))
}
switch kind {
case HolePending:
mark(words.HolePending)
case HoleWithheld:
mark(words.HoleWithheld)
holes = append(holes, hole{Kind: HoleWithheld, Chapter: ce.Chapter, Unit: ce.ChunkIdx, Reason: ce.FlagReason})
continue
case HoleStale:
mark(words.HoleStale)
case HoleIncomplete:
mark(words.HoleIncomplete, "dropped", strconv.Itoa(ce.DroppedMembers))
holes = append(holes, hole{Kind: HoleIncomplete, Chapter: ce.Chapter, Unit: ce.ChunkIdx, Reason: ce.DroppedReason, Dropped: ce.DroppedMembers})
}
if kind == HolePending || kind == HoleStale {
holes = append(holes, hole{Kind: kind, Chapter: ce.Chapter, Unit: ce.ChunkIdx})
continue
}
book.Chapters[i].Paragraphs = append(book.Chapters[i].Paragraphs, paragraphs...)
}
closeChapter()
if len(holes) > 0 {
book.Notice = append(book.Notice,
lang.FillReaderTemplate(words.NoticeHoles, "holes", strconv.Itoa(len(holes)), "total", strconv.Itoa(exp.TotalUnits)))
}
if exp.GhostRows > 0 {
book.Notice = append(book.Notice, lang.FillReaderTemplate(words.NoticeGhost, "ghost", strconv.Itoa(exp.GhostRows)))
}
book.Description = strings.Join(book.Notice, " ")
return book, holes, nil
}
// textParagraphs splits export text into paragraphs: one per non-blank line (CR, LF and CRLF all end a
// line — what text.NormalizeSource does on ingest, so a bare CR cannot make the two formats disagree),
// trimmed and cleaned of the control characters no reader's format admits (bookfile.CleanText — applied
// HERE, once, so the EPUB and the text file carry the same paragraphs; see the file comment on why the
// line is the unit).
func textParagraphs(text string) []string {
text = strings.ReplaceAll(text, "\r\n", "\n")
text = strings.ReplaceAll(text, "\r", "\n")
var out []string
for _, line := range strings.Split(text, "\n") {
if p := strings.TrimSpace(bookfile.CleanText(line)); p != "" {
out = append(out, p)
}
}
return out
}
// looksLikeLanguageTag is the syntactic shape of a BCP 47 tag (a 2–3 letter primary subtag, optional
// alphanumeric subtags): the one check that keeps an unusable dc:language a loud refusal instead of a
// file every validator rejects. It knows no language; it knows what a tag looks like.
func looksLikeLanguageTag(s string) bool {
subtags := strings.Split(s, "-")
if len(subtags[0]) < 2 || len(subtags[0]) > 3 {
return false
}
for i, sub := range subtags {
if sub == "" || len(sub) > 8 {
return false
}
for _, r := range sub {
isLetter := (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z')
isDigit := r >= '0' && r <= '9'
if !isLetter && !(isDigit && i > 0) {
return false
}
}
}
return true
}
// bookModified turns BookExport.TextModified into the file's modification time. A book that shipped
// nothing ("" — no final row) is stamped with the epoch: nothing has ever modified its text, and a
// constant is the one value that is neither a clock reading nor a claim.
func bookModified(textModified string) (time.Time, error) {
if textModified == "" {
return time.Unix(0, 0).UTC(), nil
}
t, err := time.Parse("2006-01-02T15:04:05Z", textModified)
if err != nil {
return time.Time{}, fmt.Errorf("pipeline: build: export text_modified %q is not CCYY-MM-DDThh:mm:ssZ: %w", textModified, err)
}
return t.UTC(), nil
}

657
backend/internal/pipeline/bookbuild_test.go Normal file
View file

@ -0,0 +1,657 @@
package pipeline
import (
"bytes"
"context"
"errors"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"textmachine/backend/internal/bookfile"
"textmachine/backend/internal/chunk"
"textmachine/backend/internal/lang"
"textmachine/backend/internal/obs"
)
// bookbuild_test.go pins the book writer (backlog row 236) on the runner harness: the circle
// «translated → built → read back by the engine's own ingest» closes by EQUALITY of paragraphs, and
// each of the five states a file may not lie about is driven through the real projection where a
// run can reach it (pending, withheld, ghost, drift) and through a synthetic projection where it
// cannot (incomplete, the in-chapter ghost mark, a heading, a book of no units).
// ingestParagraphs reads an EPUB back through the engine's reader and splits each chapter the way the
// chunker does (blank lines, trimmed, empties dropped — chunk.splitParagraphs).
func ingestParagraphs(t *testing.T, epubPath string) [][]string {
t.Helper()
doc, err := chunk.IngestEncoded(epubPath, "", "")
if err != nil {
t.Fatalf("ingest of the written EPUB failed: %v", err)
}
out := make([][]string, len(doc.Chapters))
for i, ch := range doc.Chapters {
for _, raw := range strings.Split(ch, "\n\n") {
if p := strings.TrimSpace(raw); p != "" {
out[i] = append(out[i], p)
}
}
}
return out
}
func assertRoundTrip(t *testing.T, book *bookfile.Book, epubPath string) {
t.Helper()
got := ingestParagraphs(t, epubPath)
if len(got) != len(book.Chapters) {
t.Fatalf("ingest read %d chapters back, the book has %d", len(got), len(book.Chapters))
}
for i := range book.Chapters {
want := book.Blocks(i)
if strings.Join(got[i], "\n|") != strings.Join(want, "\n|") {
t.Errorf("chapter %d read back differently:\n got %q\nwant %q", i+1, got[i], want)
}
}
}
// multiLineEdit makes the editor return prose with BOTH separators the real editor uses (a bare newline
// and a blank line), so the paragraph rule is exercised rather than assumed.
func multiLineEdit(body string) (string, string) {
if isEditBody(body) {
return "Первая строка абзаца.\nВторая строка.\n\nТретья, после пустой.", "stop"
}
return "ЧЕРНОВИК ПЕРЕВОДА", "stop"
}
func TestBuildBookRoundTripsThroughIngestAndPublishesItsPaths(t *testing.T) {
srv := newJSONProvider(&reqRec{}, multiLineEdit)
defer srv.Close()
bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ГЛАВАА\fГЛАВАБ\fГЛАВАВ", regenerate: 0})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r := newRunner(t, bookPath)
defer r.Close()
if _, err := r.TranslateBook(ctx); err != nil {
t.Fatal(err)
}
rep, err := r.BuildBook(BuildOptions{})
if err != nil {
t.Fatalf("BuildBook: %v", err)
}
if !rep.Complete || rep.PendingUnits+rep.WithheldUnits+rep.IncompleteUnits+rep.StaleUnits+rep.GhostRows != 0 || rep.ConfigDrift || rep.StaleUnknown {
t.Fatalf("a fully translated book must build complete, with the stale check MADE: %+v", rep)
}
if rep.Version != buildVersion || rep.TextModified == "" || rep.TextModified == "1970-01-01T00:00:00Z" {
t.Errorf("report envelope/stamp: %+v", rep)
}
dir := filepath.Dir(bookPath)
for _, f := range bookfile.Formats {
want := filepath.Join(dir, "test-book.db.book."+f)
if rep.Files[f] != want {
t.Errorf("files[%s] = %q, want %q", f, rep.Files[f], want)
}
if _, err := os.Stat(want); err != nil {
t.Errorf("%s not written: %v", want, err)
}
}
// The status envelope publishes the same places, whether or not a build ran.
st, err := r.Status(ctx)
if err != nil {
t.Fatal(err)
}
for f, p := range rep.Files {
if st.Artifacts.BookFiles[f] != p {
t.Errorf("status artifacts.book_files[%s] = %q, the build wrote %q", f, st.Artifacts.BookFiles[f], p)
}
}
// The circle: what the engine reads back is what the writer laid down, chapter by chapter.
exp, err := r.Export(false)
if err != nil {
t.Fatal(err)
}
book, _, err := assembleBook(exp, nil, r.Book.Title, r.Book.TargetLang, lang.DefaultReaderWords())
if err != nil {
t.Fatal(err)
}
if len(book.Chapters) != 3 || len(book.Chapters[0].Paragraphs) != 3 || book.Chapters[0].Paragraphs[1] != "Вторая строка." {
t.Fatalf("paragraph rule: a non-blank line is a paragraph: %+v", book.Chapters[0])
}
if book.Chapters[0].Title != "1" || len(book.Notice) != 0 || book.Description != "" {
t.Errorf("a headingless complete book: bare-number titles, no notice: %+v", book)
}
assertRoundTrip(t, book, rep.Files["epub"])
// The text file is the same blocks, with no operator vocabulary.
txt, err := os.ReadFile(rep.Files["txt"])
if err != nil {
t.Fatal(err)
}
for _, banned := range []string{"=== CHAPTER", "TEXT MISSING", "pending", "⚠"} {
if strings.Contains(string(txt), banned) {
t.Errorf("complete book's txt carries %q:\n%s", banned, txt)
}
}
if !strings.HasPrefix(string(txt), "Тест\n\n\n1\n\nПервая строка абзаца.\n\nВторая строка.\n\nТретья, после пустой.\n\n\n2\n") {
t.Errorf("txt shape:\n%s", txt)
}
// Determinism in-process (the two-process proof is the stand's cmp): a rebuild is byte-identical.
first, _ := os.ReadFile(rep.Files["epub"])
if _, err := r.BuildBook(BuildOptions{}); err != nil {
t.Fatal(err)
}
second, _ := os.ReadFile(rep.Files["epub"])
if !bytes.Equal(first, second) {
t.Error("two builds of one store differ byte-wise")
}
}
func TestBuildBookRefusesAPendingBookAndPartialMarksIt(t *testing.T) {
srv := newJSONProvider(&reqRec{}, draftEdit)
defer srv.Close()
// The ceiling pays for about one chapter, then denies (TestExportManifestPending's setup).
bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ГЛАВАА\fГЛАВАБ\fГЛАВАВ", regenerate: 0, bookUSD: 0.005})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r := newRunner(t, bookPath)
defer r.Close()
_, _ = r.TranslateBook(ctx) // the ceiling stop is expected; the partial store is the fixture
_, err := r.BuildBook(BuildOptions{})
if refusalClassOf(err) != RefusalBookIncomplete {
t.Fatalf("class = %v", err)
}
if !strings.Contains(err.Error(), "pending (not yet translated)") || !strings.Contains(err.Error(), "--partial") {
t.Errorf("the refusal must list the holes and name the way out: %v", err)
}
dir := filepath.Dir(bookPath)
if _, err := os.Stat(filepath.Join(dir, "test-book.db.book.epub")); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("a refusal must write nothing: %v", err)
}
rep, err := r.BuildBook(BuildOptions{Partial: true})
if err != nil {
t.Fatalf("--partial must write: %v", err)
}
if rep.Complete || rep.PendingUnits < 1 {
t.Fatalf("report must say the book is not whole: %+v", rep)
}
exp, _ := r.Export(false)
book, _, err := assembleBook(exp, nil, r.Book.Title, r.Book.TargetLang, lang.DefaultReaderWords())
if err != nil {
t.Fatal(err)
}
// No langpack → the non-verbal form: the notice is «⚠ holes/total», the mark «⚠ chapter.unit».
if len(book.Notice) != 1 || !strings.HasPrefix(book.Notice[0], "⚠ ") || !strings.HasSuffix(book.Notice[0], "/3") {
t.Errorf("notice = %q", book.Notice)
}
marked := 0
for _, ce := range exp.Chunks {
if ce.Disposition != exportPending {
continue
}
marked++
ch := book.Chapters[ce.Chapter-1]
if len(ch.Paragraphs) != 1 || ch.Paragraphs[0] != "⚠ "+strconv.Itoa(ce.Chapter)+"."+strconv.Itoa(ce.ChunkIdx) {
t.Errorf("pending chapter %d must carry exactly the mark: %+v", ce.Chapter, ch)
}
}
if marked != rep.PendingUnits {
t.Errorf("marked %d pending units, report says %d", marked, rep.PendingUnits)
}
assertRoundTrip(t, book, rep.Files["epub"])
txt, _ := os.ReadFile(rep.Files["txt"])
if !strings.HasPrefix(string(txt), "Тест\n\n"+book.Notice[0]+"\n\n\n") {
t.Errorf("txt must open with the title and the notice:\n%s", txt)
}
}
func TestBuildBookWithheldUnitIsAHole(t *testing.T) {
// The editor leaks a service preamble on chapter 3 → substantive sanitizer defect → the unit ships
// no text (withheld), while chapters 1–2 are clean (TestExportMatchesTranslateFinalText's ch3 path).
srv := newJSONProvider(&reqRec{}, func(body string) (string, string) {
if !isEditBody(body) {
return "черновик перевода этой главы.", "stop"
}
if strings.Contains(body, "ГЛАВАВВ") {
return "Вот перевод фрагмента:\nОн молча ушёл в туман.", "stop"
}
return "Чистый абзац.", "stop"
})
defer srv.Close()
bookPath := setupProjectOpts(t, srv.URL, projectOpts{
source: "ГЛАВААА\fГЛАВАББ\fГЛАВАВВ", gatesYAML: "gates:\n sanitizer:\n enabled: true\n",
})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r := newRunner(t, bookPath)
defer r.Close()
_, _ = r.TranslateBook(ctx) // completes with flags
exp, err := r.Export(false)
if err != nil {
t.Fatal(err)
}
if exp.PendingUnits != 0 {
t.Fatalf("the fixture must have NO pending unit — the hole is a withheld one: %+v", exp)
}
withheld, _ := HoleCounts(exp)
if withheld != 1 || UnitHole(exp.Chunks[2]) != HoleWithheld {
t.Fatalf("chapter 3 must be withheld: withheld=%d chunks=%+v", withheld, exp.Chunks)
}
_, err = r.BuildBook(BuildOptions{})
if refusalClassOf(err) != RefusalBookIncomplete || !strings.Contains(err.Error(), "chapter 3 unit 0: withheld") {
t.Fatalf("pending_units=0 with a withheld unit must still be refused, naming it: %v", err)
}
rep, err := r.BuildBook(BuildOptions{Partial: true, Formats: []string{"epub"}})
if err != nil {
t.Fatal(err)
}
if rep.WithheldUnits != 1 || rep.Complete {
t.Errorf("report: %+v", rep)
}
book, _, _ := assembleBook(exp, nil, r.Book.Title, r.Book.TargetLang, lang.DefaultReaderWords())
if got := book.Chapters[2].Paragraphs; len(got) != 1 || got[0] != "⚠ 3.0" {
t.Errorf("withheld chapter must carry exactly the mark: %q", got)
}
assertRoundTrip(t, book, rep.Files["epub"])
if _, ok := rep.Files["txt"]; ok {
t.Error("--format epub must not write the txt")
}
}
func TestBuildBookGhostRowsAreAHole(t *testing.T) {
srv := newJSONProvider(&reqRec{}, draftEdit)
defer srv.Close()
bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ГЛАВАА\fГЛАВАБ\fГЛАВАВ", regenerate: 0})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r1 := newRunner(t, bookPath)
if _, err := r1.TranslateBook(ctx); err != nil {
t.Fatal(err)
}
r1.Close()
// The source SHRINKS to two chapters: chapter 3's final row is now a GHOST — translated text the
// current cut cannot place — while pending stays 0. The «complete» look is exactly the trap.
writeFile(t, filepath.Join(filepath.Dir(bookPath), "source.txt"), "ГЛАВАА\fГЛАВАБ")
r2 := newRunner(t, bookPath)
defer r2.Close()
exp, err := r2.Export(false)
if err != nil {
t.Fatal(err)
}
if exp.PendingUnits != 0 || exp.GhostRows != 1 || len(exp.GhostUnits) != 1 || exp.GhostUnits[0] != (UnitRef{Chapter: 3, ChunkIdx: 0}) {
t.Fatalf("fixture: want pending 0, ghost 1 at 3/0; got %+v", exp)
}
_, err = r2.BuildBook(BuildOptions{})
if refusalClassOf(err) != RefusalBookIncomplete || !strings.Contains(err.Error(), "ghost rows") || !strings.Contains(err.Error(), " 3/0") {
t.Fatalf("ghost rows alone must refuse, naming the row: %v", err)
}
rep, err := r2.BuildBook(BuildOptions{Partial: true})
if err != nil {
t.Fatal(err)
}
if rep.Complete || rep.GhostRows != 1 {
t.Errorf("report: %+v", rep)
}
book, _, _ := assembleBook(exp, nil, r2.Book.Title, r2.Book.TargetLang, lang.DefaultReaderWords())
// Chapter 3 no longer exists to be marked; the book-level notice carries the count.
if len(book.Chapters) != 2 || len(book.Notice) != 1 || book.Notice[0] != "⚠ +1" {
t.Errorf("ghost-only book: 2 chapters, one ghost notice: chapters=%d notice=%q", len(book.Chapters), book.Notice)
}
assertRoundTrip(t, book, rep.Files["epub"])
}
func TestBuildBookDriftIsReportedNotMarked(t *testing.T) {
srv := newJSONProvider(&reqRec{}, draftEdit)
defer srv.Close()
bookPath := setupProjectOpts(t, srv.URL, projectOpts{regenerate: 0})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r1 := newRunner(t, bookPath)
if _, err := r1.TranslateBook(ctx); err != nil {
t.Fatal(err)
}
r1.Close()
dir := filepath.Dir(bookPath)
body, _ := os.ReadFile(filepath.Join(dir, "pipeline.yaml"))
writeFile(t, filepath.Join(dir, "pipeline.yaml"), strings.Replace(string(body), "prompt_version: v-test", "prompt_version: v-test2", 1))
r2 := newRunner(t, bookPath)
defer r2.Close()
rep, err := r2.BuildBook(BuildOptions{})
if err != nil {
t.Fatalf("drift is not a hole; the default build must write: %v", err)
}
if !rep.ConfigDrift || !rep.Complete {
t.Fatalf("report must carry the drift and call the text complete: %+v", rep)
}
txt, _ := os.ReadFile(rep.Files["txt"])
if strings.Contains(string(txt), "⚠") || strings.Contains(strings.ToLower(string(txt)), "drift") {
t.Errorf("drift must not reach the reader's file:\n%s", txt)
}
}
// TestAssembleBookMarksEveryHoleKind drives the assembler on a synthetic projection: the heading is
// stripped and becomes the title, incomplete units keep their text behind the mark, a ghost inside an
// existing chapter is marked at the chapter's end, and the words come from the target language's data.
func TestAssembleBookMarksEveryHoleKind(t *testing.T) {
exp := &BookExport{
BookID: "syn", TotalUnits: 5, PendingUnits: 1, GhostRows: 2, TextModified: "2026-08-30T12:00:00Z",
GhostUnits: []UnitRef{{Chapter: 2, ChunkIdx: 4}, {Chapter: 2, ChunkIdx: 6}},
Chunks: []ChunkExport{
{Chapter: 1, ChunkIdx: 0, Disposition: "ok", Heading: "Глава 1", FinalText: "Глава 1\n\nПодзаголовок модели\nПервый.\n\nВторой."},
{Chapter: 1, ChunkIdx: 2, Disposition: "ok", FinalText: "Третий."},
{Chapter: 2, ChunkIdx: 0, Disposition: "flagged", FlagReason: "sanitizer_substantive", FinalText: ""},
{Chapter: 2, ChunkIdx: 2, Disposition: "flagged", FlagReason: "cjk_artifact", DroppedMembers: 1, DroppedReason: "cjk_artifact", FinalText: "Уцелевшая часть."},
{Chapter: 3, ChunkIdx: 0, Disposition: "pending"},
},
}
root := t.TempDir()
if err := os.MkdirAll(filepath.Join(root, "ru"), 0o755); err != nil {
t.Fatal(err)
}
writeFile(t, filepath.Join(root, "ru", "reader.txt"), "hole.pending\tНе переведено.\nhole.withheld\tНе выпущено.\nhole.incomplete\tНедостаёт {dropped}.\nhole.stale\tУстарело.\nhole.ghost\tВне нарезки {ghost}.\nnotice.holes\tПропусков {holes} из {total}.\nnotice.ghost\tВне нарезки всего {ghost}.\n")
words, present, err := lang.LoadReaderWords(root, "ru")
if err != nil || !present {
t.Fatal(err)
}
book, holes, err := assembleBook(exp, nil, "蛊真人", "ru", words)
if err != nil {
t.Fatal(err)
}
if len(holes) != 3 {
t.Errorf("holes = %+v", holes)
}
if book.Identifier != "urn:textmachine:book:syn" || book.Language != "ru" || book.Title != "蛊真人" || bookfile.Modified(book.Modified) != "2026-08-30T12:00:00Z" {
t.Errorf("metadata: %+v", book)
}
want := []bookfile.Chapter{
{Title: "Глава 1", Paragraphs: []string{"Подзаголовок модели", "Первый.", "Второй.", "Третий."}},
{Title: "2", Paragraphs: []string{"Не выпущено.", "Недостаёт 1.", "Уцелевшая часть.", "Вне нарезки 2."}},
{Title: "3", Paragraphs: []string{"Не переведено."}},
}
if len(book.Chapters) != len(want) {
t.Fatalf("chapters: %+v", book.Chapters)
}
for i := range want {
if book.Chapters[i].Title != want[i].Title || strings.Join(book.Chapters[i].Paragraphs, "|") != strings.Join(want[i].Paragraphs, "|") {
t.Errorf("chapter %d = %+v, want %+v", i+1, book.Chapters[i], want[i])
}
}
if strings.Join(book.Notice, "|") != "Пропусков 3 из 5.|Вне нарезки всего 2." || book.Description != "Пропусков 3 из 5. Вне нарезки всего 2." {
t.Errorf("notice = %q description = %q", book.Notice, book.Description)
}
// The heading is stripped ONLY as the known prefix: a text that merely mentions the heading later
// keeps it, and a heading absent from the text strips nothing.
exp2 := &BookExport{BookID: "syn", TotalUnits: 1, TextModified: "2026-08-30T12:00:00Z",
Chunks: []ChunkExport{{Chapter: 1, ChunkIdx: 0, Disposition: "ok", Heading: "Глава 1", FinalText: "Текст, где Глава 1 упомянута."}}}
b2, _, err := assembleBook(exp2, nil, " t ", "ru", words)
if err != nil || b2.Chapters[0].Paragraphs[0] != "Текст, где Глава 1 упомянута." || b2.Chapters[0].Title != "Глава 1" || b2.Title != "t" {
t.Errorf("prefix rule / trimmed title: %+v %q %v", b2.Chapters, b2.Title, err)
}
// A book no run has written is stamped with the epoch, and a malformed stamp fails loud.
exp2.TextModified = ""
if b3, _, err := assembleBook(exp2, nil, "t", "ru", words); err != nil || bookfile.Modified(b3.Modified) != "1970-01-01T00:00:00Z" {
t.Errorf("epoch stamp: %v %v", err, b3)
}
exp2.TextModified = "yesterday"
if _, _, err := assembleBook(exp2, nil, "t", "ru", words); err == nil {
t.Error("a malformed text_modified must fail loud")
}
// The writer's own reading of a unit: text of nothing a reader can see is withheld; a bare CR ends a
// line like LF; a stale unit shows its mark and not its (outdated) text, ahead of the incomplete rule.
exp3 := &BookExport{BookID: "syn", TotalUnits: 3, TextModified: "2026-08-30T12:00:00Z", Chunks: []ChunkExport{
{Chapter: 1, ChunkIdx: 0, Disposition: "ok", FinalText: "\x01\x02"},
{Chapter: 2, ChunkIdx: 0, Disposition: "ok", FinalText: "Первая\rВторая\r\nТретья"},
{Chapter: 3, ChunkIdx: 0, Disposition: "flagged", DroppedMembers: 1, DroppedReason: "x", FinalText: "Старый текст."},
}}
b4, holes4, err := assembleBook(exp3, map[UnitRef]bool{{Chapter: 3, ChunkIdx: 0}: true}, "t", "ru", words)
if err != nil {
t.Fatal(err)
}
if got := b4.Chapters[0].Paragraphs; len(got) != 1 || got[0] != "Не выпущено." {
t.Errorf("control-only text must be a withheld hole, got %q", got)
}
if got := strings.Join(b4.Chapters[1].Paragraphs, "|"); got != "Первая|Вторая|Третья" {
t.Errorf("CR must end a line: %q", got)
}
if got := b4.Chapters[2].Paragraphs; len(got) != 1 || got[0] != "Устарело." {
t.Errorf("a stale unit shows its mark and not its old text, got %q", got)
}
if len(holes4) != 2 || holes4[0].Kind != HoleWithheld || holes4[1].Kind != HoleStale || b4.Notice[0] != "Пропусков 2 из 3." {
t.Errorf("holes = %+v notice = %q", holes4, b4.Notice)
}
}
// TestBuildBookStaleUnitsAreAHole is the final review's blocker: a source edit that keeps the unit keys
// (a typo fix) used to yield a byte-identical file under complete=true — the translation of text no
// longer in the book. The writer now reads the same fact the resume path acts on (content hash).
func TestBuildBookStaleUnitsAreAHole(t *testing.T) {
srv := newJSONProvider(&reqRec{}, draftEdit)
defer srv.Close()
bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ГЛАВАА\fГЛАВАБ", regenerate: 0})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r1 := newRunner(t, bookPath)
if _, err := r1.TranslateBook(ctx); err != nil {
t.Fatal(err)
}
rep1, err := r1.BuildBook(BuildOptions{})
if err != nil || !rep1.Complete || rep1.StaleUnknown || rep1.StaleUnits != 0 {
t.Fatalf("a fresh run must build complete with the stale check MADE: %+v %v", rep1, err)
}
r1.Close()
// The same two chapters, one of them edited in place: keys survive, the text under one does not.
writeFile(t, filepath.Join(filepath.Dir(bookPath), "source.txt"), "ГЛАВАА\fГЛАВАБ — исправлено")
r2 := newRunner(t, bookPath)
_, err = r2.BuildBook(BuildOptions{})
if refusalClassOf(err) != RefusalBookIncomplete || !strings.Contains(err.Error(), "chapter 2 unit 0: stale") || strings.Contains(err.Error(), "chapter 1 unit 0") {
t.Fatalf("the edited unit — and only it — must be a stale hole: %v", err)
}
rep2, err := r2.BuildBook(BuildOptions{Partial: true})
if err != nil || rep2.Complete || rep2.StaleUnits != 1 || rep2.StaleUnknown {
t.Fatalf("partial: %+v %v", rep2, err)
}
exp, _ := r2.Export(false)
stale, unknown := r2.staleUnits(exp)
book, _, _ := assembleBook(exp, stale, r2.Book.Title, r2.Book.TargetLang, lang.DefaultReaderWords())
if unknown || !stale[UnitRef{Chapter: 2, ChunkIdx: 0}] || strings.Join(book.Chapters[1].Paragraphs, "|") != "⚠ 2.0" {
t.Errorf("stale=%v unknown=%v chapter2=%q", stale, unknown, book.Chapters[1].Paragraphs)
}
assertRoundTrip(t, book, rep2.Files["epub"])
r2.Close()
// Re-translating (the engine re-buys the moved unit) makes the book whole again.
r3 := newRunner(t, bookPath)
defer r3.Close()
if _, err := r3.TranslateBook(ctx); err != nil {
t.Fatal(err)
}
rep3, err := r3.BuildBook(BuildOptions{})
if err != nil || !rep3.Complete || rep3.StaleUnits != 0 {
t.Fatalf("after a re-translate the book must be whole: %+v %v", rep3, err)
}
txt, _ := os.ReadFile(rep3.Files["txt"])
if strings.Contains(string(txt), "⚠") {
t.Errorf("no mark may survive the re-translate:\n%s", txt)
}
}
// TestBuildBookRemovedStageIsDrift: removing the editor stage used to make the draft rows the shipping
// rows with no drift — an unedited book that reads as complete. Now a stored row of a stage the config
// does not run is drift (export.go), and the writer's stale check says «unknown» under drift.
func TestBuildBookRemovedStageIsDrift(t *testing.T) {
srv := newJSONProvider(&reqRec{}, draftEdit)
defer srv.Close()
bookPath := setupProjectOpts(t, srv.URL, projectOpts{regenerate: 0})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r1 := newRunner(t, bookPath)
if _, err := r1.TranslateBook(ctx); err != nil {
t.Fatal(err)
}
r1.Close()
dir := filepath.Dir(bookPath)
body, _ := os.ReadFile(filepath.Join(dir, "pipeline.yaml"))
var kept []string
for _, line := range strings.Split(string(body), "\n") {
if !strings.Contains(line, "name: edit") {
kept = append(kept, line)
}
}
writeFile(t, filepath.Join(dir, "pipeline.yaml"), strings.Join(kept, "\n"))
r2 := newRunner(t, bookPath)
defer r2.Close()
rep, err := r2.BuildBook(BuildOptions{})
if err != nil {
t.Fatal(err)
}
if !rep.ConfigDrift || !rep.StaleUnknown {
t.Fatalf("a removed stage is drift, and under drift the stale check is unknown: %+v", rep)
}
}
func TestBuildBookRefusesABookOfNoUnitsAndBadOptions(t *testing.T) {
srv := newJSONProvider(&reqRec{}, draftEdit)
defer srv.Close()
r := newRunner(t, setupProject(t, srv.URL))
defer r.Close()
_, err := r.buildFromExport(&BookExport{Version: exportVersion, BookID: "empty", Chunks: []ChunkExport{}}, nil, false, bookfile.Formats, BuildOptions{})
if refusalClassOf(err) != RefusalSourceUnreadable {
t.Errorf("a book of no units: %v", err)
}
if _, err := r.BuildBook(BuildOptions{Formats: []string{"pdf"}}); err == nil || strings.Contains(err.Error(), "refus") {
t.Errorf("unknown format must be a plain error: %v", err)
}
if _, err := r.BuildBook(BuildOptions{Out: "x"}); err == nil {
t.Error("--out with two formats must be an error")
}
}
// TestBuildBookLeavesOneBuildBesideTheDatabase pins the fix of the mid-work review's blocker: a
// `--format` subset must not leave the other format's OLDER copy beside the database, where the
// envelope would present it as current — the «silently complete» outcome through a legal sequence
// (build; translate more; build --format epub).
func TestBuildBookLeavesOneBuildBesideTheDatabase(t *testing.T) {
srv := newJSONProvider(&reqRec{}, draftEdit)
defer srv.Close()
bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ГЛАВАА\fГЛАВАБ", regenerate: 0})
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r1 := newRunner(t, bookPath)
if _, err := r1.TranslateBook(ctx); err != nil {
t.Fatal(err)
}
if _, err := r1.BuildBook(BuildOptions{}); err != nil {
t.Fatal(err)
}
r1.Close()
dir := filepath.Dir(bookPath)
txtPath := filepath.Join(dir, "test-book.db.book.txt")
if _, err := os.Stat(txtPath); err != nil {
t.Fatal(err)
}
// The book grows a chapter (pending) and only the epub is rebuilt, partial.
writeFile(t, filepath.Join(dir, "source.txt"), "ГЛАВАА\fГЛАВАБ\fГЛАВАВ")
r2 := newRunner(t, bookPath)
defer r2.Close()
rep, err := r2.BuildBook(BuildOptions{Formats: []string{"epub"}, Partial: true})
if err != nil {
t.Fatal(err)
}
if rep.Complete || rep.PendingUnits != 1 {
t.Fatalf("fixture: %+v", rep)
}
if _, err := os.Stat(txtPath); !errors.Is(err, os.ErrNotExist) {
t.Fatalf("the older txt (which reads as a COMPLETE book) must not survive a subset rebuild beside the database: %v", err)
}
if _, ok := rep.Files["txt"]; ok {
t.Error("the report must list only what this build wrote")
}
// An explicit --out leaves the default place alone (it wrote elsewhere).
if _, err := r2.BuildBook(BuildOptions{Formats: []string{"txt"}, Partial: true, Out: filepath.Join(t.TempDir(), "b.txt")}); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(filepath.Join(dir, "test-book.db.book.epub")); err != nil {
t.Errorf("--out must not remove the copies beside the database: %v", err)
}
}
func TestBuildBookRefusesAConfigItCannotFileUnder(t *testing.T) {
srv := newJSONProvider(&reqRec{}, draftEdit)
defer srv.Close()
r := newRunner(t, setupProject(t, srv.URL))
defer r.Close()
exp := &BookExport{Version: exportVersion, BookID: "b", TotalUnits: 1, Chunks: []ChunkExport{{Chapter: 1, ChunkIdx: 0, Disposition: "ok", FinalText: "x"}}}
title := r.Book.Title
r.Book.Title = " "
if _, err := r.buildFromExport(exp, nil, false, bookfile.Formats, BuildOptions{}); refusalClassOf(err) != RefusalBadConfig || !strings.Contains(err.Error(), "title") {
t.Errorf("an empty title is a config refusal naming the key: %v", err)
}
r.Book.Title = title
for _, bad := range []string{"russian", "ru_RU", "", "r", "ru-", "ru-très"} {
r.Book.TargetLang = bad
if _, err := r.buildFromExport(exp, nil, false, bookfile.Formats, BuildOptions{}); refusalClassOf(err) != RefusalBadConfig || !strings.Contains(err.Error(), "target_lang") {
t.Errorf("target_lang %q must be a config refusal naming the key: %v", bad, err)
}
}
for _, good := range []string{"ru", "zh", "pt-BR", "zh-Hant-TW", "en-x-private"} {
if !looksLikeLanguageTag(good) {
t.Errorf("%q looks like a language tag and must pass", good)
}
}
}
// TestBuildBookOutNamesANewFile pins the --out rule (the final review's second blocker: --out used to
// replace ANY path, the project database included): an existing file is refused, a missing directory is
// refused, a new path is written, and the default place is left alone.
func TestBuildBookOutNamesANewFile(t *testing.T) {
srv := newJSONProvider(&reqRec{}, draftEdit)
defer srv.Close()
bookPath := setupProject(t, srv.URL)
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
r := newRunner(t, bookPath)
defer r.Close()
if _, err := r.TranslateBook(ctx); err != nil {
t.Fatal(err)
}
dbBefore, _ := os.ReadFile(r.Book.ProjectDB)
for name, out := range map[string]string{
"the project database": r.Book.ProjectDB,
"the source": r.Book.SourceFile,
"an existing file": filepath.Join(t.TempDir(), "taken.epub"),
"a missing directory": filepath.Join(t.TempDir(), "nope", "book.epub"),
} {
if name == "an existing file" {
writeFile(t, out, "somebody's bytes")
}
_, err := r.BuildBook(BuildOptions{Formats: []string{"epub"}, Out: out})
if refusalClassOf(err) != RefusalBadConfig {
t.Errorf("--out %s must be refused with the config class: %v", name, err)
}
}
dbAfter, _ := os.ReadFile(r.Book.ProjectDB)
if !bytes.Equal(dbBefore, dbAfter) {
t.Fatal("the project database was touched by a refused --out")
}
out := filepath.Join(t.TempDir(), "book.epub")
rep, err := r.BuildBook(BuildOptions{Formats: []string{"epub"}, Out: out})
if err != nil {
t.Fatal(err)
}
if rep.Files["epub"] != out {
t.Errorf("files = %v", rep.Files)
}
b, _ := os.ReadFile(out)
if !bytes.HasPrefix(b, []byte("PK")) {
t.Errorf("the explicit path must hold the EPUB: %q", b[:min(len(b), 16)])
}
if _, err := os.Stat(filepath.Join(filepath.Dir(bookPath), "test-book.db.book.epub")); !errors.Is(err, os.ErrNotExist) {
t.Error("--out must not also write the default place")
}
// No temp file survives a refusal or a write, anywhere the writer looked.
for _, d := range []string{filepath.Dir(out), filepath.Dir(bookPath)} {
entries, _ := os.ReadDir(d)
for _, e := range entries {
if strings.Contains(e.Name(), ".tmp-") {
t.Errorf("temp file left behind: %s/%s", d, e.Name())
}
}
}
}

89
backend/internal/pipeline/export.go
View file

@ -2,7 +2,9 @@ package pipeline
import (
"fmt"
"sort"
"strings"
"time"
"textmachine/backend/internal/chunk"
"textmachine/backend/internal/store"
@ -52,6 +54,21 @@ type ChunkExport struct {
// DroppedReason is why the FIRST dropped member flagged — the cause of the HOLE, which FlagReason
// is not when the edit flagged on its own account as well. "" when nothing dropped.
DroppedReason string `json:"dropped_reason,omitempty"`
// Heading is the chapter title the engine renders for this unit's chapter — the SAME literal
// chunk.ApplyHeading glued onto the front of FinalText (u.Members[0].Heading at assembly time), given
// apart so a consumer that wants the title separate from the prose strips a KNOWN prefix instead of
// guessing at the text. Non-empty only on a chapter-opening unit of a book with a heading rule
// (D39.100 п.1: "" is a legal state — a book without a rule, a chapter without a structural marker),
// and present on a pending unit too (it comes from the manifest, not from a run). Additive and
// omitempty: the export of a book without headings serializes exactly as before this field existed.
// ⚠ It is the engine's TEMPORARY render (D39.122 п.2д), not a label the book's data carries.
Heading string `json:"heading,omitempty"`
}
// UnitRef addresses one output unit by its manifest key.
type UnitRef struct {
Chapter int `json:"chapter"`
ChunkIdx int `json:"chunk_idx"`
}
// exportVersion versions the SHAPE of the export document.
@ -76,6 +93,21 @@ type BookExport struct {
// below may then not match what translate actually did. CurrentSnapshot is that projected id.
ConfigDrift bool `json:"config_drift"`
CurrentSnapshot string `json:"current_snapshot,omitempty"`
// GhostUnits are the rows GhostRows counts, BY KEY — so a consumer can say WHICH chapter has
// translated text the current cut of the book cannot place, not only that some chapter does. The
// book writer marks those chapters (a coarsened cut leaves the joined leader row carrying the text
// of a SHORTER unit than the current one; the rest of the source's translation sits in these rows
// and reaches no file). Sorted by key; omitempty (a book without ghosts serializes as before).
GhostUnits []UnitRef `json:"ghost_units,omitempty"`
// TextModified is the newest chunk_status.updated_at among the FINAL rows this export joined — the
// rows whose text (or whose withholding) is in Chunks — rendered as CCYY-MM-DDThh:mm:ssZ: the last
// time the engine wrote the shipped state of this book. Draft rows, ghost rows and rows of stages
// that do not ship are not consulted, so a $0 draft-wave re-write moves nothing here. A fact of the
// store rather than of any process's clock, which is why the book writer uses it as the file's
// modification time (two builds of one store then agree byte-for-byte). "" when no unit has a final
// row (a book that has shipped nothing). Additive and omitempty. ⚠ «Last written», not «last
// changed»: a resume that re-writes an unchanged final row moves it too (store.ChunkStatus.UpdatedAt).
TextModified string `json:"text_modified,omitempty"`
// Chunks is always non-nil (an empty book exports [] not null).
Chunks []ChunkExport `json:"chunks"`
}
@ -126,13 +158,14 @@ func (r *Runner) Export(pairs bool) (*BookExport, error) {
exp := &BookExport{Version: exportVersion, BookID: r.Book.BookID, Chunks: []ChunkExport{}}
inManifest := map[chunkKey]bool{}
var shipped []store.ChunkStatus // the final rows the export joined — what TextModified is read from
for _, u := range units {
key := chunkKey{u.Chapter, u.FirstChunkIdx}
inManifest[key] = true
exp.TotalUnits++
cs, ok := finalRow[key]
if !ok {
pend := ChunkExport{Chapter: u.Chapter, ChunkIdx: u.FirstChunkIdx, Disposition: exportPending}
pend := ChunkExport{Chapter: u.Chapter, ChunkIdx: u.FirstChunkIdx, Disposition: exportPending, Heading: u.Members[0].Heading}
if pairs {
pend.Source = u.sourceText()
}
@ -145,22 +178,57 @@ func (r *Runner) Export(pairs bool) (*BookExport, error) {
return nil, err
}
exp.Chunks = append(exp.Chunks, ce)
shipped = append(shipped, cs)
}
// F4: GHOST rows — a stored final row whose unit-leader key is NOT in the current manifest units
// (source shrunk since the run). Dropped (never exported as if live) but counted + WARNed.
for k := range finalRow {
if !inManifest[k] {
exp.GhostRows++
exp.GhostUnits = append(exp.GhostUnits, UnitRef{Chapter: k.chapter, ChunkIdx: k.chunkIdx})
}
}
sort.Slice(exp.GhostUnits, func(i, j int) bool {
a, b := exp.GhostUnits[i], exp.GhostUnits[j]
return a.Chapter < b.Chapter || (a.Chapter == b.Chapter && a.ChunkIdx < b.ChunkIdx)
})
if exp.GhostRows > 0 {
r.Log.Warn("export: dropped chunk_status rows outside the current source manifest (source shrunk since the run?)",
"book", r.Book.BookID, "ghost_rows", exp.GhostRows)
}
modified, err := newestStatusWrite(shipped)
if err != nil {
return nil, err
}
exp.TextModified = modified
r.exportConfigDrift(statuses, draftStageNames, exp)
return exp, nil
}
// storeTimeLayout is how SQLite's datetime('now') spells chunk_status.updated_at (UTC to the second).
const storeTimeLayout = "2006-01-02 15:04:05"
// newestStatusWrite renders the newest updated_at among the given rows (the joined FINAL rows) as
// BookExport.TextModified. Rows carry the store's own spelling, which sorts lexically, so the newest is
// the string maximum; it is re-parsed rather than re-spelled by hand so a row that is NOT in the store's
// form is a loud inconsistent-store error and not a silently mangled timestamp. "" when there is no row.
func newestStatusWrite(statuses []store.ChunkStatus) (string, error) {
newest := ""
for _, cs := range statuses {
if cs.UpdatedAt > newest {
newest = cs.UpdatedAt
}
}
if newest == "" {
return "", nil
}
t, err := time.Parse(storeTimeLayout, newest)
if err != nil {
return "", fmt.Errorf("pipeline: export: chunk_status.updated_at %q is not in the store's form (inconsistent store): %w", newest, err)
}
return t.UTC().Format("2006-01-02T15:04:05Z"), nil
}
// exportChunks resolves the manifest the export projects over: the persisted one when the export does
// not need source text, the full ingest+split when it does (`--pairs`). One helper so the "who needs
// text" rule lives in one place instead of being re-decided at each call.
@ -264,6 +332,7 @@ func (r *Runner) exportUnit(u editUnit, cs store.ChunkStatus, byChunk map[chunkK
// as the (heading-stripped) source manifest — the DC FP-measure aligns src↔target on the body prose,
// not the deterministic title (which no checker inspects).
ce.FinalText = chunk.ApplyHeading(u.Members[0].Heading, ce.FinalText)
ce.Heading = u.Members[0].Heading // the same literal, given apart (see the field)
if !pairs {
return ce, nil
}
@ -350,6 +419,24 @@ func (r *Runner) exportConfigDrift(statuses []store.ChunkStatus, draftStageNames
}
checkWave(draftSnaps, waveDraft)
checkWave(editSnaps, waveEdit)
// A stored row for a STAGE the current pipeline does not run is drift of the loudest kind — a stage
// renamed or removed since the run (the CAVEATS' «final-stage rename»). The per-wave check above only
// compares the snapshots of stages that still exist, so without this the removal of the editor stage
// makes the draft rows the shipping rows and nothing fires: an unedited book reads as complete.
for _, cs := range statuses {
if cs.SnapshotID == "" || draftStageNames[cs.Stage] || editStageNames[cs.Stage] {
continue
}
exp.ConfigDrift = true
if exp.CurrentSnapshot == "" {
if cur, _, serr := r.snapshotIDForWave(r.finalStageWave()); serr == nil {
exp.CurrentSnapshot = cur
}
}
r.Log.Warn("export: CONFIG-DRIFT — stored rows carry a stage the current config does not run (renamed or removed since the run); the shipping rows are not the ones the run shipped",
"book", r.Book.BookID, "stage", cs.Stage)
break
}
}
// chunkExport computes one chunk's export record from its final-stage row, exactly by prod FinalText

58
backend/internal/pipeline/holes.go Normal file
View file

@ -0,0 +1,58 @@
package pipeline
// holes.go: what a HOLE in the export is — the one predicate the plaintext render (cmd/tmctl render.go),
// the book writer (bookbuild.go) and any later consumer read the per-unit states through.
//
// BookExport's counters carry three book-level states (pending, ghost, drift). Two more are PER UNIT and
// live only in the record: a unit that reached the final stage and shipped NO text (withheld — a
// substantive flag or an upstream skip, D2: contaminated output never ships), and a unit that shipped
// text with a member's worth of it MISSING (incomplete — the c-lite editor dropped a member). A file that
// consults the counters alone comes out «silently whole» on a book with either, which is the worst thing a
// reader's copy can do; hence one definition, here, rather than an arithmetic each surface repeats.
// HoleKind names why a unit has no text, or not all of it.
type HoleKind string
const (
// HoleNone: the unit shipped its whole text.
HoleNone HoleKind = ""
// HolePending: a manifest unit with no final row — not translated yet (BookExport.PendingUnits).
HolePending HoleKind = "pending"
// HoleWithheld: a final row with no text — a substantive flag / upstream skip withheld it.
HoleWithheld HoleKind = "withheld"
// HoleIncomplete: text present, a member chunk of it missing (ChunkExport.DroppedMembers > 0).
HoleIncomplete HoleKind = "incomplete"
// HoleStale: the unit's final row was made for a source that is no longer the book's (its content
// hash differs from what the run would render now). The WRITER's kind (bookbuild.go staleUnits) —
// UnitHole never returns it, because the export record does not carry the fact.
HoleStale HoleKind = "stale"
)
// UnitHole classifies one export record. The order of the tests is the plaintext render's: pending
// first, then «no text», then «text with a dropped member» — an incomplete unit DID ship text.
func UnitHole(ce ChunkExport) HoleKind {
switch {
case ce.Disposition == exportPending:
return HolePending
case ce.FinalText == "":
return HoleWithheld
case ce.DroppedMembers > 0:
return HoleIncomplete
}
return HoleNone
}
// HoleCounts returns the per-unit hole counts of an export: withheld and incomplete (pending is the
// counter BookExport.PendingUnits). `incomplete` is a SUBSET of the exported units, not a fourth part
// of the total: those units did ship text, with a piece of it missing.
func HoleCounts(exp *BookExport) (withheld, incomplete int) {
for _, ce := range exp.Chunks {
switch UnitHole(ce) {
case HoleWithheld:
withheld++
case HoleIncomplete:
incomplete++
}
}
return withheld, incomplete
}

12
backend/internal/pipeline/refusal.go
View file

@ -90,6 +90,18 @@ const (
// acceptance's half-write probe). It is NOT the config class: the deployment is fine and the
// document is fine — the disk, at that moment, was not.
RefusalWriteIncomplete RefusalClass = "write_incomplete"
// RefusalBookIncomplete is a book file `tmctl build` will not write: the book has a HOLE — a unit not
// yet translated, a unit whose text was withheld (a substantive flag, D2), a unit missing a member's
// worth of text, or translated text the current cut of the book cannot place (ghost rows) — and the
// caller did not say `--partial`. The default is the refusal because a reader-facing copy is
// fail-closed (D29.1(б)): a file that is silently short of the book is the worst outcome the writer
// can produce. The refusal lists every hole (chapter, unit, reason), and `--partial` writes the same
// book with the notice on its first page and a mark at every hole (bookbuild.go).
//
// Nothing was written and nothing of the book's state changed — the class keeps the full «nothing
// happened» clause. It is its own class, not the config one: the deployment is fine and the book is
// fine; the book is simply not finished, and the caller decides whether an unfinished copy is wanted.
RefusalBookIncomplete RefusalClass = "book_incomplete"
)
// Refusal is an invocation the engine turned down: nothing reached a provider, nothing was spent, no

7
backend/internal/pipeline/status.go
View file

@ -130,6 +130,12 @@ type StatusArtifacts struct {
// that edits them directly is back to impersonating an operator with a text editor.
MinedDelta string `json:"mined_delta"`
MinedRejects string `json:"mined_rejects"`
// BookFiles are the READER's copies of the book that `tmctl build` writes, by format
// (bookfile.Formats): `<project_db>.book.<format>`, beside the database like every sidecar. A PLACE
// like the others — the map is complete whether or not a build has run, and a consumer opens the
// path and handles not-found. It is the channel the platform's export door reads instead of
// deriving the spelling (17-seam-inbound-law §1: the engine owns the paths of its own scheme).
BookFiles map[string]string `json:"book_files"`
}
// artifacts is the ONE producer of the envelope. Two surfaces publish it (`status --json` and
@ -141,6 +147,7 @@ func (r *Runner) artifacts() StatusArtifacts {
BankExport: absPath(r.bankExportPath()),
MinedDelta: absPath(r.Book.MinedDelta),
MinedRejects: absPath(r.Book.MinedRejects),
BookFiles: r.bookFilePaths(),
}
}

15
backend/internal/store/chunkstatus.go
View file

@ -49,6 +49,13 @@ type ChunkStatus struct {
// (D18/D19; the mini-run of 25.07 measured 0.0% where 1 of 20 drafts had in fact echoed).
// Observability only: re-derived from the stored checkpoints on every run, never a verdict, never wire.
FirstFlagReason string
// UpdatedAt is when this row was last WRITTEN, as the store spells it (`datetime('now')`: UTC,
// `YYYY-MM-DD HH:MM:SS`, NOT NULL since the table's first schema — migrate.go). It is read, never
// written from here: UpsertChunkStatus stamps it itself. The book writer takes the newest one as the
// book's modification time — a fact of the store rather than of the clock of whichever process builds
// the file, which is what keeps two builds of one store byte-identical. A resume that re-writes an
// unchanged row moves it too, so it is «last written», not «last changed».
UpdatedAt string
}
// UpsertChunkStatus writes (or overwrites) the disposition row. Overwrite is the
@ -90,12 +97,12 @@ func (s *Store) GetChunkStatus(bookID string, chapter, chunkIdx int, stage strin
var escalated int
err := s.r.QueryRowContext(ctx, `
SELECT snapshot_id, content_hash, disposition, flag_reason, attempts, final_hash, cost_usd, detail,
escalated, escalation_model, first_flag_reason
escalated, escalation_model, first_flag_reason, updated_at
FROM chunk_status
WHERE book_id = ? AND chapter = ? AND chunk_idx = ? AND stage = ?`,
bookID, chapter, chunkIdx, stage).Scan(
&cs.SnapshotID, &cs.ContentHash, &cs.Disposition, &cs.FlagReason, &cs.Attempts, &cs.FinalHash, &cs.CostUSD, &cs.Detail,
&escalated, &cs.EscalationModel, &cs.FirstFlagReason)
&escalated, &cs.EscalationModel, &cs.FirstFlagReason, &cs.UpdatedAt)
cs.Escalated = escalated != 0
if errors.Is(err, sql.ErrNoRows) {
return nil, nil
@ -111,7 +118,7 @@ func (s *Store) GetChunkStatus(bookID string, chapter, chunkIdx int, stage strin
func (s *Store) ChunkStatusesForBook(bookID string) ([]ChunkStatus, error) {
return queryAll(s.r, `
SELECT chapter, chunk_idx, stage, snapshot_id, content_hash, disposition, flag_reason,
attempts, final_hash, cost_usd, detail, escalated, escalation_model, first_flag_reason
attempts, final_hash, cost_usd, detail, escalated, escalation_model, first_flag_reason, updated_at
FROM chunk_status WHERE book_id = ?
ORDER BY chapter, chunk_idx, stage`,
func(rows *sql.Rows) (ChunkStatus, error) {
@ -119,7 +126,7 @@ func (s *Store) ChunkStatusesForBook(bookID string) ([]ChunkStatus, error) {
var escalated int
if err := rows.Scan(&cs.Chapter, &cs.ChunkIdx, &cs.Stage, &cs.SnapshotID, &cs.ContentHash,
&cs.Disposition, &cs.FlagReason, &cs.Attempts, &cs.FinalHash, &cs.CostUSD, &cs.Detail,
&escalated, &cs.EscalationModel, &cs.FirstFlagReason); err != nil {
&escalated, &cs.EscalationModel, &cs.FirstFlagReason, &cs.UpdatedAt); err != nil {
return cs, err
}
cs.Escalated = escalated != 0

164
docs/PROGRESS.md
View file

@ -279,6 +279,170 @@
> пере-прохода) · **229** (снапшот не фолдит модель внутренних гейтов, латентна) · **230** (размен
> «сходимость против поучения») · **141**-остаток · **131**. Плюс потолок объёма из `D39.165` §1б.
### 📝 ЗАПИСКА-ПЛАН — пак «писатель книги» (промт `docs/BACKEND_BOOKWRITER_PACK_SESSION_PROMPT.md`, строка 236), сессия `textmachine-main-fc`, 30.08
**Что беру (зона `backend/`, HEAD на старте `67c05be`, дерево чисто).**
1. Прод-пакет `backend/internal/bookfile` (только stdlib — `archive/zip`, `encoding/xml`; без `import "testing"`):
`container.go` — OCF-контейнер (`mimetype` первым и `zip.Store`, `container.xml`, запись entry в порядке вызовов,
нулевые таймметки) — `chunktest/epub.go` переводится на него, его OPF-заглушка и XHTML-декой остаются тестовыми;
`epub.go` — EPUB 3 из нейтральной модели `Book{Identifier, Title, Language, Modified, Description,
Chapters[]{Title, Paragraphs[]}}`: `dc:identifier`/`dc:title`/`dc:language`/`dcterms:modified`, nav
(`properties="nav"`, ⛔ НЕ в spine), один XHTML на главу, `<h1>` = заголовок главы, `<p>` = абзац;
`txt.go` — чистый текст из той же модели (заголовок строкой, абзацы через пустую строку, главы через
пустую строку; ни баннеров, ни флагов, ни `gapMarker`).
2. Сборка из `BookExport` — `backend/internal/pipeline/bookbuild.go`: `Runner.BuildBook(formats, partial)` →
`Export(false)` → дыры → отказ либо файл. Запись `writeFileAtomic` рядом с ProjectDB
(`<project_db>.book.<fmt>`, как `.manifest.json`/`.bank.json`), путь публикуется в
`StatusArtifacts.BookFiles` (`book_files`: map формат→путь, аддитивно) — приор промта принят: платформа
пути не выводит сама (17 §1). Существующий файл — перезапись: производный детерминированный артефакт, как
манифест; бэкап отказывает потому, что он точка восстановления, а файл книги пересобирается из стора.
3. Аддитивные поля `tm-export-v1` (форма не меняется, все `omitempty`): `ChunkExport.Heading` (`heading`) —
литерал `u.Members[0].Heading` в момент сборки записи (и у pending-юнита тоже, он из манифеста);
`BookExport.GhostUnits` (`ghost_units`: `[{chapter,chunk_idx}]`) — ключи ghost-строк, чтобы маркер стоял
в главе, а не только в счётчике; `BookExport.TextModified` (`text_modified`, `CCYY-MM-DDThh:mm:ssZ`) —
max(`chunk_status.updated_at`) по финальным строкам, вошедшим в файл (колонка `NOT NULL DEFAULT
datetime('now')`, `store/migrate.go:37`; в `ChunkStatus` дочитывается доп. полем). Это и есть
`dcterms:modified`: последнее изменение отгруженного текста — факт стора, не часов процесса.
4. Глагол `tmctl build --config b.yaml [--format epub,txt] [--out path] [--partial]` (по умолчанию оба формата;
`--out` — только при одном формате). Stdout — JSON-конверт `tm-build-v1` (пути, счётчики, `complete`).
Новый глагол, а не флаг `export`: `export` — чистое чтение в stdout, писать файлы из него — смешать
аудит-проекцию с артефактом; `manifest` — прецедент глагола-писателя сайдкара. `dispatchCommands` и
usage-строка расширяются — контрактное расширение того же рода, что `bank-apply` (тест
`TestParseNoArgsUsage` сам перечисляет такие расширения); чужим глаголам новые флаги ОТКАЗЫВАЮТСЯ
(дисциплина `--keys-file`/`--max-units`).
5. Отказ — новый `pipeline.RefusalBookIncomplete = "book_incomplete"` → exit **16** (полоса 10–19; таблица
тотальна по `TestTheRefusalTableIsTotal`); сообщение перечисляет дыры `глава/юнит/причина`.
6. Читательские слова — ДАННЫЕ: `<langpack_root>/<target_lang>/reader.txt` (`key<TAB>value`, плейсхолдеры
`{chapter} {unit} {dropped} {holes} {total} {ghost}`), читает ОТДЕЛЬНЫЙ загрузчик `lang.LoadReaderWords`;
в `Pack.Version()` НЕ фолдится (пин тестом «наличие reader.txt не меняет Version()» + `current_snapshot`
minirun до/после). Ключ — язык ЦЕЛИ, не пара: слова читает читатель целевого языка, и они не зависят от
исходного. Поставляю `backend/configs/langpacks/ru/reader.txt`. Каталог `<tgt>/` загрузчик пака не
читает (`Load` — только `<src>/` и `<pair>/`, `langpack.go:215-330`), так что канал вне фолда по
построению, а не по соглашению.
**Политика §3.2 — по умолчанию ОТКАЗ, `--partial` — файл с пометкой; довод:**
- Дыры: pending · withheld (`Disposition!="pending" && FinalText==""`) · incomplete (`DroppedMembers>0` при
непустом тексте) — арифметика `render.go:573-587` выносится в общий хелпер `pipeline`, которым пользуется и
`renderExport` (вывод `--plaintext` побайтово прежний) · **ghost — тоже дыра**: при укрупнении нарезки
строка-лидер несёт текст ПРЕЖНЕГО, более короткого юнита, а остаток источника лежит в ghost-строках и в
файл не попадает (acceptance: 57 строк → 37 юнитов, 20 ghost при `pending 0` — ровно «молча полная»
книга). Позиция ghost внутри главы неизвестна → маркер в конце главы с числом потерянных строк.
- Default = отказ: D29.1(б) — reader-facing выдача fail-closed; дверь платформы получит `--partial` для
контрактного «may be exported, finished or not» (14 §createExport).
- `--partial`: пометка неполноты — первый абзац первого документа spine (то, что читатель видит, открыв
книгу) + `dc:description`; маркер — отдельный абзац на месте КАЖДОЙ дыры (pending/withheld — вместо текста
юнита; incomplete — перед текстом юнита, как `gapMarker`; ghost — в конце главы). Форма — из `reader.txt`;
без пака (acceptance, `runner.go:346`) — несловесная: `⚠ {chapter}.{unit}` у дыры, `⚠ {holes}/{total}` в
пометке (символ + числа, никакого словаря в Go). В `FinalText`/`export.go` маркер не попадает.
- drift — НЕ дыра: текст файла побайтово текст прогона (`export.go`, комментарий к `exportConfigDrift`);
попадает в JSON-конверт `build` и в WARN, в файл — нет («config drift» — операторский словарь).
Следствие §4 заявляю сразу: чистого бездрейфового файла на реальной книге на этом HEAD НЕТ — чистый
путь доказывается синтетикой, реальные книги — посадки на дыры.
- Оглавление и `<h1>` при `Heading==""` — номер главы цифрами (пустая ссылка nav невалидна; цифры —
единственный лейбл, который не решает открытый К-3 «какой лейбл видит читатель у главы без заголовка»;
словесный шаблон в данных — 1 строка после ратификации, вопрос владельцу в отчёте). При `Heading!=""` —
срез известного префикса `heading+"\n\n"` из `FinalText` (`chunker.go:186-192`), `<h1>` = Heading.
- Абзац = непустая строка `FinalText`. Замерено на `minirun/export.json`: юнит 1/0 — 14 разделителей `\n\n`
и 7 одиночных `\n`, юнит 2/0 — 56 и 0: редактор чередует их без семантики, различать = выдумать
различие; строка — единица абзаца исходника-вебновеллы. Круг §4 сравнивает абзацы ингеста с абзацами,
которые положил писатель.
- Метаданные: `dc:title` = `config.Book.Title` как есть (на стенде `蛊真人` — ограничение в отчёт), `dc:language`
= `TargetLang`, `dc:identifier` = `urn:textmachine:book:<book_id>`, `dc:creator` нет (источника нет).
- 0 юнитов — отказ `sourceHasNoContent` (spine без элементов невалиден); 1 юнит — обычная книга.
**Что НЕ беру:** платформу и дверь `createExport`/`getExport`; лестницу D25.1 в коде и в `BookExport`;
строки 160/201 (`ApplyHeading` и три точки склейки не трогаю; модельные заголовки в прозе не извлекаю и не
правлю); `--resnapshot` и любые платные вызовы; `langpack_root` в копию acceptance; направление письма
(`dir`/`page-progression-direction`) — в `internal/lang` данных о направлении нет (`lang-script.txt` —
только скрипты), RTL-цели в репо нет → вопрос в отчёт, не литерал; CSS/обложку; новые зависимости.
**Чем докажу:** герметично — `bookfile` (mimetype первым/stored, nav не в spine, детерминизм двух сборок,
экранирование, 0/1 глава), `pipeline` (круг `IngestEncoded` равенством абзацев на прогоне fake-провайдера;
pending через потолок; withheld через санитайзер; **ghost — усадка исходника после прогона (посадки в
тестах не было, строю)**; drift — бамп `prompt_version`; синтетический `BookExport` на incomplete), `cmd/tmctl`
(incomplete бинарём через `setupGapProject`; парс флагов; exit 16 через totality-тест), `lang` (парс
`reader.txt`; `Version()` не меняется от его наличия). На копиях книг в песочнице с переписанными путями:
minirun — оракул июля 14/14 по `final_text/snapshot_id/disposition`, `current_snapshot` до/после
`reader.txt`, `cmp` двух ПРОЦЕССОВ, epubcheck 5.3.0 `0 fatal / 0 error`, круг ингестом; acceptance — отказ с
перечнем и `--partial`-файл с несловесными маркерами; coldrun-a — отказ 14/14 pending;
`git -C books status --short -- gu-zhenren` до/после одинаков (снят до старта: 6 строк `D …-wal/-shm`).
`make battery` + `-race`; в середине — субагент-ревью готовой части против критериев; в конце —
адверсариальный проход по направлениям §4.
### ✅ ИТОГ — пак «писатель книги» (промт `docs/BACKEND_BOOKWRITER_PACK_SESSION_PROMPT.md`, строка 236), сессия `textmachine-main-fc`, 30.08
**Что построено (зона `backend/`, поверх `67c05be`; НЕ закоммичено — дерево на лендинг оркестратору; `git status`: 10 изменённых + 8 новых путей в `backend/`, плюс этот журнал).**
- `internal/bookfile/` (новый прод-пакет, только stdlib, без `testing`): `container.go` — OCF-контейнер (`mimetype` первым, `Store`, через `CreateRaw` — без data-descriptor, флаги `0x0000`; `container.xml`; entry в порядке вызовов; нулевые zip-таймметки); `epub.go` — EPUB 3 (`dc:identifier`+`unique-identifier`, `dc:title`, `dc:language`, `dcterms:modified`, nav `properties="nav"` ⛔ не в spine, XHTML на главу: `<h1>` + `<p>`); `txt.go` — чистый текст; `model.go` — модель `Book/Chapter`, `Blocks(i)` (единственное определение содержимого документа главы), `CleanText` (общая чистка XML-недопустимых символов для обоих форматов), экранирование.
- `internal/chunk/chunktest/epub.go` переведён на `bookfile.NewContainer`; тест-декои (OPF-заглушка, `<title>c</title><style>`) остались тестовыми; 9 импортёров зелёные без правки ассертов.
- `internal/lang/reader.go` — `LoadReaderWords(root, targetLang)`: `<langpack_root>/<target>/reader.txt`, читается ОТДЕЛЬНО от `Load` (в `Pack.Version()` не фолдится по построению — `Load` читает только `<src>/` и `<pair>/`); `DefaultReaderWords()` — несловесная форма (символ + числа; тест пинит отсутствие букв). Данные: `configs/langpacks/ru/reader.txt` (7 ключей: `hole.{pending,withheld,incomplete,stale,ghost}`, `notice.{holes,ghost}`).
- `internal/pipeline/holes.go` — ОДИН предикат дыры `UnitHole`/`HoleCounts` (порядок = `renderExport`); `render.go` считает шапку `--plaintext` через него (вывод побайтово прежний).
- `internal/pipeline/export.go` — аддитивно (`omitempty`): `ChunkExport.Heading` (`heading` — литерал `u.Members[0].Heading`, у pending тоже), `BookExport.GhostUnits` (`ghost_units`, ключи ghost-строк, сортированы), `BookExport.TextModified` (`text_modified` — max `chunk_status.updated_at` по ФИНАЛЬНЫМ строкам, вошедшим в экспорт); `exportConfigDrift`: строка со стадией, которой нет в текущем конфиге, — drift (находка 10 ниже). `store/chunkstatus.go` дочитывает `updated_at` в `ChunkStatus.UpdatedAt` (колонка `NOT NULL DEFAULT datetime('now')`, `migrate.go:37`; миграции — только `ADD COLUMN`: у minirun `text_modified` = `2026-07-25T19:17:53Z`, время боевого прогона, не миграции 30.08).
- `internal/pipeline/bookbuild.go` — `Runner.BuildBook(BuildOptions{Formats, Out, Partial})`: `Export(false)` → `staleUnits` → `assembleBook` (чистая функция, возвращает список дыр) → отказ `RefusalBookIncomplete` (`refusal.go`, exit **16**) либо стейдж всех форматов + общий коммит в `<project_db>.book.<fmt>` (не запрошенные соседи удаляются — рядом с БД лежит ОДНА сборка); `StatusArtifacts.BookFiles` (`book_files`, map формат→абс. путь; в `status --json` и `manifest --json`). `--out` — только НОВЫЙ файл (существующий путь любого рода, несуществующий каталог → `config_invalid`, exit 10). Сбой между двумя `rename` → `write_incomplete` (exit 15) с перечнем landed/not landed.
- `cmd/tmctl`: глагол `build --config b.yaml [--format epub,txt] [--out path] [--partial]`; stdout — JSON `tm-build-v1` (`files`, `total/pending/withheld/incomplete/stale_units`, `stale_unknown`, `ghost_rows`, `config_drift`, `complete`, `text_modified`); чужим глаголам новые флаги отказываются, `build` отказывает `--plaintext/--pairs/--json`; usage-строка расширена (`invocation_test.go` — контрактное расширение по собственному комментарию теста «Every one is a deliberate contract extension»; это ЕДИНСТВЕННАЯ правка существующего теста). `backend/README.md` — строка `cmd/tmctl` дополнена `build`.
**Политика (§3.2) как реализована — ШЕСТЬ состояний, не пять:** дыры — pending · withheld (`Disposition!="pending" && FinalText==""`, плюс юнит, чей текст после чистки не даёт ни одного абзаца) · incomplete (`DroppedMembers>0`) · **stale** (финальная строка сделана под исходник, которого в книге уже нет: `content_hash` строки ≠ `cachedRenderedContentHashes` — тот же факт, по которому резюм перекупает юнит; считается ТОЛЬКО без дрейфа, под дрейфом `stale_unknown: true`, никогда не «нет») · **ghost** (при укрупнении нарезки строка-лидер несёт текст прежнего, более короткого юнита; остаток лежит в ghost-строках и в файл не попадает: acceptance 57→37, 20 строк при `pending 0`). По умолчанию — ОТКАЗ при любой дыре с перечнем `chapter N unit M: <вид> (<причина>)` и ghost-ключами (≤50 строк); `--partial` — файл с пометкой (первые абзацы первого документа spine + `dc:description`; в txt — после заголовка книги) и маркером на месте КАЖДОЙ дыры (pending/withheld/stale — вместо текста; incomplete — перед текстом; ghost — в конце главы, счётчиком). drift — НЕ дыра (текст файла = текст прогона): в JSON-конверте и WARN, в файле нет. Оглавление/`<h1>` при `Heading==""` — номер главы цифрами. Абзац — непустая строка `FinalText` (`\r\n`/`\r`/`\n`). `dc:title` — `book.yaml` как есть (на стенде `蛊真人`), `dc:language` = `target_lang` (форма тега BCP 47 проверяется, иначе `config_invalid`), `dc:identifier` = `urn:textmachine:book:<book_id>`, автора нет.
**Сверка состава против §3 (пункт → сделано → чем доказано; все команды — на копиях в песочнице `<SB>/stand`, бинарь `go build -o <SB>/stand/tmctl ./cmd/tmctl`).**
| §3 | Сделано | Доказательство |
|---|---|---|
| 3.1 EPUB + чистый txt из `BookExport`, ОДИН файл на формат, из `tmctl` | да: `tmctl build` → `<project_db>.book.epub` / `.book.txt` | `build-minirun.json`: 2 пути, `complete: true`; `java -jar epubcheck-5.3.0/epubcheck.jar <book>.epub` → `0 fatals / 0 errors / 0 warnings / 0 infos` на всех трёх EPUB (minirun, acceptance `--partial`, coldrun-a `--partial`) |
| 3.1 без аудит-баннеров/флагов/`gapMarker` в txt | да | `TestTXTShape`, `TestBuildBookRoundTrips…` (нет `=== CHAPTER`, `TEXT MISSING`, `pending`, `flagged`); `TestBuildRefusesTheGapAndPartialMarksIt` — реальный c-lite прогон бинарём: `⚠ 1.0 −1` перед уцелевшим текстом, ни `INCOMPLETE`, ни `cjk_artifact` |
| 3.1 `--plaintext` не тронут | да | `cmp` вывода бинаря из `git archive HEAD` и нового: IDENTICAL на minirun (171932 б), acceptance (354821 б), coldrun-a (1077 б) |
| 3.1 основа — `chunktest/epub.go`, OCF в прод без `testing` | да | `TestEPUBFixtureMimetypeIsOCFConformant` и 9 импортёров зелёные без правки; `TestContainerIsOCFConformant` (флаги `0`, extra `0`, `mimetype` на фиксированных смещениях) |
| 3.1 детерминизм двумя ПРОЦЕССАМИ | да | `cmp first/<f> <copy>/<f>` — IDENTICAL для 6 файлов (две сборки в двух процессах); zip-таймметки `1980-00-00 00:00`; sha256 EPUB minirun `084ff0a9…`, совпал с независимой сборкой ревьюера |
| 3.2 пять (шесть) состояний, отказ по умолчанию / `--partial` | да | acceptance: `build` → exit 16 «pending=0 withheld=1 incomplete=1 stale=0 ghost_rows=20 of total_units=37 (stale: UNKNOWN …) · chapter 1 unit 0: withheld (cjk_artifact) · chapter 10 unit 0: incomplete — 1 member(s)… (length) · ghost rows: 1/1 1/3 …»; `--partial` → `⚠ 2/37`, `⚠ +20`, `⚠ 1.0`, `⚠ 10.0 −1`, `⚠ 10 +1`…; coldrun-a: exit 16 (14 pending) / `--partial` со словесными маркерами из `reader.txt`; minirun (только drift): записан по умолчанию, `config_drift: true`, `stale_unknown: true`, в файле `⚠` нет; тесты `TestBuildBookRefusesAPending…`, `…WithheldUnitIsAHole`, `…GhostRowsAreAHole` (посадка построена: усадка исходника после прогона), `…StaleUnitsAreAHole`, `…DriftIsReportedNotMarked`, `…RemovedStageIsDrift`, `TestAssembleBookMarksEveryHoleKind` |
| 3.2 не в снапшот | да | `current_snapshot` minirun с `ru/reader.txt` и без — `bb0b3b2aaf0d…` оба раза (копия корня пака в песочнице); `TestReaderWordsDoNotMoveThePackVersion` |
| 3.2 книга без langpack — несловесно | да | acceptance (`langpack_root` не задан): `⚠ 1.0`, `⚠ 10.0 −1`, `⚠ 2/37`; `TestReaderWordsAbsentIsTheNonVerbalForm` |
| 3.2 маркер не в `FinalText`/`export.go` | да | JSON HEAD vs new: `existing equal True`, добавлены только `text_modified`, `heading`, `ghost_units` |
| 3.3 `ApplyHeading` и три точки не тронуты; срез только известного префикса | да | `git diff HEAD -- backend/internal/chunk/chunker.go backend/internal/chunk/ingest.go backend/internal/pipeline/waverun.go` пуст; вызов `ApplyHeading` в `export.go` — контекстная строка; minirun: 10 заголовков срезаны, круг 402/402; `TestAssembleBookMarksEveryHoleKind` (упоминание заголовка в тексте не срезается) |
| 3.3 `Heading==""` → номер цифрами в nav и `<h1>` | да | acceptance nav `1`…`25`; minirun/coldrun-a — `Глава N` из `heading` |
| 3.3-бис пять требований EPUB 3; nav не в spine; `dc:language` из `TargetLang` | да | `TestEPUBCarriesTheFiveRequiredThingsAndKeepsNavOutOfTheSpine`; OPF minirun: `urn:textmachine:book:guzhenren-minirun`, `蛊真人`, `ru`, `2026-07-25T19:17:53Z`; epubcheck |
| 3.4 место/атомарность/конверт/глагол/коды | да | `status --json`/`manifest --json` копии minirun: `book_files: {epub, txt}`; `TestTheRefusalTableIsTotal`, `TestDispatchCommandsCoversTheSwitch`, `TestParseNoArgsUsage`; `TestBuildBookLeavesOneBuildBesideTheDatabase`; `TestBuildBookOutNamesANewFile` |
| 3.5 платформа не тронута; `tm-export-v1` аддитивно; пары — данные; без зависимостей | да | `git status` — только `backend/` и `docs/PROGRESS.md`; `go.mod` не менялся; греп читательских слов по новому Go-коду — только комментарии |
| §4 `make battery` (`-race`) | да, 2 настоящих прогона | `battery-1.log` (до правок ревью) и `battery-3.log` (после всех правок): `BATTERY_EXIT=0`, 18 пакетов, скипы — три прежних стендовых (`TestMinerFullBookParity`, `TestHelperEventsRun`, `TestHelperKillLoop`). ⚠ «Батарея №2» была ложным прогоном: `make` из корня репо (`No rule to make target`), мой `echo` замаскировал код — поймано по логу, перезапущено из `backend/` |
| §4 дифф `^func Test` | ни один не исчез | `git diff HEAD -- '*_test.go' \| grep '^-func Test'` пуст; `func Test` в `backend/`: 1017 → 1044 (+27) |
| §4 круг ингестом равенством | да | `tmroundtrip` (собран `go build -overlay`, файла в дереве нет): minirun 10/10 глав, 402 абзаца; acceptance `--partial` 25/25, 1568; coldrun-a `--partial` 10/10, 25 |
| §4 `git -C books status` до/после | см. obstacle | 6 строк `D …-wal/-shm` до; после — 4: у acceptance `-wal`/`-shm` ПОЯВИЛИСЬ (идентичны трекнутым; БД не изменена) |
| §4 хирургия конфигов в копиях | список | minirun/coldrun-a `book.yaml`: удалены `mined_delta:`/`mined_rejects:`; acceptance `pipeline-acceptance.yaml`: удалены `stm_depth: 2`, `overlap_tokens: 200`, оба `prompt:`; добавлен `acceptance/pairs/zh-ru.yaml` (копия minirun/pairs с `prompts_root` на `backend/prompts` этого дерева); все абсолютные пути переписаны на песочницу/этот ворктри, греп по копиям пуст; миграции v10/v7/v14 → v16, `backups/` — в песочнице |
| §4 промежуточное и финальное ревью субагентом | да, Fable 5 ×2 | ниже |
**Промежуточное ревью (Fable 5, исполнением) — 8 находок, все закрыты:** (1) MAJOR: `--format`-подмножество оставляло соседний формат прежней сборки рядом с БД при равноправном `book_files` → стейдж всех форматов + общий коммит, не запрошенные соседи удаляются, пин `TestBuildBookLeavesOneBuildBesideTheDatabase`; (2) `text_modified` считался по всем строкам всех стадий → только по финальным строкам экспорта (acceptance `20:41:47Z`→`20:40:00Z`, coldrun-a — эпоха: ничего не отгружено); (3) пустой `title` → exit 1 → `config_invalid` с именем ключа, то же для `target_lang` не в форме BCP 47; (4) `mimetype` с data-descriptor → `CreateRaw`; (5) control-символы срезались только в EPUB → `CleanText` один раз для обоих; (6) `--format epub,epub` → дедуп; (7) `backend/README.md` без `build` → дополнено; таблицы exit-кодов в `docs/` — пинг; (8) подтверждено верным: `UnitHole` = арифметика `renderExport`, `DispOK` с пустым текстом недостижим, нулевые таймметки, `reader.txt` вне `Load`.
**Финальный адверсариальный проход (Fable 5; overlay-тесты, свой бинарь, epubcheck на adversarial-книгах: краевые символы, RTL, 2000 глав) — 10 находок:**
1. MAJOR — правка исходника НА МЕСТЕ (ключи `(chapter, chunk_idx)` уцелели, текст под ними сменился) давала файл, побайтово равный прежнему, под `complete: true`. **Исправлено — шестое состояние `stale`** (см. политику; пин `TestBuildBookStaleUnitsAreAHole`: правка главы 2 → `chapter 2 unit 0: stale`, глава 1 не задета; после `translate` — снова whole). ⚠ Побочный факт: `projectRebill` сверяет content-hash ТОЛЬКО при сдвинутом снапшоте (bank-only ветка), при том же снапшоте строка идёт как «$0 resume» без сверки — проекция денег `status` слепа к правке исходника на месте, `translate` при этом перекупает (ревьюер воспроизвёл: «2 units re-bought»). Не моя строка — пинг.
2. MAJOR — `--out` перезаписывал что угодно, включая БД проекта («file is not a database») и исходник. **Исправлено:** `--out` только для НОВОГО файла; существующий путь / отсутствующий каталог → `config_invalid`; место рядом с БД по-прежнему перезаписывается. Пин `TestBuildBookOutNamesANewFile` (БД, исходник, чужой файл, несуществующий каталог; ни одного `.tmp-` после). Стенд: `--out <project_db>` → exit 10, БД цела.
3. MINOR — сбой на фазе коммита второго формата оставлял смешанную сборку и exit 1. **Исправлено:** `write_incomplete` (exit 15) с перечнем landed/not landed; комментарий файла говорит правду. Тестом не покрыто («не проверено»: сбой `rename` в только что записанном каталоге без инъекции не воспроизводится).
4. MINOR — юнит с непустым `FinalText`, дающий ноль абзацев (control-символы/`U+FFFF`), не был дырой. **Исправлено:** сборщик читает юнит по тому, что тот реально положил (ноль абзацев → withheld); счётчики отчёта — из списка дыр сборщика; `HoleCounts` остался ровно арифметикой `renderExport`.
5. MINOR — голый `\r`: txt нёс его, XML нормализует → форматы расходились. **Исправлено:** `\r\n`/`\r`/`\n` — конец строки.
6. MINOR — `--out` в несуществующий каталог → exit 1. **Исправлено:** `config_invalid`.
7. NIT — `build` молча принимал `--plaintext/--pairs/--json`. **Исправлено:** отказ.
8. NIT — `dc:title` с паддингом vs обрезанный txt. **Исправлено:** `TrimSpace` в модели и EPUB.
9. NIT — ghost-маркер по номеру главы СТАРОЙ нарезки может встать в соседнюю главу при усадке. **Не исправлено, задокументировано** (`assembleBook`): книга всё равно помечена неполной, ложна лишь координата; иного источника координаты у ghost-строки нет.
10. MINOR (унаследовано `Export`) — удаление стадии `edit` делало черновики «полной» книгой без дрейфа. **Исправлено в `exportConfigDrift`:** строка со стадией, которой нет в конфиге, — drift; под ним stale = unknown; пин `TestBuildBookRemovedStageIsDrift`. ⚠ `Status` (`checkWave`) несёт то же слепое пятно — пинг.
Подтверждено проходом как верное: все посадки A (withheld при `pending 0`, ghost-only, исчезнувшая глава, все юниты withheld, 0 юнитов, `HoleCounts` = шапка `renderExport` на всей матрице состояний); детерминизм (нет `time.Now`/`rand`/`os.Stat`/map-итерации в байты; 2 процесса под `TZ=Asia/Tokyo LANG=C` — идентично); заголовки (срез только известного префикса; pending-первый юнит с заголовком получает заголовок; пустой заголовок недостижим — `parseHeading`); nav вне spine; круг на трёх стендовых книгах воспроизведён независимо + epubcheck чист на adversarial-книгах; RTL — валиден, без `dir`, честно неподдержан; 2000 глав — 47 мс/707 КБ, ingest 38 мс; ids/hrefs по индексу; словесных литералов в Go нет; пин `Version()` подлинный; exit 16 в полосе, таблица тотальна; JSON аддитивен; `ApplyHeading` и три точки не тронуты; ни один тест не удалён; линтер 0.
**Obstacle reporting — что НЕ удалось и что осталось непроверенным.**
- **Не проверено — открывается ли в читалке.** Читалки на машине нет; «валиден» ≠ «открылся». Проверено ТОЛЬКО валидатором W3C (epubcheck 5.3.0, `0/0/0` ×3) и кругом через собственный ингест движка.
- **Не проверено — чистый бездрейфовый файл на реальной книге.** Недостижим на этом HEAD без платного перепрогона (все три копии под `config_drift: true`); чистый путь доказан синтетикой (`TestBuildBookRoundTripsThroughIngestAndPublishesItsPaths`: `complete: true`, `stale_unknown: false`, без пометок). Следствие: **stale-проверка на реальных книгах стенда не сделана** (`stale_unknown: true` у всех трёх — под дрейфом факт неразличим), доказана только харнесом.
- **Не проверено — сбой между двумя `rename`** (находка 3): ветка `write_incomplete` написана, не исполнена.
- **Не проверено — посадка на incomplete в `pipeline`-харнесе:** реальный member-drop только через бинарь (gap-харнесс `cmd/tmctl`) и acceptance (10/0, `length`); в `internal/pipeline` — синтетический `BookExport`.
- **Не проверено — направление письма.** `dir`/`page-progression-direction` не выставляются; данных о направлении в `internal/lang` нет; RTL-файл валиден, но выровнен слева — честно неподдержано (вопрос 4).
- **Не объяснено — read-only открытие ОРИГИНАЛЬНОЙ acceptance-БД.** `git -C books status --short -- gu-zhenren` до: 6 строк `D …-wal/-shm`; после: 4 — у `acceptance/guzhenren-acc-a.db` `-wal` (0 байт, 13:47:36 +03) и `-shm` (32 КБ, 13:47:49 +03) появились, идентичны трекнутым (`git diff` пуст), сама БД (mtime 24.08) не изменена. Мои команды все шли по копиям (`project_db`/`source_file` копии — в песочнице, проверено грепом; `.lock` копии — от миграции 12:56); тесты дерева acceptance-БД не открывают (греп); субагент-ревьюер завершился раньше этого времени. В `ListAgents` живы ещё три сессии (оркестратор и две, вероятно, полигонные) — кто-то из них открыл оригинал read-only. Файлы НЕ трогал (чужой репозиторий).
- **Опровергнуто (промтом):** «глава 1 minirun остаётся `""` даже с правилом» — на HEAD у minirun заголовок у всех 10 глав (`headings 10`, nav `Глава 1…Глава 10`); `""` у всех 25 глав acceptance — подтверждено.
- **Подтверждено:** числа промта §4 (minirun 14/14 ok · drift · оракул июля 14/14 по `final_text/snapshot_id/disposition`; acceptance 37 юнитов · ghost 20 · withheld 1/0 · incomplete 10/0 · drift; coldrun-a 14/14 pending · drift); миграции v10/v7/v14 → v16; epubcheck приносится и работает.
- **Ограничения формы (по промту, не исправлялись):** `dc:title` = `蛊真人` (перевод не выдумывается); автора нет; `text_modified` — «последняя ЗАПИСЬ» финальной строки ($0-резюм, переписавший строку без изменений, двигает штамп; между сборками детерминизм сохраняется); побайтовый детерминизм гарантирован для ОДНОГО бинаря (другой toolchain Go — иной deflate-поток при тех же документах); ghost-координата — старой нарезки (находка 9); `stale` под дрейфом — «неизвестно».
- **Самопойманная ошибка:** ложный прогон «батареи №2» (см. таблицу).
- **Эхо §7:** правок задания релеем владельца не поступало; единственное сообщение оркестратора (подтверждение адресата + напоминания) заказом не является; приор «артефакт рядом с ProjectDB + `StatusArtifacts`» принят с доводом.
**Вопросы владельцу (гейт между этим паком и дверью платформы, D25.1):**
1. **Лестница `run_complete / structurally_complete / quality_reviewed / publishable`:** писатель различает ровно «без дыр» (`complete: true`) и «с дырами» (отказ / `--partial` с пометкой). Что из лестницы печатать на титуле и в имени файла и какая ступень даёт копию БЕЗ пометки — решение владельца; в код не внесено намеренно.
2. **Лейбл главы без заголовка (К-3):** сейчас — номер цифрами. Словесный шаблон — одна строка в `reader.txt`, если ратифицируется; не сделано, чтобы не решать К-3 данными.
3. **Ghost и stale как дыры по умолчанию:** довод — оба дают «молча полную» книгу (acceptance: 20 строк не в файле при `pending 0`; правка исходника на месте: старый перевод под `complete: true`). Если владелец хочет «предупреждение, не отказ», — по строке в `BuildBook`.
4. **Направление письма для RTL-цели:** поле данных цели (в `reader.txt` или рядом), не Go — когда появится RTL-пара.
**Пинги оркестратору (чужие зоны, не правил):** (а) exit-код **16 `book_incomplete`** в таблицы `docs/glossary.md:64` и `docs/architecture/14-api-contract/README.md:731,758`; (б) `docs/architecture/09-target-architecture.md`: слой «писатель книги» построен; следующий шаг — дверь платформы `createExport`/`getExport` + `export_formats` из `bookfile.Formats`; дверь обязана СТРОИТЬ (`tmctl build --format <f> [--partial]`) и отдавать то, что построила, — не читать файл, лежащий рядом с БД (копия прежней сборки); (в) `projectRebill`/`status` слепы к правке исходника на месте при том же снапшоте (перекупка без предупреждения о деньгах) — строка бэклога; (г) `Status.checkWave` не видит исчезнувшую стадию (то же, что находка 10 у `Export`) — строка бэклога; (д) строки 160/201 не тронуты, `Heading` в `tm-export-v1` — тот же временный рендер (D39.122 п.2д), помечено в комментарии поля.
### 📝 ЗАПИСКА-ПЛАН — пак «деньги» (промт `docs/archive/prompts/BACKEND_MONEY_PACK_SESSION_PROMPT_2026-08-29.md`, D39.165 §1б + строка 231), сессия `textmachine-e4`, 28.08
**Скоуп:** `backend/` — §3.1 потолок ОБЪЁМА рядом с денежным · §3.2 смета пере-прохода без покупки. `platform/` не трогаю (проводка `CeilingChapters` — не мой заказ, честная граница §0 промта принята).