From 8adcb8611b66148d14be7ccdd6afddaeb7a915c8 Mon Sep 17 00:00:00 2001 From: heaven Date: Sun, 30 Aug 2026 20:14:11 +0300 Subject: [PATCH] Land the book-writer pack: tmctl build renders EPUB 3 and clean text from BookExport with six honesty states, refusing or marking every hole, paths published in StatusArtifacts --- backend/README.md | 2 +- backend/cmd/tmctl/build_cli_test.go | 167 +++++ backend/cmd/tmctl/invocation.go | 64 +- backend/cmd/tmctl/invocation_test.go | 7 +- backend/cmd/tmctl/main.go | 30 +- backend/cmd/tmctl/render.go | 23 +- backend/configs/langpacks/ru/reader.txt | 21 + backend/internal/bookfile/bookfile_test.go | 344 ++++++++++ backend/internal/bookfile/container.go | 77 +++ backend/internal/bookfile/epub.go | 139 +++++ backend/internal/bookfile/model.go | 165 +++++ backend/internal/bookfile/txt.go | 37 ++ backend/internal/chunk/chunktest/epub.go | 40 +- backend/internal/lang/reader.go | 150 +++++ backend/internal/lang/reader_test.go | 122 ++++ backend/internal/pipeline/bookbuild.go | 598 ++++++++++++++++++ backend/internal/pipeline/bookbuild_test.go | 657 ++++++++++++++++++++ backend/internal/pipeline/export.go | 89 ++- backend/internal/pipeline/holes.go | 58 ++ backend/internal/pipeline/refusal.go | 12 + backend/internal/pipeline/status.go | 7 + backend/internal/store/chunkstatus.go | 15 +- docs/PROGRESS.md | 164 +++++ 23 files changed, 2941 insertions(+), 47 deletions(-) create mode 100644 backend/cmd/tmctl/build_cli_test.go create mode 100644 backend/configs/langpacks/ru/reader.txt create mode 100644 backend/internal/bookfile/bookfile_test.go create mode 100644 backend/internal/bookfile/container.go create mode 100644 backend/internal/bookfile/epub.go create mode 100644 backend/internal/bookfile/model.go create mode 100644 backend/internal/bookfile/txt.go create mode 100644 backend/internal/lang/reader.go create mode 100644 backend/internal/lang/reader_test.go create mode 100644 backend/internal/pipeline/bookbuild.go create mode 100644 backend/internal/pipeline/bookbuild_test.go create mode 100644 backend/internal/pipeline/holes.go diff --git a/backend/README.md b/backend/README.md index b36472e6..baf2ba2b 100644 --- a/backend/README.md +++ b/backend/README.md @@ -8,7 +8,7 @@ | Пакет | Что делает | Ключевые файлы | |---|---|---| -| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) / `manifest` (**$0-производитель персиста манифеста глав/чанков**, строка 100: нужен ДО первого прогона — дерево глав разобранной, но не запущенной книги; `--json` печатает сам документ) / `migrate` (**$0-write-open без прогона**, строка 174: read-only команды требуют ТОЧНОГО совпадения схемы и не мигрируют, поэтому апгрейд бинаря запирал все существующие книги — платформа зовёт `status --json` перед каждым спавном, а write-команда, которая мигрировала бы, не наступала никогда. Грузит ТОЛЬКО `book.yaml` — ни цен, ни ключей на $0-пути (строка 146); restore point берёт ПОД ЛОКОМ и только когда шаг реально применяется (имя `<метка>-pre-migrate.db` — вне секундного неймспейса платного пути, строка 173); несовпадение схемы у read-only путей — типизированный отказ **exit 13** с машинным токеном `schema_mismatch found=N expected=M`). **ДВА потолка прогона, и они ортогональны.** `--ceiling-usd` (строка 145) — ДЕНЬГИ: перекрывает книжный `ceilings.book_usd` ТОЛЬКО на этот прогон, в book.yaml не пишется, ноль/отрицательное = отказ запуска; каппит КУМУЛЯТИВНУЮ трату книги, а не приращение прогона. `--max-units` (D39.165 §1б) — ОБЪЁМ: не больше N ВЫХОДНЫХ ЮНИТОВ (гранулярность `units_total` манифеста, та самая, в которой платформа продаёт главы) будет ОПЛАЧЕНО этим прогоном. Юниты, которые прогон отдаёт за $0 (резюм, ре-пин), едут бесплатно и потолок не тратят; ретраи и эскалации — тоже нет, они внутри юнита. Принимает только `translate`. **Остановка по объёму — ЗАВЕРШЕНИЕ (exit 0), а не пауза:** словарь кодов выхода не расширялся, различение живёт в отчёте прогона и в логе. Отчёт различает ДОСТАВКУ (юнит, которого не было) и ПЕРЕ-ДЕЛКУ (уже доставленный юнит под сдвинутым снапшотом) — покупка, целиком ушедшая в переделку, обязана читаться как переделка. ⚠ На книге, которая МАЙНИТ банк, вторая покупка требует `--resnapshot` (авто-банк растёт между покупками и двигает edit-снапшот) — прогон предупреждает об этом в логе. main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` | +| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `build` (**писатель книги**, строка 236: EPUB 3 + чистый txt из той же экспорт-проекции в `.book.` рядом с БД, пути — в `artifacts.book_files`; книга с ЛЮБОЙ дырой — pending/withheld/incomplete/ghost — по умолчанию отказ **exit 16** с перечнем, `--partial` пишет с пометкой на первой странице и маркером на каждой дыре; слова читателя — данные `configs/langpacks/<цель>/reader.txt`, вне снапшота; `--format epub,txt`, `--out`) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) / `manifest` (**$0-производитель персиста манифеста глав/чанков**, строка 100: нужен ДО первого прогона — дерево глав разобранной, но не запущенной книги; `--json` печатает сам документ) / `migrate` (**$0-write-open без прогона**, строка 174: read-only команды требуют ТОЧНОГО совпадения схемы и не мигрируют, поэтому апгрейд бинаря запирал все существующие книги — платформа зовёт `status --json` перед каждым спавном, а write-команда, которая мигрировала бы, не наступала никогда. Грузит ТОЛЬКО `book.yaml` — ни цен, ни ключей на $0-пути (строка 146); restore point берёт ПОД ЛОКОМ и только когда шаг реально применяется (имя `<метка>-pre-migrate.db` — вне секундного неймспейса платного пути, строка 173); несовпадение схемы у read-only путей — типизированный отказ **exit 13** с машинным токеном `schema_mismatch found=N expected=M`). **ДВА потолка прогона, и они ортогональны.** `--ceiling-usd` (строка 145) — ДЕНЬГИ: перекрывает книжный `ceilings.book_usd` ТОЛЬКО на этот прогон, в book.yaml не пишется, ноль/отрицательное = отказ запуска; каппит КУМУЛЯТИВНУЮ трату книги, а не приращение прогона. `--max-units` (D39.165 §1б) — ОБЪЁМ: не больше N ВЫХОДНЫХ ЮНИТОВ (гранулярность `units_total` манифеста, та самая, в которой платформа продаёт главы) будет ОПЛАЧЕНО этим прогоном. Юниты, которые прогон отдаёт за $0 (резюм, ре-пин), едут бесплатно и потолок не тратят; ретраи и эскалации — тоже нет, они внутри юнита. Принимает только `translate`. **Остановка по объёму — ЗАВЕРШЕНИЕ (exit 0), а не пауза:** словарь кодов выхода не расширялся, различение живёт в отчёте прогона и в логе. Отчёт различает ДОСТАВКУ (юнит, которого не было) и ПЕРЕ-ДЕЛКУ (уже доставленный юнит под сдвинутым снапшотом) — покупка, целиком ушедшая в переделку, обязана читаться как переделка. ⚠ На книге, которая МАЙНИТ банк, вторая покупка требует `--resnapshot` (авто-банк растёт между покупками и двигает edit-снапшот) — прогон предупреждает об этом в логе. main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` | | `internal/llm` | OpenAI-совместимый транспорт + retry/backoff, **capability-слой** (budget_field/temperature/reasoning per-модель), провайдеры openai/local (no-proxy)/anthropic (DEPRECATED-референс); `failover.go` удалён паком-17 — маршрутизация лейблами живёт в config/pipeline (канал B, один хоп `chain[0]`, fail-closed) | `httpllm.go`, `capability.go`, `provider_*.go` | | `internal/ledger` | Цены по usage (вкл. reasoning/cache-поля), `PriceForResponse` по фактической модели | `pricing.go` | | `internal/store` | SQLite (modernc, CGO-free), цепочка миграций `schema_version` (⚠ ALTER-шаги v8+ не идемпотентны вопреки шапке — бэклог-строка 49а), reserve/settle+checkpoint, chunk_status, глоссарий (+подписной цикл терминолога), ruby, retrieval_state, request_log; `OpenReadOnly` — без flock/миграций/recovery для status/report/export (схема не совпала — типизированный `*store.SchemaMismatchError{Found,Expected}`, обе стороны); `Migrate` (+шов `beforeApply` под локом: restore point берётся там) — поверхность деплой-шага строки 174, `SchemaHead` — та самая `Expected`; write-open БД новее бинаря теперь ОТКАЗ, а не тихое открытие | `ledger.go`, `migrate.go`, `glossary.go`, `store.go` | diff --git a/backend/cmd/tmctl/build_cli_test.go b/backend/cmd/tmctl/build_cli_test.go new file mode 100644 index 00000000..93dbb6d2 --- /dev/null +++ b/backend/cmd/tmctl/build_cli_test.go @@ -0,0 +1,167 @@ +package main + +import ( + "bytes" + "encoding/json" + "errors" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// build_cli_test.go: `tmctl build` at the shell — the flag contract, and the incomplete state driven +// through the REAL binary on the c-lite harness (setupGapProject), because the claim is about the bytes +// a reader gets and the exit code a supervisor gets. + +func TestParseBuildFlags(t *testing.T) { + inv, err := parseInvocation([]string{"build", "--config", "b.yaml", "--format", "epub", "--out", "x.epub", "--partial"}, &bytes.Buffer{}) + if err != nil { + t.Fatal(err) + } + if inv.cmd != "build" || len(inv.buildFormats) != 1 || inv.buildFormats[0] != "epub" || inv.buildOut != "x.epub" || !inv.buildPartial { + t.Fatalf("parsed: %+v", inv) + } + inv, err = parseInvocation([]string{"build", "--config", "b.yaml"}, &bytes.Buffer{}) + if err != nil || inv.buildFormats != nil || inv.buildOut != "" || inv.buildPartial { + t.Fatalf("bare build: %+v %v", inv, err) + } + inv, err = parseInvocation([]string{"build", "--config", "b.yaml", "--format", "txt, epub"}, &bytes.Buffer{}) + if err != nil || strings.Join(inv.buildFormats, ",") != "txt,epub" { + t.Fatalf("comma list: %+v %v", inv.buildFormats, err) + } + for name, args := range map[string][]string{ + "export's --plaintext": {"build", "--config", "b.yaml", "--plaintext"}, + "export's --pairs": {"build", "--config", "b.yaml", "--pairs"}, + "status's --json": {"build", "--config", "b.yaml", "--json"}, + "unknown format": {"build", "--config", "b.yaml", "--format", "pdf"}, + "empty format": {"build", "--config", "b.yaml", "--format", ""}, + "out without a format": {"build", "--config", "b.yaml", "--out", "x"}, + "out with two formats": {"build", "--config", "b.yaml", "--format", "epub,txt", "--out", "x"}, + "empty out": {"build", "--config", "b.yaml", "--format", "epub", "--out", ""}, + } { + if _, err := parseInvocation(args, &bytes.Buffer{}); err == nil { + t.Errorf("%s: must be refused", name) + } + } +} + +// TestBuildFlagsAreRefusedElsewhere: a command that does not act on a flag refuses it — the rule the +// other flags follow, enumerated from dispatchCommands. +func TestBuildFlagsAreRefusedElsewhere(t *testing.T) { + for _, cmd := range dispatchCommands { + if cmd == "build" { + continue + } + for _, flag := range [][]string{{"--partial"}, {"--format", "epub"}, {"--out", "x"}} { + args := append([]string{cmd, "--config", "b.yaml"}, flag...) + if cmd == "bank-apply" { + args = append(args, "--decisions", "d.json") + } + if cmd == "seed-lint" { + args = append([]string{cmd, "--seed", "s.yaml"}, flag...) + } + _, err := parseInvocation(args, &bytes.Buffer{}) + if err == nil { + t.Errorf("%s must refuse %v", cmd, flag) + continue + } + rest := strings.Replace(err.Error(), `"`+cmd+`"`, "", 1) + if !strings.Contains(rest, cmd) && !strings.Contains(rest, "flag provided but not defined") { + t.Errorf("the reason given to `%s` names it only to refuse it: %v", cmd, err) + } + } + } +} + +// runTmctlCode is runTmctl without the exit-code assertion: it returns what the process said and how it +// ended, for the commands whose refusal IS the thing under test. +func runTmctlCode(t *testing.T, bookPath string, args ...string) (stdout, stderr string, code int) { + t.Helper() + cmd := exec.Command(buildTmctl(t), append(args, "--config", bookPath)...) + var out, errb bytes.Buffer + cmd.Stdout, cmd.Stderr = &out, &errb + err := cmd.Run() + return out.String(), errb.String(), exitCodeOf(t, err) +} + +func TestBuildRefusesTheGapAndPartialMarksIt(t *testing.T) { + const echoMarker = "禁" + srv := gapProvider(t, echoMarker) + bookPath := setupGapProject(t, srv.URL, echoMarker) + dir := filepath.Dir(bookPath) + runTmctl(t, bookPath, "translate") + + // The state really is the one under test: pending 0, one INCOMPLETE unit. + var exp map[string]any + if err := json.Unmarshal([]byte(runTmctl(t, bookPath, "export")), &exp); err != nil { + t.Fatal(err) + } + if exp["pending_units"].(float64) != 0 { + t.Fatalf("fixture: pending must be 0: %v", exp) + } + + stdout, stderr, code := runTmctlCode(t, bookPath, "build") + if code != exitBookIncomplete { + t.Fatalf("a book with a hole must be refused with exit %d, got %d\nstdout:%s\nstderr:%s", exitBookIncomplete, code, stdout, stderr) + } + if !strings.Contains(stderr, "chapter 1 unit 0: incomplete") || !strings.Contains(stderr, "cjk_artifact") { + t.Errorf("the refusal must name the hole and its cause:\n%s", stderr) + } + for _, f := range []string{"gap-book.db.book.epub", "gap-book.db.book.txt"} { + if _, err := os.Stat(filepath.Join(dir, f)); !errors.Is(err, os.ErrNotExist) { + t.Errorf("a refusal writes nothing, but %s exists (%v)", f, err) + } + } + + stdout, stderr, code = runTmctlCode(t, bookPath, "build", "--partial") + if code != 0 { + t.Fatalf("--partial must write: exit %d\nstderr:%s", code, stderr) + } + var rep struct { + Version string `json:"build_version"` + Files map[string]string `json:"files"` + IncompleteUnits int `json:"incomplete_units"` + Complete bool `json:"complete"` + } + if err := json.Unmarshal([]byte(stdout), &rep); err != nil { + t.Fatalf("stdout is not the report: %v\n%s", err, stdout) + } + if rep.Version != "tm-build-v1" || rep.IncompleteUnits != 1 || rep.Complete { + t.Errorf("report: %+v", rep) + } + txt, err := os.ReadFile(rep.Files["txt"]) + if err != nil { + t.Fatal(err) + } + shipped := string(txt) + t.Logf("READER'S TEXT:\n%s", shipped) + // No langpack → the non-verbal mark ahead of the surviving text; the edited text is there; the audit + // vocabulary of `export --plaintext` is not. + if !strings.Contains(shipped, "⚠ 1.0 −1\n\nОтредактированный текст уцелевшей части.") { + t.Errorf("the mark must stand right before the surviving text:\n%s", shipped) + } + if !strings.HasPrefix(shipped, "Тест\n\n⚠ 1/1\n\n\n1\n\n") { + t.Errorf("the notice must open the book:\n%s", shipped) + } + for _, banned := range []string{"=== CHAPTER", "TEXT MISSING", "INCOMPLETE", "cjk_artifact", "flagged"} { + if strings.Contains(shipped, banned) { + t.Errorf("operator vocabulary %q reached the reader:\n%s", banned, shipped) + } + } + // status publishes the same places the build wrote to. + var st struct { + Artifacts struct { + BookFiles map[string]string `json:"book_files"` + } `json:"artifacts"` + } + if err := json.Unmarshal([]byte(runTmctl(t, bookPath, "status", "--json")), &st); err != nil { + t.Fatal(err) + } + for f, p := range rep.Files { + if st.Artifacts.BookFiles[f] != p { + t.Errorf("status artifacts.book_files[%s] = %q, build wrote %q", f, st.Artifacts.BookFiles[f], p) + } + } +} diff --git a/backend/cmd/tmctl/invocation.go b/backend/cmd/tmctl/invocation.go index bf55115a..94b14a5c 100644 --- a/backend/cmd/tmctl/invocation.go +++ b/backend/cmd/tmctl/invocation.go @@ -6,7 +6,9 @@ import ( "io" "math" "strconv" + "strings" + "textmachine/backend/internal/bookfile" "textmachine/backend/internal/pipeline" ) @@ -49,7 +51,14 @@ type invocation struct { // invocation property for exactly the reason ceilingUSD is — how much of the book a caller is buying // right now belongs to the call, not to the book's data. maxUnits int - sel pipeline.RedriveSelector + // buildFormats / buildOut / buildPartial are `build`'s own (backlog row 236): which reader's copies + // to write, where the one file goes when the caller names a place, and whether a book with holes is + // written marked rather than refused. Invocation properties like the flags above — a caller's + // choice for one build, not a fact about the book. + buildFormats []string + buildOut string + buildPartial bool + sel pipeline.RedriveSelector } // rebillConsentValue parses `--accept-rebill[=usd]` (D20.2-Q2): the OPTIONAL-VALUE form of Р6, where a @@ -112,7 +121,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { // tells the operator they do not exist. seed-lint and bank-apply are on their own clauses because // their arguments differ — seed-lint takes --seed and no --config, bank-apply needs --decisions — // and folding them into the first clause would be the same class of lie. - return invocation{}, fmt.Errorf("usage: tmctl --config book.yaml | tmctl bank-apply --config book.yaml --decisions decisions.json | tmctl seed-lint --seed glossary.yaml") + return invocation{}, fmt.Errorf("usage: tmctl --config book.yaml | tmctl bank-apply --config book.yaml --decisions decisions.json | tmctl seed-lint --seed glossary.yaml") } cmd, rest := args[0], args[1:] @@ -135,6 +144,9 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { seed := fs.String("seed", "", "seed-lint: path to the glossary seed YAML to validate ($0, no --config)") keysFile := fs.String("keys-file", "", "translate: path to the DEPLOYMENT's provider-key file (KEY=VALUE lines). Loaded FIRST, so it wins over the .env beside book.yaml; a named file that cannot be read is a refusal, never a silent skip") decisions := fs.String("decisions", "", "bank-apply: path to the JSON decision document to apply to the book's memory bank ($0)") + buildFormat := fs.String("format", "", "build: the reader's copies to write, comma-separated ("+strings.Join(bookfile.Formats, ",")+"); default: all of them") + buildOut := fs.String("out", "", "build: write the ONE requested format to this path instead of beside the project database (needs --format with a single format)") + buildPartial := fs.Bool("partial", false, "build: write the book even when it is not whole — the notice is on its first page and every hole (a unit not translated, withheld, or missing a piece; text the current cut cannot place) is marked in the text. Without it a book with any hole is REFUSED (exit 16) and nothing is written") if err := fs.Parse(rest); err != nil { return invocation{}, err } @@ -159,7 +171,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { // is taught something false about the system. It names them all, and the test that guards it // enumerates dispatchCommands rather than a hand-written list — the hand list is what left // `bank-apply`, the very verb this pack added, undescribed by its own refusal. - return invocation{}, fmt.Errorf("--keys-file is accepted by `translate` only, not by %q: `translate` is the one command that spends. The $0 read commands (report/status/export/manifest/seed-lint/bank-apply) must not demand provider keys at all (D20.4); `redrive` does re-attack and re-bill, but takes its keys from its own conventional .env; `backup` and `migrate` are deploy steps that call no provider", cmd) + return invocation{}, fmt.Errorf("--keys-file is accepted by `translate` only, not by %q: `translate` is the one command that spends. The $0 read commands (report/status/export/build/manifest/seed-lint/bank-apply) must not demand provider keys at all (D20.4); `redrive` does re-attack and re-bill, but takes its keys from its own conventional .env; `backup` and `migrate` are deploy steps that call no provider", cmd) } // A PRESENT flag with an empty value is refused rather than ignored, for the reason the flag // exists: an unset variable in a deployment's unit file would otherwise fall back to the @@ -211,6 +223,51 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { return invocation{}, fmt.Errorf("--max-units must be a positive number of output units (it is the volume ceiling for THIS run); got %d — omit the flag to run the whole book", *maxUnits) } } + // `build`'s flags follow the same rule: a command that does not act on them refuses them. The silent + // version of --partial is the expensive one in the reader's currency rather than in dollars — a + // caller who passed it to `export` would believe they had asked for an honest, marked copy and get + // the audit projection, which marks nothing. + for _, name := range []string{"format", "out", "partial"} { + if given[name] && cmd != "build" { + return invocation{}, fmt.Errorf("--%s is accepted by `build` only, not by %q: it shapes the reader's copy of the book that `build` writes, and %q writes no such copy — accepting it would tell a caller a file was written the way they asked when nothing was", name, cmd, cmd) + } + } + // And the converse: `build` prints its report as JSON and writes files, so export's and status's + // output flags mean nothing to it — refused, not ignored (`build --plaintext` would otherwise read as + // «the text file, please» and quietly do something else). + if cmd == "build" { + for _, name := range []string{"plaintext", "pairs", "json"} { + if given[name] { + return invocation{}, fmt.Errorf("--%s is not a `build` flag: build writes the reader's copies (--format epub,txt) and prints its report as JSON regardless; accepting it would tell a caller it changed the output when it did not", name) + } + } + } + var buildFormats []string + if given["format"] { + seen := map[string]bool{} + for _, f := range strings.Split(*buildFormat, ",") { + f = strings.TrimSpace(f) + if f == "" || seen[f] { + continue // a format named twice is one file, not two writes of it + } + if !bookfile.KnownFormat(f) { + return invocation{}, fmt.Errorf("--format %q is not a format `build` writes (want one or more of %s, comma-separated)", f, strings.Join(bookfile.Formats, ",")) + } + seen[f] = true + buildFormats = append(buildFormats, f) + } + if len(buildFormats) == 0 { + return invocation{}, fmt.Errorf("--format was given with no format: an unset variable must not read as «all formats» — omit the flag for that") + } + } + if given["out"] { + if *buildOut == "" { + return invocation{}, fmt.Errorf("--out was given with no path: an unset variable must not read as «the default place»") + } + if len(buildFormats) != 1 { + return invocation{}, fmt.Errorf("--out names ONE file, so it needs --format with exactly one format (got %d)", len(buildFormats)) + } + } // seed-lint validates a standalone seed YAML — it takes --seed, not --config (no book/store/keys). if cmd == "seed-lint" { if *seed == "" { @@ -249,6 +306,7 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { cmd: cmd, cfgPath: *cfgPath, resnapshot: *resnapshot, acceptRebill: acceptRebill.c, asJSON: *asJSON, asPlaintext: *asPlaintext, asPairs: *asPairs, verifyBank: *verifyBank, ceilingUSD: *ceilingUSD, maxUnits: *maxUnits, keysFile: *keysFile, decisionsPath: *decisions, dryRun: *dryRun, + buildFormats: buildFormats, buildOut: *buildOut, buildPartial: *buildPartial, sel: pipeline.RedriveSelector{ Chapter: *chapter, ChunkIdx: *chunk, Reason: *reason, DryRun: *dryRun, }, diff --git a/backend/cmd/tmctl/invocation_test.go b/backend/cmd/tmctl/invocation_test.go index b8911ba2..4a565c83 100644 --- a/backend/cmd/tmctl/invocation_test.go +++ b/backend/cmd/tmctl/invocation_test.go @@ -26,9 +26,10 @@ func TestParseNoArgsUsage(t *testing.T) { // learned: `backup`/`migrate` (the deploy step, row 174) and `seed-lint`, which gets its own clause // because it takes --seed rather than --config (row 176, sanctioned by D39.134 п.3). `bank-apply` is // the sixth (D39.156 — the inbound door for the owner's bank decisions) and gets its own clause for - // the same reason seed-lint does: it needs --decisions on top of --config. Every one is a deliberate - // contract extension and the rest of the usage text stays frozen. - if err == nil || err.Error() != "usage: tmctl --config book.yaml | tmctl bank-apply --config book.yaml --decisions decisions.json | tmctl seed-lint --seed glossary.yaml" { + // the same reason seed-lint does: it needs --decisions on top of --config. `build` is the seventh + // (backlog row 236 — the reader's copy of the book, EPUB/text, beside the database). Every one is a + // deliberate contract extension and the rest of the usage text stays frozen. + if err == nil || err.Error() != "usage: tmctl --config book.yaml | tmctl bank-apply --config book.yaml --decisions decisions.json | tmctl seed-lint --seed glossary.yaml" { t.Fatalf("usage error text is frozen, got: %v", err) } // The line and the dispatch switch are the same list: a command reachable in run() and missing here diff --git a/backend/cmd/tmctl/main.go b/backend/cmd/tmctl/main.go index 1bf71dc5..a25839df 100644 --- a/backend/cmd/tmctl/main.go +++ b/backend/cmd/tmctl/main.go @@ -1,5 +1,5 @@ -// tmctl is the TextMachine CLI: translate / report / status / export / redrive / manifest / backup / -// migrate / seed-lint / bank-apply (dispatchCommands). +// tmctl is the TextMachine CLI: translate / report / status / export / build / redrive / manifest / +// backup / migrate / seed-lint / bank-apply (dispatchCommands). // main.go — thin wiring (package №4): argument parsing — invocation.go, // output renderers — render.go, .env — dotenv.go; here just the // «parse → env → ctx → fetch → render» wiring and exit-code mapping. @@ -72,6 +72,7 @@ const ( exitSchemaMismatch = 13 // this project's schema is not this binary's: run `tmctl migrate` (row 174) exitDecisionsRejected = 14 // the decision document was read and declined; the USER re-decides (D39.156) exitWriteIncomplete = 15 // the document was accepted and the write did not complete; the report says which file landed — re-send the same document + exitBookIncomplete = 16 // `build` refused to write a book with holes; the message lists them — finish the book, or ask for the marked copy with --partial exitRefusedOther = 19 // a refusal class this build of tmctl has no number for ) @@ -83,6 +84,7 @@ var refusalExit = map[pipeline.RefusalClass]int{ pipeline.RefusalSchemaMismatch: exitSchemaMismatch, pipeline.RefusalDecisionsRejected: exitDecisionsRejected, pipeline.RefusalWriteIncomplete: exitWriteIncomplete, + pipeline.RefusalBookIncomplete: exitBookIncomplete, } // exitCode maps a run() error onto the ratified shell contract (Milestone 2 / R1-FL-A): @@ -179,7 +181,7 @@ func traceID() string { // because the usage line and the unknown-command hint were two hand-maintained copies of it, and the // usage line fell three commands behind (row 176) while the hint stayed current. var dispatchCommands = []string{ - "translate", "report", "status", "export", "redrive", "manifest", "backup", "migrate", "seed-lint", + "translate", "report", "status", "export", "build", "redrive", "manifest", "backup", "migrate", "seed-lint", "bank-apply", } @@ -226,6 +228,8 @@ func run() error { return status(ctx, inv.cfgPath, inv.asJSON) case "export": return export(inv.cfgPath, inv.asPlaintext, inv.asPairs) + case "build": + return build(inv.cfgPath, inv.buildFormats, inv.buildOut, inv.buildPartial) case "redrive": // A dry-run redrive spends nothing and mutates nothing; only a real redrive re-attacks and re-bills. if !inv.sel.DryRun { @@ -360,6 +364,26 @@ func export(cfgPath string, asPlaintext, asPairs bool) error { return renderExport(os.Stdout, exp, asPlaintext) } +// build writes the READER's copy of the book (backlog row 236): EPUB and/or plain text from the same +// export projection `export` prints, beside the project database (pipeline.BuildBook). $0 and key-less +// like export — it reads the store and writes a file; no provider is called. A book with any hole is +// refused (exit 16, the holes listed) unless --partial asks for the marked copy. Its own verb rather than a +// flag on `export`: `export` is a pure read whose stdout IS its output, and a command that writes files +// beside the database is a different thing to script around (the `manifest` precedent). +func build(cfgPath string, formats []string, out string, partial bool) error { + r, err := pipeline.NewReadOnlyRunner(cfgPath, obs.NewLogger()) + if err != nil { + return err + } + defer r.Close() + + rep, err := r.BuildBook(pipeline.BuildOptions{Formats: formats, Out: out, Partial: partial}) + if err != nil { + return err + } + return renderBuild(os.Stdout, rep) +} + // status prints the READ-ONLY progress projection (D15.3): 0 LLM, 0 replay. --json emits the // StatusReport with stable disposition/flag_reason enums for CI/IDE; the default is a human // dashboard (unit counts, per-chapter quality passports, money, secondary ETA). BOTH modes exit 2 diff --git a/backend/cmd/tmctl/render.go b/backend/cmd/tmctl/render.go index 7b7f6523..13c9208b 100644 --- a/backend/cmd/tmctl/render.go +++ b/backend/cmd/tmctl/render.go @@ -570,16 +570,9 @@ func renderExport(w io.Writer, exp *pipeline.BookExport, asPlaintext bool) error // (exported = total − pending), so a book that shipped nothing for two units still // announced them as exported — the same lie as the mislabelled banner below, told in // numbers: a reader who trusts the summary never learns to look. - withheld, incomplete := 0, 0 - for _, ce := range exp.Chunks { - switch { - case ce.Disposition == "pending": - case ce.FinalText == "": - withheld++ - case ce.DroppedMembers > 0: - incomplete++ - } - } + // The predicate is pipeline.UnitHole — ONE definition shared with the book writer, so the + // numbers this header prints and the holes `tmctl build` marks can never disagree. + withheld, incomplete := pipeline.HoleCounts(exp) // `incomplete` is a SUBSET of `exported`, not a fourth part of the total: those units DID ship // text, with a piece of it missing. Spelled that way so a reader adding the numbers up is not // misled into thinking they partition the book. @@ -636,6 +629,16 @@ func flagParen(r string) string { return " (" + r + ")" } +// renderBuild prints the build report as indented JSON: the versioned envelope every JSON output of the +// engine carries (17-seam-inbound-law п.3), and the surface the platform's export door will read the +// file paths from. JSON only: the consumer is a caller that opens the files, and the human reads paths +// out of JSON as well as out of prose. +func renderBuild(w io.Writer, rep *pipeline.BuildReport) error { + enc := json.NewEncoder(w) + enc.SetIndent("", " ") + return enc.Encode(rep) +} + // renderManifest prints the persisted chapter/chunk manifest (row 100). The default is a human summary — // a 2283-chapter tree is not something anyone reads in a terminal — and --json emits the document itself, // byte-identical to the sidecar, so a consumer can pipe it instead of knowing where the file lives. diff --git a/backend/configs/langpacks/ru/reader.txt b/backend/configs/langpacks/ru/reader.txt new file mode 100644 index 00000000..d4d23b96 --- /dev/null +++ b/backend/configs/langpacks/ru/reader.txt @@ -0,0 +1,21 @@ +# Reader-facing words for a RUSSIAN-language book file (internal/lang/reader.go). These are RENDER data — +# what the file itself says to its reader at a hole in the text and on the first page of a book that is not +# whole. They are keyed by the TARGET language (the language the reader reads), not by the pair. +# +# ⚠ NOT part of the language pack: this directory is read by lang.LoadReaderWords only. The pack loader +# (lang.Load) reads / and -/ and folds those bytes into Pack.Version() and thus the +# run snapshot; a target directory is outside it by construction, so an edit here moves no snapshot and +# re-bills no book. Keep it that way — a wording change must never cost a re-translation. +# +# `keyvalue` per line; every key is required. Placeholders in braces are filled by the engine: +# {chapter} {unit} — the hole's coordinates (dense chapter number, output unit index within it) +# {dropped} — how many source fragments of the unit are missing from its text +# {ghost} — how many translated fragments the current cut of the book cannot place +# {holes} {total} — units with a hole / units in the book +hole.pending ⚠ Этот фрагмент ещё не переведён. +hole.withheld ⚠ Перевод этого фрагмента не выпущен: он требует проверки человеком. +hole.incomplete ⚠ В этом фрагменте недостаёт части текста ({dropped}). +hole.stale ⚠ Оригинал этого фрагмента изменился после перевода; фрагмент ждёт нового перевода. +hole.ghost ⚠ Часть перевода этой главы ({ghost}) не вошла в файл: книга была перенарезана после перевода. +notice.holes ⚠ Перевод не завершён: фрагментов с пропусками — {holes} из {total}. Каждый пропуск отмечен в тексте знаком ⚠. +notice.ghost ⚠ Книга была перенарезана после перевода: фрагментов перевода вне текущей нарезки — {ghost}; они в файл не вошли. diff --git a/backend/internal/bookfile/bookfile_test.go b/backend/internal/bookfile/bookfile_test.go new file mode 100644 index 00000000..bf10c199 --- /dev/null +++ b/backend/internal/bookfile/bookfile_test.go @@ -0,0 +1,344 @@ +package bookfile + +import ( + "archive/zip" + "bytes" + "encoding/xml" + "io" + "regexp" + "strings" + "testing" + "time" +) + +// bookfile_test.go pins the writer's two load-bearing properties — determinism and a container the +// engine's own reader round-trips — and the EPUB 3 shape epubcheck was run against (the stand proof). + +func sampleBook() *Book { + return &Book{ + Identifier: "urn:textmachine:book:test", + Title: "Тест & <проба>", + Language: "ru", + Modified: time.Date(2026, 8, 24, 1, 15, 0, 0, time.UTC), + Chapters: []Chapter{ + {Title: "Глава 1", Paragraphs: []string{"Первый абзац.", "Второй — с «кавычками» & <тегом>."}}, + {Title: "2", Paragraphs: []string{"Одна строка."}}, + }, + } +} + +func epubBytes(t *testing.T, b *Book) []byte { + t.Helper() + var buf bytes.Buffer + if err := WriteEPUB(&buf, b); err != nil { + t.Fatal(err) + } + return buf.Bytes() +} + +func zipEntries(t *testing.T, data []byte) (*zip.Reader, map[string]string) { + t.Helper() + zr, err := zip.NewReader(bytes.NewReader(data), int64(len(data))) + if err != nil { + t.Fatal(err) + } + out := map[string]string{} + for _, f := range zr.File { + rc, err := f.Open() + if err != nil { + t.Fatal(err) + } + b, _ := io.ReadAll(rc) + rc.Close() + out[f.Name] = string(b) + } + return zr, out +} + +func TestContainerIsOCFConformant(t *testing.T) { + data := epubBytes(t, sampleBook()) + zr, entries := zipEntries(t, data) + // The mimetype's LOCAL header carries its CRC and sizes itself: general-purpose flag bit 3 (a data + // descriptor after the entry) is clear, and the extra-field length is zero — the plain form a reader + // that identifies the file by its first bytes expects. Bytes 6–7 are the flags, 28–29 the extra length. + if flags := uint16(data[6]) | uint16(data[7])<<8; flags&0x8 != 0 { + t.Errorf("mimetype local header has the data-descriptor flag set (flags=%#x)", flags) + } + if extra := uint16(data[28]) | uint16(data[29])<<8; extra != 0 { + t.Errorf("mimetype local header carries an extra field of %d bytes", extra) + } + if string(data[30:38]) != "mimetype" || string(data[38:58]) != "application/epub+zip" { + t.Errorf("the mimetype entry is not at the fixed OCF offsets: %q", data[30:58]) + } + if zr.File[0].Name != "mimetype" || zr.File[0].Method != zip.Store { + t.Fatalf("mimetype must be the first, STORED entry; got %q method %d", zr.File[0].Name, zr.File[0].Method) + } + if entries["mimetype"] != "application/epub+zip" { + t.Fatalf("mimetype = %q", entries["mimetype"]) + } + if !strings.Contains(entries["META-INF/container.xml"], `full-path="OEBPS/content.opf"`) { + t.Fatalf("container.xml must point at the OPF:\n%s", entries["META-INF/container.xml"]) + } + for _, f := range zr.File { + if !f.Modified.IsZero() && f.Modified.Unix() != -2208988800 { // zero MS-DOS time decodes to 1980-00-00 → 1979/1980 epochs vary; a real clock reading is what is forbidden + if f.Modified.Year() > 1981 { + t.Errorf("entry %s carries a clock timestamp %v — the file would differ between builds", f.Name, f.Modified) + } + } + } +} + +type opfDoc struct { + UniqueID string `xml:"unique-identifier,attr"` + Metadata struct { + Identifier struct { + ID string `xml:"id,attr"` + Value string `xml:",chardata"` + } `xml:"identifier"` + Title string `xml:"title"` + Language string `xml:"language"` + Description string `xml:"description"` + Meta []struct { + Property string `xml:"property,attr"` + Value string `xml:",chardata"` + } `xml:"meta"` + } `xml:"metadata"` + Manifest []struct { + ID string `xml:"id,attr"` + Href string `xml:"href,attr"` + MediaType string `xml:"media-type,attr"` + Properties string `xml:"properties,attr"` + } `xml:"manifest>item"` + Spine []struct { + IDRef string `xml:"idref,attr"` + } `xml:"spine>itemref"` +} + +func parseOPF(t *testing.T, entries map[string]string) opfDoc { + t.Helper() + var doc opfDoc + if err := xml.Unmarshal([]byte(entries["OEBPS/content.opf"]), &doc); err != nil { + t.Fatalf("OPF does not parse: %v\n%s", err, entries["OEBPS/content.opf"]) + } + return doc +} + +func TestEPUBCarriesTheFiveRequiredThingsAndKeepsNavOutOfTheSpine(t *testing.T) { + b := sampleBook() + b.Description = "⚠ 1/2" + _, entries := zipEntries(t, epubBytes(t, b)) + doc := parseOPF(t, entries) + if doc.Metadata.Identifier.Value != b.Identifier || doc.Metadata.Identifier.ID != doc.UniqueID { + t.Errorf("dc:identifier %+v must be the one unique-identifier=%q names", doc.Metadata.Identifier, doc.UniqueID) + } + if doc.Metadata.Title != b.Title { + t.Errorf("dc:title = %q, want %q (escaped in the file, intact when parsed)", doc.Metadata.Title, b.Title) + } + padded := sampleBook() + padded.Title = " T " + if _, e := zipEntries(t, epubBytes(t, padded)); !strings.Contains(e["OEBPS/content.opf"], "T") { + t.Errorf("dc:title must be trimmed like the text file's title line:\n%s", e["OEBPS/content.opf"]) + } + if doc.Metadata.Language != "ru" { + t.Errorf("dc:language = %q", doc.Metadata.Language) + } + if doc.Metadata.Description != b.Description { + t.Errorf("dc:description = %q, want %q", doc.Metadata.Description, b.Description) + } + modified := "" + for _, m := range doc.Metadata.Meta { + if m.Property == "dcterms:modified" { + modified = m.Value + } + } + if modified != "2026-08-24T01:15:00Z" { + t.Errorf("dcterms:modified = %q, want the CCYY-MM-DDThh:mm:ssZ render of Book.Modified", modified) + } + navID := "" + for _, it := range doc.Manifest { + if it.Properties == "nav" { + navID = it.ID + if _, ok := entries["OEBPS/"+it.Href]; !ok { + t.Errorf("the nav item points at %q, which is not in the zip", it.Href) + } + } + } + if navID == "" { + t.Fatal("no manifest item carries properties=\"nav\" — EPUB 3 requires a navigation document") + } + if len(doc.Spine) != len(b.Chapters) { + t.Fatalf("spine has %d itemrefs, want one per chapter (%d) and nothing else", len(doc.Spine), len(b.Chapters)) + } + for _, ref := range doc.Spine { + if ref.IDRef == navID { + t.Fatal("the nav document is in the spine — the engine's reader would read it back as a chapter") + } + } + nav := entries["OEBPS/nav.xhtml"] + for _, ch := range b.Chapters { + if !strings.Contains(nav, ">"+xmlText(ch.Title)+"") { + t.Errorf("nav lacks an entry titled %q:\n%s", ch.Title, nav) + } + } +} + +// blocksOf pulls the text of every

/

of a chapter document, in order — what the engine's reader +// turns into paragraphs. +var blockRE = regexp.MustCompile(`(?s)<(h1|p)>(.*?)`) + +func blocksOf(doc string) []string { + var out []string + for _, m := range blockRE.FindAllStringSubmatch(doc, -1) { + out = append(out, m[2]) + } + return out +} + +func TestChapterDocumentsCarryExactlyBlocks(t *testing.T) { + b := sampleBook() + b.Notice = []string{"⚠ 1/2", "⚠ +3"} + _, entries := zipEntries(t, epubBytes(t, b)) + for i := range b.Chapters { + got := blocksOf(entries["OEBPS/"+chapterEntry(i)]) + want := b.Blocks(i) + if len(got) != len(want) { + t.Fatalf("chapter %d: %d blocks in the document, Blocks() says %d\n%v\n%v", i+1, len(got), len(want), got, want) + } + for j := range want { + if got[j] != xmlText(want[j]) { + t.Errorf("chapter %d block %d = %q, want %q", i+1, j, got[j], xmlText(want[j])) + } + } + } + // The notice is in the FIRST document only, ahead of its title. + if first := blocksOf(entries["OEBPS/ch1.xhtml"]); first[0] != "⚠ 1/2" || first[2] != xmlText("Глава 1") { + t.Errorf("first document must open with the notice, then the title: %v", first) + } + if second := blocksOf(entries["OEBPS/ch2.xhtml"]); second[0] != "2" { + t.Errorf("second document must open with its title, no notice: %v", second) + } +} + +func TestEPUBAndTXTAreDeterministic(t *testing.T) { + b := sampleBook() + first, second := epubBytes(t, b), epubBytes(t, b) + if !bytes.Equal(first, second) { + t.Fatal("two EPUB builds of one Book differ") + } + var t1, t2 bytes.Buffer + if err := WriteTXT(&t1, b); err != nil { + t.Fatal(err) + } + if err := WriteTXT(&t2, b); err != nil { + t.Fatal(err) + } + if !bytes.Equal(t1.Bytes(), t2.Bytes()) { + t.Fatal("two TXT builds of one Book differ") + } +} + +func TestTextIsEscapedAndControlCharactersAreDropped(t *testing.T) { + b := sampleBook() + b.Chapters[0].Paragraphs = []string{"a < b & c > d \"q\"", "bad\x00\x0bchars\x7f kept"} + if got := CleanText("bad\x00\x0bchars\x7f kept\xff"); got != "badchars\x7f kept\uFFFD" { + t.Errorf("CleanText = %q", got) + } + _, entries := zipEntries(t, epubBytes(t, b)) + doc := entries["OEBPS/ch1.xhtml"] + for _, want := range []string{"a < b & c > d "q"", "badchars\x7f kept"} { + if !strings.Contains(doc, want) { + t.Errorf("document lacks %q:\n%s", want, doc) + } + } + if strings.ContainsAny(doc, "\x00\x0b") { + t.Error("an XML-illegal control character reached the document") + } + if !strings.Contains(entries["OEBPS/content.opf"], "Тест & <проба>") { + t.Errorf("title not escaped:\n%s", entries["OEBPS/content.opf"]) + } +} + +func TestWritersRefuseAnInvalidBook(t *testing.T) { + cases := map[string]func(*Book){ + "no chapters": func(b *Book) { b.Chapters = nil }, + "empty title": func(b *Book) { b.Chapters[1].Title = " " }, + // Judged after CleanText: a title of control characters only would render as an empty

and + // an empty nav link — the invalid EPUB validate exists to refuse, not something to write. + "control-only chapter title": func(b *Book) { b.Chapters[1].Title = "\x01\x02 \x1f" }, + "control-only book title": func(b *Book) { b.Title = "\x03\x0b" }, + "zero modified": func(b *Book) { b.Modified = time.Time{} }, + "no language": func(b *Book) { b.Language = "" }, + "no identifier": func(b *Book) { b.Identifier = "" }, + "empty book title": func(b *Book) { b.Title = "" }, + } + for name, mutate := range cases { + b := sampleBook() + mutate(b) + if err := WriteEPUB(io.Discard, b); err == nil { + t.Errorf("%s: WriteEPUB must refuse", name) + } + if err := WriteTXT(io.Discard, b); err == nil { + t.Errorf("%s: WriteTXT must refuse", name) + } + } +} + +func TestTXTShape(t *testing.T) { + b := sampleBook() + b.Notice = []string{"⚠ 1/2"} + var buf bytes.Buffer + if err := WriteTXT(&buf, b); err != nil { + t.Fatal(err) + } + want := "Тест & <проба>\n\n⚠ 1/2\n\n\nГлава 1\n\nПервый абзац.\n\nВторой — с «кавычками» & <тегом>.\n\n\n2\n\nОдна строка.\n" + if buf.String() != want { + t.Fatalf("txt =\n%q\nwant\n%q", buf.String(), want) + } + for _, banned := range []string{"=== CHAPTER", "TEXT MISSING", "pending", "flagged"} { + if strings.Contains(buf.String(), banned) { + t.Errorf("operator vocabulary %q in the reader's text", banned) + } + } +} + +// TestTitlesAreCleanedTheSameInBothFormats pins format parity on the strings the assembler does NOT +// pre-clean — the book title, the chapter titles, the notice: the EPUB drops a control character through +// xmlText, and the text file must drop it the same way (acceptance finding: TXT wrote titles raw). +func TestTitlesAreCleanedTheSameInBothFormats(t *testing.T) { + b := sampleBook() + b.Title = "Заглавие\x01книги" + b.Notice = []string{"⚠\x03 1/2"} + b.Chapters[0].Title = "Гла\x02ва 1" + b.Chapters[0].Paragraphs = []string{"Абзац\x1fс контролем."} + _, entries := zipEntries(t, epubBytes(t, b)) + doc := parseOPF(t, entries) + if doc.Metadata.Title != "Заглавиекниги" { + t.Errorf("EPUB dc:title = %q", doc.Metadata.Title) + } + if blocks := blocksOf(entries["OEBPS/ch1.xhtml"]); len(blocks) < 3 || blocks[0] != "⚠ 1/2" || blocks[1] != "Глава 1" || blocks[2] != "Абзацс контролем." { + t.Errorf("EPUB blocks = %q", blocks) + } + var txt bytes.Buffer + if err := WriteTXT(&txt, b); err != nil { + t.Fatal(err) + } + want := "Заглавиекниги\n\n⚠ 1/2\n\n\nГлава 1\n\nАбзацс контролем.\n\n\n2\n\nОдна строка.\n" + if txt.String() != want { + t.Errorf("TXT =\n%q\nwant\n%q", txt.String(), want) + } + for name, out := range map[string]string{"epub opf": entries["OEBPS/content.opf"], "epub nav": entries["OEBPS/nav.xhtml"], "epub ch1": entries["OEBPS/ch1.xhtml"], "txt": txt.String()} { + if strings.ContainsAny(out, "\x01\x02\x03\x1f") { + t.Errorf("%s carries a control character the other format drops", name) + } + } +} + +func TestOneChapterBook(t *testing.T) { + b := &Book{Identifier: "urn:x", Title: "T", Language: "en", Modified: time.Unix(0, 0).UTC(), + Chapters: []Chapter{{Title: "1"}}} // a chapter with a title and no paragraphs is a legal, empty chapter + _, entries := zipEntries(t, epubBytes(t, b)) + doc := parseOPF(t, entries) + if len(doc.Spine) != 1 || len(blocksOf(entries["OEBPS/ch1.xhtml"])) != 1 { + t.Fatalf("one-chapter book: spine %d, blocks %v", len(doc.Spine), blocksOf(entries["OEBPS/ch1.xhtml"])) + } +} diff --git a/backend/internal/bookfile/container.go b/backend/internal/bookfile/container.go new file mode 100644 index 00000000..9a60418c --- /dev/null +++ b/backend/internal/bookfile/container.go @@ -0,0 +1,77 @@ +package bookfile + +import ( + "archive/zip" + "fmt" + "hash/crc32" + "io" +) + +// container.go: the OCF container — the zip layout every EPUB shares, independent of what the package +// inside says. It is the one place the layout is spelled, for the shipping writer (epub.go) and for the +// ingest test fixture (internal/chunk/chunktest) alike. + +// mimetypeBody is the fixed content of the OCF `mimetype` entry. +const mimetypeBody = "application/epub+zip" + +// Container is an OCF container under construction. Entries land in the order they are added, with +// ZERO timestamps (archive/zip leaves the MS-DOS date fields at zero and writes no extended-timestamp +// extra field when FileHeader.Modified is unset), so the bytes depend on the entries alone — never on the +// clock or on a filesystem — which is what makes a build reproducible across processes. +type Container struct { + zw *zip.Writer +} + +// NewContainer starts an OCF container on w whose package document (the OPF) will be added at +// rootfile — a path inside the zip such as "OEBPS/content.opf". It writes the two entries OCF fixes: +// `mimetype` FIRST and STORED (uncompressed, so a reader can identify the file by its first bytes), +// then META-INF/container.xml pointing at rootfile. The caller adds rootfile itself with Add. +func NewContainer(w io.Writer, rootfile string) (*Container, error) { + zw := zip.NewWriter(w) + // CreateRaw with the CRC and sizes given up front: the local header then carries them itself and + // no data descriptor follows the entry. OCF's letter is met either way, but a reader that + // identifies the file by its first bytes wants the plain form, and the 20 bytes cost nothing. + body := []byte(mimetypeBody) + mw, err := zw.CreateRaw(&zip.FileHeader{ + Name: "mimetype", Method: zip.Store, CRC32: crc32.ChecksumIEEE(body), + CompressedSize64: uint64(len(body)), UncompressedSize64: uint64(len(body)), + }) + if err != nil { + return nil, fmt.Errorf("bookfile: container mimetype: %w", err) + } + if _, err := mw.Write(body); err != nil { + return nil, fmt.Errorf("bookfile: container mimetype: %w", err) + } + c := &Container{zw: zw} + container := ` + + + + + +` + if err := c.Add("META-INF/container.xml", []byte(container)); err != nil { + return nil, err + } + return c, nil +} + +// Add writes one entry (deflated, zero timestamp) at name — a slash-separated path inside the zip. +func (c *Container) Add(name string, data []byte) error { + w, err := c.zw.CreateHeader(&zip.FileHeader{Name: name, Method: zip.Deflate}) + if err != nil { + return fmt.Errorf("bookfile: container entry %s: %w", name, err) + } + if _, err := w.Write(data); err != nil { + return fmt.Errorf("bookfile: container entry %s: %w", name, err) + } + return nil +} + +// Close finishes the zip (central directory). The container is complete only after it returns nil. +func (c *Container) Close() error { + if err := c.zw.Close(); err != nil { + return fmt.Errorf("bookfile: close container: %w", err) + } + return nil +} diff --git a/backend/internal/bookfile/epub.go b/backend/internal/bookfile/epub.go new file mode 100644 index 00000000..fa182cea --- /dev/null +++ b/backend/internal/bookfile/epub.go @@ -0,0 +1,139 @@ +package bookfile + +import ( + "fmt" + "io" + "strings" + "time" +) + +// epub.go: the EPUB 3 package. What the format REQUIRES and this writer supplies, in order: a package +// identifier (dc:identifier, referenced by unique-identifier), dc:title, dc:language, a +// dcterms:modified in the CCYY-MM-DDThh:mm:ssZ form, and a navigation document (the manifest item with +// properties="nav"). Each chapter is one XHTML content document in the spine. +// +// ⚠ The nav document is NOT in the spine, and there is no title page document either. The engine's own +// reader (internal/chunk/ingest.go) turns EVERY spine document into a chapter, an empty one included, so +// a nav or a title page in the spine would come back as a chapter of its own and the loaded↔received +// circle would not close. The book-level notice therefore rides inside the FIRST chapter document, ahead +// of its title (Book.Notice / Blocks). + +const ( + opfPath = "OEBPS/content.opf" + navEntry = "nav.xhtml" + // modifiedLayout is the ONLY form EPUB 3 admits for dcterms:modified (a UTC timestamp to the second). + modifiedLayout = "2006-01-02T15:04:05Z" +) + +// chapterEntry names chapter i's document (0-based) inside OEBPS/. The name carries the dense chapter +// number; reading order is the spine's, not the name's. +func chapterEntry(i int) string { return fmt.Sprintf("ch%d.xhtml", i+1) } + +// chapterID is chapter i's manifest id. +func chapterID(i int) string { return fmt.Sprintf("ch%d", i+1) } + +// WriteEPUB writes b as an EPUB 3 file to w. It is deterministic: the same Book yields the same bytes. +func WriteEPUB(w io.Writer, b *Book) error { + if err := b.validate(); err != nil { + return err + } + c, err := NewContainer(w, opfPath) + if err != nil { + return err + } + if err := c.Add(opfPath, []byte(packageDocument(b))); err != nil { + return err + } + if err := c.Add("OEBPS/"+navEntry, []byte(navDocument(b))); err != nil { + return err + } + for i := range b.Chapters { + if err := c.Add("OEBPS/"+chapterEntry(i), []byte(chapterDocument(b, i))); err != nil { + return err + } + } + return c.Close() +} + +// packageDocument renders the OPF. +func packageDocument(b *Book) string { + lang := xmlText(b.Language) + var sb strings.Builder + sb.WriteString(` + + + ` + xmlText(b.Identifier) + ` + ` + xmlText(strings.TrimSpace(b.Title)) + ` + ` + lang + ` + ` + b.Modified.UTC().Format(modifiedLayout) + ` +`) + if b.Description != "" { + sb.WriteString(` ` + xmlText(b.Description) + `` + "\n") + } + sb.WriteString(` + + +`) + for i := range b.Chapters { + sb.WriteString(` ` + "\n") + } + sb.WriteString(" \n \n") + for i := range b.Chapters { + sb.WriteString(` ` + "\n") + } + sb.WriteString(" \n\n") + return sb.String() +} + +// navDocument renders the EPUB 3 navigation document: a toc nav whose entries are the chapters, in spine +// order, each linking to its document by the chapter's Title. +func navDocument(b *Book) string { + lang := xmlText(b.Language) + var sb strings.Builder + sb.WriteString(` + + + ` + xmlText(strings.TrimSpace(b.Title)) + ` + + + \n\n\n") + return sb.String() +} + +// chapterDocument renders chapter i as an XHTML content document: the blocks of Blocks(i), the title as +//

and every other block as

. One block = one block-level element, which is what the engine's +// reader turns back into one paragraph. +func chapterDocument(b *Book, i int) string { + lang := xmlText(b.Language) + ch := b.Chapters[i] + var sb strings.Builder + sb.WriteString(` + + + ` + xmlText(ch.Title) + ` + + +`) + if i == 0 { + for _, p := range b.Notice { + sb.WriteString("

" + xmlText(p) + "

\n") + } + } + sb.WriteString("

" + xmlText(ch.Title) + "

\n") + for _, p := range ch.Paragraphs { + sb.WriteString("

" + xmlText(p) + "

\n") + } + sb.WriteString("\n\n") + return sb.String() +} + +// Modified renders t the way dcterms:modified is written, for a caller that wants to show the value it +// is about to put in the file. +func Modified(t time.Time) string { return t.UTC().Format(modifiedLayout) } diff --git a/backend/internal/bookfile/model.go b/backend/internal/bookfile/model.go new file mode 100644 index 00000000..8a8cbc47 --- /dev/null +++ b/backend/internal/bookfile/model.go @@ -0,0 +1,165 @@ +// Package bookfile writes a translated book as a READER's file — the artifact the engine hands out — from +// a neutral document model: EPUB 3 for a reading app (epub.go) and plain text (txt.go), both from the same +// Book. It is the engine's only writer of a book, and it is deliberately language- and pair-blind: every +// word a reader sees (a chapter title, a hole marker, the not-whole notice) arrives in the model as DATA +// decided upstream; nothing here knows what a chapter is called in any language (12-go-style-notes §0). +// +// Two properties are load-bearing and proven by tests rather than promised: +// - DETERMINISM: the same Book yields byte-identical files across processes. Nothing reads a clock or +// a filesystem timestamp; the zip carries zero timestamps; entries are written in a fixed order. +// - ROUND TRIP: the engine's own ingest (internal/chunk) reads an EPUB written here back into exactly +// the blocks Blocks() lays down, chapter by chapter — which is what closes the "loaded → received" +// circle the reader path was built on. +// +// The OCF container half (container.go) is ALSO what the ingest test fixture builds on +// (internal/chunk/chunktest): one zip/OCF builder, two callers, so the fixture the reader is pinned +// against and the file the writer ships cannot drift apart in the container layer. +package bookfile + +import ( + "errors" + "fmt" + "strings" + "time" +) + +// Formats are the file formats this package writes, in the order a build produces them. A format name +// is also the file extension. +var Formats = []string{"epub", "txt"} + +// KnownFormat reports whether name is one of Formats. +func KnownFormat(name string) bool { + for _, f := range Formats { + if f == name { + return true + } + } + return false +} + +// Book is the document model both writers render. +type Book struct { + // Identifier is the package identifier (dc:identifier, the OPF's unique-identifier). It is derived + // from the book's id by the caller; never minted from a clock or a random source. + Identifier string + // Title is dc:title, spelled exactly as the book config spells it. It may be the source-language + // display name — the writer has no translation of it and invents none. + Title string + // Language is dc:language and the documents' xml:lang: the TARGET language tag, from the book's data. + Language string + // Modified is dcterms:modified. It is a value from the run's data, never the wall clock, so that two + // builds of one store are byte-identical (a zero value is rejected: a missing fact must be decided + // upstream, not silently rendered as year one). + Modified time.Time + // Description is dc:description; "" omits the element. The assembler puts the not-whole notice here + // so a library shelf that shows descriptions shows it too. + Description string + // Notice are the book-level paragraphs a reader sees on OPENING the book — before the first chapter's + // title, inside the first spine document (a separate title document would be read by ingest as a + // chapter of its own), and at the head of the text file. Empty for a book with nothing to declare. + Notice []string + // Chapters in reading order; at least one (an EPUB with an empty spine is not an EPUB). + Chapters []Chapter +} + +// Chapter is one spine document. +type Chapter struct { + // Title is the chapter's heading: the

of its document, the text of its nav entry, its heading + // line in the text file. Required — an empty nav link is an invalid EPUB — and decided upstream + // (the engine's deterministic heading, or the bare chapter number when there is none). + Title string + // Paragraphs are the chapter's text blocks in order, one

each. A hole marker is an ordinary + // paragraph here: the model carries what the assembler decided to say, and nothing about why. + Paragraphs []string +} + +// Blocks returns chapter i's text blocks in the order its document carries them: the book Notice (first +// chapter only), the Title, then the Paragraphs. It is the ONE definition the EPUB writer emits from and +// the round-trip proof compares against — what ingest reads back as paragraphs is exactly this list. +func (b *Book) Blocks(i int) []string { + ch := b.Chapters[i] + out := make([]string, 0, len(b.Notice)+1+len(ch.Paragraphs)) + if i == 0 { + out = append(out, b.Notice...) + } + out = append(out, ch.Title) + out = append(out, ch.Paragraphs...) + return out +} + +// validate refuses a Book the writers could only render into an invalid file. Every failure is the +// caller's to fix upstream; the writers never paper over one. Titles are judged AFTER CleanText — the +// form they reach the file in: a title of nothing but control characters renders as an empty dc:title +// or an empty nav link, and both are invalid EPUB. +func (b *Book) validate() error { + switch { + case b.Identifier == "": + return errors.New("bookfile: the book has no identifier") + case strings.TrimSpace(CleanText(b.Title)) == "": + return errors.New("bookfile: the book has no title") + case strings.TrimSpace(b.Language) == "": + return errors.New("bookfile: the book has no language") + case b.Modified.IsZero(): + return errors.New("bookfile: the book has no modification time (it must come from the run's data, not be left unset)") + case len(b.Chapters) == 0: + return errors.New("bookfile: the book has no chapters (an empty spine is not an EPUB)") + } + for i, ch := range b.Chapters { + if strings.TrimSpace(CleanText(ch.Title)) == "" { + return fmt.Errorf("bookfile: chapter %d has no title (an empty nav entry is an invalid EPUB)", i+1) + } + } + return nil +} + +// CleanText makes s carriable by BOTH formats: invalid UTF-8 replaced, and the code points XML 1.0 +// forbids (C0 controls other than tab/newline/return, U+FFFE/U+FFFF) dropped — a control character that +// reached the export text would make an XHTML chapter unparseable to every reader, and is junk in a text +// file too. The assembler applies it to every paragraph once, so the EPUB and the text file say the same. +func CleanText(s string) string { + s = strings.ToValidUTF8(s, "\uFFFD") + if !strings.ContainsFunc(s, isXMLIllegal) { + return s + } + var b strings.Builder + b.Grow(len(s)) + for _, r := range s { + if !isXMLIllegal(r) { + b.WriteRune(r) + } + } + return b.String() +} + +func isXMLIllegal(r rune) bool { + switch { + case r == '\t' || r == '\n' || r == '\r': + return false + case r < 0x20, r == 0xFFFE, r == 0xFFFF: + return true + } + return false +} + +// xmlText renders s as XML character data: CleanText, then the three markup characters escaped, plus the +// quote so the same function serves attribute values. +func xmlText(s string) string { + s = CleanText(s) + var b strings.Builder + b.Grow(len(s)) + for _, r := range s { + switch r { + case '&': + b.WriteString("&") + case '<': + b.WriteString("<") + case '>': + b.WriteString(">") + case '"': + b.WriteString(""") + default: + b.WriteRune(r) + } + } + return b.String() +} diff --git a/backend/internal/bookfile/txt.go b/backend/internal/bookfile/txt.go new file mode 100644 index 00000000..d848a5d1 --- /dev/null +++ b/backend/internal/bookfile/txt.go @@ -0,0 +1,37 @@ +package bookfile + +import ( + "io" + "strings" +) + +// txt.go: the plain-text book. No audit banners, no flags in brackets, no operator vocabulary — the file +// is prose plus what the model carries: the title first, the notice (when there is one), then every +// chapter as its title line followed by its paragraphs. Blocks are separated by ONE blank line, chapters +// by TWO, so a chapter boundary reads differently from a paragraph boundary even in a viewer that shows +// nothing but text. The file ends with a newline. +// +// The same block list the EPUB renders (Blocks) is what lands here, so the two formats say the same thing — +// cleaned the same way too: every string the EPUB passes through xmlText goes through CleanText here, so +// a control character in a title or a notice cannot reach the text reader when the EPUB reader is spared it. + +// WriteTXT writes b as plain UTF-8 text to w. Deterministic. +func WriteTXT(w io.Writer, b *Book) error { + if err := b.validate(); err != nil { + return err + } + var sb strings.Builder + sb.WriteString(strings.TrimSpace(CleanText(b.Title))) + for _, p := range b.Notice { + sb.WriteString("\n\n" + CleanText(p)) + } + for _, ch := range b.Chapters { + sb.WriteString("\n\n\n" + CleanText(ch.Title)) + for _, p := range ch.Paragraphs { + sb.WriteString("\n\n" + CleanText(p)) + } + } + sb.WriteString("\n") + _, err := io.WriteString(w, sb.String()) + return err +} diff --git a/backend/internal/chunk/chunktest/epub.go b/backend/internal/chunk/chunktest/epub.go index 749742ee..857d6252 100644 --- a/backend/internal/chunk/chunktest/epub.go +++ b/backend/internal/chunk/chunktest/epub.go @@ -2,14 +2,22 @@ // package because two packages need the SAME epub shape: the ingest unit tests and the runner's // end-to-end tests. Duplicating a zip/OPF builder in both would let the two drift apart, and the // whole point of the e2e fixture is that it is the file the unit tests already pin. +// +// The CONTAINER layer (mimetype first and stored, container.xml, the entry writes) is not built here +// any more: it is internal/bookfile's, the same code the engine's book writer ships an EPUB through, so +// the fixture the reader is pinned against and the file the writer produces cannot drift apart at the +// zip level. What stays here is deliberately the TEST half — the bare OPF stub, the decoy /<style> +// in every chapter head that ingest_test proves are dropped, the MType/EntryName knobs for malformed +// manifests — none of which may ever reach a reader. package chunktest import ( - "archive/zip" "os" "path/filepath" "strings" "testing" + + "textmachine/backend/internal/bookfile" ) type Chapter struct { @@ -49,30 +57,18 @@ func BuildEPUBAt(t *testing.T, path string, chapters []Chapter, spineIDs []strin t.Fatal(err) } defer f.Close() - zw := zip.NewWriter(f) - add := func(name, content string) { - w, err := zw.Create(name) - if err != nil { - t.Fatal(err) - } - if _, err := w.Write([]byte(content)); err != nil { - t.Fatal(err) - } - } - - // OCF requires "mimetype" first in the archive and STORED, not deflated. Nothing in ingest - // reads it today; the fixture matches the spec so it stays a real epub as the reader grows. - mw, err := zw.CreateHeader(&zip.FileHeader{Name: "mimetype", Method: zip.Store}) + // OCF requires "mimetype" first in the archive and STORED, not deflated, and container.xml pointing + // at the OPF. The production container writes both (bookfile.NewContainer); the fixture rides it so + // it stays the real epub shape the reader will meet, by construction rather than by copy. + c, err := bookfile.NewContainer(f, "OEBPS/content.opf") if err != nil { t.Fatal(err) } - if _, err := mw.Write([]byte("application/epub+zip")); err != nil { - t.Fatal(err) + add := func(name, content string) { + if err := c.Add(name, []byte(content)); err != nil { + t.Fatal(err) + } } - add("META-INF/container.xml", `<?xml version="1.0"?> -<container version="1.0" xmlns="urn:oasis:names:tc:opendocument:xmlns:container"> - <rootfiles><rootfile full-path="OEBPS/content.opf" media-type="application/oebps-package+xml"/></rootfiles> -</container>`) var manifest, spine strings.Builder for _, c := range chapters { @@ -106,7 +102,7 @@ func BuildEPUBAt(t *testing.T, path string, chapters []Chapter, spineIDs []strin add("OEBPS/"+entry, c.Body) // non-xhtml asset (e.g. image bytes stand-in) } } - if err := zw.Close(); err != nil { + if err := c.Close(); err != nil { t.Fatal(err) } } diff --git a/backend/internal/lang/reader.go b/backend/internal/lang/reader.go new file mode 100644 index 00000000..8121d349 --- /dev/null +++ b/backend/internal/lang/reader.go @@ -0,0 +1,150 @@ +package lang + +import ( + "fmt" + "os" + "path/filepath" + "strings" +) + +// reader.go: the READER-FACING words a target language supplies for what a book FILE has to say about +// itself — the mark a reader meets at a hole in the text and the notice that the book is not whole +// (the engine's book writer, internal/pipeline bookbuild.go). They are RENDER data (what the file says), +// not TRANSLATION data (what the model sees), and the distinction is load-bearing: +// +// - They live at <langpack_root>/<TARGET lang>/reader.txt — keyed by the language the READER reads, +// not by the pair, because «this fragment is not translated» is the same sentence for a zh→ru and +// a ja→ru book. +// - They are read by THIS loader and by nothing else. Load (langpack.go) reads only the source and +// the pair directories and folds THOSE bytes into Pack.Version(), the value the run snapshot carries; +// a target directory is outside its reach by construction, so adding or editing reader.txt moves no +// snapshot and re-bills no book. That is the property the file was placed here for: a rendering fact +// must never look like a translation fact to the money. +// +// A target language that ships no file gets DefaultReaderWords: a non-verbal form — a symbol and the +// numbers the engine knows — so a book with no language data still cannot look whole when it is not, +// and no Go source needs to know how to say «chapter» or «not translated» in any language. + +// ReaderWords are the templates the book writer fills. Each is a reader-facing sentence or mark; the +// placeholders in braces are the only things the engine substitutes. A template may omit a placeholder +// it has no use for. +type ReaderWords struct { + // HolePending marks a unit the run has not reached: {chapter} {unit}. + HolePending string + // HoleWithheld marks a unit whose text was deliberately not shipped (a substantive flag upstream): + // {chapter} {unit}. + HoleWithheld string + // HoleIncomplete marks a unit whose text IS present but lost a piece (a member chunk the editor + // left out): {chapter} {unit} {dropped}. + HoleIncomplete string + // HoleStale marks a unit whose translation is of a source that has since changed (the engine re-buys + // it on the next run; the old text is not shown): {chapter} {unit}. + HoleStale string + // HoleGhost marks a chapter that has translated text the current cut of the book cannot place: + // {chapter} {ghost}. + HoleGhost string + // NoticeHoles is the book-level notice for a book with holes: {holes} {total}. + NoticeHoles string + // NoticeGhost is the book-level notice for text outside the current cut: {ghost}. + NoticeGhost string +} + +// readerKeys maps reader.txt keys to the field they fill. Every key is REQUIRED when the file exists: +// a file that names four of six marks leaves the other two to the non-verbal form in the same book, +// which is a mixed register no author intended. +var readerKeys = map[string]func(*ReaderWords, string){ + "hole.pending": func(w *ReaderWords, v string) { w.HolePending = v }, + "hole.withheld": func(w *ReaderWords, v string) { w.HoleWithheld = v }, + "hole.incomplete": func(w *ReaderWords, v string) { w.HoleIncomplete = v }, + "hole.stale": func(w *ReaderWords, v string) { w.HoleStale = v }, + "hole.ghost": func(w *ReaderWords, v string) { w.HoleGhost = v }, + "notice.holes": func(w *ReaderWords, v string) { w.NoticeHoles = v }, + "notice.ghost": func(w *ReaderWords, v string) { w.NoticeGhost = v }, +} + +// readerFile is the file name under the target-language directory. +const readerFile = "reader.txt" + +// DefaultReaderWords is the NON-VERBAL form used when the target language ships no reader.txt: a warning +// sign and the numbers the engine knows. It contains no word of any language on purpose — a word here +// would be the engine deciding how a language says something (12-go-style-notes §0). +func DefaultReaderWords() ReaderWords { + return ReaderWords{ + HolePending: "⚠ {chapter}.{unit}", + HoleWithheld: "⚠ {chapter}.{unit}", + HoleIncomplete: "⚠ {chapter}.{unit} −{dropped}", + HoleStale: "⚠ {chapter}.{unit}", + HoleGhost: "⚠ {chapter} +{ghost}", + NoticeHoles: "⚠ {holes}/{total}", + NoticeGhost: "⚠ +{ghost}", + } +} + +// LoadReaderWords reads <root>/<targetLang>/reader.txt. An empty root (a book with no langpack) or an +// absent file yields DefaultReaderWords and present=false; a present file must be whole and well-formed +// (every key once, no unknown key, no empty value) or the load fails loud — a half-written vocabulary is +// a corrupt data file, not an intended mixture. +func LoadReaderWords(root, targetLang string) (words ReaderWords, present bool, err error) { + words = DefaultReaderWords() + if root == "" || strings.TrimSpace(targetLang) == "" { + return words, false, nil + } + b, err := os.ReadFile(filepath.Join(root, targetLang, readerFile)) + if err != nil { + if os.IsNotExist(err) { + return words, false, nil + } + return words, false, fmt.Errorf("reader words %s/%s: %w", targetLang, readerFile, err) + } + parsed, err := parseReaderWords(b) + if err != nil { + return words, false, fmt.Errorf("reader words %s/%s: %w", targetLang, readerFile, err) + } + return parsed, true, nil +} + +// parseReaderWords reads the `key<TAB>value` lines of reader.txt (comments and blank lines skipped). +func parseReaderWords(b []byte) (ReaderWords, error) { + var w ReaderWords + seen := map[string]bool{} + for i, raw := range strings.Split(string(b), "\n") { + t := strings.TrimSpace(strings.TrimRight(raw, "\r")) + if t == "" || strings.HasPrefix(t, "#") { + continue + } + f := strings.SplitN(t, "\t", 2) + if len(f) != 2 { + return w, fmt.Errorf("line %d: want `key<TAB>value`, got %q", i+1, t) + } + key, val := strings.TrimSpace(f[0]), strings.TrimSpace(f[1]) + set, ok := readerKeys[key] + if !ok { + return w, fmt.Errorf("line %d: unknown key %q", i+1, key) + } + if seen[key] { + return w, fmt.Errorf("line %d: key %q given twice", i+1, key) + } + if val == "" { + return w, fmt.Errorf("line %d: key %q has an empty value", i+1, key) + } + seen[key] = true + set(&w, val) + } + for key := range readerKeys { + if !seen[key] { + return w, fmt.Errorf("missing key %q (every mark must be given; omit the file to use the non-verbal form)", key) + } + } + return w, nil +} + +// FillReaderTemplate substitutes {name} placeholders in tmpl from pairs (name, value, name, value…). +// A placeholder the template does not use is simply not there; a name the caller did not pass stays as +// written, which is visible in the output rather than silently blank. +func FillReaderTemplate(tmpl string, pairs ...string) string { + oldnew := make([]string, 0, len(pairs)) + for i := 0; i+1 < len(pairs); i += 2 { + oldnew = append(oldnew, "{"+pairs[i]+"}", pairs[i+1]) + } + return strings.NewReplacer(oldnew...).Replace(tmpl) +} diff --git a/backend/internal/lang/reader_test.go b/backend/internal/lang/reader_test.go new file mode 100644 index 00000000..61d22135 --- /dev/null +++ b/backend/internal/lang/reader_test.go @@ -0,0 +1,122 @@ +package lang + +import ( + "os" + "path/filepath" + "strings" + "testing" + "unicode" +) + +// reader_test.go: the reader words are DATA of the target language, read outside the pack loader and +// outside Pack.Version() — a wording edit must never re-bill a book. + +const sampleReader = `# ru +hole.pending ⚠ Не переведено ({chapter}.{unit}). +hole.withheld ⚠ Не выпущено. +hole.incomplete ⚠ Недостаёт {dropped}. +hole.stale ⚠ Устарело ({chapter}.{unit}). +hole.ghost ⚠ Глава {chapter}: вне нарезки {ghost}. +notice.holes ⚠ Пропусков {holes} из {total}. +notice.ghost ⚠ Вне нарезки {ghost}. +` + +func writeReader(t *testing.T, root, tgt, body string) { + t.Helper() + if err := os.MkdirAll(filepath.Join(root, tgt), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, tgt, readerFile), []byte(body), 0o644); err != nil { + t.Fatal(err) + } +} + +func TestReaderWordsLoadAndFill(t *testing.T) { + root := t.TempDir() + writeReader(t, root, "ru", sampleReader) + w, present, err := LoadReaderWords(root, "ru") + if err != nil || !present { + t.Fatalf("present=%v err=%v", present, err) + } + if got := FillReaderTemplate(w.HolePending, "chapter", "3", "unit", "0"); got != "⚠ Не переведено (3.0)." { + t.Errorf("filled = %q", got) + } + if got := FillReaderTemplate(w.NoticeHoles, "holes", "2", "total", "37"); got != "⚠ Пропусков 2 из 37." { + t.Errorf("filled = %q", got) + } + // A placeholder the caller did not pass stays visible rather than vanishing. + if got := FillReaderTemplate("{x} {y}", "x", "1"); got != "1 {y}" { + t.Errorf("unfilled placeholder must stay: %q", got) + } +} + +func TestReaderWordsAbsentIsTheNonVerbalForm(t *testing.T) { + for _, root := range []string{"", t.TempDir()} { + w, present, err := LoadReaderWords(root, "ru") + if err != nil || present { + t.Fatalf("root %q: present=%v err=%v", root, present, err) + } + if w != DefaultReaderWords() { + t.Fatalf("root %q: absent file must yield the defaults", root) + } + } + // The non-verbal form contains no letter of any language: a symbol and the numbers only. + d := DefaultReaderWords() + for name, tmpl := range map[string]string{"pending": d.HolePending, "withheld": d.HoleWithheld, "incomplete": d.HoleIncomplete, + "stale": d.HoleStale, "ghost": d.HoleGhost, "notice": d.NoticeHoles, "notice-ghost": d.NoticeGhost} { + bare := tmpl + for _, ph := range []string{"{chapter}", "{unit}", "{dropped}", "{ghost}", "{holes}", "{total}"} { + bare = strings.ReplaceAll(bare, ph, "") + } + for _, r := range bare { + if unicode.IsLetter(r) { + t.Errorf("default %s template %q carries a letter — that is a word the engine decided", name, tmpl) + break + } + } + } +} + +func TestReaderWordsFailLoud(t *testing.T) { + cases := map[string]string{ + "unknown key": sampleReader + "hole.other\tx\n", + "missing key": strings.Replace(sampleReader, "notice.ghost\t⚠ Вне нарезки {ghost}.\n", "", 1), + "duplicate key": sampleReader + "hole.pending\tагаин\n", + "empty value": strings.Replace(sampleReader, "⚠ Не выпущено.", "", 1), + "no tab": strings.Replace(sampleReader, "hole.withheld\t", "hole.withheld ", 1), + } + for name, body := range cases { + root := t.TempDir() + writeReader(t, root, "ru", body) + if _, _, err := LoadReaderWords(root, "ru"); err == nil { + t.Errorf("%s: must fail loud", name) + } + } +} + +// TestReaderWordsDoNotMoveThePackVersion is the money pin: reader.txt beside the pack changes NOTHING +// the snapshot folds. (The stand proof is `current_snapshot` of the minirun export before and after.) +func TestReaderWordsDoNotMoveThePackVersion(t *testing.T) { + root := t.TempDir() + writeSyntheticPack(t, root, "zh", "ru") + before, err := Load(root, "zh", "ru") + if err != nil { + t.Fatal(err) + } + writeReader(t, root, "ru", sampleReader) + after, err := Load(root, "zh", "ru") + if err != nil { + t.Fatal(err) + } + if before.Version() != after.Version() { + t.Fatalf("adding <root>/ru/reader.txt moved Pack.Version() %s → %s — a wording edit would re-bill every book of the pair", before.Version(), after.Version()) + } + // And the real shipped file loads. + w, present, err := LoadReaderWords(filepath.Join("..", "..", "configs", "langpacks"), "ru") + if err != nil || !present { + t.Fatalf("configs/langpacks/ru/reader.txt: present=%v err=%v", present, err) + } + if !strings.Contains(w.HoleIncomplete, "{dropped}") || !strings.Contains(w.NoticeHoles, "{total}") { + t.Errorf("shipped ru words lost a placeholder: %+v", w) + } +} diff --git a/backend/internal/pipeline/bookbuild.go b/backend/internal/pipeline/bookbuild.go new file mode 100644 index 00000000..6bd43b3a --- /dev/null +++ b/backend/internal/pipeline/bookbuild.go @@ -0,0 +1,598 @@ +package pipeline + +import ( + "bytes" + "errors" + "fmt" + "io/fs" + "os" + "path/filepath" + "strconv" + "strings" + "time" + + "textmachine/backend/internal/bookfile" + "textmachine/backend/internal/lang" +) + +// bookbuild.go: the translated book as a FILE — the `tmctl build` half of backlog row 236. It turns the +// export projection (export.go) into the reader's copy (internal/bookfile: EPUB 3 and plain text), and +// its whole job beyond that is honesty: a copy of an unfinished book must never look finished. +// +// SIX states the file may not lie about. Three are BookExport's counters (pending, ghost rows, config +// drift); two are per unit and read through UnitHole (withheld, incomplete — holes.go); the sixth is the +// writer's own, read from the store: STALE — a unit whose final row was made for a source that is no +// longer the book's (staleUnits). The rule: +// +// - pending · withheld · incomplete · stale · ghost are HOLES. By default a book with any hole is +// REFUSED (RefusalBookIncomplete, listing every hole) — a reader-facing copy is fail-closed +// (D29.1(б)). With --partial the same book is written with a NOTICE a reader meets on opening it (the +// first blocks of the first spine document, and dc:description) and a MARK at the place of every +// hole: instead of the text for a pending/withheld/stale unit, ahead of the text for an incomplete +// unit, at the end of a chapter for ghost rows (their place inside the chapter is not known). Both +// behaviours exist because the contract says an unfinished book may be exported and leaves what such +// a file contains to the engine (14 §createExport); which is the default is this file's decision. +// - ghost rows are a hole and not a footnote: when the cut coarsened since the run, the leader row the +// export joins carries the text of a SHORTER unit than the one it now stands for, and the rest of that +// source's translation sits in the ghost rows — and reaches no file. The acceptance stand book shows +// it exactly: 0 pending, 20 ghosts, a «complete» export that is not. +// - stale is a hole because the file would otherwise carry the translation of text that is no longer +// in the book, byte-identical to the previous build, under `complete: true` — reached by an ordinary +// sequence (build; fix a typo in the source; build). The engine already holds the disproving fact and +// acts on it at the next run (the resume fast-path re-buys the unit); the writer reads the same fact. +// Under config drift the fact is unreadable (see staleUnits) and the report says UNKNOWN, never none. +// - config drift is NOT a hole: the text in the file is byte-for-byte the text the run shipped (the +// drift is between the current config and that run — export.go). It is reported (BuildReport, +// the log), never written into the file: «config drift» is operator vocabulary a reader cannot use. +// +// WORDS. Every word a reader sees at a hole or in the notice is DATA of the TARGET language +// (lang.LoadReaderWords: <langpack_root>/<target>/reader.txt, outside the pack's version — a rendering +// fact must never move a snapshot). A book with no language data gets the non-verbal form: a sign and +// the numbers. No Go source here knows how any language says «chapter» or «not translated». +// +// HEADINGS. A chapter's title is the deterministic heading the engine ALREADY glued onto its first +// unit's export text (ChunkExport.Heading == chunk.ApplyHeading's literal): the writer strips that +// known prefix and gives the same literal to the chapter's <h1> and its nav entry. When the heading is "" +// — a legal state (D39.100 п.1) — the chapter is titled with its bare number, the one label that decides +// nothing (which label a reader should see there is the open K-3 question; it is not settled here). A +// title the MODEL wrote inside the prose is prose and stays prose (backlog row 160 owns that seam). +// +// PARAGRAPHS. A paragraph is a non-blank line of the export text (CR, LF or CRLF all end a line, as on +// ingest). The editor's output separates paragraphs with a blank line in one place and a bare newline in +// the next with no meaning behind the difference (measured on the minirun export: unit 1/0 has 14 of one +// and 7 of the other, unit 2/0 has 56 and 0), so a rule that tells them apart invents a distinction the +// text does not carry. The line is the unit of a paragraph in the web-novel source, and it is what +// ingest reads back (the round-trip proof). A unit whose text yields NO paragraph — control characters +// only — shipped nothing a reader can see and is a hole of the withheld kind, whatever its row says. +// +// FILES. Everything a build writes beside the database is from ONE build: every requested format is +// staged first and committed together, and a format that was NOT requested is removed, so `book_files` +// never points at an older copy next to a newer one — two files of different vintage beside one +// database is the «silently complete» outcome again, reached through a legitimate command sequence +// (build, translate more, build --format epub). A failure to PREPARE a file leaves the previous set +// whole; a failure between two commits (a rename in a directory that was just written to — rare, but +// not impossible) is RefusalWriteIncomplete naming what landed, and a rebuild converges. A copy is still +// a snapshot of the store at build time: a consumer that hands a file to a reader — the platform's export +// door — builds first and serves what it built; it does not serve whatever is lying beside the database. +// +// --out names a NEW file, and it is never replaced: an explicit path can be anything the caller typed — +// the project database, the source, the other format's copy — and the writer cannot tell a previous copy +// of the book from any of those, so it refuses to write over whatever exists there. The engine's own +// place beside the database IS replaced, because there the writer knows what it is replacing. + +// buildVersion versions the SHAPE of the build report. +const buildVersion = "tm-build-v1" + +// BuildOptions are the caller's choices for one build. +type BuildOptions struct { + // Formats to write, each one of bookfile.Formats; empty means all of them. + Formats []string + // Out is an explicit path for the ONE requested format; "" writes beside the project database. The + // path must not exist yet (see the file comment) and its directory must. + Out string + // Partial writes a book with holes, marked, instead of refusing it. + Partial bool +} + +// BuildReport is what `tmctl build` prints: where the files landed and what the book is. +type BuildReport struct { + Version string `json:"build_version"` + BookID string `json:"book_id"` + // Files maps each written format to its absolute path. + Files map[string]string `json:"files"` + // The six states, so a caller does not have to open the file to learn what it wrote. + TotalUnits int `json:"total_units"` + PendingUnits int `json:"pending_units"` + WithheldUnits int `json:"withheld_units"` + IncompleteUnits int `json:"incomplete_units"` + StaleUnits int `json:"stale_units"` + // StaleUnknown is true when the stale check could not be made for at least one shipped unit: under + // config drift (a moved config moves every rendered hash, so a source edit cannot be told from it), + // or for a row with no content hash (written before the engine kept one). Reported, never folded + // into «none». + StaleUnknown bool `json:"stale_unknown"` + GhostRows int `json:"ghost_rows"` + ConfigDrift bool `json:"config_drift"` + // Complete is true when the file has no hole of any kind — pending, withheld, incomplete, stale, + // ghost. A file written under --partial with Complete=false carries the notice and the marks. + Complete bool `json:"complete"` + // TextModified is the file's dcterms:modified (BookExport.TextModified, or the epoch for a book that + // shipped nothing), shown so the determinism of the stamp is visible without unzipping the file. + TextModified string `json:"text_modified"` +} + +// hole is one place the file cannot be whole, in reading order. +type hole struct { + Kind HoleKind + Chapter int + Unit int + Reason string // the unit's flag reason (withheld) or the dropped member's (incomplete) + Dropped int // incomplete: how many members are missing +} + +// bookIdentifierPrefix is the URN namespace of dc:identifier: the book id under it is deterministic — +// the same book yields the same identifier in every build, which is what a reader's library keys on. +const bookIdentifierPrefix = "urn:textmachine:book:" + +// bookFilePath is where format's copy of the book lives: beside the project DB, like every sidecar. +func (r *Runner) bookFilePath(format string) string { return r.Book.ProjectDB + ".book." + format } + +// bookFilePaths is the complete map StatusArtifacts publishes — every format, whether written or not. +func (r *Runner) bookFilePaths() map[string]string { + out := make(map[string]string, len(bookfile.Formats)) + for _, f := range bookfile.Formats { + out[f] = absPath(r.bookFilePath(f)) + } + return out +} + +// BuildBook writes the reader's copy of the book (see the file comment). It is a $0 read of the store +// plus a file write: no job, no reservation, no provider. Safe whenever `export` is. +// +// It writes through the atomic staging of artifact.go: a reader — the platform's export door — sees +// either the previous copy or the new one, whole. An existing copy beside the database is REPLACED: the +// file is a deterministic function of the store and re-made at will, unlike a backup, which refuses to +// overwrite because it is a restore point nothing can re-make. +func (r *Runner) BuildBook(opts BuildOptions) (*BuildReport, error) { + formats := opts.Formats + if len(formats) == 0 { + formats = bookfile.Formats + } + for _, f := range formats { + if !bookfile.KnownFormat(f) { + return nil, fmt.Errorf("pipeline: build: unknown format %q (want one of %s)", f, strings.Join(bookfile.Formats, ", ")) + } + } + if opts.Out != "" { + if len(formats) != 1 { + return nil, fmt.Errorf("pipeline: build: --out names ONE file, but %d formats were asked for (%s) — pass --format with a single format", len(formats), strings.Join(formats, ",")) + } + if err := r.checkOutPath(opts.Out); err != nil { + return nil, err + } + } + exp, err := r.Export(false) + if err != nil { + return nil, err + } + stale, staleUnknown := r.staleUnits(exp) + return r.buildFromExport(exp, stale, staleUnknown, formats, opts) +} + +// checkOutPath is the --out rule of the file comment: a new file, in a directory that exists. Refused +// with the config class — the operator's argument is the thing to fix, and a plain error would put the +// typo on exit 1 among the crashes. +func (r *Runner) checkOutPath(out string) error { + if _, err := os.Lstat(out); err == nil { + return refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: --out %s already exists and an explicit path is never replaced (it could be anything the caller typed — the project database included); name a new file, or omit --out for the engine's own place beside the database, which is replaced", r.Book.BookID, out)) + } else if !errors.Is(err, fs.ErrNotExist) { + return refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: --out %s: %w", r.Book.BookID, out, err)) + } + if fi, err := os.Stat(filepath.Dir(out)); err != nil || !fi.IsDir() { + return refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: --out %s: its directory does not exist", r.Book.BookID, out)) + } + return nil +} + +// staleUnits finds the shipped units whose FINAL row was made for a source that is no longer the book's: +// the row's content hash — the signature of the rendered messages, source text included, which the +// resume fast-path compares before serving a row for $0 (stagerun.go) — differs from the hash the run +// would render now (repin.go renderedContentHashes, the same reproduction the money projection uses). +// +// It answers only when the config has NOT drifted: a prompt bump moves every rendered hash, and a +// source edit cannot be told apart from it — under drift the answer is UNKNOWN, reported as such. A unit +// whose hash cannot be reproduced (an old row with no content hash, a template the runner has not +// loaded, a member draft missing) is unknown too, never «not stale». It never fails the build: a check +// that cannot run says so in the report and the log. +func (r *Runner) staleUnits(exp *BookExport) (stale map[UnitRef]bool, unknown bool) { + if exp.ConfigDrift { + return nil, true + } + shipped := 0 + for _, ce := range exp.Chunks { + if ce.Disposition != exportPending { + shipped++ + } + } + if shipped == 0 { + return nil, false + } + cannot := func(why string, err error) (map[UnitRef]bool, bool) { + r.Log.Warn("build: the stale check could not run; whether the source moved under the shipped rows is UNKNOWN (reported as unknown, not as none)", + "book", r.Book.BookID, "step", why, "err", err) + return nil, true + } + statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID) + if err != nil { + return cannot("read chunk_status", err) + } + if err := r.projectStoredMemory(); err != nil { + return cannot("materialize the stored bank", err) + } + full, err := r.bookChunks() + if err != nil { + return cannot("re-chunk the source", err) + } + hashes := r.cachedRenderedContentHashes(full, precomputeSticky(full, r.baseMemory, r.Pipeline.Context.GlossaryTokenBudget)) + finalRow := r.finalStageRows(statuses) + stale = map[UnitRef]bool{} + for _, ce := range exp.Chunks { + if ce.Disposition == exportPending { + continue + } + key := chunkKey{ce.Chapter, ce.ChunkIdx} + cs, ok := finalRow[key] + h, hok := hashes[key][cs.Stage] + if !ok || cs.ContentHash == "" || !hok { + unknown = true + continue + } + if h != cs.ContentHash { + stale[UnitRef{Chapter: ce.Chapter, ChunkIdx: ce.ChunkIdx}] = true + } + } + if unknown { + r.Log.Warn("build: the stale check could not be made for every shipped unit (a row without a content hash, or a position the run cannot re-render); those units are reported as unknown", "book", r.Book.BookID) + } + return stale, unknown +} + +// buildFromExport is BuildBook after the projection: the decision (refuse or write), the assembly and +// the writes. Split from the export read so the states the harness cannot reach through a run — a book +// of no units — are testable on a synthetic projection. +func (r *Runner) buildFromExport(exp *BookExport, stale map[UnitRef]bool, staleUnknown bool, formats []string, opts BuildOptions) (*BuildReport, error) { + // The two metadata the file cannot do without and the config does not require. Refused with the + // config class rather than failed: book.yaml is the thing to fix, and a plain error would put the + // operator's typo on exit 1 among the crashes. + if strings.TrimSpace(r.Book.Title) == "" { + return nil, refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: book.yaml has no `title` — the reader's copy carries the title as dc:title and the writer invents none", r.Book.BookID)) + } + if !looksLikeLanguageTag(r.Book.TargetLang) { + return nil, refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: book.yaml target_lang %q is not a language tag (dc:language wants BCP 47: `ru`, `zh`, `pt-BR`)", r.Book.BookID, r.Book.TargetLang)) + } + if exp.TotalUnits == 0 { + // A spine with no document is not an EPUB, and a text file of nothing is not a book. The class is + // the one the ingest path raises for the same fact: the source was read and cut and there is no + // book in it (refusal.go sourceHasNoContent). + return nil, sourceHasNoContent(fmt.Errorf("pipeline: build %s: the book has no output units — nothing to write", r.Book.BookID)) + } + words, present, err := lang.LoadReaderWords(r.Book.LangpackRoot, r.Book.TargetLang) + if err != nil { + // A data file the operator ships and this engine cannot read: the operator's file is the thing to + // fix, nothing about the book is wrong — the config class. + return nil, refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: %w", r.Book.BookID, err)) + } + book, holes, err := assembleBook(exp, stale, r.Book.Title, r.Book.TargetLang, words) + if err != nil { + return nil, err + } + complete := len(holes) == 0 && exp.GhostRows == 0 + if !complete && !opts.Partial { + return nil, refuse(RefusalBookIncomplete, describeHoles(r.Book.BookID, exp, holes, staleUnknown)) + } + if !complete && !present { + r.Log.Warn("build: the target language ships no reader words; holes and the notice are marked in the NON-VERBAL form (a sign and numbers)", + "book", r.Book.BookID, "target_lang", r.Book.TargetLang, "langpack_root", r.Book.LangpackRoot) + } + rep := &BuildReport{ + Version: buildVersion, BookID: exp.BookID, Files: map[string]string{}, + TotalUnits: exp.TotalUnits, GhostRows: exp.GhostRows, ConfigDrift: exp.ConfigDrift, StaleUnknown: staleUnknown, + Complete: complete, TextModified: bookfile.Modified(book.Modified), + } + for _, h := range holes { + switch h.Kind { + case HolePending: + rep.PendingUnits++ + case HoleWithheld: + rep.WithheldUnits++ + case HoleIncomplete: + rep.IncompleteUnits++ + case HoleStale: + rep.StaleUnits++ + } + } + + // Stage every format, then commit every format (the writeDecisionFiles discipline, artifact.go): an + // environment failure while PREPARING the second file must not leave a new epub beside the previous txt. + var staged []*stagedFile + abort := func() { + for _, s := range staged { + s.abort() + } + } + for _, f := range formats { + var buf bytes.Buffer + switch f { + case "epub": + err = bookfile.WriteEPUB(&buf, book) + case "txt": + err = bookfile.WriteTXT(&buf, book) + } + if err != nil { + abort() + return nil, err + } + path := r.bookFilePath(f) + if opts.Out != "" { + path = opts.Out + } + s, err := stageFileAtomic(path, buf.Bytes()) + if err != nil { + abort() + if opts.Out != "" { + // The caller named the place and the place cannot take a file: their argument, the + // config class — not an engine failure on exit 1. + return nil, refuse(RefusalBadConfig, fmt.Errorf("pipeline: build %s: --out %s cannot be written: %w", r.Book.BookID, opts.Out, err)) + } + return nil, err + } + staged = append(staged, s) + rep.Files[f] = absPath(path) + } + for i, s := range staged { + if err := s.commit(); err != nil { + for _, rest := range staged[i+1:] { + rest.abort() + } + landed := make([]string, 0, i) + for _, done := range staged[:i] { + landed = append(landed, done.path) + } + return nil, refuse(RefusalWriteIncomplete, fmt.Errorf("pipeline: build %s: the copies were prepared and the write did not complete — %w; landed: [%s]; not landed: %s and the rest — rebuild to converge", + r.Book.BookID, err, strings.Join(landed, " "), s.path)) + } + } + if opts.Out == "" { + // The set beside the database is from THIS build: a format not asked for is removed rather than + // left as an older copy the envelope would present as current (see the file comment). + for _, f := range bookfile.Formats { + if _, wanted := rep.Files[f]; wanted { + continue + } + if err := os.Remove(r.bookFilePath(f)); err != nil && !errors.Is(err, fs.ErrNotExist) { + return nil, fmt.Errorf("pipeline: build %s: remove the previous %s copy: %w", r.Book.BookID, f, err) + } + } + } + if exp.ConfigDrift { + r.Log.Warn("build: CONFIG-DRIFT — the file carries the text the run shipped; the current config would render a different snapshot (not written into the file)", + "book", r.Book.BookID, "current", exp.CurrentSnapshot) + } + r.Log.Info("build: book written", "book", r.Book.BookID, "files", rep.Files, "complete", complete, + "pending", rep.PendingUnits, "withheld", rep.WithheldUnits, "incomplete", rep.IncompleteUnits, "stale", rep.StaleUnits, + "stale_unknown", staleUnknown, "ghost_rows", exp.GhostRows) + return rep, nil +} + +// describeListMax bounds the per-hole lines of a refusal: a never-run 2000-chapter book has thousands of +// pending units, and the refusal is a diagnostic, not the manifest. +const describeListMax = 50 + +// describeHoles is the refusal's text: the counts, then every hole by place and reason. +func describeHoles(bookID string, exp *BookExport, holes []hole, staleUnknown bool) error { + counts := map[HoleKind]int{} + for _, h := range holes { + counts[h.Kind]++ + } + var sb strings.Builder + fmt.Fprintf(&sb, "pipeline: build %s: the book is not whole and --partial was not given — nothing written: pending=%d withheld=%d incomplete=%d stale=%d ghost_rows=%d of total_units=%d", + bookID, counts[HolePending], counts[HoleWithheld], counts[HoleIncomplete], counts[HoleStale], exp.GhostRows, exp.TotalUnits) + if staleUnknown { + sb.WriteString(" (stale: UNKNOWN for some or all units — the config drifted, or a row carries no content hash)") + } + for i, h := range holes { + if i == describeListMax { + fmt.Fprintf(&sb, "\n … and %d more", len(holes)-describeListMax) + break + } + switch h.Kind { + case HolePending: + fmt.Fprintf(&sb, "\n chapter %d unit %d: pending (not yet translated)", h.Chapter, h.Unit) + case HoleWithheld: + fmt.Fprintf(&sb, "\n chapter %d unit %d: withheld%s", h.Chapter, h.Unit, parenReason(h.Reason)) + case HoleIncomplete: + fmt.Fprintf(&sb, "\n chapter %d unit %d: incomplete — %d member(s) of the unit missing from its text%s", h.Chapter, h.Unit, h.Dropped, parenReason(h.Reason)) + case HoleStale: + fmt.Fprintf(&sb, "\n chapter %d unit %d: stale (the source changed under this unit since the run; `translate` re-buys it)", h.Chapter, h.Unit) + } + } + if len(exp.GhostUnits) > 0 { + sb.WriteString("\n ghost rows (translated text the current cut of the book cannot place):") + for i, g := range exp.GhostUnits { + if i == describeListMax { + fmt.Fprintf(&sb, " … and %d more", len(exp.GhostUnits)-describeListMax) + break + } + fmt.Fprintf(&sb, " %d/%d", g.Chapter, g.ChunkIdx) + } + } + return fmt.Errorf("%s", sb.String()) +} + +func parenReason(reason string) string { + if reason == "" { + return "" + } + return " (" + reason + ")" +} + +// assembleBook turns the export into the document model: chapters from the units in manifest order, +// the known heading prefix stripped and re-used as the title, a mark at every hole, the notice when +// there is one — and returns the holes it marked, in reading order, which is what the refusal and the +// report count (the writer's own reading of the units, not the counters'). Pure: the same export, stale +// set and words yield the same Book. +func assembleBook(exp *BookExport, stale map[UnitRef]bool, title, language string, words lang.ReaderWords) (*bookfile.Book, []hole, error) { + modified, err := bookModified(exp.TextModified) + if err != nil { + return nil, nil, err + } + book := &bookfile.Book{ + Identifier: bookIdentifierPrefix + exp.BookID, + Title: strings.TrimSpace(title), + Language: language, + Modified: modified, + } + // Ghost rows by the chapter number they were STORED under — the old cut's numbering. Chapter numbers + // are dense, so a source edit that empties a chapter shifts every later number, and a ghost's mark can + // then sit in a neighbouring chapter's text; the book is still marked not-whole, only the mark's + // coordinate is the old cut's. A ghost whose chapter the current cut no longer has is in the notice's + // count and nowhere else. + ghostsByChapter := map[int]int{} + for _, g := range exp.GhostUnits { + ghostsByChapter[g.Chapter]++ + } + var holes []hole + chapterNo := 0 // dense chapter number of the chapter being assembled; 0 = none yet (numbers are 1-based) + closeChapter := func() { + if chapterNo == 0 { + return + } + if n := ghostsByChapter[chapterNo]; n > 0 { + i := len(book.Chapters) - 1 + book.Chapters[i].Paragraphs = append(book.Chapters[i].Paragraphs, + lang.FillReaderTemplate(words.HoleGhost, "chapter", strconv.Itoa(chapterNo), "ghost", strconv.Itoa(n))) + delete(ghostsByChapter, chapterNo) + } + } + for _, ce := range exp.Chunks { + if ce.Chapter != chapterNo { + closeChapter() + chapterNo = ce.Chapter + t := ce.Heading + if t == "" { + t = strconv.Itoa(ce.Chapter) + } + book.Chapters = append(book.Chapters, bookfile.Chapter{Title: t}) + } + i := len(book.Chapters) - 1 + chapter, unit := strconv.Itoa(ce.Chapter), strconv.Itoa(ce.ChunkIdx) + text := ce.FinalText + if ce.Heading != "" { + // The literal chunk.ApplyHeading glued on: `heading + "\n\n" + text`. A KNOWN prefix, stripped + // only when it is there; never a guess at the text. + text = strings.TrimPrefix(text, ce.Heading+"\n\n") + } + kind := UnitHole(ce) + var paragraphs []string + switch kind { + case HoleNone, HoleIncomplete: + if stale[UnitRef{Chapter: ce.Chapter, ChunkIdx: ce.ChunkIdx}] { + // The text is the translation of a source that is no longer the book's: not shown. + kind = HoleStale + break + } + paragraphs = textParagraphs(text) + if len(paragraphs) == 0 { + // A row that says «text» over characters no reader's file can carry shipped nothing a + // reader can see; the file says so rather than carrying an empty chapter. + kind = HoleWithheld + } + } + mark := func(tmpl string, extra ...string) { + args := append([]string{"chapter", chapter, "unit", unit}, extra...) + book.Chapters[i].Paragraphs = append(book.Chapters[i].Paragraphs, lang.FillReaderTemplate(tmpl, args...)) + } + switch kind { + case HolePending: + mark(words.HolePending) + case HoleWithheld: + mark(words.HoleWithheld) + holes = append(holes, hole{Kind: HoleWithheld, Chapter: ce.Chapter, Unit: ce.ChunkIdx, Reason: ce.FlagReason}) + continue + case HoleStale: + mark(words.HoleStale) + case HoleIncomplete: + mark(words.HoleIncomplete, "dropped", strconv.Itoa(ce.DroppedMembers)) + holes = append(holes, hole{Kind: HoleIncomplete, Chapter: ce.Chapter, Unit: ce.ChunkIdx, Reason: ce.DroppedReason, Dropped: ce.DroppedMembers}) + } + if kind == HolePending || kind == HoleStale { + holes = append(holes, hole{Kind: kind, Chapter: ce.Chapter, Unit: ce.ChunkIdx}) + continue + } + book.Chapters[i].Paragraphs = append(book.Chapters[i].Paragraphs, paragraphs...) + } + closeChapter() + if len(holes) > 0 { + book.Notice = append(book.Notice, + lang.FillReaderTemplate(words.NoticeHoles, "holes", strconv.Itoa(len(holes)), "total", strconv.Itoa(exp.TotalUnits))) + } + if exp.GhostRows > 0 { + book.Notice = append(book.Notice, lang.FillReaderTemplate(words.NoticeGhost, "ghost", strconv.Itoa(exp.GhostRows))) + } + book.Description = strings.Join(book.Notice, " ") + return book, holes, nil +} + +// textParagraphs splits export text into paragraphs: one per non-blank line (CR, LF and CRLF all end a +// line — what text.NormalizeSource does on ingest, so a bare CR cannot make the two formats disagree), +// trimmed and cleaned of the control characters no reader's format admits (bookfile.CleanText — applied +// HERE, once, so the EPUB and the text file carry the same paragraphs; see the file comment on why the +// line is the unit). +func textParagraphs(text string) []string { + text = strings.ReplaceAll(text, "\r\n", "\n") + text = strings.ReplaceAll(text, "\r", "\n") + var out []string + for _, line := range strings.Split(text, "\n") { + if p := strings.TrimSpace(bookfile.CleanText(line)); p != "" { + out = append(out, p) + } + } + return out +} + +// looksLikeLanguageTag is the syntactic shape of a BCP 47 tag (a 2–3 letter primary subtag, optional +// alphanumeric subtags): the one check that keeps an unusable dc:language a loud refusal instead of a +// file every validator rejects. It knows no language; it knows what a tag looks like. +func looksLikeLanguageTag(s string) bool { + subtags := strings.Split(s, "-") + if len(subtags[0]) < 2 || len(subtags[0]) > 3 { + return false + } + for i, sub := range subtags { + if sub == "" || len(sub) > 8 { + return false + } + for _, r := range sub { + isLetter := (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') + isDigit := r >= '0' && r <= '9' + if !isLetter && !(isDigit && i > 0) { + return false + } + } + } + return true +} + +// bookModified turns BookExport.TextModified into the file's modification time. A book that shipped +// nothing ("" — no final row) is stamped with the epoch: nothing has ever modified its text, and a +// constant is the one value that is neither a clock reading nor a claim. +func bookModified(textModified string) (time.Time, error) { + if textModified == "" { + return time.Unix(0, 0).UTC(), nil + } + t, err := time.Parse("2006-01-02T15:04:05Z", textModified) + if err != nil { + return time.Time{}, fmt.Errorf("pipeline: build: export text_modified %q is not CCYY-MM-DDThh:mm:ssZ: %w", textModified, err) + } + return t.UTC(), nil +} diff --git a/backend/internal/pipeline/bookbuild_test.go b/backend/internal/pipeline/bookbuild_test.go new file mode 100644 index 00000000..ad1eb01f --- /dev/null +++ b/backend/internal/pipeline/bookbuild_test.go @@ -0,0 +1,657 @@ +package pipeline + +import ( + "bytes" + "context" + "errors" + "os" + "path/filepath" + "strconv" + "strings" + "testing" + + "textmachine/backend/internal/bookfile" + "textmachine/backend/internal/chunk" + "textmachine/backend/internal/lang" + "textmachine/backend/internal/obs" +) + +// bookbuild_test.go pins the book writer (backlog row 236) on the runner harness: the circle +// «translated → built → read back by the engine's own ingest» closes by EQUALITY of paragraphs, and +// each of the five states a file may not lie about is driven through the real projection where a +// run can reach it (pending, withheld, ghost, drift) and through a synthetic projection where it +// cannot (incomplete, the in-chapter ghost mark, a heading, a book of no units). + +// ingestParagraphs reads an EPUB back through the engine's reader and splits each chapter the way the +// chunker does (blank lines, trimmed, empties dropped — chunk.splitParagraphs). +func ingestParagraphs(t *testing.T, epubPath string) [][]string { + t.Helper() + doc, err := chunk.IngestEncoded(epubPath, "", "") + if err != nil { + t.Fatalf("ingest of the written EPUB failed: %v", err) + } + out := make([][]string, len(doc.Chapters)) + for i, ch := range doc.Chapters { + for _, raw := range strings.Split(ch, "\n\n") { + if p := strings.TrimSpace(raw); p != "" { + out[i] = append(out[i], p) + } + } + } + return out +} + +func assertRoundTrip(t *testing.T, book *bookfile.Book, epubPath string) { + t.Helper() + got := ingestParagraphs(t, epubPath) + if len(got) != len(book.Chapters) { + t.Fatalf("ingest read %d chapters back, the book has %d", len(got), len(book.Chapters)) + } + for i := range book.Chapters { + want := book.Blocks(i) + if strings.Join(got[i], "\n|") != strings.Join(want, "\n|") { + t.Errorf("chapter %d read back differently:\n got %q\nwant %q", i+1, got[i], want) + } + } +} + +// multiLineEdit makes the editor return prose with BOTH separators the real editor uses (a bare newline +// and a blank line), so the paragraph rule is exercised rather than assumed. +func multiLineEdit(body string) (string, string) { + if isEditBody(body) { + return "Первая строка абзаца.\nВторая строка.\n\nТретья, после пустой.", "stop" + } + return "ЧЕРНОВИК ПЕРЕВОДА", "stop" +} + +func TestBuildBookRoundTripsThroughIngestAndPublishesItsPaths(t *testing.T) { + srv := newJSONProvider(&reqRec{}, multiLineEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ГЛАВАА\fГЛАВАБ\fГЛАВАВ", regenerate: 0}) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + r := newRunner(t, bookPath) + defer r.Close() + if _, err := r.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + + rep, err := r.BuildBook(BuildOptions{}) + if err != nil { + t.Fatalf("BuildBook: %v", err) + } + if !rep.Complete || rep.PendingUnits+rep.WithheldUnits+rep.IncompleteUnits+rep.StaleUnits+rep.GhostRows != 0 || rep.ConfigDrift || rep.StaleUnknown { + t.Fatalf("a fully translated book must build complete, with the stale check MADE: %+v", rep) + } + if rep.Version != buildVersion || rep.TextModified == "" || rep.TextModified == "1970-01-01T00:00:00Z" { + t.Errorf("report envelope/stamp: %+v", rep) + } + dir := filepath.Dir(bookPath) + for _, f := range bookfile.Formats { + want := filepath.Join(dir, "test-book.db.book."+f) + if rep.Files[f] != want { + t.Errorf("files[%s] = %q, want %q", f, rep.Files[f], want) + } + if _, err := os.Stat(want); err != nil { + t.Errorf("%s not written: %v", want, err) + } + } + // The status envelope publishes the same places, whether or not a build ran. + st, err := r.Status(ctx) + if err != nil { + t.Fatal(err) + } + for f, p := range rep.Files { + if st.Artifacts.BookFiles[f] != p { + t.Errorf("status artifacts.book_files[%s] = %q, the build wrote %q", f, st.Artifacts.BookFiles[f], p) + } + } + + // The circle: what the engine reads back is what the writer laid down, chapter by chapter. + exp, err := r.Export(false) + if err != nil { + t.Fatal(err) + } + book, _, err := assembleBook(exp, nil, r.Book.Title, r.Book.TargetLang, lang.DefaultReaderWords()) + if err != nil { + t.Fatal(err) + } + if len(book.Chapters) != 3 || len(book.Chapters[0].Paragraphs) != 3 || book.Chapters[0].Paragraphs[1] != "Вторая строка." { + t.Fatalf("paragraph rule: a non-blank line is a paragraph: %+v", book.Chapters[0]) + } + if book.Chapters[0].Title != "1" || len(book.Notice) != 0 || book.Description != "" { + t.Errorf("a headingless complete book: bare-number titles, no notice: %+v", book) + } + assertRoundTrip(t, book, rep.Files["epub"]) + + // The text file is the same blocks, with no operator vocabulary. + txt, err := os.ReadFile(rep.Files["txt"]) + if err != nil { + t.Fatal(err) + } + for _, banned := range []string{"=== CHAPTER", "TEXT MISSING", "pending", "⚠"} { + if strings.Contains(string(txt), banned) { + t.Errorf("complete book's txt carries %q:\n%s", banned, txt) + } + } + if !strings.HasPrefix(string(txt), "Тест\n\n\n1\n\nПервая строка абзаца.\n\nВторая строка.\n\nТретья, после пустой.\n\n\n2\n") { + t.Errorf("txt shape:\n%s", txt) + } + + // Determinism in-process (the two-process proof is the stand's cmp): a rebuild is byte-identical. + first, _ := os.ReadFile(rep.Files["epub"]) + if _, err := r.BuildBook(BuildOptions{}); err != nil { + t.Fatal(err) + } + second, _ := os.ReadFile(rep.Files["epub"]) + if !bytes.Equal(first, second) { + t.Error("two builds of one store differ byte-wise") + } +} + +func TestBuildBookRefusesAPendingBookAndPartialMarksIt(t *testing.T) { + srv := newJSONProvider(&reqRec{}, draftEdit) + defer srv.Close() + // The ceiling pays for about one chapter, then denies (TestExportManifestPending's setup). + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ГЛАВАА\fГЛАВАБ\fГЛАВАВ", regenerate: 0, bookUSD: 0.005}) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + r := newRunner(t, bookPath) + defer r.Close() + _, _ = r.TranslateBook(ctx) // the ceiling stop is expected; the partial store is the fixture + + _, err := r.BuildBook(BuildOptions{}) + if refusalClassOf(err) != RefusalBookIncomplete { + t.Fatalf("class = %v", err) + } + if !strings.Contains(err.Error(), "pending (not yet translated)") || !strings.Contains(err.Error(), "--partial") { + t.Errorf("the refusal must list the holes and name the way out: %v", err) + } + dir := filepath.Dir(bookPath) + if _, err := os.Stat(filepath.Join(dir, "test-book.db.book.epub")); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("a refusal must write nothing: %v", err) + } + + rep, err := r.BuildBook(BuildOptions{Partial: true}) + if err != nil { + t.Fatalf("--partial must write: %v", err) + } + if rep.Complete || rep.PendingUnits < 1 { + t.Fatalf("report must say the book is not whole: %+v", rep) + } + exp, _ := r.Export(false) + book, _, err := assembleBook(exp, nil, r.Book.Title, r.Book.TargetLang, lang.DefaultReaderWords()) + if err != nil { + t.Fatal(err) + } + // No langpack → the non-verbal form: the notice is «⚠ holes/total», the mark «⚠ chapter.unit». + if len(book.Notice) != 1 || !strings.HasPrefix(book.Notice[0], "⚠ ") || !strings.HasSuffix(book.Notice[0], "/3") { + t.Errorf("notice = %q", book.Notice) + } + marked := 0 + for _, ce := range exp.Chunks { + if ce.Disposition != exportPending { + continue + } + marked++ + ch := book.Chapters[ce.Chapter-1] + if len(ch.Paragraphs) != 1 || ch.Paragraphs[0] != "⚠ "+strconv.Itoa(ce.Chapter)+"."+strconv.Itoa(ce.ChunkIdx) { + t.Errorf("pending chapter %d must carry exactly the mark: %+v", ce.Chapter, ch) + } + } + if marked != rep.PendingUnits { + t.Errorf("marked %d pending units, report says %d", marked, rep.PendingUnits) + } + assertRoundTrip(t, book, rep.Files["epub"]) + txt, _ := os.ReadFile(rep.Files["txt"]) + if !strings.HasPrefix(string(txt), "Тест\n\n"+book.Notice[0]+"\n\n\n") { + t.Errorf("txt must open with the title and the notice:\n%s", txt) + } +} + +func TestBuildBookWithheldUnitIsAHole(t *testing.T) { + // The editor leaks a service preamble on chapter 3 → substantive sanitizer defect → the unit ships + // no text (withheld), while chapters 1–2 are clean (TestExportMatchesTranslateFinalText's ch3 path). + srv := newJSONProvider(&reqRec{}, func(body string) (string, string) { + if !isEditBody(body) { + return "черновик перевода этой главы.", "stop" + } + if strings.Contains(body, "ГЛАВАВВ") { + return "Вот перевод фрагмента:\nОн молча ушёл в туман.", "stop" + } + return "Чистый абзац.", "stop" + }) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{ + source: "ГЛАВААА\fГЛАВАББ\fГЛАВАВВ", gatesYAML: "gates:\n sanitizer:\n enabled: true\n", + }) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + r := newRunner(t, bookPath) + defer r.Close() + _, _ = r.TranslateBook(ctx) // completes with flags + + exp, err := r.Export(false) + if err != nil { + t.Fatal(err) + } + if exp.PendingUnits != 0 { + t.Fatalf("the fixture must have NO pending unit — the hole is a withheld one: %+v", exp) + } + withheld, _ := HoleCounts(exp) + if withheld != 1 || UnitHole(exp.Chunks[2]) != HoleWithheld { + t.Fatalf("chapter 3 must be withheld: withheld=%d chunks=%+v", withheld, exp.Chunks) + } + _, err = r.BuildBook(BuildOptions{}) + if refusalClassOf(err) != RefusalBookIncomplete || !strings.Contains(err.Error(), "chapter 3 unit 0: withheld") { + t.Fatalf("pending_units=0 with a withheld unit must still be refused, naming it: %v", err) + } + rep, err := r.BuildBook(BuildOptions{Partial: true, Formats: []string{"epub"}}) + if err != nil { + t.Fatal(err) + } + if rep.WithheldUnits != 1 || rep.Complete { + t.Errorf("report: %+v", rep) + } + book, _, _ := assembleBook(exp, nil, r.Book.Title, r.Book.TargetLang, lang.DefaultReaderWords()) + if got := book.Chapters[2].Paragraphs; len(got) != 1 || got[0] != "⚠ 3.0" { + t.Errorf("withheld chapter must carry exactly the mark: %q", got) + } + assertRoundTrip(t, book, rep.Files["epub"]) + if _, ok := rep.Files["txt"]; ok { + t.Error("--format epub must not write the txt") + } +} + +func TestBuildBookGhostRowsAreAHole(t *testing.T) { + srv := newJSONProvider(&reqRec{}, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ГЛАВАА\fГЛАВАБ\fГЛАВАВ", regenerate: 0}) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + r1.Close() + + // The source SHRINKS to two chapters: chapter 3's final row is now a GHOST — translated text the + // current cut cannot place — while pending stays 0. The «complete» look is exactly the trap. + writeFile(t, filepath.Join(filepath.Dir(bookPath), "source.txt"), "ГЛАВАА\fГЛАВАБ") + r2 := newRunner(t, bookPath) + defer r2.Close() + exp, err := r2.Export(false) + if err != nil { + t.Fatal(err) + } + if exp.PendingUnits != 0 || exp.GhostRows != 1 || len(exp.GhostUnits) != 1 || exp.GhostUnits[0] != (UnitRef{Chapter: 3, ChunkIdx: 0}) { + t.Fatalf("fixture: want pending 0, ghost 1 at 3/0; got %+v", exp) + } + _, err = r2.BuildBook(BuildOptions{}) + if refusalClassOf(err) != RefusalBookIncomplete || !strings.Contains(err.Error(), "ghost rows") || !strings.Contains(err.Error(), " 3/0") { + t.Fatalf("ghost rows alone must refuse, naming the row: %v", err) + } + rep, err := r2.BuildBook(BuildOptions{Partial: true}) + if err != nil { + t.Fatal(err) + } + if rep.Complete || rep.GhostRows != 1 { + t.Errorf("report: %+v", rep) + } + book, _, _ := assembleBook(exp, nil, r2.Book.Title, r2.Book.TargetLang, lang.DefaultReaderWords()) + // Chapter 3 no longer exists to be marked; the book-level notice carries the count. + if len(book.Chapters) != 2 || len(book.Notice) != 1 || book.Notice[0] != "⚠ +1" { + t.Errorf("ghost-only book: 2 chapters, one ghost notice: chapters=%d notice=%q", len(book.Chapters), book.Notice) + } + assertRoundTrip(t, book, rep.Files["epub"]) +} + +func TestBuildBookDriftIsReportedNotMarked(t *testing.T) { + srv := newJSONProvider(&reqRec{}, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{regenerate: 0}) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + r1.Close() + dir := filepath.Dir(bookPath) + body, _ := os.ReadFile(filepath.Join(dir, "pipeline.yaml")) + writeFile(t, filepath.Join(dir, "pipeline.yaml"), strings.Replace(string(body), "prompt_version: v-test", "prompt_version: v-test2", 1)) + + r2 := newRunner(t, bookPath) + defer r2.Close() + rep, err := r2.BuildBook(BuildOptions{}) + if err != nil { + t.Fatalf("drift is not a hole; the default build must write: %v", err) + } + if !rep.ConfigDrift || !rep.Complete { + t.Fatalf("report must carry the drift and call the text complete: %+v", rep) + } + txt, _ := os.ReadFile(rep.Files["txt"]) + if strings.Contains(string(txt), "⚠") || strings.Contains(strings.ToLower(string(txt)), "drift") { + t.Errorf("drift must not reach the reader's file:\n%s", txt) + } +} + +// TestAssembleBookMarksEveryHoleKind drives the assembler on a synthetic projection: the heading is +// stripped and becomes the title, incomplete units keep their text behind the mark, a ghost inside an +// existing chapter is marked at the chapter's end, and the words come from the target language's data. +func TestAssembleBookMarksEveryHoleKind(t *testing.T) { + exp := &BookExport{ + BookID: "syn", TotalUnits: 5, PendingUnits: 1, GhostRows: 2, TextModified: "2026-08-30T12:00:00Z", + GhostUnits: []UnitRef{{Chapter: 2, ChunkIdx: 4}, {Chapter: 2, ChunkIdx: 6}}, + Chunks: []ChunkExport{ + {Chapter: 1, ChunkIdx: 0, Disposition: "ok", Heading: "Глава 1", FinalText: "Глава 1\n\nПодзаголовок модели\nПервый.\n\nВторой."}, + {Chapter: 1, ChunkIdx: 2, Disposition: "ok", FinalText: "Третий."}, + {Chapter: 2, ChunkIdx: 0, Disposition: "flagged", FlagReason: "sanitizer_substantive", FinalText: ""}, + {Chapter: 2, ChunkIdx: 2, Disposition: "flagged", FlagReason: "cjk_artifact", DroppedMembers: 1, DroppedReason: "cjk_artifact", FinalText: "Уцелевшая часть."}, + {Chapter: 3, ChunkIdx: 0, Disposition: "pending"}, + }, + } + root := t.TempDir() + if err := os.MkdirAll(filepath.Join(root, "ru"), 0o755); err != nil { + t.Fatal(err) + } + writeFile(t, filepath.Join(root, "ru", "reader.txt"), "hole.pending\tНе переведено.\nhole.withheld\tНе выпущено.\nhole.incomplete\tНедостаёт {dropped}.\nhole.stale\tУстарело.\nhole.ghost\tВне нарезки {ghost}.\nnotice.holes\tПропусков {holes} из {total}.\nnotice.ghost\tВне нарезки всего {ghost}.\n") + words, present, err := lang.LoadReaderWords(root, "ru") + if err != nil || !present { + t.Fatal(err) + } + book, holes, err := assembleBook(exp, nil, "蛊真人", "ru", words) + if err != nil { + t.Fatal(err) + } + if len(holes) != 3 { + t.Errorf("holes = %+v", holes) + } + if book.Identifier != "urn:textmachine:book:syn" || book.Language != "ru" || book.Title != "蛊真人" || bookfile.Modified(book.Modified) != "2026-08-30T12:00:00Z" { + t.Errorf("metadata: %+v", book) + } + want := []bookfile.Chapter{ + {Title: "Глава 1", Paragraphs: []string{"Подзаголовок модели", "Первый.", "Второй.", "Третий."}}, + {Title: "2", Paragraphs: []string{"Не выпущено.", "Недостаёт 1.", "Уцелевшая часть.", "Вне нарезки 2."}}, + {Title: "3", Paragraphs: []string{"Не переведено."}}, + } + if len(book.Chapters) != len(want) { + t.Fatalf("chapters: %+v", book.Chapters) + } + for i := range want { + if book.Chapters[i].Title != want[i].Title || strings.Join(book.Chapters[i].Paragraphs, "|") != strings.Join(want[i].Paragraphs, "|") { + t.Errorf("chapter %d = %+v, want %+v", i+1, book.Chapters[i], want[i]) + } + } + if strings.Join(book.Notice, "|") != "Пропусков 3 из 5.|Вне нарезки всего 2." || book.Description != "Пропусков 3 из 5. Вне нарезки всего 2." { + t.Errorf("notice = %q description = %q", book.Notice, book.Description) + } + // The heading is stripped ONLY as the known prefix: a text that merely mentions the heading later + // keeps it, and a heading absent from the text strips nothing. + exp2 := &BookExport{BookID: "syn", TotalUnits: 1, TextModified: "2026-08-30T12:00:00Z", + Chunks: []ChunkExport{{Chapter: 1, ChunkIdx: 0, Disposition: "ok", Heading: "Глава 1", FinalText: "Текст, где Глава 1 упомянута."}}} + b2, _, err := assembleBook(exp2, nil, " t ", "ru", words) + if err != nil || b2.Chapters[0].Paragraphs[0] != "Текст, где Глава 1 упомянута." || b2.Chapters[0].Title != "Глава 1" || b2.Title != "t" { + t.Errorf("prefix rule / trimmed title: %+v %q %v", b2.Chapters, b2.Title, err) + } + // A book no run has written is stamped with the epoch, and a malformed stamp fails loud. + exp2.TextModified = "" + if b3, _, err := assembleBook(exp2, nil, "t", "ru", words); err != nil || bookfile.Modified(b3.Modified) != "1970-01-01T00:00:00Z" { + t.Errorf("epoch stamp: %v %v", err, b3) + } + exp2.TextModified = "yesterday" + if _, _, err := assembleBook(exp2, nil, "t", "ru", words); err == nil { + t.Error("a malformed text_modified must fail loud") + } + // The writer's own reading of a unit: text of nothing a reader can see is withheld; a bare CR ends a + // line like LF; a stale unit shows its mark and not its (outdated) text, ahead of the incomplete rule. + exp3 := &BookExport{BookID: "syn", TotalUnits: 3, TextModified: "2026-08-30T12:00:00Z", Chunks: []ChunkExport{ + {Chapter: 1, ChunkIdx: 0, Disposition: "ok", FinalText: "\x01\x02"}, + {Chapter: 2, ChunkIdx: 0, Disposition: "ok", FinalText: "Первая\rВторая\r\nТретья"}, + {Chapter: 3, ChunkIdx: 0, Disposition: "flagged", DroppedMembers: 1, DroppedReason: "x", FinalText: "Старый текст."}, + }} + b4, holes4, err := assembleBook(exp3, map[UnitRef]bool{{Chapter: 3, ChunkIdx: 0}: true}, "t", "ru", words) + if err != nil { + t.Fatal(err) + } + if got := b4.Chapters[0].Paragraphs; len(got) != 1 || got[0] != "Не выпущено." { + t.Errorf("control-only text must be a withheld hole, got %q", got) + } + if got := strings.Join(b4.Chapters[1].Paragraphs, "|"); got != "Первая|Вторая|Третья" { + t.Errorf("CR must end a line: %q", got) + } + if got := b4.Chapters[2].Paragraphs; len(got) != 1 || got[0] != "Устарело." { + t.Errorf("a stale unit shows its mark and not its old text, got %q", got) + } + if len(holes4) != 2 || holes4[0].Kind != HoleWithheld || holes4[1].Kind != HoleStale || b4.Notice[0] != "Пропусков 2 из 3." { + t.Errorf("holes = %+v notice = %q", holes4, b4.Notice) + } +} + +// TestBuildBookStaleUnitsAreAHole is the final review's blocker: a source edit that keeps the unit keys +// (a typo fix) used to yield a byte-identical file under complete=true — the translation of text no +// longer in the book. The writer now reads the same fact the resume path acts on (content hash). +func TestBuildBookStaleUnitsAreAHole(t *testing.T) { + srv := newJSONProvider(&reqRec{}, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ГЛАВАА\fГЛАВАБ", regenerate: 0}) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + rep1, err := r1.BuildBook(BuildOptions{}) + if err != nil || !rep1.Complete || rep1.StaleUnknown || rep1.StaleUnits != 0 { + t.Fatalf("a fresh run must build complete with the stale check MADE: %+v %v", rep1, err) + } + r1.Close() + // The same two chapters, one of them edited in place: keys survive, the text under one does not. + writeFile(t, filepath.Join(filepath.Dir(bookPath), "source.txt"), "ГЛАВАА\fГЛАВАБ — исправлено") + r2 := newRunner(t, bookPath) + _, err = r2.BuildBook(BuildOptions{}) + if refusalClassOf(err) != RefusalBookIncomplete || !strings.Contains(err.Error(), "chapter 2 unit 0: stale") || strings.Contains(err.Error(), "chapter 1 unit 0") { + t.Fatalf("the edited unit — and only it — must be a stale hole: %v", err) + } + rep2, err := r2.BuildBook(BuildOptions{Partial: true}) + if err != nil || rep2.Complete || rep2.StaleUnits != 1 || rep2.StaleUnknown { + t.Fatalf("partial: %+v %v", rep2, err) + } + exp, _ := r2.Export(false) + stale, unknown := r2.staleUnits(exp) + book, _, _ := assembleBook(exp, stale, r2.Book.Title, r2.Book.TargetLang, lang.DefaultReaderWords()) + if unknown || !stale[UnitRef{Chapter: 2, ChunkIdx: 0}] || strings.Join(book.Chapters[1].Paragraphs, "|") != "⚠ 2.0" { + t.Errorf("stale=%v unknown=%v chapter2=%q", stale, unknown, book.Chapters[1].Paragraphs) + } + assertRoundTrip(t, book, rep2.Files["epub"]) + r2.Close() + // Re-translating (the engine re-buys the moved unit) makes the book whole again. + r3 := newRunner(t, bookPath) + defer r3.Close() + if _, err := r3.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + rep3, err := r3.BuildBook(BuildOptions{}) + if err != nil || !rep3.Complete || rep3.StaleUnits != 0 { + t.Fatalf("after a re-translate the book must be whole: %+v %v", rep3, err) + } + txt, _ := os.ReadFile(rep3.Files["txt"]) + if strings.Contains(string(txt), "⚠") { + t.Errorf("no mark may survive the re-translate:\n%s", txt) + } +} + +// TestBuildBookRemovedStageIsDrift: removing the editor stage used to make the draft rows the shipping +// rows with no drift — an unedited book that reads as complete. Now a stored row of a stage the config +// does not run is drift (export.go), and the writer's stale check says «unknown» under drift. +func TestBuildBookRemovedStageIsDrift(t *testing.T) { + srv := newJSONProvider(&reqRec{}, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{regenerate: 0}) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + r1.Close() + dir := filepath.Dir(bookPath) + body, _ := os.ReadFile(filepath.Join(dir, "pipeline.yaml")) + var kept []string + for _, line := range strings.Split(string(body), "\n") { + if !strings.Contains(line, "name: edit") { + kept = append(kept, line) + } + } + writeFile(t, filepath.Join(dir, "pipeline.yaml"), strings.Join(kept, "\n")) + r2 := newRunner(t, bookPath) + defer r2.Close() + rep, err := r2.BuildBook(BuildOptions{}) + if err != nil { + t.Fatal(err) + } + if !rep.ConfigDrift || !rep.StaleUnknown { + t.Fatalf("a removed stage is drift, and under drift the stale check is unknown: %+v", rep) + } +} + +func TestBuildBookRefusesABookOfNoUnitsAndBadOptions(t *testing.T) { + srv := newJSONProvider(&reqRec{}, draftEdit) + defer srv.Close() + r := newRunner(t, setupProject(t, srv.URL)) + defer r.Close() + _, err := r.buildFromExport(&BookExport{Version: exportVersion, BookID: "empty", Chunks: []ChunkExport{}}, nil, false, bookfile.Formats, BuildOptions{}) + if refusalClassOf(err) != RefusalSourceUnreadable { + t.Errorf("a book of no units: %v", err) + } + if _, err := r.BuildBook(BuildOptions{Formats: []string{"pdf"}}); err == nil || strings.Contains(err.Error(), "refus") { + t.Errorf("unknown format must be a plain error: %v", err) + } + if _, err := r.BuildBook(BuildOptions{Out: "x"}); err == nil { + t.Error("--out with two formats must be an error") + } +} + +// TestBuildBookLeavesOneBuildBesideTheDatabase pins the fix of the mid-work review's blocker: a +// `--format` subset must not leave the other format's OLDER copy beside the database, where the +// envelope would present it as current — the «silently complete» outcome through a legal sequence +// (build; translate more; build --format epub). +func TestBuildBookLeavesOneBuildBesideTheDatabase(t *testing.T) { + srv := newJSONProvider(&reqRec{}, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "ГЛАВАА\fГЛАВАБ", regenerate: 0}) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + r1 := newRunner(t, bookPath) + if _, err := r1.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + if _, err := r1.BuildBook(BuildOptions{}); err != nil { + t.Fatal(err) + } + r1.Close() + dir := filepath.Dir(bookPath) + txtPath := filepath.Join(dir, "test-book.db.book.txt") + if _, err := os.Stat(txtPath); err != nil { + t.Fatal(err) + } + // The book grows a chapter (pending) and only the epub is rebuilt, partial. + writeFile(t, filepath.Join(dir, "source.txt"), "ГЛАВАА\fГЛАВАБ\fГЛАВАВ") + r2 := newRunner(t, bookPath) + defer r2.Close() + rep, err := r2.BuildBook(BuildOptions{Formats: []string{"epub"}, Partial: true}) + if err != nil { + t.Fatal(err) + } + if rep.Complete || rep.PendingUnits != 1 { + t.Fatalf("fixture: %+v", rep) + } + if _, err := os.Stat(txtPath); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("the older txt (which reads as a COMPLETE book) must not survive a subset rebuild beside the database: %v", err) + } + if _, ok := rep.Files["txt"]; ok { + t.Error("the report must list only what this build wrote") + } + // An explicit --out leaves the default place alone (it wrote elsewhere). + if _, err := r2.BuildBook(BuildOptions{Formats: []string{"txt"}, Partial: true, Out: filepath.Join(t.TempDir(), "b.txt")}); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(filepath.Join(dir, "test-book.db.book.epub")); err != nil { + t.Errorf("--out must not remove the copies beside the database: %v", err) + } +} + +func TestBuildBookRefusesAConfigItCannotFileUnder(t *testing.T) { + srv := newJSONProvider(&reqRec{}, draftEdit) + defer srv.Close() + r := newRunner(t, setupProject(t, srv.URL)) + defer r.Close() + exp := &BookExport{Version: exportVersion, BookID: "b", TotalUnits: 1, Chunks: []ChunkExport{{Chapter: 1, ChunkIdx: 0, Disposition: "ok", FinalText: "x"}}} + title := r.Book.Title + r.Book.Title = " " + if _, err := r.buildFromExport(exp, nil, false, bookfile.Formats, BuildOptions{}); refusalClassOf(err) != RefusalBadConfig || !strings.Contains(err.Error(), "title") { + t.Errorf("an empty title is a config refusal naming the key: %v", err) + } + r.Book.Title = title + for _, bad := range []string{"russian", "ru_RU", "", "r", "ru-", "ru-très"} { + r.Book.TargetLang = bad + if _, err := r.buildFromExport(exp, nil, false, bookfile.Formats, BuildOptions{}); refusalClassOf(err) != RefusalBadConfig || !strings.Contains(err.Error(), "target_lang") { + t.Errorf("target_lang %q must be a config refusal naming the key: %v", bad, err) + } + } + for _, good := range []string{"ru", "zh", "pt-BR", "zh-Hant-TW", "en-x-private"} { + if !looksLikeLanguageTag(good) { + t.Errorf("%q looks like a language tag and must pass", good) + } + } +} + +// TestBuildBookOutNamesANewFile pins the --out rule (the final review's second blocker: --out used to +// replace ANY path, the project database included): an existing file is refused, a missing directory is +// refused, a new path is written, and the default place is left alone. +func TestBuildBookOutNamesANewFile(t *testing.T) { + srv := newJSONProvider(&reqRec{}, draftEdit) + defer srv.Close() + bookPath := setupProject(t, srv.URL) + ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()}) + r := newRunner(t, bookPath) + defer r.Close() + if _, err := r.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + dbBefore, _ := os.ReadFile(r.Book.ProjectDB) + for name, out := range map[string]string{ + "the project database": r.Book.ProjectDB, + "the source": r.Book.SourceFile, + "an existing file": filepath.Join(t.TempDir(), "taken.epub"), + "a missing directory": filepath.Join(t.TempDir(), "nope", "book.epub"), + } { + if name == "an existing file" { + writeFile(t, out, "somebody's bytes") + } + _, err := r.BuildBook(BuildOptions{Formats: []string{"epub"}, Out: out}) + if refusalClassOf(err) != RefusalBadConfig { + t.Errorf("--out %s must be refused with the config class: %v", name, err) + } + } + dbAfter, _ := os.ReadFile(r.Book.ProjectDB) + if !bytes.Equal(dbBefore, dbAfter) { + t.Fatal("the project database was touched by a refused --out") + } + out := filepath.Join(t.TempDir(), "book.epub") + rep, err := r.BuildBook(BuildOptions{Formats: []string{"epub"}, Out: out}) + if err != nil { + t.Fatal(err) + } + if rep.Files["epub"] != out { + t.Errorf("files = %v", rep.Files) + } + b, _ := os.ReadFile(out) + if !bytes.HasPrefix(b, []byte("PK")) { + t.Errorf("the explicit path must hold the EPUB: %q", b[:min(len(b), 16)]) + } + if _, err := os.Stat(filepath.Join(filepath.Dir(bookPath), "test-book.db.book.epub")); !errors.Is(err, os.ErrNotExist) { + t.Error("--out must not also write the default place") + } + // No temp file survives a refusal or a write, anywhere the writer looked. + for _, d := range []string{filepath.Dir(out), filepath.Dir(bookPath)} { + entries, _ := os.ReadDir(d) + for _, e := range entries { + if strings.Contains(e.Name(), ".tmp-") { + t.Errorf("temp file left behind: %s/%s", d, e.Name()) + } + } + } +} diff --git a/backend/internal/pipeline/export.go b/backend/internal/pipeline/export.go index 45ebad11..9c5ba533 100644 --- a/backend/internal/pipeline/export.go +++ b/backend/internal/pipeline/export.go @@ -2,7 +2,9 @@ package pipeline import ( "fmt" + "sort" "strings" + "time" "textmachine/backend/internal/chunk" "textmachine/backend/internal/store" @@ -52,6 +54,21 @@ type ChunkExport struct { // DroppedReason is why the FIRST dropped member flagged — the cause of the HOLE, which FlagReason // is not when the edit flagged on its own account as well. "" when nothing dropped. DroppedReason string `json:"dropped_reason,omitempty"` + // Heading is the chapter title the engine renders for this unit's chapter — the SAME literal + // chunk.ApplyHeading glued onto the front of FinalText (u.Members[0].Heading at assembly time), given + // apart so a consumer that wants the title separate from the prose strips a KNOWN prefix instead of + // guessing at the text. Non-empty only on a chapter-opening unit of a book with a heading rule + // (D39.100 п.1: "" is a legal state — a book without a rule, a chapter without a structural marker), + // and present on a pending unit too (it comes from the manifest, not from a run). Additive and + // omitempty: the export of a book without headings serializes exactly as before this field existed. + // ⚠ It is the engine's TEMPORARY render (D39.122 п.2д), not a label the book's data carries. + Heading string `json:"heading,omitempty"` +} + +// UnitRef addresses one output unit by its manifest key. +type UnitRef struct { + Chapter int `json:"chapter"` + ChunkIdx int `json:"chunk_idx"` } // exportVersion versions the SHAPE of the export document. @@ -76,6 +93,21 @@ type BookExport struct { // below may then not match what translate actually did. CurrentSnapshot is that projected id. ConfigDrift bool `json:"config_drift"` CurrentSnapshot string `json:"current_snapshot,omitempty"` + // GhostUnits are the rows GhostRows counts, BY KEY — so a consumer can say WHICH chapter has + // translated text the current cut of the book cannot place, not only that some chapter does. The + // book writer marks those chapters (a coarsened cut leaves the joined leader row carrying the text + // of a SHORTER unit than the current one; the rest of the source's translation sits in these rows + // and reaches no file). Sorted by key; omitempty (a book without ghosts serializes as before). + GhostUnits []UnitRef `json:"ghost_units,omitempty"` + // TextModified is the newest chunk_status.updated_at among the FINAL rows this export joined — the + // rows whose text (or whose withholding) is in Chunks — rendered as CCYY-MM-DDThh:mm:ssZ: the last + // time the engine wrote the shipped state of this book. Draft rows, ghost rows and rows of stages + // that do not ship are not consulted, so a $0 draft-wave re-write moves nothing here. A fact of the + // store rather than of any process's clock, which is why the book writer uses it as the file's + // modification time (two builds of one store then agree byte-for-byte). "" when no unit has a final + // row (a book that has shipped nothing). Additive and omitempty. ⚠ «Last written», not «last + // changed»: a resume that re-writes an unchanged final row moves it too (store.ChunkStatus.UpdatedAt). + TextModified string `json:"text_modified,omitempty"` // Chunks is always non-nil (an empty book exports [] not null). Chunks []ChunkExport `json:"chunks"` } @@ -126,13 +158,14 @@ func (r *Runner) Export(pairs bool) (*BookExport, error) { exp := &BookExport{Version: exportVersion, BookID: r.Book.BookID, Chunks: []ChunkExport{}} inManifest := map[chunkKey]bool{} + var shipped []store.ChunkStatus // the final rows the export joined — what TextModified is read from for _, u := range units { key := chunkKey{u.Chapter, u.FirstChunkIdx} inManifest[key] = true exp.TotalUnits++ cs, ok := finalRow[key] if !ok { - pend := ChunkExport{Chapter: u.Chapter, ChunkIdx: u.FirstChunkIdx, Disposition: exportPending} + pend := ChunkExport{Chapter: u.Chapter, ChunkIdx: u.FirstChunkIdx, Disposition: exportPending, Heading: u.Members[0].Heading} if pairs { pend.Source = u.sourceText() } @@ -145,22 +178,57 @@ func (r *Runner) Export(pairs bool) (*BookExport, error) { return nil, err } exp.Chunks = append(exp.Chunks, ce) + shipped = append(shipped, cs) } // F4: GHOST rows — a stored final row whose unit-leader key is NOT in the current manifest units // (source shrunk since the run). Dropped (never exported as if live) but counted + WARNed. for k := range finalRow { if !inManifest[k] { exp.GhostRows++ + exp.GhostUnits = append(exp.GhostUnits, UnitRef{Chapter: k.chapter, ChunkIdx: k.chunkIdx}) } } + sort.Slice(exp.GhostUnits, func(i, j int) bool { + a, b := exp.GhostUnits[i], exp.GhostUnits[j] + return a.Chapter < b.Chapter || (a.Chapter == b.Chapter && a.ChunkIdx < b.ChunkIdx) + }) if exp.GhostRows > 0 { r.Log.Warn("export: dropped chunk_status rows outside the current source manifest (source shrunk since the run?)", "book", r.Book.BookID, "ghost_rows", exp.GhostRows) } + modified, err := newestStatusWrite(shipped) + if err != nil { + return nil, err + } + exp.TextModified = modified r.exportConfigDrift(statuses, draftStageNames, exp) return exp, nil } +// storeTimeLayout is how SQLite's datetime('now') spells chunk_status.updated_at (UTC to the second). +const storeTimeLayout = "2006-01-02 15:04:05" + +// newestStatusWrite renders the newest updated_at among the given rows (the joined FINAL rows) as +// BookExport.TextModified. Rows carry the store's own spelling, which sorts lexically, so the newest is +// the string maximum; it is re-parsed rather than re-spelled by hand so a row that is NOT in the store's +// form is a loud inconsistent-store error and not a silently mangled timestamp. "" when there is no row. +func newestStatusWrite(statuses []store.ChunkStatus) (string, error) { + newest := "" + for _, cs := range statuses { + if cs.UpdatedAt > newest { + newest = cs.UpdatedAt + } + } + if newest == "" { + return "", nil + } + t, err := time.Parse(storeTimeLayout, newest) + if err != nil { + return "", fmt.Errorf("pipeline: export: chunk_status.updated_at %q is not in the store's form (inconsistent store): %w", newest, err) + } + return t.UTC().Format("2006-01-02T15:04:05Z"), nil +} + // exportChunks resolves the manifest the export projects over: the persisted one when the export does // not need source text, the full ingest+split when it does (`--pairs`). One helper so the "who needs // text" rule lives in one place instead of being re-decided at each call. @@ -264,6 +332,7 @@ func (r *Runner) exportUnit(u editUnit, cs store.ChunkStatus, byChunk map[chunkK // as the (heading-stripped) source manifest — the DC FP-measure aligns src↔target on the body prose, // not the deterministic title (which no checker inspects). ce.FinalText = chunk.ApplyHeading(u.Members[0].Heading, ce.FinalText) + ce.Heading = u.Members[0].Heading // the same literal, given apart (see the field) if !pairs { return ce, nil } @@ -350,6 +419,24 @@ func (r *Runner) exportConfigDrift(statuses []store.ChunkStatus, draftStageNames } checkWave(draftSnaps, waveDraft) checkWave(editSnaps, waveEdit) + // A stored row for a STAGE the current pipeline does not run is drift of the loudest kind — a stage + // renamed or removed since the run (the CAVEATS' «final-stage rename»). The per-wave check above only + // compares the snapshots of stages that still exist, so without this the removal of the editor stage + // makes the draft rows the shipping rows and nothing fires: an unedited book reads as complete. + for _, cs := range statuses { + if cs.SnapshotID == "" || draftStageNames[cs.Stage] || editStageNames[cs.Stage] { + continue + } + exp.ConfigDrift = true + if exp.CurrentSnapshot == "" { + if cur, _, serr := r.snapshotIDForWave(r.finalStageWave()); serr == nil { + exp.CurrentSnapshot = cur + } + } + r.Log.Warn("export: CONFIG-DRIFT — stored rows carry a stage the current config does not run (renamed or removed since the run); the shipping rows are not the ones the run shipped", + "book", r.Book.BookID, "stage", cs.Stage) + break + } } // chunkExport computes one chunk's export record from its final-stage row, exactly by prod FinalText diff --git a/backend/internal/pipeline/holes.go b/backend/internal/pipeline/holes.go new file mode 100644 index 00000000..53fcb293 --- /dev/null +++ b/backend/internal/pipeline/holes.go @@ -0,0 +1,58 @@ +package pipeline + +// holes.go: what a HOLE in the export is — the one predicate the plaintext render (cmd/tmctl render.go), +// the book writer (bookbuild.go) and any later consumer read the per-unit states through. +// +// BookExport's counters carry three book-level states (pending, ghost, drift). Two more are PER UNIT and +// live only in the record: a unit that reached the final stage and shipped NO text (withheld — a +// substantive flag or an upstream skip, D2: contaminated output never ships), and a unit that shipped +// text with a member's worth of it MISSING (incomplete — the c-lite editor dropped a member). A file that +// consults the counters alone comes out «silently whole» on a book with either, which is the worst thing a +// reader's copy can do; hence one definition, here, rather than an arithmetic each surface repeats. + +// HoleKind names why a unit has no text, or not all of it. +type HoleKind string + +const ( + // HoleNone: the unit shipped its whole text. + HoleNone HoleKind = "" + // HolePending: a manifest unit with no final row — not translated yet (BookExport.PendingUnits). + HolePending HoleKind = "pending" + // HoleWithheld: a final row with no text — a substantive flag / upstream skip withheld it. + HoleWithheld HoleKind = "withheld" + // HoleIncomplete: text present, a member chunk of it missing (ChunkExport.DroppedMembers > 0). + HoleIncomplete HoleKind = "incomplete" + // HoleStale: the unit's final row was made for a source that is no longer the book's (its content + // hash differs from what the run would render now). The WRITER's kind (bookbuild.go staleUnits) — + // UnitHole never returns it, because the export record does not carry the fact. + HoleStale HoleKind = "stale" +) + +// UnitHole classifies one export record. The order of the tests is the plaintext render's: pending +// first, then «no text», then «text with a dropped member» — an incomplete unit DID ship text. +func UnitHole(ce ChunkExport) HoleKind { + switch { + case ce.Disposition == exportPending: + return HolePending + case ce.FinalText == "": + return HoleWithheld + case ce.DroppedMembers > 0: + return HoleIncomplete + } + return HoleNone +} + +// HoleCounts returns the per-unit hole counts of an export: withheld and incomplete (pending is the +// counter BookExport.PendingUnits). `incomplete` is a SUBSET of the exported units, not a fourth part +// of the total: those units did ship text, with a piece of it missing. +func HoleCounts(exp *BookExport) (withheld, incomplete int) { + for _, ce := range exp.Chunks { + switch UnitHole(ce) { + case HoleWithheld: + withheld++ + case HoleIncomplete: + incomplete++ + } + } + return withheld, incomplete +} diff --git a/backend/internal/pipeline/refusal.go b/backend/internal/pipeline/refusal.go index 2e4b8eeb..b902480c 100644 --- a/backend/internal/pipeline/refusal.go +++ b/backend/internal/pipeline/refusal.go @@ -90,6 +90,18 @@ const ( // acceptance's half-write probe). It is NOT the config class: the deployment is fine and the // document is fine — the disk, at that moment, was not. RefusalWriteIncomplete RefusalClass = "write_incomplete" + // RefusalBookIncomplete is a book file `tmctl build` will not write: the book has a HOLE — a unit not + // yet translated, a unit whose text was withheld (a substantive flag, D2), a unit missing a member's + // worth of text, or translated text the current cut of the book cannot place (ghost rows) — and the + // caller did not say `--partial`. The default is the refusal because a reader-facing copy is + // fail-closed (D29.1(б)): a file that is silently short of the book is the worst outcome the writer + // can produce. The refusal lists every hole (chapter, unit, reason), and `--partial` writes the same + // book with the notice on its first page and a mark at every hole (bookbuild.go). + // + // Nothing was written and nothing of the book's state changed — the class keeps the full «nothing + // happened» clause. It is its own class, not the config one: the deployment is fine and the book is + // fine; the book is simply not finished, and the caller decides whether an unfinished copy is wanted. + RefusalBookIncomplete RefusalClass = "book_incomplete" ) // Refusal is an invocation the engine turned down: nothing reached a provider, nothing was spent, no diff --git a/backend/internal/pipeline/status.go b/backend/internal/pipeline/status.go index 51cecc18..a6ba425d 100644 --- a/backend/internal/pipeline/status.go +++ b/backend/internal/pipeline/status.go @@ -130,6 +130,12 @@ type StatusArtifacts struct { // that edits them directly is back to impersonating an operator with a text editor. MinedDelta string `json:"mined_delta"` MinedRejects string `json:"mined_rejects"` + // BookFiles are the READER's copies of the book that `tmctl build` writes, by format + // (bookfile.Formats): `<project_db>.book.<format>`, beside the database like every sidecar. A PLACE + // like the others — the map is complete whether or not a build has run, and a consumer opens the + // path and handles not-found. It is the channel the platform's export door reads instead of + // deriving the spelling (17-seam-inbound-law §1: the engine owns the paths of its own scheme). + BookFiles map[string]string `json:"book_files"` } // artifacts is the ONE producer of the envelope. Two surfaces publish it (`status --json` and @@ -141,6 +147,7 @@ func (r *Runner) artifacts() StatusArtifacts { BankExport: absPath(r.bankExportPath()), MinedDelta: absPath(r.Book.MinedDelta), MinedRejects: absPath(r.Book.MinedRejects), + BookFiles: r.bookFilePaths(), } } diff --git a/backend/internal/store/chunkstatus.go b/backend/internal/store/chunkstatus.go index cea1485d..47accee6 100644 --- a/backend/internal/store/chunkstatus.go +++ b/backend/internal/store/chunkstatus.go @@ -49,6 +49,13 @@ type ChunkStatus struct { // (D18/D19; the mini-run of 25.07 measured 0.0% where 1 of 20 drafts had in fact echoed). // Observability only: re-derived from the stored checkpoints on every run, never a verdict, never wire. FirstFlagReason string + // UpdatedAt is when this row was last WRITTEN, as the store spells it (`datetime('now')`: UTC, + // `YYYY-MM-DD HH:MM:SS`, NOT NULL since the table's first schema — migrate.go). It is read, never + // written from here: UpsertChunkStatus stamps it itself. The book writer takes the newest one as the + // book's modification time — a fact of the store rather than of the clock of whichever process builds + // the file, which is what keeps two builds of one store byte-identical. A resume that re-writes an + // unchanged row moves it too, so it is «last written», not «last changed». + UpdatedAt string } // UpsertChunkStatus writes (or overwrites) the disposition row. Overwrite is the @@ -90,12 +97,12 @@ func (s *Store) GetChunkStatus(bookID string, chapter, chunkIdx int, stage strin var escalated int err := s.r.QueryRowContext(ctx, ` SELECT snapshot_id, content_hash, disposition, flag_reason, attempts, final_hash, cost_usd, detail, - escalated, escalation_model, first_flag_reason + escalated, escalation_model, first_flag_reason, updated_at FROM chunk_status WHERE book_id = ? AND chapter = ? AND chunk_idx = ? AND stage = ?`, bookID, chapter, chunkIdx, stage).Scan( &cs.SnapshotID, &cs.ContentHash, &cs.Disposition, &cs.FlagReason, &cs.Attempts, &cs.FinalHash, &cs.CostUSD, &cs.Detail, - &escalated, &cs.EscalationModel, &cs.FirstFlagReason) + &escalated, &cs.EscalationModel, &cs.FirstFlagReason, &cs.UpdatedAt) cs.Escalated = escalated != 0 if errors.Is(err, sql.ErrNoRows) { return nil, nil @@ -111,7 +118,7 @@ func (s *Store) GetChunkStatus(bookID string, chapter, chunkIdx int, stage strin func (s *Store) ChunkStatusesForBook(bookID string) ([]ChunkStatus, error) { return queryAll(s.r, ` SELECT chapter, chunk_idx, stage, snapshot_id, content_hash, disposition, flag_reason, - attempts, final_hash, cost_usd, detail, escalated, escalation_model, first_flag_reason + attempts, final_hash, cost_usd, detail, escalated, escalation_model, first_flag_reason, updated_at FROM chunk_status WHERE book_id = ? ORDER BY chapter, chunk_idx, stage`, func(rows *sql.Rows) (ChunkStatus, error) { @@ -119,7 +126,7 @@ func (s *Store) ChunkStatusesForBook(bookID string) ([]ChunkStatus, error) { var escalated int if err := rows.Scan(&cs.Chapter, &cs.ChunkIdx, &cs.Stage, &cs.SnapshotID, &cs.ContentHash, &cs.Disposition, &cs.FlagReason, &cs.Attempts, &cs.FinalHash, &cs.CostUSD, &cs.Detail, - &escalated, &cs.EscalationModel, &cs.FirstFlagReason); err != nil { + &escalated, &cs.EscalationModel, &cs.FirstFlagReason, &cs.UpdatedAt); err != nil { return cs, err } cs.Escalated = escalated != 0 diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index 7fa1fdb1..4752dab7 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -279,6 +279,170 @@ > пере-прохода) · **229** (снапшот не фолдит модель внутренних гейтов, латентна) · **230** (размен > «сходимость против поучения») · **141**-остаток · **131**. Плюс потолок объёма из `D39.165` §1б. +### 📝 ЗАПИСКА-ПЛАН — пак «писатель книги» (промт `docs/BACKEND_BOOKWRITER_PACK_SESSION_PROMPT.md`, строка 236), сессия `textmachine-main-fc`, 30.08 + +**Что беру (зона `backend/`, HEAD на старте `67c05be`, дерево чисто).** +1. Прод-пакет `backend/internal/bookfile` (только stdlib — `archive/zip`, `encoding/xml`; без `import "testing"`): + `container.go` — OCF-контейнер (`mimetype` первым и `zip.Store`, `container.xml`, запись entry в порядке вызовов, + нулевые таймметки) — `chunktest/epub.go` переводится на него, его OPF-заглушка и XHTML-декой остаются тестовыми; + `epub.go` — EPUB 3 из нейтральной модели `Book{Identifier, Title, Language, Modified, Description, + Chapters[]{Title, Paragraphs[]}}`: `dc:identifier`/`dc:title`/`dc:language`/`dcterms:modified`, nav + (`properties="nav"`, ⛔ НЕ в spine), один XHTML на главу, `<h1>` = заголовок главы, `<p>` = абзац; + `txt.go` — чистый текст из той же модели (заголовок строкой, абзацы через пустую строку, главы через + пустую строку; ни баннеров, ни флагов, ни `gapMarker`). +2. Сборка из `BookExport` — `backend/internal/pipeline/bookbuild.go`: `Runner.BuildBook(formats, partial)` → + `Export(false)` → дыры → отказ либо файл. Запись `writeFileAtomic` рядом с ProjectDB + (`<project_db>.book.<fmt>`, как `.manifest.json`/`.bank.json`), путь публикуется в + `StatusArtifacts.BookFiles` (`book_files`: map формат→путь, аддитивно) — приор промта принят: платформа + пути не выводит сама (17 §1). Существующий файл — перезапись: производный детерминированный артефакт, как + манифест; бэкап отказывает потому, что он точка восстановления, а файл книги пересобирается из стора. +3. Аддитивные поля `tm-export-v1` (форма не меняется, все `omitempty`): `ChunkExport.Heading` (`heading`) — + литерал `u.Members[0].Heading` в момент сборки записи (и у pending-юнита тоже, он из манифеста); + `BookExport.GhostUnits` (`ghost_units`: `[{chapter,chunk_idx}]`) — ключи ghost-строк, чтобы маркер стоял + в главе, а не только в счётчике; `BookExport.TextModified` (`text_modified`, `CCYY-MM-DDThh:mm:ssZ`) — + max(`chunk_status.updated_at`) по финальным строкам, вошедшим в файл (колонка `NOT NULL DEFAULT + datetime('now')`, `store/migrate.go:37`; в `ChunkStatus` дочитывается доп. полем). Это и есть + `dcterms:modified`: последнее изменение отгруженного текста — факт стора, не часов процесса. +4. Глагол `tmctl build --config b.yaml [--format epub,txt] [--out path] [--partial]` (по умолчанию оба формата; + `--out` — только при одном формате). Stdout — JSON-конверт `tm-build-v1` (пути, счётчики, `complete`). + Новый глагол, а не флаг `export`: `export` — чистое чтение в stdout, писать файлы из него — смешать + аудит-проекцию с артефактом; `manifest` — прецедент глагола-писателя сайдкара. `dispatchCommands` и + usage-строка расширяются — контрактное расширение того же рода, что `bank-apply` (тест + `TestParseNoArgsUsage` сам перечисляет такие расширения); чужим глаголам новые флаги ОТКАЗЫВАЮТСЯ + (дисциплина `--keys-file`/`--max-units`). +5. Отказ — новый `pipeline.RefusalBookIncomplete = "book_incomplete"` → exit **16** (полоса 10–19; таблица + тотальна по `TestTheRefusalTableIsTotal`); сообщение перечисляет дыры `глава/юнит/причина`. +6. Читательские слова — ДАННЫЕ: `<langpack_root>/<target_lang>/reader.txt` (`key<TAB>value`, плейсхолдеры + `{chapter} {unit} {dropped} {holes} {total} {ghost}`), читает ОТДЕЛЬНЫЙ загрузчик `lang.LoadReaderWords`; + в `Pack.Version()` НЕ фолдится (пин тестом «наличие reader.txt не меняет Version()» + `current_snapshot` + minirun до/после). Ключ — язык ЦЕЛИ, не пара: слова читает читатель целевого языка, и они не зависят от + исходного. Поставляю `backend/configs/langpacks/ru/reader.txt`. Каталог `<tgt>/` загрузчик пака не + читает (`Load` — только `<src>/` и `<pair>/`, `langpack.go:215-330`), так что канал вне фолда по + построению, а не по соглашению. + +**Политика §3.2 — по умолчанию ОТКАЗ, `--partial` — файл с пометкой; довод:** +- Дыры: pending · withheld (`Disposition!="pending" && FinalText==""`) · incomplete (`DroppedMembers>0` при + непустом тексте) — арифметика `render.go:573-587` выносится в общий хелпер `pipeline`, которым пользуется и + `renderExport` (вывод `--plaintext` побайтово прежний) · **ghost — тоже дыра**: при укрупнении нарезки + строка-лидер несёт текст ПРЕЖНЕГО, более короткого юнита, а остаток источника лежит в ghost-строках и в + файл не попадает (acceptance: 57 строк → 37 юнитов, 20 ghost при `pending 0` — ровно «молча полная» + книга). Позиция ghost внутри главы неизвестна → маркер в конце главы с числом потерянных строк. +- Default = отказ: D29.1(б) — reader-facing выдача fail-closed; дверь платформы получит `--partial` для + контрактного «may be exported, finished or not» (14 §createExport). +- `--partial`: пометка неполноты — первый абзац первого документа spine (то, что читатель видит, открыв + книгу) + `dc:description`; маркер — отдельный абзац на месте КАЖДОЙ дыры (pending/withheld — вместо текста + юнита; incomplete — перед текстом юнита, как `gapMarker`; ghost — в конце главы). Форма — из `reader.txt`; + без пака (acceptance, `runner.go:346`) — несловесная: `⚠ {chapter}.{unit}` у дыры, `⚠ {holes}/{total}` в + пометке (символ + числа, никакого словаря в Go). В `FinalText`/`export.go` маркер не попадает. +- drift — НЕ дыра: текст файла побайтово текст прогона (`export.go`, комментарий к `exportConfigDrift`); + попадает в JSON-конверт `build` и в WARN, в файл — нет («config drift» — операторский словарь). + Следствие §4 заявляю сразу: чистого бездрейфового файла на реальной книге на этом HEAD НЕТ — чистый + путь доказывается синтетикой, реальные книги — посадки на дыры. +- Оглавление и `<h1>` при `Heading==""` — номер главы цифрами (пустая ссылка nav невалидна; цифры — + единственный лейбл, который не решает открытый К-3 «какой лейбл видит читатель у главы без заголовка»; + словесный шаблон в данных — 1 строка после ратификации, вопрос владельцу в отчёте). При `Heading!=""` — + срез известного префикса `heading+"\n\n"` из `FinalText` (`chunker.go:186-192`), `<h1>` = Heading. +- Абзац = непустая строка `FinalText`. Замерено на `minirun/export.json`: юнит 1/0 — 14 разделителей `\n\n` + и 7 одиночных `\n`, юнит 2/0 — 56 и 0: редактор чередует их без семантики, различать = выдумать + различие; строка — единица абзаца исходника-вебновеллы. Круг §4 сравнивает абзацы ингеста с абзацами, + которые положил писатель. +- Метаданные: `dc:title` = `config.Book.Title` как есть (на стенде `蛊真人` — ограничение в отчёт), `dc:language` + = `TargetLang`, `dc:identifier` = `urn:textmachine:book:<book_id>`, `dc:creator` нет (источника нет). +- 0 юнитов — отказ `sourceHasNoContent` (spine без элементов невалиден); 1 юнит — обычная книга. + +**Что НЕ беру:** платформу и дверь `createExport`/`getExport`; лестницу D25.1 в коде и в `BookExport`; +строки 160/201 (`ApplyHeading` и три точки склейки не трогаю; модельные заголовки в прозе не извлекаю и не +правлю); `--resnapshot` и любые платные вызовы; `langpack_root` в копию acceptance; направление письма +(`dir`/`page-progression-direction`) — в `internal/lang` данных о направлении нет (`lang-script.txt` — +только скрипты), RTL-цели в репо нет → вопрос в отчёт, не литерал; CSS/обложку; новые зависимости. + +**Чем докажу:** герметично — `bookfile` (mimetype первым/stored, nav не в spine, детерминизм двух сборок, +экранирование, 0/1 глава), `pipeline` (круг `IngestEncoded` равенством абзацев на прогоне fake-провайдера; +pending через потолок; withheld через санитайзер; **ghost — усадка исходника после прогона (посадки в +тестах не было, строю)**; drift — бамп `prompt_version`; синтетический `BookExport` на incomplete), `cmd/tmctl` +(incomplete бинарём через `setupGapProject`; парс флагов; exit 16 через totality-тест), `lang` (парс +`reader.txt`; `Version()` не меняется от его наличия). На копиях книг в песочнице с переписанными путями: +minirun — оракул июля 14/14 по `final_text/snapshot_id/disposition`, `current_snapshot` до/после +`reader.txt`, `cmp` двух ПРОЦЕССОВ, epubcheck 5.3.0 `0 fatal / 0 error`, круг ингестом; acceptance — отказ с +перечнем и `--partial`-файл с несловесными маркерами; coldrun-a — отказ 14/14 pending; +`git -C books status --short -- gu-zhenren` до/после одинаков (снят до старта: 6 строк `D …-wal/-shm`). +`make battery` + `-race`; в середине — субагент-ревью готовой части против критериев; в конце — +адверсариальный проход по направлениям §4. + +### ✅ ИТОГ — пак «писатель книги» (промт `docs/BACKEND_BOOKWRITER_PACK_SESSION_PROMPT.md`, строка 236), сессия `textmachine-main-fc`, 30.08 + +**Что построено (зона `backend/`, поверх `67c05be`; НЕ закоммичено — дерево на лендинг оркестратору; `git status`: 10 изменённых + 8 новых путей в `backend/`, плюс этот журнал).** +- `internal/bookfile/` (новый прод-пакет, только stdlib, без `testing`): `container.go` — OCF-контейнер (`mimetype` первым, `Store`, через `CreateRaw` — без data-descriptor, флаги `0x0000`; `container.xml`; entry в порядке вызовов; нулевые zip-таймметки); `epub.go` — EPUB 3 (`dc:identifier`+`unique-identifier`, `dc:title`, `dc:language`, `dcterms:modified`, nav `properties="nav"` ⛔ не в spine, XHTML на главу: `<h1>` + `<p>`); `txt.go` — чистый текст; `model.go` — модель `Book/Chapter`, `Blocks(i)` (единственное определение содержимого документа главы), `CleanText` (общая чистка XML-недопустимых символов для обоих форматов), экранирование. +- `internal/chunk/chunktest/epub.go` переведён на `bookfile.NewContainer`; тест-декои (OPF-заглушка, `<title>c