Land backend package 4: runner decomposed to eight files, golden determinism guard, OpenReadOnly status during live runs, operator logging, tmctl unit tests
This commit is contained in:
parent
a96248f14b
commit
296419c8ae
41 changed files with 3318 additions and 1713 deletions
|
|
@ -6,13 +6,13 @@
|
||||||
|
|
||||||
| Пакет | Что делает | Ключевые файлы |
|
| Пакет | Что делает | Ключевые файлы |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `cmd/tmctl` | CLI: `translate` / `report` / `status` (read-only проекция N/M+паспорта глав+деньги, `--json`) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) | `main.go`, `internal/pipeline/status.go` |
|
| `cmd/tmctl` | CLI: `translate` / `report` / `status` (read-only проекция N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3). main — тонкая обвязка: разбор аргументов/exit-коды/.env/рендеры вынесены в тестируемые функции (пакет №4) | `main.go`, `invocation.go`, `render.go`, `dotenv.go`, `internal/pipeline/status.go` |
|
||||||
| `internal/llm` | OpenAI-совместимый транспорт + retry/backoff, **capability-слой** (budget_field/temperature/reasoning per-модель), failover (написан, НЕ подключён — D4: только local-роли), нативный Anthropic = DEPRECATED-референс под будущий Gemini | `httpllm.go`, `capability.go`, `failover.go` |
|
| `internal/llm` | OpenAI-совместимый транспорт + retry/backoff, **capability-слой** (budget_field/temperature/reasoning per-модель), failover (написан, НЕ подключён — D4: только local-роли), нативный Anthropic = DEPRECATED-референс под будущий Gemini | `httpllm.go`, `capability.go`, `failover.go` |
|
||||||
| `internal/ledger` | Цены по usage (вкл. reasoning/cache-поля), `PriceForResponse` по фактической модели | `pricing.go` |
|
| `internal/ledger` | Цены по usage (вкл. reasoning/cache-поля), `PriceForResponse` по фактической модели | `pricing.go` |
|
||||||
| `internal/store` | SQLite (modernc, CGO-free), идемпотентные миграции v1–v7, reserve/settle+checkpoint, chunk_status, глоссарий, ruby, retrieval_state (вкл. стиль-флаги v7), request_log | `ledger.go`, `migrate.go`, `glossary.go` |
|
| `internal/store` | SQLite (modernc, CGO-free), идемпотентные миграции v1–v7, reserve/settle+checkpoint, chunk_status, глоссарий, ruby, retrieval_state (вкл. стиль-флаги v7), request_log; `OpenReadOnly` — без flock/миграций/recovery для status/report | `ledger.go`, `migrate.go`, `glossary.go`, `store.go` |
|
||||||
| `internal/config` | fail-fast загрузка models/pipeline/book; **эхо-мина-гейт** `echoMineViolation`; `CheckRunnable` блокирует неисполнимое (C2/fanout/judge) | `models.go`, `pipeline.go` |
|
| `internal/config` | fail-fast загрузка models/pipeline/book; **эхо-мина-гейт** `echoMineViolation`; `CheckRunnable` блокирует неисполнимое (C2/fanout/judge) | `models.go`, `pipeline.go` |
|
||||||
| `internal/pipeline` | Раннер (циклы, disposition, single-hop эскалация с ре-гейтом, resume), чанкер v4, ingest txt/epub+ruby, classify/coverage, **банк памяти v2** (Aho-Corasick, спойлер-окна, disposition, post-check), рендер+инъекция | `runner.go`, `memory.go`, `disposition.go`, `coverage.go` |
|
| `internal/pipeline` | Раннер, декомпозированный по связным единицам (пакет №4): сетап `runner.go` → снапшот `snapshot.go` → сид `seeding.go` → цикл книги `bookrun.go` → петля чанка `chunkrun.go` → стадия `stagerun.go` → эскалация с ре-гейтом `escalation.go` → resume `resume.go`; чанкер v4, ingest txt/epub+ruby, classify/coverage, **банк памяти v2** (Aho-Corasick, спойлер-окна, disposition, post-check), рендер+инъекция. Ф2-стройки добавляются файлом: канал B → `escalation.go`, annotator/voice-инъекция → `chunkrun.go` | `bookrun.go`, `stagerun.go`, `memory.go`, `disposition.go`, `coverage.go` |
|
||||||
| `internal/obs` | trace_id, структурные логи, safego | |
|
| `internal/obs` | trace_id, структурные логи (contextHandler несёт book/chapter/chunk/stage/role из ctx), safego | |
|
||||||
|
|
||||||
## Инварианты — ЛОМАТЬ НЕЛЬЗЯ (каждый закреплён тестами)
|
## Инварианты — ЛОМАТЬ НЕЛЬЗЯ (каждый закреплён тестами)
|
||||||
|
|
||||||
|
|
@ -40,8 +40,10 @@ go run ./cmd/tmctl report --config example/book.yaml # $0, читает store
|
||||||
set -a; . ./.env; set +a; TM_LIVE=1 go test -tags live -run TestLive -v ./internal/pipeline/
|
set -a; . ./.env; set +a; TM_LIVE=1 go test -tags live -run TestLive -v ./internal/pipeline/
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Golden-гард детерминизма** (`internal/pipeline/golden_test.go` + `testdata/golden/`): на статичной фикстурной книге пинит бит-в-бит snapshotID, request_hash всех вызовов (вкл. эскалацию), wire-тела, вердикты и resume-байты. Красный golden = рефакторинг изменил wire/вердикты = `--resnapshot` = переоплата книги (D15). Обновлять ТОЛЬКО на осознанной, ратифицированной смене поведения: `TM_UPDATE_GOLDEN=1 go test ./internal/pipeline/ -run TestGolden`.
|
||||||
|
|
||||||
Финал каждой вехи — агентское адверсариальное селфревью (несколько дименсий → независимая верификация каждой находки → фиксы mutation-verified: реверт фикса валит именно его тест). Внешнее ревью — оркестратор.
|
Финал каждой вехи — агентское адверсариальное селфревью (несколько дименсий → независимая верификация каждой находки → фиксы mutation-verified: реверт фикса валит именно его тест). Внешнее ревью — оркестратор.
|
||||||
|
|
||||||
## Известный техдолг (не трогать молча — см. D-лог)
|
## Известный техдолг (не трогать молча — см. D-лог)
|
||||||
|
|
||||||
F3 at-most-once (после D15.2); Escal.Chains — валидируемый мёртвый конфиг (раннер читает только `escalate_to`, полные цепочки = шаг 7); D3-цепочка/канал B ЗАВЕДЕНЫ в `models.yaml` (пакет №3, слаги live-фактчекнуты 2026-07-10 — gemini ИМЕННО `-preview`); `escalation.budget_usd>0` требуется, чтобы single-hop `escalate_to` черновика стрелял (приёмка); min-budget Kimi≥16k/Gemini≥8k — комментарии, не схема; реализация content-addressed resume (D15.2 v3) остаётся заблокированной до ратификации оркестратором.
|
F3 at-most-once (после D15.2); Escal.Chains — валидируемый мёртвый конфиг (раннер читает только `escalate_to`, полные цепочки = шаг 7); D3-цепочка/канал B ЗАВЕДЕНЫ в `models.yaml` (пакет №3, слаги live-фактчекнуты 2026-07-10 — gemini ИМЕННО `-preview`); `escalation.budget_usd>0` требуется, чтобы single-hop `escalate_to` черновика стрелял (приёмка); min-budget Kimi≥16k/Gemini≥8k — комментарии, не схема; реализация content-addressed resume (D15.2) — после обязательных v3.1-правок (D22.2: langs→verdictSnapshotID, развязка cache_ttl).
|
||||||
|
|
|
||||||
58
backend/cmd/tmctl/dotenv.go
Normal file
58
backend/cmd/tmctl/dotenv.go
Normal file
|
|
@ -0,0 +1,58 @@
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// dotenv.go: чтение backend/.env (gitignored) в окружение. Парсер выделен чистой
|
||||||
|
// функцией (пакет №4): правила «парные кавычки» и «не перекрывать выставленное»
|
||||||
|
// — находки прошлых ревью, жившие только в комментариях; теперь они закреплены
|
||||||
|
// юнитами (регресс «упростили в strings.Trim» ломал бы значения ключей API и
|
||||||
|
// всплывал бы необъяснимыми 401 посреди приёмочного прогона).
|
||||||
|
|
||||||
|
// dotenvPair is one KEY=VALUE line of a .env file.
|
||||||
|
type dotenvPair struct{ K, V string }
|
||||||
|
|
||||||
|
// parseDotEnv reads KEY=VALUE lines: пропускает пустые/комментарии/без «=»,
|
||||||
|
// триммит ключ и значение, снимает только ПАРНЫЕ обрамляющие кавычки — Trim по
|
||||||
|
// набору символов откусил бы легитимную кавычку в конце значения (находка ревью).
|
||||||
|
func parseDotEnv(r io.Reader) []dotenvPair {
|
||||||
|
var out []dotenvPair
|
||||||
|
sc := bufio.NewScanner(r)
|
||||||
|
for sc.Scan() {
|
||||||
|
line := strings.TrimSpace(sc.Text())
|
||||||
|
if line == "" || strings.HasPrefix(line, "#") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
k, v, ok := strings.Cut(line, "=")
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
|
||||||
|
if n := len(v); n >= 2 && (v[0] == '"' || v[0] == '\'') && v[n-1] == v[0] {
|
||||||
|
v = v[1 : n-1]
|
||||||
|
}
|
||||||
|
out = append(out, dotenvPair{K: k, V: v})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// loadDotEnv reads KEY=VALUE lines into the environment without overriding
|
||||||
|
// already-set variables. Ключи — из backend/.env (gitignored); секреты никогда
|
||||||
|
// не попадают в конфиги/репо. Заметь: переменная, выставленная в ПУСТУЮ строку,
|
||||||
|
// считается невыставленной и будет перекрыта (текущее поведение, заморожено).
|
||||||
|
func loadDotEnv(path string) {
|
||||||
|
f, err := os.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer f.Close()
|
||||||
|
for _, p := range parseDotEnv(f) {
|
||||||
|
if os.Getenv(p.K) == "" {
|
||||||
|
os.Setenv(p.K, p.V)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
51
backend/cmd/tmctl/dotenv_test.go
Normal file
51
backend/cmd/tmctl/dotenv_test.go
Normal file
|
|
@ -0,0 +1,51 @@
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// dotenv_test.go pins the .env parsing rules that used to live only in comments:
|
||||||
|
// парные кавычки (регресс «упростили в strings.Trim» портил бы API-ключи и
|
||||||
|
// всплывал бы 401-ми посреди прогона) и «выставленное окружение не перекрывать».
|
||||||
|
|
||||||
|
func TestParseDotEnvPairedQuotes(t *testing.T) {
|
||||||
|
in := strings.Join([]string{
|
||||||
|
`A="v1"`, // парные двойные — снять
|
||||||
|
`B='v2'`, // парные одинарные — снять
|
||||||
|
`C="v3'`, // РАЗНЫЕ кавычки — не трогать
|
||||||
|
`D=v4"`, // хвостовая кавычка без парной — оставить (главный регресс-кейс)
|
||||||
|
`E="`, // одиночная кавычка (n<2) — оставить
|
||||||
|
`F=""`, // пустые парные — пустая строка
|
||||||
|
` G = v7 `, // трим ключа и значения
|
||||||
|
`# comment`, // комментарий — пропустить
|
||||||
|
``, // пустая — пропустить
|
||||||
|
`no-equals-line`, // без «=» — пропустить
|
||||||
|
}, "\n")
|
||||||
|
got := parseDotEnv(strings.NewReader(in))
|
||||||
|
want := []dotenvPair{
|
||||||
|
{"A", "v1"}, {"B", "v2"}, {"C", `"v3'`}, {"D", `v4"`}, {"E", `"`}, {"F", ""}, {"G", "v7"},
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("parseDotEnv:\n got %+v\n want %+v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadDotEnvDoesNotOverride(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), ".env")
|
||||||
|
if err := os.WriteFile(path, []byte("TM_TEST_SET=from_file\nTM_TEST_UNSET=filled\n"), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Setenv("TM_TEST_SET", "from_env")
|
||||||
|
t.Setenv("TM_TEST_UNSET", "") // выставленная В ПУСТОТУ считается невыставленной (заморожено)
|
||||||
|
loadDotEnv(path)
|
||||||
|
if got := os.Getenv("TM_TEST_SET"); got != "from_env" {
|
||||||
|
t.Fatalf("an already-set variable must win over .env, got %q", got)
|
||||||
|
}
|
||||||
|
if got := os.Getenv("TM_TEST_UNSET"); got != "filled" {
|
||||||
|
t.Fatalf("an empty variable is treated as unset (frozen behaviour), got %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
65
backend/cmd/tmctl/invocation.go
Normal file
65
backend/cmd/tmctl/invocation.go
Normal file
|
|
@ -0,0 +1,65 @@
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/pipeline"
|
||||||
|
)
|
||||||
|
|
||||||
|
// invocation.go: разбор аргументов CLI, выделенный в чистую функцию (пакет №4 —
|
||||||
|
// у tmctl было 0 тестов на контрактные инварианты: bad-flag→exit 1, а не 2;
|
||||||
|
// селектор redrive; порядок валидации). Контракт флагов/текстов ошибок ЗАМОРОЖЕН
|
||||||
|
// (D12/D15.3) — эта функция только переносит его в тестируемое место.
|
||||||
|
|
||||||
|
// invocation is one parsed tmctl command line.
|
||||||
|
type invocation struct {
|
||||||
|
cmd string
|
||||||
|
cfgPath string
|
||||||
|
resnapshot bool
|
||||||
|
asJSON bool
|
||||||
|
sel pipeline.RedriveSelector
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseInvocation parses os.Args[1:] into an invocation. flagOut receives the
|
||||||
|
// stdlib's flag diagnostics ("flag provided but not defined" + usage) — main
|
||||||
|
// passes os.Stderr, tests a buffer; the bytes and their destination are part of
|
||||||
|
// the frozen contract.
|
||||||
|
//
|
||||||
|
// Deliberately preserved quirks (менять = менять контракт):
|
||||||
|
// - the command NAME is not validated here — `tmctl bogus --config x` reaches
|
||||||
|
// the dispatch switch (after dotenv/ctx setup) and errors there, while
|
||||||
|
// `tmctl bogus` without --config errors "--config book.yaml is required";
|
||||||
|
// - ContinueOnError, НЕ ExitOnError: стдлибовский ExitOnError звал бы
|
||||||
|
// os.Exit(2) на плохом флаге и КОЛЛАПСИРОВАЛ бы с exit-кодом 2
|
||||||
|
// (completed-with-flags); возврат ошибки ведёт плохой флаг в default-ветку
|
||||||
|
// main() → exit 1, оставляя 2 эксклюзивным для флагнутых чанков.
|
||||||
|
func parseInvocation(args []string, flagOut io.Writer) (invocation, error) {
|
||||||
|
if len(args) < 1 {
|
||||||
|
return invocation{}, fmt.Errorf("usage: tmctl <translate|report|status|redrive> --config book.yaml")
|
||||||
|
}
|
||||||
|
cmd, rest := args[0], args[1:]
|
||||||
|
|
||||||
|
fs := flag.NewFlagSet(cmd, flag.ContinueOnError)
|
||||||
|
fs.SetOutput(flagOut)
|
||||||
|
cfgPath := fs.String("config", "", "path to book.yaml")
|
||||||
|
resnapshot := fs.Bool("resnapshot", false, "re-pin existing jobs to the current config snapshot (пере-перевод оплаченных чанков — явное согласие)")
|
||||||
|
asJSON := fs.Bool("json", false, "status: emit the projection as JSON (stable disposition/flag_reason enums) for CI/IDE")
|
||||||
|
chapter := fs.Int("chapter", -1, "redrive: restrict to this chapter (default: any)")
|
||||||
|
chunk := fs.Int("chunk", -1, "redrive: restrict to this chunk index within the chapter (default: any)")
|
||||||
|
reason := fs.String("reason", "", "redrive: restrict to this flag_reason (default: any)")
|
||||||
|
dryRun := fs.Bool("dry-run", false, "redrive: report what would be re-attacked without touching anything")
|
||||||
|
if err := fs.Parse(rest); err != nil {
|
||||||
|
return invocation{}, err
|
||||||
|
}
|
||||||
|
if *cfgPath == "" {
|
||||||
|
return invocation{}, fmt.Errorf("--config book.yaml is required")
|
||||||
|
}
|
||||||
|
return invocation{
|
||||||
|
cmd: cmd, cfgPath: *cfgPath, resnapshot: *resnapshot, asJSON: *asJSON,
|
||||||
|
sel: pipeline.RedriveSelector{
|
||||||
|
Chapter: *chapter, ChunkIdx: *chunk, Reason: *reason, DryRun: *dryRun,
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
108
backend/cmd/tmctl/invocation_test.go
Normal file
108
backend/cmd/tmctl/invocation_test.go
Normal file
|
|
@ -0,0 +1,108 @@
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/pipeline"
|
||||||
|
)
|
||||||
|
|
||||||
|
// invocation_test.go pins the frozen CLI parsing contract (пакет №4): exit-код 2
|
||||||
|
// эксклюзивен для completed-with-flags (плохой флаг → 1), порядок валидации,
|
||||||
|
// селектор redrive, приёмник диагностики стдлибовского flag.
|
||||||
|
|
||||||
|
func TestParseNoArgsUsage(t *testing.T) {
|
||||||
|
_, err := parseInvocation(nil, &bytes.Buffer{})
|
||||||
|
if err == nil || err.Error() != "usage: tmctl <translate|report|status|redrive> --config book.yaml" {
|
||||||
|
t.Fatalf("usage error text is frozen, got: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseMissingConfig(t *testing.T) {
|
||||||
|
// Порядок валидации заморожен: `tmctl bogus` без --config жалуется на config,
|
||||||
|
// НЕ на неизвестную команду (та ловится позже, в dispatch-свиче run()).
|
||||||
|
for _, args := range [][]string{{"translate"}, {"bogus"}} {
|
||||||
|
_, err := parseInvocation(args, &bytes.Buffer{})
|
||||||
|
if err == nil || err.Error() != "--config book.yaml is required" {
|
||||||
|
t.Fatalf("args %v: missing-config error text is frozen, got: %v", args, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseBadFlagIsExit1NotExit2(t *testing.T) {
|
||||||
|
var diag bytes.Buffer
|
||||||
|
_, err := parseInvocation([]string{"status", "--no-such-flag"}, &diag)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("a bad flag must be an error")
|
||||||
|
}
|
||||||
|
// The collision guard: a parse failure maps to exit 1 — NEVER to 2, which is
|
||||||
|
// exclusive to completed-with-flags (the reason for ContinueOnError).
|
||||||
|
if code := exitCode(err); code != 1 {
|
||||||
|
t.Fatalf("bad flag must exit 1, got %d", code)
|
||||||
|
}
|
||||||
|
// The stdlib's diagnostics ("flag provided but not defined" + usage) must land
|
||||||
|
// in the provided writer (main passes os.Stderr).
|
||||||
|
if !strings.Contains(diag.String(), "flag provided but not defined") {
|
||||||
|
t.Fatalf("flag diagnostics must go to the provided writer, got: %q", diag.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseUnknownCommandPassesThrough(t *testing.T) {
|
||||||
|
inv, err := parseInvocation([]string{"bogus", "--config", "b.yaml"}, &bytes.Buffer{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("command name is validated in the dispatch switch, not the parser: %v", err)
|
||||||
|
}
|
||||||
|
if inv.cmd != "bogus" || inv.cfgPath != "b.yaml" {
|
||||||
|
t.Fatalf("inv = %+v", inv)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseRedriveSelectorDefaults(t *testing.T) {
|
||||||
|
inv, err := parseInvocation([]string{"redrive", "--config", "b.yaml"}, &bytes.Buffer{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
want := pipeline.RedriveSelector{Chapter: -1, ChunkIdx: -1, Reason: "", DryRun: false}
|
||||||
|
if inv.sel != want {
|
||||||
|
t.Fatalf("default selector must be «any» (-1/-1/\"\"/false), got %+v", inv.sel)
|
||||||
|
}
|
||||||
|
if inv.resnapshot {
|
||||||
|
t.Fatal("resnapshot must default to false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseRedriveSelectorExplicit(t *testing.T) {
|
||||||
|
inv, err := parseInvocation([]string{"redrive", "--config", "b.yaml",
|
||||||
|
"--chapter", "3", "--chunk", "7", "--reason", "soft_refusal", "--dry-run", "--resnapshot"}, &bytes.Buffer{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
want := pipeline.RedriveSelector{Chapter: 3, ChunkIdx: 7, Reason: "soft_refusal", DryRun: true}
|
||||||
|
if inv.sel != want {
|
||||||
|
t.Fatalf("selector wiring lost a flag (класс регресса D20.4 «parsed but ignored»): %+v", inv.sel)
|
||||||
|
}
|
||||||
|
// D20.4 regression class: --resnapshot was once parsed but never wired.
|
||||||
|
if !inv.resnapshot {
|
||||||
|
t.Fatal("--resnapshot must reach the invocation")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExitCodeContract(t *testing.T) {
|
||||||
|
if exitCode(nil) != 0 {
|
||||||
|
t.Fatal("nil → 0")
|
||||||
|
}
|
||||||
|
if exitCode(errors.New("boom")) != 1 {
|
||||||
|
t.Fatal("generic error → 1")
|
||||||
|
}
|
||||||
|
sentinel := &pipeline.CompletedWithFlags{Flagged: 2, Total: 5}
|
||||||
|
if exitCode(sentinel) != 2 {
|
||||||
|
t.Fatal("sentinel → 2")
|
||||||
|
}
|
||||||
|
// The sentinel must survive %w-wrapping (errors.As, not a type switch).
|
||||||
|
if exitCode(fmt.Errorf("outer: %w", sentinel)) != 2 {
|
||||||
|
t.Fatal("wrapped sentinel → 2")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -1,68 +1,55 @@
|
||||||
// tmctl is the TextMachine CLI (Фаза 0: translate один чанк + report).
|
// tmctl is the TextMachine CLI: translate / report / status / redrive.
|
||||||
|
// main.go — тонкая обвязка (пакет №4): разбор аргументов — invocation.go,
|
||||||
|
// рендеры вывода — render.go, .env — dotenv.go; здесь только связка
|
||||||
|
// «parse → env → ctx → fetch → render» и маппинг exit-кодов.
|
||||||
package main
|
package main
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"bufio"
|
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
|
||||||
"errors"
|
"errors"
|
||||||
"flag"
|
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"os/signal"
|
"os/signal"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
|
||||||
"syscall"
|
"syscall"
|
||||||
|
|
||||||
"textmachine/backend/internal/obs"
|
"textmachine/backend/internal/obs"
|
||||||
"textmachine/backend/internal/pipeline"
|
"textmachine/backend/internal/pipeline"
|
||||||
|
"textmachine/backend/internal/store"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Exit codes (Веха 2): 0 clean · 2 completed-with-flags (приёмка допускает N
|
|
||||||
// флагов) · 1 infra failure. The exit-2 case is carried up as a typed sentinel
|
|
||||||
// (*pipeline.CompletedWithFlags) so a "clean run, attention needed" never
|
|
||||||
// masquerades as an infra crash and vice-versa.
|
|
||||||
func main() {
|
func main() {
|
||||||
err := run()
|
err := run()
|
||||||
|
if err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, "tmctl:", err)
|
||||||
|
}
|
||||||
|
os.Exit(exitCode(err))
|
||||||
|
}
|
||||||
|
|
||||||
|
// exitCode maps a run() error onto the ratified shell contract (Веха 2): 0 clean ·
|
||||||
|
// 2 completed-with-flags (приёмка допускает N флагов; типизированный сентинел
|
||||||
|
// *pipeline.CompletedWithFlags, переживает %w-обёртки через errors.As) · 1 infra
|
||||||
|
// failure и всё остальное, включая ошибку разбора флагов (ContinueOnError в
|
||||||
|
// parseInvocation держит 2 эксклюзивным для флагнутых чанков).
|
||||||
|
func exitCode(err error) int {
|
||||||
var flagged *pipeline.CompletedWithFlags
|
var flagged *pipeline.CompletedWithFlags
|
||||||
switch {
|
switch {
|
||||||
case err == nil:
|
case err == nil:
|
||||||
return
|
return 0
|
||||||
case errors.As(err, &flagged):
|
case errors.As(err, &flagged):
|
||||||
fmt.Fprintln(os.Stderr, "tmctl:", err)
|
return 2
|
||||||
os.Exit(2)
|
|
||||||
default:
|
default:
|
||||||
fmt.Fprintln(os.Stderr, "tmctl:", err)
|
return 1
|
||||||
os.Exit(1)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func run() error {
|
func run() error {
|
||||||
if len(os.Args) < 2 {
|
inv, err := parseInvocation(os.Args[1:], os.Stderr)
|
||||||
return fmt.Errorf("usage: tmctl <translate|report|status|redrive> --config book.yaml")
|
if err != nil {
|
||||||
}
|
|
||||||
cmd, args := os.Args[1], os.Args[2:]
|
|
||||||
|
|
||||||
// ContinueOnError (not ExitOnError): the stdlib's ExitOnError calls os.Exit(2)
|
|
||||||
// on a parse failure, which would COLLIDE with exit code 2 (completed-with-
|
|
||||||
// flags). Returning the error routes a bad flag to main()'s default branch →
|
|
||||||
// exit 1, keeping 2 exclusive to the flagged-chunks case (self-review Веха 2).
|
|
||||||
fs := flag.NewFlagSet(cmd, flag.ContinueOnError)
|
|
||||||
cfgPath := fs.String("config", "", "path to book.yaml")
|
|
||||||
resnapshot := fs.Bool("resnapshot", false, "re-pin existing jobs to the current config snapshot (пере-перевод оплаченных чанков — явное согласие)")
|
|
||||||
asJSON := fs.Bool("json", false, "status: emit the projection as JSON (stable disposition/flag_reason enums) for CI/IDE")
|
|
||||||
chapter := fs.Int("chapter", -1, "redrive: restrict to this chapter (default: any)")
|
|
||||||
chunk := fs.Int("chunk", -1, "redrive: restrict to this chunk index within the chapter (default: any)")
|
|
||||||
reason := fs.String("reason", "", "redrive: restrict to this flag_reason (default: any)")
|
|
||||||
dryRun := fs.Bool("dry-run", false, "redrive: report what would be re-attacked without touching anything")
|
|
||||||
if err := fs.Parse(args); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if *cfgPath == "" {
|
|
||||||
return fmt.Errorf("--config book.yaml is required")
|
|
||||||
}
|
|
||||||
|
|
||||||
loadDotEnv(filepath.Join(filepath.Dir(*cfgPath), ".env"))
|
loadDotEnv(filepath.Join(filepath.Dir(inv.cfgPath), ".env"))
|
||||||
loadDotEnv(".env")
|
loadDotEnv(".env")
|
||||||
|
|
||||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||||
|
|
@ -74,19 +61,17 @@ func run() error {
|
||||||
LogBodies: os.Getenv("LOG_LLM_BODIES") == "1",
|
LogBodies: os.Getenv("LOG_LLM_BODIES") == "1",
|
||||||
})
|
})
|
||||||
|
|
||||||
switch cmd {
|
switch inv.cmd {
|
||||||
case "translate":
|
case "translate":
|
||||||
return translate(ctx, *cfgPath, *resnapshot)
|
return translate(ctx, inv.cfgPath, inv.resnapshot)
|
||||||
case "report":
|
case "report":
|
||||||
return report(*cfgPath)
|
return report(inv.cfgPath)
|
||||||
case "status":
|
case "status":
|
||||||
return status(ctx, *cfgPath, *asJSON)
|
return status(ctx, inv.cfgPath, inv.asJSON)
|
||||||
case "redrive":
|
case "redrive":
|
||||||
return redrive(ctx, *cfgPath, *resnapshot, pipeline.RedriveSelector{
|
return redrive(ctx, inv.cfgPath, inv.resnapshot, inv.sel)
|
||||||
Chapter: *chapter, ChunkIdx: *chunk, Reason: *reason, DryRun: *dryRun,
|
|
||||||
})
|
|
||||||
default:
|
default:
|
||||||
return fmt.Errorf("unknown command %q (want translate|report|status|redrive)", cmd)
|
return fmt.Errorf("unknown command %q (want translate|report|status|redrive)", inv.cmd)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -102,52 +87,9 @@ func translate(ctx context.Context, cfgPath string, resnapshot bool) error {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
return renderTranslate(os.Stdout, res, func() (float64, float64, error) {
|
||||||
for _, ch := range res.Chunks {
|
return r.Store.SpentUSD(res.BookID)
|
||||||
fmt.Printf("=== ГЛАВА %d ЧАНК %d — %s%s ===\n", ch.Chapter, ch.ChunkIdx, ch.Disposition, flagSuffix(ch.FlagReason))
|
})
|
||||||
if ch.Disposition == pipeline.DispOK {
|
|
||||||
fmt.Println(ch.FinalText)
|
|
||||||
} else {
|
|
||||||
fmt.Printf("[ФЛАГ %s] чанк не переведён — черновик/редактура непригодны, помечен для человека\n", ch.FlagReason)
|
|
||||||
}
|
|
||||||
for _, st := range ch.Stages {
|
|
||||||
how := "call"
|
|
||||||
switch {
|
|
||||||
case st.Disposition == pipeline.DispSkipped:
|
|
||||||
how = "skipped"
|
|
||||||
case st.FromResume:
|
|
||||||
how = "resume"
|
|
||||||
}
|
|
||||||
fmt.Printf(" %-8s %-22s %-8s %-8s%s $%.6f (cum $%.6f) in=%d (cached=%d) out=%d+%d att=%d %dms finish=%s\n",
|
|
||||||
st.Stage, st.Model, how, st.Disposition, flagSuffix(st.FlagReason),
|
|
||||||
st.CostUSD, st.CumCostUSD,
|
|
||||||
st.Usage.PromptTokens, st.Usage.CachedTokens,
|
|
||||||
st.Usage.CompletionTokens, st.Usage.ReasoningTokens, st.Attempts, st.LatencyMS, st.FinishReason)
|
|
||||||
}
|
|
||||||
fmt.Println()
|
|
||||||
}
|
|
||||||
|
|
||||||
fmt.Printf("ИТОГО (этот запуск): $%.6f — чанков %d, флагов %d\n", res.TotalUSD, len(res.Chunks), res.Flagged)
|
|
||||||
committed, reserved, err := r.Store.SpentUSD(res.BookID)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
fmt.Printf("Ledger книги: committed=$%.6f reserved=$%.6f\n", committed, reserved)
|
|
||||||
|
|
||||||
// Exit code 2 is a typed sentinel, not an infra error: the report above is
|
|
||||||
// already on stdout; main() maps this to a non-zero exit for the operator.
|
|
||||||
if res.Flagged > 0 {
|
|
||||||
return &pipeline.CompletedWithFlags{Flagged: res.Flagged, Total: len(res.Chunks)}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// flagSuffix renders a non-empty flag reason as "(reason)".
|
|
||||||
func flagSuffix(r pipeline.FlagReason) string {
|
|
||||||
if r == "" {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
return "(" + string(r) + ")"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func report(cfgPath string) error {
|
func report(cfgPath string) error {
|
||||||
|
|
@ -158,95 +100,13 @@ func report(cfgPath string) error {
|
||||||
}
|
}
|
||||||
defer r.Close()
|
defer r.Close()
|
||||||
|
|
||||||
rows, err := r.Store.RequestLogRows(r.Book.BookID)
|
// Чтения — коллбеками (interleaved с печатью, как исторически): провал чтения
|
||||||
if err != nil {
|
// посреди аудита оставляет уже напечатанную часть на stdout (см. renderReport).
|
||||||
return err
|
return renderReport(os.Stdout,
|
||||||
}
|
func() ([]store.RequestLogView, error) { return r.Store.RequestLogRows(r.Book.BookID) },
|
||||||
|
func() ([]store.ChunkStatus, error) { return r.Store.ChunkStatusesForBook(r.Book.BookID) },
|
||||||
fmt.Printf("%-20s %-8s %-12s %-22s %8s %8s %8s %8s %8s %10s %8s %-8s %-5s %-3s\n",
|
func() ([]store.RetrievalState, error) { return r.Store.RetrievalStatesForBook(r.Book.BookID) },
|
||||||
"ts", "stage", "role", "model", "prompt", "cached", "cwrite", "compl", "reason", "cost_usd", "ms", "finish", "tmhit", "ok")
|
func() (float64, float64, error) { return r.Store.SpentUSD(r.Book.BookID) })
|
||||||
for _, row := range rows {
|
|
||||||
fmt.Printf("%-20s %-8s %-12s %-22s %8d %8d %8d %8d %8d %10.6f %8d %-8s %-5d %-3d\n",
|
|
||||||
row.TS, row.Stage, row.Role, row.ModelActual, row.PromptTokens, row.CachedTokens,
|
|
||||||
row.CacheCreationTokens, row.CompletionTokens, row.ReasoningTokens, row.CostUSD,
|
|
||||||
row.LatencyMS, row.FinishReason, row.TMHit, row.OK)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Flag section (Веха 2): every chunk×stage whose disposition ≠ ok — the
|
|
||||||
// "флаг редактору" the plan requires (02-mvp Фаза-1 приёмка допускает N).
|
|
||||||
flags, err := r.Store.ChunkStatusesForBook(r.Book.BookID)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
headerPrinted := false
|
|
||||||
for _, f := range flags {
|
|
||||||
if f.Disposition == "ok" {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if !headerPrinted {
|
|
||||||
fmt.Printf("\n=== ФЛАГИ (disposition ≠ ok) ===\n")
|
|
||||||
fmt.Printf("%-4s %-6s %-8s %-9s %-16s %5s %10s %s\n",
|
|
||||||
"ch", "chunk", "stage", "disp", "reason", "att", "cost_usd", "detail")
|
|
||||||
headerPrinted = true
|
|
||||||
}
|
|
||||||
fmt.Printf("%-4d %-6d %-8s %-9s %-16s %5d %10.6f %s\n",
|
|
||||||
f.Chapter, f.ChunkIdx, f.Stage, f.Disposition, f.FlagReason, f.Attempts, f.CostUSD, f.Detail)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Memory section (шаг 4): the per-chunk retrieval-state, surfaced so silent glossary
|
|
||||||
// degradation is LOUD. The aggregate line always prints when a glossary is in use;
|
|
||||||
// each chunk with a post-check miss is listed (the flagger-mode signal — the model
|
|
||||||
// ignored an approved term, or we injected the wrong dst) with the offending src→dst.
|
|
||||||
states, err := r.Store.RetrievalStatesForBook(r.Book.BookID)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if len(states) > 0 {
|
|
||||||
var injected, sticky, ambiguous, spoiler, evicted, misses, style int
|
|
||||||
for _, rs := range states {
|
|
||||||
injected += rs.NExactHits
|
|
||||||
sticky += rs.NSticky
|
|
||||||
ambiguous += rs.NAmbiguousFlagged
|
|
||||||
spoiler += rs.NSpoilerBlocked
|
|
||||||
evicted += rs.NEvicted
|
|
||||||
misses += rs.NPostcheckMiss
|
|
||||||
style += rs.NStyleFlags
|
|
||||||
}
|
|
||||||
fmt.Printf("\n=== ПАМЯТЬ (retrieval-state) ===\n")
|
|
||||||
fmt.Printf("инъекций(точных)=%d sticky=%d ambiguous=%d спойлер-блок=%d вытеснено=%d post-check-промахов=%d\n",
|
|
||||||
injected, sticky, ambiguous, spoiler, evicted, misses)
|
|
||||||
printed := false
|
|
||||||
for _, rs := range states {
|
|
||||||
if rs.NPostcheckMiss == 0 {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if !printed {
|
|
||||||
fmt.Printf("%-4s %-6s %6s %s\n", "ch", "chunk", "misses", "detail (src→dst не найден в выводе)")
|
|
||||||
printed = true
|
|
||||||
}
|
|
||||||
fmt.Printf("%-4d %-6d %6d %s\n", rs.Chapter, rs.ChunkIdx, rs.NPostcheckMiss, rs.PostcheckDetail)
|
|
||||||
}
|
|
||||||
// Cheap style/number flaggers (observability, not gates): total + per-chunk detail.
|
|
||||||
fmt.Printf("\n=== СТИЛЬ-ГЕЙТЫ (наблюдаемость: тире-диалоги, ё, транслит-междометия, разряды 万/億) — всего %d ===\n", style)
|
|
||||||
stylePrinted := false
|
|
||||||
for _, rs := range states {
|
|
||||||
if rs.NStyleFlags == 0 {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if !stylePrinted {
|
|
||||||
fmt.Printf("%-4s %-6s %6s %s\n", "ch", "chunk", "flags", "detail")
|
|
||||||
stylePrinted = true
|
|
||||||
}
|
|
||||||
fmt.Printf("%-4d %-6d %6d %s\n", rs.Chapter, rs.ChunkIdx, rs.NStyleFlags, rs.StyleDetail)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
committed, reserved, err := r.Store.SpentUSD(r.Book.BookID)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
fmt.Printf("\nLedger книги: committed=$%.6f reserved=$%.6f\n", committed, reserved)
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// status prints the READ-ONLY progress projection (D15.3): 0 LLM, 0 replay. --json emits the
|
// status prints the READ-ONLY progress projection (D15.3): 0 LLM, 0 replay. --json emits the
|
||||||
|
|
@ -266,86 +126,10 @@ func status(ctx context.Context, cfgPath string, asJSON bool) error {
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
if asJSON {
|
if asJSON {
|
||||||
enc := json.NewEncoder(os.Stdout)
|
return renderStatusJSON(os.Stdout, rep)
|
||||||
enc.SetIndent("", " ")
|
|
||||||
if err := enc.Encode(rep); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
// Symmetric with the human mode (minor 1d): flagged>0 is exit 2 (completed-with-flags), not
|
|
||||||
// exit 0. The full JSON is already on stdout; the sentinel only sets the shell disposition
|
|
||||||
// (its message goes to stderr in main), so a CI/IDE consumer both parses the projection AND
|
|
||||||
// sees "attention needed" in the exit code instead of silently reading 0.
|
|
||||||
if rep.Flagged > 0 {
|
|
||||||
return &pipeline.CompletedWithFlags{Flagged: rep.Flagged, Total: rep.TotalChunks}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
|
return renderStatusHuman(os.Stdout, rep, cfgPath)
|
||||||
snap := rep.Snapshot
|
|
||||||
if snap == "" {
|
|
||||||
snap = "—"
|
|
||||||
} else if len(snap) > 12 {
|
|
||||||
snap = snap[:12]
|
|
||||||
}
|
|
||||||
drift := ""
|
|
||||||
if rep.SnapshotDrift {
|
|
||||||
drift += " ⚠ SNAPSHOT-DRIFT (несколько снапшотов в chunk_status — конфиг менялся; --resnapshot)"
|
|
||||||
}
|
|
||||||
if rep.ConfigDrift {
|
|
||||||
drift += " ⚠ CONFIG-DRIFT (текущий конфиг рендерит другой снапшот — правка после прогона; translate потребует --resnapshot = переоплата книги)"
|
|
||||||
}
|
|
||||||
fmt.Printf("=== СТАТУС: %s (snapshot %s)%s ===\n", rep.BookID, snap, drift)
|
|
||||||
fmt.Printf("Прогресс: %d/%d чанков (%.1f%%) — done=%d in_progress=%d flagged=%d pending=%d\n",
|
|
||||||
rep.Done, rep.TotalChunks, rep.PercentDone, rep.Done, rep.InProgress, rep.Flagged, rep.Pending)
|
|
||||||
fmt.Printf("Эскалаций: %d · post-check промахов (confirmed): %d · стиль-флагов (наблюдаемость): %d\n",
|
|
||||||
rep.Escalations, rep.PostcheckMisses, rep.StyleFlags)
|
|
||||||
ceil := ""
|
|
||||||
if rep.BookCeilingUSD > 0 {
|
|
||||||
ceil = fmt.Sprintf(" (потолок $%.2f — %.1f%%)", rep.BookCeilingUSD, rep.CeilingPct)
|
|
||||||
}
|
|
||||||
fmt.Printf("Деньги: committed=$%.6f reserved=$%.6f%s · прогноз книги ~$%.6f\n",
|
|
||||||
rep.CommittedUSD, rep.ReservedUSD, ceil, rep.ProjectedBookUSD)
|
|
||||||
if rep.ETASeconds > 0 {
|
|
||||||
fmt.Printf("ETA: ~%.0fс (средний throughput fresh-вызовов, НЕ EWMA — D12-отступление; вторично)\n", rep.ETASeconds)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(rep.Chapters) > 0 {
|
|
||||||
// style = cheap deterministic style/number flags (observability, not a disposition) — a per-
|
|
||||||
// chapter glance count so a human sees where the linters fired without opening `report` (D20.4).
|
|
||||||
fmt.Printf("\n%-5s %-9s %-10s %-6s %-4s %-6s %-16s %10s\n", "ch", "done/all", "verdict", "flag", "esc", "style", "worst_flag", "cost_usd")
|
|
||||||
for _, p := range rep.Chapters {
|
|
||||||
fmt.Printf("%-5d %d/%-7d %-10s %-6d %-4d %-6d %-16s %10.6f\n",
|
|
||||||
p.Chapter, p.ChunksDone, p.ChunksTotal, p.Verdict, p.ChunksFlagged, p.Escalations, p.StyleFlags,
|
|
||||||
dashIfEmpty(p.WorstFlagReason), p.CostUSD)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
// flagged≠failed (jobs.status.failed = infra-only, D12): flagged chunks are a "clean run,
|
|
||||||
// attention needed" signal. Two kinds need DIFFERENT actions (minor 1d): a chunk_status flag
|
|
||||||
// (soft_refusal, length, …) is re-drivable with `tmctl redrive`; a gate-promoted glossary_miss
|
|
||||||
// is NOT (redrive is a no-op — the miss re-derives from the unchanged glossary), so it needs a
|
|
||||||
// seed fix + `translate --resnapshot`. Advise each set separately instead of one blanket hint.
|
|
||||||
if rep.Flagged > 0 {
|
|
||||||
if redrivable := rep.Flagged - rep.GlossaryMissFlagged; redrivable > 0 {
|
|
||||||
fmt.Printf("\n%d флагнутых чанк(ов) ждут внимания — переатаковать: tmctl redrive --config %s [--chapter N --chunk M --reason R]\n",
|
|
||||||
redrivable, cfgPath)
|
|
||||||
}
|
|
||||||
if rep.GlossaryMissFlagged > 0 {
|
|
||||||
fmt.Printf("\n%d чанк(ов) флагнуты post-check-гейтом (glossary_miss) — redrive для них no-op: исправьте сид-глоссарий (термин/dst) и перезапустите `tmctl translate --config %s --resnapshot` (переоплата затронутых чанков)\n",
|
|
||||||
rep.GlossaryMissFlagged, cfgPath)
|
|
||||||
}
|
|
||||||
return &pipeline.CompletedWithFlags{Flagged: rep.Flagged, Total: rep.TotalChunks}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// dashIfEmpty renders "" as "—" for a table cell.
|
|
||||||
func dashIfEmpty(s string) string {
|
|
||||||
if s == "" {
|
|
||||||
return "—"
|
|
||||||
}
|
|
||||||
return s
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// redrive re-attacks the FLAGGED chunks matching the selector (D15.3): it resets their terminal
|
// redrive re-attacks the FLAGGED chunks matching the selector (D15.3): it resets their terminal
|
||||||
|
|
@ -368,63 +152,7 @@ func redrive(ctx context.Context, cfgPath string, resnapshot bool, sel pipeline.
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
return renderRedrive(os.Stdout, r.Book.BookID, summary, res, func() (float64, float64, error) {
|
||||||
fmt.Printf("=== REDRIVE: %s ===\n", r.Book.BookID)
|
return r.Store.SpentUSD(r.Book.BookID)
|
||||||
if len(summary.Targets) == 0 {
|
})
|
||||||
fmt.Println("Флагнутых чанков под селектор не найдено — нечего переатаковать.")
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
fmt.Printf("Целей: %d флагнутых чанк(ов) — сброс терминального флага + свежий retry/escalation-бюджет (DispOK не тронут):\n", len(summary.Targets))
|
|
||||||
for _, t := range summary.Targets {
|
|
||||||
fmt.Printf(" ch%d/chunk%d (%s) → стадии: %s\n", t.Chapter, t.ChunkIdx, t.FlagReason, strings.Join(t.Stages, ", "))
|
|
||||||
}
|
|
||||||
if summary.DryRun {
|
|
||||||
fmt.Println("[dry-run: ничего не сброшено, вызовов не сделано]")
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
fmt.Println("[сброшено; перезапуск durable-цикла — DispOK чанки резюмируются за $0, сброшенные переатакуются]")
|
|
||||||
fmt.Println()
|
|
||||||
|
|
||||||
if res != nil {
|
|
||||||
fmt.Printf("ИТОГО re-run (этот запуск): $%.6f — чанков %d, флагов %d\n", res.TotalUSD, len(res.Chunks), res.Flagged)
|
|
||||||
committed, reserved, serr := r.Store.SpentUSD(res.BookID)
|
|
||||||
if serr == nil {
|
|
||||||
fmt.Printf("Ledger книги: committed=$%.6f reserved=$%.6f\n", committed, reserved)
|
|
||||||
}
|
|
||||||
if res.Flagged > 0 {
|
|
||||||
return &pipeline.CompletedWithFlags{Flagged: res.Flagged, Total: len(res.Chunks)}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// loadDotEnv reads KEY=VALUE lines into the environment without overriding
|
|
||||||
// already-set variables. Ключи — из backend/.env (gitignored); секреты никогда
|
|
||||||
// не попадают в конфиги/репо.
|
|
||||||
func loadDotEnv(path string) {
|
|
||||||
f, err := os.Open(path)
|
|
||||||
if err != nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
defer f.Close()
|
|
||||||
sc := bufio.NewScanner(f)
|
|
||||||
for sc.Scan() {
|
|
||||||
line := strings.TrimSpace(sc.Text())
|
|
||||||
if line == "" || strings.HasPrefix(line, "#") {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
k, v, ok := strings.Cut(line, "=")
|
|
||||||
if !ok {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
k, v = strings.TrimSpace(k), strings.TrimSpace(v)
|
|
||||||
// Снимаем только ПАРНЫЕ обрамляющие кавычки: Trim по набору символов
|
|
||||||
// откусил бы легитимную кавычку в конце ключа (находка ревью).
|
|
||||||
if n := len(v); n >= 2 && (v[0] == '"' || v[0] == '\'') && v[n-1] == v[0] {
|
|
||||||
v = v[1 : n-1]
|
|
||||||
}
|
|
||||||
if os.Getenv(k) == "" {
|
|
||||||
os.Setenv(k, v)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
324
backend/cmd/tmctl/render.go
Normal file
324
backend/cmd/tmctl/render.go
Normal file
|
|
@ -0,0 +1,324 @@
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/pipeline"
|
||||||
|
"textmachine/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// render.go: рендеры человеческого/JSON-вывода команд, выделенные из main.go в
|
||||||
|
// чистые функции над структурами pipeline/store (пакет №4). Все байты вывода —
|
||||||
|
// ЗАМОРОЖЕННЫЙ контракт (стабильные enum в --json — D12; человеческие таблицы —
|
||||||
|
// поверхность аудита D20.4): fmt.Printf заменён на fmt.Fprintf(w, …) с
|
||||||
|
// w=os.Stdout — байт-в-байт; каждое сознательное изменение перечислено в
|
||||||
|
// PROGRESS (в этом пакете одно: report несёт ch/chunk/model_requested/err —
|
||||||
|
// пост-мортем упавшего вызова был «пустой строкой», боль smoke-прогона).
|
||||||
|
// ledger-коллбеки сохраняют ТОЧНЫЙ порядок «печать → чтение SpentUSD → печать»
|
||||||
|
// исходного кода: подъём чтения до рендера менял бы частичный вывод на ошибке.
|
||||||
|
|
||||||
|
// renderTranslate prints the per-chunk translation report and returns the
|
||||||
|
// CompletedWithFlags sentinel when chunks were flagged (exit 2).
|
||||||
|
func renderTranslate(w io.Writer, res *pipeline.BookResult, ledger func() (committed, reserved float64, err error)) error {
|
||||||
|
for _, ch := range res.Chunks {
|
||||||
|
fmt.Fprintf(w, "=== ГЛАВА %d ЧАНК %d — %s%s ===\n", ch.Chapter, ch.ChunkIdx, ch.Disposition, flagSuffix(ch.FlagReason))
|
||||||
|
if ch.Disposition == pipeline.DispOK {
|
||||||
|
fmt.Fprintln(w, ch.FinalText)
|
||||||
|
} else {
|
||||||
|
fmt.Fprintf(w, "[ФЛАГ %s] чанк не переведён — черновик/редактура непригодны, помечен для человека\n", ch.FlagReason)
|
||||||
|
}
|
||||||
|
for _, st := range ch.Stages {
|
||||||
|
how := "call"
|
||||||
|
switch {
|
||||||
|
case st.Disposition == pipeline.DispSkipped:
|
||||||
|
how = "skipped"
|
||||||
|
case st.FromResume:
|
||||||
|
how = "resume"
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, " %-8s %-22s %-8s %-8s%s $%.6f (cum $%.6f) in=%d (cached=%d) out=%d+%d att=%d %dms finish=%s\n",
|
||||||
|
st.Stage, st.Model, how, st.Disposition, flagSuffix(st.FlagReason),
|
||||||
|
st.CostUSD, st.CumCostUSD,
|
||||||
|
st.Usage.PromptTokens, st.Usage.CachedTokens,
|
||||||
|
st.Usage.CompletionTokens, st.Usage.ReasoningTokens, st.Attempts, st.LatencyMS, st.FinishReason)
|
||||||
|
}
|
||||||
|
fmt.Fprintln(w)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Fprintf(w, "ИТОГО (этот запуск): $%.6f — чанков %d, флагов %d\n", res.TotalUSD, len(res.Chunks), res.Flagged)
|
||||||
|
committed, reserved, err := ledger()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "Ledger книги: committed=$%.6f reserved=$%.6f\n", committed, reserved)
|
||||||
|
|
||||||
|
// Exit code 2 is a typed sentinel, not an infra error: the report above is
|
||||||
|
// already on stdout; main() maps this to a non-zero exit for the operator.
|
||||||
|
if res.Flagged > 0 {
|
||||||
|
return &pipeline.CompletedWithFlags{Flagged: res.Flagged, Total: len(res.Chunks)}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// flagSuffix renders a non-empty flag reason as "(reason)".
|
||||||
|
func flagSuffix(r pipeline.FlagReason) string {
|
||||||
|
if r == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return "(" + string(r) + ")"
|
||||||
|
}
|
||||||
|
|
||||||
|
// renderReport prints the request_log table, the flag/memory/style sections and
|
||||||
|
// the ledger line. Осознанное изменение пакета №4: колонки ch/chunk, модель с
|
||||||
|
// фолбэком на ЗАПРОШЕННУЮ (у упавшего вызова model_actual пуст — строка была
|
||||||
|
// анонимной) и хвост degraded/err — БД эти колонки хранит с Вехи 2, принтер их
|
||||||
|
// терял, и пост-мортем требовал лезть в sqlite3.
|
||||||
|
//
|
||||||
|
// Все три чтения store — КОЛЛБЕКАМИ, как ledger: исторический report читал и
|
||||||
|
// печатал вперемешку, и провал чтения посреди аудита оставлял на stdout уже
|
||||||
|
// напечатанную часть; подъём чтений до рендера молча менял бы этот частичный
|
||||||
|
// вывод на пустой (находка селфревью — тот же принцип, что задокументирован
|
||||||
|
// ниже для ledger-строки).
|
||||||
|
func renderReport(w io.Writer,
|
||||||
|
fetchRows func() ([]store.RequestLogView, error),
|
||||||
|
fetchFlags func() ([]store.ChunkStatus, error),
|
||||||
|
fetchStates func() ([]store.RetrievalState, error),
|
||||||
|
ledger func() (committed, reserved float64, err error)) error {
|
||||||
|
rows, err := fetchRows()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "%-20s %-4s %-5s %-8s %-12s %-22s %8s %8s %8s %8s %8s %10s %8s %-8s %-5s %-3s %s\n",
|
||||||
|
"ts", "ch", "chunk", "stage", "role", "model", "prompt", "cached", "cwrite", "compl", "reason", "cost_usd", "ms", "finish", "tmhit", "ok", "err")
|
||||||
|
for _, row := range rows {
|
||||||
|
model := row.ModelActual
|
||||||
|
if model == "" && row.ModelRequested != "" {
|
||||||
|
model = row.ModelRequested + "(req)"
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "%-20s %-4d %-5d %-8s %-12s %-22s %8d %8d %8d %8d %8d %10.6f %8d %-8s %-5d %-3d %s\n",
|
||||||
|
row.TS, row.Chapter, row.ChunkIdx, row.Stage, row.Role, model, row.PromptTokens, row.CachedTokens,
|
||||||
|
row.CacheCreationTokens, row.CompletionTokens, row.ReasoningTokens, row.CostUSD,
|
||||||
|
row.LatencyMS, row.FinishReason, row.TMHit, row.OK, errTail(row.Degraded, row.Err))
|
||||||
|
}
|
||||||
|
|
||||||
|
// Flag section (Веха 2): every chunk×stage whose disposition ≠ ok — the
|
||||||
|
// "флаг редактору" the plan requires (02-mvp Фаза-1 приёмка допускает N).
|
||||||
|
flags, err := fetchFlags()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
headerPrinted := false
|
||||||
|
for _, f := range flags {
|
||||||
|
if f.Disposition == "ok" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !headerPrinted {
|
||||||
|
fmt.Fprintf(w, "\n=== ФЛАГИ (disposition ≠ ok) ===\n")
|
||||||
|
fmt.Fprintf(w, "%-4s %-6s %-8s %-9s %-16s %5s %10s %s\n",
|
||||||
|
"ch", "chunk", "stage", "disp", "reason", "att", "cost_usd", "detail")
|
||||||
|
headerPrinted = true
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "%-4d %-6d %-8s %-9s %-16s %5d %10.6f %s\n",
|
||||||
|
f.Chapter, f.ChunkIdx, f.Stage, f.Disposition, f.FlagReason, f.Attempts, f.CostUSD, f.Detail)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Memory section (шаг 4): the per-chunk retrieval-state, surfaced so silent glossary
|
||||||
|
// degradation is LOUD. The aggregate line always prints when a glossary is in use;
|
||||||
|
// each chunk with a post-check miss is listed (the flagger-mode signal — the model
|
||||||
|
// ignored an approved term, or we injected the wrong dst) with the offending src→dst.
|
||||||
|
states, err := fetchStates()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if len(states) > 0 {
|
||||||
|
var injected, sticky, ambiguous, spoiler, evicted, misses, style int
|
||||||
|
for _, rs := range states {
|
||||||
|
injected += rs.NExactHits
|
||||||
|
sticky += rs.NSticky
|
||||||
|
ambiguous += rs.NAmbiguousFlagged
|
||||||
|
spoiler += rs.NSpoilerBlocked
|
||||||
|
evicted += rs.NEvicted
|
||||||
|
misses += rs.NPostcheckMiss
|
||||||
|
style += rs.NStyleFlags
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "\n=== ПАМЯТЬ (retrieval-state) ===\n")
|
||||||
|
fmt.Fprintf(w, "инъекций(точных)=%d sticky=%d ambiguous=%d спойлер-блок=%d вытеснено=%d post-check-промахов=%d\n",
|
||||||
|
injected, sticky, ambiguous, spoiler, evicted, misses)
|
||||||
|
printed := false
|
||||||
|
for _, rs := range states {
|
||||||
|
if rs.NPostcheckMiss == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !printed {
|
||||||
|
fmt.Fprintf(w, "%-4s %-6s %6s %s\n", "ch", "chunk", "misses", "detail (src→dst не найден в выводе)")
|
||||||
|
printed = true
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "%-4d %-6d %6d %s\n", rs.Chapter, rs.ChunkIdx, rs.NPostcheckMiss, rs.PostcheckDetail)
|
||||||
|
}
|
||||||
|
// Cheap style/number flaggers (observability, not gates): total + per-chunk detail.
|
||||||
|
fmt.Fprintf(w, "\n=== СТИЛЬ-ГЕЙТЫ (наблюдаемость: тире-диалоги, ё, транслит-междометия, разряды 万/億) — всего %d ===\n", style)
|
||||||
|
stylePrinted := false
|
||||||
|
for _, rs := range states {
|
||||||
|
if rs.NStyleFlags == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !stylePrinted {
|
||||||
|
fmt.Fprintf(w, "%-4s %-6s %6s %s\n", "ch", "chunk", "flags", "detail")
|
||||||
|
stylePrinted = true
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "%-4d %-6d %6d %s\n", rs.Chapter, rs.ChunkIdx, rs.NStyleFlags, rs.StyleDetail)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
committed, reserved, err := ledger()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "\nLedger книги: committed=$%.6f reserved=$%.6f\n", committed, reserved)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// errTail collapses the degraded/err columns into one bounded table tail (empty
|
||||||
|
// when the call was clean). Усечение — по границе руны: err несёт сырые куски
|
||||||
|
// тел провайдеров (CJK/кириллица), байтовый срез печатал бы битый UTF-8
|
||||||
|
// (находка селфревью).
|
||||||
|
func errTail(degraded, errText string) string {
|
||||||
|
s := degraded
|
||||||
|
if errText != "" {
|
||||||
|
if s != "" {
|
||||||
|
s += " "
|
||||||
|
}
|
||||||
|
s += errText
|
||||||
|
}
|
||||||
|
if len(s) > 120 {
|
||||||
|
cut := 120
|
||||||
|
for cut > 0 && !utf8.RuneStart(s[cut]) {
|
||||||
|
cut--
|
||||||
|
}
|
||||||
|
s = s[:cut] + "…"
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// renderStatusJSON emits the StatusReport as indented JSON (stable enums — the
|
||||||
|
// ratified CI/IDE contract, D12/D15.3) and maps flagged>0 to the exit-2 sentinel.
|
||||||
|
func renderStatusJSON(w io.Writer, rep *pipeline.StatusReport) error {
|
||||||
|
enc := json.NewEncoder(w)
|
||||||
|
enc.SetIndent("", " ")
|
||||||
|
if err := enc.Encode(rep); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
// Symmetric with the human mode (minor 1d): flagged>0 is exit 2 (completed-with-flags), not
|
||||||
|
// exit 0. The full JSON is already on stdout; the sentinel only sets the shell disposition
|
||||||
|
// (its message goes to stderr in main), so a CI/IDE consumer both parses the projection AND
|
||||||
|
// sees "attention needed" in the exit code instead of silently reading 0.
|
||||||
|
if rep.Flagged > 0 {
|
||||||
|
return &pipeline.CompletedWithFlags{Flagged: rep.Flagged, Total: rep.TotalChunks}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// renderStatusHuman prints the operator dashboard (unit counts, per-chapter
|
||||||
|
// passports, money, secondary ETA) and maps flagged>0 to the exit-2 sentinel.
|
||||||
|
// cfgPath попадает в советы «что делать» (командные строки redrive/translate).
|
||||||
|
func renderStatusHuman(w io.Writer, rep *pipeline.StatusReport, cfgPath string) error {
|
||||||
|
snap := rep.Snapshot
|
||||||
|
if snap == "" {
|
||||||
|
snap = "—"
|
||||||
|
} else if len(snap) > 12 {
|
||||||
|
snap = snap[:12]
|
||||||
|
}
|
||||||
|
drift := ""
|
||||||
|
if rep.SnapshotDrift {
|
||||||
|
drift += " ⚠ SNAPSHOT-DRIFT (несколько снапшотов в chunk_status — конфиг менялся; --resnapshot)"
|
||||||
|
}
|
||||||
|
if rep.ConfigDrift {
|
||||||
|
drift += " ⚠ CONFIG-DRIFT (текущий конфиг рендерит другой снапшот — правка после прогона; translate потребует --resnapshot = переоплата книги)"
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "=== СТАТУС: %s (snapshot %s)%s ===\n", rep.BookID, snap, drift)
|
||||||
|
fmt.Fprintf(w, "Прогресс: %d/%d чанков (%.1f%%) — done=%d in_progress=%d flagged=%d pending=%d\n",
|
||||||
|
rep.Done, rep.TotalChunks, rep.PercentDone, rep.Done, rep.InProgress, rep.Flagged, rep.Pending)
|
||||||
|
fmt.Fprintf(w, "Эскалаций: %d · post-check промахов (confirmed): %d · стиль-флагов (наблюдаемость): %d\n",
|
||||||
|
rep.Escalations, rep.PostcheckMisses, rep.StyleFlags)
|
||||||
|
ceil := ""
|
||||||
|
if rep.BookCeilingUSD > 0 {
|
||||||
|
ceil = fmt.Sprintf(" (потолок $%.2f — %.1f%%)", rep.BookCeilingUSD, rep.CeilingPct)
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "Деньги: committed=$%.6f reserved=$%.6f%s · прогноз книги ~$%.6f\n",
|
||||||
|
rep.CommittedUSD, rep.ReservedUSD, ceil, rep.ProjectedBookUSD)
|
||||||
|
if rep.ETASeconds > 0 {
|
||||||
|
fmt.Fprintf(w, "ETA: ~%.0fс (средний throughput fresh-вызовов, НЕ EWMA — D12-отступление; вторично)\n", rep.ETASeconds)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(rep.Chapters) > 0 {
|
||||||
|
// style = cheap deterministic style/number flags (observability, not a disposition) — a per-
|
||||||
|
// chapter glance count so a human sees where the linters fired without opening `report` (D20.4).
|
||||||
|
fmt.Fprintf(w, "\n%-5s %-9s %-10s %-6s %-4s %-6s %-16s %10s\n", "ch", "done/all", "verdict", "flag", "esc", "style", "worst_flag", "cost_usd")
|
||||||
|
for _, p := range rep.Chapters {
|
||||||
|
fmt.Fprintf(w, "%-5d %d/%-7d %-10s %-6d %-4d %-6d %-16s %10.6f\n",
|
||||||
|
p.Chapter, p.ChunksDone, p.ChunksTotal, p.Verdict, p.ChunksFlagged, p.Escalations, p.StyleFlags,
|
||||||
|
dashIfEmpty(p.WorstFlagReason), p.CostUSD)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// flagged≠failed (jobs.status.failed = infra-only, D12): flagged chunks are a "clean run,
|
||||||
|
// attention needed" signal. Two kinds need DIFFERENT actions (minor 1d): a chunk_status flag
|
||||||
|
// (soft_refusal, length, …) is re-drivable with `tmctl redrive`; a gate-promoted glossary_miss
|
||||||
|
// is NOT (redrive is a no-op — the miss re-derives from the unchanged glossary), so it needs a
|
||||||
|
// seed fix + `translate --resnapshot`. Advise each set separately instead of one blanket hint.
|
||||||
|
if rep.Flagged > 0 {
|
||||||
|
if redrivable := rep.Flagged - rep.GlossaryMissFlagged; redrivable > 0 {
|
||||||
|
fmt.Fprintf(w, "\n%d флагнутых чанк(ов) ждут внимания — переатаковать: tmctl redrive --config %s [--chapter N --chunk M --reason R]\n",
|
||||||
|
redrivable, cfgPath)
|
||||||
|
}
|
||||||
|
if rep.GlossaryMissFlagged > 0 {
|
||||||
|
fmt.Fprintf(w, "\n%d чанк(ов) флагнуты post-check-гейтом (glossary_miss) — redrive для них no-op: исправьте сид-глоссарий (термин/dst) и перезапустите `tmctl translate --config %s --resnapshot` (переоплата затронутых чанков)\n",
|
||||||
|
rep.GlossaryMissFlagged, cfgPath)
|
||||||
|
}
|
||||||
|
return &pipeline.CompletedWithFlags{Flagged: rep.Flagged, Total: rep.TotalChunks}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// dashIfEmpty renders "" as "—" for a table cell.
|
||||||
|
func dashIfEmpty(s string) string {
|
||||||
|
if s == "" {
|
||||||
|
return "—"
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// renderRedrive prints the redrive plan and (after a live re-run) the re-run
|
||||||
|
// totals. ledger может вернуть ошибку — она СОЗНАТЕЛЬНО глотается без ledger-
|
||||||
|
// строки (исходное поведение: флаки чтения не должны менять exit-код redrive).
|
||||||
|
func renderRedrive(w io.Writer, bookID string, summary *pipeline.RedriveSummary, res *pipeline.BookResult, ledger func() (committed, reserved float64, err error)) error {
|
||||||
|
fmt.Fprintf(w, "=== REDRIVE: %s ===\n", bookID)
|
||||||
|
if len(summary.Targets) == 0 {
|
||||||
|
fmt.Fprintln(w, "Флагнутых чанков под селектор не найдено — нечего переатаковать.")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
fmt.Fprintf(w, "Целей: %d флагнутых чанк(ов) — сброс терминального флага + свежий retry/escalation-бюджет (DispOK не тронут):\n", len(summary.Targets))
|
||||||
|
for _, t := range summary.Targets {
|
||||||
|
fmt.Fprintf(w, " ch%d/chunk%d (%s) → стадии: %s\n", t.Chapter, t.ChunkIdx, t.FlagReason, strings.Join(t.Stages, ", "))
|
||||||
|
}
|
||||||
|
if summary.DryRun {
|
||||||
|
fmt.Fprintln(w, "[dry-run: ничего не сброшено, вызовов не сделано]")
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
fmt.Fprintln(w, "[сброшено; перезапуск durable-цикла — DispOK чанки резюмируются за $0, сброшенные переатакуются]")
|
||||||
|
fmt.Fprintln(w)
|
||||||
|
|
||||||
|
if res != nil {
|
||||||
|
fmt.Fprintf(w, "ИТОГО re-run (этот запуск): $%.6f — чанков %d, флагов %d\n", res.TotalUSD, len(res.Chunks), res.Flagged)
|
||||||
|
committed, reserved, serr := ledger()
|
||||||
|
if serr == nil {
|
||||||
|
fmt.Fprintf(w, "Ledger книги: committed=$%.6f reserved=$%.6f\n", committed, reserved)
|
||||||
|
}
|
||||||
|
if res.Flagged > 0 {
|
||||||
|
return &pipeline.CompletedWithFlags{Flagged: res.Flagged, Total: len(res.Chunks)}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
226
backend/cmd/tmctl/render_test.go
Normal file
226
backend/cmd/tmctl/render_test.go
Normal file
|
|
@ -0,0 +1,226 @@
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/llm"
|
||||||
|
"textmachine/backend/internal/pipeline"
|
||||||
|
"textmachine/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// render_test.go pins the human/JSON output contracts of the extracted renders
|
||||||
|
// (пакет №4): sentinel-возвраты (exit 2), советы «что делать», секции только при
|
||||||
|
// непустых данных, err-хвост report, dry-run redrive.
|
||||||
|
|
||||||
|
func okLedger() (float64, float64, error) { return 0.123456, 0, nil }
|
||||||
|
|
||||||
|
func TestRenderTranslateOKAndLedger(t *testing.T) {
|
||||||
|
var b strings.Builder
|
||||||
|
res := &pipeline.BookResult{BookID: "b1", TotalUSD: 0.01, Chunks: []pipeline.ChunkOutcome{{
|
||||||
|
Chapter: 1, ChunkIdx: 0, Disposition: pipeline.DispOK, FinalText: "ТЕКСТ",
|
||||||
|
Stages: []pipeline.StageResult{{Stage: "draft", Model: "m", Disposition: pipeline.DispOK,
|
||||||
|
Usage: llm.Usage{PromptTokens: 10, CompletionTokens: 5}, Attempts: 1, FinishReason: "stop"}},
|
||||||
|
}}}
|
||||||
|
if err := renderTranslate(&b, res, okLedger); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
out := b.String()
|
||||||
|
for _, want := range []string{
|
||||||
|
"=== ГЛАВА 1 ЧАНК 0 — ok ===", "ТЕКСТ",
|
||||||
|
"ИТОГО (этот запуск): $0.010000 — чанков 1, флагов 0",
|
||||||
|
"Ledger книги: committed=$0.123456 reserved=$0.000000",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(out, want) {
|
||||||
|
t.Fatalf("translate output must contain %q, got:\n%s", want, out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderTranslateFlaggedSentinel(t *testing.T) {
|
||||||
|
var b strings.Builder
|
||||||
|
res := &pipeline.BookResult{BookID: "b1", Flagged: 1, Chunks: []pipeline.ChunkOutcome{{
|
||||||
|
Chapter: 2, ChunkIdx: 1, Disposition: pipeline.DispFlagged, FlagReason: pipeline.FlagSoftRefusal,
|
||||||
|
}}}
|
||||||
|
err := renderTranslate(&b, res, okLedger)
|
||||||
|
var flagged *pipeline.CompletedWithFlags
|
||||||
|
if !errors.As(err, &flagged) || flagged.Flagged != 1 {
|
||||||
|
t.Fatalf("flagged run must return the exit-2 sentinel, got: %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(b.String(), "[ФЛАГ soft_refusal]") {
|
||||||
|
t.Fatalf("flagged chunk banner missing:\n%s", b.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderTranslateLedgerErrorAfterTotals(t *testing.T) {
|
||||||
|
// Замороженный порядок частичного вывода: итоговая строка уже напечатана,
|
||||||
|
// ошибка чтения ledger обрывает ПОСЛЕ неё (подъём чтения до рендера менял бы
|
||||||
|
// байты частичного вывода — риск-карта извлечения, п.4).
|
||||||
|
var b strings.Builder
|
||||||
|
res := &pipeline.BookResult{BookID: "b1"}
|
||||||
|
err := renderTranslate(&b, res, func() (float64, float64, error) { return 0, 0, errors.New("boom") })
|
||||||
|
if err == nil || err.Error() != "boom" {
|
||||||
|
t.Fatalf("ledger error must propagate, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(b.String(), "ИТОГО") || strings.Contains(b.String(), "Ledger книги") {
|
||||||
|
t.Fatalf("totals must be printed, ledger line must not:\n%s", b.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderReportColumnsAndErrTail(t *testing.T) {
|
||||||
|
var b strings.Builder
|
||||||
|
rows := []store.RequestLogView{
|
||||||
|
{TS: "2026-07-10 17:00:00", Chapter: 3, ChunkIdx: 2, Stage: "draft", Role: "translator",
|
||||||
|
ModelRequested: "fake-model", ModelActual: "", Err: "connect: refused", OK: 0},
|
||||||
|
{TS: "2026-07-10 17:00:05", Chapter: 3, ChunkIdx: 2, Stage: "draft", Role: "translator",
|
||||||
|
ModelRequested: "fake-model", ModelActual: "fake-model", Degraded: "cjk_artifact", OK: 0},
|
||||||
|
}
|
||||||
|
if err := renderReport(&b,
|
||||||
|
func() ([]store.RequestLogView, error) { return rows, nil },
|
||||||
|
func() ([]store.ChunkStatus, error) { return nil, nil },
|
||||||
|
func() ([]store.RetrievalState, error) { return nil, nil },
|
||||||
|
okLedger); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
out := b.String()
|
||||||
|
// Пакет №4: у упавшего вызова model_actual пуст — строка обязана нести
|
||||||
|
// запрошенную модель, позицию и текст ошибки (пост-мортем без sqlite3).
|
||||||
|
if !strings.Contains(out, "fake-model(req)") {
|
||||||
|
t.Fatalf("failed call must show the requested model, got:\n%s", out)
|
||||||
|
}
|
||||||
|
if !strings.Contains(out, "connect: refused") || !strings.Contains(out, "cjk_artifact") {
|
||||||
|
t.Fatalf("err/degraded tail missing:\n%s", out)
|
||||||
|
}
|
||||||
|
// Секции флагов/памяти не печатаются без данных.
|
||||||
|
if strings.Contains(out, "ФЛАГИ") || strings.Contains(out, "ПАМЯТЬ") {
|
||||||
|
t.Fatalf("empty sections must not print headers:\n%s", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderReportSections(t *testing.T) {
|
||||||
|
var b strings.Builder
|
||||||
|
flags := []store.ChunkStatus{
|
||||||
|
{Chapter: 1, ChunkIdx: 0, Stage: "draft", Disposition: "ok"},
|
||||||
|
{Chapter: 2, ChunkIdx: 1, Stage: "draft", Disposition: "flagged", FlagReason: "hard_refusal", Attempts: 1, Detail: "provider refused"},
|
||||||
|
}
|
||||||
|
states := []store.RetrievalState{{Chapter: 2, ChunkIdx: 1, NExactHits: 3, NPostcheckMiss: 1,
|
||||||
|
PostcheckDetail: `[{"src":"鈴木"}]`, NStyleFlags: 2, StyleDetail: `{"yo":2}`}}
|
||||||
|
if err := renderReport(&b,
|
||||||
|
func() ([]store.RequestLogView, error) { return nil, nil },
|
||||||
|
func() ([]store.ChunkStatus, error) { return flags, nil },
|
||||||
|
func() ([]store.RetrievalState, error) { return states, nil },
|
||||||
|
okLedger); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
out := b.String()
|
||||||
|
for _, want := range []string{"=== ФЛАГИ (disposition ≠ ok) ===", "hard_refusal",
|
||||||
|
"=== ПАМЯТЬ (retrieval-state) ===", "post-check-промахов=1", `[{"src":"鈴木"}]`,
|
||||||
|
"— всего 2 ===", "Ledger книги"} {
|
||||||
|
if !strings.Contains(out, want) {
|
||||||
|
t.Fatalf("report must contain %q, got:\n%s", want, out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if strings.Contains(out, "1 0") && strings.Contains(out, "ok") && strings.Count(out, "draft") > 1 {
|
||||||
|
t.Fatalf("ok rows must not enter the flag section:\n%s", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderReportPartialOutputOnMidAuditError(t *testing.T) {
|
||||||
|
// Замороженное поведение (селфревью №4): чтения store интерливятся с печатью,
|
||||||
|
// провал ПОСЛЕ первой секции оставляет её на stdout (аудит не «пустеет» молча).
|
||||||
|
var b strings.Builder
|
||||||
|
rows := []store.RequestLogView{{TS: "t", Chapter: 1, Stage: "draft", Role: "translator", ModelActual: "m", OK: 1}}
|
||||||
|
err := renderReport(&b,
|
||||||
|
func() ([]store.RequestLogView, error) { return rows, nil },
|
||||||
|
func() ([]store.ChunkStatus, error) { return nil, errors.New("chunk_status corrupted") },
|
||||||
|
func() ([]store.RetrievalState, error) { t.Fatal("must not be reached"); return nil, nil },
|
||||||
|
okLedger)
|
||||||
|
if err == nil || err.Error() != "chunk_status corrupted" {
|
||||||
|
t.Fatalf("mid-audit error must propagate, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(b.String(), "draft") {
|
||||||
|
t.Fatalf("the request_log table printed before the failure must remain on the writer:\n%s", b.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderStatusJSONSchemaAndSentinel(t *testing.T) {
|
||||||
|
var b strings.Builder
|
||||||
|
rep := &pipeline.StatusReport{BookID: "b1", TotalChunks: 4, Done: 3, Flagged: 1}
|
||||||
|
err := renderStatusJSON(&b, rep)
|
||||||
|
var flagged *pipeline.CompletedWithFlags
|
||||||
|
if !errors.As(err, &flagged) {
|
||||||
|
t.Fatalf("--json with flags must return the exit-2 sentinel (minor 1d), got %v", err)
|
||||||
|
}
|
||||||
|
var decoded map[string]any
|
||||||
|
if jerr := json.Unmarshal([]byte(b.String()), &decoded); jerr != nil {
|
||||||
|
t.Fatalf("output must be valid JSON: %v", jerr)
|
||||||
|
}
|
||||||
|
if decoded["book_id"] != "b1" {
|
||||||
|
t.Fatalf("stable schema field book_id missing: %v", decoded)
|
||||||
|
}
|
||||||
|
// Clean book → nil error → exit 0.
|
||||||
|
if err := renderStatusJSON(&strings.Builder{}, &pipeline.StatusReport{TotalChunks: 4, Done: 4}); err != nil {
|
||||||
|
t.Fatalf("clean book must exit 0, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderStatusHumanDriftAndAdvice(t *testing.T) {
|
||||||
|
var b strings.Builder
|
||||||
|
rep := &pipeline.StatusReport{
|
||||||
|
BookID: "b1", Snapshot: "abcdef0123456789", TotalChunks: 10, Done: 7,
|
||||||
|
Flagged: 3, GlossaryMissFlagged: 1, SnapshotDrift: true, ConfigDrift: true,
|
||||||
|
}
|
||||||
|
err := renderStatusHuman(&b, rep, "book.yaml")
|
||||||
|
var flagged *pipeline.CompletedWithFlags
|
||||||
|
if !errors.As(err, &flagged) || flagged.Flagged != 3 {
|
||||||
|
t.Fatalf("flagged status must return the exit-2 sentinel, got %v", err)
|
||||||
|
}
|
||||||
|
out := b.String()
|
||||||
|
for _, want := range []string{
|
||||||
|
"snapshot abcdef012345", "⚠ SNAPSHOT-DRIFT", "⚠ CONFIG-DRIFT",
|
||||||
|
// Совет расщеплён (minor 1d): re-drivable отдельно от glossary_miss (redrive там no-op).
|
||||||
|
"2 флагнутых чанк(ов) ждут внимания — переатаковать: tmctl redrive --config book.yaml",
|
||||||
|
"1 чанк(ов) флагнуты post-check-гейтом (glossary_miss)",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(out, want) {
|
||||||
|
t.Fatalf("status must contain %q, got:\n%s", want, out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderRedriveDryRunAndSentinel(t *testing.T) {
|
||||||
|
var b strings.Builder
|
||||||
|
sum := &pipeline.RedriveSummary{DryRun: true, Targets: []pipeline.RedriveTarget{
|
||||||
|
{Chapter: 1, ChunkIdx: 2, FlagReason: "cjk_artifact", Stages: []string{"draft", "edit"}}}}
|
||||||
|
if err := renderRedrive(&b, "b1", sum, nil, okLedger); err != nil {
|
||||||
|
t.Fatalf("dry-run must exit 0, got %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(b.String(), "[dry-run: ничего не сброшено, вызовов не сделано]") {
|
||||||
|
t.Fatalf("dry-run banner missing:\n%s", b.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
// No targets → informational exit 0.
|
||||||
|
b.Reset()
|
||||||
|
if err := renderRedrive(&b, "b1", &pipeline.RedriveSummary{}, nil, okLedger); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(b.String(), "нечего переатаковать") {
|
||||||
|
t.Fatalf("no-targets banner missing:\n%s", b.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
// Live re-run that still has flags → sentinel; a flaky ledger read is
|
||||||
|
// swallowed (no ledger line, exit code unchanged) — замороженное поведение.
|
||||||
|
b.Reset()
|
||||||
|
sum = &pipeline.RedriveSummary{Targets: []pipeline.RedriveTarget{{Chapter: 1, ChunkIdx: 2, FlagReason: "x", Stages: []string{"draft"}}}}
|
||||||
|
res := &pipeline.BookResult{BookID: "b1", Flagged: 1, Chunks: make([]pipeline.ChunkOutcome, 3)}
|
||||||
|
err := renderRedrive(&b, "b1", sum, res, func() (float64, float64, error) { return 0, 0, errors.New("flaky") })
|
||||||
|
var flagged *pipeline.CompletedWithFlags
|
||||||
|
if !errors.As(err, &flagged) {
|
||||||
|
t.Fatalf("re-run with flags must return the exit-2 sentinel, got %v", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(b.String(), "Ledger книги") {
|
||||||
|
t.Fatalf("flaky ledger read must swallow the ledger line, not fail:\n%s", b.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -124,7 +124,9 @@ func (c *failoverClient) Complete(ctx context.Context, req LLMRequest) (*LLMResp
|
||||||
// A 2xx with EMPTY content from the local leg (thinking ate the token
|
// A 2xx with EMPTY content from the local leg (thinking ate the token
|
||||||
// budget, a quirk of the abliterated model). The local call is free, so
|
// budget, a quirk of the abliterated model). The local call is free, so
|
||||||
// retry the turn on the cloud. The backend is up: not a breaker trip.
|
// retry the turn on the cloud. The backend is up: not a breaker trip.
|
||||||
c.log.DebugContext(ctx, "local llm returned empty content; retrying on cloud")
|
// INFO, не DEBUG: каждый такой ретрай — реальные облачные деньги на роли,
|
||||||
|
// задуманной бесплатной; un-budgeted local→cloud обязан быть виден (D4.3).
|
||||||
|
c.log.InfoContext(ctx, "local llm returned empty content; retrying on cloud (paid)")
|
||||||
return c.fallback.Complete(ctx, req)
|
return c.fallback.Complete(ctx, req)
|
||||||
}
|
}
|
||||||
if ctx.Err() != nil {
|
if ctx.Err() != nil {
|
||||||
|
|
@ -139,7 +141,10 @@ func (c *failoverClient) Complete(ctx context.Context, req LLMRequest) (*LLMResp
|
||||||
}
|
}
|
||||||
|
|
||||||
c.trip()
|
c.trip()
|
||||||
c.log.DebugContext(ctx, "local llm failed; falling back to cloud", "err", err)
|
// WARN с причиной: сам trip() логирует переход БЕЗ err (он под мьютексом и не
|
||||||
|
// знает запроса), а на дефолтном INFO оператор иначе видел бы «tripped offline»
|
||||||
|
// с выброшенной причиной — первый вопрос дебага канала B (пакет №4).
|
||||||
|
c.log.WarnContext(ctx, "local llm failed; falling back to cloud", "err", err)
|
||||||
resp, ferr := c.fallback.Complete(ctx, req)
|
resp, ferr := c.fallback.Complete(ctx, req)
|
||||||
if ferr != nil {
|
if ferr != nil {
|
||||||
c.log.WarnContext(ctx, "local llm and cloud fallback both failed", "local_err", err, "cloud_err", ferr)
|
c.log.WarnContext(ctx, "local llm and cloud fallback both failed", "local_err", err, "cloud_err", ferr)
|
||||||
|
|
|
||||||
|
|
@ -79,31 +79,20 @@ const maxRetryAfterWait = 5 * time.Minute
|
||||||
// and the native Anthropic adapter): exponential backoff with cap, Retry-After
|
// and the native Anthropic adapter): exponential backoff with cap, Retry-After
|
||||||
// override, jitter, ctx-done select. Один экземпляр политики — чтобы фикс
|
// override, jitter, ctx-done select. Один экземпляр политики — чтобы фикс
|
||||||
// ретраев не «уезжал» от одного провайдера, забыв другой.
|
// ретраев не «уезжал» от одного провайдера, забыв другой.
|
||||||
|
//
|
||||||
|
// Логи — продукт для оператора (пакет №4, боли сняты smoke-прогоном): старт
|
||||||
|
// каждой попытки виден на DEBUG (иначе «висит на таймауте» неотличимо от
|
||||||
|
// «умер»), WARN несёт ФАКТИЧЕСКОЕ ожидание перед следующей попыткой (Retry-After
|
||||||
|
// может молча растянуть его до 5 мин против ожидаемого backoff_cap) и говорит
|
||||||
|
// «will retry» ТОЛЬКО когда попытка действительно остаётся — раньше последний
|
||||||
|
// провал тоже логировался как «will retry» прямо перед exhausted-ошибкой.
|
||||||
func retryLoop[T any](ctx context.Context, profile RetryProfile, name string, log *slog.Logger, attempt func() (T, bool, error)) (T, error) {
|
func retryLoop[T any](ctx context.Context, profile RetryProfile, name string, log *slog.Logger, attempt func() (T, bool, error)) (T, error) {
|
||||||
var zero T
|
var zero T
|
||||||
var lastErr error
|
var lastErr error
|
||||||
for att := 0; att < profile.MaxAttempts; att++ {
|
for att := 0; att < profile.MaxAttempts; att++ {
|
||||||
if att > 0 {
|
if log != nil {
|
||||||
shift := att - 1
|
log.DebugContext(ctx, name+" attempt start", "attempt", att+1, "max", profile.MaxAttempts,
|
||||||
if shift > 20 {
|
"attempt_timeout", profile.AttemptTimeout.String())
|
||||||
shift = 20 // защита сдвига от переполнения при больших max_attempts
|
|
||||||
}
|
|
||||||
backoff := profile.BackoffBase << uint(shift)
|
|
||||||
if backoff <= 0 || backoff > profile.BackoffCap {
|
|
||||||
backoff = profile.BackoffCap
|
|
||||||
}
|
|
||||||
if ra := retryAfterOf(lastErr); ra > 0 {
|
|
||||||
if ra > maxRetryAfterWait {
|
|
||||||
ra = maxRetryAfterWait
|
|
||||||
}
|
|
||||||
backoff = ra
|
|
||||||
}
|
|
||||||
backoff += time.Duration(rand.Intn(250)) * time.Millisecond
|
|
||||||
select {
|
|
||||||
case <-ctx.Done():
|
|
||||||
return zero, ctx.Err()
|
|
||||||
case <-time.After(backoff):
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
resp, retryable, err := attempt()
|
resp, retryable, err := attempt()
|
||||||
if err == nil {
|
if err == nil {
|
||||||
|
|
@ -116,13 +105,45 @@ func retryLoop[T any](ctx context.Context, profile RetryProfile, name string, lo
|
||||||
if !retryable {
|
if !retryable {
|
||||||
return zero, err
|
return zero, err
|
||||||
}
|
}
|
||||||
|
if att+1 >= profile.MaxAttempts {
|
||||||
|
break // attempts exhausted — no retry remains, no sleep
|
||||||
|
}
|
||||||
|
backoff := nextBackoff(profile, att, lastErr)
|
||||||
if log != nil {
|
if log != nil {
|
||||||
log.WarnContext(ctx, name+" attempt failed, will retry", "attempt", att+1, "max", profile.MaxAttempts, "err", err)
|
log.WarnContext(ctx, name+" attempt failed, will retry", "attempt", att+1, "max", profile.MaxAttempts,
|
||||||
|
"retry_in", backoff.Round(time.Millisecond).String(), "err", err)
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return zero, ctx.Err()
|
||||||
|
case <-time.After(backoff):
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return zero, fmt.Errorf("%s: exhausted %d attempts: %w", name, profile.MaxAttempts, lastErr)
|
return zero, fmt.Errorf("%s: exhausted %d attempts: %w", name, profile.MaxAttempts, lastErr)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// nextBackoff computes the wait before attempt att+1 (0-based att just failed):
|
||||||
|
// exponential base<<att with cap, overridden by an honoured Retry-After (bounded
|
||||||
|
// by maxRetryAfterWait), plus jitter. Extracted from the loop so the WARN above
|
||||||
|
// can log the REAL wait it is about to sleep.
|
||||||
|
func nextBackoff(profile RetryProfile, att int, lastErr error) time.Duration {
|
||||||
|
shift := att
|
||||||
|
if shift > 20 {
|
||||||
|
shift = 20 // защита сдвига от переполнения при больших max_attempts
|
||||||
|
}
|
||||||
|
backoff := profile.BackoffBase << uint(shift)
|
||||||
|
if backoff <= 0 || backoff > profile.BackoffCap {
|
||||||
|
backoff = profile.BackoffCap
|
||||||
|
}
|
||||||
|
if ra := retryAfterOf(lastErr); ra > 0 {
|
||||||
|
if ra > maxRetryAfterWait {
|
||||||
|
ra = maxRetryAfterWait
|
||||||
|
}
|
||||||
|
backoff = ra
|
||||||
|
}
|
||||||
|
return backoff + time.Duration(rand.Intn(250))*time.Millisecond
|
||||||
|
}
|
||||||
|
|
||||||
// openAIClient performs OpenAI-compatible /chat/completions calls with retry.
|
// openAIClient performs OpenAI-compatible /chat/completions calls with retry.
|
||||||
type openAIClient struct {
|
type openAIClient struct {
|
||||||
name string // provider label for logs/errors ("deepseek", "glm", "local")
|
name string // provider label for logs/errors ("deepseek", "glm", "local")
|
||||||
|
|
@ -298,7 +319,13 @@ func (c *openAIClient) attempt(ctx context.Context, payload []byte) (*openAIResp
|
||||||
|
|
||||||
resp, err := c.http.Do(req)
|
resp, err := c.http.Do(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
// Network error / timeout — retryable (unless the parent ctx is done).
|
// Network error / timeout — retryable (unless the parent ctx is done). A
|
||||||
|
// per-attempt deadline is annotated with the configured timeout: the bare
|
||||||
|
// «context deadline exceeded» не говорит оператору, ЧЕЙ это дедлайн —
|
||||||
|
// таймаут попытки (лечится timeouts.attempt_s) или отмена всего прогона.
|
||||||
|
if errors.Is(attemptCtx.Err(), context.DeadlineExceeded) && ctx.Err() == nil {
|
||||||
|
err = fmt.Errorf("attempt timed out after %s (timeouts.attempt_s): %w", c.profile.AttemptTimeout, err)
|
||||||
|
}
|
||||||
return nil, ctx.Err() == nil, err
|
return nil, ctx.Err() == nil, err
|
||||||
}
|
}
|
||||||
defer resp.Body.Close()
|
defer resp.Body.Close()
|
||||||
|
|
|
||||||
179
backend/internal/pipeline/bookrun.go
Normal file
179
backend/internal/pipeline/bookrun.go
Normal file
|
|
@ -0,0 +1,179 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"textmachine/backend/internal/llm"
|
||||||
|
)
|
||||||
|
|
||||||
|
// bookrun.go: цикл уровня КНИГИ (Веха 2) — ingest → сид/материализация памяти →
|
||||||
|
// снапшот → последовательный обход чанков с D2-дисциплиной (флаг не роняет прогон)
|
||||||
|
// и sticky-окном сцены (A5). Плюс типы результатов прогона и exit-контракт CLI.
|
||||||
|
|
||||||
|
// StageResult reports one executed stage of a chunk.
|
||||||
|
type StageResult struct {
|
||||||
|
Stage string
|
||||||
|
Role string
|
||||||
|
Model string // фактически ответившая модель
|
||||||
|
FromResume bool // no provider call was made THIS run (fully served from checkpoints)
|
||||||
|
Usage llm.Usage
|
||||||
|
CostUSD float64 // THIS run's spend (0 when fully served from checkpoints)
|
||||||
|
CumCostUSD float64 // sum across ALL attempts of this chunk×stage (F3-honest, incl. retries)
|
||||||
|
LatencyMS int
|
||||||
|
FinishReason string
|
||||||
|
Text string // the usable output (only on an ok disposition)
|
||||||
|
Disposition Disposition
|
||||||
|
FlagReason FlagReason // "" when ok
|
||||||
|
Detail string
|
||||||
|
Attempts int
|
||||||
|
// Escalated — a single-hop fallback draft was tried this stage (D12); when the
|
||||||
|
// fallback passed the re-gate, Model above is the fallback (it answered).
|
||||||
|
Escalated bool
|
||||||
|
EscalationModel string // the fallback model when Escalated ("" otherwise)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ChunkOutcome is one chunk's result across the stage list.
|
||||||
|
type ChunkOutcome struct {
|
||||||
|
Chapter int
|
||||||
|
ChunkIdx int
|
||||||
|
Stages []StageResult
|
||||||
|
FinalText string // the last stage's output; "" when the chunk is flagged
|
||||||
|
Disposition Disposition // ok | flagged (a chunk has no "skipped" — that is a per-later-stage state)
|
||||||
|
FlagReason FlagReason // the flagging stage's reason ("" when ok)
|
||||||
|
CostUSD float64 // THIS run's spend on this chunk
|
||||||
|
}
|
||||||
|
|
||||||
|
// BookResult aggregates a whole run over every chapter×chunk.
|
||||||
|
type BookResult struct {
|
||||||
|
BookID string
|
||||||
|
Chunks []ChunkOutcome
|
||||||
|
TotalUSD float64 // THIS run's spend across all chunks
|
||||||
|
Flagged int // number of flagged chunks (acceptance допускает N)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ExitCode is the run's shell disposition: 0 clean, 2 completed-with-flags
|
||||||
|
// (acceptance допускает N флагов — приёмка это разрешает). An infra failure is
|
||||||
|
// an error from TranslateBook, which the CLI maps to 1.
|
||||||
|
func (b *BookResult) ExitCode() int {
|
||||||
|
if b.Flagged > 0 {
|
||||||
|
return 2
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// CompletedWithFlags is the typed sentinel the CLI maps to exit code 2: the book
|
||||||
|
// finished end-to-end but N chunks were flagged for a human. It is NOT an infra
|
||||||
|
// failure (a plain error → exit 1); it is a "clean run, attention needed" signal
|
||||||
|
// carried up through `func run() error` in the idiomatic Go way.
|
||||||
|
type CompletedWithFlags struct {
|
||||||
|
Flagged int
|
||||||
|
Total int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *CompletedWithFlags) Error() string {
|
||||||
|
return fmt.Sprintf("completed with %d/%d chunk(s) flagged for review", e.Flagged, e.Total)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TranslateBook runs the whole book: split the normalized source into
|
||||||
|
// chapter/chunk units (chunker.go) and drive each chunk through the stage list.
|
||||||
|
// A bad chunk is flagged and the loop CONTINUES (D2); only an infra failure
|
||||||
|
// (ceiling, config change without --resnapshot, unbilled call failure, store
|
||||||
|
// error) aborts with an error, from which resume continues.
|
||||||
|
func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) {
|
||||||
|
// Ingest reads + normalizes the source (txt/epub) into ordered per-chapter text
|
||||||
|
// and captures ruby readings (ingest.go), decoding the txt source per book.encoding
|
||||||
|
// (auto/utf8/gb18030). Offline and deterministic ($0, no LLM).
|
||||||
|
doc, err := IngestEncoded(r.Book.SourceFile, r.Book.Encoding, r.Book.SourceLang)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Persist captured ruby readings (idempotent; consumed by seedGlossary below into
|
||||||
|
// auto glossary candidates — шаг 4). Never injected into a prompt here (§7d); a
|
||||||
|
// resume re-persists the same rows at $0.
|
||||||
|
if err := r.persistRuby(doc.Ruby); err != nil {
|
||||||
|
return nil, fmt.Errorf("pipeline: persist ruby readings: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Seed + materialize the glossary BEFORE the snapshot: the frozen approved rows feed
|
||||||
|
// memoryVersion() → the snapshot (F1), so the snapshot must be computed with the bank
|
||||||
|
// already materialized. Deterministic and $0 (seed file + classified ruby, no LLM).
|
||||||
|
if err := r.seedGlossary(ctx); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Snapshot is book-level (brief + stage plan + memory), computed once and
|
||||||
|
// upserted before the loop; every job pins to it. memoryVersion() now reflects the
|
||||||
|
// materialized bank, so a changed approved glossary re-pins loudly (F1).
|
||||||
|
snapID, snapPayload, err := r.snapshotID()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := r.Store.UpsertSnapshot(snapID, r.Book.BriefHash(), snapPayload); err != nil {
|
||||||
|
return nil, fmt.Errorf("pipeline: upsert snapshot %.12s: %w", snapID, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
chunks := SplitChunks(doc.Chapters)
|
||||||
|
if len(chunks) == 0 {
|
||||||
|
return nil, fmt.Errorf("pipeline: source file %s produced no chunks after normalization", r.Book.SourceFile)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Масштаб прогона — одной строкой на stderr (боль smoke-прогона: N/M и знаменатель
|
||||||
|
// прогресса не появлялись в логах вообще, только в stdout/status).
|
||||||
|
r.Log.InfoContext(ctx, "book run started", "book", r.Book.BookID, "snapshot", snapID[:12],
|
||||||
|
"chapters", len(doc.Chapters), "chunks", len(chunks), "stages_per_chunk", len(r.Pipeline.Stages))
|
||||||
|
|
||||||
|
res := &BookResult{BookID: r.Book.BookID}
|
||||||
|
// Sticky scene-inertia state (A5): the exact-matched ids of the recent chunks in
|
||||||
|
// the CURRENT chapter, reset at a chapter boundary (a new chapter is a scene change).
|
||||||
|
// Rebuilt deterministically each run because translateChunk recomputes the ($0)
|
||||||
|
// selection for EVERY chunk, resumed ones included — so sticky is resume-stable.
|
||||||
|
var stickyWin []map[string]injectionDisposition
|
||||||
|
prevChapter := 0
|
||||||
|
for i, ch := range chunks {
|
||||||
|
if ch.Chapter != prevChapter {
|
||||||
|
stickyWin = nil
|
||||||
|
prevChapter = ch.Chapter
|
||||||
|
// Пульс прогресса на границе главы: done/total + деньги ЭТОГО прогона —
|
||||||
|
// оператор длинной книги видит движение, не открывая status.
|
||||||
|
r.Log.InfoContext(ctx, "chapter started", "chapter", ch.Chapter,
|
||||||
|
"chunks_done", i, "chunks_total", len(chunks), "run_usd", fmt.Sprintf("%.6f", res.TotalUSD))
|
||||||
|
}
|
||||||
|
outcome, activeIDs, err := r.translateChunk(ctx, snapID, ch, unionSticky(stickyWin))
|
||||||
|
if err != nil {
|
||||||
|
// Infra failure: abort. Chunks already committed are checkpointed;
|
||||||
|
// resume continues from here at $0 for the done work.
|
||||||
|
return res, err
|
||||||
|
}
|
||||||
|
res.Chunks = append(res.Chunks, *outcome)
|
||||||
|
res.TotalUSD += outcome.CostUSD
|
||||||
|
if outcome.Disposition == DispFlagged {
|
||||||
|
res.Flagged++
|
||||||
|
}
|
||||||
|
// Advance the sticky window (keep the last stickyDepth chunks' exact matches).
|
||||||
|
stickyWin = append(stickyWin, activeIDs)
|
||||||
|
if len(stickyWin) > stickyDepth {
|
||||||
|
stickyWin = stickyWin[len(stickyWin)-stickyDepth:]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
r.Log.InfoContext(ctx, "book run finished", "book", r.Book.BookID,
|
||||||
|
"chunks", len(res.Chunks), "flagged", res.Flagged, "run_usd", fmt.Sprintf("%.6f", res.TotalUSD))
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// unionSticky merges the recent chunks' exact-matched ids into one sticky_prev, carrying
|
||||||
|
// each id's DISPOSITION (D16.2). When an id fired in several recent chunks, the MOST RECENT
|
||||||
|
// firing wins (win is ordered oldest→newest), so a re-established CONFIRMED match overrides
|
||||||
|
// an older collision-prone AMBIGUOUS one — and vice-versa, never silently upgrading.
|
||||||
|
func unionSticky(win []map[string]injectionDisposition) map[string]injectionDisposition {
|
||||||
|
if len(win) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := map[string]injectionDisposition{}
|
||||||
|
for _, s := range win {
|
||||||
|
for id, disp := range s {
|
||||||
|
out[id] = disp
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
@ -12,7 +12,7 @@ import (
|
||||||
// whole paragraphs to a ~1–2k-TOKEN budget, and when a single paragraph exceeds
|
// whole paragraphs to a ~1–2k-TOKEN budget, and when a single paragraph exceeds
|
||||||
// the budget descend to SENTENCE boundaries — never splitting a sentence.
|
// the budget descend to SENTENCE boundaries — never splitting a sentence.
|
||||||
//
|
//
|
||||||
// Contract kept intact for the loop (runner.go depends on it): SplitChunks emits
|
// Contract kept intact for the loop (bookrun.go/chunkrun.go depend on it): SplitChunks emits
|
||||||
// Chunk{Chapter,ChunkIdx,Text}; ChunkIdx resets to 0 per chapter; the function is
|
// Chunk{Chapter,ChunkIdx,Text}; ChunkIdx resets to 0 per chapter; the function is
|
||||||
// a PURE, deterministic function of its input (no time/rand, no map iteration).
|
// a PURE, deterministic function of its input (no time/rand, no map iteration).
|
||||||
// Its behavior is versioned by chunkerVersion (render.go), folded into the job
|
// Its behavior is versioned by chunkerVersion (render.go), folded into the job
|
||||||
|
|
|
||||||
234
backend/internal/pipeline/chunkrun.go
Normal file
234
backend/internal/pipeline/chunkrun.go
Normal file
|
|
@ -0,0 +1,234 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"maps"
|
||||||
|
"slices"
|
||||||
|
"textmachine/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// chunkrun.go: disposition-петля уровня ЧАНКА (D2) — стадии по порядку, первый флаг
|
||||||
|
// останавливает чанк (дальше skipped, платной редактуры мусора нет), плюс горячий
|
||||||
|
// путь памяти v2 (Select→инъекция per-role→post-check E1) и дешёвые стиль-флаггеры.
|
||||||
|
// Новые роли-потребители инъекции Ф2 (annotator, voice-слой) добавляются здесь.
|
||||||
|
|
||||||
|
// classifyOutput resolves a completion's disposition: FIRST the always-on intrinsic
|
||||||
|
// classifier (echo/empty/refusal/length — Веха 2), then, only if that is ok AND the
|
||||||
|
// configurable coverage gate is enabled, the excision gate (шаг 6 / D12 Q3). Both are
|
||||||
|
// pure and deterministic over (source, output, finish), so a resumed checkpoint
|
||||||
|
// reproduces the identical verdict for free — the intrinsic part unconditionally, the
|
||||||
|
// gate part under the same coverage config (folded into the snapshot, so a gate change
|
||||||
|
// re-pins loudly). The coverage gate runs ONLY on the TRANSLATOR role's output (vs the
|
||||||
|
// original source): a monolingual editor legitimately restructures sentences, so gating
|
||||||
|
// it against the source would false-flag a correct edit (see the role check below).
|
||||||
|
func (r *Runner) classifyOutput(role, source, output, finish string) classification {
|
||||||
|
cls := classify(classifyInput{Source: source, Output: output, Finish: finish, TargetLang: r.Book.TargetLang})
|
||||||
|
if !cls.ok() || !r.Pipeline.Gates.Coverage.Enabled {
|
||||||
|
return cls
|
||||||
|
}
|
||||||
|
// The coverage gate compares the output against the ORIGINAL source, which is a
|
||||||
|
// source→target translation only for the TRANSLATOR role. A monolingual editor
|
||||||
|
// legitimately restructures/merges sentences, so gating ITS output against the
|
||||||
|
// source with the translation corridor false-flags a correct edit as excision
|
||||||
|
// (self-review finding). Editor/other-role fidelity is a Phase-2 concern (a
|
||||||
|
// bilingual judge over the draft, §04 mode 2), not this excision gate.
|
||||||
|
if role != roleTranslator {
|
||||||
|
return cls
|
||||||
|
}
|
||||||
|
if cov := coverageCheck(r.Pipeline.Gates.Coverage, source, output, r.Book.SourceLang, r.Book.TargetLang); !cov.cls.ok() {
|
||||||
|
return cov.cls
|
||||||
|
}
|
||||||
|
return cls
|
||||||
|
}
|
||||||
|
|
||||||
|
// translateChunk drives ONE chunk through the stage list. Stages run in order,
|
||||||
|
// each feeding the next; the FIRST flagged stage stops the chunk — later stages
|
||||||
|
// are recorded `skipped` (no paid edit over a garbage draft, D2). It also runs the
|
||||||
|
// memory bank v2 hot path: it Selects the glossary injection once ($0, deterministic),
|
||||||
|
// injects it into the TRANSLATOR stage, post-checks the translator output (E1), and
|
||||||
|
// persists the per-chunk retrieval-state (observability). Returns the chunk outcome, the
|
||||||
|
// exact-matched entity ids (the next chunk's sticky_prev, A5), and an error only on an
|
||||||
|
// infra failure. stickyPrev is the prior chunks' exact matches in this chapter.
|
||||||
|
func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, stickyPrev map[string]injectionDisposition) (*ChunkOutcome, map[string]injectionDisposition, error) {
|
||||||
|
out := &ChunkOutcome{Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Disposition: DispOK}
|
||||||
|
prev := ""
|
||||||
|
flagged := false
|
||||||
|
var flagReason FlagReason
|
||||||
|
|
||||||
|
// Hot path: select the glossary records for this chunk ONCE (deterministic, $0), and
|
||||||
|
// serialize the translator injection block. Recomputed on every run (resumed chunks
|
||||||
|
// included) so the sticky window and the injected bytes are resume-stable.
|
||||||
|
var memSel memorySelection
|
||||||
|
var translatorInjection, editorInjection string
|
||||||
|
activeIDs := map[string]injectionDisposition{}
|
||||||
|
if r.memory != nil {
|
||||||
|
memSel = r.memory.Select(ch.Text, ch.Chapter, stickyPrev, r.Pipeline.Context.GlossaryTokenBudget)
|
||||||
|
translatorInjection = renderGlossaryBlock(memSel.injected)
|
||||||
|
editorInjection = renderEditorConstraintBlock(memSel.injected)
|
||||||
|
activeIDs = memSel.activeIDs
|
||||||
|
}
|
||||||
|
|
||||||
|
for stageIdx, st := range r.Pipeline.Stages {
|
||||||
|
if flagged {
|
||||||
|
// An earlier stage flagged → this stage is not attempted or billed.
|
||||||
|
detail := fmt.Sprintf("skipped: an upstream stage was flagged (%s)", flagReason)
|
||||||
|
if err := r.Store.UpsertChunkStatus(store.ChunkStatus{
|
||||||
|
BookID: r.Book.BookID, Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Stage: st.Name,
|
||||||
|
SnapshotID: snapID, Disposition: string(DispSkipped), FlagReason: string(flagReason),
|
||||||
|
Detail: detail,
|
||||||
|
}); err != nil {
|
||||||
|
return out, activeIDs, fmt.Errorf("pipeline: record skipped chunk_status: %w", err)
|
||||||
|
}
|
||||||
|
out.Stages = append(out.Stages, StageResult{
|
||||||
|
Stage: st.Name, Role: st.Role, Model: st.Model,
|
||||||
|
Disposition: DispSkipped, FlagReason: flagReason, Detail: detail,
|
||||||
|
})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// Inject the glossary per role. The TRANSLATOR gets the src→dst block (its keys are
|
||||||
|
// matched against the source chunk); the monolingual EDITOR gets the CONFIRMED dst
|
||||||
|
// forms as target-consistency constraints (no source — decisions-log D1); every other
|
||||||
|
// stage gets none. Empty injection → the plain 2-message layout. The injection is a
|
||||||
|
// message, so it enters this stage's request_hash automatically (a resumed chunk
|
||||||
|
// reproduces it deterministically from the frozen bank).
|
||||||
|
injection := ""
|
||||||
|
switch st.Role {
|
||||||
|
case roleTranslator:
|
||||||
|
injection = translatorInjection
|
||||||
|
case roleEditor:
|
||||||
|
injection = editorInjection
|
||||||
|
}
|
||||||
|
sr, err := r.runStage(ctx, st, stageIdx, snapID, ch, prev, injection)
|
||||||
|
if err != nil {
|
||||||
|
return out, activeIDs, err
|
||||||
|
}
|
||||||
|
out.Stages = append(out.Stages, *sr)
|
||||||
|
out.CostUSD += sr.CostUSD
|
||||||
|
if sr.Disposition == DispFlagged {
|
||||||
|
flagged = true
|
||||||
|
flagReason = sr.FlagReason
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
prev = sr.Text
|
||||||
|
}
|
||||||
|
|
||||||
|
// Post-check the FINAL, exported output (E1): the reader sees the last stage's text
|
||||||
|
// (the editor's), not the translator draft — the monolingual editor can drift an
|
||||||
|
// approved term the translator got right, so checking the draft alone would miss it.
|
||||||
|
// Runs on the fresh OR fully-resumed chunk (both carry the final text through `prev`),
|
||||||
|
// so it is resume-reproducible. In the default FLAGGER mode a miss is recorded only in
|
||||||
|
// the retrieval-state (observability); with the opt-in gate a miss flags the chunk
|
||||||
|
// (glossary_miss). Skipped when a stage already flagged (no usable output to check).
|
||||||
|
var postMisses []postcheckMiss
|
||||||
|
outputChecked := false
|
||||||
|
if r.memory != nil && !flagged && prev != "" {
|
||||||
|
outputChecked = true
|
||||||
|
postMisses = r.memory.postcheck(memSel.injected, prev)
|
||||||
|
// The gate flips ONLY on CONFIRMED (approved) misses — an AMBIGUOUS miss is an
|
||||||
|
// unverified candidate the model may legitimately reject, so it must not discard a
|
||||||
|
// correct translation (external-review major #1).
|
||||||
|
if r.Pipeline.Gates.Glossary.PostcheckGate && countConfirmedMisses(postMisses) > 0 {
|
||||||
|
flagged = true
|
||||||
|
flagReason = FlagGlossaryMiss
|
||||||
|
r.Log.WarnContext(ctx, "glossary post-check gate flagged the chunk",
|
||||||
|
"chapter", ch.Chapter, "chunk", ch.ChunkIdx, "confirmed_misses", countConfirmedMisses(postMisses))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cheap deterministic style/number flaggers on the FINAL text (cheapgates.go): observability
|
||||||
|
// only, never a disposition. Runs on the same fresh-or-resumed output as the post-check, so it
|
||||||
|
// is resume-reproducible; skipped when a stage flagged (no usable output). Source is ch.Text
|
||||||
|
// (the 万/億 magnitude gate compares source↔output).
|
||||||
|
var cheap cheapGateResult
|
||||||
|
if !flagged && prev != "" {
|
||||||
|
cheap = runCheapGates(ch.Text, prev, r.cheapGateConfig())
|
||||||
|
if cheap.total() > 0 {
|
||||||
|
r.Log.InfoContext(ctx, "cheap style gates flagged the chunk (observability, not a gate)",
|
||||||
|
"chapter", ch.Chapter, "chunk", ch.ChunkIdx, "style_flags", cheap.total(),
|
||||||
|
"dialogue_dash", cheap.DialogueDash, "yo", cheap.YoInconsistent,
|
||||||
|
"translit_interj", cheap.TranslitInterj, "number_magnitude", cheap.NumberMagnitude)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Persist the per-chunk retrieval-state (registry gate #4: convert silent memory
|
||||||
|
// degradation into a loud, visible record) plus the cheap style-flag counts. Deterministic +
|
||||||
|
// idempotent, so a resume re-derives the identical row. Only when a glossary is materialized
|
||||||
|
// (always true in a normal run — seedGlossary sets an at-least-empty bank).
|
||||||
|
if r.memory != nil {
|
||||||
|
if err := r.persistRetrievalState(snapID, ch, memSel, postMisses, outputChecked, cheap); err != nil {
|
||||||
|
return out, activeIDs, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if flagged {
|
||||||
|
out.Disposition = DispFlagged
|
||||||
|
out.FlagReason = flagReason
|
||||||
|
out.FinalText = "" // garbage/refusal/glossary-miss never propagates to the next chunk or export
|
||||||
|
} else {
|
||||||
|
out.FinalText = prev
|
||||||
|
}
|
||||||
|
return out, activeIDs, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// persistRetrievalState writes the per-chunk observability record from the deterministic
|
||||||
|
// selection + the post-check result. n_exact_hits/n_sticky/n_ambiguous count the INJECTED
|
||||||
|
// records (what the model saw); spoiler/eviction are the dropped-and-logged totals;
|
||||||
|
// post-check misses are recorded only when the translator actually produced text.
|
||||||
|
func (r *Runner) persistRetrievalState(snapID string, ch Chunk, sel memorySelection, misses []postcheckMiss, outputChecked bool, cheap cheapGateResult) error {
|
||||||
|
rs := store.RetrievalState{
|
||||||
|
BookID: r.Book.BookID, Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, SnapshotID: snapID,
|
||||||
|
NStyleFlags: cheap.total(),
|
||||||
|
}
|
||||||
|
if cheap.total() > 0 {
|
||||||
|
if b, err := json.Marshal(cheap); err == nil {
|
||||||
|
rs.StyleDetail = string(b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, p := range sel.injected {
|
||||||
|
if p.via == "sticky" {
|
||||||
|
rs.NSticky++
|
||||||
|
} else {
|
||||||
|
rs.NExactHits++
|
||||||
|
}
|
||||||
|
if p.disp == memAmbiguous {
|
||||||
|
rs.NAmbiguousFlagged++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rs.NSpoilerBlocked = len(sel.rejected)
|
||||||
|
rs.NEvicted = len(sel.evicted)
|
||||||
|
if outputChecked {
|
||||||
|
// n_postcheck_miss is the CONFIRMED-miss count (the actionable consistency-failure
|
||||||
|
// signal, external-review #1); the detail carries ALL misses (confirmed AND the
|
||||||
|
// ambiguous forced-post-check ones, each disp-tagged) for the human.
|
||||||
|
rs.NPostcheckMiss = countConfirmedMisses(misses)
|
||||||
|
if len(misses) > 0 {
|
||||||
|
if b, err := json.Marshal(misses); err == nil {
|
||||||
|
rs.PostcheckDetail = string(b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ids := slices.Sorted(maps.Keys(sel.activeIDs))
|
||||||
|
if ids == nil {
|
||||||
|
// slices.Sorted даёт NIL на пустой карте → json.Marshal рендерил бы "null",
|
||||||
|
// а исторический формат персистентной колонки — "[]" (чанк без точных
|
||||||
|
// матчей — обычный кейс); держим байты стабильными (находка селфревью №4).
|
||||||
|
ids = []string{}
|
||||||
|
}
|
||||||
|
if b, err := json.Marshal(ids); err == nil {
|
||||||
|
rs.InjectedIDs = string(b)
|
||||||
|
}
|
||||||
|
return r.Store.UpsertRetrievalState(rs)
|
||||||
|
}
|
||||||
|
|
||||||
|
// cheapGateConfig builds the cheap-gate knobs from the book brief: the ё-policy and the
|
||||||
|
// lower-cased per-project interjection allowlist. Pure, no store access.
|
||||||
|
func (r *Runner) cheapGateConfig() cheapGateConfig {
|
||||||
|
allow := make(map[string]bool, len(r.Book.StyleAllowlist))
|
||||||
|
for _, s := range r.Book.StyleAllowlist {
|
||||||
|
allow[s] = true
|
||||||
|
}
|
||||||
|
return cheapGateConfig{yoPolicy: r.Book.YoPolicy, allowlist: allow}
|
||||||
|
}
|
||||||
107
backend/internal/pipeline/escalation.go
Normal file
107
backend/internal/pipeline/escalation.go
Normal file
|
|
@ -0,0 +1,107 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/config"
|
||||||
|
"textmachine/backend/internal/llm"
|
||||||
|
"textmachine/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// escalation.go: single-hop эскалация детерминированного контент-провала (D12/D3) —
|
||||||
|
// один хоп на другую модель под budget_usd, результат РЕ-гейтится, retry-бюджет не
|
||||||
|
// сбрасывается. Канал B (18+) Ф2 расширяет ИМЕННО этот файл (permissive-цепочки).
|
||||||
|
|
||||||
|
// errReserveCeiling is wrapped into the error runAttempt returns when a book/day USD
|
||||||
|
// ceiling denies a reservation. The PRIMARY path propagates it (the book durably
|
||||||
|
// pauses and resumes once the ceiling is raised, D4); the OPTIONAL escalation hop
|
||||||
|
// catches it and degrades to keeping the primary flag instead of aborting the whole
|
||||||
|
// book on every run (self-review finding).
|
||||||
|
var errReserveCeiling = errors.New("reserve ceiling reached")
|
||||||
|
|
||||||
|
// escalationBudgetRemains reports whether the book may still spend on a single-hop
|
||||||
|
// fallback draft: escalation is OPT-IN via escalation.budget_usd (0 = disabled, the
|
||||||
|
// boevoy default — a stage's escalate_to is inert until a premium budget is set), and
|
||||||
|
// capped at that budget summed over the book's escalation checkpoints (money-path
|
||||||
|
// durable, resume-safe). A PRE-HOP soft cap: the gate admits a hop while spent <
|
||||||
|
// budget and does NOT pre-estimate the hop's own cost, so a single hop may overshoot
|
||||||
|
// the budget by up to its full cost; the NEXT chunk's escalation is then denied. Size
|
||||||
|
// the budget with that worst case in mind — it bounds TOTAL escalation, not per-hop.
|
||||||
|
func (r *Runner) escalationBudgetRemains() (bool, error) {
|
||||||
|
budget := r.Pipeline.Escal.BudgetUSD
|
||||||
|
if budget <= 0 {
|
||||||
|
return false, nil
|
||||||
|
}
|
||||||
|
spent, err := r.Store.EscalationSpentUSD(r.Book.BookID)
|
||||||
|
if err != nil {
|
||||||
|
return false, err
|
||||||
|
}
|
||||||
|
return spent < budget, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// escalationOutcome is maybeEscalate's report to runStage. attempted is true ONLY
|
||||||
|
// when a hop actually executed (fresh or replayed from its checkpoint) — a chunk
|
||||||
|
// whose hop was skipped (not escalatable / no escalate_to / budget exhausted) or
|
||||||
|
// denied by a USD ceiling reports attempted=false, so runStage keeps the primary
|
||||||
|
// flag and records Escalated=false (the exact pre-extraction semantics).
|
||||||
|
type escalationOutcome struct {
|
||||||
|
attempted bool
|
||||||
|
fb stageAttempt // the fallback attempt; meaningful only when attempted
|
||||||
|
}
|
||||||
|
|
||||||
|
// maybeEscalate runs the single-hop escalation (D12 deterministic-content-failure
|
||||||
|
// class): a flag that a DIFFERENT model might fix (echo / excision / refusal) is
|
||||||
|
// routed ONCE to the stage's named fallback, under the book's escalation.budget_usd,
|
||||||
|
// and RE-GATED (classifyOutput runs on the fallback output too — §3.8). The editor
|
||||||
|
// declares no escalate_to → pinned per book (its style must not drift to a foreign
|
||||||
|
// model, D12/2605.13368). The fallback uses its OWN model (the request_hash axis), so
|
||||||
|
// its checkpoint never collides with the failed primary's, and it is exactly ONE call
|
||||||
|
// (not a fresh retry budget): total calls/chunk = primary attempts + 1 hop, never
|
||||||
|
// reset on the fallback (the LiteLLM #19985 retry×fallback blow-up guard). primary is
|
||||||
|
// the attempt loop's terminal (flagged) attempt.
|
||||||
|
func (r *Runner) maybeEscalate(ctx context.Context, st config.Stage, snapID string, ch Chunk, job *store.Job, baseMaxTokens int, msgs []llm.Message, primary stageAttempt) (escalationOutcome, error) {
|
||||||
|
var out escalationOutcome
|
||||||
|
if !primary.cls.Reason.escalatable() || st.EscalateTo == "" {
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
// Idempotency (self-review): if the fallback hop already happened its
|
||||||
|
// checkpoint exists and was already paid — REPLAY it for free regardless of the
|
||||||
|
// budget, so a crash AFTER the hop settled but BEFORE chunk_status was written
|
||||||
|
// re-serves it on resume rather than discarding a paid, successful translation
|
||||||
|
// and flipping the verdict OK→flagged. Only a FRESH hop is budget-gated. The
|
||||||
|
// hash mirrors runAttempt's for (model=EscalateTo, attempt=0, maxTokens=base).
|
||||||
|
fbHash := RequestHash(r.Book.BookID, ch.Chapter, ch.ChunkIdx, 0, st.Name, st.Role, st.EscalateTo,
|
||||||
|
st.Temperature, st.Reasoning, false, baseMaxTokens, snapID, msgs)
|
||||||
|
fbExists, err := r.Store.GetCheckpoint(fbHash)
|
||||||
|
if err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
mayHop := fbExists != nil
|
||||||
|
if !mayHop {
|
||||||
|
if mayHop, err = r.escalationBudgetRemains(); err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !mayHop {
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
fb, err := r.runAttempt(ctx, st, st.EscalateTo, snapID, ch, job, 0, baseMaxTokens, msgs, true)
|
||||||
|
if err != nil {
|
||||||
|
// An OPTIONAL hop that trips a USD ceiling must NOT abort the whole book
|
||||||
|
// (and re-abort on every resume): the chunk is already flagged, so keep
|
||||||
|
// the primary flag and continue. Any other infra error still aborts.
|
||||||
|
if !errors.Is(err, errReserveCeiling) {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
r.Log.WarnContext(ctx, "escalation hop denied by a USD ceiling; keeping the primary flag",
|
||||||
|
"stage", st.Name, "chapter", ch.Chapter, "chunk", ch.ChunkIdx, "reason", string(primary.cls.Reason))
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
out.attempted, out.fb = true, fb
|
||||||
|
r.Log.WarnContext(ctx, "stage escalated to a fallback model",
|
||||||
|
"stage", st.Name, "chapter", ch.Chapter, "chunk", ch.ChunkIdx,
|
||||||
|
"primary_reason", string(primary.cls.Reason), "fallback", st.EscalateTo,
|
||||||
|
"fallback_disposition", string(fb.cls.Reason.disposition()))
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
277
backend/internal/pipeline/golden_test.go
Normal file
277
backend/internal/pipeline/golden_test.go
Normal file
|
|
@ -0,0 +1,277 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/obs"
|
||||||
|
)
|
||||||
|
|
||||||
|
// golden_test.go — the refactor determinism guard (бэкенд-пакет №4, железное
|
||||||
|
// ограничение №1): on the static fixture book (testdata/golden/) it pins BIT-FOR-BIT
|
||||||
|
// the snapshotID + payload, every stage's request_hash and wire body, the chunk_status
|
||||||
|
// / retrieval_state rows and the final rendered texts — for a fresh run AND a resume
|
||||||
|
// run. Any refactoring step that changes a single byte of the wire or of a resolved
|
||||||
|
// verdict fails this test loudly: a changed request_hash = a missed checkpoint =
|
||||||
|
// --resnapshot = the whole book re-billed (D15), which no code-health win can pay for.
|
||||||
|
//
|
||||||
|
// The fixture deliberately walks the money-relevant paths in one book: a 2-chunk
|
||||||
|
// chapter (sticky window + injection), a spoiler-blocked term (since_ch: 2), a nested
|
||||||
|
// glossary key (紋章 in 竜の紋章 → collision/AMBIGUOUS machinery), a CJK-echo draft
|
||||||
|
// cured by the single-hop escalation re-gate (ch2), and a hard refusal where the
|
||||||
|
// fallback also refuses → flag + skipped edit + exit 2 (ch3).
|
||||||
|
//
|
||||||
|
// Update (ONLY on a deliberate, ratified behaviour change — never to "fix" a red
|
||||||
|
// refactor): TM_UPDATE_GOLDEN=1 go test ./internal/pipeline/ -run TestGolden
|
||||||
|
const goldenFile = "testdata/golden/capture.golden"
|
||||||
|
|
||||||
|
// goldenEchoMarker / goldenRefusalMarker are unique substrings of chapters 2/3 of the
|
||||||
|
// fixture source; the mock provider keys its echo/refusal behaviour off them.
|
||||||
|
const (
|
||||||
|
goldenEchoMarker = "響動計画"
|
||||||
|
goldenRefusalMarker = "拒絶計画"
|
||||||
|
)
|
||||||
|
|
||||||
|
// goldenRespond is the deterministic mock provider brain. The response text is a pure
|
||||||
|
// function of the request body (no time, no randomness): default drafts/edits carry a
|
||||||
|
// body-hash suffix so every chunk×stage output is distinct, plus glossary dst forms so
|
||||||
|
// the post-check exercises both hit and miss.
|
||||||
|
func goldenRespond(body string) (text, finish string) {
|
||||||
|
var req struct {
|
||||||
|
Model string `json:"model"`
|
||||||
|
}
|
||||||
|
_ = json.Unmarshal([]byte(body), &req)
|
||||||
|
sum := sha256.Sum256([]byte(body))
|
||||||
|
tag := hex.EncodeToString(sum[:])[:12]
|
||||||
|
switch {
|
||||||
|
case strings.Contains(body, goldenRefusalMarker):
|
||||||
|
// Both the primary and the fallback refuse → the chunk stays flagged (1 hop).
|
||||||
|
return "Не могу помочь с этим фрагментом.", "refusal"
|
||||||
|
case strings.Contains(body, goldenEchoMarker) && !isEditBody(body):
|
||||||
|
if req.Model == "fake-fallback" {
|
||||||
|
// The escalation hop returns a clean translation → the re-gate passes it.
|
||||||
|
return "ЭСКАЛАЦИОННЫЙ ПЕРЕВОД " + tag + ". Судзуки вынес Драконью печать из Академии магии.", "stop"
|
||||||
|
}
|
||||||
|
// The primary draft echoes CJK instead of translating → cjk_artifact.
|
||||||
|
return "夜明け前、鈴木は再び書庫に戻り、竜の紋章を布に包んで持ち出した。", "stop"
|
||||||
|
case isEditBody(body):
|
||||||
|
return "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД " + tag + ". Судзуки шёл по коридорам Академии магии.", "stop"
|
||||||
|
default:
|
||||||
|
return "ЧЕРНОВИК ПЕРЕВОДА " + tag + ". Судзуки шёл по коридорам Академии магии.", "stop"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// setupGoldenProject copies the static fixture into a temp dir and writes models.yaml
|
||||||
|
// (the only runtime-generated file: it carries the live mock URL and a fresh
|
||||||
|
// prices_checked date — neither enters the snapshot, request hashes or wire bodies).
|
||||||
|
func setupGoldenProject(t *testing.T, providerURL string) string {
|
||||||
|
t.Helper()
|
||||||
|
dir := t.TempDir()
|
||||||
|
src := filepath.Join("testdata", "golden")
|
||||||
|
for _, f := range []string{"book.yaml", "pipeline.yaml", "source.txt", "glossary-seed.yaml",
|
||||||
|
filepath.Join("prompts", "translator.md"), filepath.Join("prompts", "editor.md")} {
|
||||||
|
data, err := os.ReadFile(filepath.Join(src, f))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
writeFile(t, filepath.Join(dir, f), string(data))
|
||||||
|
}
|
||||||
|
writeFile(t, filepath.Join(dir, "models.yaml"), fmt.Sprintf(`
|
||||||
|
prices_checked: %q
|
||||||
|
default_model: fake-model
|
||||||
|
providers:
|
||||||
|
fake:
|
||||||
|
kind: openai
|
||||||
|
base_url: %q
|
||||||
|
timeouts: { attempt_s: 5, max_attempts: 2, backoff_cap_s: 1 }
|
||||||
|
models:
|
||||||
|
fake-model:
|
||||||
|
provider: fake
|
||||||
|
price: { input_per_m: 1.0, cached_per_m: 0.1, cache_write_per_m: 0, output_per_m: 2.0 }
|
||||||
|
fake-fallback:
|
||||||
|
provider: fake
|
||||||
|
price: { input_per_m: 3.0, cached_per_m: 0.3, cache_write_per_m: 0, output_per_m: 6.0 }
|
||||||
|
`, time.Now().UTC().Format("2006-01-02"), providerURL))
|
||||||
|
return filepath.Join(dir, "book.yaml")
|
||||||
|
}
|
||||||
|
|
||||||
|
// captureGolden renders the full deterministic state of a run as one canonical text
|
||||||
|
// blob. Everything time/host-dependent (latency, timestamps, temp paths, provider
|
||||||
|
// URL) is deliberately excluded; everything byte/verdict-relevant is included.
|
||||||
|
func captureGolden(t *testing.T, label string, r *Runner, res *BookResult, wireBodies []string) string {
|
||||||
|
t.Helper()
|
||||||
|
var b strings.Builder
|
||||||
|
w := func(format string, args ...any) { fmt.Fprintf(&b, format+"\n", args...) }
|
||||||
|
fl := func(f float64) string { return strconv.FormatFloat(f, 'f', -1, 64) }
|
||||||
|
|
||||||
|
w("==== run %s ====", label)
|
||||||
|
snapID, payload, err := r.snapshotID()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
w("snapshot_id: %s", snapID)
|
||||||
|
w("snapshot_payload: %s", payload)
|
||||||
|
w("brief_hash: %s", r.Book.BriefHash())
|
||||||
|
w("memory_version: %s", r.memoryVersion())
|
||||||
|
|
||||||
|
w("-- book result --")
|
||||||
|
w("chunks=%d flagged=%d exit=%d total_usd=%s", len(res.Chunks), res.Flagged, res.ExitCode(), fl(res.TotalUSD))
|
||||||
|
for _, ch := range res.Chunks {
|
||||||
|
w("chunk ch%d/%d disposition=%s flag=%q final_text=%q cost=%s",
|
||||||
|
ch.Chapter, ch.ChunkIdx, ch.Disposition, ch.FlagReason, ch.FinalText, fl(ch.CostUSD))
|
||||||
|
for _, st := range ch.Stages {
|
||||||
|
w(" stage=%s role=%s model=%s resume=%t disp=%s flag=%q attempts=%d escalated=%t esc_model=%q finish=%q cum_usd=%s detail=%q",
|
||||||
|
st.Stage, st.Role, st.Model, st.FromResume, st.Disposition, st.FlagReason,
|
||||||
|
st.Attempts, st.Escalated, st.EscalationModel, st.FinishReason, fl(st.CumCostUSD), st.Detail)
|
||||||
|
w(" stage_text=%q", st.Text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
w("-- chunk_status --")
|
||||||
|
css, err := r.Store.ChunkStatusesForBook(r.Book.BookID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
sort.Slice(css, func(i, j int) bool {
|
||||||
|
a, c := css[i], css[j]
|
||||||
|
if a.Chapter != c.Chapter {
|
||||||
|
return a.Chapter < c.Chapter
|
||||||
|
}
|
||||||
|
if a.ChunkIdx != c.ChunkIdx {
|
||||||
|
return a.ChunkIdx < c.ChunkIdx
|
||||||
|
}
|
||||||
|
return a.Stage < c.Stage
|
||||||
|
})
|
||||||
|
for _, cs := range css {
|
||||||
|
w("ch%d/%d %s snap_match=%t content_hash=%s disp=%s flag=%q attempts=%d final_hash=%s cost=%s escalated=%t esc_model=%q detail=%q",
|
||||||
|
cs.Chapter, cs.ChunkIdx, cs.Stage, cs.SnapshotID == snapID, cs.ContentHash, cs.Disposition,
|
||||||
|
cs.FlagReason, cs.Attempts, cs.FinalHash, fl(cs.CostUSD), cs.Escalated, cs.EscalationModel, cs.Detail)
|
||||||
|
}
|
||||||
|
|
||||||
|
w("-- request_log (insertion order) --")
|
||||||
|
rls, err := r.Store.RequestLogRows(r.Book.BookID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, rl := range rls {
|
||||||
|
w("ch%d/%d %s role=%s req=%s actual=%s hash=%s tm_hit=%d ok=%d finish=%q degraded=%q cost=%s tokens=%d/%d/%d/%d/%d err=%q",
|
||||||
|
rl.Chapter, rl.ChunkIdx, rl.Stage, rl.Role, rl.ModelRequested, rl.ModelActual, rl.RequestHash,
|
||||||
|
rl.TMHit, rl.OK, rl.FinishReason, rl.Degraded, fl(rl.CostUSD),
|
||||||
|
rl.PromptTokens, rl.CachedTokens, rl.CacheCreationTokens, rl.CompletionTokens, rl.ReasoningTokens, rl.Err)
|
||||||
|
}
|
||||||
|
|
||||||
|
w("-- retrieval_state --")
|
||||||
|
rss, err := r.Store.RetrievalStatesForBook(r.Book.BookID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
sort.Slice(rss, func(i, j int) bool {
|
||||||
|
if rss[i].Chapter != rss[j].Chapter {
|
||||||
|
return rss[i].Chapter < rss[j].Chapter
|
||||||
|
}
|
||||||
|
return rss[i].ChunkIdx < rss[j].ChunkIdx
|
||||||
|
})
|
||||||
|
for _, rs := range rss {
|
||||||
|
w("ch%d/%d snap_match=%t exact=%d sticky=%d ambiguous=%d spoiler=%d evicted=%d postcheck_miss=%d style_flags=%d",
|
||||||
|
rs.Chapter, rs.ChunkIdx, rs.SnapshotID == snapID, rs.NExactHits, rs.NSticky, rs.NAmbiguousFlagged,
|
||||||
|
rs.NSpoilerBlocked, rs.NEvicted, rs.NPostcheckMiss, rs.NStyleFlags)
|
||||||
|
w(" injected_ids=%s postcheck_detail=%s style_detail=%s", rs.InjectedIDs, rs.PostcheckDetail, rs.StyleDetail)
|
||||||
|
}
|
||||||
|
|
||||||
|
w("-- wire bodies (call order) --")
|
||||||
|
for i, body := range wireBodies {
|
||||||
|
w("[%d] %s", i, body)
|
||||||
|
}
|
||||||
|
return b.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestGoldenDeterminism is the refactor guard: fresh run + resume run against the
|
||||||
|
// static fixture must match testdata/golden/capture.golden byte-for-byte.
|
||||||
|
func TestGoldenDeterminism(t *testing.T) {
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
body, _ := io.ReadAll(r.Body)
|
||||||
|
rec.record(string(body))
|
||||||
|
text, finish := goldenRespond(string(body))
|
||||||
|
tb, _ := json.Marshal(text)
|
||||||
|
var req struct {
|
||||||
|
Model string `json:"model"`
|
||||||
|
}
|
||||||
|
_ = json.Unmarshal(body, &req)
|
||||||
|
fmt.Fprintf(w, `{"id":"fake","model":%q,"choices":[{"message":{"content":%s},"finish_reason":%q}],
|
||||||
|
"usage":{"prompt_tokens":1000,"completion_tokens":500,"prompt_tokens_details":{"cached_tokens":200}}}`,
|
||||||
|
req.Model, tb, finish)
|
||||||
|
}))
|
||||||
|
defer srv.Close()
|
||||||
|
bookPath := setupGoldenProject(t, srv.URL)
|
||||||
|
ctx := obs.WithReqInfo(t.Context(), obs.ReqInfo{TraceID: "golden-trace"})
|
||||||
|
|
||||||
|
// Run 1: fresh book, every call is a fresh reserve→call→settle.
|
||||||
|
r1 := newRunner(t, bookPath)
|
||||||
|
res1, err := r1.TranslateBook(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
capture := captureGolden(t, "1 (fresh)", r1, res1, rec.all())
|
||||||
|
freshCalls := rec.count()
|
||||||
|
r1.Close()
|
||||||
|
|
||||||
|
// Run 2: resume from checkpoints/chunk_status — must make ZERO provider calls
|
||||||
|
// and reproduce the identical texts/verdicts (the wire-bodies section stays as
|
||||||
|
// captured in run 1; a resume that re-called the provider would append to it).
|
||||||
|
r2 := newRunner(t, bookPath)
|
||||||
|
defer r2.Close()
|
||||||
|
res2, err := r2.TranslateBook(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if rec.count() != freshCalls {
|
||||||
|
t.Fatalf("resume made %d extra provider calls", rec.count()-freshCalls)
|
||||||
|
}
|
||||||
|
capture += captureGolden(t, "2 (resume)", r2, res2, nil)
|
||||||
|
|
||||||
|
if os.Getenv("TM_UPDATE_GOLDEN") == "1" {
|
||||||
|
if err := os.WriteFile(goldenFile, []byte(capture), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("golden updated: %s (%d bytes, %d provider calls)", goldenFile, len(capture), freshCalls)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
want, err := os.ReadFile(goldenFile)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("golden file missing (generate once with TM_UPDATE_GOLDEN=1): %v", err)
|
||||||
|
}
|
||||||
|
if string(want) != capture {
|
||||||
|
t.Fatalf("golden mismatch: the run's bytes diverged from %s.\nЭто значит, что рефакторинг изменил wire-байты, хеши или вердикты — "+
|
||||||
|
"любое такое изменение = --resnapshot = переоплата книги (D15). Найди и убери причину; обновлять golden можно только "+
|
||||||
|
"на осознанной, ратифицированной смене поведения.\n%s", goldenFile, diffHint(string(want), capture))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// diffHint points a human at the first diverging line — the full capture is too big
|
||||||
|
// for a useful t.Fatalf dump.
|
||||||
|
func diffHint(want, got string) string {
|
||||||
|
wl, gl := strings.Split(want, "\n"), strings.Split(got, "\n")
|
||||||
|
n := len(wl)
|
||||||
|
if len(gl) < n {
|
||||||
|
n = len(gl)
|
||||||
|
}
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
if wl[i] != gl[i] {
|
||||||
|
return fmt.Sprintf("first divergence at line %d:\n want: %s\n got: %s", i+1, wl[i], gl[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("line counts differ: want %d, got %d", len(wl), len(gl))
|
||||||
|
}
|
||||||
|
|
@ -23,7 +23,7 @@ import (
|
||||||
// into an ordered list of per-chapter NORMALIZED text plus the ruby/furigana
|
// into an ordered list of per-chapter NORMALIZED text plus the ruby/furigana
|
||||||
// readings captured on the way (04-unhappy §4 / D9). It is the ONLY place that
|
// readings captured on the way (04-unhappy §4 / D9). It is the ONLY place that
|
||||||
// reads the source; the runner then feeds doc.Chapters to SplitChunks and persists
|
// reads the source; the runner then feeds doc.Chapters to SplitChunks and persists
|
||||||
// doc.Ruby (runner.go). It is offline and deterministic — no LLM, no time/rand — so
|
// doc.Ruby (seeding.go). It is offline and deterministic — no LLM, no time/rand — so
|
||||||
// the whole path is $0 and reproducible.
|
// the whole path is $0 and reproducible.
|
||||||
//
|
//
|
||||||
// epub v1 is a text extraction (02-mvp:25): read the chapters in spine order,
|
// epub v1 is a text extraction (02-mvp:25): read the chapters in spine order,
|
||||||
|
|
@ -41,7 +41,7 @@ import (
|
||||||
const chapterSep = "\f"
|
const chapterSep = "\f"
|
||||||
|
|
||||||
// RubyReading is one captured (base, reading) pair and the chapter it appeared in.
|
// RubyReading is one captured (base, reading) pair and the chapter it appeared in.
|
||||||
// The pipeline aggregates these (min chapter, count) before persisting (runner.go).
|
// The pipeline aggregates these (min chapter, count) before persisting (seeding.go).
|
||||||
type RubyReading struct {
|
type RubyReading struct {
|
||||||
Base string // the ruby body: the kanji/base surface form
|
Base string // the ruby body: the kanji/base surface form
|
||||||
Reading string // the <rt> reading (furigana)
|
Reading string // the <rt> reading (furigana)
|
||||||
|
|
|
||||||
|
|
@ -25,7 +25,7 @@ import (
|
||||||
// the post-check substring test: NFC → Unicode lower → ё→е fold.
|
// the post-check substring test: NFC → Unicode lower → ё→е fold.
|
||||||
//
|
//
|
||||||
// The whole artifact is versioned by memoryNormVersion, which is FOLDED INTO the
|
// The whole artifact is versioned by memoryNormVersion, which is FOLDED INTO the
|
||||||
// snapshot via memoryVersion() (runner.go): a change to the algorithm OR to the
|
// snapshot via memoryVersion() (snapshot.go): a change to the algorithm OR to the
|
||||||
// embedded trad→simp table is a loud --resnapshot, never a silent match-behaviour
|
// embedded trad→simp table is a loud --resnapshot, never a silent match-behaviour
|
||||||
// change on already-checkpointed chunks. Determinism is load-bearing — the injected
|
// change on already-checkpointed chunks. Determinism is load-bearing — the injected
|
||||||
// glossary block is a pure function of (frozen glossary, normalized chunk), and it
|
// glossary block is a pure function of (frozen glossary, normalized chunk), and it
|
||||||
|
|
@ -47,7 +47,7 @@ var (
|
||||||
trad2simp map[rune]rune
|
trad2simp map[rune]rune
|
||||||
// memoryNormVersion is the snapshot-load-bearing version of the entire
|
// memoryNormVersion is the snapshot-load-bearing version of the entire
|
||||||
// normalization artifact: the algorithm tag + a content hash of the embedded
|
// normalization artifact: the algorithm tag + a content hash of the embedded
|
||||||
// trad→simp table. memoryVersion() (runner.go) folds it into the job snapshot.
|
// trad→simp table. memoryVersion() (snapshot.go) folds it into the job snapshot.
|
||||||
memoryNormVersion string
|
memoryNormVersion string
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -4,6 +4,8 @@ import (
|
||||||
"crypto/sha256"
|
"crypto/sha256"
|
||||||
"encoding/hex"
|
"encoding/hex"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"maps"
|
||||||
|
"slices"
|
||||||
"sort"
|
"sort"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
@ -715,11 +717,7 @@ func injectivityCollisions(rows []store.GlossaryEntry) []string {
|
||||||
}
|
}
|
||||||
var out []string
|
var out []string
|
||||||
// Deterministic order: iterate a sorted key list, not the map.
|
// Deterministic order: iterate a sorted key list, not the map.
|
||||||
keys := make([]string, 0, len(bySurface))
|
keys := slices.Sorted(maps.Keys(bySurface))
|
||||||
for k := range bySurface {
|
|
||||||
keys = append(keys, k)
|
|
||||||
}
|
|
||||||
sort.Strings(keys)
|
|
||||||
for _, k := range keys {
|
for _, k := range keys {
|
||||||
srcs := bySurface[k]
|
srcs := bySurface[k]
|
||||||
if len(distinct(srcs)) > 1 {
|
if len(distinct(srcs)) > 1 {
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,9 @@ package pipeline
|
||||||
import (
|
import (
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"maps"
|
||||||
"os"
|
"os"
|
||||||
|
"slices"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"unicode"
|
"unicode"
|
||||||
|
|
@ -206,11 +208,7 @@ func approvedSharedKeyCollisions(entries []store.GlossaryEntry) []string {
|
||||||
owners[k] = append(owners[k], i)
|
owners[k] = append(owners[k], i)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
keys := make([]string, 0, len(owners))
|
keys := slices.Sorted(maps.Keys(owners))
|
||||||
for k := range owners {
|
|
||||||
keys = append(keys, k)
|
|
||||||
}
|
|
||||||
sort.Strings(keys)
|
|
||||||
var out []string
|
var out []string
|
||||||
seenPair := map[[2]int]bool{}
|
seenPair := map[[2]int]bool{}
|
||||||
for _, k := range keys {
|
for _, k := range keys {
|
||||||
|
|
|
||||||
|
|
@ -222,7 +222,7 @@ func RequestHash(bookID string, chapter, chunkIdx, attempt int, stage, role, mod
|
||||||
}
|
}
|
||||||
|
|
||||||
// msgsContentHash is a content signature of the rendered messages, INDEPENDENT
|
// msgsContentHash is a content signature of the rendered messages, INDEPENDENT
|
||||||
// of attempt/max_tokens. It guards the chunk_status resume fast-path (runner.go):
|
// of attempt/max_tokens. It guards the chunk_status resume fast-path (stagerun.go):
|
||||||
// the source bytes are NOT folded into the snapshot (only the chunker RULES and
|
// the source bytes are NOT folded into the snapshot (only the chunker RULES and
|
||||||
// the semantic brief are), so a positional chunk_status row must be re-validated
|
// the semantic brief are), so a positional chunk_status row must be re-validated
|
||||||
// against the current rendered content — otherwise an edited source would serve
|
// against the current rendered content — otherwise an edited source would serve
|
||||||
|
|
|
||||||
70
backend/internal/pipeline/resume.go
Normal file
70
backend/internal/pipeline/resume.go
Normal file
|
|
@ -0,0 +1,70 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"textmachine/backend/internal/config"
|
||||||
|
"textmachine/backend/internal/llm"
|
||||||
|
"textmachine/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// resume.go: обслуживание уже РЕШЁННОГО чанк×стадии из chunk_status/чекпоинтов —
|
||||||
|
// $0, без вызова провайдера; ok-стадия отдаёт авторитетный текст следующей стадии,
|
||||||
|
// терминальный флаг никогда не переатакуется (анти-ведж №1).
|
||||||
|
|
||||||
|
// resumeFromChunkStatus serves a chunk×stage whose disposition is already
|
||||||
|
// resolved (read before any render — no re-billing). An ok stage returns its
|
||||||
|
// authoritative checkpoint text to feed the next stage; a flagged stage returns
|
||||||
|
// the flag WITHOUT re-attacking (a terminal flag never re-enters the paid loop).
|
||||||
|
func (r *Runner) resumeFromChunkStatus(ctx context.Context, st config.Stage, ch Chunk, cs *store.ChunkStatus) (*StageResult, error) {
|
||||||
|
sr := &StageResult{
|
||||||
|
Stage: st.Name, Role: st.Role, Model: st.Model, FromResume: true,
|
||||||
|
CostUSD: 0, CumCostUSD: cs.CostUSD,
|
||||||
|
Disposition: Disposition(cs.Disposition), FlagReason: FlagReason(cs.FlagReason),
|
||||||
|
Detail: cs.Detail, Attempts: cs.Attempts,
|
||||||
|
// Escalation attribution now travels on chunk_status (D15.3), so it is preserved
|
||||||
|
// across resume for BOTH an ok escalation AND a flagged-then-escalated chunk (the
|
||||||
|
// status read-model reads it without re-opening the checkpoint).
|
||||||
|
Escalated: cs.Escalated, EscalationModel: cs.EscalationModel,
|
||||||
|
}
|
||||||
|
if cs.Disposition == string(DispOK) {
|
||||||
|
cp, err := r.Store.GetCheckpoint(cs.FinalHash)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("pipeline: read final checkpoint %.12s for %s/ch%d/chunk%d/%s: %w",
|
||||||
|
cs.FinalHash, r.Book.BookID, ch.Chapter, ch.ChunkIdx, st.Name, err)
|
||||||
|
}
|
||||||
|
if cp == nil || strings.TrimSpace(cp.ResponseText) == "" {
|
||||||
|
// chunk_status ok is only written AFTER its checkpoint is durably
|
||||||
|
// settled, so this is a torn/tampered store — fail loud rather than
|
||||||
|
// feed the next stage an empty draft.
|
||||||
|
return nil, fmt.Errorf("pipeline: chunk_status ok for %s/ch%d/chunk%d/%s references checkpoint %.12s which is missing/empty — inconsistent store",
|
||||||
|
r.Book.BookID, ch.Chapter, ch.ChunkIdx, st.Name, cs.FinalHash)
|
||||||
|
}
|
||||||
|
var usage llm.Usage
|
||||||
|
if uerr := json.Unmarshal([]byte(cp.UsageJSON), &usage); uerr != nil {
|
||||||
|
r.Log.DebugContext(ctx, "checkpoint usage_json unreadable; tokens report as zero", "hash", cs.FinalHash[:12], "err", uerr)
|
||||||
|
}
|
||||||
|
sr.Usage = usage
|
||||||
|
sr.Model = cp.ModelActual
|
||||||
|
sr.FinishReason = cp.FinishReason
|
||||||
|
sr.Text = cp.ResponseText
|
||||||
|
}
|
||||||
|
rl := r.baseRequestLog(st, ch, st.Model, cs.FinalHash)
|
||||||
|
rl.ModelActual = sr.Model
|
||||||
|
rl.TMHit, rl.OK = true, cs.Disposition == string(DispOK)
|
||||||
|
rl.FinishReason, rl.Degraded = sr.FinishReason, nonOKTag(cs.Disposition, cs.FlagReason)
|
||||||
|
r.Store.LogRequest(ctx, r.Log, rl)
|
||||||
|
r.Log.InfoContext(ctx, "stage resolved from chunk_status", "stage", st.Name,
|
||||||
|
"chapter", ch.Chapter, "chunk", ch.ChunkIdx, "disposition", cs.Disposition, "reason", cs.FlagReason)
|
||||||
|
return sr, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// nonOKTag is the request_log `degraded` value for a resumed disposition row.
|
||||||
|
func nonOKTag(disposition, reason string) string {
|
||||||
|
if disposition == string(DispOK) {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return reason
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load diff
78
backend/internal/pipeline/runner_readonly_test.go
Normal file
78
backend/internal/pipeline/runner_readonly_test.go
Normal file
|
|
@ -0,0 +1,78 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"textmachine/backend/internal/obs"
|
||||||
|
)
|
||||||
|
|
||||||
|
// runner_readonly_test.go pins the operator-visibility contract (пакет №4): the
|
||||||
|
// read-only runner (tmctl status/report) must work WHILE a write runner owns the
|
||||||
|
// project — «оператор слеп ровно когда висит 300-я глава» was the top smoke
|
||||||
|
// finding, caused by status/report sharing the writer's exclusive flock.
|
||||||
|
|
||||||
|
// TestReadOnlyRunnerWorksDuringLiveRun: a NewReadOnlyRunner must open and project
|
||||||
|
// Status while a NewRunner (write path, flock held) is alive on the same book.
|
||||||
|
func TestReadOnlyRunnerWorksDuringLiveRun(t *testing.T) {
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := newJSONProvider(rec, draftEdit)
|
||||||
|
defer srv.Close()
|
||||||
|
bookPath := setupProject(t, srv.URL)
|
||||||
|
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
|
||||||
|
|
||||||
|
// The write runner owns the project (flock held for its whole lifetime) and
|
||||||
|
// has completed a run, so there are rows to project.
|
||||||
|
w := newRunner(t, bookPath)
|
||||||
|
defer w.Close()
|
||||||
|
if _, err := w.TranslateBook(ctx); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Simulates `tmctl status` from a second shell mid-run: the write runner is
|
||||||
|
// still open. The old store.Open path failed here with "project database is
|
||||||
|
// in use by another tmctl process".
|
||||||
|
ro, err := NewReadOnlyRunner(bookPath, obs.NewLogger())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read-only runner must open during a live run: %v", err)
|
||||||
|
}
|
||||||
|
defer ro.Close()
|
||||||
|
|
||||||
|
rep, err := ro.Status(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if rep.TotalChunks != 1 || rep.Done != 1 || rep.Flagged != 0 {
|
||||||
|
t.Fatalf("status projection over a live-locked book = %+v", rep)
|
||||||
|
}
|
||||||
|
if rep.CommittedUSD == 0 {
|
||||||
|
t.Fatal("status must see the writer's committed spend")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReadOnlyRunnerFirstTouchCreates: before the first run there is no DB file —
|
||||||
|
// the read-only runner falls back to the creating Open (мигрирует пустую базу), so
|
||||||
|
// «status до первого прогона» keeps showing 0/N pending instead of erroring.
|
||||||
|
func TestReadOnlyRunnerFirstTouchCreates(t *testing.T) {
|
||||||
|
rec := &reqRec{}
|
||||||
|
srv := newJSONProvider(rec, draftEdit)
|
||||||
|
defer srv.Close()
|
||||||
|
bookPath := setupProject(t, srv.URL)
|
||||||
|
ctx := obs.WithReqInfo(context.Background(), obs.ReqInfo{TraceID: obs.NewTraceID()})
|
||||||
|
|
||||||
|
ro, err := NewReadOnlyRunner(bookPath, obs.NewLogger())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer ro.Close()
|
||||||
|
rep, err := ro.Status(ctx)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if rep.TotalChunks != 1 || rep.Done != 0 || rep.Pending != 1 {
|
||||||
|
t.Fatalf("pre-run status = %+v", rep)
|
||||||
|
}
|
||||||
|
if rec.count() != 0 {
|
||||||
|
t.Fatalf("read-only status must make zero provider calls, got %d", rec.count())
|
||||||
|
}
|
||||||
|
}
|
||||||
118
backend/internal/pipeline/seeding.go
Normal file
118
backend/internal/pipeline/seeding.go
Normal file
|
|
@ -0,0 +1,118 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"textmachine/backend/internal/store"
|
||||||
|
)
|
||||||
|
|
||||||
|
// seeding.go: детерминированные $0-прелюдии джобы — REPLACE-сид глоссария из
|
||||||
|
// ручного файла + ruby-чтений (D16.4) и персист ruby-агрегатов. Выполняются ДО
|
||||||
|
// snapshotID: замороженные approved-строки входят в memoryVersion (F1).
|
||||||
|
|
||||||
|
// seedGlossary REPLACES the book's glossary from its deterministic inputs — the manual
|
||||||
|
// seed file (curated approved/draft terms) and the captured ruby readings (classified
|
||||||
|
// into auto candidates) — then MATERIALIZES the frozen bank for this job (r.memory). Run
|
||||||
|
// once before snapshotID: the frozen APPROVED rows are hashed into memoryVersion (F1), so
|
||||||
|
// editing the seed is a loud --resnapshot. Idempotent (full replace), $0, no LLM. B2
|
||||||
|
// approved dst-collisions are logged (not fatal — some collisions are legitimate).
|
||||||
|
func (r *Runner) seedGlossary(ctx context.Context) error {
|
||||||
|
var entries []store.GlossaryEntry
|
||||||
|
if r.Book.GlossarySeed != "" {
|
||||||
|
seed, err := loadGlossarySeed(r.Book.GlossarySeed)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
entries = append(entries, seed...)
|
||||||
|
}
|
||||||
|
manualSrcs := map[string]bool{}
|
||||||
|
for _, e := range entries {
|
||||||
|
manualSrcs[e.Src] = true
|
||||||
|
}
|
||||||
|
ruby, err := r.Store.RubyReadingsForBook(r.Book.BookID)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("pipeline: read ruby readings for %s: %w", r.Book.BookID, err)
|
||||||
|
}
|
||||||
|
// D16.4: attach a manual term's kana ruby-reading as an alias (kana spelling matchable)
|
||||||
|
// BEFORE appending the auto-candidates, so it only touches the curated manual entries. A
|
||||||
|
// reading that would collide with a different seeded term (homophone) is skipped+logged,
|
||||||
|
// not attached (which would fail the book loud on an alias the operator cannot edit out).
|
||||||
|
if skipped := attachRubyAliasesToManual(entries, ruby); len(skipped) > 0 {
|
||||||
|
r.Log.WarnContext(ctx, "ruby kana-alias skipped as a homophone collision (kana form left unmatchable; disambiguate in the seed if needed)",
|
||||||
|
"skipped", strings.Join(skipped, "; "))
|
||||||
|
}
|
||||||
|
entries = append(entries, rubyToCandidates(ruby, manualSrcs)...)
|
||||||
|
for i := range entries {
|
||||||
|
entries[i].BookID = r.Book.BookID
|
||||||
|
}
|
||||||
|
// Task-6 re-audit: fail loud on a firing key shared by two DIFFERENT approved terms with
|
||||||
|
// different dst + overlapping windows (the alias generalization of the D16.1 polysemy
|
||||||
|
// livelock) — checked over the FULL entry set (incl. ruby-attached aliases) before persisting.
|
||||||
|
if cols := approvedSharedKeyCollisions(entries); len(cols) > 0 {
|
||||||
|
return fmt.Errorf("pipeline: glossary shared-key collisions (A2 / D16.1 livelock class):\n - %s", strings.Join(cols, "\n - "))
|
||||||
|
}
|
||||||
|
if err := r.Store.ReplaceGlossary(r.Book.BookID, entries); err != nil {
|
||||||
|
return fmt.Errorf("pipeline: replace glossary for %s (%d entries): %w", r.Book.BookID, len(entries), err)
|
||||||
|
}
|
||||||
|
rows, err := r.Store.GlossaryForBook(r.Book.BookID)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("pipeline: read glossary for %s: %w", r.Book.BookID, err)
|
||||||
|
}
|
||||||
|
r.memory = materializeMemory(rows, r.Pipeline.Gates.Glossary.PostcheckGate)
|
||||||
|
if cols := injectivityCollisions(rows); len(cols) > 0 {
|
||||||
|
r.Log.WarnContext(ctx, "glossary approved dst-collisions (B2: two source terms share one Russian surface — the reader cannot tell them apart)",
|
||||||
|
"collisions", strings.Join(cols, "; "))
|
||||||
|
}
|
||||||
|
r.Log.InfoContext(ctx, "glossary materialized", "book", r.Book.BookID,
|
||||||
|
"entries", len(rows), "memory_version", r.memory.Version()[:12])
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// persistRuby aggregates the ingested ruby occurrences into one row per
|
||||||
|
// (base, reading) — first_chapter = MIN, occurrences = full-book count — and
|
||||||
|
// REPLACES the book's whole ruby set (store.ReplaceRubyReadings). Idempotent: the
|
||||||
|
// aggregation is recomputed identically on every ingest, so a resume re-writes the
|
||||||
|
// same rows; a source edit converges every column — a pair removed by the edit
|
||||||
|
// disappears instead of lingering as a phantom (external-review #6). Called
|
||||||
|
// unconditionally (even for zero readings — a txt book or a source that dropped its
|
||||||
|
// furigana) so the full-replace clears any stale rows. The write order is sorted for
|
||||||
|
// deterministic, test-stable behavior. Memory v2 (шаг 4) consumes ruby_readings into
|
||||||
|
// a glossary name-lock (D9); nothing here injects into a prompt (§7d).
|
||||||
|
func (r *Runner) persistRuby(readings []RubyReading) error {
|
||||||
|
type agg struct {
|
||||||
|
first int
|
||||||
|
count int
|
||||||
|
}
|
||||||
|
seen := map[[2]string]*agg{}
|
||||||
|
order := make([][2]string, 0, len(readings))
|
||||||
|
for _, rr := range readings {
|
||||||
|
key := [2]string{rr.Base, rr.Reading}
|
||||||
|
a, ok := seen[key]
|
||||||
|
if !ok {
|
||||||
|
a = &agg{first: rr.Chapter}
|
||||||
|
seen[key] = a
|
||||||
|
order = append(order, key)
|
||||||
|
}
|
||||||
|
if rr.Chapter < a.first {
|
||||||
|
a.first = rr.Chapter
|
||||||
|
}
|
||||||
|
a.count++
|
||||||
|
}
|
||||||
|
sort.Slice(order, func(i, j int) bool {
|
||||||
|
if order[i][0] != order[j][0] {
|
||||||
|
return order[i][0] < order[j][0]
|
||||||
|
}
|
||||||
|
return order[i][1] < order[j][1]
|
||||||
|
})
|
||||||
|
rows := make([]store.RubyReading, 0, len(order))
|
||||||
|
for _, key := range order {
|
||||||
|
a := seen[key]
|
||||||
|
rows = append(rows, store.RubyReading{
|
||||||
|
BookID: r.Book.BookID, Base: key[0], Reading: key[1],
|
||||||
|
FirstChapter: a.first, Occurrences: a.count,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return r.Store.ReplaceRubyReadings(r.Book.BookID, rows)
|
||||||
|
}
|
||||||
262
backend/internal/pipeline/snapshot.go
Normal file
262
backend/internal/pipeline/snapshot.go
Normal file
|
|
@ -0,0 +1,262 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
)
|
||||||
|
|
||||||
|
// snapshot.go: материализация snapshotID (§3.2/D5.2/D8) — контент-хеш всего, что
|
||||||
|
// влияет на wire-байты ИЛИ на резолв вердикта чекпоинта. Правка любого входа —
|
||||||
|
// громкий --resnapshot (= переоплата книги, D15), никогда тихий false-hit.
|
||||||
|
|
||||||
|
// contextSnap freezes the context-assembly knobs (§3.8) inside the snapshot.
|
||||||
|
// Fixed field order — it is part of a content hash.
|
||||||
|
type contextSnap struct {
|
||||||
|
GlossaryInjection string `json:"glossary_injection"`
|
||||||
|
GlossaryTokenBudget int `json:"glossary_token_budget"`
|
||||||
|
STMDepth int `json:"stm_depth"`
|
||||||
|
OverlapTokens int `json:"overlap_tokens"`
|
||||||
|
CacheTTL string `json:"cache_ttl"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// coverageSnap freezes the excision coverage-gate config inside the snapshot, so
|
||||||
|
// enabling the gate OR tuning its thresholds is a loud --resnapshot, never a silent
|
||||||
|
// mismatch between an old checkpoint's stored disposition and a changed gate. HONEST
|
||||||
|
// COST (D15): the snapshotID is folded into every call's RequestHash (render.go), so a
|
||||||
|
// --resnapshot changes EVERY request_hash → every checkpoint misses → the whole book is
|
||||||
|
// RE-BILLED, even for byte-identical wire requests whose verdict merely needs
|
||||||
|
// re-classifying. There is no free "re-classify from the checkpoint" here: the
|
||||||
|
// content-addressed reuse that COULD make an unchanged wire request free (msgsContentHash,
|
||||||
|
// render.go) is gated on an UNCHANGED snapshot (stagerun.go), so it never fires across a
|
||||||
|
// resnapshot. This all-or-nothing re-pay is ACCEPTED for the static acceptance book (the
|
||||||
|
// gate is loud, divergences do not occur) but is exactly why content-addressed checkpoint
|
||||||
|
// reuse must be designed before the ongoing/append-chapters mode (D15.2). Only folded when
|
||||||
|
// ENABLED: tweaking a disabled gate's bounds must not force a re-pin (the gate produces no
|
||||||
|
// verdicts while off). json.Marshal sorts the LenRatio map keys → deterministic render.
|
||||||
|
// Mirrors the estimator/max_tokens-policy discipline (render.go) for a NON-wire input that
|
||||||
|
// still determines a checkpoint's resolved verdict.
|
||||||
|
type coverageSnap struct {
|
||||||
|
Enabled bool `json:"enabled"`
|
||||||
|
Version string `json:"version,omitempty"`
|
||||||
|
LenRatio map[string][]float64 `json:"len_ratio,omitempty"`
|
||||||
|
SentCovMin float64 `json:"sent_cov_min,omitempty"`
|
||||||
|
MinChunkChars int `json:"min_chunk_chars,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// coverageSnapshot renders the gate's snapshot component: {enabled:false} when off
|
||||||
|
// (so toggling on is a visible change), the full config + algorithm version when on.
|
||||||
|
func (r *Runner) coverageSnapshot() coverageSnap {
|
||||||
|
cov := coverageSnap{Enabled: r.Pipeline.Gates.Coverage.Enabled}
|
||||||
|
if cov.Enabled {
|
||||||
|
cov.Version = coverageGateVersion
|
||||||
|
cov.LenRatio = r.Pipeline.Gates.Coverage.LenRatio
|
||||||
|
cov.SentCovMin = r.Pipeline.Gates.Coverage.SentCovMin
|
||||||
|
cov.MinChunkChars = r.Pipeline.Gates.Coverage.MinChunkChars
|
||||||
|
}
|
||||||
|
return cov
|
||||||
|
}
|
||||||
|
|
||||||
|
// memoryVersion is the content-hash of the deterministically materialized injected
|
||||||
|
// memory (the frozen APPROVED glossary rows + the normalization/matcher algorithm
|
||||||
|
// versions), the memory component of the snapshot (D5.2/D8, F1 CLOSED). It is the
|
||||||
|
// Version() of the bank materialized once before the loop, so a change to the approved
|
||||||
|
// glossary — or to the deterministic machinery (trad→simp table, matcher) — fires the
|
||||||
|
// resnapshot gate LOUDLY instead of a stale checkpoint re-paying a diverged translation.
|
||||||
|
// nil bank (report path / a book with no glossary) → the empty-materialization hash, a
|
||||||
|
// stable constant. STM is excluded (rebuilt from checkpoints, §3.2). Auto/draft rows are
|
||||||
|
// excluded per D8 (their injected-content changes are caught at the per-chunk
|
||||||
|
// content_hash level; a future autopopulation milestone revisits this).
|
||||||
|
func (r *Runner) memoryVersion() string {
|
||||||
|
if r.memory != nil {
|
||||||
|
return r.memory.Version()
|
||||||
|
}
|
||||||
|
return computeMemoryVersion(nil, r.Pipeline.Gates.Glossary.PostcheckGate)
|
||||||
|
}
|
||||||
|
|
||||||
|
// snapshotID materializes the job-context snapshot (§3.2): brief_hash, chunker
|
||||||
|
// version, context-assembly knobs, the memory version and the full stage plan
|
||||||
|
// (model, prompt version + CONTENT hash, sampling, resolved capability).
|
||||||
|
// Payload is rendered with a fixed field order — the id is a content hash, so
|
||||||
|
// identical context re-upserts idempotently.
|
||||||
|
func (r *Runner) snapshotID() (id, payload string, err error) {
|
||||||
|
type stageSnap struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Role string `json:"role"`
|
||||||
|
Model string `json:"model"`
|
||||||
|
PromptVersion string `json:"prompt_version"`
|
||||||
|
PromptSHA256 string `json:"prompt_sha256"`
|
||||||
|
Temperature float64 `json:"temperature"`
|
||||||
|
Reasoning string `json:"reasoning"`
|
||||||
|
// ExtraBody модели меняет ТЕЛО запроса (GLM thinking-off и т.п.) —
|
||||||
|
// без него правка ручки в models.yaml молча не инвалидировала бы
|
||||||
|
// чекпоинты (находка ревью). json.Marshal карты сортирует ключи →
|
||||||
|
// детерминированный рендер.
|
||||||
|
ModelExtra json.RawMessage `json:"model_extra,omitempty"`
|
||||||
|
// Провайдер-уровневые оверрайды (local kind переопределяет wire
|
||||||
|
// temperature/max_tokens ПОСЛЕ вычисления request-hash) — тоже влияют
|
||||||
|
// на фактический запрос; без них правка providers.local.max_tokens
|
||||||
|
// давала бы тот же snapshot и ложный checkpoint-hit на стендовом
|
||||||
|
// local-пути (находка внешнего ревью F2).
|
||||||
|
ProviderTemp float64 `json:"provider_temp,omitempty"`
|
||||||
|
ProviderMaxTok int `json:"provider_max_tok,omitempty"`
|
||||||
|
// ProviderModel — the local-kind backend tag the provider swaps onto the wire
|
||||||
|
// AFTER the request-hash (the actual model that answers). The MOST impactful
|
||||||
|
// local override, yet it was missing here while its weaker temp/max_tok siblings
|
||||||
|
// were folded: a local swap 8b→14b mid-book keeps the same snapID and resume
|
||||||
|
// serves the old model (external-review). Folded so it is a loud --resnapshot.
|
||||||
|
ProviderModel string `json:"provider_model,omitempty"`
|
||||||
|
// Capability — резолвнутая wire-форма модели (D3.1): budget-ключ,
|
||||||
|
// temperature-режим, reasoning-контроль. Меняет ТЕЛО запроса (max_tokens
|
||||||
|
// vs max_completion_tokens, отправлять ли temperature, thinking-выключа-
|
||||||
|
// тель), поэтому обязана входить в снапшот — иначе правка каппы молча
|
||||||
|
// false-хитит чекпоинты (тот же класс D5.2, что и payload ниже).
|
||||||
|
Capability json.RawMessage `json:"capability,omitempty"`
|
||||||
|
// EscalateTo + EscalateCapability — the single-hop fallback model and its
|
||||||
|
// resolved wire shape (Веха 2.5). The fallback's request_hash uses ITS model,
|
||||||
|
// and its wire body uses ITS capability — both must be in the snapshot, or
|
||||||
|
// changing the escalation model / its caps would silently false-hit an
|
||||||
|
// escalated chunk's checkpoint (closes the D5.2 escalation-capability techdebt:
|
||||||
|
// stages were folded, escalation models were not, until the cycle landed).
|
||||||
|
EscalateTo string `json:"escalate_to,omitempty"`
|
||||||
|
EscalateCapability json.RawMessage `json:"escalate_capability,omitempty"`
|
||||||
|
// The fallback model's OTHER wire-affecting inputs, mirroring the primary
|
||||||
|
// fold above: its top-level extra_body (merged into the escalation wire body)
|
||||||
|
// and its provider-level temperature/max_tokens overrides (local kind). Without
|
||||||
|
// these, editing the fallback's extra_body / provider knobs would change the
|
||||||
|
// escalation wire but leave the snapshot identical → a silent false-hit on a
|
||||||
|
// resumed escalated chunk (self-review: the primary path guards this, escalation
|
||||||
|
// did not).
|
||||||
|
EscalateExtra json.RawMessage `json:"escalate_extra,omitempty"`
|
||||||
|
EscalateProviderTemp float64 `json:"escalate_provider_temp,omitempty"`
|
||||||
|
EscalateProviderMaxTok int `json:"escalate_provider_max_tok,omitempty"`
|
||||||
|
EscalateProviderModel string `json:"escalate_provider_model,omitempty"`
|
||||||
|
}
|
||||||
|
snap := struct {
|
||||||
|
BriefHash string `json:"brief_hash"`
|
||||||
|
ChunkerVersion string `json:"chunker_version"`
|
||||||
|
EstimatorVersion string `json:"estimator_version"`
|
||||||
|
// MaxTokensPolicy versions the attempt→max_tokens scaling (Веха 2): a
|
||||||
|
// change to the retry doubling shifts attempt≥1 request_hashes, so it
|
||||||
|
// belongs in the snapshot as a loud invalidation (same class as
|
||||||
|
// estimator_version, applied to the regeneration axis).
|
||||||
|
MaxTokensPolicy string `json:"max_tokens_policy"`
|
||||||
|
// ClassifierVersion versions the intrinsic classify() verdict logic (thresholds
|
||||||
|
// + order), so a re-verdict on a resumed checkpoint is a loud --resnapshot, not
|
||||||
|
// a silent flagged↔ok divergence (external-review; symmetric to coverage).
|
||||||
|
ClassifierVersion string `json:"classifier_version"`
|
||||||
|
PipelineCore string `json:"pipeline_core"`
|
||||||
|
// Defaults влияют на maxTokens, а тот входит в request-hash: без них
|
||||||
|
// правка max_output_ratio молча инвалидировала бы все чекпоинты в
|
||||||
|
// обход snapshot-гейта (находка ревью).
|
||||||
|
MaxOutputRatio float64 `json:"max_output_ratio"`
|
||||||
|
MinMaxTokens int `json:"min_max_tokens"`
|
||||||
|
// ContextAssembly — ручки сборки контекста (глоссарий/STM/overlap/TTL,
|
||||||
|
// §3.8). Их правка меняет ВХОД модели, когда память войдёт в msgs; сворачи-
|
||||||
|
// ваем ДО инъекции, чтобы смена budget'а инъекции не промахнулась мимо
|
||||||
|
// resnapshot-гейта (D5.2). Фикс-порядок полей — это content-hash.
|
||||||
|
ContextAssembly contextSnap `json:"context_assembly"`
|
||||||
|
// MemoryVersion — content-hash детерминированно материализованной инъекти-
|
||||||
|
// руемой памяти (approved-глоссарий+резюме+series-bible). Не bump-счётчик
|
||||||
|
// (D8): забыть пересчитать нельзя, а забытый bump переоткрыл бы класс тихой
|
||||||
|
// переоплаты D5.2. До миграции банка памяти (v2) материализация пуста —
|
||||||
|
// хэш стабилен; когда память войдёт в msgs, поле сменится и resnapshot-
|
||||||
|
// гейт сработает громко. STM сюда НЕ входит (пересобирается из чекпоинтов).
|
||||||
|
MemoryVersion string `json:"memory_version"`
|
||||||
|
// PostcheckGate — the memory-bank post-check gate (E1). When true a post-check
|
||||||
|
// miss flips the chunk to flagged; folding its on/off state makes toggling it a
|
||||||
|
// loud --resnapshot (it changes a checkpoint's RESOLVED disposition), not a silent
|
||||||
|
// re-verdict on resume (same class as coverage/classifier). The post-check
|
||||||
|
// ALGORITHM version is already inside MemoryVersion (memoryMatchVersion).
|
||||||
|
PostcheckGate bool `json:"postcheck_gate"`
|
||||||
|
// Coverage — excision QA-gate config (D12 Q3). It does not touch the wire,
|
||||||
|
// but it determines a checkpoint's RESOLVED verdict, so a gate change must be
|
||||||
|
// a loud --resnapshot, not a silent re-verdict on resume (coverageSnapshot).
|
||||||
|
Coverage coverageSnap `json:"coverage"`
|
||||||
|
// StyleCheckVersion versions the cheap deterministic post-check rules (dialogue-dash,
|
||||||
|
// yofikator, translit-interjection blocklist, 万/億 magnitude gate — cheapgates.go). They
|
||||||
|
// are observability, not wire, but editing a rule shifts the recorded style-flag counts, so
|
||||||
|
// a bump is a loud --resnapshot (same verdict class as ClassifierVersion). The ё-policy is
|
||||||
|
// part of BriefHash (a book field), so a policy change already re-pins via brief_hash.
|
||||||
|
StyleCheckVersion string `json:"style_check_version"`
|
||||||
|
Stages []stageSnap `json:"stages"`
|
||||||
|
}{
|
||||||
|
BriefHash: r.Book.BriefHash(),
|
||||||
|
ChunkerVersion: chunkerVersion,
|
||||||
|
EstimatorVersion: estimatorVersion,
|
||||||
|
MaxTokensPolicy: maxTokensPolicyVersion,
|
||||||
|
ClassifierVersion: classifierVersion,
|
||||||
|
PipelineCore: r.Pipeline.Core,
|
||||||
|
MaxOutputRatio: r.Pipeline.Defaults.MaxOutputRatio,
|
||||||
|
MinMaxTokens: r.Pipeline.Defaults.MinMaxTokens,
|
||||||
|
ContextAssembly: contextSnap{
|
||||||
|
GlossaryInjection: r.Pipeline.Context.GlossaryInjection,
|
||||||
|
GlossaryTokenBudget: r.Pipeline.Context.GlossaryTokenBudget,
|
||||||
|
STMDepth: r.Pipeline.Context.STMDepth,
|
||||||
|
OverlapTokens: r.Pipeline.Context.OverlapTokens,
|
||||||
|
CacheTTL: r.Pipeline.Context.CacheTTL,
|
||||||
|
},
|
||||||
|
MemoryVersion: r.memoryVersion(),
|
||||||
|
PostcheckGate: r.Pipeline.Gates.Glossary.PostcheckGate,
|
||||||
|
Coverage: r.coverageSnapshot(),
|
||||||
|
StyleCheckVersion: cheapGateVersion,
|
||||||
|
}
|
||||||
|
for _, st := range r.Pipeline.Stages {
|
||||||
|
ss := stageSnap{
|
||||||
|
Name: st.Name, Role: st.Role, Model: st.Model,
|
||||||
|
PromptVersion: st.PromptVersion, PromptSHA256: r.templates[st.Name].SHA256,
|
||||||
|
Temperature: st.Temperature, Reasoning: st.Reasoning,
|
||||||
|
}
|
||||||
|
if prov, ok := r.Models.Providers[r.Models.Models[st.Model].Provider]; ok {
|
||||||
|
ss.ProviderTemp, ss.ProviderMaxTok, ss.ProviderModel = prov.Temperature, prov.MaxTokens, prov.Model
|
||||||
|
}
|
||||||
|
if extra := r.Models.Models[st.Model].ExtraBody; len(extra) > 0 {
|
||||||
|
raw, merr := json.Marshal(extra)
|
||||||
|
if merr != nil {
|
||||||
|
return "", "", merr
|
||||||
|
}
|
||||||
|
ss.ModelExtra = raw
|
||||||
|
}
|
||||||
|
// Резолвнутая каппа — тем же ResolveCapability, что и у клиента, поэтому
|
||||||
|
// хэшируемое всегда совпадает с отправляемым. json.Marshal сортирует
|
||||||
|
// ключи карт (off_extra_body) → детерминированный рендер.
|
||||||
|
capRaw, cerr := json.Marshal(r.Models.ResolveCapability(st.Model))
|
||||||
|
if cerr != nil {
|
||||||
|
return "", "", cerr
|
||||||
|
}
|
||||||
|
ss.Capability = capRaw
|
||||||
|
// Fold the single-hop fallback model + its resolved capability (Веха 2.5):
|
||||||
|
// both are wire-affecting inputs of an escalated call, so changing them is a
|
||||||
|
// loud --resnapshot, not a silent false-hit on an escalated checkpoint.
|
||||||
|
// TRIPWIRE (пакет №4): этот блок — зеркало primary-фолда выше (2×, ниже порога
|
||||||
|
// извлечения ≥3). Когда канал B/annotator добавит ТРЕТЬЮ модельную ось стадии —
|
||||||
|
// сначала извлечь общий foldModelWire(model) (prov-тройка, extra, capability),
|
||||||
|
// потом добавлять ось: копипаста здесь тише всего расходится и портит
|
||||||
|
// идентичность снапшота (false-hit/false-miss на эскалированных чекпоинтах).
|
||||||
|
if st.EscalateTo != "" {
|
||||||
|
ss.EscalateTo = st.EscalateTo
|
||||||
|
escCap, eerr := json.Marshal(r.Models.ResolveCapability(st.EscalateTo))
|
||||||
|
if eerr != nil {
|
||||||
|
return "", "", eerr
|
||||||
|
}
|
||||||
|
ss.EscalateCapability = escCap
|
||||||
|
if prov, ok := r.Models.Providers[r.Models.Models[st.EscalateTo].Provider]; ok {
|
||||||
|
ss.EscalateProviderTemp, ss.EscalateProviderMaxTok, ss.EscalateProviderModel = prov.Temperature, prov.MaxTokens, prov.Model
|
||||||
|
}
|
||||||
|
if extra := r.Models.Models[st.EscalateTo].ExtraBody; len(extra) > 0 {
|
||||||
|
raw, merr := json.Marshal(extra)
|
||||||
|
if merr != nil {
|
||||||
|
return "", "", merr
|
||||||
|
}
|
||||||
|
ss.EscalateExtra = raw
|
||||||
|
}
|
||||||
|
}
|
||||||
|
snap.Stages = append(snap.Stages, ss)
|
||||||
|
}
|
||||||
|
data, err := json.Marshal(snap)
|
||||||
|
if err != nil {
|
||||||
|
return "", "", err
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256(data)
|
||||||
|
return hex.EncodeToString(sum[:]), string(data), nil
|
||||||
|
}
|
||||||
458
backend/internal/pipeline/stagerun.go
Normal file
458
backend/internal/pipeline/stagerun.go
Normal file
|
|
@ -0,0 +1,458 @@
|
||||||
|
package pipeline
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"textmachine/backend/internal/config"
|
||||||
|
"textmachine/backend/internal/ledger"
|
||||||
|
"textmachine/backend/internal/llm"
|
||||||
|
"textmachine/backend/internal/obs"
|
||||||
|
"textmachine/backend/internal/store"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// stagerun.go: исполнение ОДНОЙ стадии одного чанка до терминального disposition —
|
||||||
|
// resume-fast-path по chunk_status (контент-верифицированный), ось attempt
|
||||||
|
// (retry {length,empty} с удвоением бюджета до капа), затем один вызов
|
||||||
|
// reserve→call→settle+checkpoint на попытку (инвариант денег №1).
|
||||||
|
|
||||||
|
// runStage runs ONE stage of ONE chunk to a terminal disposition. It first
|
||||||
|
// resumes a resolved verdict (chunk_status read BEFORE any render — anti-wedge
|
||||||
|
// #1), otherwise walks the attempt axis: render → per-attempt checkpoint resume
|
||||||
|
// or a fresh reserve/call/settle → classify → retry the retryable {length,empty}
|
||||||
|
// subset with a doubled budget up to the regenerate cap, then flag. Returns an
|
||||||
|
// error ONLY on an infra failure; a bad completion is a disposition, never an
|
||||||
|
// error.
|
||||||
|
func (r *Runner) runStage(ctx context.Context, st config.Stage, stageIdx int, snapID string, ch Chunk, prev, injection string) (*StageResult, error) {
|
||||||
|
// Дополняем ReqInfo ПЕРВЫМ делом, СОХРАНЯЯ решения допуска (LogBodies) —
|
||||||
|
// перезапись с нуля отрезала бы задокументированный debug-канал (находка
|
||||||
|
// ревью). Раньше обогащение стояло ПОСЛЕ resume-fast-path — и все строки
|
||||||
|
// resume/re-pin шли без book/role оси contextHandler (боль smoke-прогона №4).
|
||||||
|
ri, _ := obs.ReqInfoFromContext(ctx)
|
||||||
|
ri.Book, ri.Chapter, ri.Chunk, ri.Stage, ri.Role = r.Book.BookID, ch.Chapter, ch.ChunkIdx, st.Name, st.Role
|
||||||
|
ctx = obs.WithReqInfo(ctx, ri)
|
||||||
|
|
||||||
|
job, err := r.Store.EnsureJob(r.Book.BookID, ch.Chapter, st.Name, snapID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("pipeline: ensure job %s/ch%d/%s: %w", r.Book.BookID, ch.Chapter, st.Name, err)
|
||||||
|
}
|
||||||
|
// Snapshot pinning (Р6): a job frozen on a stale snapshot is a loud stop
|
||||||
|
// unless --resnapshot explicitly accepts the re-translation.
|
||||||
|
if job.SnapshotID != snapID {
|
||||||
|
if !r.Resnapshot {
|
||||||
|
return nil, fmt.Errorf("pipeline: job %s/ch%d/%s was started under snapshot %.12s, current config renders snapshot %.12s — конфиг/промпты изменились; уже оплаченные чекпоинты станут недействительны и вызовы будут пере-оплачены; повторить с --resnapshot, чтобы принять это явно",
|
||||||
|
r.Book.BookID, ch.Chapter, st.Name, job.SnapshotID, snapID)
|
||||||
|
}
|
||||||
|
if err := r.Store.UpdateJobSnapshot(job.ID, snapID); err != nil {
|
||||||
|
return nil, fmt.Errorf("pipeline: re-pin job %s/ch%d/%s to snapshot %.12s: %w", r.Book.BookID, ch.Chapter, st.Name, snapID, err)
|
||||||
|
}
|
||||||
|
r.Log.WarnContext(ctx, "job re-pinned to new snapshot (--resnapshot)", "stage", st.Name, "old", job.SnapshotID[:12], "new", snapID[:12])
|
||||||
|
}
|
||||||
|
|
||||||
|
// Render the wire messages up front. This is cheap (string substitution — the
|
||||||
|
// expensive part is the LLM call, still gated below) and it lets the resume
|
||||||
|
// fast-path be CONTENT-VERIFIED: the source bytes are not in the snapshot, so
|
||||||
|
// a positional chunk_status row must be checked against the current rendered
|
||||||
|
// content, else an edited source would serve a stale, divergent translation.
|
||||||
|
msgs, err := MessagesWithInjection(r.templates[st.Name], RenderVars{Book: r.Book, Text: ch.Text, Draft: prev}, injection)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
contentHash := msgsContentHash(msgs)
|
||||||
|
|
||||||
|
// Anti-wedge #1 + content-guard (self-review): resume resolves from
|
||||||
|
// chunk_status BEFORE any LLM call. A flagged chunk is not in TM (garbage
|
||||||
|
// never commits), so absence-in-TM ≠ "not done" — the disposition row is the
|
||||||
|
// resume authority, and a terminally-flagged chunk is NOT re-attacked (no
|
||||||
|
// infinite paid loop, even if the regenerate budget was raised). The row is
|
||||||
|
// trusted ONLY when snapshot AND content both match: a stale-snapshot row
|
||||||
|
// (post --resnapshot) or a stale-content row (source edited, not in the
|
||||||
|
// snapshot) is ignored → the attempt loop below re-derives it content-safely
|
||||||
|
// (a source edit becomes a silent per-chunk re-translate, §3.4, not a stale
|
||||||
|
// serve). A `skipped` row is ignored too: translateChunk re-derives the skip.
|
||||||
|
if cs, err := r.Store.GetChunkStatus(r.Book.BookID, ch.Chapter, ch.ChunkIdx, st.Name); err != nil {
|
||||||
|
return nil, fmt.Errorf("pipeline: read chunk_status %s/ch%d/chunk%d/%s: %w", r.Book.BookID, ch.Chapter, ch.ChunkIdx, st.Name, err)
|
||||||
|
} else if cs != nil && cs.SnapshotID == snapID && cs.ContentHash == contentHash && cs.Disposition != string(DispSkipped) {
|
||||||
|
return r.resumeFromChunkStatus(ctx, st, ch, cs)
|
||||||
|
}
|
||||||
|
|
||||||
|
// max_tokens base is sized from the text THIS stage processes: the source for
|
||||||
|
// the translator, the prior draft for later stages (D2.5 — the monolingual
|
||||||
|
// editor works over the Russian draft ≈1.9× the CJK source, so sizing edit
|
||||||
|
// from source under-budgets and false-triggers a length retry).
|
||||||
|
sizingText := ch.Text
|
||||||
|
if stageIdx > 0 {
|
||||||
|
sizingText = prev
|
||||||
|
}
|
||||||
|
baseMaxTokens := int(float64(EstimateTokens(sizingText)) * r.Pipeline.Defaults.MaxOutputRatio)
|
||||||
|
if baseMaxTokens < r.Pipeline.Defaults.MinMaxTokens {
|
||||||
|
baseMaxTokens = r.Pipeline.Defaults.MinMaxTokens
|
||||||
|
}
|
||||||
|
|
||||||
|
maxRegen := r.Pipeline.Retries.RegenerateBeforeEscalate
|
||||||
|
if maxRegen < 0 {
|
||||||
|
maxRegen = 0
|
||||||
|
}
|
||||||
|
|
||||||
|
var cumCost, runCost float64
|
||||||
|
var last stageAttempt
|
||||||
|
anyFresh := false
|
||||||
|
attemptsMade := 0
|
||||||
|
for attempt := 0; ; attempt++ {
|
||||||
|
maxTokens := maxTokensForAttempt(baseMaxTokens, attempt)
|
||||||
|
att, err := r.runAttempt(ctx, st, st.Model, snapID, ch, job, attempt, maxTokens, msgs, false)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err // infra failure
|
||||||
|
}
|
||||||
|
attemptsMade = attempt + 1
|
||||||
|
cumCost += att.cumCost
|
||||||
|
runCost += att.runCost
|
||||||
|
anyFresh = anyFresh || att.freshCall
|
||||||
|
last = att
|
||||||
|
if att.cls.ok() {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
// Flagged: re-attack only the retryable subset, only while regenerations
|
||||||
|
// remain (a bigger budget on the attempt axis, D2.3). Everything else is
|
||||||
|
// deterministic — a same-model retry would re-refuse and re-bill (D2.2).
|
||||||
|
if att.cls.Reason.retryable() && attempt < maxRegen {
|
||||||
|
r.Log.WarnContext(ctx, "stage flagged, regenerating with a larger budget",
|
||||||
|
"stage", st.Name, "chapter", ch.Chapter, "chunk", ch.ChunkIdx,
|
||||||
|
"attempt", attempt, "reason", string(att.cls.Reason), "next_max_tokens", maxTokensForAttempt(baseMaxTokens, attempt+1))
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
// Single-hop escalation (D12; the full policy lives in escalation.go). The hop
|
||||||
|
// runs at most ONCE, is re-gated, and never resets the retry budget; a ceiling-
|
||||||
|
// denied hop keeps the primary flag instead of aborting the book.
|
||||||
|
escalated, escModel := false, ""
|
||||||
|
esc, err := r.maybeEscalate(ctx, st, snapID, ch, job, baseMaxTokens, msgs, last)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if esc.attempted {
|
||||||
|
cumCost += esc.fb.cumCost
|
||||||
|
runCost += esc.fb.runCost
|
||||||
|
anyFresh = anyFresh || esc.fb.freshCall
|
||||||
|
escalated, escModel = true, st.EscalateTo
|
||||||
|
if esc.fb.cls.ok() {
|
||||||
|
last = esc.fb // the fallback draft passed the re-gate → it is authoritative
|
||||||
|
}
|
||||||
|
// else: the fallback also failed → keep the primary flag (last unchanged);
|
||||||
|
// the fallback call is billed and counted, the chunk stays flagged (1 hop).
|
||||||
|
}
|
||||||
|
|
||||||
|
disposition := last.cls.Reason.disposition()
|
||||||
|
finalHash := ""
|
||||||
|
if disposition == DispOK {
|
||||||
|
finalHash = last.reqHash
|
||||||
|
}
|
||||||
|
// chunk_status is a RESOLVE over the checkpoints: cost_usd sums every attempt
|
||||||
|
// (F3-honest — retries are counted), final_hash points at the authoritative
|
||||||
|
// checkpoint the ok path serves on resume.
|
||||||
|
if err := r.Store.UpsertChunkStatus(store.ChunkStatus{
|
||||||
|
BookID: r.Book.BookID, Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Stage: st.Name,
|
||||||
|
SnapshotID: snapID, ContentHash: contentHash, Disposition: string(disposition), FlagReason: string(last.cls.Reason),
|
||||||
|
Attempts: attemptsMade, FinalHash: finalHash, CostUSD: cumCost, Detail: last.cls.Detail,
|
||||||
|
Escalated: escalated, EscalationModel: escModel,
|
||||||
|
}); err != nil {
|
||||||
|
return nil, fmt.Errorf("pipeline: record chunk_status: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
sr := &StageResult{
|
||||||
|
Stage: st.Name, Role: st.Role, Model: last.modelActual,
|
||||||
|
FromResume: !anyFresh,
|
||||||
|
Usage: last.usage,
|
||||||
|
CostUSD: runCost,
|
||||||
|
CumCostUSD: cumCost,
|
||||||
|
LatencyMS: last.latency,
|
||||||
|
FinishReason: last.finish,
|
||||||
|
Disposition: disposition,
|
||||||
|
FlagReason: last.cls.Reason,
|
||||||
|
Detail: last.cls.Detail,
|
||||||
|
Attempts: attemptsMade,
|
||||||
|
Escalated: escalated,
|
||||||
|
EscalationModel: escModel,
|
||||||
|
}
|
||||||
|
if disposition == DispOK {
|
||||||
|
sr.Text = last.text
|
||||||
|
} else {
|
||||||
|
r.Log.WarnContext(ctx, "stage flagged", "stage", st.Name, "chapter", ch.Chapter,
|
||||||
|
"chunk", ch.ChunkIdx, "reason", string(last.cls.Reason), "attempts", attemptsMade,
|
||||||
|
"cum_cost_usd", fmt.Sprintf("%.6f", cumCost))
|
||||||
|
}
|
||||||
|
return sr, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// stageAttempt is the result of one attempt (a checkpoint hit or a fresh call).
|
||||||
|
type stageAttempt struct {
|
||||||
|
reqHash string
|
||||||
|
cls classification
|
||||||
|
text string
|
||||||
|
usage llm.Usage
|
||||||
|
finish string
|
||||||
|
modelActual string
|
||||||
|
latency int
|
||||||
|
runCost float64 // billed THIS run (0 on a checkpoint hit)
|
||||||
|
cumCost float64 // this attempt's cost (checkpoint cost on a hit, fresh cost on a call)
|
||||||
|
freshCall bool // a provider call was made this run
|
||||||
|
}
|
||||||
|
|
||||||
|
// runAttempt executes exactly one attempt on the request-hash axis: a checkpoint
|
||||||
|
// hit is classified for free (self-heal, incl. legacy Фаза-0 empty/decode
|
||||||
|
// checkpoints — no re-billing), otherwise a fresh reserve → call → settle+
|
||||||
|
// checkpoint. It returns an error only on an infra failure; a bad completion
|
||||||
|
// comes back as a classification on the attempt.
|
||||||
|
func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID string, ch Chunk, job *store.Job, attempt, maxTokens int, msgs []llm.Message, escalation bool) (stageAttempt, error) {
|
||||||
|
reqHash := RequestHash(r.Book.BookID, ch.Chapter, ch.ChunkIdx, attempt, st.Name, st.Role, model,
|
||||||
|
st.Temperature, st.Reasoning, false, maxTokens, snapID, msgs)
|
||||||
|
att := stageAttempt{reqHash: reqHash, modelActual: model}
|
||||||
|
|
||||||
|
// Resume on the attempt axis: a checkpoint means THIS attempt already happened
|
||||||
|
// and was billed — classify its text and never re-bill (kill -9 loses ≤1 call;
|
||||||
|
// legacy empty/decode checkpoints self-heal by classification, not a crash).
|
||||||
|
if cp, err := r.Store.GetCheckpoint(reqHash); err != nil {
|
||||||
|
return att, fmt.Errorf("pipeline: read checkpoint for %s/ch%d/chunk%d/%s attempt %d: %w", r.Book.BookID, ch.Chapter, ch.ChunkIdx, st.Name, attempt, err)
|
||||||
|
} else if cp != nil {
|
||||||
|
var usage llm.Usage
|
||||||
|
if uerr := json.Unmarshal([]byte(cp.UsageJSON), &usage); uerr != nil {
|
||||||
|
// Деньги не страдают (cumCost — из cp.CostUSD), но токены отчёта будут
|
||||||
|
// нулями — не молчать об этом совсем (боль аудита цепочек, info-класс).
|
||||||
|
r.Log.DebugContext(ctx, "checkpoint usage_json unreadable; tokens report as zero", "hash", reqHash[:12], "err", uerr)
|
||||||
|
}
|
||||||
|
att.usage = usage
|
||||||
|
att.text = cp.ResponseText
|
||||||
|
att.finish = cp.FinishReason
|
||||||
|
att.modelActual = cp.ModelActual
|
||||||
|
att.cumCost = cp.CostUSD
|
||||||
|
att.cls = r.classifyOutput(st.Role, ch.Text, cp.ResponseText, cp.FinishReason)
|
||||||
|
rl := r.baseRequestLog(st, ch, model, reqHash)
|
||||||
|
rl.ModelActual = cp.ModelActual
|
||||||
|
rl.TMHit, rl.OK, rl.FinishReason, rl.Degraded = true, att.cls.ok(), cp.FinishReason, degradedTag(att.cls)
|
||||||
|
r.Store.LogRequest(ctx, r.Log, rl)
|
||||||
|
r.setJobStatus(ctx, job.ID, "done")
|
||||||
|
r.Log.InfoContext(ctx, "attempt served from checkpoint", "stage", st.Name,
|
||||||
|
"attempt", attempt, "hash", reqHash[:12], "disposition", string(att.cls.Reason.disposition()))
|
||||||
|
return att, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fresh call: reserve → call → settle+checkpoint (atomic, §3.3).
|
||||||
|
price := r.Pricer.PriceFor(model)
|
||||||
|
promptEst := 0
|
||||||
|
for _, m := range msgs {
|
||||||
|
promptEst += EstimateTokens(m.Content)
|
||||||
|
}
|
||||||
|
// D13.6: reserve the additive reasoning buffer for the ACTUAL model being called (the
|
||||||
|
// escalate_to fallback may differ from st.Model and have its own provider). 0 for
|
||||||
|
// subset providers / reasoning-off — the ceiling then sees only completion, as before.
|
||||||
|
reasoningBudget := r.Models.AdditiveReasoningTokens(model, st.Reasoning, st.ReasoningMaxTokens)
|
||||||
|
estimate := ledger.EstimateUSD(price, promptEst, maxTokens, reasoningBudget)
|
||||||
|
|
||||||
|
resv, verdict, err := r.Store.Reserve(r.Book.BookID, estimate, store.Ceilings{
|
||||||
|
BookUSD: r.Book.Ceilings.BookUSD, DayUSD: r.Book.Ceilings.DayUSD,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
r.setJobStatus(ctx, job.ID, "failed")
|
||||||
|
return att, fmt.Errorf("pipeline: reserve $%.6f for %s/ch%d/chunk%d/%s: %w", estimate, r.Book.BookID, ch.Chapter, ch.ChunkIdx, st.Name, err)
|
||||||
|
}
|
||||||
|
switch verdict {
|
||||||
|
case store.ReserveDeniedBook, store.ReserveDeniedDay:
|
||||||
|
// Ceiling is a hard, book-wide stop (not a per-chunk flag): the job stays
|
||||||
|
// 'pending' and resume continues once the ceiling is raised. Wrapped so an
|
||||||
|
// optional escalation hop can degrade instead of aborting the book. Деньги
|
||||||
|
// в тексте — это ПОСЛЕДНЯЯ строка, которую оператор читает в 3 часа ночи:
|
||||||
|
// без committed/reserved/estimate он не может выбрать новый потолок, а
|
||||||
|
// «%.2f» рендерил маленький потолок как «0.00$» (боль smoke-прогона).
|
||||||
|
committed, reserved, serr := r.Store.SpentUSD(r.Book.BookID)
|
||||||
|
money := ""
|
||||||
|
if serr != nil {
|
||||||
|
// Не молчать (собственная доктрина пакета): деталь денег пропадает из
|
||||||
|
// сообщения — пусть хотя бы след останется в логе.
|
||||||
|
r.Log.WarnContext(ctx, "SpentUSD read failed while formatting the ceiling error; money detail omitted", "err", serr)
|
||||||
|
} else {
|
||||||
|
money = fmt.Sprintf(" (committed=$%.6f reserved=$%.6f, denied estimate=$%.6f, ch%d/chunk%d/%s)",
|
||||||
|
committed, reserved, estimate, ch.Chapter, ch.ChunkIdx, st.Name)
|
||||||
|
}
|
||||||
|
if verdict == store.ReserveDeniedBook {
|
||||||
|
return att, fmt.Errorf("pipeline: book USD ceiling reached ($%g)%s — raise ceilings.book_usd or stop: %w", r.Book.Ceilings.BookUSD, money, errReserveCeiling)
|
||||||
|
}
|
||||||
|
return att, fmt.Errorf("pipeline: daily USD ceiling reached ($%g)%s: %w", r.Book.Ceilings.DayUSD, money, errReserveCeiling)
|
||||||
|
}
|
||||||
|
|
||||||
|
client, err := r.client(model)
|
||||||
|
if err != nil {
|
||||||
|
r.releaseReservation(ctx, resv)
|
||||||
|
r.setJobStatus(ctx, job.ID, "failed")
|
||||||
|
return att, err
|
||||||
|
}
|
||||||
|
if err := r.Store.SetJobStatus(job.ID, "running"); err != nil {
|
||||||
|
r.releaseReservation(ctx, resv)
|
||||||
|
return att, err
|
||||||
|
}
|
||||||
|
att.freshCall = true
|
||||||
|
|
||||||
|
// «Вызов в полёте» обязан быть виден: между этой строкой и «attempt completed»
|
||||||
|
// могут пройти минуты (attempt_timeout × ретраи транспорта), и без неё «висит
|
||||||
|
// провайдер» неотличимо от «умер процесс» — главная боль smoke-прогона №4.
|
||||||
|
// Ось book/chapter/chunk/stage/role приезжает из ctx через contextHandler.
|
||||||
|
r.Log.InfoContext(ctx, "calling model", "model", model, "attempt", attempt,
|
||||||
|
"max_tokens", maxTokens, "escalation", escalation, "estimate_usd", fmt.Sprintf("%.6f", estimate))
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
resp, err := client.Complete(ctx, llm.LLMRequest{
|
||||||
|
Model: model,
|
||||||
|
Messages: msgs,
|
||||||
|
MaxTokens: maxTokens,
|
||||||
|
Temperature: st.Temperature,
|
||||||
|
ReasoningEffort: st.Reasoning,
|
||||||
|
})
|
||||||
|
att.latency = int(time.Since(start).Milliseconds())
|
||||||
|
if err != nil {
|
||||||
|
var bde *llm.BilledDecodeError
|
||||||
|
if errors.As(err, &bde) {
|
||||||
|
// 2xx with an unreadable body: the provider ALREADY billed. Do not
|
||||||
|
// release; conservatively settle the ESTIMATE with an empty decode
|
||||||
|
// checkpoint, then FLAG (anti-wedge #2 — the loop continues, this is
|
||||||
|
// not an infra crash). A settle failure here is a real infra fault.
|
||||||
|
if serr := r.Store.SettleWithCheckpoint(resv, estimate, store.Checkpoint{
|
||||||
|
RequestHash: reqHash, JobID: job.ID, ChunkIdx: ch.ChunkIdx, Attempt: attempt,
|
||||||
|
Stage: st.Name, Role: st.Role, ModelRequested: model, ModelActual: model,
|
||||||
|
ResponseText: "", UsageJSON: "{}", CostUSD: estimate, FinishReason: decodeErrorFinish,
|
||||||
|
Escalation: escalation,
|
||||||
|
}); serr != nil {
|
||||||
|
return att, fmt.Errorf("pipeline: settle after billed decode failure: %w", serr)
|
||||||
|
}
|
||||||
|
att.finish = decodeErrorFinish
|
||||||
|
att.cumCost, att.runCost = estimate, estimate
|
||||||
|
att.cls = classification{FlagDecodeError, "billed 2xx with an unreadable body: " + err.Error()}
|
||||||
|
rl := r.baseRequestLog(st, ch, model, reqHash)
|
||||||
|
rl.CostUSD, rl.LatencyMS, rl.FinishReason = estimate, att.latency, decodeErrorFinish
|
||||||
|
rl.Degraded, rl.Err, rl.OK = "billed_2xx_decode_failed", err.Error(), false
|
||||||
|
r.Store.LogRequest(ctx, r.Log, rl)
|
||||||
|
r.setJobStatus(ctx, job.ID, "done")
|
||||||
|
return att, nil
|
||||||
|
}
|
||||||
|
// No 2xx ever arrived: nothing was billed. Release and surface as an INFRA
|
||||||
|
// failure — a long book pauses/resumes on an outage or terminal 4xx (D4),
|
||||||
|
// rather than flag-storming every remaining chunk on a dead provider.
|
||||||
|
r.releaseReservation(ctx, resv)
|
||||||
|
rl := r.baseRequestLog(st, ch, model, reqHash)
|
||||||
|
rl.LatencyMS, rl.Err, rl.OK = att.latency, err.Error(), false
|
||||||
|
r.Store.LogRequest(ctx, r.Log, rl)
|
||||||
|
r.setJobStatus(ctx, job.ID, "failed")
|
||||||
|
// Полная ось в ПОСЛЕДНЕЙ строке, которую читает оператор: одна «stage draft
|
||||||
|
// call: …» не говорит, КАКОЙ чанк какой моделью умер (боль smoke-прогона).
|
||||||
|
return att, fmt.Errorf("pipeline: stage %s call (ch%d/chunk%d, model %s): %w", st.Name, ch.Chapter, ch.ChunkIdx, model, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
modelActual := resp.Model
|
||||||
|
if modelActual == "" {
|
||||||
|
modelActual = model
|
||||||
|
}
|
||||||
|
att.modelActual = modelActual
|
||||||
|
att.text = resp.Text
|
||||||
|
att.finish = resp.FinishReason
|
||||||
|
att.usage = resp.Usage
|
||||||
|
|
||||||
|
// Цена по фактически ответившей модели, с fallback на ЗАПРОШЕННУЮ (не на
|
||||||
|
// дешёвый глобальный якорь), если провайдер вернул канонизированный слаг.
|
||||||
|
price = r.Pricer.PriceForResponse(model, modelActual)
|
||||||
|
cost := ledger.CostUSD(price, resp.Usage)
|
||||||
|
// Платная модель (InputPerM>0) вернула 2xx с нулевым usage — $0 ослепил бы
|
||||||
|
// потолок: берём консервативную оценку. Для local ($0 цена) нулевой usage
|
||||||
|
// штатен — остаётся $0.
|
||||||
|
if cost == 0 && price.InputPerM > 0 && resp.Usage.PromptTokens == 0 && resp.Usage.CompletionTokens == 0 {
|
||||||
|
r.Log.WarnContext(ctx, "paid 2xx with zero usage; settling the reservation estimate to keep the ceiling honest",
|
||||||
|
"stage", st.Name, "model", modelActual, "estimate_usd", fmt.Sprintf("%.6f", estimate))
|
||||||
|
cost = estimate
|
||||||
|
}
|
||||||
|
usageJSON, err := json.Marshal(resp.Usage)
|
||||||
|
if err != nil {
|
||||||
|
return att, err
|
||||||
|
}
|
||||||
|
att.cumCost, att.runCost = cost, cost
|
||||||
|
|
||||||
|
// Деньги: settle + сырой ответ — одна транзакция (§3.3). ВСЕГДА, даже для
|
||||||
|
// пустого/усечённого/отказного ответа: он оплачен провайдером, чекпоинт
|
||||||
|
// хранит его до цента, а годность решает classify ПОСЛЕ (деньги ≠ вердикт).
|
||||||
|
if err := r.Store.SettleWithCheckpoint(resv, cost, store.Checkpoint{
|
||||||
|
RequestHash: reqHash, JobID: job.ID, ChunkIdx: ch.ChunkIdx, Attempt: attempt,
|
||||||
|
Stage: st.Name, Role: st.Role,
|
||||||
|
ModelRequested: model, ModelActual: modelActual, ResponseText: resp.Text,
|
||||||
|
UsageJSON: string(usageJSON), CostUSD: cost, FinishReason: resp.FinishReason,
|
||||||
|
ProviderRequestID: resp.ProviderRequestID, Escalation: escalation,
|
||||||
|
}); err != nil {
|
||||||
|
// The provider HAS billed this 2xx; failing to persist means the money
|
||||||
|
// state is behind reality — fail loud, never continue on top.
|
||||||
|
return att, fmt.Errorf("pipeline: settle stage %s: %w", st.Name, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Classify AFTER the money is durably settled+checkpointed. F4 lives here: a
|
||||||
|
// non-empty truncated length draft is now classified (flagged/retried), never
|
||||||
|
// silently passed downstream as OK; an empty completion is flagged too, not a
|
||||||
|
// run-crash.
|
||||||
|
att.cls = r.classifyOutput(st.Role, ch.Text, resp.Text, resp.FinishReason)
|
||||||
|
|
||||||
|
rl := r.baseRequestLog(st, ch, model, reqHash)
|
||||||
|
rl.ModelActual = modelActual
|
||||||
|
rl.PromptTokens, rl.CachedTokens = resp.Usage.PromptTokens, resp.Usage.CachedTokens
|
||||||
|
rl.CacheCreationTokens = resp.Usage.CacheCreationTokens
|
||||||
|
rl.CompletionTokens, rl.ReasoningTokens = resp.Usage.CompletionTokens, resp.Usage.ReasoningTokens
|
||||||
|
rl.CostUSD, rl.LatencyMS, rl.FinishReason = cost, att.latency, resp.FinishReason
|
||||||
|
rl.OK, rl.Degraded = att.cls.ok(), degradedTag(att.cls)
|
||||||
|
r.Store.LogRequest(ctx, r.Log, rl)
|
||||||
|
r.setJobStatus(ctx, job.ID, "done")
|
||||||
|
r.Log.InfoContext(ctx, "attempt completed", "stage", st.Name, "attempt", attempt, "model", modelActual,
|
||||||
|
"cost_usd", fmt.Sprintf("%.6f", cost), "latency_ms", att.latency,
|
||||||
|
"disposition", string(att.cls.Reason.disposition()), "reason", string(att.cls.Reason))
|
||||||
|
return att, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// baseRequestLog builds the attribution prefix EVERY request_log row must carry
|
||||||
|
// (book/chapter/chunk/stage/role/model/request_hash) — the join keys that tie a
|
||||||
|
// telemetry row to spend and chunk_status. One constructor instead of five hand
|
||||||
|
// copies: a new call path (annotator stage, channel B hop) cannot forget a key
|
||||||
|
// and produce an unjoinable row. The outcome-specific tail (tokens, cost, tm_hit,
|
||||||
|
// err, degraded) deliberately stays at each call site — the asymmetries there
|
||||||
|
// are real (a decode-failure row has no ModelActual, a resume row no latency).
|
||||||
|
func (r *Runner) baseRequestLog(st config.Stage, ch Chunk, model, reqHash string) store.RequestLog {
|
||||||
|
return store.RequestLog{
|
||||||
|
BookID: r.Book.BookID, Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx,
|
||||||
|
Stage: st.Name, Role: st.Role, ModelRequested: model, RequestHash: reqHash,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// setJobStatus updates the advisory jobs.status row, WARNING on failure instead of
|
||||||
|
// silently swallowing it (6 call sites used to `_ =` the error): job.status is
|
||||||
|
// telemetry, not the resume authority (chunk_status/checkpoints are), so a failed
|
||||||
|
// write must not abort the pipeline — but a jobs table silently stuck in
|
||||||
|
// running/failed desyncs the status/redrive read-models with zero trace.
|
||||||
|
func (r *Runner) setJobStatus(ctx context.Context, jobID int64, status string) {
|
||||||
|
if err := r.Store.SetJobStatus(jobID, status); err != nil {
|
||||||
|
r.Log.WarnContext(ctx, "job status update failed (non-fatal; jobs table may lag chunk_status)",
|
||||||
|
"job_id", jobID, "status", status, "err", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// releaseReservation releases an unspent reservation, logging an ERROR on failure
|
||||||
|
// (the level the one non-swallowing site already used): a leaked reserved_usd
|
||||||
|
// silently tightens the book/day ceiling until the next process restart recovers
|
||||||
|
// it (2 of 3 release paths used to swallow this — аудит цепочек).
|
||||||
|
func (r *Runner) releaseReservation(ctx context.Context, resv store.Reservation) {
|
||||||
|
if err := r.Store.Release(resv); err != nil {
|
||||||
|
r.Log.ErrorContext(ctx, "reservation release failed (reserved_usd leaks and tightens ceilings until the next process restart)",
|
||||||
|
"err", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// degradedTag surfaces a flag reason into the request_log `degraded` column.
|
||||||
|
func degradedTag(cls classification) string {
|
||||||
|
if cls.ok() {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return string(cls.Reason)
|
||||||
|
}
|
||||||
|
|
@ -303,7 +303,12 @@ func (r *Runner) Status(ctx context.Context) (*StatusReport, error) {
|
||||||
// seed FILE that was not re-run is NOT detected here (the stored glossary is what status
|
// seed FILE that was not re-run is NOT detected here (the stored glossary is what status
|
||||||
// projects); it surfaces on the next translate's re-seed.
|
// projects); it surfaces on the next translate's re-seed.
|
||||||
if rep.Snapshot != "" {
|
if rep.Snapshot != "" {
|
||||||
if curSnap, serr := r.currentSnapshotProjected(); serr == nil && curSnap != rep.Snapshot {
|
if curSnap, serr := r.currentSnapshotProjected(); serr != nil {
|
||||||
|
// Не молчать: провал проекции раньше тихо читался как «дрифта нет», и
|
||||||
|
// оператор верил чистому статусу при реально изменённом конфиге (аудит
|
||||||
|
// цепочек). Drift остаётся false (проекция неизвестна), но с WARN.
|
||||||
|
r.Log.WarnContext(ctx, "config-drift check failed; drift state unknown (reported as none)", "err", serr)
|
||||||
|
} else if curSnap != rep.Snapshot {
|
||||||
rep.ConfigDrift = true
|
rep.ConfigDrift = true
|
||||||
rep.CurrentSnapshot = curSnap
|
rep.CurrentSnapshot = curSnap
|
||||||
}
|
}
|
||||||
|
|
|
||||||
19
backend/internal/pipeline/testdata/golden/book.yaml
vendored
Normal file
19
backend/internal/pipeline/testdata/golden/book.yaml
vendored
Normal file
|
|
@ -0,0 +1,19 @@
|
||||||
|
# Golden-fixture book. models.yaml is written by the test (live mock URL + fresh
|
||||||
|
# prices_checked date); everything hash-relevant lives in the static files here.
|
||||||
|
book_id: golden-book
|
||||||
|
title: Золотая книга
|
||||||
|
source_lang: ja
|
||||||
|
target_lang: ru
|
||||||
|
genre: ранобэ
|
||||||
|
audience: взрослые
|
||||||
|
venuti: 0.5
|
||||||
|
honorifics: keep
|
||||||
|
transcription: polivanov
|
||||||
|
footnotes: minimal
|
||||||
|
yo_policy: auto
|
||||||
|
style_allowlist: [ня]
|
||||||
|
pipeline: pipeline.yaml
|
||||||
|
models: models.yaml
|
||||||
|
source_file: source.txt
|
||||||
|
glossary_seed: glossary-seed.yaml
|
||||||
|
ceilings: { book_usd: 5.0, day_usd: 10.0 }
|
||||||
154
backend/internal/pipeline/testdata/golden/capture.golden
vendored
Normal file
154
backend/internal/pipeline/testdata/golden/capture.golden
vendored
Normal file
File diff suppressed because one or more lines are too long
24
backend/internal/pipeline/testdata/golden/glossary-seed.yaml
vendored
Normal file
24
backend/internal/pipeline/testdata/golden/glossary-seed.yaml
vendored
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
# Golden-fixture seed: exercises exact hits, decl forms, a spoiler window (since_ch: 2)
|
||||||
|
# and a nested key (紋章 inside 竜の紋章) for the collision/AMBIGUOUS path.
|
||||||
|
terms:
|
||||||
|
- src: 鈴木
|
||||||
|
dst: Судзуки
|
||||||
|
type: name
|
||||||
|
gender: m
|
||||||
|
decl: { invariant: true }
|
||||||
|
- src: 魔法学院
|
||||||
|
dst: Академия магии
|
||||||
|
type: org
|
||||||
|
decl:
|
||||||
|
forms: [Академия магии, Академии магии, Академию магии, Академией магии]
|
||||||
|
- src: 竜の紋章
|
||||||
|
dst: Драконья печать
|
||||||
|
type: term
|
||||||
|
since_ch: 2
|
||||||
|
decl:
|
||||||
|
forms: [Драконья печать, Драконьей печати, Драконью печать]
|
||||||
|
- src: 紋章
|
||||||
|
dst: герб
|
||||||
|
type: term
|
||||||
|
decl:
|
||||||
|
forms: [герб, герба, гербу, гербом]
|
||||||
10
backend/internal/pipeline/testdata/golden/pipeline.yaml
vendored
Normal file
10
backend/internal/pipeline/testdata/golden/pipeline.yaml
vendored
Normal file
|
|
@ -0,0 +1,10 @@
|
||||||
|
# Golden-fixture pipeline: C1 draft→edit with a single-hop escalation on the draft.
|
||||||
|
core: C1
|
||||||
|
version: 1
|
||||||
|
defaults: { max_output_ratio: 2.0, min_max_tokens: 512 }
|
||||||
|
retries: { regenerate_before_escalate: 1 }
|
||||||
|
context: { glossary_injection: selective, glossary_token_budget: 800 }
|
||||||
|
escalation: { budget_usd: 0.5 }
|
||||||
|
stages:
|
||||||
|
- { name: draft, role: translator, model: fake-model, prompt: prompts/translator.md, prompt_version: v-golden, temperature: 0.3, reasoning: "off", escalate_to: fake-fallback }
|
||||||
|
- { name: edit, role: editor, model: fake-model, prompt: prompts/editor.md, prompt_version: v-golden, temperature: 0.4, reasoning: "off" }
|
||||||
3
backend/internal/pipeline/testdata/golden/prompts/editor.md
vendored
Normal file
3
backend/internal/pipeline/testdata/golden/prompts/editor.md
vendored
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: {{title}}.
|
||||||
|
---USER---
|
||||||
|
Черновик перевода для редактуры: {{draft}}
|
||||||
3
backend/internal/pipeline/testdata/golden/prompts/translator.md
vendored
Normal file
3
backend/internal/pipeline/testdata/golden/prompts/translator.md
vendored
Normal file
|
|
@ -0,0 +1,3 @@
|
||||||
|
Переводи художественный текст с {{source_lang}} на {{target_lang}}. Жанр: {{genre}}. Аудитория: {{audience}}. Баланс Венути: {{venuti}}. Хонорифики: {{honorifics}}. Транскрипция: {{transcription}}.
|
||||||
|
---USER---
|
||||||
|
{{text}}
|
||||||
35
backend/internal/pipeline/testdata/golden/source.txt
vendored
Normal file
35
backend/internal/pipeline/testdata/golden/source.txt
vendored
Normal file
|
|
@ -0,0 +1,35 @@
|
||||||
|
第一章 図書館の秘密
|
||||||
|
|
||||||
|
一番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ一層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。鈴木の指先に刻まれた印は、書庫に近づくたびに淡く光った。司書たちはその光を恐れ、誰も彼に声をかけなかった。
|
||||||
|
|
||||||
|
二番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ二層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。回廊の壁には歴代の院長の肖像が並び、その視線は生きているかのように彼の背中を追った。埃の匂いが記憶を刺激した。
|
||||||
|
|
||||||
|
三番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ三層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。鈴木の指先に刻まれた印は、書庫に近づくたびに淡く光った。司書たちはその光を恐れ、誰も彼に声をかけなかった。
|
||||||
|
|
||||||
|
四番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ四層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。回廊の壁には歴代の院長の肖像が並び、その視線は生きているかのように彼の背中を追った。埃の匂いが記憶を刺激した。
|
||||||
|
|
||||||
|
五番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ五層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。鈴木の指先に刻まれた印は、書庫に近づくたびに淡く光った。司書たちはその光を恐れ、誰も彼に声をかけなかった。
|
||||||
|
|
||||||
|
六番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ六層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。回廊の壁には歴代の院長の肖像が並び、その視線は生きているかのように彼の背中を追った。埃の匂いが記憶を刺激した。扉の奥で彼が見たものは、台座に置かれた竜の紋章だった。それは禁じられた歴史の最後の欠片だと、後に彼は知ることになる。
|
||||||
|
|
||||||
|
七番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ七層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。鈴木の指先に刻まれた印は、書庫に近づくたびに淡く光った。司書たちはその光を恐れ、誰も彼に声をかけなかった。
|
||||||
|
|
||||||
|
八番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ八層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。回廊の壁には歴代の院長の肖像が並び、その視線は生きているかのように彼の背中を追った。埃の匂いが記憶を刺激した。
|
||||||
|
|
||||||
|
第二章 響動計画
|
||||||
|
|
||||||
|
夜明け前、鈴木は再び書庫に戻り、竜の紋章を布に包んで持ち出した。魔法学院の鐘が三度鳴り、計画の始まりを告げた。
|
||||||
|
|
||||||
|
「響動計画は今夜動き出す」と老司書が言った。その声は震えていたが、目は確かな決意に満ちていた。
|
||||||
|
|
||||||
|
第三章 拒絶計画
|
||||||
|
|
||||||
|
その手紙には拒絶計画という言葉だけが記されていた。鈴木は封を閉じ、暖炉の火にかざした。
|
||||||
|
|
||||||
|
炎は紙を包み、言葉は灰になった。しかし灰の中で、何かが微かに光り続けていた。
|
||||||
|
|
||||||
|
第四章 静かな終わり
|
||||||
|
|
||||||
|
夜が明けると、街は霧に包まれていた。人々は何も知らないまま、いつもの朝を迎えた。
|
||||||
|
|
||||||
|
遠くの丘の上で、旅人がひとり、東の空を眺めていた。風は冷たく、道はまだ長い。
|
||||||
|
|
@ -92,30 +92,22 @@ func (s *Store) GetChunkStatus(bookID string, chapter, chunkIdx int, stage strin
|
||||||
// ChunkStatusesForBook returns every disposition row for a book, ordered for a
|
// ChunkStatusesForBook returns every disposition row for a book, ordered for a
|
||||||
// stable report (tmctl report: the flag section).
|
// stable report (tmctl report: the flag section).
|
||||||
func (s *Store) ChunkStatusesForBook(bookID string) ([]ChunkStatus, error) {
|
func (s *Store) ChunkStatusesForBook(bookID string) ([]ChunkStatus, error) {
|
||||||
ctx, cancel := opContext()
|
return queryAll(s.r, `
|
||||||
defer cancel()
|
|
||||||
rows, err := s.r.QueryContext(ctx, `
|
|
||||||
SELECT chapter, chunk_idx, stage, snapshot_id, content_hash, disposition, flag_reason,
|
SELECT chapter, chunk_idx, stage, snapshot_id, content_hash, disposition, flag_reason,
|
||||||
attempts, final_hash, cost_usd, detail, escalated, escalation_model
|
attempts, final_hash, cost_usd, detail, escalated, escalation_model
|
||||||
FROM chunk_status WHERE book_id = ?
|
FROM chunk_status WHERE book_id = ?
|
||||||
ORDER BY chapter, chunk_idx, stage`, bookID)
|
ORDER BY chapter, chunk_idx, stage`,
|
||||||
if err != nil {
|
func(rows *sql.Rows) (ChunkStatus, error) {
|
||||||
return nil, err
|
cs := ChunkStatus{BookID: bookID}
|
||||||
}
|
var escalated int
|
||||||
defer rows.Close()
|
if err := rows.Scan(&cs.Chapter, &cs.ChunkIdx, &cs.Stage, &cs.SnapshotID, &cs.ContentHash,
|
||||||
var out []ChunkStatus
|
&cs.Disposition, &cs.FlagReason, &cs.Attempts, &cs.FinalHash, &cs.CostUSD, &cs.Detail,
|
||||||
for rows.Next() {
|
&escalated, &cs.EscalationModel); err != nil {
|
||||||
cs := ChunkStatus{BookID: bookID}
|
return cs, err
|
||||||
var escalated int
|
}
|
||||||
if err := rows.Scan(&cs.Chapter, &cs.ChunkIdx, &cs.Stage, &cs.SnapshotID, &cs.ContentHash,
|
cs.Escalated = escalated != 0
|
||||||
&cs.Disposition, &cs.FlagReason, &cs.Attempts, &cs.FinalHash, &cs.CostUSD, &cs.Detail,
|
return cs, nil
|
||||||
&escalated, &cs.EscalationModel); err != nil {
|
}, bookID)
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
cs.Escalated = escalated != 0
|
|
||||||
out = append(out, cs)
|
|
||||||
}
|
|
||||||
return out, rows.Err()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ResetChunkStages deletes the chunk_status rows AND the checkpoints for the given stages of
|
// ResetChunkStages deletes the chunk_status rows AND the checkpoints for the given stages of
|
||||||
|
|
|
||||||
|
|
@ -217,25 +217,15 @@ func (s *Store) GlossaryForBook(bookID string) ([]GlossaryEntry, error) {
|
||||||
// GlossaryRevisionsForBook returns the editorial-time journal (B1) for a book,
|
// GlossaryRevisionsForBook returns the editorial-time journal (B1) for a book,
|
||||||
// newest first, for the report / a human audit.
|
// newest first, for the report / a human audit.
|
||||||
func (s *Store) GlossaryRevisionsForBook(bookID string) ([]GlossaryRevision, error) {
|
func (s *Store) GlossaryRevisionsForBook(bookID string) ([]GlossaryRevision, error) {
|
||||||
ctx, cancel := opContext()
|
return queryAll(s.r, `
|
||||||
defer cancel()
|
|
||||||
rows, err := s.r.QueryContext(ctx, `
|
|
||||||
SELECT src, sense, old_dst, new_dst, editorial_ts, reason
|
SELECT src, sense, old_dst, new_dst, editorial_ts, reason
|
||||||
FROM glossary_revisions WHERE book_id = ?
|
FROM glossary_revisions WHERE book_id = ?
|
||||||
ORDER BY id DESC`, bookID)
|
ORDER BY id DESC`,
|
||||||
if err != nil {
|
func(rows *sql.Rows) (GlossaryRevision, error) {
|
||||||
return nil, err
|
var r GlossaryRevision
|
||||||
}
|
err := rows.Scan(&r.Src, &r.Sense, &r.OldDst, &r.NewDst, &r.EditorialTS, &r.Reason)
|
||||||
defer rows.Close()
|
return r, err
|
||||||
var out []GlossaryRevision
|
}, bookID)
|
||||||
for rows.Next() {
|
|
||||||
var r GlossaryRevision
|
|
||||||
if err := rows.Scan(&r.Src, &r.Sense, &r.OldDst, &r.NewDst, &r.EditorialTS, &r.Reason); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
out = append(out, r)
|
|
||||||
}
|
|
||||||
return out, rows.Err()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// UpsertRetrievalState writes (or overwrites) the per-chunk retrieval-state record.
|
// UpsertRetrievalState writes (or overwrites) the per-chunk retrieval-state record.
|
||||||
|
|
@ -298,30 +288,20 @@ func (s *Store) GetRetrievalState(bookID string, chapter, chunkIdx int) (*Retrie
|
||||||
// RetrievalStatesForBook returns every retrieval-state record for a book, ordered for
|
// RetrievalStatesForBook returns every retrieval-state record for a book, ordered for
|
||||||
// a stable report (the memory section of tmctl report).
|
// a stable report (the memory section of tmctl report).
|
||||||
func (s *Store) RetrievalStatesForBook(bookID string) ([]RetrievalState, error) {
|
func (s *Store) RetrievalStatesForBook(bookID string) ([]RetrievalState, error) {
|
||||||
ctx, cancel := opContext()
|
return queryAll(s.r, `
|
||||||
defer cancel()
|
|
||||||
rows, err := s.r.QueryContext(ctx, `
|
|
||||||
SELECT chapter, chunk_idx, snapshot_id, n_exact_hits, n_sticky, n_ambiguous_flagged,
|
SELECT chapter, chunk_idx, snapshot_id, n_exact_hits, n_sticky, n_ambiguous_flagged,
|
||||||
n_spoiler_blocked, n_evicted, embedding_tier_used, n_postcheck_miss, postcheck_detail, injected_ids,
|
n_spoiler_blocked, n_evicted, embedding_tier_used, n_postcheck_miss, postcheck_detail, injected_ids,
|
||||||
n_style_flags, style_detail
|
n_style_flags, style_detail
|
||||||
FROM retrieval_state WHERE book_id = ?
|
FROM retrieval_state WHERE book_id = ?
|
||||||
ORDER BY chapter, chunk_idx`, bookID)
|
ORDER BY chapter, chunk_idx`,
|
||||||
if err != nil {
|
func(rows *sql.Rows) (RetrievalState, error) {
|
||||||
return nil, err
|
rs := RetrievalState{BookID: bookID}
|
||||||
}
|
err := rows.Scan(&rs.Chapter, &rs.ChunkIdx, &rs.SnapshotID, &rs.NExactHits, &rs.NSticky,
|
||||||
defer rows.Close()
|
&rs.NAmbiguousFlagged, &rs.NSpoilerBlocked, &rs.NEvicted, &rs.EmbeddingTierUsed,
|
||||||
var out []RetrievalState
|
&rs.NPostcheckMiss, &rs.PostcheckDetail, &rs.InjectedIDs,
|
||||||
for rows.Next() {
|
&rs.NStyleFlags, &rs.StyleDetail)
|
||||||
rs := RetrievalState{BookID: bookID}
|
return rs, err
|
||||||
if err := rows.Scan(&rs.Chapter, &rs.ChunkIdx, &rs.SnapshotID, &rs.NExactHits, &rs.NSticky,
|
}, bookID)
|
||||||
&rs.NAmbiguousFlagged, &rs.NSpoilerBlocked, &rs.NEvicted, &rs.EmbeddingTierUsed,
|
|
||||||
&rs.NPostcheckMiss, &rs.PostcheckDetail, &rs.InjectedIDs,
|
|
||||||
&rs.NStyleFlags, &rs.StyleDetail); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
out = append(out, rs)
|
|
||||||
}
|
|
||||||
return out, rows.Err()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func boolToInt(b bool) int {
|
func boolToInt(b bool) int {
|
||||||
|
|
|
||||||
115
backend/internal/store/readonly_test.go
Normal file
115
backend/internal/store/readonly_test.go
Normal file
|
|
@ -0,0 +1,115 @@
|
||||||
|
package store
|
||||||
|
|
||||||
|
import (
|
||||||
|
"database/sql"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// readonly_test.go pins the OpenReadOnly contract (пакет №4): no flock (status
|
||||||
|
// works during a live run), hard read-only connections, loud errors on a missing
|
||||||
|
// DB and on a schema the binary does not match.
|
||||||
|
|
||||||
|
// TestOpenReadOnlyConcurrentWithWriter is the headline: a READ-ONLY open must
|
||||||
|
// succeed while a WRITER owns the project (the exclusive flock is held), reads
|
||||||
|
// must see the writer's committed rows, and a stray write through the read-only
|
||||||
|
// store must fail loudly instead of corrupting the live writer's state.
|
||||||
|
func TestOpenReadOnlyConcurrentWithWriter(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "book.db")
|
||||||
|
w, err := Open(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer w.Close()
|
||||||
|
if err := w.InsertRequestLog(RequestLog{BookID: "b1", Chapter: 1, Stage: "draft", Role: "translator", OK: true}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Writer still open (flock held) — the old Open here would fail with
|
||||||
|
// "project database is in use by another tmctl process".
|
||||||
|
ro, err := OpenReadOnly(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read-only open must not need the writer's flock: %v", err)
|
||||||
|
}
|
||||||
|
defer ro.Close()
|
||||||
|
|
||||||
|
rows, err := ro.RequestLogRows("b1")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(rows) != 1 || rows[0].Stage != "draft" {
|
||||||
|
t.Fatalf("read-only store must see the writer's committed rows, got %+v", rows)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A stray write must be a loud SQLITE_READONLY error, never a silent success
|
||||||
|
// (the read-only Store aliases its write pool to the query_only connections).
|
||||||
|
if err := ro.InsertRequestLog(RequestLog{BookID: "b1", Chapter: 2, Stage: "edit"}); err == nil {
|
||||||
|
t.Fatal("a write through the read-only store must fail")
|
||||||
|
}
|
||||||
|
|
||||||
|
// The writer keeps working while the reader is open (WAL concurrent access).
|
||||||
|
if err := w.InsertRequestLog(RequestLog{BookID: "b1", Chapter: 2, Stage: "edit", Role: "editor", OK: true}); err != nil {
|
||||||
|
t.Fatalf("writer must not be disturbed by an open reader: %v", err)
|
||||||
|
}
|
||||||
|
rows, err = ro.RequestLogRows("b1")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(rows) != 2 {
|
||||||
|
t.Fatalf("reader must see the writer's new committed row, got %d rows", len(rows))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestOpenReadOnlyMissingDB: a read of a never-run project must not silently
|
||||||
|
// create an empty database — the error wraps os.ErrNotExist so the runner can
|
||||||
|
// fall back to the creating Open on first touch.
|
||||||
|
func TestOpenReadOnlyMissingDB(t *testing.T) {
|
||||||
|
_, err := OpenReadOnly(filepath.Join(t.TempDir(), "nope.db"))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("read-only open of a missing DB must fail")
|
||||||
|
}
|
||||||
|
if !errors.Is(err, os.ErrNotExist) {
|
||||||
|
t.Fatalf("error must wrap os.ErrNotExist for the first-touch fallback, got: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestOpenReadOnlySchemaMismatch: read-only open never migrates, so a database
|
||||||
|
// behind (or ahead of) this binary's schema is a loud, actionable error — not a
|
||||||
|
// query that fails later on a missing column.
|
||||||
|
func TestOpenReadOnlySchemaMismatch(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "book.db")
|
||||||
|
w, err := Open(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
w.Close()
|
||||||
|
|
||||||
|
// Roll the recorded version back one step (simulates a DB from an older binary).
|
||||||
|
db, err := sql.Open("sqlite", "file:"+path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := db.Exec(`DELETE FROM schema_version WHERE version = (SELECT MAX(version) FROM schema_version)`); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
db.Close()
|
||||||
|
|
||||||
|
if _, err := OpenReadOnly(path); err == nil {
|
||||||
|
t.Fatal("read-only open of an older-schema DB must fail loudly (it never migrates)")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ahead of the binary: an unknown future version must fail too.
|
||||||
|
db, err = sql.Open("sqlite", "file:"+path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := db.Exec(`INSERT INTO schema_version (version) VALUES (?), (?)`, len(migrations), len(migrations)+7); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
db.Close()
|
||||||
|
if _, err := OpenReadOnly(path); err == nil {
|
||||||
|
t.Fatal("read-only open of a newer-schema DB must fail loudly")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -2,6 +2,7 @@ package store
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"database/sql"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
|
|
||||||
"textmachine/backend/internal/obs"
|
"textmachine/backend/internal/obs"
|
||||||
|
|
@ -82,12 +83,18 @@ func (s *Store) FreshCallLatencyMS(bookID string) (totalMS int64, calls int, err
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// RequestLogView is one request_log row for inspection (tmctl report).
|
// RequestLogView is one request_log row for inspection (tmctl report; the golden
|
||||||
|
// determinism guard reads Chapter/ChunkIdx/RequestHash/Degraded to pin every call's
|
||||||
|
// request_hash byte-for-byte).
|
||||||
type RequestLogView struct {
|
type RequestLogView struct {
|
||||||
TS string
|
TS string
|
||||||
|
Chapter int
|
||||||
|
ChunkIdx int
|
||||||
Stage string
|
Stage string
|
||||||
Role string
|
Role string
|
||||||
|
ModelRequested string
|
||||||
ModelActual string
|
ModelActual string
|
||||||
|
RequestHash string
|
||||||
PromptTokens int
|
PromptTokens int
|
||||||
CachedTokens int
|
CachedTokens int
|
||||||
CacheCreationTokens int
|
CacheCreationTokens int
|
||||||
|
|
@ -97,36 +104,28 @@ type RequestLogView struct {
|
||||||
LatencyMS int
|
LatencyMS int
|
||||||
FinishReason string
|
FinishReason string
|
||||||
TMHit int
|
TMHit int
|
||||||
|
Degraded string
|
||||||
|
Err string
|
||||||
OK int
|
OK int
|
||||||
}
|
}
|
||||||
|
|
||||||
// RequestLogRows returns all request_log rows for a book (tmctl report / приёмка
|
// RequestLogRows returns all request_log rows for a book (tmctl report / приёмка
|
||||||
// Фазы 0). Строки МАТЕРИАЛИЗУЮТСЯ под op-таймаутом и возвращаются срезом —
|
// Фазы 0). Материализация под op-таймаутом и rows.Err() — гарантии queryAll
|
||||||
// отдавать *sql.Rows нельзя: ленивая итерация у вызывающего переживает
|
// («context canceled» посреди чтения / молча усечённая таблица — находки приёмки).
|
||||||
// `defer cancel()` этого метода, и контекст отменяется ПОСРЕДИ чтения
|
|
||||||
// («context canceled», обрыв таблицы report — находка реальной приёмки).
|
|
||||||
func (s *Store) RequestLogRows(bookID string) ([]RequestLogView, error) {
|
func (s *Store) RequestLogRows(bookID string) ([]RequestLogView, error) {
|
||||||
ctx, cancel := opContext()
|
return queryAll(s.r, `
|
||||||
defer cancel()
|
SELECT ts, chapter, chunk_idx, stage, role, model_requested, model_actual,
|
||||||
rows, err := s.r.QueryContext(ctx, `
|
request_hash, prompt_tokens, cached_tokens,
|
||||||
SELECT ts, stage, role, model_actual, prompt_tokens, cached_tokens,
|
|
||||||
cache_creation_tokens, completion_tokens, reasoning_tokens,
|
cache_creation_tokens, completion_tokens, reasoning_tokens,
|
||||||
cost_usd, latency_ms, finish_reason, tm_hit, ok
|
cost_usd, latency_ms, finish_reason, tm_hit, degraded, err, ok
|
||||||
FROM request_log WHERE book_id = ? ORDER BY id`, bookID)
|
FROM request_log WHERE book_id = ? ORDER BY id`,
|
||||||
if err != nil {
|
func(rows *sql.Rows) (RequestLogView, error) {
|
||||||
return nil, err
|
var v RequestLogView
|
||||||
}
|
err := rows.Scan(&v.TS, &v.Chapter, &v.ChunkIdx, &v.Stage, &v.Role,
|
||||||
defer rows.Close()
|
&v.ModelRequested, &v.ModelActual, &v.RequestHash,
|
||||||
var out []RequestLogView
|
&v.PromptTokens, &v.CachedTokens, &v.CacheCreationTokens,
|
||||||
for rows.Next() {
|
&v.CompletionTokens, &v.ReasoningTokens, &v.CostUSD,
|
||||||
var v RequestLogView
|
&v.LatencyMS, &v.FinishReason, &v.TMHit, &v.Degraded, &v.Err, &v.OK)
|
||||||
if err := rows.Scan(&v.TS, &v.Stage, &v.Role, &v.ModelActual,
|
return v, err
|
||||||
&v.PromptTokens, &v.CachedTokens, &v.CacheCreationTokens,
|
}, bookID)
|
||||||
&v.CompletionTokens, &v.ReasoningTokens, &v.CostUSD,
|
|
||||||
&v.LatencyMS, &v.FinishReason, &v.TMHit, &v.OK); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
out = append(out, v)
|
|
||||||
}
|
|
||||||
return out, rows.Err()
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,7 @@
|
||||||
package store
|
package store
|
||||||
|
|
||||||
|
import "database/sql"
|
||||||
|
|
||||||
// ruby.go: the ruby/furigana readings captured on ingest (шаг 3a, v4 schema).
|
// ruby.go: the ruby/furigana readings captured on ingest (шаг 3a, v4 schema).
|
||||||
// This is CAPTURE-ONLY storage — the reading layer that epub-v1 text extraction
|
// This is CAPTURE-ONLY storage — the reading layer that epub-v1 text extraction
|
||||||
// would otherwise silently drop (04-unhappy §4 / D9). Memory v2 (шаг 4) reads it
|
// would otherwise silently drop (04-unhappy §4 / D9). Memory v2 (шаг 4) reads it
|
||||||
|
|
@ -52,23 +54,13 @@ func (s *Store) ReplaceRubyReadings(bookID string, rows []RubyReading) error {
|
||||||
// RubyReadingsForBook returns every captured reading for a book, ordered
|
// RubyReadingsForBook returns every captured reading for a book, ordered
|
||||||
// deterministically (first_chapter, base, reading) for a stable report/consumer.
|
// deterministically (first_chapter, base, reading) for a stable report/consumer.
|
||||||
func (s *Store) RubyReadingsForBook(bookID string) ([]RubyReading, error) {
|
func (s *Store) RubyReadingsForBook(bookID string) ([]RubyReading, error) {
|
||||||
ctx, cancel := opContext()
|
return queryAll(s.r, `
|
||||||
defer cancel()
|
|
||||||
rows, err := s.r.QueryContext(ctx, `
|
|
||||||
SELECT base, reading, first_chapter, occurrences
|
SELECT base, reading, first_chapter, occurrences
|
||||||
FROM ruby_readings WHERE book_id = ?
|
FROM ruby_readings WHERE book_id = ?
|
||||||
ORDER BY first_chapter, base, reading`, bookID)
|
ORDER BY first_chapter, base, reading`,
|
||||||
if err != nil {
|
func(rows *sql.Rows) (RubyReading, error) {
|
||||||
return nil, err
|
rr := RubyReading{BookID: bookID}
|
||||||
}
|
err := rows.Scan(&rr.Base, &rr.Reading, &rr.FirstChapter, &rr.Occurrences)
|
||||||
defer rows.Close()
|
return rr, err
|
||||||
var out []RubyReading
|
}, bookID)
|
||||||
for rows.Next() {
|
|
||||||
rr := RubyReading{BookID: bookID}
|
|
||||||
if err := rows.Scan(&rr.Base, &rr.Reading, &rr.FirstChapter, &rr.Occurrences); err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
out = append(out, rr)
|
|
||||||
}
|
|
||||||
return out, rows.Err()
|
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -92,9 +92,65 @@ func Open(path string) (*Store, error) {
|
||||||
return s, nil
|
return s, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// OpenReadOnly opens an EXISTING project database for the $0 read-only projections
|
||||||
|
// (`tmctl status`/`report`, D15.3/D20.4) WITHOUT the exclusive flock, migrations or
|
||||||
|
// the reservation-recovery pass. Раньше эти команды шли через Open → эксклюзивный
|
||||||
|
// flock, и оператор был заперт от status РОВНО на время живого прогона — «висит
|
||||||
|
// 300-я глава, а посмотреть нечем» (боль smoke-прогона пакета №4). WAL штатно
|
||||||
|
// допускает конкурентных читателей при живом писателе; замок нужен ТОЛЬКО из-за
|
||||||
|
// recoverReservations (обнулил бы живые резервы бегущего процесса) — а read-путь
|
||||||
|
// этот проход не выполняет, поэтому и замок ему не нужен.
|
||||||
|
//
|
||||||
|
// Гарантии: соединения жёстко read-only (PRAGMA query_only=1 — случайная запись
|
||||||
|
// падает громко, а не портит состояние живого писателя); отсутствующая БД — громкая
|
||||||
|
// ошибка (не создаём пустую); схема старше бинаря — громкая ошибка «прогоните
|
||||||
|
// write-команду» (миграции применяет только писатель). Известная честная граница:
|
||||||
|
// после краха прогона reserved_usd остаётся ненулевым до следующей WRITE-команды
|
||||||
|
// (recovery-проход только там) — status покажет этот хвост как reserved.
|
||||||
|
func OpenReadOnly(path string) (*Store, error) {
|
||||||
|
if _, err := os.Stat(path); err != nil {
|
||||||
|
return nil, fmt.Errorf("store: project database %s does not exist yet — run `tmctl translate` first: %w", path, err)
|
||||||
|
}
|
||||||
|
// Без journal_mode(WAL): режим журнала — свойство ФАЙЛА (Open всегда создаёт
|
||||||
|
// WAL, флаг персистентен), а попытка его выставить на query_only-соединении
|
||||||
|
// была бы записью. foreign_keys/synchronous читателю нерелевантны.
|
||||||
|
v := url.Values{}
|
||||||
|
v.Add("_pragma", "busy_timeout(5000)")
|
||||||
|
v.Add("_pragma", "query_only(1)")
|
||||||
|
r, err := sql.Open("sqlite", "file:"+path+"?"+v.Encode())
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("store: open read-only pool: %w", err)
|
||||||
|
}
|
||||||
|
r.SetMaxOpenConns(4)
|
||||||
|
// w намеренно указывает на тот же query_only-пул: заблудившаяся запись через
|
||||||
|
// read-only Store вернёт SQLITE_READONLY громко, а не nil-panic и не тихий успех.
|
||||||
|
s := &Store{w: r, r: r, lock: nil}
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), opTimeout)
|
||||||
|
defer cancel()
|
||||||
|
var current int
|
||||||
|
if err := s.r.QueryRowContext(ctx, `SELECT COALESCE(MAX(version), 0) FROM schema_version`).Scan(¤t); err != nil {
|
||||||
|
s.Close()
|
||||||
|
return nil, fmt.Errorf("store: read schema version of %s (пустая/битая база?): %w", path, err)
|
||||||
|
}
|
||||||
|
if current < len(migrations) {
|
||||||
|
s.Close()
|
||||||
|
return nil, fmt.Errorf("store: %s is at schema v%d, this binary expects v%d — run a write command (`tmctl translate`/`redrive`) to migrate first (read-only open never migrates)",
|
||||||
|
path, current, len(migrations))
|
||||||
|
}
|
||||||
|
if current > len(migrations) {
|
||||||
|
s.Close()
|
||||||
|
return nil, fmt.Errorf("store: %s is at schema v%d, NEWER than this binary's v%d — обновите tmctl (читать новую схему старым кодом небезопасно)",
|
||||||
|
path, current, len(migrations))
|
||||||
|
}
|
||||||
|
return s, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (s *Store) Close() error {
|
func (s *Store) Close() error {
|
||||||
err1 := s.w.Close()
|
err1 := s.w.Close()
|
||||||
err2 := s.r.Close()
|
var err2 error
|
||||||
|
if s.r != s.w {
|
||||||
|
err2 = s.r.Close()
|
||||||
|
}
|
||||||
releaseLock(s.lock)
|
releaseLock(s.lock)
|
||||||
if err1 != nil {
|
if err1 != nil {
|
||||||
return err1
|
return err1
|
||||||
|
|
@ -129,6 +185,32 @@ func opContext() (context.Context, context.CancelFunc) {
|
||||||
return context.WithTimeout(context.Background(), opTimeout)
|
return context.WithTimeout(context.Background(), opTimeout)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// queryAll runs a read query and scans every row into a slice (пакет №4 — пять
|
||||||
|
// изоморфных scan-циклов; новые read-models D15.2/аннотатора получают его даром).
|
||||||
|
// Гарантии, которые рукописный цикл забывает: rows.Err() всегда проверен (ошибка
|
||||||
|
// драйвера ПОСРЕДИ итерации не может молча усечь результат), строки
|
||||||
|
// МАТЕРИАЛИЗУЮТСЯ под op-таймаутом (ленивая итерация у вызывающего пережила бы
|
||||||
|
// defer cancel() и рвалась «context canceled» посреди чтения — находка реальной
|
||||||
|
// приёмки), rows.Close всегда закрыт.
|
||||||
|
func queryAll[T any](db *sql.DB, query string, scan func(*sql.Rows) (T, error), args ...any) ([]T, error) {
|
||||||
|
ctx, cancel := opContext()
|
||||||
|
defer cancel()
|
||||||
|
rows, err := db.QueryContext(ctx, query, args...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
var out []T
|
||||||
|
for rows.Next() {
|
||||||
|
v, err := scan(rows)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, v)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
// recoverReservations zeroes reserved_usd left over by a crashed process. The
|
// recoverReservations zeroes reserved_usd left over by a crashed process. The
|
||||||
// project file is owned by ONE process at a time (CLI), so any reservation
|
// project file is owned by ONE process at a time (CLI), so any reservation
|
||||||
// present at open belongs to a run that never settled — the recovery pass из
|
// present at open belongs to a run that never settled — the recovery pass из
|
||||||
|
|
|
||||||
|
|
@ -647,6 +647,40 @@ keep-alive (Ф1–2), инъекция глоссария (`selective`), пор
|
||||||
4. **`escalation.budget_usd>0`** приёмочная сессия 蛊真人 обязана выставить, иначе echo-эскалация черновика (D18) не стреляет (документировано в pipeline-c1.yaml; дефолт 0 держит CI без gemini/mistral-ключей).
|
4. **`escalation.budget_usd>0`** приёмочная сессия 蛊真人 обязана выставить, иначе echo-эскалация черновика (D18) не стреляет (документировано в pipeline-c1.yaml; дефолт 0 держит CI без gemini/mistral-ключей).
|
||||||
5. gpt-5-mini жив пробой, но ушёл с текущей прайс-страницы (там gpt-5.4/5.5/5.6) — оставлен кандидатом D3, не в цепочке Ф1; пересмотр к Ф2.
|
5. gpt-5-mini жив пробой, но ушёл с текущей прайс-страницы (там gpt-5.4/5.5/5.6) — оставлен кандидатом D3, не в цепочке Ф1; пересмотр к Ф2.
|
||||||
|
|
||||||
|
### 2026-07-10 (пакет-4) — Код-хелс: golden-гард + план рефакторинга (инвентаризация)
|
||||||
|
|
||||||
|
**Golden-гард ПОСТРОЕН ДО любых правок** (`internal/pipeline/golden_test.go` + фикстура `testdata/golden/`): на статичной 3-главной книге (2-чанковая глава со sticky/инъекцией, спойлер-окно since_ch:2, вложенный ключ 紋章⊂竜の紋章, CJK-эхо→эскалация→ре-гейт ok, hard refusal→фолбэк тоже отказал→flag+skip+exit 2) пинится бит-в-бит: snapshotID+payload, request_hash ВСЕХ вызовов (вкл. эскалационный), wire-байты всех 9 тел запросов *(оркестратор: финальная фикстура после селфревью-расширения — 4 главы / 5 чанков / 11 тел)*, chunk_status/retrieval_state, финальные тексты — для свежего прогона И resume ($0, 0 вызовов). Обновление — только `TM_UPDATE_GOLDEN=1` на осознанной смене поведения. Попутно расширен read-view `RequestLogView` (chapter/chunk/model_requested/request_hash/degraded/err — колонки были в таблице, view их терял).
|
||||||
|
|
||||||
|
**Инвентаризация болей — исполнением, не чтением** (workflow 4 аудитора: smoke 6 сценариев на моках через реальный tmctl [hang/500-storm/429+Retry-After/refusal/ceiling/clean+resume], аудит цепочек ошибок, tmctl-тестируемость, скан дублей ≥3):
|
||||||
|
- **Мажоры (все воспроизведены прогоном):** (1) эксклюзивный flock `store.Open` запирает read-only `status`/`report` на всё время живого прогона — оператор слеп ровно когда «висит 300-я глава» (lock нужен только ради recoverReservations, который read-пути не нужен); (2) ноль строк лога, пока вызов в полёте (до ~15 мин тишины при дефолтных таймаутах) — «висит» неотличимо от «работает» даже на debug; (3) backoff/Retry-After спит молча (доказан 8-с разрыв) и WARN «will retry» врёт на последней попытке; (4) store-ошибки всплывают голым SQLite-текстом без операции/книги/главы/чанка.
|
||||||
|
- **Миноры:** терминальная ошибка стадии без ch/chunk/model; `report` не печатает ch/chunk/err; ceiling-ошибка «0.00$» при 0.001 и без committed/reserved; release-фейл резервации глотается на 2 из 3 путей; ReqInfo вешается на ctx ПОСЛЕ resume fast-path (resume-строки без book/role); status глотает ошибку drift-проекции («нет дрифта» по умолчанию); failover-причины только на debug.
|
||||||
|
- **Что уже хорошо (не сломать):** все money-цепочки %w целы (errReserveCeiling/BilledDecodeError/CompletedWithFlags — верифицировано трассировкой), per-stage строки логов несут полную ось, exit-коды 0/1/2 держатся, retry/backoff — единственный движок.
|
||||||
|
- **Дубли ≥3 (честный счёт):** 5×RequestLog-префикс атрибуции и 6×`_ = SetJobStatus` в runner.go; 5 изоморфных scan-циклов в store. Ниже порога (осознанно не трогать): snapshotID primary/escalate зеркало (2×, tripwire — третья модельная ось канала B), тела транспортов (2×, разные wire-форматы), tmctl RW/RO прологи (4×, но это контракт D20.4).
|
||||||
|
|
||||||
|
**План (по приоритету промта, каждый шаг: build+vet+test -race зелёные + golden бит-в-бит):** (1) декомпозиция runner.go 1383 → ~7 связных файлов одного пакета (snapshot/сид/цикл-книги/цикл-чанка/стадия/эскалация/resume; единственная правка формы — выделение эскалационного блока runStage в метод с точным сохранением семантики) — критерий: канал B/annotator/voice-инъекция добавляются файлом; (2) логирование по снятым болям (in-flight строка, честный backoff-лог, обёртки store-ошибок с осью, ceiling с деньгами, release-warn, ReqInfo hoist, run-start/run-end строки) + read-only store open без flock (query_only-пул, skip migrate/recover, fail-loud на отставшей схеме); (3) tmctl: parseInvocation/exitCode/parseDotEnv + fetch/render split (io.Writer, байт-в-байт) + юниты на контрактные инварианты (bad-flag→1 не 2, селектор redrive, парные кавычки dotenv); (4) хелперы дублей ≥3; (5) границы пакета pipeline — НЕ трогаю (оценка после (1): под-пакеты не дают очевидного выигрыша при высокой связности memory↔runner↔render; в «осознанно не тронуто»). Оценка диффа: перемещений ~1.3k строк, содержательных правок ~150–250 строк, новых тестов ~400–600 строк.
|
||||||
|
|
||||||
|
### 2026-07-10 (пакет-4, итог) — Код-хелс рефакторинг ЗАВЕРШЁН
|
||||||
|
|
||||||
|
Весь план исполнен; `go build ./... && go vet ./... && go test ./... -race` зелёные на каждом шаге; **golden бит-в-бит после каждого шага** (перегенерирован один раз ОСОЗНАННО в финале: расширение фикстуры по находке селфревью + починка verb'ов capture-формата — поведение пайплайна не менялось, эквивалентность против HEAD доказана исполнением, см. селфревью). **Ничего не коммичено** (лендинг — оркестратор). Дифф: 41 файл, +3317/−1712 (счёт испр. оркестратором; backend-only 40 файлов +3283; «+3295» снят до дописи журнала), из них новых тестов ровно 855 строк; чужие зоны не тронуты.
|
||||||
|
|
||||||
|
**Сделано (что удешевляет / какой класс багов исключает):**
|
||||||
|
1. **runner.go 1383 → 8 файлов** (`runner` 168 сетап · `snapshot` 262 · `seeding` 118 · `bookrun` 179 · `chunkrun` 234 · `stagerun` 458 · `escalation` 107 · `resume` 70): перенос дословный (эквивалентность верифицирована AST-диффом селфревью), единственная правка формы — `maybeEscalate` из runStage (семантика `escalated`/ceiling-degrade/replay сохранена точно). Ф2-стройки садятся файлом: канал B → escalation.go, annotator/voice → chunkrun.go.
|
||||||
|
2. **Логирование как продукт** (все фиксы — по болям, снятым ИСПОЛНЕНИЕМ, не воображаемым): in-flight строка `calling model` перед каждым вызовом (висящий провайдер ≠ мёртвый процесс; было до ~15 мин тишины); retryLoop — фактическое ожидание `retry_in` в WARN (Retry-After до 5 мин был невидим), «will retry» больше НЕ врёт на последней попытке, debug-строка старта попытки, deadline-exceeded аннотирован `timeouts.attempt_s`; `book run started/chapter started/book run finished` (N/M и деньги прогона на stderr); ~12 store-ошибок обёрнуты осью book/ch/chunk/stage; терминальная ошибка стадии несёт ch/chunk/model; ceiling-ошибка — committed/reserved/estimate и `%g` (было «0.00$» при потолке 0.001); release-фейлы больше не глотаются (Error); ReqInfo hoist — resume/re-pin строки получили book/role; status: провал drift-проекции — WARN, не тихое «дрифта нет»; failover: причина трипа Warn, платный local→cloud ретрай Info (D4.3).
|
||||||
|
3. **`store.OpenReadOnly` + wiring в `NewReadOnlyRunner`** — главный операционный фикс: `tmctl status/report` работают ПРИ ЖИВОМ прогоне (раньше эксклюзивный flock запирал оператора ровно на время «висит 300-я глава»). Без flock/миграций/recoverReservations; соединения `query_only` (заблудшая запись падает громко); нет базы → fallback на создающий Open (перворазовый status как раньше); схема старше/новее бинаря → громкая ошибка. Тесты: конкурентный reader-при-writer, запрет записи, missing DB, schema mismatch, runner-уровень «status при живом прогоне».
|
||||||
|
4. **tmctl 430 → тонкий main 158 + `invocation/render/dotenv`.go, 0 → 18 юнитов** на замороженные контракты: exit 2 эксклюзивен (bad flag → 1, обёрнутый сентинел → 2), порядок валидации, селектор redrive (класс регресса D20.4 «parsed but ignored»), парные кавычки dotenv, sentinel-возвраты рендеров, частичный вывод report при ошибке чтения. Рендеры — `fmt.Fprintf(w)` байт-в-байт; чтения store — коллбеками с ИСТОРИЧЕСКИМ interleaved-порядком.
|
||||||
|
5. **Дубли ≥3:** `baseRequestLog` (5 копий префикса атрибуции — новый путь вызова не может забыть join-ключи телеметрии), `setJobStatus` c WARN (6 «тихих» `_ =` — рассинхрон jobs↔chunk_status теперь виден), store `queryAll` generic (5 изоморфных сканов; rows.Err()/материализация гарантированы для будущих read-models D15.2), `slices.Sorted(maps.Keys())` ×3.
|
||||||
|
6. **Расширен `RequestLogView`** (ch/chunk/model_requested/request_hash/degraded/err — колонки БД, которые view терял) — нужен golden-гарду и report'у. **Осознанное изменение human-таблицы `report`**: колонки ch/chunk, модель с фолбэком «(req)», хвост degraded/err (пост-мортем упавшего вызова был пустой анонимной строкой; `--json`-схема status НЕ менялась). README: карта пакетов обновлена + пункт про golden-гард.
|
||||||
|
|
||||||
|
**Селфревью (агентское адверсариальное, 5 осей find → refute-by-default verify, 14 агентов ~880k ток.): 21 кандидат → 7 CONFIRMED (все закрыты), 1 downgraded→info, 1 refuted, 12 info.** Ключевое: (major) `slices.Sorted` на пустой карте даёт nil → `retrieval_state.injected_ids` персистился бы `"null"` вместо исторического `"[]"` на каждом чанке без точных матчей, а golden-фикстура была слепа к пустой выборке → нормализация в `[]` + **фикстура расширена 4-й главой без матчей** (пустая строка теперь запинена); (minor) printf-дефекты capture-формата golden (`err=%!q(MISSING)`, `%t` на int — err-колонка была НЕ запинена; vet не ловит через клоужер) → починены, golden перегенерирован; (minor) errTail резал UTF-8 по байту → срез по границе руны; (minor) report хоистил чтения до печати — частичный аудит при ошибке чтения молча пустел → возвращён interleaved-порядок коллбеками + тест; (minor) 7 устаревших указателей «(runner.go)» в комментариях + шапка runner.go описывала монолит → исправлены. Позитив ревью: AST-дифф всех 36 деклараций — дословно кроме документированных правок; **новый golden прогнан против HEAD-пайплайна — PASS** (прямое доказательство эквивалентности wire/вердиктов); truth-table retryLoop — эквивалентность по всем (attempt, retryable, ctx); money-цепочки и sentinel-цепочки целы.
|
||||||
|
|
||||||
|
**Осознанно не тронуто (причины):** под-пакеты pipeline (memory/gates/ingest делят неэкспортируемые типы с раннером; выигрыша нет — «не тащить через силу»); snapshotID primary/escalate зеркало 2× (< порога 3; **tripwire-коммент в snapshot.go**: извлечь foldModelWire ПЕРЕД третьей модельной осью канала B); тела транспортов openai/anthropic 2× (разные wire-форматы); tmctl RW/RO прологи 4× (контракт D20.4); persistRetrievalState marshal ×3 (недостижимый failure mode); F3 at-most-once, Escal.Chains, D15.2-реализация, store→ORM, перф — анти-скоуп промта.
|
||||||
|
|
||||||
|
**Честные оговорки / пинги оркестратору:**
|
||||||
|
1. Пошаговая история «golden зелёный после каждого шага» не восстановима из git (пакет — одно рабочее дерево без промежуточных коммитов); компенсация — прогон финального golden против HEAD-кода селфревью (PASS) + полный сьют на каждом шаге. *(Оркестратор, та же оговорка шире: smoke-сценарии инвентаризации [hang/500-storm/429/ceiling] следов в диффе не оставили, а лог-фиксы retryLoop/errTail тест-ассертов не имеют — реверт errTail-фикса выживает сьют; в fix-лист D23.4.)*
|
||||||
|
2. **Документированное поведенческое следствие OpenReadOnly:** после краха прогона `status` показывает хвост `reserved_usd` до следующей WRITE-команды (recovery-проход только у писателя; раньше read-команда сама обнуляла резервы — что и было причиной flock). Задокументировано в store.go.
|
||||||
|
3. **Пинг (зона docs/):** `architecture/06-memory-risk-registry.md:71` цитирует «snapshotID (runner.go:145-163)» и `07-strategic-review` содержит runner.go-линки — после декомпозиции указатели ведут в сетап-файл; поправить при следующем касании доков (сами механизмы не менялись: snapshotID/memoryVersion → snapshot.go).
|
||||||
|
4. Известная info-грань: OpenReadOnly на read-only ФС (бэкап-снапшот) не работает (WAL требует создать -shm) и создаёт -shm/-wal рядом с БД при чтении голой копии; хинт ошибки «пустая/битая база?» в этом кейсе вводит в заблуждение. Не регресс (старый путь падал раньше и жёстче); лечить — только формулировкой хинта, авто-`immutable=1` небезопасен при живом писателе.
|
||||||
|
|
||||||
## Полигон
|
## Полигон
|
||||||
(секция параллельной сессии — записи добавлять сюда)
|
(секция параллельной сессии — записи добавлять сюда)
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue