diff --git a/backend/README.md b/backend/README.md index 23a5227b..f9f1551b 100644 --- a/backend/README.md +++ b/backend/README.md @@ -8,10 +8,10 @@ | Пакет | Что делает | Ключевые файлы | |---|---|---| -| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) / `manifest` (**$0-производитель персиста манифеста глав/чанков**, строка 100: нужен ДО первого прогона — дерево глав разобранной, но не запущенной книги; `--json` печатает сам документ). Денежный аргумент прогона: `--ceiling-usd` (строка 145) перекрывает книжный `ceilings.book_usd` ТОЛЬКО на этот прогон, в book.yaml не пишется, ноль/отрицательное = отказ запуска. main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` | +| `cmd/tmctl` | CLI: `translate` / `status` (read-only N/M+паспорта глав+деньги, `--json`; работает ПРИ живом прогоне — store без flock) / `report` (request-log+леджер книги **+ per-run quality-report**: структурный KPI предл./нарратив-абзац, тире, cosmetic-strip/echo-rates, глоссарий-промахи, trust-gated — наблюдаемость, НЕ гейт; D39.2-T4) / `export` (**экспорт-поверхность для полигона**: final_hash→checkpoint→checks.ExportNormalize, manifest-join c pending/ghost-учётом, ConfigDrift-поле, `--plaintext`; D39.5) / `redrive` (переатака флагнутых: `--chapter/--chunk/--reason/--dry-run`, D15.3) / `manifest` (**$0-производитель персиста манифеста глав/чанков**, строка 100: нужен ДО первого прогона — дерево глав разобранной, но не запущенной книги; `--json` печатает сам документ) / `migrate` (**$0-write-open без прогона**, строка 174: read-only команды требуют ТОЧНОГО совпадения схемы и не мигрируют, поэтому апгрейд бинаря запирал все существующие книги — платформа зовёт `status --json` перед каждым спавном, а write-команда, которая мигрировала бы, не наступала никогда. Грузит ТОЛЬКО `book.yaml` — ни цен, ни ключей на $0-пути (строка 146); restore point берёт ПОД ЛОКОМ и только когда шаг реально применяется (имя `<метка>-pre-migrate.db` — вне секундного неймспейса платного пути, строка 173); несовпадение схемы у read-only путей — типизированный отказ **exit 13** с машинным токеном `schema_mismatch found=N expected=M`). Денежный аргумент прогона: `--ceiling-usd` (строка 145) перекрывает книжный `ceilings.book_usd` ТОЛЬКО на этот прогон, в book.yaml не пишется, ноль/отрицательное = отказ запуска. main — тонкая обвязка: разбор/exit-коды/.env/рендеры в тестируемых функциях | `main.go`, `invocation.go`, `render.go`, `dotenv.go` | | `internal/llm` | OpenAI-совместимый транспорт + retry/backoff, **capability-слой** (budget_field/temperature/reasoning per-модель), провайдеры openai/local (no-proxy)/anthropic (DEPRECATED-референс); `failover.go` удалён паком-17 — маршрутизация лейблами живёт в config/pipeline (канал B, один хоп `chain[0]`, fail-closed) | `httpllm.go`, `capability.go`, `provider_*.go` | | `internal/ledger` | Цены по usage (вкл. reasoning/cache-поля), `PriceForResponse` по фактической модели | `pricing.go` | -| `internal/store` | SQLite (modernc, CGO-free), цепочка миграций `schema_version` (⚠ ALTER-шаги v8+ не идемпотентны вопреки шапке — бэклог-строка 49а), reserve/settle+checkpoint, chunk_status, глоссарий (+подписной цикл терминолога), ruby, retrieval_state, request_log; `OpenReadOnly` — без flock/миграций/recovery для status/report/export | `ledger.go`, `migrate.go`, `glossary.go`, `store.go` | +| `internal/store` | SQLite (modernc, CGO-free), цепочка миграций `schema_version` (⚠ ALTER-шаги v8+ не идемпотентны вопреки шапке — бэклог-строка 49а), reserve/settle+checkpoint, chunk_status, глоссарий (+подписной цикл терминолога), ruby, retrieval_state, request_log; `OpenReadOnly` — без flock/миграций/recovery для status/report/export (схема не совпала — типизированный `*store.SchemaMismatchError{Found,Expected}`, обе стороны); `Migrate` (+шов `beforeApply` под локом: restore point берётся там) — поверхность деплой-шага строки 174, `SchemaHead` — та самая `Expected`; write-open БД новее бинаря теперь ОТКАЗ, а не тихое открытие | `ledger.go`, `migrate.go`, `glossary.go`, `store.go` | | `internal/config` | fail-fast загрузка models/pipeline/book; **эхо-мина-гейт** `echoMineViolation`; `CheckRunnable` блокирует неисполнимое (C2/fanout/judge); **пар-слой** (D39.23): `configs/pairs/<пара>.yaml` — калибровка нарезки + коридор длины пары; **промпт-резолв КОНВЕНЦИЕЙ** `prompts/<пара>/<роль>.md` из `Book.LangPair()`+роли, **fail-loud на пару без пакета** (ja-книга не едет молча через zh-промпт), осознанное исключение — `prompt_override` на стадии | `models.go`, `pipeline.go`, `pair.go` | | `internal/text` | Нормализация и рун-примитивы, общие для банка/майнера/чекеров (script-agnostic слой, `lang` НЕ импортирует): `NormalizeSourceKey`/`NormalizeTargetForm` (версионированный артефакт `NormVersion()`, фолдится в снапшот), `NormalizeSource`, `TokenizeScript` (алфавит цели из данных, Mn-фолд U+0301; `TokenizeCyrillic` = ru-дефолт-обёртка), `DenseScript` — единый дом sizing-таксономии (D39.64) | `norm.go`, `runes.go`, `source.go` | | `internal/seed` | Схема сид-YAML книги (`File`/`Term`/`Decl`/`Alias`) — общий артефакт: майнер её ПИШЕТ (mined-дельта), банк ЧИТАЕТ (`LoadGlossarySeed`); тип общий, поэтому эмиссия не может разъехаться с загрузчиком | `seed.go` | @@ -65,6 +65,7 @@ go run ./cmd/tmctl status --config example/book.yaml --json # $0, N/M+пасп go run ./cmd/tmctl report --config example/book.yaml # $0, quality-report (KPI/rates) go run ./cmd/tmctl export --config example/book.yaml # $0, экспорт-JSON для полигона (--plaintext для человека) go run ./cmd/tmctl manifest --config example/book.yaml # $0, пере-строить персист манифеста глав/чанков (--json — сам документ) +go run ./cmd/tmctl migrate --config example/book.yaml # $0, довести схему проекта до головы бинаря (деплой-шаг: стоп прогонов → НОВЫЙ бинарь на место → migrate ИМ по каждой книге без открытых попыток → прогоны в работу) go run ./cmd/tmctl translate --config example/book.yaml --ceiling-usd 0.5 # потолок ТОЛЬКО на этот прогон, book.yaml не пишется # live-conformance (реальные провайдеры, платно, вне CI): set -a; . ./.env; set +a; TM_LIVE=1 go test -tags live -run TestLive -v ./internal/pipeline/ diff --git a/backend/cmd/tmctl/main.go b/backend/cmd/tmctl/main.go index 40c5048b..4ed4113c 100644 --- a/backend/cmd/tmctl/main.go +++ b/backend/cmd/tmctl/main.go @@ -1,4 +1,5 @@ -// tmctl is the TextMachine CLI: translate / report / status / export / redrive / manifest / backup. +// tmctl is the TextMachine CLI: translate / report / status / export / redrive / manifest / backup / +// migrate. // main.go — thin wiring (package №4): argument parsing — invocation.go, // output renderers — render.go, .env — dotenv.go; here just the // «parse → env → ctx → fetch → render» wiring and exit-code mapping. @@ -45,6 +46,7 @@ const ( exitConfigInvalid = 10 // the configuration will not run: unreadable, unparseable, invalid, no key exitSourceUnreadable = 11 // the BOOK's source cannot be read or decoded — the one class about the text exitProjectLocked = 12 // another tmctl owns this project right now; come back later + exitSchemaMismatch = 13 // this project's schema is not this binary's: run `tmctl migrate` (row 174) exitRefusedOther = 19 // a refusal class this build of tmctl has no number for ) @@ -53,6 +55,7 @@ var refusalExit = map[pipeline.RefusalClass]int{ pipeline.RefusalBadConfig: exitConfigInvalid, pipeline.RefusalSourceUnreadable: exitSourceUnreadable, pipeline.RefusalProjectLocked: exitProjectLocked, + pipeline.RefusalSchemaMismatch: exitSchemaMismatch, } // exitCode maps a run() error onto the ratified shell contract (Milestone 2 / R1-FL-A): @@ -183,10 +186,12 @@ func run() error { return manifestCmd(inv.cfgPath, inv.asJSON) case "backup": return backupCmd(inv.cfgPath, os.Stdout) + case "migrate": + return migrateCmd(inv.cfgPath, os.Stdout) case "seed-lint": return seedLint(inv.seedPath) default: - return fmt.Errorf("unknown command %q (want translate|report|status|export|redrive|manifest|backup|seed-lint)", inv.cmd) + return fmt.Errorf("unknown command %q (want translate|report|status|export|redrive|manifest|backup|migrate|seed-lint)", inv.cmd) } } diff --git a/backend/cmd/tmctl/migrate.go b/backend/cmd/tmctl/migrate.go new file mode 100644 index 00000000..703b7e4c --- /dev/null +++ b/backend/cmd/tmctl/migrate.go @@ -0,0 +1,108 @@ +package main + +// migrate.go: the $0 `tmctl migrate` verb — a project's WRITE open with no run behind it (backlog +// row 174). +// +// The deadlock it breaks: the read-only commands (status, report, export, manifest) demand this +// binary's exact schema and never migrate — only a write open applies the chain. So upgrading the +// engine binary locks every existing book at once: the platform calls `tmctl status --json` before each +// spawn and for the money, that call refuses the older schema, and the write command that WOULD have +// migrated the project never comes. The deploy step is "stop the runs → put the NEW binary in place → +// run THIS command over the books with it → let the runs start again", and a caller that meets exit 13 +// in flight can also self-heal from it. +// +// ⚠ WHICH books: the sweep belongs on projects with no OPEN attempt. A migrated project is above the +// schema of every older binary, and the platform settles a finished attempt with the binary that +// attempt was PINNED to — so migrating underneath an attempt whose settle has not happened yet would +// trade this deadlock for a hold that can never be closed. The engine has no notion of an attempt, so +// that rule lives in the caller's sweep (row 175); what the engine guarantees is that a project a live +// run holds is refused rather than migrated (exit 12). +// +// It deliberately loads the BOOK config only. A migration touches no prompt, no provider and no price, +// and pulling in the rest of the stack would drag its gates onto a $0 path — `prices_checked` refuses a +// models.yaml whose prices are over 120 days old (backlog row 146), and an engine deployed onto a stand +// with stale prices would then meet the deadlock again in exactly the shape this command exists to +// remove. Same reason the command needs no provider key: it is an operator safety verb, like `backup`. + +import ( + "errors" + "fmt" + "io" + "io/fs" + "os" + "time" + + "textmachine/backend/internal/config" + "textmachine/backend/internal/pipeline" + "textmachine/backend/internal/store" +) + +// migrateSuffix names what the restore point below is FOR, and it is what keeps this command out of the +// paid path's namespace. `translate`'s pre-flight guard writes backups/.db with a stamp of +// second precision, and BackupSQLite refuses to overwrite an existing one (backlog row 173) — so +// sharing the name would make the very sequence this command enables (migrate, then start the run it +// unblocked) fail inside the pre-flight guard whenever both land in the same second, turning a healthy +// deploy into exit 1 before anything reached a provider. +const migrateSuffix = "-pre-migrate" + +// migrateCmd brings the book's project database to this binary's schema and reports the transition. +// $0: no LLM call, no network, no ledger movement — the only writes are the migration steps themselves +// and the stale-reservation recovery every write open performs. +// +// Idempotent: a project already at head is opened, found current and closed — no step applied, no +// restore point taken, no money touched. +func migrateCmd(cfgPath string, w io.Writer) error { + book, err := config.LoadBook(cfgPath) + if err != nil { + // Like the pre-flight backup guard, this runs before any runner exists, so it is the first thing + // a broken config meets — and a refusal classified here rather than collapsed onto exit 1 is the + // whole of PD-196 (see pipeline/refusal.go). + return pipeline.RefuseConfig(err) + } + // Only ABSENT means "never run": a database this call is about to CREATE has no history to protect, + // and it is the one case that gets no restore point. Any other stat error read that way would skip + // the guard on a file that is there. + dbExists := true + if _, err := os.Stat(book.ProjectDB); err != nil { + if !errors.Is(err, fs.ErrNotExist) { + return fmt.Errorf("tmctl migrate: cannot stat the project database %s: %w", book.ProjectDB, err) + } + dbExists = false + } + + // The restore point is taken INSIDE the project lock, and only when a step is actually going to run + // (store.Migrate's seam). A migration is the one $0 operation that rewrites the file holding the + // owner's signed bank, the checkpoints and the ledger, and the ALTER steps do not converge on a + // half-applied database (row 49а) — a damaged file is not something a re-run repairs. Taking it + // before the lock instead would copy the whole database of a project a live run owns and then refuse + // the migration anyway, once per retry. + backup := func(m store.Migration) error { + if !dbExists { + return nil + } + path, err := store.BackupSQLite(book.ProjectDB, + backupDirFor(book.ProjectDB), backupStamp(time.Now())+migrateSuffix) + if err != nil { + return fmt.Errorf("tmctl migrate: refusing to migrate a project it could not back up first: %w", err) + } + fmt.Fprintf(w, "pre-migrate: backed up the project DB at schema v%d to %s (integrity_check green)\n", m.From, path) + return nil + } + + applied, err := store.Migrate(book.ProjectDB, backup) + if err != nil { + // The two classified open failures a caller acts on: another tmctl holds the project (come back + // later, exit 12) and a database NEWER than this binary (upgrade tmctl, exit 13 — a migration + // cannot repair that direction, so a caller looping on "migrate and retry" has to be told). + return pipeline.RefuseStoreOpen(err) + } + switch { + case !dbExists: + fmt.Fprintf(w, "migrate: created %s at schema v%d\n", book.ProjectDB, applied.To) + case applied.From == applied.To: + fmt.Fprintf(w, "migrate: %s already at schema v%d\n", book.ProjectDB, applied.To) + default: + fmt.Fprintf(w, "migrate: %s schema v%d -> v%d\n", book.ProjectDB, applied.From, applied.To) + } + return nil +} diff --git a/backend/cmd/tmctl/migrate_cli_test.go b/backend/cmd/tmctl/migrate_cli_test.go new file mode 100644 index 00000000..8de41109 --- /dev/null +++ b/backend/cmd/tmctl/migrate_cli_test.go @@ -0,0 +1,398 @@ +package main + +import ( + "bytes" + "context" + "database/sql" + "fmt" + "io" + "net/http" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "strings" + "syscall" + "testing" + "time" + + "textmachine/backend/internal/pipeline" + "textmachine/backend/internal/store" +) + +// migrate_cli_test.go is the CLI half of backlog row 174: the deploy deadlock is a SHELL fact — the +// platform learns what happened from an exit code and a line of stderr — so the contract is exercised +// where it is a contract. The migration mechanics themselves are pinned generically in +// internal/store/migrate_test.go. + +// staleTheProjectSchema rolls the project database's recorded version back one step, which is what an +// older engine binary left behind. +// +// ⚠ This fixture re-applies the newest migration on the next write open, which is fine while that step +// is `CREATE … IF NOT EXISTS` and is NOT guaranteed for a future ALTER step (backlog row 49а). It is +// used here because these tests are about the exit codes and the output, and the deep fixture — the +// migration chain truncated to an older vintage — is only reachable from inside the store package, +// where those properties are tested. If a future migration makes this fail, move the setup rather than +// weakening the assertions. +func staleTheProjectSchema(t *testing.T, dbPath string) int { + t.Helper() + db, err := sql.Open("sqlite", "file:"+dbPath) + if err != nil { + t.Fatal(err) + } + defer db.Close() + if _, err := db.Exec(`DELETE FROM schema_version WHERE version = (SELECT MAX(version) FROM schema_version)`); err != nil { + t.Fatal(err) + } + return schemaVersionOf(t, dbPath) +} + +// schemaVersionOf reads the recorded schema version through its own connection — the state under test +// must be asserted independently of the product code that reports it. +func schemaVersionOf(t *testing.T, dbPath string) int { + t.Helper() + db, err := sql.Open("sqlite", "file:"+dbPath) + if err != nil { + t.Fatal(err) + } + defer db.Close() + var v int + if err := db.QueryRow(`SELECT COALESCE(MAX(version), 0) FROM schema_version`).Scan(&v); err != nil { + t.Fatal(err) + } + return v +} + +// holdTheProject takes the project's exclusive flock the way another tmctl process would, WITHOUT +// opening the store — because a write open would migrate the database, and the state under test is a +// project that is both stale and held. +func holdTheProject(t *testing.T, dbPath string) { + t.Helper() + f, err := os.OpenFile(dbPath+".lock", os.O_CREATE|os.O_RDWR, 0o644) + if err != nil { + t.Fatal(err) + } + if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB); err != nil { + t.Fatal(err) + } + t.Cleanup(func() { + _ = syscall.Flock(int(f.Fd()), syscall.LOCK_UN) + f.Close() + }) +} + +func projectDBOf(bookPath string) string { + return filepath.Join(filepath.Dir(bookPath), "cli-book.db") +} + +func backupsOf(t *testing.T, dbPath string) []string { + t.Helper() + entries, err := os.ReadDir(backupDirFor(dbPath)) + if os.IsNotExist(err) { + return nil + } + if err != nil { + t.Fatal(err) + } + var names []string + for _, e := range entries { + names = append(names, e.Name()) + } + return names +} + +// mockProvider answers every call with a usable draft, so a `translate` really settles money. +func mockProvider(t *testing.T) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = io.ReadAll(r.Body) + fmt.Fprint(w, `{"id":"fake","model":"fake-model","choices":[{"message":{"content":"ЧЕРНОВИК ПЕРЕВОДА"},"finish_reason":"stop"}], + "usage":{"prompt_tokens":1000,"completion_tokens":500}}`) + })) + t.Cleanup(srv.Close) + return srv +} + +// TestTheDeployDeadlockIsBrokenAtTheShell drives the real binary through the exact sequence the +// platform meets on an engine upgrade: `status --json` before a spawn refuses the older schema, and the +// contract must make that refusal actionable — its own number inside the refusal band, and the two +// versions machine-readable — so the caller can migrate and start the run instead of stopping the world. +func TestTheDeployDeadlockIsBrokenAtTheShell(t *testing.T) { + if testing.Short() { + t.Skip("builds and runs the binary") + } + bin := buildTmctl(t) + bookPath := setupCLIProject(t, mockProvider(t).URL) + dbPath := projectDBOf(bookPath) + + // A book an older binary left behind. + st, err := store.Open(dbPath) + if err != nil { + t.Fatal(err) + } + st.Close() + from := staleTheProjectSchema(t, dbPath) + + var stderr bytes.Buffer + status := exec.Command(bin, "status", "--config", bookPath, "--json") + status.Stderr = &stderr + if got := exitCodeOf(t, status.Run()); got != exitSchemaMismatch { + t.Fatalf("`status --json` on an older schema exited %d, want %d", got, exitSchemaMismatch) + } + // The number says WHAT; these two say which way and how far — without them a caller cannot tell the + // direction a migration repairs from the one where it would loop forever. + want := fmt.Sprintf("schema_mismatch found=%d expected=%d", from, store.SchemaHead()) + if !strings.Contains(stderr.String(), want) { + t.Fatalf("the refusal must carry %q machine-readably; got:\n%s", want, stderr.String()) + } + if exitSchemaMismatch < refusalFirst || exitSchemaMismatch > refusalLast { + t.Fatalf("%d is outside the refusal band [%d,%d] every consumer keys on", exitSchemaMismatch, refusalFirst, refusalLast) + } + + out, err := exec.Command(bin, "migrate", "--config", bookPath).Output() + if got := exitCodeOf(t, err); got != 0 { + t.Fatalf("`migrate` exited %d: %s", got, out) + } + if want := fmt.Sprintf("schema v%d -> v%d", from, store.SchemaHead()); !strings.Contains(string(out), want) { + t.Fatalf("migrate must report the transition %q; got:\n%s", want, out) + } + if got := exitCodeOf(t, exec.Command(bin, "status", "--config", bookPath, "--json").Run()); got != 0 { + t.Fatalf("`status --json` after the migration exited %d, want 0 — the deadlock is not broken", got) + } + // …and the run the migration unblocked actually starts. It follows within milliseconds, which is + // what put migrate's restore point and the paid path's pre-flight guard in the same second. + if run := exec.Command(bin, "translate", "--config", bookPath); exitCodeOf(t, run.Run()) != 0 { + t.Fatal("the run spawned right after the migration must start, not die in its pre-flight guard") + } +} + +// TestTheMigrationRestorePointNeverCollidesWithThePaidPath is the deterministic half of the same fact: +// both producers are handed the SAME stamp, because in the deploy sequence they really do land in the +// same second. `store.BackupSQLite` refuses to overwrite a restore point (backlog row 173), so sharing +// the name would make the run refuse to start with exit 1 — outside the refusal band, and after a +// migration that already succeeded. +func TestTheMigrationRestorePointNeverCollidesWithThePaidPath(t *testing.T) { + bookPath := setupCLIProject(t, "http://127.0.0.1:1") + dbPath := projectDBOf(bookPath) + st, err := store.Open(dbPath) + if err != nil { + t.Fatal(err) + } + st.Close() + staleTheProjectSchema(t, dbPath) + + // PRODUCTION takes the restore point — the point of the test. Supplying the name here instead would + // only pin that the constant is non-empty, and the regression to catch is the call site dropping it. + if err := migrateCmd(bookPath, io.Discard); err != nil { + t.Fatal(err) + } + names := backupsOf(t, dbPath) + if len(names) != 1 { + t.Fatalf("restore points after the migration: %v, want exactly one", names) + } + + // Strip the tag production chose and the bare second-stamp must still be free, so translate's + // pre-flight guard can take its own backup in the SAME second. Untagged, the two names are equal and + // BackupSQLite refuses to overwrite — the run the migration just unblocked would die in its + // pre-flight guard with exit 1. No wall clock is involved: the collision is decided by the name. + tagged := strings.TrimSuffix(names[0], ".db") + bare := strings.TrimSuffix(tagged, migrateSuffix) + if bare == tagged { + t.Fatalf("the migration's restore point %q carries no tag of its own — it is inside the paid path's namespace", names[0]) + } + if _, err := store.BackupSQLite(dbPath, backupDirFor(dbPath), bare); err != nil { + t.Fatalf("the paid path's pre-flight backup in the SAME second must still succeed: %v", err) + } +} + +// TestMigrateNeedsNeitherPricesNorModels: the $0 path must not carry the paid path's gates. A +// models.yaml whose prices are older than 120 days refuses to load (backlog row 146), and if the +// migration were built on the full config stack a stand with stale prices would meet the same deadlock +// this command exists to remove — the read command refuses, and the one thing that could migrate +// refuses too. +func TestMigrateNeedsNeitherPricesNorModels(t *testing.T) { + bookPath := setupCLIProject(t, mockProvider(t).URL) + modelsPath := filepath.Join(filepath.Dir(bookPath), "models.yaml") + raw, err := os.ReadFile(modelsPath) + if err != nil { + t.Fatal(err) + } + stale := time.Now().UTC().AddDate(0, 0, -200).Format("2006-01-02") + writeCLIFile(t, modelsPath, strings.Replace(string(raw), time.Now().UTC().Format("2006-01-02"), stale, 1)) + + // The contrast: the read-only projection is refused by the price gate before it reads a schema. + if got := exitCode(status(context.Background(), bookPath, true)); got != exitConfigInvalid { + t.Fatalf("setup: `status` on stale prices exited %d, want %d", got, exitConfigInvalid) + } + var out bytes.Buffer + if err := migrateCmd(bookPath, &out); err != nil { + t.Fatalf("migrate must not consult prices: %v", err) + } + if !strings.Contains(out.String(), fmt.Sprintf("v%d", store.SchemaHead())) { + t.Fatalf("migrate must report the head it brought the project to; got %q", out.String()) + } + + // The structural version of the same claim, which also covers the provider keys the models file + // carries: with no models.yaml at all the migration still runs, so it cannot be consulting either. + if err := os.Remove(modelsPath); err != nil { + t.Fatal(err) + } + if got := exitCode(status(context.Background(), bookPath, true)); got != exitConfigInvalid { + t.Fatalf("setup: `status` without models.yaml exited %d, want %d", got, exitConfigInvalid) + } + if err := migrateCmd(bookPath, io.Discard); err != nil { + t.Fatalf("migrate must not load the model stack at all: %v", err) + } +} + +// TestMigrateKeepsTheMoneyAndRepeatsAsANoOp is the money half at the command level: a real run's +// settled spend, a reservation its process never settled, and a migration between them. `committed` is +// what the platform's ceiling argument is computed from (PD-158) and what a settle reconciles against, +// so it must survive the deploy step untouched; `reserved` is advisory and the write open's recovery +// pass is entitled to zero it. +func TestMigrateKeepsTheMoneyAndRepeatsAsANoOp(t *testing.T) { + bookPath := setupCLIProject(t, mockProvider(t).URL) + dbPath := projectDBOf(bookPath) + if err := translate(context.Background(), bookPath, false, pipeline.RebillConsent{}, false, 0); err != nil { + t.Fatalf("setup run: %v", err) + } + + st, err := store.Open(dbPath) + if err != nil { + t.Fatal(err) + } + committed, _, err := st.SpentUSD("cli-book") + if err != nil { + t.Fatal(err) + } + if committed <= 0 { + t.Fatalf("setup: the run must have settled money, committed=%v", committed) + } + // The reservation a killed run leaves behind. + if _, verdict, err := st.Reserve("cli-book", 0.25, store.Ceilings{BookUSD: 10, DayUSD: 10}); err != nil || verdict != store.ReserveOK { + t.Fatalf("stale reserve: %v %v", verdict, err) + } + st.Close() + from := staleTheProjectSchema(t, dbPath) + + for i := range 2 { + var out bytes.Buffer + if err := migrateCmd(bookPath, &out); err != nil { + t.Fatalf("call %d: %v", i, err) + } + ro, err := store.OpenReadOnly(dbPath) + if err != nil { + t.Fatalf("call %d: the read path must work after migrating: %v", i, err) + } + gotCommitted, gotReserved, err := ro.SpentUSD("cli-book") + ro.Close() + if err != nil { + t.Fatal(err) + } + if gotCommitted != committed { + t.Fatalf("call %d: committed %v -> %v; a migration must not move settled money by a cent", i, committed, gotCommitted) + } + if gotReserved != 0 { + t.Fatalf("call %d: reserved=%v, want the recovery pass to have zeroed it", i, gotReserved) + } + // The restore point is taken for the call that MIGRATES and for no other, and it holds the + // database as it was BEFORE the step — a copy taken afterwards is not a restore point at all. + names := backupsOf(t, dbPath) + if len(names) != 1 { + t.Fatalf("call %d: restore points %v, want exactly the one the migrating call took", i, names) + } + if v := schemaVersionOf(t, filepath.Join(backupDirFor(dbPath), names[0])); v != from { + t.Fatalf("call %d: the restore point is at schema v%d, want the pre-migration v%d", i, v, from) + } + if i == 1 && !strings.Contains(out.String(), "already at schema") { + t.Fatalf("the repeat call must report a no-op; got %q", out.String()) + } + } +} + +func TestMigrateRefusesWithTheSameVocabularyAsEveryOtherCommand(t *testing.T) { + t.Run("a broken config", func(t *testing.T) { + bookPath := setupCLIProject(t, "http://127.0.0.1:1") + writeCLIFile(t, bookPath, "book_id: cli-book\n bad indentation: [\n") + if got := exitCode(migrateCmd(bookPath, io.Discard)); got != exitConfigInvalid { + t.Fatalf("exit %d, want %d", got, exitConfigInvalid) + } + }) + + t.Run("a project another process holds", func(t *testing.T) { + // The deploy step runs while the operator believes the runs are stopped. If one is not, the + // answer is "come back later" — never a migration under a live writer's feet, whose reservations + // the recovery pass would zero. And nothing may be written for that refusal: a restore point + // taken before the lock would copy the whole database once per retry. + bookPath := setupCLIProject(t, "http://127.0.0.1:1") + dbPath := projectDBOf(bookPath) + st, err := store.Open(dbPath) + if err != nil { + t.Fatal(err) + } + st.Close() + staleTheProjectSchema(t, dbPath) + holdTheProject(t, dbPath) + + if got := exitCode(migrateCmd(bookPath, io.Discard)); got != exitProjectLocked { + t.Fatalf("exit %d, want %d", got, exitProjectLocked) + } + if got := backupsOf(t, dbPath); len(got) != 0 { + t.Fatalf("a refused migration wrote %v; nothing may be written for a project it does not own", got) + } + // The retry the refusal invites reaches the lock too, instead of dying on the leftovers of the + // previous attempt. + if got := exitCode(migrateCmd(bookPath, io.Discard)); got != exitProjectLocked { + t.Fatalf("retry exited %d, want %d again", got, exitProjectLocked) + } + }) + + t.Run("a project newer than this binary", func(t *testing.T) { + // The direction a migration cannot repair. It must not read as success, or a caller looping + // "status refused → migrate → retry" would loop forever on a rolled-back binary. + bookPath := setupCLIProject(t, "http://127.0.0.1:1") + dbPath := projectDBOf(bookPath) + st, err := store.Open(dbPath) + if err != nil { + t.Fatal(err) + } + st.Close() + db, err := sql.Open("sqlite", "file:"+dbPath) + if err != nil { + t.Fatal(err) + } + if _, err := db.Exec(`INSERT INTO schema_version (version) VALUES (?)`, store.SchemaHead()+7); err != nil { + t.Fatal(err) + } + db.Close() + err = migrateCmd(bookPath, io.Discard) + if got := exitCode(err); got != exitSchemaMismatch { + t.Fatalf("exit %d, want %d", got, exitSchemaMismatch) + } + if !strings.Contains(err.Error(), fmt.Sprintf("schema_mismatch found=%d expected=%d", store.SchemaHead()+7, store.SchemaHead())) { + t.Fatalf("the refusal must name both versions machine-readably: %v", err) + } + if got := backupsOf(t, dbPath); len(got) != 0 { + t.Fatalf("a refused migration wrote %v", got) + } + }) + + t.Run("a book that has never been run", func(t *testing.T) { + // The deploy sweep does not have to know which books have a database yet. Nothing is backed up: + // a database this call creates has no history to lose. + bookPath := setupCLIProject(t, "http://127.0.0.1:1") + var out bytes.Buffer + if err := migrateCmd(bookPath, &out); err != nil { + t.Fatal(err) + } + if !strings.Contains(out.String(), "created") { + t.Fatalf("migrate must say it created the project; got %q", out.String()) + } + if v := schemaVersionOf(t, projectDBOf(bookPath)); v != store.SchemaHead() { + t.Fatalf("created at v%d, want v%d", v, store.SchemaHead()) + } + if got := backupsOf(t, projectDBOf(bookPath)); len(got) != 0 { + t.Fatalf("a project being created was backed up: %v", got) + } + }) +} diff --git a/backend/internal/pipeline/refusal.go b/backend/internal/pipeline/refusal.go index 16e7da62..2d76374c 100644 --- a/backend/internal/pipeline/refusal.go +++ b/backend/internal/pipeline/refusal.go @@ -1,5 +1,11 @@ package pipeline +import ( + "errors" + + "textmachine/backend/internal/store" +) + // refusal.go: the shell contract's REFUSAL classes (row 165 / PD-196 of the platform). // // The defect this closes was not cosmetic. `tmctl` mapped every failure it did not recognise onto exit @@ -27,6 +33,17 @@ const ( // RefusalProjectLocked is another tmctl process owning the project. Nothing is wrong with anything — // the answer is to come back later. RefusalProjectLocked RefusalClass = "project_locked" + // RefusalSchemaMismatch is a project database whose schema version is not this binary's. Nothing is + // wrong with the book either: the answer is `tmctl migrate` (or, in the other direction, a newer + // binary), and the two numbers a caller needs to tell those apart ride the message as the stable + // token `schema_mismatch found= expected=` (store.SchemaMismatchError). + // + // It is its own class because read-only opens never migrate, so an engine upgrade refuses every + // existing book until a write command touches it — and the platform's own pre-spawn call is one of + // those read-only opens, so that write command never came (the deploy deadlock, backlog row 174). + // A caller that can SEE this class self-heals ("caught it → migrate → retry") where it used to stop + // the world over what looked like a broken project. + RefusalSchemaMismatch RefusalClass = "schema_mismatch" ) // Refusal is an invocation the engine turned down before doing any work of its own: nothing reached a @@ -57,6 +74,27 @@ func RefuseConfig(err error) error { return refuse(RefusalBadConfig, err) } +// RefuseStoreOpen classifies a store-open failure. It is exported and it is the ONE place the mapping +// lives, because two callers open a project store: openRunner below, and `tmctl migrate`, which opens +// one with no runner at all. A second copy of this switch is how "another process holds it" and "this +// binary does not match the schema" drift back into a single exit 1 on one of the two paths. +// +// An unrecognised failure is returned UNCHANGED rather than swept into the band: the band means "turned +// down before doing any work", and a caller destroys or waits on data because of it. +func RefuseStoreOpen(err error) error { + var mismatch *store.SchemaMismatchError + switch { + case err == nil: + return nil + case errors.Is(err, store.ErrLocked): + return refuse(RefusalProjectLocked, err) + case errors.As(err, &mismatch): + return refuse(RefusalSchemaMismatch, err) + default: + return err + } +} + // refuseSource classifies a failure to obtain the book's text, and it classifies almost all of them as a // CONFIG fault. That is deliberate, and it is the most consequential decision in this file. // diff --git a/backend/internal/pipeline/refusal_test.go b/backend/internal/pipeline/refusal_test.go new file mode 100644 index 00000000..3cd29cd2 --- /dev/null +++ b/backend/internal/pipeline/refusal_test.go @@ -0,0 +1,58 @@ +package pipeline + +import ( + "errors" + "fmt" + "testing" + + "textmachine/backend/internal/store" +) + +// refusal_test.go pins RefuseStoreOpen, the one place a store-open failure becomes a shell class. Two +// callers share it (openRunner and `tmctl migrate`), and its arms decide what an automated caller DOES: +// wait, migrate, or treat the project as broken. + +func TestRefuseStoreOpenClassifiesWhatACallerActsOn(t *testing.T) { + locked := fmt.Errorf("opening the project: %w", store.ErrLocked) + mismatch := fmt.Errorf("opening the project: %w", &store.SchemaMismatchError{Path: "/b.db", Found: 14, Expected: 15}) + + for _, tc := range []struct { + name string + err error + want RefusalClass + }{ + {"another process holds it", locked, RefusalProjectLocked}, + {"the schema is not this binary's", mismatch, RefusalSchemaMismatch}, + } { + t.Run(tc.name, func(t *testing.T) { + var refusal *Refusal + if !errors.As(RefuseStoreOpen(tc.err), &refusal) { + t.Fatalf("%v was not classified as a refusal", tc.err) + } + if refusal.Class != tc.want { + t.Fatalf("class %q, want %q", refusal.Class, tc.want) + } + // Both arms must survive a %w-wrap, because that is how they arrive: neither caller opens the + // store at the top of its own stack. + if !errors.Is(refusal, store.ErrLocked) && !errors.As(refusal, new(*store.SchemaMismatchError)) { + t.Fatal("the classification dropped the cause it was made from") + } + }) + } + + // The safety rule, and the reason the default arm is a passthrough: the refusal band means "turned + // down before doing ANY work", and the platform's intake acts on it — it waits instead of failing, + // and one number in it deletes the user's upload. A disk error during an open is not that; it must + // stay exit 1 (infra failure) rather than borrow a class that promises nothing happened. + disk := errors.New("store: open write pool: input/output error") + got := RefuseStoreOpen(disk) + if !errors.Is(got, disk) { + t.Fatalf("an unrecognised open failure must come back with its cause intact, got %v", got) + } + if errors.As(got, new(*Refusal)) { + t.Fatal("an unrecognised open failure was swept into the refusal band") + } + if RefuseStoreOpen(nil) != nil { + t.Fatal("a nil failure must stay nil: the classifier never invents one") + } +} diff --git a/backend/internal/pipeline/runner.go b/backend/internal/pipeline/runner.go index fc0bf246..d145048e 100644 --- a/backend/internal/pipeline/runner.go +++ b/backend/internal/pipeline/runner.go @@ -262,10 +262,7 @@ func openRunner(bookPath string, logger *slog.Logger, forWrite bool) (*Runner, e } } if err != nil { - if errors.Is(err, store.ErrLocked) { - return nil, refuse(RefusalProjectLocked, err) - } - return nil, err + return nil, RefuseStoreOpen(err) } r := &Runner{ Book: book, diff --git a/backend/internal/store/migrate.go b/backend/internal/store/migrate.go index d8f6d8d6..746fd767 100644 --- a/backend/internal/store/migrate.go +++ b/backend/internal/store/migrate.go @@ -2,6 +2,7 @@ package store import ( "context" + "database/sql" "fmt" ) @@ -463,32 +464,169 @@ var migrations = []string{ // which is membank.SpoilerLeaks over the selection's Rejected set, no schema at all. Scope amended // by D39.55 (form, not intent). -// migrate runs all pending migrations on the write pool, one transaction per -// step, recording each in schema_version. -func (s *Store) migrate(ctx context.Context) error { - if _, err := s.w.ExecContext(ctx, `CREATE TABLE IF NOT EXISTS schema_version (version INTEGER PRIMARY KEY)`); err != nil { - return fmt.Errorf("store: schema_version: %w", err) +// SchemaHead is the schema version this binary migrates a project database to. It is the version every +// read-only open demands (OpenReadOnly) and the version `tmctl migrate` brings a project to. +func SchemaHead() int { return len(migrations) } + +// SchemaMismatchError is a project database whose recorded schema version is not this binary's. It is a +// TYPE rather than a message because it is the one open failure a caller can repair by itself, and it +// could not act on the difference while every failure arrived as the same text and the same exit 1. +// +// The engine's read-only opens never migrate — only a write open does — so upgrading the binary leaves +// every existing book unreadable until something writes to it. The platform calls `tmctl status --json` +// before each spawn and for the money, that call refuses the older schema, and the write command that +// would have migrated the project never comes: the deploy deadlock of backlog row 174. +// +// The numbers therefore travel BOTH in the type (for a Go caller) and in the message (for the shell), +// where they carry the stable token +// +// schema_mismatch found= expected= +// +// Found < Expected is the direction `tmctl migrate` repairs — "caught it, migrated, retried" instead of +// stopping the world. Found > Expected is a binary OLDER than the project, which no command here can fix +// (the operator upgrades tmctl), so a caller that blindly re-ran the migration would loop forever. +type SchemaMismatchError struct { + Path string + Found int + Expected int +} + +func (e *SchemaMismatchError) Error() string { + repair := "the database is NEWER than this tmctl — update the binary (an old build must neither read nor write a newer schema)" + if e.Stale() { + repair = "the database is older than this tmctl — run `tmctl migrate --config ` (read-only opens never migrate)" + } + return fmt.Sprintf("store: schema_mismatch found=%d expected=%d path=%s — %s", e.Found, e.Expected, e.Path, repair) +} + +// Stale reports whether the database is BEHIND the binary — the one direction a migration repairs. +func (e *SchemaMismatchError) Stale() bool { return e.Found < e.Expected } + +// Migration is the schema transition a write open performed: From is the version the project database +// was recorded at, To the version it is at now. From == To is a database that was already at head. +type Migration struct { + From int + To int +} + +// Migrate applies a project database's pending migrations and reports the transition — the WRITE open +// (exclusive flock · migration chain · stale-reservation recovery) with no run behind it, which is all +// `tmctl migrate` is (backlog row 174). It sends nothing, spends nothing and holds the flock only for +// as long as the migration takes. +// +// A database already at head is a no-op; one that does not exist is CREATED at head, exactly as the +// first touch of a project by any other write command creates it. A database a NEWER binary wrote is +// refused (SchemaMismatchError) before anything of this process reaches the file. +// +// beforeApply, when non-nil, runs INSIDE the project's exclusive lock, after the pending set is known +// and before the first step is applied — and only when there IS a step to apply. That is where a caller +// takes its restore point, and the placement is the whole point of the parameter: a backup taken before +// the lock is a full copy written for a migration that a live run may then refuse (litter on every +// retry of an exit-12), and one taken after the steps is not a restore point at all. An error from it +// aborts the migration with nothing applied. +func Migrate(path string, beforeApply func(Migration) error) (Migration, error) { + s, err := open(path, beforeApply) + if err != nil { + return Migration{}, err + } + applied := s.applied + if err := s.Close(); err != nil { + return applied, fmt.Errorf("store: close %s after migrating: %w", path, err) + } + return applied, nil +} + +// schemaVersion reads the recorded schema version through db. +// +// A database that exists but has no schema_version TABLE — a fresh file, or a process killed between +// creating one and committing the first migration — is version 0, which is what it is: nothing has been +// applied. That is asked as a separate question rather than read off a failed SELECT, so a file that is +// not a database at all still fails loudly instead of reading as an empty project. +func schemaVersion(ctx context.Context, db *sql.DB) (int, error) { + var recorded int + if err := db.QueryRowContext(ctx, + `SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'schema_version'`).Scan(&recorded); err != nil { + return 0, err + } + if recorded == 0 { + return 0, nil } var current int - if err := s.w.QueryRowContext(ctx, `SELECT COALESCE(MAX(version), 0) FROM schema_version`).Scan(¤t); err != nil { - return fmt.Errorf("store: read version: %w", err) + if err := db.QueryRowContext(ctx, `SELECT COALESCE(MAX(version), 0) FROM schema_version`).Scan(¤t); err != nil { + return 0, err + } + return current, nil +} + +// migrate runs all pending migrations on the write pool, one transaction per +// step, recording each in schema_version. +// +// Every phase gets its own opTimeout budget, and the caller's seam gets NONE. That is not tidiness: the +// seam exists to take a restore point, whose cost scales with the book, and while it shared one deadline +// with the steps a large project made the migration fail on the copy it had just paid for (see open). +func (s *Store) migrate(path string, beforeApply func(Migration) error) error { + current, err := s.schemaBaseline() + if err != nil { + return err + } + // A project a NEWER binary wrote is REFUSED rather than opened. The loop below would silently do + // nothing with it — there is no step left to apply — and the caller would then read and, worse, + // WRITE a schema through code that predates it. OpenReadOnly has always refused that direction + // ("reading a newer schema with old code is unsafe"); the write path used to accept it, which is + // the more dangerous half of the same fact. It is refused here, BEFORE recoverReservations, so a + // refused open leaves the file exactly as it found it. + if current > len(migrations) { + return &SchemaMismatchError{Path: path, Found: current, Expected: len(migrations)} + } + // The caller's seam (Migrate): the lock is held, the pending set is known, nothing has been applied, + // and no deadline of this package is running — the work is the caller's and so is its bounding. + if beforeApply != nil && current < len(migrations) { + if err := beforeApply(Migration{From: current, To: len(migrations)}); err != nil { + return err + } } for v := current; v < len(migrations); v++ { - tx, err := s.w.BeginTx(ctx, nil) - if err != nil { - return fmt.Errorf("store: begin migration %d: %w", v+1, err) - } - if _, err := tx.ExecContext(ctx, migrations[v]); err != nil { - _ = tx.Rollback() - return fmt.Errorf("store: apply migration %d: %w", v+1, err) - } - if _, err := tx.ExecContext(ctx, `INSERT INTO schema_version (version) VALUES (?)`, v+1); err != nil { - _ = tx.Rollback() - return fmt.Errorf("store: record migration %d: %w", v+1, err) - } - if err := tx.Commit(); err != nil { - return fmt.Errorf("store: commit migration %d: %w", v+1, err) + if err := s.applyStep(v); err != nil { + return err } } + s.applied = Migration{From: current, To: len(migrations)} + return nil +} + +// schemaBaseline makes sure the version table exists and reports what is recorded in it. +func (s *Store) schemaBaseline() (int, error) { + ctx, cancel := opContext() + defer cancel() + if _, err := s.w.ExecContext(ctx, `CREATE TABLE IF NOT EXISTS schema_version (version INTEGER PRIMARY KEY)`); err != nil { + return 0, fmt.Errorf("store: schema_version: %w", err) + } + current, err := schemaVersion(ctx, s.w) + if err != nil { + return 0, fmt.Errorf("store: read version: %w", err) + } + return current, nil +} + +// applyStep applies migrations[v] and records v+1 in ONE transaction (DDL is transactional in SQLite), +// so a process that dies mid-step leaves the database at its previous version rather than half-applied. +func (s *Store) applyStep(v int) error { + ctx, cancel := opContext() + defer cancel() + tx, err := s.w.BeginTx(ctx, nil) + if err != nil { + return fmt.Errorf("store: begin migration %d: %w", v+1, err) + } + if _, err := tx.ExecContext(ctx, migrations[v]); err != nil { + _ = tx.Rollback() + return fmt.Errorf("store: apply migration %d: %w", v+1, err) + } + if _, err := tx.ExecContext(ctx, `INSERT INTO schema_version (version) VALUES (?)`, v+1); err != nil { + _ = tx.Rollback() + return fmt.Errorf("store: record migration %d: %w", v+1, err) + } + if err := tx.Commit(); err != nil { + return fmt.Errorf("store: commit migration %d: %w", v+1, err) + } return nil } diff --git a/backend/internal/store/migrate_test.go b/backend/internal/store/migrate_test.go new file mode 100644 index 00000000..03a4bb3d --- /dev/null +++ b/backend/internal/store/migrate_test.go @@ -0,0 +1,450 @@ +package store + +import ( + "database/sql" + "errors" + "os" + "path/filepath" + "testing" + "time" +) + +// migrate_test.go pins the mechanics `tmctl migrate` stands on (backlog row 174): a database an OLDER +// binary wrote converges to head and unlocks the read-only path, the MONEY it holds survives the +// conversion untouched, a database a NEWER binary wrote is refused before anything is written, and a +// step that fails leaves nothing of itself behind. + +// atVersion runs fn with the migration chain TRUNCATED to v steps, so anything opened inside it sees +// exactly the database a binary of that vintage would have written and worked with. +// +// The fixture truncates the chain instead of deleting rows from schema_version because those are not the +// same thing: rolling the counter back leaves HEAD's tables in place and makes the next open re-apply +// the newest step, which backlog row 49а says is not idempotent for an ALTER. Truncation also keeps the +// fixture correct when a new migration lands — it never names a version or a table. +// +// It mutates a package-level var, which is safe because these tests do not run in parallel (nothing in +// this package calls t.Parallel). +func atVersion(t *testing.T, v int, fn func()) { + t.Helper() + full := migrations + migrations = migrations[:v] + defer func() { migrations = full }() + fn() +} + +// openAtVersion creates a project database exactly as a binary with v migrations would have left it. +func openAtVersion(t *testing.T, path string, v int) { + t.Helper() + atVersion(t, v, func() { + s, err := Open(path) + if err != nil { + t.Fatal(err) + } + if err := s.Close(); err != nil { + t.Fatal(err) + } + }) +} + +// rawVersion reads MAX(schema_version) through its own connection, deliberately NOT through the +// product's own reader: the state a migration left behind must be asserted independently of the code +// that reports it, and this way it can also be read from a database the product is currently refusing. +func rawVersion(t *testing.T, path string) int { + t.Helper() + db, err := sql.Open("sqlite", "file:"+path) + if err != nil { + t.Fatal(err) + } + defer db.Close() + var v int + if err := db.QueryRow(`SELECT COALESCE(MAX(version), 0) FROM schema_version`).Scan(&v); err != nil { + t.Fatal(err) + } + return v +} + +func TestMigrateBringsAnOlderProjectToHeadAndUnlocksTheReadPath(t *testing.T) { + head := SchemaHead() + path := filepath.Join(t.TempDir(), "book.db") + openAtVersion(t, path, head-1) + + // The deadlock itself: the read-only projections the platform calls before every spawn refuse a + // database an older binary wrote, and they never migrate it. + _, err := OpenReadOnly(path) + var mismatch *SchemaMismatchError + if !errors.As(err, &mismatch) { + t.Fatalf("a read-only open of an older schema must return the typed refusal, got %v", err) + } + if mismatch.Found != head-1 || mismatch.Expected != head || !mismatch.Stale() { + t.Fatalf("found=%d expected=%d stale=%v, want %d/%d/true", mismatch.Found, mismatch.Expected, mismatch.Stale(), head-1, head) + } + + applied, err := Migrate(path, nil) + if err != nil { + t.Fatalf("migrate: %v", err) + } + if applied.From != head-1 || applied.To != head { + t.Fatalf("transition %+v, want v%d -> v%d", applied, head-1, head) + } + if got := rawVersion(t, path); got != head { + t.Fatalf("schema is at v%d after migrating, want v%d", got, head) + } + ro, err := OpenReadOnly(path) + if err != nil { + t.Fatalf("the read-only path must work after a migration: %v", err) + } + ro.Close() +} + +func TestMigrateIsANoOpOnAProjectAlreadyAtHead(t *testing.T) { + head := SchemaHead() + path := filepath.Join(t.TempDir(), "book.db") + openAtVersion(t, path, head) + + for i := range 2 { + applied, err := Migrate(path, nil) + if err != nil { + t.Fatalf("call %d: %v", i, err) + } + if applied.From != head || applied.To != head { + t.Fatalf("call %d: transition %+v, want a no-op at v%d", i, applied, head) + } + } +} + +func TestMigrateCreatesAProjectThatWasNeverRun(t *testing.T) { + // The deploy sweep runs over every book, including one that has been accepted but never translated. + // Creating it at head is what every other write open already does on first touch, and it keeps the + // sweep uniform instead of making the caller ask which books exist yet. + path := filepath.Join(t.TempDir(), "fresh.db") + applied, err := Migrate(path, nil) + if err != nil { + t.Fatal(err) + } + if applied.From != 0 || applied.To != SchemaHead() { + t.Fatalf("transition %+v, want v0 -> v%d", applied, SchemaHead()) + } + ro, err := OpenReadOnly(path) + if err != nil { + t.Fatalf("a created project must be readable at head: %v", err) + } + ro.Close() +} + +// TestMigrateKeepsCommittedMoneyAndZeroesOnlyTheStaleReservation is the money test. +// +// It is the one property the platform's ratified ceiling formula rests on (PD-158: the argument is +// committed + increment, WITHOUT reserved, precisely because a write open zeroes leftover reservations +// and a read-only status does not). A migration is a write open, so it performs that recovery — and it +// must move `committed` by not one cent while doing it, or every ceiling computed from a status taken +// after a deploy would be wrong. +func TestMigrateKeepsCommittedMoneyAndZeroesOnlyTheStaleReservation(t *testing.T) { + head := SchemaHead() + path := filepath.Join(t.TempDir(), "book.db") + caps := Ceilings{BookUSD: 10, DayUSD: 10} + + // A book an older binary ran: one settled call, and one reservation whose process died before it + // settled — exactly the state a crashed or killed run leaves behind. + atVersion(t, head-1, func() { + s, err := Open(path) + if err != nil { + t.Fatal(err) + } + defer s.Close() + job := mustSnapshotAndJob(t, s) + res, verdict, err := s.Reserve("book", 0.10, caps) + if err != nil || verdict != ReserveOK { + t.Fatalf("reserve: %v %v", verdict, err) + } + cp := Checkpoint{RequestHash: "h1", JobID: job.ID, Stage: "draft", Role: "translator", + ModelRequested: "m", ModelActual: "m", ResponseText: "текст", UsageJSON: "{}", CostUSD: 0.04} + if err := s.SettleWithCheckpoint(res, 0.04, cp, nil); err != nil { + t.Fatal(err) + } + if _, verdict, err = s.Reserve("book", 0.25, caps); err != nil || verdict != ReserveOK { + t.Fatalf("stale reserve: %v %v", verdict, err) + } + committed, reserved, err := s.SpentUSD("book") + if err != nil || committed != 0.04 || reserved != 0.25 { + t.Fatalf("fixture: committed=%v reserved=%v err=%v", committed, reserved, err) + } + }) + + if _, err := Migrate(path, nil); err != nil { + t.Fatalf("migrate: %v", err) + } + + // Twice: the second call is the idempotency half — a deploy sweep that runs over a book it already + // migrated must not move money either. + for i := range 2 { + if i == 1 { + if _, err := Migrate(path, nil); err != nil { + t.Fatalf("repeat migrate: %v", err) + } + } + ro, err := OpenReadOnly(path) + if err != nil { + t.Fatalf("call %d: %v", i, err) + } + committed, reserved, err := ro.SpentUSD("book") + ro.Close() + if err != nil { + t.Fatal(err) + } + if committed != 0.04 { + t.Fatalf("call %d: committed moved to %v — a migration must not touch settled money", i, committed) + } + if reserved != 0 { + t.Fatalf("call %d: reserved=%v, want 0 — the write open owes the recovery pass", i, reserved) + } + } + if got := rawVersion(t, path); got != head { + t.Fatalf("schema v%d after migrating, want v%d", got, head) + } + // The checkpoint the money is anchored to is still there: `committed == SUM(checkpoints)` is the + // invariant the platform settles from, so a migration that kept the number and lost the row would + // pass the assertion above and still have broken the book. + ro, err := OpenReadOnly(path) + if err != nil { + t.Fatal(err) + } + defer ro.Close() + cp, err := ro.GetCheckpoint("h1") + if err != nil || cp == nil || cp.CostUSD != 0.04 { + t.Fatalf("checkpoint after migration: %+v err=%v", cp, err) + } +} + +// TestOpenRefusesAProjectNewerThanTheBinary: the write path used to accept a database it could not +// understand and would then have written through pre-dating code. It is refused in both open modes, and +// the refusal happens BEFORE the recovery pass — a live reservation still standing afterwards is the +// proof that nothing of the refused process reached the file. +func TestOpenRefusesAProjectNewerThanTheBinary(t *testing.T) { + head := SchemaHead() + path := filepath.Join(t.TempDir(), "book.db") + s, err := Open(path) + if err != nil { + t.Fatal(err) + } + if _, verdict, err := s.Reserve("book", 0.25, Ceilings{BookUSD: 10, DayUSD: 10}); err != nil || verdict != ReserveOK { + t.Fatalf("reserve: %v %v", verdict, err) + } + s.Close() + + db, err := sql.Open("sqlite", "file:"+path) + if err != nil { + t.Fatal(err) + } + if _, err := db.Exec(`INSERT INTO schema_version (version) VALUES (?)`, head+7); err != nil { + t.Fatal(err) + } + db.Close() + + for name, open := range map[string]func(string) (*Store, error){"write": Open, "read-only": OpenReadOnly} { + st, err := open(path) + if err == nil { + st.Close() + t.Fatalf("%s open of a newer schema must be refused", name) + } + var mismatch *SchemaMismatchError + if !errors.As(err, &mismatch) { + t.Fatalf("%s open: %v, want a typed SchemaMismatchError", name, err) + } + if mismatch.Found != head+7 || mismatch.Expected != head || mismatch.Stale() { + t.Fatalf("%s open: found=%d expected=%d stale=%v", name, mismatch.Found, mismatch.Expected, mismatch.Stale()) + } + } + if _, err := Migrate(path, nil); !errors.As(err, new(*SchemaMismatchError)) { + t.Fatalf("migrate of a newer schema must be refused with the typed error, got %v", err) + } + + // Nothing was written: the reservation the fixture left standing is still standing, so the recovery + // pass never ran on a schema this binary does not know. + ro, err := sql.Open("sqlite", "file:"+path) + if err != nil { + t.Fatal(err) + } + defer ro.Close() + var reserved float64 + if err := ro.QueryRow(`SELECT COALESCE(SUM(reserved_usd), 0) FROM spend WHERE book_id = 'book'`).Scan(&reserved); err != nil { + t.Fatal(err) + } + if reserved != 0.25 { + t.Fatalf("reserved=%v after three refused opens, want the fixture's 0.25 untouched", reserved) + } +} + +// TestAFailedMigrationStepLeavesTheDatabaseAtItsPreviousVersion answers, by execution, the question +// `tmctl migrate` raises: is APPLYING a step atomic? It is — DDL is transactional in SQLite and the +// version row is inserted in the SAME transaction as the step — so a process that dies mid-migration +// leaves a database at its previous version rather than a half-applied one whose ALTER steps (row 49а) +// will not converge on the next try. The migration bodies themselves are untouched by this pack. +func TestAFailedMigrationStepLeavesTheDatabaseAtItsPreviousVersion(t *testing.T) { + head := SchemaHead() + path := filepath.Join(t.TempDir(), "book.db") + + // A step whose FIRST statement succeeds and whose second does not: if application were not atomic, + // `probe` would outlive the failure. Restored with defer, so a panic inside Open cannot leave the + // package-level chain rewritten for whatever runs next. + full := migrations + defer func() { migrations = full }() + migrations = append(append([]string(nil), migrations...), `CREATE TABLE probe (x INTEGER); THIS IS NOT SQL;`) + s, err := Open(path) + if err == nil { + s.Close() + t.Fatal("a migration step that does not execute must fail the open") + } + + if got := rawVersion(t, path); got != head { + t.Fatalf("schema recorded v%d after a failed step, want the previous v%d", got, head) + } + db, err := sql.Open("sqlite", "file:"+path) + if err != nil { + t.Fatal(err) + } + defer db.Close() + var probes int + if err := db.QueryRow(`SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'probe'`).Scan(&probes); err != nil { + t.Fatal(err) + } + if probes != 0 { + t.Fatal("the failed step's first statement survived: applying a migration is not atomic") + } +} + +// TestADatabaseWithNothingAppliedIsAVersionAndNotAnError: a file that exists but has no +// schema_version table — a process killed between creating one and committing the first migration — +// used to fail the read path with prose and exit 1, i.e. as an unrepairable project. It is version 0: +// the SAME typed refusal as any other mismatch, so the same "caught it → migrate → retry" repairs it. +func TestADatabaseWithNothingAppliedIsAVersionAndNotAnError(t *testing.T) { + dir := t.TempDir() + empty := filepath.Join(dir, "empty.db") + if err := os.WriteFile(empty, nil, 0o644); err != nil { + t.Fatal(err) + } + _, err := OpenReadOnly(empty) + var mismatch *SchemaMismatchError + if !errors.As(err, &mismatch) { + t.Fatalf("a database with nothing applied must be the typed refusal, got %v", err) + } + if mismatch.Found != 0 || mismatch.Expected != SchemaHead() || !mismatch.Stale() { + t.Fatalf("found=%d expected=%d stale=%v, want 0/%d/true", mismatch.Found, mismatch.Expected, mismatch.Stale(), SchemaHead()) + } + if _, err := Migrate(empty, nil); err != nil { + t.Fatalf("and a migration must repair it: %v", err) + } + ro, err := OpenReadOnly(empty) + if err != nil { + t.Fatalf("read path after the repair: %v", err) + } + ro.Close() + + // A file that is not a database at all must NOT read as "version 0, migrate me". + junk := filepath.Join(dir, "junk.db") + if err := os.WriteFile(junk, []byte("this is not a database, it is a text file"), 0o644); err != nil { + t.Fatal(err) + } + if _, err := OpenReadOnly(junk); err == nil || errors.As(err, &mismatch) { + t.Fatalf("a corrupt file must fail loudly and NOT as a schema mismatch, got %v", err) + } +} + +// TestTheMigrationSeamRunsUnderTheLockAndOnlyWhenAStepWillRun pins the placement `tmctl migrate` takes +// its restore point in. The three properties are what make a backup a backup: it runs BEFORE the first +// step (a copy taken afterwards is not a restore point), only when there IS a step (else a repeat call +// litters, and its second-precision stamp collides), and inside the lock (else a project a live run +// owns is copied in full for a migration that is refused anyway — once per retry). +func TestTheMigrationSeamRunsUnderTheLockAndOnlyWhenAStepWillRun(t *testing.T) { + head := SchemaHead() + path := filepath.Join(t.TempDir(), "book.db") + openAtVersion(t, path, head-1) + + var seen []Migration + versionAtHook := -1 + applied, err := Migrate(path, func(m Migration) error { + seen = append(seen, m) + versionAtHook = rawVersion(t, path) + // The lock is ours while the hook runs: a second opener must be turned away, which is what + // makes "back up here" different from "back up before calling". + if s, err := Open(path); !errors.Is(err, ErrLocked) { + if err == nil { + s.Close() + } + t.Errorf("the seam must run INSIDE the project lock; a concurrent open got %v", err) + } + return nil + }) + if err != nil { + t.Fatal(err) + } + if len(seen) != 1 || seen[0] != (Migration{From: head - 1, To: head}) { + t.Fatalf("the seam saw %+v, want exactly one v%d -> v%d", seen, head-1, head) + } + if versionAtHook != head-1 { + t.Fatalf("the database was at v%d when the seam ran, want the PRE-migration v%d", versionAtHook, head-1) + } + if applied.From != head-1 || applied.To != head { + t.Fatalf("transition %+v", applied) + } + + // Nothing left to apply → the seam does not run at all. + seen = nil + if _, err := Migrate(path, func(m Migration) error { seen = append(seen, m); return nil }); err != nil { + t.Fatal(err) + } + if len(seen) != 0 { + t.Fatalf("the seam ran %d time(s) on a project already at head", len(seen)) + } +} + +// TestTheSeamIsNotChargedToTheStoreOperationBudget: the seam's work is the CALLER's (a restore point +// whose cost scales with the book), and opTimeout bounds operations that are O(1) in the size of the +// book. While the two shared one deadline, a large project failed the migration on the copy it had just +// paid for — «context deadline exceeded» on the first step, exit 1 outside the refusal band, and a +// retry that copied the whole database again (round-2 review, reproduced on 2.6 GB). +// +// The budget is shrunk instead of the seam sleeping ten real seconds: same property, no dead time in +// the battery. It is restored on the way out and nothing here runs in parallel. +func TestTheSeamIsNotChargedToTheStoreOperationBudget(t *testing.T) { + head := SchemaHead() + defer func(original time.Duration) { opTimeout = original }(opTimeout) + opTimeout = 150 * time.Millisecond + + path := filepath.Join(t.TempDir(), "book.db") + openAtVersion(t, path, head-1) + + applied, err := Migrate(path, func(Migration) error { + time.Sleep(3 * opTimeout) // a restore point on a book far larger than the budget + return nil + }) + if err != nil { + t.Fatalf("a slow seam must not fail the migration it protects: %v", err) + } + if applied.From != head-1 || applied.To != head { + t.Fatalf("transition %+v, want v%d -> v%d", applied, head-1, head) + } + if got := rawVersion(t, path); got != head { + t.Fatalf("schema at v%d, want v%d", got, head) + } +} + +// TestASeamFailureAbortsTheMigrationWithNothingApplied: a caller that cannot take its restore point is +// telling the engine not to migrate — the answer is to stop with the database exactly as it was, not to +// proceed unprotected. +func TestASeamFailureAbortsTheMigrationWithNothingApplied(t *testing.T) { + head := SchemaHead() + path := filepath.Join(t.TempDir(), "book.db") + openAtVersion(t, path, head-1) + + refuse := errors.New("no room for a restore point") + if _, err := Migrate(path, func(Migration) error { return refuse }); !errors.Is(err, refuse) { + t.Fatalf("the seam's error must abort the migration, got %v", err) + } + if got := rawVersion(t, path); got != head-1 { + t.Fatalf("schema moved to v%d after a refused seam, want v%d", got, head-1) + } + // And the lock is released, so the next attempt is not locked out by the failed one. + if _, err := Migrate(path, nil); err != nil { + t.Fatalf("a retry after a refused seam: %v", err) + } +} diff --git a/backend/internal/store/store.go b/backend/internal/store/store.go index 2357c52b..60fa6f51 100644 --- a/backend/internal/store/store.go +++ b/backend/internal/store/store.go @@ -29,13 +29,24 @@ import ( // opTimeout bounds every store operation: SQLite is a local file and // effectively never blocks; the cap keeps a wedged filesystem from hanging a // pipeline goroutine forever. -const opTimeout = 10 * time.Second +// +// It is a PER-OPERATION budget and only holds while every operation under it is O(1) in the size of the +// book — which is what the migration chain and the recovery pass are (metadata-only DDL and one UPDATE). +// A caller's own work must therefore never be charged to it; see the seam in migrate. +// +// A var, not a const, so a test can shrink it: the property "the seam is not charged to this budget" is +// otherwise only testable by sleeping ten real seconds in the battery. +var opTimeout = 10 * time.Second // Store is one project database. type Store struct { w *sql.DB // single-connection write pool; all mutations go through it r *sql.DB // read pool lock *os.File + // applied is the schema transition this open performed — the zero value on a read-only open, which + // never migrates. It is recorded because a write open is the only thing that MAY migrate, so it is + // also the only thing that knows what it migrated (`tmctl migrate` reports it, row 174). + applied Migration } // Open opens (or creates) the project database at path, applies pending @@ -45,7 +56,11 @@ type Store struct { // owns the project file" is enforced here, not assumed. Without the lock a // concurrent `tmctl report` during a translation would zero a running process's live // reservations via the recovery pass and blind its ceilings (review finding). -func Open(path string) (*Store, error) { +func Open(path string) (*Store, error) { return open(path, nil) } + +// open is Open with the migration seam exposed: beforeApply, when non-nil, runs after the lock is held +// and the pending set is known, and before the first step is applied. See Migrate. +func open(path string, beforeApply func(Migration) error) (*Store, error) { lock, err := acquireLock(path + ".lock") if err != nil { return nil, err @@ -80,12 +95,18 @@ func Open(path string) (*Store, error) { r.SetMaxOpenConns(4) s := &Store{w: w, r: r, lock: lock} - ctx, cancel := context.WithTimeout(context.Background(), opTimeout) - defer cancel() - if err := s.migrate(ctx); err != nil { + // Each phase takes its OWN budget rather than sharing one across the whole open. The migration is + // the only phase that can host caller work of unbounded size (the seam below), and a single shared + // deadline made that work eat the budget of the steps it was called to protect: on a large project + // the restore point finished and the first step then failed «context deadline exceeded» — an exit 1 + // outside the refusal band, after an expensive write, with every retry copying the database again + // (round-2 review, reproduced on a 2.6 GB project). + if err := s.migrate(path, beforeApply); err != nil { s.Close() return nil, err } + ctx, cancel := opContext() + defer cancel() if err := s.recoverReservations(ctx); err != nil { s.Close() return nil, err @@ -104,9 +125,9 @@ func Open(path string) (*Store, error) { // // Guarantees: connections are strictly read-only (PRAGMA query_only=1 — an accidental // write fails loudly rather than corrupting a live writer's state); a missing DB is a loud -// error (we don't create an empty one); a schema older than the binary is a loud error "run a -// write command" (only the writer applies migrations). A known, honest boundary: -// after a run crashes, reserved_usd stays non-zero until the next WRITE command +// error (we don't create an empty one); a schema this binary does not match is a TYPED refusal naming +// `tmctl migrate` (only a write open applies migrations — see SchemaMismatchError). A known, honest +// boundary: after a run crashes, reserved_usd stays non-zero until the next WRITE command // (the recovery pass runs only there) — status will show this leftover as reserved. func OpenReadOnly(path string) (*Store, error) { if _, err := os.Stat(path); err != nil { @@ -128,20 +149,18 @@ func OpenReadOnly(path string) (*Store, error) { s := &Store{w: r, r: r, lock: nil} ctx, cancel := context.WithTimeout(context.Background(), opTimeout) defer cancel() - var current int - if err := s.r.QueryRowContext(ctx, `SELECT COALESCE(MAX(version), 0) FROM schema_version`).Scan(¤t); err != nil { + current, err := schemaVersion(ctx, s.r) + if err != nil { s.Close() - return nil, fmt.Errorf("store: read schema version of %s (empty/corrupt database?): %w", path, err) + return nil, fmt.Errorf("store: read schema version of %s (not a database?): %w", path, err) } - if current < len(migrations) { + // A version this binary does not match is a TYPED refusal, in both directions: the caller's repair + // differs ("migrate this project" versus "upgrade this binary"), and while both arrived as prose and + // exit 1 an automated caller could act on neither — which is what turned a binary upgrade into a + // deadlock across every existing book (row 174, SchemaMismatchError). + if current != len(migrations) { s.Close() - return nil, fmt.Errorf("store: %s is at schema v%d, this binary expects v%d — run a write command (`tmctl translate`/`redrive`) to migrate first (read-only open never migrates)", - path, current, len(migrations)) - } - if current > len(migrations) { - s.Close() - return nil, fmt.Errorf("store: %s is at schema v%d, NEWER than this binary's v%d — update tmctl (reading a newer schema with old code is unsafe)", - path, current, len(migrations)) + return nil, &SchemaMismatchError{Path: path, Found: current, Expected: len(migrations)} } return s, nil }