Land platform P5 after three acceptance rounds: streaming book intake with storage sentinel, stop and resume with recorded intent, Prometheus observability, enforced Go 1.26.6 floor
This commit is contained in:
parent
d1f42c5470
commit
69d485a16f
48 changed files with 8090 additions and 364 deletions
|
|
@ -18,6 +18,8 @@
|
|||
| П-7 | **Кредитный баланс и фри-тир** (решения владельца 05.08; ПРОДАЖИ НЕТ — бета на предоплаченных ключах владельца): append-only леджер в целых микро-долларах, грант фри-тира из админки (дефолт $5, настраиваемый), холд ДО спавна прогона + пер-книжный потолок движку (жёсткий стоп исполняет движок), расчёт на границе попытки. ⚠ Окон с обнулением НЕТ — баланс, а не подписка: черновик `usage_windows` в подписочной форме заменяется. Платёжный провайдер не проектируется до решения продавать. Разбор — `docs/PLATFORM_DIRECTION.md` §2 | P1+, после П-6 | владелец 04–05.08 |
|
||||
| П-8 | **Минимальная админ-поверхность**: начислить/списать кредиты аккаунту (одна запись леджера), посмотреть баланс и состояние прогонов. Защищённая ручка либо CLI — форму предлагает сессия. Без неё фри-тир неуправляем | вместе с П-7 | владелец 05.08 |
|
||||
| П-5 | **API лимитов/использования + оповещение стопа по потолку** (решение владельца 04.08, D39.100/ПТ-35, механизм «как Claude Code»): страница лимитов в настройках читает СТАТУС использования (не суммы — деньги на провод не идут, D39.84); стоп по потолку → статус `paused` + оповещение «перевод остановлен: лимиты исчерпаны»; сырьё у движка есть (`request_log`/`ledger`, потолки конфига), форму API предлагает P0 | Ф3, вместе с П-1 | D39.100, ПТ-35, контракт К-8 |
|
||||
| П-9 | **Загрузка книги `POST /books` (multipart)** — единственная ручка контракта 0.2.0, которую пак раннера НЕ взял. Причина названа, а не умолчана: это не «ещё один хендлер», а хранилище (куда лёг файл, кто его чистит при отказе), разбор (статусы `uploading`/`parsing`/`rejected` существуют в схеме и ни одним писателем не заполняются), каталог проекта движка (кто пишет `book.yaml` — платформа его НЕ правит, D39.110) и пер-маршрутный потолок тела вместе с тестом, которого ждёт PD-72. Пока её нет, книги заводятся дев-инструментом `tmplatformctl book add`, и библиотека читает их так же, как читала бы загруженные: read-model один. ⚠ PD-72 закрывается ВМЕСТЕ с этой ручкой, не раньше | Ф3, следующий пак | сессия P4 (границей промта D39.119 §5а) |
|
||||
| П-9 | **Загрузка книги `POST /books` (multipart)** — единственная ручка контракта 0.2.0, которую пак раннера НЕ взял. Причина названа, а не умолчана: это не «ещё один хендлер», а хранилище (куда лёг файл, кто его чистит при отказе), разбор (статусы `uploading`/`parsing`/`rejected` существуют в схеме и ни одним писателем не заполняются), каталог проекта движка (кто пишет `book.yaml` — платформа его НЕ правит, D39.110) и пер-маршрутный потолок тела вместе с тестом, которого ждёт PD-72. Пока её нет, книги заводятся дев-инструментом `tmplatformctl book add`, и библиотека читает их так же, как читала бы загруженные: read-model один. ⚠ PD-72 закрывается ВМЕСТЕ с этой ручкой, не раньше — **ИСПОЛНЕНО P5 (11.08), с одной НАЗВАННОЙ половиной на ратификации:** ручка построена потоково (`r.MultipartReader`, пер-маршрутный потолок тела, свой дедлайн чтения), хранилище — каталог книги под `TM_PLATFORM_BOOKS_DIR`, статусы `uploading → parsing → not_started \| rejected` получили писателей, разбор зовёт `tmctl manifest`, PD-72 закрыт своим тестом. НЕ построено и вынесено вопросом в журнал: кто пишет стартовый `book.yaml` при интейке (D39.110 §2b) — шов `books.ErrNotProvisioned` назван, без конфигурации книга отказывает разбором | Ф3, следующий пак | сессия P4 (границей промта D39.119 §5а), исполнено P5 |
|
||||
| П-10 | **Честная оценка «$/глава» от ДВИЖКА.** Сейчас ставка — константа платформы ($0.03, провенанс exp08 v2 через D30.4, `STACK_DECISIONS` §20), и это осознанная бета-мера: движковой поверхности оценки не существует, а выдумывать её запрещено. Ставка решает только ДЛИНУ шкалы (деньги защищены холдом и потолком движка), но на книге, которая заметно дороже или дешевле средней, шкала врёт пользователю о том, сколько глав он покупает. Нужна оценка от движка по конкретной книге — запрос уходит строкой ЕДИНОГО бэклога через оркестратора, не сюда | когда-нибудь (до первого платящего) | сессия P4 |
|
||||
| П-11 | **Наблюдаемость раннера.** Метрик и трейсинга в зоне нет вовсе (грепнуто: ни prometheus, ни otel, ни expvar, ни pprof), а у раннера появились величины, которые без них не видны: глубина очереди, возраст незакрытых холдов, число прогонов в карантине, отставание тейлера, длительность свипа. Сегодня всё это читается только глазами по логам и SQL. Связано с PD-115 (у зоны нет внешнего эталона ни по одной оси, кроме безопасности) | Ф3 | сессия P4 |
|
||||
| П-11 | **Наблюдаемость раннера.** Метрик и трейсинга в зоне нет вовсе (грепнуто: ни prometheus, ни otel, ни expvar, ни pprof), а у раннера появились величины, которые без них не видны: глубина очереди, возраст незакрытых холдов, число прогонов в карантине, отставание тейлера, длительность свипа. Сегодня всё это читается только глазами по логам и SQL. Связано с PD-115 (у зоны нет внешнего эталона ни по одной оси, кроме безопасности) — **ИСПОЛНЕНО P5 (11.08):** `prometheus/client_golang` v1.24.1 на отдельном слушателе `TM_PLATFORM_METRICS_ADDR`; глубина очереди · возраст самого старого открытого холда · карантины · отставание тейлера · книги в интейке · длительность свипа и счётчик недоведённых проходов · запросы и задержки по паттерну маршрута. Ось наблюдаемости получила внешний эталон (практики именования Prometheus + золотые сигналы, `ENGINEERING_STANDARDS` §2) — половина PD-115 | Ф3 | сессия P4, исполнено P5 |
|
||||
| П-12 | **Квота интейка и ретеншен отклонённых книг.** `POST /books` даёт аутентифицированному аккаунту писать на диск оператора: один аплоад ограничен (64 МиБ), число аплоадов — ничем. Отклонённая по вине источника книга каталог теряет, отклонённая по вине деплоя — сохраняет намеренно, и не чистит их никто. Нужны лимит книг на аккаунт (сколько книг входит во фри-тир — продуктовая политика, не инженерная) и свип ретеншена. Строка регистра — PD-175 | до первого чужого пользователя | сессия P5 |
|
||||
| П-13 | **Оценка размера книги в символах — у ДВИЖКА.** `character_count` контракта платформа считает потоково на приёме (байты, не являющиеся продолжением UTF-8), что точно для UTF-8 и приблизительно для GB18030/UTF-16, которые движок принимает и декодирует сам. Манифест несёт `source_bytes` и `encoding`, но не число символов. Запрос уходит строкой ЕДИНОГО бэклога через оркестратора; строка регистра — PD-177 | когда-нибудь | сессия P5 |
|
||||
|
|
|
|||
|
|
@ -3,13 +3,24 @@
|
|||
# under you on someone else's machine is not a gate. Shape mirrors backend/Makefile deliberately.
|
||||
|
||||
GO ?= go
|
||||
# go.mod's floor is 1.26.4 (the engine's), but the BUILD toolchain floor here is 1.26.5: it carries
|
||||
# the crypto/tls and os security fixes, and this module is the one exposed to the network.
|
||||
GO_MIN_VERSION := 1.26.5
|
||||
# go.mod's floor is 1.26.4 (the engine's), but the BUILD toolchain floor here is HIGHER, and for one
|
||||
# reason: this module is the one exposed to the network, so it takes the standard library's security
|
||||
# fixes as soon as they are released.
|
||||
#
|
||||
# ⚠ Raised 1.26.5 → 1.26.6 on 13.08 by `make vuln`, not by choice: the advisory database published
|
||||
# five standard-library vulnerabilities against 1.26.5 — net/http, crypto/tls, net/url, encoding/xml,
|
||||
# encoding/asn1 (GO-2026-6218/6090/6089/6088/5972) — all fixed in 1.26.6, and govulncheck traces two
|
||||
# of them into paths this service calls. The same battery is green on 1.26.6 and the scan is clean.
|
||||
# HOW the floor is actually enforced, because the first version of this raise enforced nothing: the
|
||||
# check below COMPARES versions (its predecessor was a regex that matched the very 1.26.5 it was
|
||||
# written to refuse), and go.mod carries `toolchain go1.26.6`, which every build reads whether or not
|
||||
# it calls make — with GOTOOLCHAIN=auto such a host fetches this toolchain, with =local it stops.
|
||||
# The comparison itself is pinned in internal/gates.
|
||||
GO_MIN_VERSION := 1.26.6
|
||||
GOLANGCI_LINT ?= golangci-lint
|
||||
GOLANGCI_VERSION := 2.12.2
|
||||
|
||||
.PHONY: build vet fmt lint test check tools-check vuln fuzz
|
||||
.PHONY: build vet fmt lint test check tools-check version-check vuln fuzz
|
||||
|
||||
build: tools-check
|
||||
$(GO) build ./...
|
||||
|
|
@ -21,9 +32,26 @@ vet:
|
|||
fmt:
|
||||
@test -z "$$(gofmt -l .)" || { echo "gofmt: not formatted:"; gofmt -l .; exit 1; }
|
||||
|
||||
tools-check:
|
||||
@$(GO) version | grep -qE 'go1\.26\.([5-9]|[0-9]{2,})|go1\.(2[7-9]|[3-9][0-9])' || { \
|
||||
echo "Go $(GO_MIN_VERSION)+ required (security fixes in a network-facing module); got: $$($(GO) version)"; exit 1; }
|
||||
# GO_VERSION is what the gate JUDGES, and it is a variable so the judgement can be tested with
|
||||
# versions this host does not have: `make version-check GO_VERSION=go1.26.5` must fail.
|
||||
GO_VERSION ?= $$($(GO) env GOVERSION)
|
||||
|
||||
# version-check COMPARES rather than matches. The predecessor of this rule was a regex over
|
||||
# `go version`, and it accepted the very release it was written to refuse — 1.26.5 matched
|
||||
# `go1\.26\.([5-9]|…)` — while GO_MIN_VERSION lived only in the failure message. A comparison has no
|
||||
# such gap: `sort -V` orders 1.26.10 above 1.26.9, which a regex over digits gets wrong too, and it
|
||||
# is the same tool the release engineering of this repo already relies on.
|
||||
version-check:
|
||||
@have="$$(printf '%s' "$(GO_VERSION)" | sed 's/^go//')"; \
|
||||
case "$$have" in *[!0-9.]*) \
|
||||
echo "Go $(GO_MIN_VERSION)+ required, and a release: a prerelease or development toolchain does not carry the fixes its number promises; got: go$$have"; \
|
||||
exit 1;; esac; \
|
||||
lowest="$$(printf '%s\n%s\n' "$$have" "$(GO_MIN_VERSION)" | sort -V | head -1)"; \
|
||||
test "$$lowest" = "$(GO_MIN_VERSION)" || { \
|
||||
echo "Go $(GO_MIN_VERSION)+ required (standard-library security fixes in a network-facing module); got: go$$have"; \
|
||||
exit 1; }
|
||||
|
||||
tools-check: version-check
|
||||
@$(GOLANGCI_LINT) --version 2>/dev/null | grep -q " $(GOLANGCI_VERSION) " || { \
|
||||
echo "golangci-lint $(GOLANGCI_VERSION) required (findings are version-dependent)."; \
|
||||
echo "install: https://github.com/golangci/golangci-lint/releases/tag/v$(GOLANGCI_VERSION)"; exit 1; }
|
||||
|
|
|
|||
|
|
@ -14,6 +14,11 @@
|
|||
Бинари: `cmd/tmplatformd` (сервис) и `cmd/tmplatformctl` (админ: гранты, КОРРЕКТИРОВКИ (`adjust`), баланс, журнал входов,
|
||||
отзыв сессий, дев-интейк `book add`, и `exit-marker` — его зовёт systemd на конце прогона). Деплой — `deploy/`.
|
||||
|
||||
Метрики — отдельным слушателем (`TM_PLATFORM_METRICS_ADDR`, дефолт `127.0.0.1:9464`), формат
|
||||
Prometheus; на контрактную поверхность они не выходят и наружу не привязываются (`docs/STACK_DECISIONS.md` §24).
|
||||
Эффективная конфигурация печатается на старте с источником каждой настройки; секреты и денежные
|
||||
суммы — фактом наличия, без значения (PD-114).
|
||||
|
||||
⚠ **Прогон — транзиентный systemd-юнит, а не ребёнок сервиса** (D39.106). Установка требует
|
||||
`loginctl enable-linger`, иначе не стартует ни один прогон; почему именно так и что было
|
||||
измерено — `docs/STACK_DECISIONS.md` §15–20.
|
||||
|
|
@ -35,12 +40,21 @@
|
|||
- аутентификация и аккаунты — **есть (P1)**: вход через OIDC даёт только СОБЫТИЕ входа, сессия
|
||||
своя; ключ личности `(provider, subject)`, почта не ключ. Оплаты нет и в бете не будет
|
||||
(владелец 05.08): аккаунты живут на кредитном балансе, фри-тир — запись `grant` в леджер;
|
||||
- библиотека книг: чья книга, права доступа, хранение исходников и экспортов;
|
||||
- библиотека книг: чья книга, права доступа, хранение исходников и экспортов — **приём есть (P5)**:
|
||||
`POST /v0/books` принимает multipart потоково со своим потолком тела и своим дедлайном чтения,
|
||||
кладёт исходник в каталог книги под `TM_PLATFORM_BOOKS_DIR` и ведёт книгу по статусам
|
||||
`uploading → parsing → not_started | rejected`; разбор — $0-команда движка `tmctl manifest`.
|
||||
⚠ Кто кладёт стартовый `book.yaml` в каталог новой книги — открытый вопрос на ратификации
|
||||
(D39.110 §2b): без него разбор честно отказывает, шов назван `books.ErrNotProvisioned`;
|
||||
- учёт денег **на пользователя** — **схема и операции есть (P1)**: append-only леджер в целых
|
||||
микро-долларах, резервации, кэш баланса с инвариантом `balance == SUM(ledger)`. Защита прогона —
|
||||
холд ДО спавна плюс книжный потолок движку (жёсткий стоп исполняет движок). ⚠ Потолок, который
|
||||
получает движок, — это НАКОПЛЕННЫЙ потолок КНИГИ (`committed + reserved` за всю её историю плюс
|
||||
купленный прирост), а не бюджет прогона: платформа переводит одно в другое сама (D39.122);
|
||||
- остановка и продолжение перевода — **есть (P5)**: `POST /v0/runs/{id}/stop` пишет НАМЕРЕНИЕ стопа
|
||||
в Postgres до сигнала (иначе стоп и авария — один и тот же выход движка, PD-152) и просит systemd;
|
||||
`/resume` открывает новую попытку с ОСТАТКОМ бюджета прогона, переиспользуя механику
|
||||
перезапуска реконсилятора;
|
||||
- очередь задач и запуск воркеров, статусы прогонов, ретраи — **есть (P4)**: River на том же
|
||||
Postgres; задание очереди выдаёт только РАЗРЕШЕНИЕ стартовать, а жизнь прогона ведёт
|
||||
реконсилятор, который читает мир (Postgres · журнал книги · маркер выхода) и не ждёт процесса;
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import (
|
|||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
|
|
@ -17,6 +18,7 @@ import (
|
|||
"textmachine/platform/internal/config"
|
||||
"textmachine/platform/internal/httpapi"
|
||||
"textmachine/platform/internal/login"
|
||||
"textmachine/platform/internal/metrics"
|
||||
"textmachine/platform/internal/pgstore"
|
||||
"textmachine/platform/internal/reqid"
|
||||
)
|
||||
|
|
@ -40,10 +42,20 @@ func run(log *slog.Logger) error {
|
|||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Before anything else is decided: an environment-only configuration leaves no artifact to read
|
||||
// back, so a deployment whose EnvironmentFile= never loaded looks exactly like one whose values
|
||||
// were chosen (register row PD-114, ratified 09.08). Secrets and money amounts are named, never
|
||||
// printed.
|
||||
cfg.LogEffective(log)
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
// One registry for the process, whether or not anything scrapes it: the instruments are wired
|
||||
// into the request path and the sweep either way, and a nil registry would make the telemetry a
|
||||
// second code path that only runs in production.
|
||||
telemetry := metrics.New()
|
||||
|
||||
var db *pgstore.Store
|
||||
if cfg.DSN == "" {
|
||||
// Deliberate: the process still serves liveness so a supervisor can start it before the
|
||||
|
|
@ -72,7 +84,8 @@ func run(log *slog.Logger) error {
|
|||
Log: log,
|
||||
Deny: httpapi.ProblemHandler(http.StatusUnauthorized, "Session missing or invalid"),
|
||||
}
|
||||
deps := httpapi.Deps{Log: log, Auth: authn, TrustedOrigins: cfg.TrustedOrigins, HSTS: !cfg.InsecureCookies}
|
||||
deps := httpapi.Deps{Log: log, Auth: authn, TrustedOrigins: cfg.TrustedOrigins,
|
||||
HSTS: !cfg.InsecureCookies, Observe: telemetry.Middleware()}
|
||||
if db != nil {
|
||||
deps.DB = db
|
||||
authn.Sessions = db
|
||||
|
|
@ -107,7 +120,7 @@ func run(log *slog.Logger) error {
|
|||
}
|
||||
|
||||
if db != nil {
|
||||
stopRunner, err := startRunner(ctx, cfg, db, log, &deps)
|
||||
stopRunner, err := startRunner(ctx, cfg, db, log, telemetry, &deps)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
|
@ -119,6 +132,9 @@ func run(log *slog.Logger) error {
|
|||
return err
|
||||
}
|
||||
|
||||
stopMetrics := serveMetrics(ctx, cfg.MetricsAddr, telemetry, log)
|
||||
defer stopMetrics()
|
||||
|
||||
// A second signal must kill rather than wait: once the drain starts, the handler is
|
||||
// unregistered and the next SIGTERM goes back to being fatal.
|
||||
go func() {
|
||||
|
|
@ -135,6 +151,57 @@ func run(log *slog.Logger) error {
|
|||
return srv.Run(ctx, ln)
|
||||
}
|
||||
|
||||
// serveMetrics exposes the exposition endpoint on a listener of its OWN, and returns the function
|
||||
// that stops it.
|
||||
//
|
||||
// Separate from the API on purpose, and the reason is that this surface has no session behind it.
|
||||
// The contract's surface answers 401 before 404 so that an anonymous caller cannot map it (STACK §4),
|
||||
// and metrics carry operational shape — queue depth, run counts, the size of this deployment —
|
||||
// which is the same class of fact. The two ways out of that are inventing a second authorization
|
||||
// model for scrapers or binding the endpoint where only the host can reach it; the second is what
|
||||
// every control plane does, and it is one line of deployment rather than one more thing to get
|
||||
// right. Default 127.0.0.1; empty turns it off.
|
||||
//
|
||||
// A listener that cannot bind is a WARN and not a fatal: telemetry is how a service is watched, not
|
||||
// how it serves, and taking the control plane down because a port is busy would make the watching
|
||||
// more dangerous than the not-watching.
|
||||
func serveMetrics(ctx context.Context, addr string, m *metrics.Metrics, log *slog.Logger) func() {
|
||||
if addr == "" {
|
||||
log.Warn("no TM_PLATFORM_METRICS_ADDR: this instance exposes no metrics")
|
||||
return func() {}
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
mux.Handle("GET /metrics", m.Handler())
|
||||
srv := &http.Server{
|
||||
Addr: addr,
|
||||
Handler: mux,
|
||||
// The same deadlines the API carries, and for once a WriteTimeout too: nothing here streams,
|
||||
// and a scrape that hangs is a connection held for nothing.
|
||||
ReadHeaderTimeout: 5 * time.Second,
|
||||
ReadTimeout: 10 * time.Second,
|
||||
WriteTimeout: 30 * time.Second,
|
||||
IdleTimeout: 60 * time.Second,
|
||||
BaseContext: func(net.Listener) context.Context { return context.WithoutCancel(ctx) },
|
||||
}
|
||||
var lc net.ListenConfig
|
||||
ln, err := lc.Listen(ctx, "tcp", addr)
|
||||
if err != nil {
|
||||
log.Warn("metrics endpoint could not be served", "addr", addr, "err", err)
|
||||
return func() {}
|
||||
}
|
||||
log.Info("serving metrics", "addr", ln.Addr().String())
|
||||
go func() {
|
||||
if err := srv.Serve(ln); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||
log.Error("metrics endpoint stopped", "err", err)
|
||||
}
|
||||
}()
|
||||
return func() {
|
||||
stop, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
|
||||
defer cancel()
|
||||
_ = srv.Shutdown(stop)
|
||||
}
|
||||
}
|
||||
|
||||
// sweepSessions deletes rows past their absolute expiry (PD-7). A failed sweep is logged and
|
||||
// retried on the next tick: it is housekeeping, and it must never take the service down.
|
||||
func sweepSessions(ctx context.Context, db *pgstore.Store, log *slog.Logger) {
|
||||
|
|
|
|||
|
|
@ -9,8 +9,11 @@ import (
|
|||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"textmachine/platform/internal/books"
|
||||
"textmachine/platform/internal/config"
|
||||
"textmachine/platform/internal/httpapi"
|
||||
"textmachine/platform/internal/jobs"
|
||||
"textmachine/platform/internal/metrics"
|
||||
"textmachine/platform/internal/money"
|
||||
"textmachine/platform/internal/pgstore"
|
||||
"textmachine/platform/internal/pricing"
|
||||
|
|
@ -18,16 +21,17 @@ import (
|
|||
"textmachine/platform/internal/runs"
|
||||
)
|
||||
|
||||
// startRunner wires the run lifecycle and returns the function that stops it.
|
||||
// startRunner wires the run lifecycle and the book intake, and returns the function that stops them.
|
||||
//
|
||||
// The reads are mounted whether or not runs can be spawned: a library and a book card are not run
|
||||
// machinery, and an instance without an engine binary is a perfectly useful read replica. What an
|
||||
// unconfigured runner refuses is starting a run, and it says so at boot rather than at the first
|
||||
// click.
|
||||
func startRunner(ctx context.Context, cfg config.Config, db *pgstore.Store, log *slog.Logger, deps *httpapi.Deps) (func(), error) {
|
||||
func startRunner(ctx context.Context, cfg config.Config, db *pgstore.Store, log *slog.Logger,
|
||||
m *metrics.Metrics, deps *httpapi.Deps) (func(), error) {
|
||||
deps.Library = db
|
||||
if !cfg.RunsEnabled() {
|
||||
log.Warn("no TM_PLATFORM_ENGINE_BIN: the library is served read-only and no run can be started")
|
||||
log.Warn("no TM_PLATFORM_ENGINE_BIN: the library is served read-only, no run can be started and no book can be uploaded")
|
||||
return func() {}, nil
|
||||
}
|
||||
model, err := pricing.New(money.MicroUSD(cfg.Runner.PerChapterMicroUSD))
|
||||
|
|
@ -73,16 +77,31 @@ func startRunner(ctx context.Context, cfg config.Config, db *pgstore.Store, log
|
|||
},
|
||||
Log: log,
|
||||
}
|
||||
queue, err := runs.NewQueue(db.Pool(), svc, log, cfg.Runner.Workers)
|
||||
intake, err := startIntake(cfg, db, rn, log, deps)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// A typed nil is not nil once it is inside an interface, and the queue decides which workers to
|
||||
// register by exactly that check. Spelled out so an instance without intake registers no parse
|
||||
// worker rather than one that calls a nil service.
|
||||
var parser jobs.Parser
|
||||
if intake != nil {
|
||||
parser = intake
|
||||
}
|
||||
queue, err := jobs.New(db.Pool(), svc, parser, log, cfg.Runner.Workers)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
svc.Queue = queue
|
||||
deps.Runs = svc
|
||||
if intake != nil {
|
||||
intake.Queue = queue
|
||||
}
|
||||
if err := queue.Start(ctx); err != nil {
|
||||
return nil, fmt.Errorf("start queue: %w", err)
|
||||
}
|
||||
go sweepRuns(ctx, svc, cfg.Runner.SweepEvery, log)
|
||||
go sweep(ctx, sweeps{runs: svc, books: intake, db: db, metrics: m},
|
||||
cfg.Runner.SweepEvery, log)
|
||||
return func() {
|
||||
// The queue is drained; the RUNS are not touched. They are transient units, not children,
|
||||
// and outliving this process is the whole point of the seam (D39.106).
|
||||
|
|
@ -94,19 +113,95 @@ func startRunner(ctx context.Context, cfg config.Config, db *pgstore.Store, log
|
|||
}, nil
|
||||
}
|
||||
|
||||
// sweepRuns runs the reconciler at boot and then on a ticker.
|
||||
// startIntake wires the book upload, or explains at boot why this deployment takes none.
|
||||
//
|
||||
// A nil service leaves POST /books unmounted, which is the same shape every unbuilt contract route
|
||||
// has: an instance that accepted a file it had nowhere to put would take the whole upload before it
|
||||
// could say so.
|
||||
func startIntake(cfg config.Config, db *pgstore.Store, engine books.Manifester, log *slog.Logger,
|
||||
deps *httpapi.Deps) (*books.Service, error) {
|
||||
if !cfg.IntakeEnabled() {
|
||||
log.Warn("no TM_PLATFORM_BOOKS_DIR: this instance accepts no book uploads")
|
||||
return nil, nil
|
||||
}
|
||||
// Created at boot, where a permission problem is an operator's to see, rather than on the first
|
||||
// upload, where it would be one user's mysterious failure.
|
||||
if err := os.MkdirAll(cfg.Intake.BooksDir, 0o750); err != nil {
|
||||
return nil, fmt.Errorf("books directory %s: %w", cfg.Intake.BooksDir, err)
|
||||
}
|
||||
svc := &books.Service{
|
||||
Store: db,
|
||||
Engine: engine,
|
||||
Cfg: books.Config{
|
||||
BooksDir: cfg.Intake.BooksDir,
|
||||
EngineBinary: cfg.Runner.EngineBinary,
|
||||
},
|
||||
Log: log,
|
||||
}
|
||||
deps.Intake = svc
|
||||
deps.Upload = httpapi.UploadLimits{
|
||||
MaxBytes: cfg.Intake.MaxUploadBytes,
|
||||
Deadline: cfg.Intake.UploadDeadline,
|
||||
}
|
||||
return svc, nil
|
||||
}
|
||||
|
||||
// sweepBudget is what ONE pass of the RUN sweep may take. Per pass rather than per tick: see the
|
||||
// note inside sweep().
|
||||
const sweepBudget = 2 * time.Minute
|
||||
|
||||
// intakeSweepBudget is the same for the intake, and it is larger because the work inside it is:
|
||||
// re-driving one book's parse is an engine call the queue itself bounds at jobs.JobTimeout, and a
|
||||
// pass shorter than that turns "this book is large" into "this host cannot run the engine".
|
||||
const intakeSweepBudget = jobs.JobTimeout + time.Minute
|
||||
|
||||
// sweeps is everything one tick of the reconciler covers.
|
||||
type sweeps struct {
|
||||
runs *runs.Service
|
||||
books *books.Service
|
||||
db *pgstore.Store
|
||||
metrics *metrics.Metrics
|
||||
}
|
||||
|
||||
// sweep runs the reconcilers at boot and then on a ticker.
|
||||
//
|
||||
// The boot pass is not a special case and is not skipped: after a reboot every transient unit is
|
||||
// gone, and this is what notices and restarts the runs they were carrying (unified backlog row 138).
|
||||
func sweepRuns(ctx context.Context, svc *runs.Service, every time.Duration, log *slog.Logger) {
|
||||
sweep := func() {
|
||||
c, cancel := context.WithTimeout(ctx, 2*time.Minute)
|
||||
// The intake's own pass rides the same ticker — it answers the same question about a different
|
||||
// object, "whose walk stopped and nobody is coming back for it".
|
||||
func sweep(ctx context.Context, s sweeps, every time.Duration, log *slog.Logger) {
|
||||
// EACH pass gets its OWN budget, and that is not tidiness: sharing one deadline let the runs pass
|
||||
// spend all of it — two runs whose engine hangs are two minutes — and hand an already-expired
|
||||
// context to the intake sweep and to the telemetry, every tick, for as long as those runs sat
|
||||
// there. The starvation the per-run budget bounds inside one pass would simply have moved up a
|
||||
// level, and the metric that shows it would have stopped updating with it.
|
||||
pass := func(name string, budget time.Duration, fn func(context.Context) error) {
|
||||
c, cancel := context.WithTimeout(ctx, budget)
|
||||
defer cancel()
|
||||
if err := svc.Sweep(c); err != nil && !errors.Is(err, context.Canceled) {
|
||||
log.Error("run sweep failed", "err", err)
|
||||
start := time.Now()
|
||||
err := fn(c)
|
||||
// A pass that ran out of its budget left work untouched, and the list is ordered the same way
|
||||
// every time, so the tail starves. Counted rather than only logged: this is the number that
|
||||
// says whether it is happening (register row PD-169).
|
||||
s.metrics.ObserveSweep(name, time.Since(start), errors.Is(err, context.DeadlineExceeded))
|
||||
if err != nil && !errors.Is(err, context.Canceled) {
|
||||
log.Error(name+" sweep failed", "err", err)
|
||||
}
|
||||
}
|
||||
sweep()
|
||||
one := func() {
|
||||
pass("runs", sweepBudget, s.runs.Sweep)
|
||||
if s.books != nil {
|
||||
// The intake's pass is allowed MORE than the runs' pass, and the number is not a taste: one
|
||||
// book's parse is the same engine call the queue gives `jobs.JobTimeout`, and a pass that
|
||||
// cannot contain one kills it — which the intake then counts as a host that cannot run the
|
||||
// engine and spends an attempt on. The pass has to be able to hold at least one book.
|
||||
pass("intake", intakeSweepBudget, s.books.Sweep)
|
||||
}
|
||||
c, cancel := context.WithTimeout(ctx, sweepBudget)
|
||||
defer cancel()
|
||||
observe(c, s, log)
|
||||
}
|
||||
one()
|
||||
t := time.NewTicker(every)
|
||||
defer t.Stop()
|
||||
for {
|
||||
|
|
@ -114,11 +209,35 @@ func sweepRuns(ctx context.Context, svc *runs.Service, every time.Duration, log
|
|||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
sweep()
|
||||
one()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// observe publishes the state of the control plane. A failure to measure never fails the sweep: it
|
||||
// is one WARN and the next tick tries again.
|
||||
func observe(ctx context.Context, s sweeps, log *slog.Logger) {
|
||||
o, err := s.db.Observe(ctx)
|
||||
if err != nil {
|
||||
log.Warn("the control plane's own state could not be read", "err", err)
|
||||
return
|
||||
}
|
||||
s.metrics.ObserveRunner(metrics.Runner{
|
||||
QueueDepth: o.QueueDepth,
|
||||
OldestHoldSeconds: o.OldestHoldSeconds,
|
||||
QuarantinedAttempts: o.QuarantinedAttempts,
|
||||
LiveRuns: o.LiveRuns,
|
||||
BooksUploading: o.BooksUploading,
|
||||
BooksParsing: o.BooksParsing,
|
||||
})
|
||||
lag, err := s.runs.Lag(ctx)
|
||||
if err != nil {
|
||||
log.Warn("the tailer's lag could not be read", "err", err)
|
||||
return
|
||||
}
|
||||
s.metrics.ObserveTailerLag(lag)
|
||||
}
|
||||
|
||||
// markerArgv is the command systemd runs when a unit ends. It defaults to the admin CLI next to the
|
||||
// running daemon, because that is the one binary guaranteed to be the same build as this process.
|
||||
func markerArgv(configured string) ([]string, error) {
|
||||
|
|
|
|||
|
|
@ -78,8 +78,31 @@ TM_PLATFORM_OIDC_CLIENT_ID=...
|
|||
TM_PLATFORM_OIDC_REDIRECT_URL=https://app.example.org/auth/callback
|
||||
TM_PLATFORM_AFTER_LOGIN=/library
|
||||
TM_PLATFORM_SIGNUP_GRANT_USD=5
|
||||
TM_PLATFORM_BOOKS_DIR=/srv/textmachine/books
|
||||
TM_PLATFORM_METRICS_ADDR=127.0.0.1:9464
|
||||
```
|
||||
|
||||
⚠ **`TM_PLATFORM_BOOKS_DIR` — каталог, который сервис ПИШЕТ и, на отклонённом интейке, УДАЛЯЕТ.**
|
||||
Всё под ним создано загрузкой книг; книга, заведённая дев-инструментом с чужим `--workdir`, туда не
|
||||
попадает и не удаляется никогда (гард `books.owns`). Каталог должен существовать и принадлежать
|
||||
`tmplatform`, и он обязан быть в `ReadWritePaths=` юнита — при `ProtectSystem=strict` отсутствующий
|
||||
путь валит сборку mount-namespace целиком (PD-91, та же грабля, что и с `/srv/textmachine`):
|
||||
|
||||
```sh
|
||||
install -d -m0750 -o tmplatform -g tmplatform /srv/textmachine/books
|
||||
```
|
||||
|
||||
⚠ **`/metrics` не аутентифицирован** — его защищает только адрес привязки. Дефолт `127.0.0.1`, то
|
||||
есть снаружи он недостижим; поднять его на внешний адрес значит открыть операционную форму деплоя
|
||||
(глубина очереди, число прогонов, возраст холдов) всем, кто до него дотянется. Скрейпер живёт на том
|
||||
же хосте либо ходит через тот же edge, что и API. Разбор — `docs/STACK_DECISIONS.md` §24, строка
|
||||
риска — PD-179.
|
||||
|
||||
⚠ **Загрузка книги идёт минуты, и это касается edge-прокси.** Маршрут `POST /v0/books` принимает до
|
||||
`TM_PLATFORM_MAX_UPLOAD_BYTES` (64 МиБ) и держит соединение до `TM_PLATFORM_UPLOAD_DEADLINE`
|
||||
(10 минут). Прокси перед сервисом обязан разрешать столько же: у него свои `client_max_body_size` и
|
||||
свои таймауты чтения, и молчаливо режет их он, а не мы.
|
||||
|
||||
⚠ **`TM_PLATFORM_STATE_DIR` меняется только когда живых прогонов нет.** Exit-маркер пишется по пути,
|
||||
вычисленному при СПАВНЕ, а читается по пути из текущей конфигурации: после смены каталога конец
|
||||
идущего прогона становится невидим, и реконсилятор перезапускает его как потерянный (PD-155). Путь
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -40,9 +40,19 @@
|
|||
- Отказ зависимости — деградация с диагностикой, не тишина: проглоченная ошибка без лога и без
|
||||
комментария-обоснования = дефект (класс PD-5/PD-16).
|
||||
|
||||
**Наблюдаемость:** структурные логи (route-pattern, не сырой путь — id пользователя/книги в логи не
|
||||
текут), `X-Request-Id` на каждом ответе, ошибки видимы на своём уровне. Money-числа — только в
|
||||
приватных таблицах, ни в логах, ни в ответах (кроме P-5-статуса процентами).
|
||||
**Наблюдаемость** — базовая линия **практики именования Prometheus** (базовые единицы: секунды и
|
||||
байты; `_total` у счётчиков; единица в имени, не в лейбле) + **четыре золотых сигнала** SRE на
|
||||
вопрос «что мерить» (внесено P5 по ратифицированному направлению PD-115: у оси появился внешний
|
||||
эталон, а не только собственная проза зоны). Конкретика:
|
||||
- структурные логи (route-pattern, не сырой путь — id пользователя/книги в логи не текут),
|
||||
`X-Request-Id` на каждом ответе, ошибки видимы на своём уровне;
|
||||
- метрики отдаются в формате Prometheus на ОТДЕЛЬНОМ слушателе (`STACK_DECISIONS` §24); лейбл несёт
|
||||
паттерн маршрута, никогда путь — иначе это и неограниченная кардинальность, и библиотека
|
||||
пользователя в индексе оператора;
|
||||
- Money-числа — только в приватных таблицах, ни в логах, ни в ответах, ни в метриках (кроме
|
||||
P-5-статуса процентами);
|
||||
- ⚠ оси **ops** (релиз/откат, ёмкость, восстановление) и **конфигурация** внешнего эталона
|
||||
по-прежнему не имеют — PD-115 открыт ими.
|
||||
|
||||
**Контракт-первичность:** поверхность = `docs/architecture/14-api-contract/openapi.yaml`; расхождение
|
||||
кода со спекой = дефект чей-то один: либо спека правится через ратификацию, либо код. Генерация
|
||||
|
|
|
|||
|
|
@ -6,15 +6,15 @@
|
|||
>
|
||||
> Общая записка по обоим новым сервисам — `frontend/docs/STACK_DECISIONS.md` §5 (02.08). Здесь —
|
||||
> платформенная часть с датами релизов и сверкой. Что изменилось за два дня: три библиотечных пина
|
||||
> §5 (pgx · goose · River) на 04.08 всё ещё последние; по Go последним патчем стенда идёт 1.26.5
|
||||
> (07.07) — floor `go.mod` оставлен общим с движком, а тулчейн сборки поднят до 1.26.5 (см. ниже).
|
||||
> §5 (pgx · goose · River) на 04.08 всё ещё последние; по Go последним патчем стенда идёт **1.26.6**
|
||||
> (13.08, поднят security-адвизори — см. таблицу) — floor `go.mod` оставлен общим с движком.
|
||||
|
||||
## Пины
|
||||
|
||||
| Что | Пин | Релиз пина | Зачем нам |
|
||||
|---|---|---|---|
|
||||
| Go (язык, `go.mod`) | **1.26.4** | 02.06.2026 | Тот же floor, что у движка (`backend/go.mod`) — общий стенд собирает оба модуля одним тулчейном |
|
||||
| Go (тулчейн сборки, `make tools-check`) | **≥1.26.5** | 07.07.2026 | 1.26.5 несёт security-фиксы `crypto/tls` и `os`; сетевой модуль собирается ими, а не «чем-нибудь 1.26» |
|
||||
| Go (тулчейн сборки, `make tools-check`) | **≥1.26.6** | 13.08.2026 | Поднят с 1.26.5 не по вкусу, а по `make vuln`: база адвизори опубликовала пять уязвимостей stdlib против 1.26.5 — `net/http`, `crypto/tls`, `net/url`, `encoding/xml`, `encoding/asn1` (GO-2026-6218/6090/6089/6088/5972), все закрыты в 1.26.6, и две трассируются в пути, которые эта служба зовёт (`pgstore.Open → pgx.ParseConfig → asn1.Unmarshal`). На 1.26.6 батарея зелёная и скан чист. ⚠ Как floor РЕАЛЬНО держится (первая редакция этого подъёма не держала ничего — регекс `go1\.26\.([5-9]…)` принимал ту самую 1.26.5, а `GO_MIN_VERSION` жил только в echo): `make version-check` СРАВНИВАЕТ версии (`sort -V`, префиксы `rc`/`devel` отвергаются), и `go.mod` несёт `toolchain go1.26.6` — его читает всякая сборка, даже мимо make: при `GOTOOLCHAIN=auto` хост скачает нужный тулчейн, при `=local` остановится с ошибкой. Само сравнение запинено (`internal/gates`) |
|
||||
| PostgreSQL | **18.x** (проверено на 18.4), floor **16** | 18.4 — май 2026 | 18 — текущая мажорная (19 в бете, в прод не берём); floor 16, потому что River тестируется на трёх последних мажорных |
|
||||
| HTTP | stdlib `net/http` + `ServeMux` | — | Роутер-библиотека не нужна: `ServeMux` с 1.22 умеет метод+wildcards, а `Request.Pattern` даёт лог по маршруту, не по пути |
|
||||
| CSRF | stdlib `http.CrossOriginProtection` | Go 1.25 | Ровно тот механизм, что описан в §5 (Sec-Fetch-Site → Origin), теперь в тулчейне — свой велосипед не пишем |
|
||||
|
|
@ -221,6 +221,28 @@
|
|||
|
||||
23. **«Ошибка материализации» — два разных факта, и различает их `runs.quarantines`.** Транзиентное — повтор следующим свипом; сюда входят не только дедлок и сериализация (`40P01`/`40001`), но и всё, во что превращается ШТАТНЫЙ рестарт управляемого Postgres: класс 08, `57P01`/`57P02`/`57P03`, `pgconn.SafeToRetry`, любой `net.Error`. Первая редакция знала только первые два, и рестарт базы карантинил проекцию живого платного прогона навсегда — снятия карантина в дереве нет (найдено ре-чеком V2, исполнением). Граница держится на типах: ошибка чтения ФАЙЛА — `*fs.PathError`, а он `net.Error` не удовлетворяет; пропасть, конфликт payload, битая строка — карантин ПОПЫТКИ (её проекции), прогон при этом продолжается и продолжает платить. До различения дедлок, который разрешился сам, ослеплял проекцию живого платного прогона навсегда.
|
||||
|
||||
## Что решено сессией P5 (11.08) — загрузка книги, стоп/резюм, наблюдаемость
|
||||
|
||||
24. **Метрики — `prometheus/client_golang` v1.24.1, на ОТДЕЛЬНОМ слушателе, и это выбор против stdlib.** Норма зоны требует stdlib прежде библиотеки (`ENGINEERING_STANDARDS` §1), поэтому первым рассмотрен `expvar` — и он этой работы не несёт: нет лейблов (⇒ «запросы по маршруту и коду» не выражаются вовсе), нет гистограмм (⇒ на вопрос о задержке остаётся среднее — единственная статистика, которая прячет хвост), а его JSON не читает ни один скрейпер без переводчика. Сэкономил бы он зависимость, а стоил бы написания недостающих трёх руками — то есть ровно того самописного пути, который та же норма и запрещает. OpenTelemetry для этого деплоя тяжелее: коллектор процессом, протокол экспорта настройкой, и всё равно scrape-эндпоинт на конце. Пин сверен живьём 11.08 (`proxy.golang.org/@latest`), релиз 24.07.2026.
|
||||
|
||||
**Внешний эталон оси (половина PD-115):** практики именования Prometheus (базовые единицы — секунды и байты; `_total` у счётчиков; единица не в лейбле) плюс «четыре золотых сигнала» на вопрос «что мерить». Пин на форму — `metrics.TestTheRunnersStateIsExposedWithItsUnits`, он же ловит единицу в имени.
|
||||
|
||||
**Кардинальность:** лейбл несёт ПАТТЕРН маршрута, никогда путь — сырой путь это библиотека пользователя в индексе оператора (PD-3) и неограниченное число рядов; неразобранный запрос сворачивается в один ряд `(unmatched)`, потому что там лейбл выбирает не сервер. Пин — `TestRequestsAreCountedByRoutePatternAndNeverByPath`.
|
||||
|
||||
**Отдельный слушатель (`TM_PLATFORM_METRICS_ADDR`, дефолт `127.0.0.1:9464`), а не маршрут под `/v0`.** Контрактная поверхность отвечает 401 раньше 404, чтобы аноним не картографировал её (§4), а экспозиция несёт ту же породу фактов — глубину очереди, число прогонов, размер деплоя. Выходов два: вторая модель авторизации для скрейпера или привязка туда, где достаёт только хост; второе — то, что делает каждый контрол-плейн, и это строка деплоя вместо ещё одной вещи, которую надо не сломать (та же логика, по которой админ-поверхность стала CLI, §10). Не поднявшийся слушатель — WARN, не фатал: телеметрия это как за сервисом смотрят, а не как он служит.
|
||||
|
||||
**Значения снимает СВИП, а не скрейп.** Гейджи ставятся раз в проход реконсилятора, который и так читает базу по таймеру; коллектор, ходящий в Postgres на каждый скрейп, отдал бы нагрузку на контрол-плейн тому, у кого есть доступ к эндпоинту.
|
||||
|
||||
25. **Дедлайн чтения маршрута загрузки РАСШИРЯЕТСЯ, но никогда не снимается.** `ReadTimeout` сервера (30 с) покрывает ВЕСЬ запрос вместе с телом — это и есть закрытие PD-2, — а книга это десятки мегабайт и минуты бытового аплинка. Маршрут `POST /books` ставит собственный конечный дедлайн через `http.ResponseController` (`TM_PLATFORM_UPLOAD_DEADLINE`, дефолт 10 минут); §12 запрещает СНЯТИЕ дедлайна, и запрет в силе — снятый дедлайн возвращает полу-кормленный запрос к неограниченному удержанию, а расширенный оставляет его ограниченным. Первоисточник тот же, что и у самого запрета: доккоммент `net/http.Server.ReadTimeout` называет пер-запросный дедлайн именно для этого случая («Because ReadTimeout does not let Handlers make per-request decisions on each request body's acceptable deadline or upload rate…»). Пин — `TestTheUploadRouteExtendsItsOwnReadDeadlineAndOnlyItsOwn`, и он двусторонний: тот же медленный запрос на ОБЫЧНОМ маршруте обязан быть отрезан. ⚠ Следствие для обёрток: любая обёртка `ResponseWriter` обязана нести `Unwrap`, иначе расширение молча становится no-op (посадка «убрать `Unwrap` из метрик» падает).
|
||||
|
||||
26. **Приём книги: строка ПЕРЕД байтами, файл в каталоге книги, разбор — отдельным шагом.** Порядок «строка, потом байты» делает `uploading` состоянием, которое кто-то может наблюдать (вторая вкладка видит книгу, пока файл ещё идёт), и — что важнее операционно — делает брошенную загрузку НАХОДИМОЙ: строка единственное, что говорит, чей каталог под `BooksDir`. Цена порядка названа: языки обязаны прийти ДО файла, потому что потоковый читатель отдаёт части в порядке провода; тем же правилом специфицирована браузерная загрузка S3, а в спеке нашего контракта его нет — заведено вопросом владельцу контракта (PD-172).
|
||||
|
||||
Тело читается `r.MultipartReader` и льётся в файл `source.<расширение>`; расширение сохраняется, потому что движок диспетчеризует читатель по нему (`.epub` → epub, иначе текст), а не потому, что платформа знает форматы — списка форматов у неё нет и не должно быть. Разбор = `tmctl manifest --json`, $0-команда, которая и создаёт БД проекта; она даёт число глав и идентичность разреза (`chunker_version`, `source_sha256`).
|
||||
|
||||
**Два разных отказа и две разных судьбы файла.** Движок ОТВЕТИЛ «не разобрать» (`*exec.ExitError`) — это про книгу, терминально с первого ответа, каталог удаляется: перепарсить нечем, скачать нечем, а аутентифицированный маршрут, который пишет на диск оператора и ничего не убирает, был бы дырой, которую этот пак открыл бы сам. Движок НЕ ЗАПУСТИЛСЯ — это про хост: заявка возвращается, следующий свип пробует снова, и только после бюджета попыток книга становится `rejected` с файлом НА МЕСТЕ (удалять чужую загрузку из-за своей поломки — не наше решение).
|
||||
|
||||
27. **Стоп различается НАМЕРЕНИЕМ, записанным до сигнала, а не догадкой по коду выхода.** Движок ловит SIGTERM и выходит кодом 1, поэтому маркер пользовательского стопа и маркер аварии — одни и те же байты (PD-152). Платформа пишет `runs.stop_requested_at` в Postgres и только потом просит systemd; на гонку «стоп против самостоятельного финиша» стоит сверка времени маркера, а чистые исходы движка (0/2/3) намерение перебивают — завершённый перевод не должен показываться отменённым. Следствия, каждое запинено: прогон с намерением НЕ перезапускается реконсилятором (иначе деньги уходят на работу, которую владелец только что отменил), живой юнит с намерением получает повторный стоп (закрывает «платформа умерла между записью и вызовом»), а стоп до спавна заканчивает прогон и возвращает холд целиком. ⚠ Что этим НЕ закрыто и названо: штатная перезагрузка хоста по-прежнему выглядит как авария — там намерения нет ни у кого, и различить это может только сам движок различимым кодом выхода (строка 165 единого бэклога).
|
||||
|
||||
## Как поднять локально
|
||||
|
||||
```sh
|
||||
|
|
@ -236,6 +258,11 @@ curl -s localhost:8080/readyz # ready
|
|||
`TM_PLATFORM_SESSION_MAX_AGE` · `TM_PLATFORM_INSECURE_COOKIES` (dev, по HTTP) ·
|
||||
`TM_PLATFORM_OIDC_ISSUER` · `_OIDC_CLIENT_ID` · `_OIDC_CLIENT_SECRET` (или `_FILE`) ·
|
||||
`_OIDC_REDIRECT_URL` · `TM_PLATFORM_AFTER_LOGIN` · `TM_PLATFORM_SIGNUP_GRANT_USD`.
|
||||
Приём книги: `TM_PLATFORM_BOOKS_DIR` (только АБСОЛЮТНЫЙ путь, корень каталогов книг; пусто =
|
||||
инстанс загрузок не принимает и `POST /books` не смонтирован) · `TM_PLATFORM_MAX_UPLOAD_BYTES`
|
||||
(дефолт 64 МиБ — потолок ОДНОЙ загрузки, §25) · `TM_PLATFORM_UPLOAD_DEADLINE` (дефолт 10 минут —
|
||||
сколько тело одной загрузки вправе идти).
|
||||
Наблюдаемость: `TM_PLATFORM_METRICS_ADDR` (дефолт `127.0.0.1:9464`; пусто = метрики не отдаются).
|
||||
Раннер: `TM_PLATFORM_ENGINE_BIN` (версионированный путь tmctl; пусто = инстанс только читает) ·
|
||||
`TM_PLATFORM_STATE_DIR` (только АБСОЛЮТНЫЙ путь: маркер пишет юнит из каталога книги, читает демон
|
||||
из своего — относительный назвал бы два разных файла; отказ на буте) · `TM_PLATFORM_CTL_BIN` ·
|
||||
|
|
@ -246,20 +273,50 @@ curl -s localhost:8080/readyz # ready
|
|||
на той сборке движка, с которой прогон начался — строка 139).
|
||||
Вход монтируется, только если задана ВСЯ четвёрка OIDC; половина конфигурации — отказ на старте.
|
||||
|
||||
⚠ Все они печатаются на старте с источником (`default`/`environment`/`file`) — PD-114; секреты и
|
||||
денежные суммы печатаются фактом наличия, без значения.
|
||||
|
||||
Админ-команды: `tmplatformctl grant --user <id> --usd 5 [--note ...] [--key ...]` ·
|
||||
`balance --user <id>` · `logins --user <id>` · `revoke --user <id>`.
|
||||
|
||||
### Postgres на стенде без root
|
||||
|
||||
**Основной способ (13.08): ПОЛНЫЙ дистрибутив в домашнем каталоге, тоже без sudo.**
|
||||
|
||||
```sh
|
||||
# micromamba — один статический бинарь с conda-forge; прокси стенда пускает conda-forge
|
||||
curl -sL -o mm.tar.bz2 https://conda.anaconda.org/conda-forge/linux-64/micromamba-2.9.0-0.tar.bz2
|
||||
tar -xjf mm.tar.bz2 bin/micromamba && install -m755 bin/micromamba ~/.local/bin/micromamba
|
||||
|
||||
MAMBA_ROOT_PREFIX=$HOME/.local/share/mamba \
|
||||
~/.local/bin/micromamba create -y -p ~/.local/pgsql -c conda-forge 'postgresql=18.4'
|
||||
|
||||
~/.local/pgsql/bin/initdb -D ~/.local/share/tmstand/pgdata -U postgres -A trust \
|
||||
--no-locale --encoding=UTF8
|
||||
~/.local/pgsql/bin/pg_ctl -D ~/.local/share/tmstand/pgdata \
|
||||
-l ~/.local/share/tmstand/pg.log -o "-k /tmp -p 55433 -c listen_addresses=''" start
|
||||
export TM_PLATFORM_TEST_DSN='postgres://postgres@/postgres?host=/tmp&port=55433&sslmode=disable'
|
||||
```
|
||||
|
||||
164 МБ, ставится в `~/.local` (та же схема, что у Go и Node на этой машине), sudo не нужен, и в
|
||||
пакете есть `psql`, `pg_dump`, `pg_basebackup`, contrib — то есть БД можно смотреть руками. Порт
|
||||
55433 выбран, чтобы не спорить со стендом-скрэтчпадом на 55432, если тот поднят.
|
||||
⚠ Проверено исполнением 13.08: батарея зоны `make check` под `-race` на этом стенде — EXIT=0,
|
||||
скипов 0, все пакеты зелёные. Стенд переживает уборку скрэтчпадов сессий; после рестарта машины
|
||||
поднимается тем же `pg_ctl … start`.
|
||||
|
||||
**Запасной способ — бинарники `io.zonky.test.postgres` с Maven Central в скрэтчпад:**
|
||||
|
||||
```sh
|
||||
# бинарники io.zonky.test.postgres с Maven Central, распакованные в скрэтчпад
|
||||
pg/bin/initdb -D pgdata -U postgres -A trust --no-locale --encoding=UTF8
|
||||
pg/bin/pg_ctl -D pgdata -l pg.log -o "-k /tmp -p 55432 -c listen_addresses=" start
|
||||
export TM_PLATFORM_TEST_DSN='postgres://postgres@/postgres?host=/tmp&port=55432&sslmode=disable'
|
||||
```
|
||||
|
||||
⚠ Сокет кладём в `/tmp` (`-k`): полный путь скрэтчпада длиннее лимита Unix-сокета.
|
||||
⚠ `psql` в пакете zonky НЕТ — только `initdb`/`pg_ctl`/`postgres`; проверять из Go.
|
||||
⚠ В пакете zonky ТОЛЬКО `initdb`/`pg_ctl`/`postgres` — проверено содержимым архива
|
||||
(1071 файл, в `bin/` три штуки); `psql` нет, смотреть БД приходится из Go. Живёт в скрэтчпаде
|
||||
сессии, поэтому умирает вместе с ним — для долгой работы предпочтителен способ выше.
|
||||
|
||||
⚠ Postgres на стенде отсутствует как системный пакет и sudo нет. Схема и запросы этой сессии
|
||||
проверены на ЖИВОМ PostgreSQL **18.4**, поднятом без root из бинарников zonky
|
||||
|
|
|
|||
|
|
@ -6,15 +6,32 @@
|
|||
|
||||
## Текущее состояние
|
||||
|
||||
- **P5 ПОСТРОЕН и ДОФИКШЕН по приёмке — дерево передаётся на лендинг (11.08).** `POST /v0/books`
|
||||
потоково с пер-маршрутным потолком тела и своим дедлайном чтения · статусы
|
||||
`uploading → parsing → not_started | rejected` получили писателей, разбор — $0-команда движка
|
||||
`tmctl manifest` · `POST /v0/runs/{id}/stop|resume` с намерением стопа в Postgres ДО сигнала ·
|
||||
метрики Prometheus на отдельном слушателе · печать эффективной конфигурации.
|
||||
- **Приёмка 11.08 (8 находок) → дофикс → кросс-семейное ревью дофикса (5 находок) → ре-чек
|
||||
оркестратора (3 находки, 2 high) → ТРЕТИЙ РАУНД ЗАКРЫТ (14.08)**: разделы «Третий раунд»,
|
||||
«Дофикс-2» и «Дофикс P5» ниже, движение — записками-планами. Батарея на стенде `~/.local/pgsql`
|
||||
(порт 55433) под `-race`: EXIT=0, 16 пакетов, линтер 0 issues, скипов 0, `make vuln` чист, тестов
|
||||
**261 → 359**. Посадки: 10/10 · 6/6 · 6/6 (две пережившие за все раунды — оба раза виноват был
|
||||
слепой пин, переписан пин, не мутация); списки — в журнале.
|
||||
⚠ **Тулчейн 1.26.6 НА РАТИФИКАЦИЮ**, и вчерашний клейм про него был ЛОЖЕН: гейт принимал 1.26.5
|
||||
(регекс вместо сравнения). Теперь floor держат два механизма — `make version-check` сравнивает
|
||||
версии, `go.mod` несёт `toolchain go1.26.6`, который читает всякая сборка; при `GOTOOLCHAIN=auto`
|
||||
хост скачает нужный тулчейн, при `=local` остановится. Само сравнение запинено (`internal/gates`).
|
||||
**ВОПРОС на ратификацию — кто пишет `book.yaml` при интейке**: половина П-9 стоит на нём, раздел
|
||||
«Развилка» ниже. Дерево не коммичено.
|
||||
- **P4 «раннер» + дофикс + V2 ПРИНЯТЫ и ЗАЛЕНДЕНЫ — D39.123, код `d29e30c`** (ревью-шапка ниже):
|
||||
транзиентный systemd-юнит на прогон · очередь River с холдом-до-спавна в одной транзакции ·
|
||||
реконсилятор с базовой линией расчёта · тейлер `events.jsonl` с карантином проекции · пять
|
||||
ручек `/v0` контракта **0.2.1** · дев-интейк. Батарея с живым PG 18.4 под `-race` зелёная,
|
||||
скипов 0, тестов зоны **261**. Формула аргумента потолка — `committed + прирост` (PD-158,
|
||||
ратифицирована ПОПРАВКОЙ к пингу оркестратора).
|
||||
- **Регистр — 171 строка** (`python3 docs/scripts/counts.py`): 124 закрыто · 1 закрыт ратификацией (PD-59) · 3 риском · 43 открыто
|
||||
(**1 major — PD-113**: стоп по потолку отдаётся `failed` до эмиттера; после PD-158 его цена
|
||||
выросла — потолок срабатывает ровно на исчерпании холда).
|
||||
- **Регистр — 198 строк** (`python3 docs/scripts/counts.py`): 145 закрыто · 1 закрыт ратификацией
|
||||
(PD-59) · 4 риском · 48 открыто (**1 major — PD-113**: стоп по потолку отдаётся `failed` до
|
||||
эмиттера). Форма — 13 секций (пинг оркестратора №16).
|
||||
- **Промт P5 ВЫДАН оркестратором 10.08** (`PLATFORM_P5_SESSION_PROMPT.md`, D39.128): П-9
|
||||
`POST /books` (+PD-72 потолок тела; развилка «кто пишет book.yaml при интейке» — вопросом в этот
|
||||
журнал ДО стройки той половины) · стоп/резюм-ручки PD-140 (различение «потолок vs авария» —
|
||||
|
|
@ -25,6 +42,812 @@
|
|||
(D39.107/109/112/114); разделы — в [archive/platform-PROGRESS-P0-P3.md](archive/platform-PROGRESS-P0-P3.md).
|
||||
Стек и рецепт стенда — `STACK_DECISIONS.md`; критерии приёмки — `ENGINEERING_STANDARDS.md`.
|
||||
|
||||
## Третий раунд (14.08): ре-чек оркестратора FP5-9…FP5-11 и хвосты
|
||||
|
||||
Все три находки подтвердились на коде до правки; возражений нет. Две из трёх — мои вчерашние фиксы,
|
||||
которые ЗАЯВЛЕНЫ шире, чем сделаны, и это ровно тот класс, за который приёмка уже била.
|
||||
|
||||
### FP5-9 [HIGH] Гейт тулчейна не отказывал 1.26.5 — и три дока это утверждали
|
||||
|
||||
Проверено исполнением: `make version-check GO_VERSION=go1.26.5` на прежнем правиле проходил.
|
||||
Регекс `go1\.26\.([5-9]|[0-9]{2,})` принимал ровно ту версию, ради отказа от которой floor и
|
||||
поднимали, а `GO_MIN_VERSION` жил только в тексте сообщения. Тот же регекс ставил 1.26.10 НИЖЕ
|
||||
1.26.9. Клейм «хост на 1.26.5 получит отказ» был ложен в трёх местах (журнал ×2, `STACK_DECISIONS`,
|
||||
коммент Makefile) — гейт «поднят» был только на словах.
|
||||
|
||||
Сделано, двумя независимыми механизмами:
|
||||
|
||||
1. `make version-check` СРАВНИВАЕТ версии (`sort -V`), а не матчит; префиксы `rc`/`devel` отвергаются
|
||||
отдельной веткой — пререлиз не несёт фиксов, которые обещает его номер. Цель отделена от
|
||||
`tools-check` и берёт версию из переменной `GO_VERSION`, чтобы её можно было судить версиями,
|
||||
которых на хосте нет.
|
||||
2. `go.mod` получил `toolchain go1.26.6`. **Решение и обоснование** (оркестратор просил решить): floor
|
||||
ЯЗЫКА (`go 1.26.4`) остаётся общим с движком — один стенд собирает оба модуля; директива
|
||||
`toolchain` поднимает только тулчейн, то есть stdlib, которым линкуется сетевой модуль. Её читает
|
||||
ВСЯКАЯ сборка, включая ту, что не зовёт make: при `GOTOOLCHAIN=auto` (дефолт) хост скачает 1.26.6
|
||||
вместо того, чтобы собрать уязвимый бинарь, при `=local` — остановится с ошибкой, называющей
|
||||
версию. Обе ветки — тот ответ, который нужен; ни одна не зависит от того, прочитал ли кто-то
|
||||
Makefile. Цена названа: офлайн-хост с `=local` и только 1.26.5 теперь не соберёт зону — это и есть
|
||||
гейт.
|
||||
|
||||
Пин — на СРАВНЕНИЕ, а не на прогон: новый пакет `internal/gates` гоняет `version-check` таблицей
|
||||
версий, которых на этой машине нет (1.26.4/1.26.5/1.25.9/1.9.9/rc/devel — отказ;
|
||||
1.26.6/1.26.7/1.26.10/1.27.0/2.0.0 — приём), и вторым тестом требует, чтобы `go.mod` и
|
||||
`GO_MIN_VERSION` называли одну версию. Две посадки падают. Три ложных клейма переписаны на описание
|
||||
механизма. Строка PD-197.
|
||||
|
||||
### FP5-10 [HIGH] Гард корня не переживал unmount и маскировался бутовым `MkdirAll`
|
||||
|
||||
Тоже подтверждено: `Stat(BooksDir)` отвечает «есть» про пустой mountpoint, оставшийся после
|
||||
размонтирования тома, смонтированного РОВНО в `BooksDir`, — а бут безусловным `MkdirAll` пересоздаёт
|
||||
корень после рестарта. То есть мой вчерашний фикс закрывал только форму «каталог удалён вместе с
|
||||
родителем» и не закрывал ту, ради которой писался.
|
||||
|
||||
Сделано по кандидату оркестратора — **сентинел провижининга**: файл `.tmplatform-books` в корне,
|
||||
который пишет ПЕРВАЯ загрузка (`markStorage`, `O_EXCL`, поэтому две одновременные загрузки обе правы)
|
||||
и не пишет больше никто — в частности, не пишет бут. Решение «хранилище на месте» принимается по
|
||||
СЕНТИНЕЛУ (`storageIsThere`), а не по существованию каталога: ни unmount, ни `MkdirAll` его не
|
||||
подделывают. Хост, который никогда не принимал загрузок, отвечает «нет» — безопасная сторона: у него
|
||||
нет книг, которые можно отклонить.
|
||||
|
||||
Пин — `books.TestAnUnmountedVolumeLooksLikeAnEmptyRootAndStillIsNotTheBooksFault`: удаляем всё под
|
||||
корнем и пересоздаём сам корень (ровно то, что оставляют unmount и бут), десять циклов ожидания —
|
||||
книга жива, бюджет цел, движок не спрошен, сентинел не воскрес. **Первая редакция этого пина посадку
|
||||
ПЕРЕЖИЛА** (мутация «не писать сентинел вовсе»): без сентинела всё ждёт, и тест этого не отличал —
|
||||
добавлено утверждение, что принятая загрузка сентинел ПИШЕТ, иначе терминальность крэш-окна (FP5-3)
|
||||
теряется. Обе посадки падают. PD-192 дополнена.
|
||||
|
||||
### FP5-11 Правило ревизии применено не ко всем писателям
|
||||
|
||||
Пять писателей жизненного цикла ПРОГОНА остались на `revision + 1` против правила, которое дофикс
|
||||
ввёл для трёх интейковых: старт, реоткрытие, пауза (`runs.go`) и два закрытия (`sink.go`). Значит
|
||||
staleness библиотеки, которую вчера закрыли на статусах интейка, оставалась на статусах прогона:
|
||||
книга не-максимума аккаунта уходила `translating` — библиотека не двигалась.
|
||||
|
||||
Сделано: все пять берут `nextRevisionOfThisBooksLibrary`. Пин —
|
||||
`pgstore.TestARunTransitionOnAnOlderBookStillMovesTheLibrary` (вторая книга держит максимум 41,
|
||||
прогон стартует на первой; проверяются обе половины жизненного цикла — старт и пауза). Посадка
|
||||
падает.
|
||||
|
||||
⚠ **Мотивированное исключение, названное вслух:** два писателя `books.revision` НЕ переведены —
|
||||
прогресс-путь синка (`bumpBook` на `unitDone`/`progress`). Причина не вкусовая: `unitDone` вычисляет
|
||||
`chapters.revision` из `books.revision + 1` ДО инкремента, и прыжок книги через пол развязал бы
|
||||
книжную и главную шкалы, а это отдельная scope-семантика контракта. Она и не нужна: прогресс
|
||||
существует только у книги, у которой ИДЁТ прогон, а старт прогона теперь сам поднимает книгу на
|
||||
уровень библиотеки — дальше `+1` монотонен и виден. Если правило когда-нибудь понадобится и там,
|
||||
оно приедет вместе с проверкой главной шкалы, а не молча.
|
||||
|
||||
### Хвосты ре-чека
|
||||
|
||||
- **(а)** Коммент `PauseRun` обещал проверку «в том же стейтменте», а стоит отдельный
|
||||
`select … for update` в той же транзакции. Текст приведён к коду (транзакция и её замки).
|
||||
- **(б)** Комменты `parse.go` («terminal on the first answer») описывали до-дофиксное поведение на
|
||||
пути, который УДАЛЯЕТ файл пользователя, — самое опасное место для устаревшего текста. Переписаны:
|
||||
через бюджет попыток идёт КАЖДЫЙ ответ движка, удаляет только последний.
|
||||
- **(в)** Остаток FP5-5 закрыт кодом, а не строкой: проход, которому осталось меньше, чем нужно
|
||||
разбору, книгу НЕ НАЧИНАЕТ (клейм берётся внутри `Parse`, поэтому отложенная книга не тратит
|
||||
ничего) и говорит об этом одной строкой лога вместо ошибки на книгу. Пин —
|
||||
`books.TestAPassTooShortForAParseStartsNoneAtAll`, посадка падает.
|
||||
- **(г)** Мета-пин systemd-гейта перечислителен — принято как названный остаток: он ловит возврат к
|
||||
сверке сообщений по словам systemd и требует, чтобы гейт вообще спрашивал менеджер, но исчерпать
|
||||
все будущие формулировки перечислением нельзя.
|
||||
|
||||
### Посадки третьего раунда
|
||||
|
||||
| Посадка (что откатывается) | Пин, который обязан упасть | Итог |
|
||||
|---|---|---|
|
||||
| гейт снова матчит регексом (принимает 1.26.5) | `gates.TestTheToolchainGateComparesVersionsRatherThanMatchingThem` | поймана |
|
||||
| `go.mod` снова без `toolchain` | `gates.TestGoModPinsTheSameToolchainTheBatteryDemands` | поймана |
|
||||
| гард снова смотрит на существование корня | `books.TestAnUnmountedVolumeLooksLikeAnEmptyRootAndStillIsNotTheBooksFault` | поймана |
|
||||
| сентинел не пишется загрузкой | тот же | ПЕРЕЖИЛА → пин усилен → поймана |
|
||||
| старт прогона снова `revision + 1` | `pgstore.TestARunTransitionOnAnOlderBookStillMovesTheLibrary` | поймана |
|
||||
| проход снова начинает книгу, на которую нет времени | `books.TestAPassTooShortForAParseStartsNoneAtAll` | поймана |
|
||||
|
||||
### Батарея третьего раунда
|
||||
|
||||
- `make check` с живым PG 18.4 (стенд `~/.local/pgsql`, порт 55433) под `-race`: **EXIT=0, 16
|
||||
пакетов, скипов 0**, линтер **0 issues**, `make vuln` — чисто.
|
||||
- Тесты `^func Test` исполнением: **354 → 359** (+5, удалённых 0).
|
||||
- Посадки: **6 из 6** (одна со второго захода, см. таблицу).
|
||||
|
||||
### Записка-план третьего раунда: ID → статус → улика
|
||||
|
||||
| ID | Статус | Улика |
|
||||
|---|---|---|
|
||||
| FP5-9 | ЗАКРЫТО | `version-check` сравнивает (`sort -V`) + `toolchain go1.26.6` в `go.mod`; `internal/gates` ×2 пина, две посадки падают; три клейма переписаны. PD-197 |
|
||||
| FP5-10 | ЗАКРЫТО | сентинел `.tmplatform-books`, пишет только загрузка; `books.TestAnUnmountedVolumeLooksLikeAnEmptyRootAndStillIsNotTheBooksFault`; две посадки падают. PD-192 дополнена |
|
||||
| FP5-11 | ЗАКРЫТО с названным исключением | пять писателей прогона на `nextRevisionOfThisBooksLibrary`; `pgstore.TestARunTransitionOnAnOlderBookStillMovesTheLibrary`; посадка падает. Прогресс-путь синка НЕ тронут — обоснование выше. PD-122 дополнена |
|
||||
| хвост (а) | ЗАКРЫТО | текст `PauseRun` приведён к коду |
|
||||
| хвост (б) | ЗАКРЫТО | текст `parse.go` про терминальность приведён к коду. PD-198 |
|
||||
| хвост (в) | ЗАКРЫТО кодом | проход не начинает книгу без полного бюджета; `books.TestAPassTooShortForAParseStartsNoneAtAll`; посадка падает. PD-189 дополнена |
|
||||
| хвост (г) | ПРИНЯТО как остаток | перечислительность мета-пина названа, не «закрыта» |
|
||||
|
||||
|
||||
## Дофикс-2 (13.08): кросс-семейное ревью САМОГО дофикса
|
||||
|
||||
Пропуск, который я закрыл только по вопросу владельца: дофикс приёмки был проверен исполнением
|
||||
(пины · посадки · батарея · живая проба), но НЕ был отревьюен независимо — author≠reviewer для
|
||||
дофикс-диффа не соблюдался. Запущены два ревьюера другой семьи (Fable 5, я Opus), read-only по
|
||||
дереву, каждый со своей линзой — «деньги и гонки» и «интейк, потеря данных, правдивость чисел».
|
||||
Вернули **пять дефектов с воспроизведением, одну гипотезу и два расхождения отчётности**. Все
|
||||
подтверждены мной на коде до правки; возражений нет.
|
||||
|
||||
### Н1 [medium-high, интейк] Пропавший КОРЕНЬ хранилища читался как вина каждой книги
|
||||
|
||||
`os.Stat(workdir)` даёт ENOENT и когда пропал каталог одной книги, и когда не смонтирован сам
|
||||
`BooksDir`. Первое — терминальный конец этой книги (и лечение крэш-окна отказа, FP5-3), второе —
|
||||
беда хоста. Читая второе как первое, ОДИН проход свипа терминально отклонял ВСЕ книги в интейке с
|
||||
`source_unreadable` — причиной, которая винит файл пользователя, терминальна по устройству и не
|
||||
имеет обратного хода (ни перепарса, ни удаления, PD-175). То есть мой же фикс FP5-3 создал путь
|
||||
массового уничтожения загрузок из-за размонтированного тома.
|
||||
|
||||
Сделано: `ErrStorageGone` отличён от `ErrDirectoryGone` — корень спрашивается ПЕРЕД тем, как винить
|
||||
книгу; причина `storage_unavailable` никогда не терминальна, бюджет не тратит (движка не звали) и на
|
||||
книге не хранится. Предикат `waitsForTheDeployment` собрал оба таких случая в одном месте.
|
||||
Пин — `books.TestAVanishedStorageRootIsNotEveryBooksFault` (две книги, снесённый корень, десять
|
||||
циклов ожидания: обе живы, бюджет цел, движок не спрошен ни разу, том вернулся — следующий проход
|
||||
разбирает). Посадка падает. Строка PD-192.
|
||||
|
||||
### Н2 [minor, ревизия] Пол ревизии глотал весь статусный проход книги
|
||||
|
||||
Продолжение FP5-1 на соседнем пути: вставка научилась брать `greatest(max, пол) + 1`, а смены
|
||||
статуса остались на `revision + 1`. Книга, загруженная ДО чужой отменённой загрузки, идёт
|
||||
`uploading → parsing → not_started` тремя инкрементами СВОЕГО счётчика и всё ещё под полом —
|
||||
`greatest(max, пол)` не двигается вовсе, а клиент по контракту отбрасывает чтение, которое не выше
|
||||
применённого. Экран, только что загрузивший книгу, показывает её «приходящей», пока библиотеку не
|
||||
сдвинет постороннее событие. Монотонность при этом не нарушена — это staleness, и именно поэтому
|
||||
прошлый круг её не заметил.
|
||||
|
||||
Сделано: три писателя статуса (`StartParsing`, `FinishParse`, `RejectBook`) берут следующий номер
|
||||
БИБЛИОТЕКИ, а не свой + 1 (`nextRevisionOfThisBooksLibrary`, владелец читается из обновляемой
|
||||
строки). Пин — `books.TestAStatusChangeIsVisibleEvenUnderTheLibrarysFloor`, посадка падает. PD-122
|
||||
переформулирована: остаток — собственный счётчик области (двое ОДНОВРЕМЕННЫХ писателей могут
|
||||
вычислить одно и то же число; значение при этом растёт, поэтому клиент перезапрашивает).
|
||||
|
||||
### М1 [medium, деньги] `PauseRun` — третий закрывающий путь без гарда живой попытки
|
||||
|
||||
Тот же класс, что PD-181 и FP5-2, на пути, который прошлый круг не проверил: пауза закрывала прогон,
|
||||
не спрашивая, жива ли ещё закрываемая попытка И этого ли она прогона (апдейт попытки шёл даже без
|
||||
`run_id`). Сценарий ревьюера: два поколения реконсилятора при перекрывающемся деплое, у старого ниже
|
||||
`PerChapter` в конфиге → его проход считает остаток исчерпанным и паузит прогон, который уже
|
||||
рестартован в живую попытку 2. Итог — прогон `paused/credit_exhausted` под живым тратящим движком, а
|
||||
открытый холд попытки 2 не виден НИ в `ListLiveRuns` (прогон завершён), НИ в `UnsettledRuns` (попытка
|
||||
не завершена).
|
||||
|
||||
Сделано: тот же `exists (… a.id = $4 and a.run_id = runs.id and a.ended_at is null)`, что у соседей,
|
||||
плюс `run_id` в апдейте попытки. Пин — `pgstore.TestAPauseFromAnOldSnapshotDoesNotCloseARunOverALiveAttempt`
|
||||
(проверяет и то, что прогон жив, и то, что холд живой попытки виден в списке). Посадка падает. PD-193.
|
||||
|
||||
### М2 [low, проекция] Синк законченной попытки усыновлял чужой хендшейк
|
||||
|
||||
Форма, которой до P5 не существовало: стоп ДО спавна оставляет попытку закрытой и без
|
||||
`engine_run_id`. Устаревший материализатор с такой попыткой принимал hello попытки, которая её
|
||||
заменила, биндил чужой `engine_run_id` на себя и материализовал тот же журнал второй раз — счётчики
|
||||
глав и юнитов удваивались. Деньги не двигались.
|
||||
|
||||
Сделано: бинд отказан для законченной попытки (`ended_at is null` в CAS). Пин —
|
||||
`pgstore.TestAMaterializerOfAnEndedAttemptDoesNotAdoptTheNextAttemptsEngine`, посадка падает. PD-194.
|
||||
|
||||
### М3 [low, деньги] Возврат холда «прогона, который не запускался», ждал ответа движка
|
||||
|
||||
Ревьюер дал это гипотезой; я перепроверил и подтвердил. `settle` сперва требовал успешного
|
||||
`tmctl status`, и только потом ветка «юнита не было — вернуть холод целиком». Но хост, который
|
||||
производит эту ситуацию, — ровно тот, где движок не запускается: `Status` падает на каждом проходе,
|
||||
и деньги прогона, который не начинался, остаются зарезервированными навсегда (видимыми, но
|
||||
запертыми).
|
||||
|
||||
Сделано: ветка «нет юнита и нет базовой линии» идёт ДО вызова движка; `ReleaseUnspawned`
|
||||
перепроверяет строку под замком, поэтому снапшот, который успели заспавнить, отвергается там и
|
||||
settle идёт обычным путём. Пин —
|
||||
`runs.TestTheHoldOfARunThatNeverStartedComesBackOnAHostWhoseEngineCannotAnswer` (движок отвечает
|
||||
«no such file» на любой запрос; холд обязан вернуться, движок — не быть спрошенным). Посадка падает.
|
||||
PD-195.
|
||||
|
||||
### Хвосты того же круга
|
||||
|
||||
- **Н3.** `StartParsing` и оба `abandon` в `Accept` шли на `context.WithoutCancel` без своего
|
||||
дедлайна: отсоединённый — не значит бесконечный, повисший запрос держал бы горутину запроса,
|
||||
которого уже нет. Переведены на `writeCtx` (30 с). **Пина нет и не заявляется**: это граница, а не
|
||||
наблюдаемое поведение — чтобы его пинить, нужен стор, умеющий висеть.
|
||||
- **Н4.** Опечатка оператора в рукописном `book.yaml` по-прежнему стоит файла пользователя: движок
|
||||
отвечает exit 1 на всё, через пять циклов книга отклоняется как `source_unreadable` и исходник
|
||||
удаляется. FP5-4 закрыл только «конфигурации нет вовсе». Заведено отдельной строкой риска PD-196,
|
||||
а не спрятано в комментарий: настоящее лечение — различающий код выхода или `--dry-run` на стороне
|
||||
движка, это не правка платформы.
|
||||
- **Н5.** Пин FP5-7 сверял ДВЕ точные формулировки старого стринг-матча — третье написание прошло бы
|
||||
мимо. Теперь тест разбирает файл гейта парсером и смотрит только КОД (комментарии там цитируют ту
|
||||
самую строку — текстовый скан принял бы историю за дефект), запрещает в коде слова systemd и
|
||||
требует, чтобы гейт вообще спрашивал менеджер. Посадка «вернуть матч по сообщению» падает.
|
||||
- **Отчётность.** Ревьюер не смог проверить клейм «посадки 10 из 10»: списки лежали в скрэтчпаде
|
||||
сессии, а не в дереве. Исправлено — списки посадок теперь ниже, в самом журнале.
|
||||
- Ловушка счёта, в которую ревьюер едва не попал и которую стоит знать: `git grep '^func Test' HEAD --
|
||||
platform` даёт 262 — лишняя строка из код-блока в архивном доке; счёт по `*_test.go` даёт 261.
|
||||
|
||||
### Списки посадок (то, чего не хватало отчёту)
|
||||
|
||||
| Раунд | Посадка (что откатывается) | Пин, который обязан упасть | Итог |
|
||||
|---|---|---|---|
|
||||
| Дофикс | вставка снова читает только `max(revision)` | `TestABookThatJoinsAfterACancelledUploadStillMovesTheRevision` | поймана |
|
||||
| Дофикс | `FinishUnspawnedStop` без гарда живой попытки | `TestAStaleUnspawnedStopDoesNotCloseAResumedRun` | поймана |
|
||||
| Дофикс | пропавший каталог снова = «нет конфигурации» | `TestABookWhoseDirectoryIsGoneIsRejectedRatherThanLeftWaiting` | поймана |
|
||||
| Дофикс | ожидание конфигурации снова жжёт бюджет | `TestWaitingForAConfigurationDoesNotBringDeletionCloser` | поймана |
|
||||
| Дофикс | проход интейка снова под бюджетом прогонов | `TestOneBookInTheSweepGetsABudgetAParseCanLiveIn` | поймана |
|
||||
| Дофикс | просроченный дедлайн снова 500 | `TestAnUploadThatOutlivesItsDeadlineIsNotAnInternalError` | поймана |
|
||||
| Дофикс | гейт systemd снова по тексту сообщения | `TestTheSystemdGateAsksAboutTheCapabilityAndNotAMessage` | поймана |
|
||||
| Дофикс | пауза перебивает стоп | `TestAStopDuringSettlementOutranksThePause` | поймана |
|
||||
| Дофикс | потолок частей формы снова `n > cap` | `TestAFormWithTooManyPartsIsRefusedAtTheCap` | ПЕРЕЖИЛА → пин переписан → поймана |
|
||||
| Дофикс | wire-тест ревизии без различающей силы | `TestTheCardProjectsTheRevisionTheStoreGaveAndInventsNone` | поймана |
|
||||
| Дофикс-2 | пропавший корень снова читается как вина книги | `TestAVanishedStorageRootIsNotEveryBooksFault` | поймана |
|
||||
| Дофикс-2 | смена статуса снова `revision + 1` | `TestAStatusChangeIsVisibleEvenUnderTheLibrarysFloor` | поймана |
|
||||
| Дофикс-2 | пауза снова без гарда живой попытки | `TestAPauseFromAnOldSnapshotDoesNotCloseARunOverALiveAttempt` | поймана |
|
||||
| Дофикс-2 | синк снова биндит законченную попытку | `TestAMaterializerOfAnEndedAttemptDoesNotAdoptTheNextAttemptsEngine` | поймана |
|
||||
| Дофикс-2 | возврат холда снова ждёт ответа движка | `TestTheHoldOfARunThatNeverStartedComesBackOnAHostWhoseEngineCannotAnswer` | поймана |
|
||||
| Дофикс-2 | гейт systemd снова сверяет сообщение (усиленный пин) | тот же | поймана |
|
||||
|
||||
### Батарея дофикса-2
|
||||
|
||||
- `make check` с живым PG 18.4 под `-race`: **EXIT=0, 15 пакетов, скипов 0**, линтер **0 issues**.
|
||||
- Тесты `^func Test` исполнением: **349 → 354** (+5 пинов, удалённых 0).
|
||||
- Посадки: **6 из 6** (таблица выше, нижние шесть строк).
|
||||
- ⚠ **`make vuln` покраснел между двумя прогонами ОДНОГО дня, и не из-за дерева.** База адвизори
|
||||
опубликовала пять уязвимостей stdlib против Go **1.26.5** — `net/http`, `crypto/tls`, `net/url`,
|
||||
`encoding/xml`, `encoding/asn1` (GO-2026-6218 / 6090 / 6089 / 6088 / 5972), все закрыты в
|
||||
**1.26.6**, и govulncheck трассирует две из них в пути, которые эта служба зовёт (например
|
||||
`pgstore.Open → pgx.ParseConfig → asn1.Unmarshal`). Поставил 1.26.6 в `~/.local` (sha256 сверен с
|
||||
go.dev), поднял `GO_MIN_VERSION` в Makefile 1.26.5 → 1.26.6 с обоснованием на месте, обновил пин в
|
||||
`STACK_DECISIONS.md`. На 1.26.6 батарея зелёная и скан чист.
|
||||
**НА РАТИФИКАЦИЮ:** хост сборки, оставшийся на 1.26.5, должен получать отказ.
|
||||
⚠ **Правка 14.08: в этой редакции он его НЕ получал** — гейт сравнивал регексом и принимал 1.26.5
|
||||
(ре-чек, FP5-9). Как floor держится теперь — раздел «Третий раунд» выше.
|
||||
|
||||
### Записка-план дофикса-2: ID → статус → улика
|
||||
|
||||
| ID | Статус | Улика |
|
||||
|---|---|---|
|
||||
| Н1 | ЗАКРЫТО | `ErrStorageGone` + `waitsForTheDeployment`; `books.TestAVanishedStorageRootIsNotEveryBooksFault`; посадка падает. PD-192 |
|
||||
| Н2 | ЗАКРЫТО | `nextRevisionOfThisBooksLibrary` у трёх писателей статуса; `books.TestAStatusChangeIsVisibleEvenUnderTheLibrarysFloor`; посадка падает. PD-122 переформулирована |
|
||||
| М1 | ЗАКРЫТО | гард живой попытки в `PauseRun` + `run_id` в апдейте попытки; `pgstore.TestAPauseFromAnOldSnapshotDoesNotCloseARunOverALiveAttempt`; посадка падает. PD-193 |
|
||||
| М2 | ЗАКРЫТО | `ended_at is null` в бинде; `pgstore.TestAMaterializerOfAnEndedAttemptDoesNotAdoptTheNextAttemptsEngine`; посадка падает. PD-194 |
|
||||
| М3 | ЗАКРЫТО (гипотеза ревью подтверждена мной) | ветка «не спавнился» ДО вызова движка; `runs.TestTheHoldOfARunThatNeverStartedComesBackOnAHostWhoseEngineCannotAnswer`; посадка падает. PD-195 |
|
||||
| Н3 | ЗАКРЫТО БЕЗ ПИНА (честно) | `writeCtx` на `StartParsing` и обоих `abandon`; граница, а не поведение — пин не заявляется |
|
||||
| Н4 | СТРОКОЙ | PD-196: exit 1 движка на опечатку в `book.yaml` стоит файла; лечение — на стороне движка |
|
||||
| Н5 | ЗАКРЫТО | пин FP5-7 разбирает КОД гейта парсером, а не текст файла; посадка падает |
|
||||
| Отчётность | ЗАКРЫТО | списки посадок перенесены из скрэтчпада в журнал (таблица выше) |
|
||||
| Тулчейн | НА РАТИФИКАЦИЮ | Go 1.26.5 → 1.26.6 по пяти адвизори stdlib; `make vuln` чист, батарея зелёная |
|
||||
|
||||
|
||||
## Дофикс P5 (11.08): фикс-лист приёмки FP5-1…FP5-8
|
||||
|
||||
Пак принят УСЛОВНО с восемью находками (одна high). Ниже — что сделано по каждой, чем проверено и
|
||||
что осталось. Каждое число снято исполнением, команда рядом. Возражений НЕТ: все восемь
|
||||
подтвердились на коде до правки — три из них были прямыми последствиями МОИХ же решений, принятых в
|
||||
разных файлах и не сверенных между собой.
|
||||
|
||||
### FP5-1 [HIGH] Пол ревизии библиотеки игнорировался на вставке
|
||||
|
||||
Приёмка права, и посылка перепроверена мной на дереве, не по памяти: `DeleteUpload` поднимал
|
||||
`users.library_revision`, `ListBooks` читал `greatest(max, пол)`, а `nextLibraryRevision` — ТОЛЬКО
|
||||
`max(books.revision)`. Книга, загруженная после отменённой загрузки, входила на или ниже пола, и
|
||||
одна ревизия отвечала за три разных состояния библиотеки: до отменённой загрузки, во время неё и
|
||||
после прихода следующей книги. Клейм «PD-122 закрыта в половине добавления» был ЛОЖЕН — тесты
|
||||
покрывали только свежий аккаунт, где пол ещё ноль.
|
||||
|
||||
Сделано: вставка берёт `greatest(max(revision), users.library_revision) + 1`. Пин —
|
||||
`books.TestABookThatJoinsAfterACancelledUploadStillMovesTheRevision`, и он гоняет именно ту
|
||||
последовательность, которой не было: книга → отмена → книга; проверяет и ревизию БИБЛИОТЕКИ, и
|
||||
ревизию самой новой книги (иначе её карточка устарела бы в момент появления). PD-122 пере-формулирована:
|
||||
открытым остаётся один случай — смена статуса книги, которая не самая новая, не двигает ни одну из
|
||||
половин.
|
||||
|
||||
### FP5-2 [деньги] `FinishUnspawnedStop` без гарда живой попытки
|
||||
|
||||
Тот же гард, что `FinishRun` получил кругом раньше (PD-181), на соседнем пути отсутствовал: путь
|
||||
закрытия «стопа до спавна» проверял только `unit_name is null`. Устаревший проход мог закрыть уже
|
||||
резюмированный прогон, и холд второй попытки выпадал из обоих списков; приёмка добавила к этому
|
||||
следствие, которого я не назвал: каждый следующий резюм отвечал бы 409 навсегда, потому что
|
||||
продолжать пришлось бы уже завершённый прогон.
|
||||
|
||||
Сделано: попытка обязана быть живой и принадлежать этому прогону (`ended_at is null and run_id = $`).
|
||||
Пин — `runs.TestAStaleUnspawnedStopDoesNotCloseAResumedRun`; посадка «снять гард» падает.
|
||||
Строка PD-186.
|
||||
|
||||
### FP5-3 Крэш-окно «снос каталога → запись строки»
|
||||
|
||||
Клейм «порядок самоизлечивающийся» был ложен, и ложным его сделала МОЯ ЖЕ вторая правка того же
|
||||
круга: снесённый каталог читается как «нет конфигурации», а эту причину я тем же паком сделал
|
||||
НЕтерминальной — книга оставалась в `parsing` навсегда, ожидая конфигурацию, которую положить некуда.
|
||||
Диспозиция F11 («порядок выбран самоизлечивающийся») переписана: она стояла на посылке, которая к
|
||||
моменту записи уже не была верна.
|
||||
|
||||
Сделано: пропавший КАТАЛОГ отличён от отсутствующей конфигурации (`ErrDirectoryGone`) и терминален
|
||||
сразу — движка при этом не зовём вовсе. Пин —
|
||||
`books.TestABookWhoseDirectoryIsGoneIsRejectedRatherThanLeftWaiting`. Строка PD-187.
|
||||
|
||||
### FP5-4 Ожидание конфигурации жгло бюджет разбора
|
||||
|
||||
`ClaimParse` считает КАЖДУЮ заявку, а книга без конфигурации заявляется раз в грацию бесконечно.
|
||||
После пяти циклов ожидания бюджет исчерпан — и первый же ответ движка становится терминальным
|
||||
мгновенно, вместе с удалением исходника. Движок отдаёт exit 1 и на опечатку в `book.yaml`, так что
|
||||
цена ошибки оператора — файл пользователя.
|
||||
|
||||
Сделано: попытка возвращается (`RefundParseAttempt`), когда движок не был спрошен вовсе; возврат
|
||||
условен на клейме, как и все прочие записи, заканчивающие проход. Пин —
|
||||
`books.TestWaitingForAConfigurationDoesNotBringDeletionCloser`: после пятнадцати циклов ожидания
|
||||
бюджет цел, и первый отказ движка после этого НЕ терминален и ничего не удаляет. Строка PD-188.
|
||||
|
||||
### FP5-5 Бэкстоп гнал разбор под бюджетом прохода
|
||||
|
||||
Проход интейка шёл под теми же двумя минутами, что и проход прогонов, а разбор внутри — это тот же
|
||||
вызов движка, которому очередь даёт пятнадцать минут. Большая книга убивалась дедлайном свипа,
|
||||
убийство читалось как «хост не может запустить движок», попытка списывалась — и так каждый проход,
|
||||
пока книга не отклонялась ЗА СВОЙ РАЗМЕР. Вторую половину приёмка назвала точно: запись отказа шла
|
||||
на уже просроченном контексте и терялась, то есть цикл был бесконечным.
|
||||
|
||||
Сделано: у прохода интейка свой бюджет (`jobs.JobTimeout + 1m`), у каждой книги внутри прохода —
|
||||
свой (`jobs.JobTimeout`), а терминальные записи (`FinishParse`/`RejectBook`/возврат попытки) идут на
|
||||
контексте, переживающем дедлайн вызова. Пин — `books.TestOneBookInTheSweepGetsABudgetAParseCanLiveIn`
|
||||
(фейк движка читает дедлайн, который ему выдали). Строка PD-189.
|
||||
|
||||
### FP5-6 Просроченный дедлайн загрузки уходил 500
|
||||
|
||||
Сделано: 408 problem+json. Выбор кода назван вслух: медленный клиент — не сломанный сервис, а 408 по
|
||||
RFC 9110 §15.5.9 означает ровно «полный запрос не пришёл за время, которое сервер готов был ждать», и
|
||||
сообщает клиенту, что лечение — повтор. Кода 408 в перечне операции нет, поэтому он внесён в тот же
|
||||
пакет вопросов владельцу контракта, что PD-172/PD-174/PD-180. Пин —
|
||||
`httpapi.TestAnUploadThatOutlivesItsDeadlineIsNotAnInternalError`. Строка PD-190.
|
||||
|
||||
### FP5-7 Скип-гард systemd-тестов пинил СООБЩЕНИЕ
|
||||
|
||||
Приёмка права дважды, и вторая половина — исправление МОЕЙ ошибки в фактах. Гейт существовал и
|
||||
скипал по подстроке «Failed to connect to bus», а systemd 259 отвечает «Failed to connect to user
|
||||
scope bus» — гейт перестал гейтить от чужой правки строки. Моя строка PD-178 при этом утверждала, что
|
||||
формы скипа для systemd в зоне НЕТ ВОВСЕ: это было неверно, и я записал предположение как факт.
|
||||
|
||||
⚠ И вторая посылка PD-178 устарела прямо в ходе дофикса: **на этом стенде пользовательский менеджер
|
||||
systemd теперь ЖИВ** (`systemctl --user show` → `Version=259.5`), и все три теста проходят — тогда
|
||||
как несколькими часами раньше в этой же сессии его не было (`Linger=no`, sudo нет, ручной запуск
|
||||
выходил кодом 1). То есть состояние менеджера ПЛАВАЕТ между сессиями стенда, и строка переписана
|
||||
именно так.
|
||||
|
||||
Сделано: гейт спрашивает СПОСОБНОСТЬ — доходит ли процесс до своего менеджера, — и скипает громко и
|
||||
поимённо, симметрично форме БД-тестов. Пин —
|
||||
`runner.TestTheSystemdGateAsksAboutTheCapabilityAndNotAMessage`: он падает, если гейт снова начнёт
|
||||
сверять текст. PD-178 закрыта.
|
||||
|
||||
### FP5-8 Хвосты
|
||||
|
||||
- **(а) деньги.** Стоп, пришедший в окно расчёта, отвечал `paused/credit_exhausted` — «кончились
|
||||
деньги» вместо «владелец остановил». Пауза теперь отказывает при висящем интенте
|
||||
(`ErrStopRequested`), реконсилятор заканчивает прогон стопом. Пин
|
||||
`runs.TestAStopDuringSettlementOutranksThePause`. Строка PD-191.
|
||||
- **(б)** PD-153 дополнена: у СТОП-пути окно закрыто с обеих сторон (заявка не выдаётся прогону с
|
||||
интентом; закрытие «стопа до спавна» спрашивает systemd, если у попытки есть базовая линия). Само
|
||||
окно PD-153 — грация от `started_at` — не тронуто.
|
||||
- **(в)** Потолок частей формы пропускал N+1 (`n > cap` вместо `n >= cap`). Пин
|
||||
`httpapi.TestAFormWithTooManyPartsIsRefusedAtTheCap` — и ПЕРВАЯ его редакция посадку ПЕРЕЖИЛА:
|
||||
она сверяла только код 400, а форма, у которой просто кончились части, тоже 400, так что обе
|
||||
версии потолка проходили. Переписан на наблюдаемое следствие: файл стоит ровно на части `cap+1`,
|
||||
и тест требует, чтобы интейку его НЕ отдали (`in.read == 0`) — при потолке на единицу больше файл
|
||||
дочитывается и книга принимается. Посадка теперь падает.
|
||||
- **(г)** Выключатель metrics-листенера был недостижим через окружение: пустое значение переменной
|
||||
неотличимо от неустановленного и берёт дефолт. Теперь выключает СЛОВО (`off`/`none`), и это
|
||||
записано там же, где документирован сам выключатель.
|
||||
- **(д)** Wire-тест ревизии вернул различающую силу: фикстура снова несёт РАЗНЫЕ числа (книга 41,
|
||||
прогон 12), и тест утверждает то, за что отвечает этот слой — он отдаёт число стора и не выдумывает
|
||||
своё. Правило, что число берётся из книги, пинится в сторе.
|
||||
- **(е)** `help` метрики `queue_depth` называет все состояния, которые она считает, включая `pending`.
|
||||
- **(ж)** Счёт тестов в таблице задач пересчитан исполнением.
|
||||
|
||||
### Батарея дофикса
|
||||
|
||||
- `make check` с живым PostgreSQL 18.4 под `-race`: **все пакеты зелёные, включая `internal/runner`**
|
||||
(менеджер systemd на стенде жив — см. FP5-7), линтер **0 issues**.
|
||||
- Скипы: 0 при живом менеджере; при мёртвом — три ИМЕНОВАННЫХ скипа вместо трёх падений.
|
||||
- Тесты `^func Test` исполнением: **340 → 349**, добавленных **9**, удалённых **0**, переименован
|
||||
**1** (`TestTheCardsRevisionIsTheBooksAndNotTheRuns` → `TestTheCardProjectsTheRevisionTheStoreGaveAndInventsNone`,
|
||||
FP5-8д — свойство то же, имя стало отвечать за то, что тест на самом деле утверждает). Против HEAD:
|
||||
**261 → 349**. Команда: `comm` двух списков `^func Test` (HEAD и дерево) — единственная строка
|
||||
расхождения и есть это переименование.
|
||||
- Посадки дофикса: **10, поймано 10** — но честно: первый круг дал **9 из 10**, пережившая посадка
|
||||
(в) показала, что пин сверял не то следствие; переписан пин, не мутация. Списки — `mut9.json`.
|
||||
- `make vuln`: чисто.
|
||||
|
||||
**Живая проба дофикса на собранном бинаре** (`go build ./cmd/tmplatformd`, боевой PostgreSQL 18.4,
|
||||
сессия заведена прямой вставкой в стенд и убрана после пробы):
|
||||
|
||||
- FP5-6: сырым сокетом отправлена шапка `POST /v0/books` с `Content-Length: 100000` и одной частью,
|
||||
дальше клиент «думает» 4 секунды при `TM_PLATFORM_UPLOAD_DEADLINE=2s`. Ответ —
|
||||
`HTTP/1.1 408 Request Timeout`, `Content-Type: application/problem+json`,
|
||||
тело `{"type":"about:blank","title":"The upload did not finish in time","status":408}`, соединение
|
||||
закрыто. Юнит-тест это же место видит через фейковый ридер: живой прогон подтверждает, что дедлайн
|
||||
РЕАЛЬНО срабатывает на соединении, а не только маппинг ошибки.
|
||||
- FP5-8г: с `TM_PLATFORM_METRICS_ADDR=off` в печати эффективной конфигурации стоит
|
||||
`value=off source=environment`, в логе — `no TM_PLATFORM_METRICS_ADDR: this instance exposes no
|
||||
metrics`, слушателя на порту нет (`ss -ltn` пуст). Со значением-адресом скрейп отдаёт 200.
|
||||
- FP5-8е: `# HELP tm_platform_queue_depth Jobs in the queue that have not finished (pending,
|
||||
available, running, scheduled or retryable)` — из живого скрейпа. Там же видно 9 семейств зоны из
|
||||
десяти: `sweep_unfinished_total` — счётчик с лейблами и серии до первого инкремента не имеет,
|
||||
это нормальная семантика prometheus, а не пропавшая метрика.
|
||||
- Заодно: `tm_platform_http_requests_total{code="200",method="GET",route="GET /v0/books"}` — лейбл
|
||||
маршрута это ПАТТЕРН (F9), и `GET /v0/books` живым Bearer'ом отвечает 200.
|
||||
|
||||
### Записка-план дофикса: ID → статус → улика
|
||||
|
||||
| ID | Статус | Улика |
|
||||
|---|---|---|
|
||||
| FP5-1 | ЗАКРЫТО | `nextLibraryRevision` = `greatest(max(revision), users.library_revision)+1`; `books.TestABookThatJoinsAfterACancelledUploadStillMovesTheRevision`; посадка «снова только максимум» падает |
|
||||
| FP5-2 | ЗАКРЫТО | гард живой попытки в `FinishUnspawnedStop` (`ended_at is null and run_id = $2`); `runs.TestAStaleUnspawnedStopDoesNotCloseAResumedRun`; посадка падает. PD-186 |
|
||||
| FP5-3 | ЗАКРЫТО | `ErrDirectoryGone` проверяется ПЕРЕД `ErrNotProvisioned` и терминален; `books.TestABookWhoseDirectoryIsGoneIsRejectedRatherThanLeftWaiting`; посадка падает. PD-187 |
|
||||
| FP5-4 | ЗАКРЫТО | `RefundParseAttempt` условно на клейме; `books.TestWaitingForAConfigurationDoesNotBringDeletionCloser`; посадка падает. PD-188 |
|
||||
| FP5-5 | ЗАКРЫТО | свой бюджет у прохода интейка и у каждой книги, терминальные записи на пережившем контексте; `books.TestOneBookInTheSweepGetsABudgetAParseCanLiveIn`; посадка падает. PD-189 |
|
||||
| FP5-6 | ЗАКРЫТО кодом, вопрос владельцу контракта СТРОКОЙ | `os.ErrDeadlineExceeded` → 408; `httpapi.TestAnUploadThatOutlivesItsDeadlineIsNotAnInternalError`; посадка падает. PD-190 |
|
||||
| FP5-7 | ЗАКРЫТО, PD-178 переписана и закрыта | гейт спрашивает способность соединиться с менеджером; `runner.TestTheSystemdGateAsksAboutTheCapabilityAndNotAMessage`; посадка падает. Менеджер на стенде сейчас ЖИВ (259.5) — три теста прошли |
|
||||
| FP5-8а | ЗАКРЫТО | `PauseRun` отказывает при висящем интенте (`ErrStopRequested`); `runs.TestAStopDuringSettlementOutranksThePause`; посадка падает. PD-191 |
|
||||
| FP5-8б | СТРОКОЙ | PD-153 дополнена: стоп-путь закрыт с обеих сторон; само окно грации не тронуто |
|
||||
| FP5-8в | ЗАКРЫТО со второго захода | `n >= cap`; пин переписан на «файл за потолком не отдан интейку»; посадка падает |
|
||||
| FP5-8г | ЗАКРЫТО | выключатель metrics-листенера — СЛОВО (`off`/`none`), пустая переменная неотличима от неустановленной; `config/effective_test.go` |
|
||||
| FP5-8д | ЗАКРЫТО | фикстура снова несёт разные числа (книга 41, прогон 12); `httpapi.TestTheCardProjectsTheRevisionTheStoreGaveAndInventsNone`; посадка падает |
|
||||
| FP5-8е | ЗАКРЫТО | `help` у `queue_depth` называет все состояния, включая `pending` |
|
||||
| FP5-8ж | ЗАКРЫТО | числа в таблице задач пересчитаны исполнением (см. батарею выше) |
|
||||
|
||||
|
||||
## Приёмка P5 оркестратором (11.08): ПРИНЯТ УСЛОВНО — дофикс-раунд ДО лендинга (прецедент P4)
|
||||
|
||||
**Что подтверждено исполнением.** Батарея пере-прогнана мной на СВЕЖЕПОДНЯТОМ стенде (zonky PG
|
||||
18.4, рецепт §«Postgres без root»): **все пакеты зелёные, ВКЛЮЧАЯ `internal/runner`** — на текущем
|
||||
стенде пользовательский менеджер systemd ЖИВ (`user@1000.service` active, systemd 259), и три
|
||||
теста PD-178 прошли; линтер 0 issues; тестов **340**, удалённых против HEAD **0** (comm пуст).
|
||||
Живые числа проб и мутационных прогонов read-only не пере-снимаются — приняты «со слов» с
|
||||
пометкой. Ревью 10 агентами (6 линз + скептики): **8 находок подтверждено (1 high), 0
|
||||
опровергнуто, 9 минорных.** Задеты деньги и данные пользователей ⇒ по прецеденту P4 пак
|
||||
принимается УСЛОВНО, лендинг после дофикса и ре-чека.
|
||||
|
||||
**Фикс-лист (движение — запиской-планом ID → статус → улика, D39.121):**
|
||||
|
||||
| ID | Находка | Лечение |
|
||||
|---|---|---|
|
||||
| FP5-1 **HIGH** | **Пол ревизии библиотеки игнорируется на вставке** (`pgstore/books.go:99`): `nextLibraryRevision` читает только `max(books.revision)`, а `DeleteUpload` поднимает `users.library_revision`; книга, загруженная после отменённой загрузки, входит НА/НИЖЕ текущей ревизии — три разных состояния библиотеки под одной ревизией, revision-keyed refetch слеп. Клейм закрытия PD-122 ложен; тесты покрывают только свежий аккаунт | вставка учитывает пол (`greatest`); тест со сценарием «отмена → повторная загрузка»; PD-122 переформулировать |
|
||||
| FP5-2 деньги | **`FinishUnspawnedStop` без гарда живой попытки** (`pgstore/sink.go:362`): закрытие по устаревшему снапшоту РЕЗЮМИРОВАННОГО прогона — холд попытки-2 вне `ListLiveRuns` И вне `UnsettledRuns`, движок без надзора, последующий резюм вечно 409. Ровно та гонка, от которой `FinishRun` уже получил гард (PD-181) | тот же гард «закрываемая попытка ещё живая»; пин + посадка |
|
||||
| FP5-3 | **Крашокно `removeDir → RejectBook`** (`books/parse.go:188`): смерть между ними = книга навечно в `parsing` (ENOENT читается как `not_configured`, терминального пути нет) — клейм «порядок самоизлечивающийся» ложен, диспозиция F11 стояла на ложной посылке | терминальность переживает краш (запись/маркер до сноса либо «каталога нет вовсе» = терминал); диспозицию F11 переписать |
|
||||
| FP5-4 | **`not_configured`-проходы жгут бюджет разбора** (`parse.go:138`): `ClaimParse` инкрементит `parse_attempts` безусловно ⇒ после ~5 циклов ожидания конфигурации ПЕРВЫЙ отказ движка терминален мгновенно — включая удаление исходника при опечатке `book.yaml` (exit 1 = «вина файла») | `not_configured` не тратит бюджет; пин «ожидание не приближает удаление» |
|
||||
| FP5-5 | **Свип-бэкстоп гонит разбор под общим бюджетом прохода 2 мин против 15 мин очереди** (`cmd/tmplatformd/runner.go:151`): восстановление большой книги убивается дедлайном свипа, каждый цикл жжёт попытку, а reject на просроченном контексте не записывается — kill-loop | у бэкстоп-разбора свой бюджет ≥ `jobs.JobTimeout` (пара тестом, как `claimGrace`); reject на свежем контексте |
|
||||
| FP5-6 | **Просроченный дедлайн загрузки уходит 500** (`httpapi/v0.go:404`) — код вне спеки и вне декларации PD-180 | замаппить в честный код problem+json; внести в пакет вопросов PD-180 |
|
||||
| FP5-7 | **Скип-гард systemd-тестов пинит сообщение, которого systemd 259 не даёт** (`runner/systemd_test.go:25`: ждёт «Failed to connect to bus», стенд говорит «…user scope bus…») — на стенде без менеджера тесты ПАДАЮТ, а не скипаются; PD-178 не знает об этом гарде и устарела фактами (менеджер сейчас ЖИВ, батарея зелёная) | гейт по СПОСОБНОСТИ (проба соединения), скип поимённо — симметрия DSN-формы; PD-178 переписать: «состояние менеджера плавает между сессиями стенда» |
|
||||
| FP5-8 хвосты | (а) стоп в окне settle отвечает `paused/credit_exhausted` вместо `stopped` (`reconcile.go:543`) · (б) PD-153-окно: полный возврат холда под стартующим движком через новый стоп-путь — дополнить строку · (в) потолок частей формы пропускает N+1 · (г) выключатель metrics-листенера недостижим через env · (д) wire-тест ревизии потерял различающую силу (фикстура 12→41 сравняла прогон с книгой) · (е) help `queue_depth` не называет `pending` · (ж) счёт тестов в таблице задач устарел | по весу: (а)(б)(д) — фиксы с пинами; (в)(г)(е)(ж) — мелкие правки/честные строки |
|
||||
|
||||
Развилка `book.yaml` (вопрос зоны) и четыре контракт-вопроса — на мне, ратифицирую при лендинге
|
||||
дофикса. Дерево остаётся незакоммиченным до ре-чека.
|
||||
|
||||
**Ре-чек дофикса-2 (11.08, оркестратор): батарея воспроизведена зелёной (354 теста, Go 1.26.6,
|
||||
стенд ~/.local/pgsql:55433), НО 3 находки подтверждены (2 high) — третий раунд по списку ниже,
|
||||
лендинг после него:**
|
||||
|
||||
| ID | Находка | Лечение |
|
||||
|---|---|---|
|
||||
| FP5-9 **HIGH** | **Гейт тулчейна НЕ отказывает 1.26.5**: `Makefile:33` — регекс `go1\.26\.([5-9]\|…)` принимает 1.26.5, `GO_MIN_VERSION=1.26.6` живёт только в echo; `go.mod` — `go 1.26.4` БЕЗ `toolchain`; клейм «хост на 1.26.5 получит отказ» ложен в трёх доках (журнал ×2, STACK_DECISIONS §, коммент Makefile) | регекс/сравнение версий чинится по-настоящему (+ `toolchain go1.26.6` в go.mod — реши и обоснуй); пин на функцию сравнения, не на прогон; три клейма поправить |
|
||||
| FP5-10 **HIGH** | **PD-192 не закрывает заявленный сценарий**: гард — только `Stat(BooksDir)` (`parse.go:235`); unmount тома, смонтированного РОВНО в BooksDir, оставляет пустой mountpoint → Stat успешен → терминальный reject ВСЕХ книг с удалением; безусловный `MkdirAll` на буте (`runner.go:129`) маскирует пропажу после рестарта | различать «хранилище пропало» от «книги нет» устойчиво к обеим формам (кандидат: сентинел-файл провижининга в BooksDir, который бут НЕ пересоздаёт; нет сентинела → ErrStorageGone, не терминалить, бюджет не жечь); пин на unmount-форму |
|
||||
| FP5-11 | **Н2 закрыт не для всех писателей статуса**: пять писателей жизненного цикла прогона остались на `revision+1` (`pgstore/runs.go:108,517,620` · `sink.go:318,396`) против собственного правила дофикса (`books.go:111`) — staleness библиотеки на статусах прогона | те же «следующий номер библиотеки», что у трёх интейковых; пин на прогонный переход у книги не-максимума аккаунта |
|
||||
| хвосты | (а) коммент `PauseRun` «в том же стейтменте» против фактического SELECT FOR UPDATE (`runs.go:580`) — текст к коду; (б) комменты `parse.go:41-52` «terminal on the first answer» описывают до-дофиксное поведение на пути удаления файла; (в) остаток FP5-5: в одном проходе бэкстопа полный бюджет получает только первая книга — вторая жжёт попытки на обрезанном контексте (узкий триггер — честная строка либо бюджет на КНИГУ); (г) мета-пин systemd-гейта перечислителен — остаток названный, не дефект | (а)(б) — правки текста; (в) — строка или фикс; (г) — знать |
|
||||
|
||||
## Сессия P5 (11.08): загрузка книги · стоп и резюм · наблюдаемость · форма регистра
|
||||
|
||||
**Каждое число ниже снято исполнением, команда стоит рядом.** Дерево не коммичено, в индексе ничего
|
||||
не держу; зона записи — только `platform/`.
|
||||
|
||||
### Таблица задач
|
||||
|
||||
| Задача промта | Что сделано | Чем проверено |
|
||||
|---|---|---|
|
||||
| 1. `POST /books` (П-9, PD-72) | Потоковый multipart (`r.MultipartReader`), пер-маршрутный потолок тела, свой дедлайн чтения, каталог книги под `TM_PLATFORM_BOOKS_DIR`, статусы `uploading → parsing → not_started \| rejected` с писателями, разбор = `tmctl manifest --json`, свип-бэкстоп интейка | **44 теста** (пак + пины дофикса): 22 `internal/books` + 11 `httpapi/intake_test.go` + 6 `pgstore/intake_test.go` + 5 на шов манифеста (`runner/manifest_test.go`). 21 посадка пака, живая проба на боевом бинаре |
|
||||
| ⚠ половина 1: кто пишет `book.yaml` | **НЕ построена — вопрос ниже.** Построен ШОВ: `books.ErrNotProvisioned`, книга без конфигурации отказывает разбором с причиной `not_configured` | `TestABookWithNoEngineConfigurationWaitsRatherThanDies` |
|
||||
| 2. Стоп и резюм (PD-140) | `POST /v0/runs/{id}/stop` пишет намерение (`runs.stop_requested_at`, миграция 00014) ДО сигнала; `/resume` переиспользует `reopen` реконсилятора и ставит спавн в очередь, а не в запрос; реконсилятор классифицирует маркер по намерению, не рестартит стопнутый прогон и повторяет стоп живому юниту | **32 теста** (пак + пины дофикса): 28 `runs/control_test.go` + 4 `httpapi/control_test.go`. 25 посадок пака, живая проба |
|
||||
| 3. Наблюдаемость (П-11) + PD-114 | `prometheus/client_golang` v1.24.1 на отдельном слушателе; 10 семейств метрик зоны, включая запросы и задержки по паттерну маршрута; бюджет свипа НА ПРОГОН (PD-169); печать эффективной конфигурации с источником и редакцией секретов и денег | **12 тестов** (пак + пин дофикса): 5 `internal/metrics` + 6 `config/effective_test.go` + 1 `internal/jobs`. 8 посадок пака, живая проба |
|
||||
| 4. Регистр секциями (пинг №16) | Таблица разложена на 11 секций (открытые по весу · принятый риск · закрытые ратификацией · закрытые по эрам P1–P5), построчная форма сохранена | `python3 docs/scripts/counts.py --check` — зелено; список ID до и после перекладки совпадает посимвольно |
|
||||
|
||||
### ⚠ ВОПРОС НА РАТИФИКАЦИЮ: кто пишет `book.yaml` при интейке
|
||||
|
||||
Промт гейтит эту половину явно, и она НЕ построена. Что известно и чего стоит каждый вариант:
|
||||
|
||||
**Факт.** Каталог новой книги создаёт платформа (иначе `POST /books` некуда писать), а разбор —
|
||||
`tmctl manifest --config <каталог>/book.yaml`; движок требует в конфиге `book_id`, пару языков,
|
||||
пути `pipeline`/`models`/`source_file` и хотя бы один потолок (`backend/internal/config/book.go`
|
||||
проверяет всё это на загрузке). D39.110 §2b говорит «платформа `book.yaml` не правит» — в контексте
|
||||
того решения речь шла о ПОТОЛКАХ, которые платформа передаёт аргументом прогона, а не о создании
|
||||
файла для книги, которой ещё не существует.
|
||||
|
||||
| Вариант | Что делает платформа | Цена |
|
||||
|---|---|---|
|
||||
| **А. Оператор кладёт файл** (то, что построено) | Ничего: каталог и исходник создаются, конфигурации ждём; без неё разбор отказывает и после бюджета попыток книга становится `rejected` с причиной `not_configured` | Загрузка через UI не доходит до конца без ручной работы оператора НА КАЖДУЮ книгу; бета с чужими пользователями на этом не живёт. Зато D39.110 §2b соблюдён буквально |
|
||||
| **Б. Рендер из деплой-шаблона при интейке** (кандидат промта) | Один раз, при создании каталога, рендерит `book.yaml` из шаблона деплоя (`TM_PLATFORM_BOOK_TEMPLATE`): языки и жанр из `BookIntake`, `source_file` — имя, которое платформа же и записала, потолки и пути пайплайна — из шаблона оператора. Дальше файл принадлежит оператору: платформа его не читает и не трогает | Платформа становится писателем файла движка ОДИН раз. Нужен шаблон в деплое (ещё одна переменная и ещё один файл, который можно забыть). Если оператор потом файл поправил — ничего не происходит, платформа туда не возвращается |
|
||||
| **В. Движок сам заводит книгу** | Платформа зовёт новую $0-команду движка (`tmctl init --source-lang zh --target-lang ru …`), движок пишет свой конфиг сам | Архитектурно чище всех: файл движка пишет движок, и требования к полям живут там же, где проверяются. Но это строка ЕДИНОГО бэклога и релиз движка, то есть П-9 стоит до неё |
|
||||
|
||||
**Что предлагает зона:** Б как бета-меру и В как правильную форму, строкой единого бэклога. При Б
|
||||
шов уже готов — `books.ErrNotProvisioned` заменяется рендером в ОДНОМ месте
|
||||
(`books.Service.manifest`), остальной интейк не меняется. **Ратифицирует оркестратор; тихо
|
||||
интерпретировать эту развилку сессия не стала.**
|
||||
|
||||
Вторым вопросом того же корня: **название книги.** `BookIntake` поля `title` не несёт, манифест
|
||||
движка названия не даёт вовсе (он даёт id, счёты и идентичность разреза). Сегодня заголовок берётся
|
||||
из ИМЕНИ ЗАГРУЖЕННОГО ФАЙЛА (`книга.epub` → «книга»), пустое остаётся пустым — синтезировать
|
||||
«Книга 1» значит показать читателю ярлык, которого никто не писал. Если ратифицируется Б, имя
|
||||
логично брать оттуда же; если владелец хочет поле `title` в форме — это правка спеки, не зоны.
|
||||
|
||||
### Дизайн стопа: намерение, а не догадка по коду выхода
|
||||
|
||||
Кандидат-дизайн промта взят целиком; вот его границы.
|
||||
|
||||
- **Что записывается.** `runs.stop_requested_at` пишется ОДНИМ оператором с проверкой владения и
|
||||
живости (`pgstore.RequestStop`: `update … from books where owner_id = $2 and finished_at is null
|
||||
returning …` плюс имя юнита живой попытки). Ноль строк — два разных ответа, и они разделены
|
||||
вторым запросом ТОЛЬКО на этом пути: видит владелец прогон → `ErrRunNotLive` (409), не видит →
|
||||
`ErrNoRun` (404). Идемпотентно через `coalesce`: второй стоп не сдвигает ПЕРВУЮ отметку, потому
|
||||
что она — улика о том, что случилось раньше.
|
||||
- **Порядок.** Запись коммитится ДО обращения к systemd. Запинено не косвенно: фейк раннера зовёт
|
||||
хук ВНУТРИ своего `Stop`, хук читает `stop_requested_at` из БД и требует, чтобы она уже была
|
||||
(`TestTheStopIsRecordedBeforeSystemdIsAsked`). Посадка «спросить systemd раньше» падает.
|
||||
- **Гонка «стоп против самостоятельного финиша».** Намерение перекрашивает исход, только если оно
|
||||
РАНЬШЕ маркера (`stoppedOnRequest`: `!l.StopRequestedAt.After(m.At)`), а чистые коды движка
|
||||
(0 · 2 · 3) намерение перебивают всегда — завершённый перевод не должен показываться отменённым.
|
||||
Три пина: стоп → `stopped`, маркер раньше стопа → `failed`, чистый выход → `ready`.
|
||||
- **Три следствия, каждое запинено.** Прогон с намерением НЕ перезапускается реконсилятором (иначе
|
||||
деньги уходят на работу, которую владелец отменил) · живой юнит с намерением получает стоп
|
||||
ПОВТОРНО (закрывает «платформа умерла между записью и вызовом») · стоп до спавна заканчивает
|
||||
прогон и возвращает холд ЦЕЛИКОМ (проверено балансом, не статусом).
|
||||
- **Чего дизайн НЕ закрывает, и это названо:** штатная перезагрузка хоста. Там намерения нет ни у
|
||||
кого, маркер честно скажет `exited/1`, и прогон закроется как `failed` вместо перезапуска.
|
||||
Различить это может только движок различимым кодом выхода graceful-stop (строка 165 единого
|
||||
бэклога) — платформенной догадки здесь быть не должно. PD-152 остаётся открытой этой половиной.
|
||||
|
||||
Резюм по контракту: `stopped`/`paused` → новая попытка с ОСТАТКОМ бюджета (`потолок − RunSpent`);
|
||||
`awaiting_bank` → 409 «подпись неполна», и она неполна у любого деплоя, потому что банк сегодня не
|
||||
материализуется вовсе (компаньон §3); нечего продолжать → 202 с прогоном КАК ЕСТЬ, без переписывания
|
||||
его состояния (прогон, который остановил пользователь, остаётся `stopped`, а не становится `paused`
|
||||
из-за того, что резюм не нашёл денег). Механика перезапуска переиспользована: `restart`
|
||||
реконсилятора и `Resume` — один `reopen`; решение «паузить» вынесено к вызывающим, потому что
|
||||
реконсилятор паузит ЖИВОЙ прогон, а резюм работает с уже завершённым и переписывать ему исход не
|
||||
вправе.
|
||||
|
||||
### Наблюдаемость: что выбрано и почему не stdlib
|
||||
|
||||
`expvar` рассмотрен первым (норма §1 «stdlib прежде библиотеки») и этой работы не несёт: нет
|
||||
лейблов — «запросы по маршруту и коду» не выражаются вовсе; нет гистограмм — на вопрос о задержке
|
||||
остаётся среднее, единственная статистика, которая прячет хвост; его JSON не читает ни один скрейпер.
|
||||
Сэкономил бы он зависимость, а стоил бы написания недостающих трёх руками — ровно того самописного
|
||||
пути, который та же норма и запрещает. Взят `prometheus/client_golang` **v1.24.1** (релиз
|
||||
24.07.2026, пин сверен живьём через `proxy.golang.org`), OpenTelemetry отклонён как тяжёлый для
|
||||
одной VM: коллектор процессом, протокол экспорта настройкой и всё равно scrape-эндпоинт на конце.
|
||||
Эталон оси (половина PD-115) — практики именования Prometheus плюс четыре золотых сигнала, внесены в
|
||||
`ENGINEERING_STANDARDS` §2. Разбор — `STACK_DECISIONS` §24.
|
||||
|
||||
Отдаётся ОТДЕЛЬНЫМ слушателем (`TM_PLATFORM_METRICS_ADDR`, дефолт `127.0.0.1:9464`), не маршрутом
|
||||
под `/v0`: экспозиция несёт операционную форму деплоя, а второй модели авторизации ради скрейпера
|
||||
зона не заводит — тот же довод, что сделал админ-поверхность CLI (§10). Риск принят строкой PD-179 и
|
||||
записан в `deploy/README.md`. Величины снимает СВИП, а не скрейп: коллектор, ходящий в Postgres на
|
||||
каждый запрос, отдал бы нагрузку на контрол-плейн тому, у кого есть доступ к порту.
|
||||
|
||||
Метрики зоны: глубина очереди · возраст самого старого открытого холда · попытки в карантине ·
|
||||
живые прогоны · отставание тейлера в байтах · книги в интейке по статусу · длительность свипа и
|
||||
счётчик проходов, не уложившихся в бюджет (PD-169) · запросы и задержки по паттерну маршрута.
|
||||
|
||||
### Что нашла ЖИВАЯ ПРОБА, чего не нашли тесты
|
||||
|
||||
Проба гоняет боевые `tmplatformd`/`tmplatformctl` против живого Postgres реальными HTTP-вызовами;
|
||||
подменены две чужие стороны и обе процессом — движок (как в P4) и systemd (стенда с пользовательским
|
||||
менеджером нет, PD-178). Нашла две вещи, и обе исправлены:
|
||||
|
||||
1. **Резюм спавнил движок ВНУТРИ запроса** — запрос висел, пока прогон не кончится. На фейке это
|
||||
видно как таймаут 120 с; на настоящем systemd висело бы меньше, но по-прежнему секундами:
|
||||
`spawnAttempt` читает отсчёт книги (`tmctl status` — секунды CPU движка) и ходит к systemd.
|
||||
Исправлено: резюм ставит спавн в ОЧЕРЕДЬ (`EnqueueRunNow`), как это делает допуск прогона;
|
||||
реконсилятор — бэкстоп, если запись потерялась. Пин переписан на новое свойство, посадка
|
||||
«спавнить в запросе» падает.
|
||||
2. **Идентификатор книги тёк в INFO-логи** (`book accepted`, `book parsed`, `book rejected`) —
|
||||
против нормы `ENGINEERING_STANDARDS` §Наблюдаемость («id пользователя/книги в логи не текут») и
|
||||
против собственной дисциплины раннера, который в INFO называет ПРОГОН и никогда книгу. Снято;
|
||||
на ERROR путь книги остаётся в двух местах намеренно (терминальный отказ разбора и неудавшееся
|
||||
удаление каталога) — это открытый класс PD-139, строка дополнена. Пин —
|
||||
`TestNoBookIdentifierReachesAnInfoLine`.
|
||||
|
||||
Прогон пробы после исправлений (числа с экрана):
|
||||
|
||||
| Шаг | Наблюдение |
|
||||
|---|---|
|
||||
| печать конфигурации | 29 настроек с источником; DSN — `(set)/environment`, обе денежные — `(an amount; not logged)`; утечек значений в лог **0** |
|
||||
| `POST /v0/books` (38 символов) | **201**, `status: parsing`, `title: 蛊真人` из имени файла, `character_count: 38` (руны, не байты) |
|
||||
| статусы | `parsing → not_started`, `chapter_count: 2` из манифеста, каталог книги: `book.yaml`, `source.txt` |
|
||||
| файл больше 64 КиБ | **413** problem+json (PD-72) |
|
||||
| источник, который движок не разобрал | **`rejected`**, каталог книги удалён |
|
||||
| `POST .../runs` (2 главы) | **202**, юнит создан, движок запущен, холд **$0.06** |
|
||||
| **стоп живого прогона** | **202**, и в САМОМ ответе прогон ещё `translating` — стоп ПРИНЯТ, а не завершён (движок дописывает чекпойнт). Через свип: юнит погашен, маркер `run_…-1.exit` записан, книга и прогон → **`stopped`**, `finished_at` проставлен, расчёт: баланс **9.990000** (списано $0.01 — то, что отчитал движок), холд закрыт |
|
||||
| **резюм** | **202**; прогон → `translating`, попытка **2** с НОВЫМ юнитом, холд **$0.05** = потолок $0.06 минус потраченное |
|
||||
| резюм живого прогона | **409** · стоп несуществующего прогона — **404** |
|
||||
| метрики | `queue_depth 0` · `live_runs 1` · `oldest_open_hold_seconds 2.63` · `quarantined_attempts 0` · `tailer_lag_bytes 0` · `books_in_intake{uploading,parsing} 0/0` · `sweep_duration_seconds_count{runs} 8` · `http_requests_total{code="201",route="POST /v0/books"} 2` |
|
||||
| гигиена логов | сумм, argv движка и id книг — **0**; ERROR-строк — **0** |
|
||||
|
||||
⚠ В пробе `book.yaml` кладёт СТЕНД, играя оператора, — это и есть та половина, что стоит на
|
||||
ратификации. Заодно этим прогоняется путь «поломка деплоя ретраится, а не отвергает книгу»: первая
|
||||
попытка разбора отказывает как `not_configured`, следующий свип разбирает книгу уже с конфигурацией.
|
||||
|
||||
### Батарея, посадки и счёт
|
||||
|
||||
- `make check` с живым PostgreSQL 18.4 под `-race`: **все пакеты зелёные, кроме `internal/runner`**
|
||||
(три systemd-теста, PD-178 — разбор ниже), линтер **0 issues**.
|
||||
- Скипы: `go test ./... -count=1 -v | grep -c -- '--- SKIP'` = **0**.
|
||||
- `make vuln`: **No vulnerabilities found**.
|
||||
- Тесты `^func Test` исполнением: **261 → 340**, добавленных **79**, удалённых **0**
|
||||
(`comm -23` списка HEAD и списка дерева — пусто).
|
||||
- **Посадки: 74 прогона в КОПИИ зоны вне репозитория** (скрипт восстанавливает копию после каждой;
|
||||
списки — `mut1…mut6.json` рабочего каталога сессии). Итог: **65 пойманы своим кругом, 7 пережили,
|
||||
2 не применились по образцу — и все девять разобраны, а не подчищены**:
|
||||
- ДВЕ пережили по-настоящему, и обе были дефектами ТЕСТА, а не кода: «брошенная загрузка чистится
|
||||
на отменённом контексте» (тест не отменял контекст, то есть проверял не то свойство) и «отмена
|
||||
загрузки роняет ревизию» (тест сравнивал с числом ДО загрузки вместо того, которое клиент мог
|
||||
увидеть ВО ВРЕМЯ неё). Оба теста усилены — первый отменяет контекст ровно так, как ушедший
|
||||
клиент, второй читает библиотеку из середины аплоада, — и пере-посадки ПОЙМАНЫ;
|
||||
- три пережили как СЛАБЫЕ мутации: `_ = unit` не меняет порядка вызовов; переименование ветки
|
||||
`awaiting_bank` эквивалентно, потому что обе ветки отвечают отказом; «заявка разбора отдаётся
|
||||
назад» не моделировала снятия заявки. Все три переписаны в настоящие (перестановка вызовов,
|
||||
«пустить `awaiting_bank` в резюмируемые», снятие условия свежести) и ПОЙМАНЫ;
|
||||
- одна пережила на УСТАРЕВШЕЙ копии (посадка на `Unwrap` метрик гонялась до того, как телеметрия
|
||||
была вписана в цепочку теста дедлайна) — пере-посадка ПОЙМАНА;
|
||||
- две не применились: образец встречался дважды либо расходился пробелами; переписаны и ПОЙМАНЫ.
|
||||
⚠ Чего эта цифра НЕ значит (урок PD-142/PD-151): поимённого соответствия «каждый пин — своя
|
||||
посадка» нет и не заявляется. 62 посадки против 73 новых тестов покрывают несущие свойства —
|
||||
деньги, гонки снапшота, переходы статусов, потолки, редакцию логов, гарды файловой системы; не
|
||||
покрыты вспомогательные, где тест утверждает форму ответа или таблицу значений.
|
||||
Несущие посадки: потолок тела возвращается к дефолту · дедлайн загрузки не расширяется · `Unwrap`
|
||||
снят · стоп классифицируется по коду выхода · гард гонки «стоп vs финиш» снят · стопнутый прогон
|
||||
рестартится · рестарт стирает свежий интент · закрытие неспавненного стопа не перепроверяет юнит ·
|
||||
право на спавн выдаётся завершённому прогону · спавн игнорирует интент · резюм берёт второй холд ·
|
||||
резюм даёт полный потолок · резюм спавнит в запросе · резюм судит деплой раньше владения · конфликт
|
||||
второго живого прогона снова 500 · прогон стартует на книге вне интейка · книга, отклонённая
|
||||
движком, сохраняет файл · поломка деплоя считается виной книги · бюджет попыток снят · заявка
|
||||
разбора не compare-and-set и не спейсит ретраи · свип сносит каталог живой книги · новая книга
|
||||
входит с ревизией 0 · отмена загрузки роняет ревизию · пол ревизии игнорируется · ревизия прогона
|
||||
берётся из своей колонки (два места) · корневой гард каталога принимает любой путь · символы
|
||||
считаются байтами · строка книги пишется после тела · сумма и секрет печатаются · лейбл метрики —
|
||||
путь вместо маршрута · настройка читается без записи · свип без бюджета на прогон · вердикт
|
||||
остановленной попытки переписывается резюмом.
|
||||
|
||||
### ⚠ Оспаривание посылки: батарея зоны на этом стенде не может быть зелёной (PD-178)
|
||||
|
||||
Промт требует «0 FAIL». Три теста `internal/runner` падают, и это свойство СТЕНДА, а не пака: на
|
||||
хосте нет пользовательского менеджера systemd (`/run/systemd/system` есть, PID 1 — systemd **259**,
|
||||
но `user@1000.service` не поднят, `Linger=no`, sudo нет; попытка поднять менеджер руками выходит
|
||||
кодом 1 без вывода). Проверено, что это не регрессия: `git archive HEAD platform`, распакованный в
|
||||
копию вне репозитория, даёт те же три падения с тем же текстом «Failed to connect to user scope bus».
|
||||
Форма зоны для внешнего предусловия существует и она другая — БД-тесты СКИПАЮТСЯ вслух по
|
||||
`TM_PLATFORM_TEST_DSN`, и `make check` называет скипы поимённо. Приводить одно к другому сессия не
|
||||
стала: правка теста ради зелени запрещена (D39.121), решение — оркестратора.
|
||||
|
||||
⚠ Отсюда же ограничение живой пробы, названное прямо: **systemd-клеймы P4 этим паком НЕ пере-мерены**
|
||||
(cgroup прогона, `--collect`, поведение маркера на трёх исходах). Проба подменяет systemd процессом,
|
||||
моделируя ровно ту семантику, от которой зависит стоп: юнит переживает создателя, `ExecStopPost`
|
||||
отрабатывает с `SERVICE_RESULT`/`EXIT_CODE`/`EXIT_STATUS`, `stop` = SIGTERM главному процессу.
|
||||
⚠ Стенд с P4 при этом изменился: там был systemd 255, здесь 259.
|
||||
|
||||
### Регистр: закрыто, сужено, заведено
|
||||
|
||||
- **Закрыто (4):** PD-72 (потолок тела приехал с маршрутом и своим тестом) · PD-140 (ручки стопа и
|
||||
резюма) · PD-114 (печать эффективной конфигурации) · PD-169 (бюджет свипа на прогон плюс метрика).
|
||||
- **Сужено, но открыто (4 + 1):** PD-152 (пользовательский стоп закрыт, ребут — за движком) ·
|
||||
PD-115 (ось наблюдаемости получила эталон; ops и конфигурация — нет) · PD-122 (добавление книги
|
||||
двигает ревизию области; удаление — по-прежнему нет) · PD-162 (терминальное состояние появилось у
|
||||
ИНТЕЙКА; клин живого прогона на удалённом каталоге не тронут) · PD-139 дополнена вторым местом.
|
||||
- **Заведено (14, из них 4 закрыты тем же паком):** PD-172 «файл последним» не записано в контракте · PD-173 у `rejected` нет
|
||||
причины на проводе · PD-174 `POST /books` отвечает 404 там, где спека кода не даёт (и резюм —
|
||||
503) · PD-175 квоты интейка нет · PD-176 `requestTooLarge` не доходит через обёртки ·
|
||||
PD-177 `character_count` для не-UTF-8 — оценка · PD-178 systemd-тесты падают, а не скипаются ·
|
||||
PD-179 `/metrics` без аутентификации (принятый риск) · PD-180 201 несёт `parsing`, а не
|
||||
`uploading`, плюс отказы интейка вне спеки · PD-181…PD-184 (кросс-семейное ревью, закрыты
|
||||
здесь же) · PD-185 `finalizing` без писателя.
|
||||
- Регистр: **185 строк** (`python3 docs/scripts/counts.py`): 132 закрыто · 1 ратификацией · 4 риском ·
|
||||
48 открыто (1 major — PD-113, прежний).
|
||||
- Форма файла по пингу №16: 11 секций, построчная форма `| PD-N | … |` сохранена, `--check` зелёный.
|
||||
Проверено, что ни одна строка не потеряна: список ID до и после перекладки сортированно совпадает
|
||||
(`diff <(grep -o '^| PD-[0-9]* |' до) <(… после)` — пусто).
|
||||
|
||||
### Вопросы владельцу контракта (через оркестратора)
|
||||
|
||||
1. **Порядок частей формы.** `POST /books` требует, чтобы `file` шёл ПОСЛЕДНИМ (PD-172). Это
|
||||
свойство потокового приёма, а не прихоть: строка книги пишется до тела, а языки в ней
|
||||
обязательны. Тем же правилом специфицирована браузерная загрузка S3. Спека молчит — вносить?
|
||||
2. **Причина отказа.** `rejected` без причины на проводе (PD-173) не даёт экрану различить «файл не
|
||||
тот» и «наш движок был недоступен», а это разные советы пользователю. Поле заводить не право
|
||||
зоны; в БД причина есть.
|
||||
3. **Инстанс без интейка** отвечает на `POST /books` охраняемым 404 (PD-174) — по форме зоны для
|
||||
несмонтированных маршрутов, но кода этого спека не перечисляет (класс PD-112). Там же: резюм на
|
||||
деплое без маркер-команды отвечает 503, а 0.2.1 ратифицировала 503 только для СТАРТА прогона.
|
||||
4. **«Responds immediately; the book enters `uploading`»** недостижимо как класс (PD-180): по HTTP
|
||||
ответ не может уйти раньше, чем прочитано тело, поэтому 201 несёт `parsing`. Плюс отказы интейка,
|
||||
которых спека не описывает вовсе: число частей формы, длина текстового поля, порог 413 и обрыв
|
||||
соединения по дедлайну маршрута.
|
||||
|
||||
### Кросс-семейное ревью (D39.120 п.1а): что нашла ДРУГАЯ модель после трёх своих
|
||||
|
||||
⚠ **Сначала о разрыве, который это закрывает.** Три ревьюера выше запускались без указания модели и
|
||||
унаследовали модель сессии — то есть все четверо (автор и трое) были одной семьи. D39.120 п.1(а)
|
||||
просит в панели ≥1 опровергателя ДРУГОЙ моделью на оценочных линзах; механические линзы (сверка
|
||||
провода, посадки, батарея) от смены модели не выигрывают, а «деньги и гонки» — линза именно
|
||||
оценочная. Пробел назван владельцем, не найден сессией. Запущены два ревьюера **Fable 5** (слепой
|
||||
поиск по диффу и атака на денежные инварианты), обоим предписано воспроизводить исполнением.
|
||||
⚠ Оговорка из той же ноты, п.1(в): Fable — та же семья Anthropic, поэтому семейный слепой участок
|
||||
этим НЕ снимается; первичны пере-ран и исполнение, а внешняя калибровка — не право сессии.
|
||||
|
||||
**Результат: другая модель нашла 5 дефектов, которые три однофамильца пропустили, и два из них —
|
||||
уничтожение пользовательских данных.** Все воспроизведены исполнением, все закрыты, каждый с пином и
|
||||
посадкой.
|
||||
|
||||
| Находка | Чем это было | Как закрыто |
|
||||
|---|---|---|
|
||||
| **Грация клейма (10 мин) КОРОЧЕ таймаута задания очереди (15 мин)** — обе константы поставил этот же пак, в разных файлах, не сверив | В окне 10–15 минут свип отдаёт клейм второму парсеру: два `tmctl manifest` на одной директории, проигравший умирает на эксклюзивном локе движка с exit 1 — а exit 1 у движка означает «источник не разобрать». Книга отклонялась ТЕРМИНАЛЬНО, исходник пользователя удалялся. Воспроизведено тестом ревьюера | Грация написана как `jobs.JobTimeout + 5m`, пара утверждается тестом; терминальная запись возможна только пока клейм ЕЩЁ НАШ (`parse_started_at = <наш>`). PD-183 |
|
||||
| **Один exit 1 движка = удаление загрузки** | Движок маппит на exit 1 ВСЕ свои отказы (его собственный комментарий): полный диск, лок от ручного `tmctl` оператора, украденный клейм. Первый же такой отказ уничтожал файл безвозвратно — ре-парса и удаления в контракте нет | Отказ ИСТОЧНИКА идёт через тот же бюджет попыток, что и поломка деплоя; удаление — только на терминальном шаге. Пин расширен: первый отказ обязан оставить файл на месте |
|
||||
| **`not_configured` был терминальным** — а конфигурацию не пишет никто, пока развилка не ратифицирована | Значит на СЕГОДНЯШНЕМ деплое любая загрузка умирала бы в течение часа, с удалением, из-за пробела, которого пользователь не видит, а оператор закрывает одним файлом | `not_configured` НЕ становится терминальным никогда: книга ждёт в `parsing` с целым исходником, и это видно метрикой интейка. ⚠ Тест на этом свойстве ПЕРЕВЁРНУТ намеренно — он пинил «пять попыток и отказ», и это и было ошибкой |
|
||||
| **Устаревший снапшот до-финиширует РЕЗЮМИРОВАННЫЙ прогон** (деньги) | Маркер завершённой попытки остаётся на диске; проход со старым снапшотом закрывал вернувшийся к жизни прогон, и холд попытки 2 выпадал из ОБОИХ списков — ни живых прогонов, ни несведённых. Достижимо стало ровно с появлением резюма | `FinishRun` пишет, только если закрываемая попытка ещё живая, и возвращает признак «закрыл». PD-181 |
|
||||
| **Стоп закрывал прогон под ЖИВЫМ движком** (деньги) | «Заявка отдана назад» ≠ «процесса нет»: `systemd-run`, убитый после запроса, оставляет движок работать — зона это уже знает и ради этого хранит базовую линию, но стоп-путь читал пустое имя юнита как «его не было» | Ненулевая базовая линия при пустом имени = надгробие спавна; имя юнита детерминировано, платформа спрашивает systemd `Alive` и живому юниту шлёт стоп. PD-182 |
|
||||
| Заявка на спавн игнорировала висящий интент стопа (мой комментарий утверждал обратное) | Стоп, попавший в окно, пока воркер читает отсчёт книги, не мешал старту движка | `stop_requested_at is null` — в том же CAS; комментарий приведён к коду |
|
||||
| `TM_PLATFORM_UPLOAD_DEADLINE` мог пережить грацию свипа | Оператор, поднявший дедлайн выше часа, получал удаление строки и каталога из-под ещё идущей загрузки | Пара проверяется на буте |
|
||||
| Полностью принятая загрузка выбрасывалась, если клиент ушёл до 201 | Телефон уснул — файл и минуты линии в корзину | `StartParsing` идёт на контексте, переживающем запрос |
|
||||
| Лейбл `method` — токен вызывающего | Для маршрута кардинальность закрыта, для метода — нет | Закрытый список, прочее — `(other)`. PD-184 |
|
||||
| `runner.Manifest` подвисал/убивал движок на превышении потолка | Правку я собирался внести раньше, и она НЕ доехала (скрипт упал на другой ассерции) — поймано только здесь | Остаток дренится, превышение — честная ошибка |
|
||||
| Статус `finalizing` без писателя | Слово контракта, которого не пишет ни один путь — тот же класс, что интейк-статусы до этого пака | Строкой PD-185, не кодом: писателя выдумывать не право зоны |
|
||||
|
||||
**Посадок этого круга: 12, поймано 11.** Пережила одна и она названа, а не подчищена: «отказ сносит
|
||||
каталог ПОСЛЕ строки, а не до» — свойство про ПОРЯДОК двух побочных эффектов, наблюдаемое только
|
||||
через смерть процесса между ними; конечное состояние в обоих порядках одинаково, и юнит-тестом это
|
||||
не ловится. Порядок выбран самоизлечивающийся (сначала каталог, потом строка) и обоснован в коде.
|
||||
|
||||
⚠ **Что этот круг говорит о предыдущем.** Три однофамильных ревьюера прошли по тем же файлам и не
|
||||
увидели ни рассинхрона двух констант, поставленных в одном паке, ни того, что «exit 1 = вина файла»
|
||||
уничтожает данные на первом же транзиенте. Это ровно та коррелированная слепота, ради которой
|
||||
D39.120 просит другую модель, — и здесь она подтверждена эмпирически, а не принята на веру.
|
||||
|
||||
### Записка-план после ревью: ID → статус → улика (D39.121)
|
||||
|
||||
| ID | Что | Статус | Улика |
|
||||
|---|---|---|---|
|
||||
| R1 | Право на спавн выдаётся законченному прогону (деньги, критично) | ЗАКРЫТО | гард в `RecordSpawn`; `TestTheQueueWorkerDoesNotStartARunThatWasAlreadyStopped`; посадка «снять гард» падает |
|
||||
| R2 | Рестарт стирает свежий интент стопа (деньги) | ЗАКРЫТО | `RestartRun` берёт строку прогона `for update` и отдаёт `ErrStopRequested`; `TestAStopPressedWhileTheReconcilerRestartsIsNotLost`; посадка падает |
|
||||
| R3 | Закрытие неспавненного стопа роняет только что стартовавший движок (деньги) | ЗАКРЫТО | `FinishUnspawnedStop` с перепроверкой под замком; `TestARunSpawnedWhileTheSweepWasClosingItIsNotAbandoned`; посадка падает |
|
||||
| R4 | Спавн не читает интент стопа (деньги) | ЗАКРЫТО | отказ в `spawnAttempt`; тот же пин, что у R1; посадка падает |
|
||||
| R5 | Интейк и телеметрия голодают на общем дедлайне | ЗАКРЫТО | у каждого прохода свой бюджет (`cmd/tmplatformd/runner.go`); прогон пробы: `sweep_duration_seconds_count{intake}` растёт вместе с `{runs}` |
|
||||
| R6 | Свип сносит каталог книги, которую не удалил | ЗАКРЫТО | `abandon` возвращается на ошибке удаления; `TestTheSweepNeverRemovesTheSourceOfABookItCouldNotDelete`; посадка падает |
|
||||
| R7 | Отмена загрузки роняет ревизию библиотеки назад | ЗАКРЫТО | пол `users.library_revision`; `TestCancellingAnUploadDoesNotWindTheLibraryBack`; две посадки падают |
|
||||
| R8 | Бюджет разбора считает тики, а не время | ЗАКРЫТО | заявка держится; пин расширен проверкой «до грации не делается ничего»; посадка падает |
|
||||
| R9 | Таймаут задания очереди — молчаливая минута River | ЗАКРЫТО | `JobTimeout` выбран явно; `jobs_test.go` пинит вид и политику заданий |
|
||||
| R10 | `runner.Manifest` подвисает на превышении потолка | ЗАКРЫТО | остаток дренится, превышение — ошибка; `TestAManifestThatIsNotJSONIsAnError` и соседние гоняют реальный процесс |
|
||||
| R11 | Ревизия прогона на проводе — из его колонки | ЗАКРЫТО | стор отдаёт книжную на всех путях; `pgstore.TestEveryRunTheStoreHandsOutCarriesItsBooksRevision`, `runs.TestEveryRunCarryingAnswerUsesTheBooksRevision`; посадка падает |
|
||||
| R12 | Резюм судит деплой раньше владения | ЗАКРЫТО | порядок переставлен; `TestOwnershipIsJudgedBeforeTheDeploymentsHealth` |
|
||||
| R13 | Резюм второго живого прогона книги → 500 | ЗАКРЫТО | `ErrRunInFlight` → 409; `TestResumeIsRefusedWhenTheBookHasAnotherLiveRun` |
|
||||
| R14 | Два резюма подряд → 404 проигравшему | ЗАКРЫТО | проигравший перечитывает прогон; `TestTwoResumesOfOneRunTakeOneHoldAndBothAnswer` |
|
||||
| R15 | Каталог-сирота, если строка исчезла между телом и `parsing` | ЗАКРЫТО | `Accept` убирает за собой и на этой ветке |
|
||||
| R16 | 201 несёт `parsing`, а не `uploading` | СТРОКОЙ, не кодом | PD-180: по HTTP ответ не может уйти раньше тела; `uploading` наблюдаем параллельным чтением, пин есть |
|
||||
| R17 | Отказы интейка, которых спека не описывает (части формы, длина поля, порог 413, обрыв по дедлайну) | СТРОКОЙ | PD-180, вопросом владельцу контракта одним пакетом с PD-172/PD-174 |
|
||||
| R18 | 503 на резюме у несконфигурированного деплоя | СТРОКОЙ | PD-174 расширен: класс тот же, что у ратифицированного 503 на старте |
|
||||
| R19 | Любая загрузка сегодня приходит к `rejected`, и выхода из `rejected` нет | СТРОКОЙ | PD-175 расширен; корень — развилка `book.yaml`, вопрос выше |
|
||||
| R20 | Пути книг в ERROR-логах интейка | СТРОКОЙ | PD-139 дополнена; на INFO/WARN идентификаторов нет, и это запинено |
|
||||
| R21 | `requestTooLarge` не доходит через обёртки | СТРОКОЙ | PD-176: 413 отдаётся штатно, соединение закрывается обычным путём под `ReadTimeout` |
|
||||
| R22 | Порядок блокировок в `RestartRun` (строка прогона бралась последней) | ЗАКРЫТО попутно | R2 берёт её сразу после книги — документированный порядок восстановлен |
|
||||
| F1 | Грация клейма короче таймаута задания: свип крадёт парс, книга терминально отклоняется с удалением файла | ЗАКРЫТО | `claimGrace = jobs.JobTimeout + 5m`; `books.TestTheClaimGraceOutlivesTheQueuesJobTimeout`; посадка падает |
|
||||
| F2 | Один exit 1 движка уничтожал загрузку | ЗАКРЫТО | отказ источника идёт через бюджет; удаление только на терминальном шаге; посадка «отклонять с первого раза» падает |
|
||||
| F3 | `not_configured` терминален — то есть каждая сегодняшняя загрузка обречена | ЗАКРЫТО (тест перевёрнут намеренно) | `TestABookWithNoEngineConfigurationWaitsRatherThanDies`; посадка «отклонять после бюджета» падает |
|
||||
| F4 | Устаревший снапшот до-финиширует резюмированный прогон, холд вне списков | ЗАКРЫТО | гард живой попытки в `FinishRun`; `TestAStaleSweepDoesNotReFinishAResumedRunFromTheOldAttemptsMarker`; посадка падает |
|
||||
| F5 | Стоп закрывает прогон под живым движком после отданной заявки | ЗАКРЫТО | `Alive` по детерминированному имени юнита; `TestAStopDoesNotCloseARunWhoseGivenBackClaimLeftAnEngineRunning`; посадка падает |
|
||||
| F6 | Заявка на спавн игнорирует интент стопа | ЗАКРЫТО | условие в CAS; `TestAStopCommittedWhileTheWorkerReadsTheMeterStopsTheSpawn`; посадка падает |
|
||||
| F7 | Дедлайн загрузки мог пережить грацию свипа | ЗАКРЫТО | проверка пары на буте; `config.TestAnUploadDeadlineLongerThanTheSweepsGraceIsRefused`; посадка падает |
|
||||
| F8 | Принятая загрузка выбрасывалась, если клиент ушёл до 201 | ЗАКРЫТО | контекст переживает запрос; `TestAnUploadSurvivesAClientThatHangsUpAfterTheLastByte`; посадка падает |
|
||||
| F9 | Лейбл `method` — токен вызывающего | ЗАКРЫТО | закрытый список; посадка падает |
|
||||
| F10 | `runner.Manifest` подвисает на превышении потолка | ЗАКРЫТО | дренаж остатка + ошибка; правка была задумана раньше и НЕ доехала — поймана только кросс-семейным кругом |
|
||||
| F11 | Порядок «каталог до строки» в отказе | ⚠ ПЕРЕСМОТРЕНО дофиксом → FP5-3 | диспозиция «порядок самоизлечивающийся» стояла на посылке, которую тот же пак отменил (`not_configured` стал НЕтерминальным), поэтому крэш-окно оставляло книгу в `parsing` навсегда; закрыто отдельной причиной `ErrDirectoryGone` (PD-187), пин и посадка есть |
|
||||
| F12 | `finalizing` — статус без писателя | СТРОКОЙ | PD-185 |
|
||||
|
||||
### Что НЕ делалось (по промту)
|
||||
|
||||
Эскроу/`uncertain`/`closing` (строка 136) · формула потолка PD-158 и SpendBound PD-159 · ставка
|
||||
$0.03 · эмиттер-сторона движка и словарь `events.go` · П-2 · платёжный провайдер · удаление аккаунта
|
||||
(PD-107) · счётчик ревизии ОБЛАСТИ целиком (PD-122 сужена, не закрыта) · правка контракта (четыре
|
||||
вопроса выше вместо неё).
|
||||
|
||||
## Ратификация приёмкой P4 (оркестратор №15, 09.08)
|
||||
|
||||
**Вердикт: P4 + дофикс + V2 ПРИНЯТЫ и ЗАЛЕНДЕНЫ — D39.123, код `d29e30c` (53 файла, +9867/−170,
|
||||
|
|
|
|||
|
|
@ -2,10 +2,21 @@ module textmachine/platform
|
|||
|
||||
go 1.26.4
|
||||
|
||||
// The LANGUAGE floor stays the engine's (one stand builds both modules). The TOOLCHAIN is pinned
|
||||
// higher and separately, because it is what supplies the standard library this network-facing module
|
||||
// links: 1.26.5 carries five advisories govulncheck traces into paths this service calls
|
||||
// (GO-2026-6218/6090/6089/6088/5972), all fixed in 1.26.6. With the default GOTOOLCHAIN=auto a host
|
||||
// on an older toolchain fetches this one instead of building a vulnerable binary; with
|
||||
// GOTOOLCHAIN=local it stops with an error naming the version. Both are the answer we want, and
|
||||
// neither depends on anybody reading a Makefile. `make version-check` is the same floor for the
|
||||
// battery, which runs where GOTOOLCHAIN may be pinned to local.
|
||||
toolchain go1.26.6
|
||||
|
||||
require (
|
||||
github.com/coreos/go-oidc/v3 v3.20.0
|
||||
github.com/jackc/pgx/v5 v5.10.0
|
||||
github.com/pressly/goose/v3 v3.27.3
|
||||
github.com/prometheus/client_golang v1.24.1
|
||||
github.com/riverqueue/river v0.42.0
|
||||
github.com/riverqueue/river/riverdriver/riverpgxv5 v0.42.0
|
||||
golang.org/x/oauth2 v0.36.0
|
||||
|
|
@ -13,13 +24,19 @@ require (
|
|||
)
|
||||
|
||||
require (
|
||||
github.com/beorn7/perks v1.0.1 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/davecgh/go-spew v1.1.1 // indirect
|
||||
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
github.com/mfridman/interpolate v0.0.2 // indirect
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
||||
github.com/pmezard/go-difflib v1.0.0 // indirect
|
||||
github.com/prometheus/client_model v0.6.2 // indirect
|
||||
github.com/prometheus/common v0.70.1 // indirect
|
||||
github.com/prometheus/procfs v0.21.1 // indirect
|
||||
github.com/riverqueue/river/riverdriver v0.42.0 // indirect
|
||||
github.com/riverqueue/river/rivershared v0.42.0 // indirect
|
||||
github.com/riverqueue/river/rivertype v0.42.0 // indirect
|
||||
|
|
@ -32,6 +49,8 @@ require (
|
|||
go.uber.org/goleak v1.3.0 // indirect
|
||||
go.uber.org/multierr v1.11.0 // indirect
|
||||
golang.org/x/sync v0.22.0 // indirect
|
||||
golang.org/x/sys v0.47.0 // indirect
|
||||
golang.org/x/text v0.40.0 // indirect
|
||||
google.golang.org/protobuf v1.36.11 // indirect
|
||||
gopkg.in/yaml.v3 v3.0.1 // indirect
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,3 +1,7 @@
|
|||
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
|
||||
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/coreos/go-oidc/v3 v3.20.0 h1:EtE0WIBHk03N+DqGkY4+UONzzZHk7amKt6IyNd7OsZE=
|
||||
github.com/coreos/go-oidc/v3 v3.20.0/go.mod h1:DYCf24+ncYi+XkIH97GY1+dqoRlbaSI26KVTCI9SrY4=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
|
|
@ -7,6 +11,8 @@ github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkp
|
|||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/go-jose/go-jose/v4 v4.1.4 h1:moDMcTHmvE6Groj34emNPLs/qtYXRVcd6S7NHbHz3kA=
|
||||
github.com/go-jose/go-jose/v4 v4.1.4/go.mod h1:x4oUasVrzR7071A4TnHLGSPpNOm2a21K9Kf04k1rs08=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/jackc/pgerrcode v0.0.0-20240316143900-6e2875d9b438 h1:Dj0L5fhJ9F82ZJyVOmBx6msDp/kfd1t9GRfny/mfJA0=
|
||||
|
|
@ -19,20 +25,34 @@ github.com/jackc/pgx/v5 v5.10.0 h1:VhSvgU2jSli8o3AqIEOTJr7rZwAEUVo4E4XhR94Zfr0=
|
|||
github.com/jackc/pgx/v5 v5.10.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
||||
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0=
|
||||
github.com/kr/pretty v0.3.0/go.mod h1:640gp4NfQd8pI5XOwp5fnNeVWj67G7CFk/SaSQn7NBk=
|
||||
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
|
||||
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
|
||||
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
|
||||
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
|
||||
github.com/mattn/go-isatty v0.0.23 h1:cYwCQTQf3HB6xUC+BtyCLZNr7IzbOmoZbmssVNzSyiQ=
|
||||
github.com/mattn/go-isatty v0.0.23/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
|
||||
github.com/mfridman/interpolate v0.0.2 h1:pnuTK7MQIxxFz1Gr+rjSIx9u7qVjf5VOoM/u6BbAxPY=
|
||||
github.com/mfridman/interpolate v0.0.2/go.mod h1:p+7uk6oE07mpE/Ik1b8EckO0O4ZXiGAfshKBWLUM9Xg=
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
|
||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
|
||||
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
|
||||
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/pressly/goose/v3 v3.27.3 h1:pIglVHjw99r4e/hDHHwbl9vfOsDMqUokfkXo6+n/RxA=
|
||||
github.com/pressly/goose/v3 v3.27.3/go.mod h1:Dag+xpV6o20HR2LFY1j0q6MDwc3f7vPUFDA77R+0yGY=
|
||||
github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU=
|
||||
github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE=
|
||||
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
|
||||
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
|
||||
github.com/prometheus/common v0.70.1 h1:1HvjP4D5oL3t8RsPlwxA9onvvStjtIHYE5XuuwOi/PY=
|
||||
github.com/prometheus/common v0.70.1/go.mod h1:VdFUQDMZK3VLkurFUVhia6uys/0suUp86TJz5qbJRhc=
|
||||
github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI=
|
||||
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
||||
github.com/riverqueue/river v0.42.0 h1:WDx0F0loJg41XvkHVyQVr9PUN2p9SPLr2Tx7zs5ONlY=
|
||||
|
|
@ -71,6 +91,8 @@ go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
|||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
|
||||
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
|
||||
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
|
||||
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
|
||||
golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs=
|
||||
golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q=
|
||||
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
|
||||
|
|
@ -81,6 +103,8 @@ golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
|
|||
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
|
||||
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
|
||||
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
|
||||
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
|
||||
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
|
||||
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
|
||||
|
|
|
|||
386
platform/internal/books/books.go
Normal file
386
platform/internal/books/books.go
Normal file
|
|
@ -0,0 +1,386 @@
|
|||
// Package books owns a book's INTAKE: receiving the file the user uploads, putting it where the
|
||||
// engine will look for it, and turning it into a chapter tree.
|
||||
//
|
||||
// The shape mirrors the run lifecycle deliberately (package runs): the request only gets the book as
|
||||
// far as "the bytes are here", and everything after that is a step some later sweep can finish. A
|
||||
// parse is one $0 call of the engine that takes seconds on a large book, and holding a request open
|
||||
// for it — or losing the book when the process doing it is restarted — are the two failures this
|
||||
// split exists to avoid.
|
||||
package books
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"textmachine/platform/internal/ingest"
|
||||
"textmachine/platform/internal/pgstore"
|
||||
)
|
||||
|
||||
// Manifester is the engine's $0 producer of a chapter tree: `tmctl manifest`. An interface so the
|
||||
// intake's decisions can be pinned without an engine binary.
|
||||
type Manifester interface {
|
||||
Manifest(ctx context.Context, binary, workdir string) (ingest.Manifest, error)
|
||||
}
|
||||
|
||||
// Enqueuer hands a book to the queue inside the caller's transaction.
|
||||
type Enqueuer interface {
|
||||
EnqueueParse(ctx context.Context, tx pgstore.Tx, bookID string) error
|
||||
}
|
||||
|
||||
// Config is what an operator chooses about intake.
|
||||
type Config struct {
|
||||
// BooksDir is the root the platform creates book directories under. Absolute, and NOT the place
|
||||
// an operator keeps hand-made books: everything below it is written and, on a rejected intake,
|
||||
// removed by this service.
|
||||
BooksDir string
|
||||
// EngineBinary is the versioned tmctl path used to parse. Intake is not mounted without it: a
|
||||
// deployment that cannot parse would take uploads it can only reject.
|
||||
EngineBinary string
|
||||
}
|
||||
|
||||
// Service is the intake.
|
||||
type Service struct {
|
||||
Store *pgstore.Store
|
||||
Engine Manifester
|
||||
Queue Enqueuer
|
||||
Cfg Config
|
||||
Log *slog.Logger
|
||||
// Now is injectable so the graces below are testable without sleeping.
|
||||
Now func() time.Time
|
||||
}
|
||||
|
||||
func (s *Service) now() time.Time {
|
||||
if s.Now != nil {
|
||||
return s.Now()
|
||||
}
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
func (s *Service) log() *slog.Logger {
|
||||
if s.Log == nil {
|
||||
return slog.New(slog.DiscardHandler)
|
||||
}
|
||||
return s.Log
|
||||
}
|
||||
|
||||
// ErrBadIntake is a request this route cannot make a book out of. It is the contract's 400.
|
||||
var ErrBadIntake = errors.New("books: the intake form is not usable")
|
||||
|
||||
// SourceName is the file name the intake writes a book's source under, extension aside. The name is
|
||||
// FIXED and predictable because the engine finds the source through `source_file:` in the book's own
|
||||
// config, which somebody else writes (see the provisioning seam in parse.go).
|
||||
const SourceName = "source"
|
||||
|
||||
// maxTitle bounds the title taken from the uploaded file's name. The library lists it, and the
|
||||
// client-supplied name is otherwise the one unbounded string on that screen.
|
||||
const maxTitle = 200
|
||||
|
||||
// langCode is the contract's LangCode: a code, never a name (§LangCode). Validated here as well as
|
||||
// in the read model because this is where a value from a browser enters.
|
||||
var langCode = regexp.MustCompile(`^[a-z]{2,3}(-[A-Za-z0-9]{2,8})*$`)
|
||||
|
||||
// Intake is one accepted call of POST /books.
|
||||
type Intake struct {
|
||||
UserID string
|
||||
SourceLang string
|
||||
TargetLang string
|
||||
Genre string
|
||||
// Filename is the name the client gave the part. It is used for two things and trusted for
|
||||
// neither: a title for the library and the extension that tells the engine which reader to use.
|
||||
Filename string
|
||||
// File is the body of the upload, already bounded by the route's own limit. It is streamed to
|
||||
// disk and never held in memory.
|
||||
File io.Reader
|
||||
}
|
||||
|
||||
// Accept receives one book.
|
||||
//
|
||||
// The order — row, then bytes — is what makes `uploading` a state anything can observe: a second tab
|
||||
// listing the library while a 60 MB file is still on the wire sees the book with its languages and
|
||||
// without its size. It also makes an abandoned upload FINDABLE, which is the half that matters
|
||||
// operationally: the row is the only record that a directory under BooksDir belongs to anyone.
|
||||
func (s *Service) Accept(ctx context.Context, in Intake) (pgstore.Book, error) {
|
||||
if !langCode.MatchString(in.SourceLang) || !langCode.MatchString(in.TargetLang) {
|
||||
return pgstore.Book{}, fmt.Errorf("%w: the languages must be codes", ErrBadIntake)
|
||||
}
|
||||
if s.Cfg.BooksDir == "" {
|
||||
return pgstore.Book{}, errors.New("books: no books directory is configured")
|
||||
}
|
||||
id := pgstore.NewBookID()
|
||||
dir := filepath.Join(s.Cfg.BooksDir, id)
|
||||
if err := os.MkdirAll(dir, 0o750); err != nil {
|
||||
return pgstore.Book{}, fmt.Errorf("books: create book directory: %w", err)
|
||||
}
|
||||
// The first upload is what marks this storage as ours — see storageIsThere. Written here and
|
||||
// nowhere else, and deliberately NOT at boot: a marker the boot recreates says "the storage is
|
||||
// here" about a directory the boot itself just made.
|
||||
if err := s.markStorage(); err != nil {
|
||||
s.removeDir(dir)
|
||||
return pgstore.Book{}, err
|
||||
}
|
||||
// The created row is not kept: what the caller gets back is the row as it stands AFTER the file
|
||||
// landed, and between the two the status has moved from `uploading` to `parsing`.
|
||||
_, err := s.Store.CreateUpload(ctx, id, pgstore.NewUpload{
|
||||
OwnerID: in.UserID,
|
||||
Title: titleFrom(in.Filename),
|
||||
SourceLang: in.SourceLang,
|
||||
TargetLang: in.TargetLang,
|
||||
Genre: in.Genre,
|
||||
Workdir: dir,
|
||||
Now: s.now(),
|
||||
})
|
||||
if err != nil {
|
||||
s.removeDir(dir)
|
||||
return pgstore.Book{}, err
|
||||
}
|
||||
characters, err := s.receive(filepath.Join(dir, SourceName+extensionOf(in.Filename)), in.File)
|
||||
if err != nil {
|
||||
// The upload did not finish, so there is nothing to parse and nothing to keep. The row is
|
||||
// DELETED rather than rejected: `rejected` means the file could not be parsed (contract
|
||||
// §BookStatus), and there is no delete handle in the contract for the user to clear a row an
|
||||
// abandoned upload would otherwise leave in their library forever.
|
||||
//
|
||||
// On its own context: the ordinary cause of getting here is the client going away, and the
|
||||
// request's context is already cancelled by then.
|
||||
c, cancel := writeCtx(ctx)
|
||||
defer cancel()
|
||||
s.abandon(c, id, dir)
|
||||
return pgstore.Book{}, err
|
||||
}
|
||||
// On a context that outlives the request: every byte is in, and a client that hung up while
|
||||
// waiting for the 201 must not cost the upload it already finished. The ROW is what makes the
|
||||
// book findable, so it is written even when nobody is left to read the answer. Bounded on its own
|
||||
// (writeCtx): detached is not the same as unlimited, and a hung statement here would hold the
|
||||
// goroutine of a request that is already over.
|
||||
start, cancelStart := writeCtx(ctx)
|
||||
defer cancelStart()
|
||||
book, err := s.Store.StartParsing(start, id, characters, s.enqueue)
|
||||
if err != nil {
|
||||
// The row is gone or unreachable, and the directory holds a file nothing points at — which is
|
||||
// the one thing the row-first order exists to prevent, so it is undone here too. The ordinary
|
||||
// cause is the sweep having abandoned this upload while it was still arriving.
|
||||
c, cancel := writeCtx(ctx)
|
||||
defer cancel()
|
||||
s.abandon(c, id, dir)
|
||||
return pgstore.Book{}, err
|
||||
}
|
||||
// No book id: an INFO line must not identify a user's library (ENGINEERING_STANDARDS
|
||||
// §Наблюдаемость, the same rule that keeps raw paths out of the access log — PD-3). The request
|
||||
// id the handler carries is what ties this line to the response the user got.
|
||||
s.log().InfoContext(ctx, "book accepted", "characters", characters)
|
||||
return book, nil
|
||||
}
|
||||
|
||||
// receive streams the upload to disk and counts what went past.
|
||||
//
|
||||
// Streamed, never buffered: the route's limit is tens of megabytes and reading that into memory
|
||||
// would make one upload per concurrent request the platform's memory profile. The file is created
|
||||
// with O_EXCL so a book directory can never be written twice by two requests that somehow minted the
|
||||
// same id.
|
||||
func (s *Service) receive(path string, body io.Reader) (int64, error) {
|
||||
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o640)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("books: create source file: %w", err)
|
||||
}
|
||||
c := &counter{w: f}
|
||||
if _, err := io.Copy(c, body); err != nil {
|
||||
f.Close()
|
||||
// The error is returned as it came: the handler tells a body that outgrew the route's limit
|
||||
// from a client that went away, and both are the caller's to name (http.MaxBytesError).
|
||||
return 0, err
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
return 0, fmt.Errorf("books: close source file: %w", err)
|
||||
}
|
||||
return c.characters, nil
|
||||
}
|
||||
|
||||
// counter writes through and counts CHARACTERS, which is what the contract's Book carries — a size
|
||||
// in characters, known to the platform from intake and to nobody else (§Book.character_count).
|
||||
//
|
||||
// It counts every byte that is not a UTF-8 continuation byte, which is exactly the rune count of
|
||||
// valid UTF-8 and needs no state across the chunk boundaries io.Copy hands it. ⚠ For a source in
|
||||
// another encoding — the engine accepts GB18030 and UTF-16 and decodes them itself — the figure is
|
||||
// an approximation of the character count, not the count. Naming a size for a file whose encoding is
|
||||
// not yet known is what intake can do; the exact one would need the engine's decode, which happens a
|
||||
// step later and reports bytes.
|
||||
type counter struct {
|
||||
w io.Writer
|
||||
characters int64
|
||||
}
|
||||
|
||||
func (c *counter) Write(p []byte) (int, error) {
|
||||
n, err := c.w.Write(p)
|
||||
for _, b := range p[:n] {
|
||||
if b&0xC0 != 0x80 {
|
||||
c.characters++
|
||||
}
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
func (s *Service) enqueue(ctx context.Context, tx pgstore.Tx, bookID string) error {
|
||||
if s.Queue == nil {
|
||||
return nil // no queue configured: the sweep picks the book up on its next pass
|
||||
}
|
||||
return s.Queue.EnqueueParse(ctx, tx, bookID)
|
||||
}
|
||||
|
||||
// abandon undoes an upload that did not finish. Both halves are best effort and both are logged:
|
||||
// what must not happen is a silent leak of either the row or the directory.
|
||||
func (s *Service) abandon(ctx context.Context, id, dir string) {
|
||||
// The ROW is deleted first here, and unlike the reject path that order is forced: the directory
|
||||
// may only go once it is certain nobody owns it. The sweep decides from a snapshot, and a request
|
||||
// that finished in the meantime has moved the book to `parsing` — DeleteUpload's status guard then
|
||||
// refuses, and removing the directory anyway would delete the source of a live book under it.
|
||||
//
|
||||
// The crash window that leaves is a directory with no row, and it is named rather than closed:
|
||||
// nothing walks BooksDir looking for orphans (register row PD-175, where the retention sweep that
|
||||
// would is filed).
|
||||
if err := s.Store.DeleteUpload(ctx, id); err != nil {
|
||||
if errors.Is(err, pgstore.ErrNoBook) {
|
||||
// The ordinary race: another instance's sweep got there first, or the upload finished. Not
|
||||
// an error — an ERROR line on a routine race is noise that teaches operators to skim.
|
||||
s.log().InfoContext(ctx, "the abandoned upload was already gone; its directory is left alone")
|
||||
return
|
||||
}
|
||||
s.log().ErrorContext(ctx, "an abandoned upload was not removed; its directory is left alone", "err", err)
|
||||
return
|
||||
}
|
||||
s.removeDir(dir)
|
||||
}
|
||||
|
||||
// removeDir deletes a directory this service created, and NOTHING else.
|
||||
//
|
||||
// The guard is not ceremony: a book registered by the dev CLI carries a workdir the operator chose —
|
||||
// their own project directory, with their own source and their own project database — and no path
|
||||
// here may ever remove one. Everything under BooksDir was created by this service and holds nothing
|
||||
// the platform did not put there.
|
||||
func (s *Service) removeDir(dir string) {
|
||||
if !s.owns(dir) {
|
||||
s.log().Error("refusing to remove a directory this service did not create", "dir", dir)
|
||||
return
|
||||
}
|
||||
if err := os.RemoveAll(dir); err != nil {
|
||||
s.log().Error("book directory could not be removed", "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// StorageMarker is the file that says "this is the storage this platform has been writing books
|
||||
// into". Exported so an operator provisioning a volume by hand can put one there.
|
||||
const StorageMarker = ".tmplatform-books"
|
||||
|
||||
const storageMarkerText = `This directory holds book sources written by tmplatformd.
|
||||
Its presence is what tells the intake that the storage is mounted: without it a book whose directory
|
||||
is missing is treated as "the storage is gone" and WAITS, instead of being rejected with its source
|
||||
deleted. Do not remove it while books live here.
|
||||
`
|
||||
|
||||
// markStorage writes the marker if it is not there. Idempotent and race-free by O_EXCL: two uploads
|
||||
// arriving together are both correct, and the loser's EEXIST is the state it wanted.
|
||||
func (s *Service) markStorage() error {
|
||||
f, err := os.OpenFile(filepath.Join(s.Cfg.BooksDir, StorageMarker),
|
||||
os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o640)
|
||||
if errors.Is(err, fs.ErrExist) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("books: mark the storage: %w", err)
|
||||
}
|
||||
defer f.Close()
|
||||
if _, err := f.WriteString(storageMarkerText); err != nil {
|
||||
return fmt.Errorf("books: mark the storage: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// storageIsThere reports whether the books storage is the one this platform wrote into.
|
||||
//
|
||||
// ⚠ It asks about the MARKER and not about the directory, and the difference is the whole point. A
|
||||
// volume mounted AT BooksDir leaves an empty mountpoint behind when it is unmounted, so the directory
|
||||
// still exists and `Stat` still succeeds — which is how the first version of this guard (PD-192) let
|
||||
// an unmount reject every book in intake, with their sources deleted, exactly as if each book's own
|
||||
// directory had been removed. The boot's MkdirAll made it worse by recreating the root after a
|
||||
// restart. The marker is written by the FIRST upload and by nothing else, so neither an unmount nor a
|
||||
// fresh MkdirAll can forge it (re-check of the dofix, FP5-10).
|
||||
//
|
||||
// A host that has never taken an upload has no marker either, and answers "not there" — which is the
|
||||
// safe direction: it has no books to reject.
|
||||
func (s *Service) storageIsThere() bool {
|
||||
if s.Cfg.BooksDir == "" {
|
||||
return false
|
||||
}
|
||||
_, err := os.Stat(filepath.Join(s.Cfg.BooksDir, StorageMarker))
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func (s *Service) owns(dir string) bool {
|
||||
if s.Cfg.BooksDir == "" || dir == "" {
|
||||
return false
|
||||
}
|
||||
rel, err := filepath.Rel(s.Cfg.BooksDir, dir)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
return rel != "." && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator))
|
||||
}
|
||||
|
||||
// titleFrom is the book's name until something better exists.
|
||||
//
|
||||
// ⚠ Named limitation, not a design: `BookIntake` carries no title field (contract §BookIntake) and
|
||||
// inventing one is not this zone's right, while the engine's manifest reports counts and identity
|
||||
// and no title either. What is left is the name the user gave the file, which is a real thing they
|
||||
// chose. An empty result stays empty: a synthesized "Book 1" would be a label nobody wrote.
|
||||
func titleFrom(filename string) string {
|
||||
name := filepath.Base(filepath.FromSlash(filename))
|
||||
if i := strings.LastIndexByte(name, '.'); i > 0 {
|
||||
name = name[:i]
|
||||
}
|
||||
name = strings.Map(func(r rune) rune {
|
||||
if unicode.IsControl(r) {
|
||||
return -1
|
||||
}
|
||||
return r
|
||||
}, name)
|
||||
name = strings.TrimSpace(name)
|
||||
if name == "." || name == ".." || name == string(filepath.Separator) {
|
||||
return ""
|
||||
}
|
||||
if utf8.RuneCountInString(name) > maxTitle {
|
||||
name = string([]rune(name)[:maxTitle])
|
||||
}
|
||||
return name
|
||||
}
|
||||
|
||||
// extensionOf is the one thing about the FORMAT the platform is allowed to know: the engine
|
||||
// dispatches its reader by extension (`.epub` → the epub reader, anything else → plain text,
|
||||
// backend/internal/chunk/ingest.go), so the extension has to survive intake or an EPUB is read as
|
||||
// text.
|
||||
//
|
||||
// It is not a format allowlist, and that is deliberate: which formats exist is the engine's
|
||||
// question, and a platform that refused an extension the engine had just learned would be a second
|
||||
// place to teach. What it does refuse is a name that is not an extension — the value goes into a
|
||||
// path, so anything but lowercase alphanumerics is dropped and the source becomes plain text.
|
||||
func extensionOf(filename string) string {
|
||||
ext := strings.ToLower(filepath.Ext(filepath.Base(filepath.FromSlash(filename))))
|
||||
if len(ext) < 2 || len(ext) > 9 {
|
||||
return ".txt"
|
||||
}
|
||||
for _, r := range ext[1:] {
|
||||
if (r < 'a' || r > 'z') && (r < '0' || r > '9') {
|
||||
return ".txt"
|
||||
}
|
||||
}
|
||||
return ext
|
||||
}
|
||||
1099
platform/internal/books/books_test.go
Normal file
1099
platform/internal/books/books_test.go
Normal file
File diff suppressed because it is too large
Load diff
342
platform/internal/books/parse.go
Normal file
342
platform/internal/books/parse.go
Normal file
|
|
@ -0,0 +1,342 @@
|
|||
package books
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io/fs"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"textmachine/platform/internal/ingest"
|
||||
"textmachine/platform/internal/jobs"
|
||||
"textmachine/platform/internal/pgstore"
|
||||
"textmachine/platform/internal/runner"
|
||||
)
|
||||
|
||||
// Graces and budgets of the intake walk. Constants rather than settings: they are properties of the
|
||||
// walk itself, and every one of them is a number an operator would have to reason about the
|
||||
// reconciler to choose.
|
||||
const (
|
||||
// claimGrace is how long a parse may be somebody's business before another pass may take it. It
|
||||
// covers the ordinary case — the queue job is claimed in milliseconds — and the failure it exists
|
||||
// for: the process holding the claim was restarted mid-parse.
|
||||
//
|
||||
// ⚠ It MUST outlive the queue's own job timeout, and it is written as that constant plus a margin
|
||||
// so the two cannot drift apart. Shorter, the sweep steals the claim from a parse that is still
|
||||
// legitimately running: thief and holder meet on one project directory, the loser dies on the
|
||||
// engine's exclusive lock with exit 1 — and exit 1 is how the engine says "I cannot cut this
|
||||
// source", so a book that was parsing perfectly well would be rejected for it.
|
||||
claimGrace = jobs.JobTimeout + 5*time.Minute
|
||||
// UploadGrace is how long a book may stay `uploading`. A request that is still arriving holds the
|
||||
// row, so anything older than this is an upload whose request is gone — and that reasoning holds
|
||||
// only while the route's own read deadline is SHORTER. Exported so the boot can refuse a
|
||||
// configuration where it is not: an operator who raises TM_PLATFORM_UPLOAD_DEADLINE past this
|
||||
// would have the sweep delete a book's row and directory out from under a request still writing
|
||||
// into it.
|
||||
UploadGrace = time.Hour
|
||||
// parseAttempts is how many times a book may be handed to the engine before intake gives up. EVERY
|
||||
// answer of the engine goes through it — a refusal of the source as much as a host that cannot run
|
||||
// the binary — because the engine gives both the same exit code, so what this really bounds is how
|
||||
// long a broken host re-runs the engine over every book uploaded to it, and how many chances a
|
||||
// genuinely unreadable file gets before its source is removed.
|
||||
parseAttempts = 5
|
||||
)
|
||||
|
||||
// Reasons a book is rejected. The platform's OWN closed vocabulary: the engine's text reads like
|
||||
// pipeline internals and never crosses this seam (contract §Problem, PT-33), and these are stored
|
||||
// for an operator rather than projected — contract v0 gives a rejected book no reason field.
|
||||
const (
|
||||
// ReasonSourceUnreadable — the engine ran and refused the file. Terminal, but NOT on the first
|
||||
// answer: the engine maps every failure it has onto exit 1, so a refusal goes through the attempt
|
||||
// budget like everything else and only the last one deletes the source (see defer_).
|
||||
ReasonSourceUnreadable = "source_unreadable"
|
||||
// ReasonNotConfigured — the book has no project configuration, so there is nothing to parse
|
||||
// against. See ErrNotProvisioned: today that is a deployment's state, not a user's mistake.
|
||||
ReasonNotConfigured = "not_configured"
|
||||
// ReasonParserUnavailable — the engine could not be RUN, repeatedly, until the attempt budget was
|
||||
// spent.
|
||||
ReasonParserUnavailable = "parser_unavailable"
|
||||
// ReasonStorageUnavailable — the storage root itself is not there. Never terminal, and never
|
||||
// stored on a book: it says something about the host, and the book it happened to be read for is
|
||||
// no more at fault than any other. See ErrStorageGone.
|
||||
ReasonStorageUnavailable = "storage_unavailable"
|
||||
)
|
||||
|
||||
// ErrNotProvisioned is a book directory with no engine configuration in it.
|
||||
//
|
||||
// ⚠ This is the seam of an OPEN question, and it is named here rather than papered over. A book's
|
||||
// project configuration (`book.yaml`) carries the pair, the ceilings and the paths the engine reads,
|
||||
// and D39.110 §2b says the platform does not write that file. Creating a directory for an uploaded
|
||||
// book therefore leaves somebody having to put the first one there, and who that is — the operator
|
||||
// from a deployment template, or the platform rendering one at intake and never touching it again —
|
||||
// is a ratification the zone asked for and does not have (zone journal, P5).
|
||||
//
|
||||
// Until it arrives the intake is complete up to this line: the file is received, stored, counted and
|
||||
// handed to the engine, and a book whose directory carries no configuration is rejected with a
|
||||
// reason that says so to an operator.
|
||||
var ErrNotProvisioned = errors.New("books: the book has no engine configuration")
|
||||
|
||||
// ErrDirectoryGone is a book whose whole project directory is missing.
|
||||
//
|
||||
// Told apart from ErrNotProvisioned deliberately, and the difference is terminal-vs-not: a missing
|
||||
// CONFIGURATION is a deployment question somebody can still answer, while a missing DIRECTORY means
|
||||
// the source this platform received is not there any more and no amount of waiting brings it back.
|
||||
// It is also the crash window of the rejection path — the directory is removed before the row is
|
||||
// written — and reading it as "not configured" left such a book in `parsing` for good.
|
||||
var ErrDirectoryGone = errors.New("books: the book's directory is gone")
|
||||
|
||||
// ErrStorageGone is the ROOT of the book storage missing — an unmounted volume, or a deployment
|
||||
// pointed at a path that is not there yet.
|
||||
//
|
||||
// Told apart from ErrDirectoryGone because the two look identical from one book (both are ENOENT on
|
||||
// the same Stat) and mean opposite things. A book's own directory being gone is that book's own
|
||||
// terminal end; the root being gone is the host's, and reading it as the first would have ONE sweep
|
||||
// reject EVERY book in intake with "the source cannot be read" — a reason that blames the user's
|
||||
// file, is terminal by design, and has no way back (there is no re-parse and no un-reject). Found by
|
||||
// cross-family review of the acceptance dofix.
|
||||
var ErrStorageGone = errors.New("books: the book storage root is gone")
|
||||
|
||||
// Parse turns a received file into a chapter tree, or into a reason it is not one.
|
||||
//
|
||||
// It is the body of the queue's worker AND of the backstop sweep, and it is safe to call twice: the
|
||||
// claim is a compare-and-set, so the loser does nothing. Losing that race is the ordinary case, not
|
||||
// a failure — two `tmctl manifest` processes on one project directory would be two writers of a file
|
||||
// the engine holds exclusively.
|
||||
func (s *Service) Parse(ctx context.Context, bookID string) error {
|
||||
now := s.now()
|
||||
claim, err := s.Store.ClaimParse(ctx, bookID, now, now.Add(-claimGrace))
|
||||
if errors.Is(err, pgstore.ErrParseClaimed) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
m, err := s.manifest(ctx, claim.Workdir)
|
||||
if err == nil {
|
||||
if m.ChaptersTotal < 1 {
|
||||
// A source the engine read and found nothing in. The book's own fault — but it goes through
|
||||
// the same budget as everything else, because "the engine answered no" and "the engine could
|
||||
// not answer" are told apart by an exit code the engine gives BOTH.
|
||||
return s.defer_(ctx, claim, ReasonSourceUnreadable)
|
||||
}
|
||||
// Counts and versions, no book id: the same rule as everywhere else on this side of the log
|
||||
// (ENGINEERING_STANDARDS §Наблюдаемость). What an operator needs per book — including why a
|
||||
// book was rejected — is a column, not a log line.
|
||||
s.log().InfoContext(ctx, "book parsed", "chapters", m.ChaptersTotal,
|
||||
"manifest_version", m.Version, "chunker", m.ChunkerVersion)
|
||||
c, cancel := writeCtx(ctx)
|
||||
defer cancel()
|
||||
return s.Store.FinishParse(c, bookID, claim.At, pgstore.ParsedBook{
|
||||
Chapters: m.ChaptersTotal,
|
||||
SourceSHA256: m.SourceSHA256Bytes(),
|
||||
ChunkerVersion: m.ChunkerVersion,
|
||||
})
|
||||
}
|
||||
if errors.Is(err, ErrStorageGone) {
|
||||
// The host cannot see its own storage. Nothing about this book is known yet, so it waits with
|
||||
// the budget untouched — exactly like a book waiting for a configuration.
|
||||
s.log().ErrorContext(ctx, "the book storage root is not there: intake waits rather than rejects",
|
||||
"reason", ReasonStorageUnavailable)
|
||||
return s.defer_(ctx, claim, ReasonStorageUnavailable)
|
||||
}
|
||||
if errors.Is(err, ErrDirectoryGone) {
|
||||
// Nothing to wait for and nothing to retry: what this platform received is not on disk. It is
|
||||
// also how the rejection path heals from a crash between removing the directory and writing
|
||||
// the row — the next pass finds no directory and finishes the job.
|
||||
s.log().ErrorContext(ctx, "book rejected: its directory is gone", "reason", ReasonSourceUnreadable)
|
||||
return s.reject(ctx, claim, ReasonSourceUnreadable)
|
||||
}
|
||||
reason := ReasonParserUnavailable
|
||||
switch {
|
||||
case errors.Is(err, ErrNotProvisioned):
|
||||
reason = ReasonNotConfigured
|
||||
case refusedTheSource(err):
|
||||
reason = ReasonSourceUnreadable
|
||||
}
|
||||
// ⚠ err can carry the book's path (the engine is asked about a directory), which is the open class
|
||||
// of PD-139, so it travels only at WARN/ERROR and never at INFO.
|
||||
return s.defer_(ctx, claim, reason)
|
||||
}
|
||||
|
||||
// defer_ spends one attempt of the budget and, when the budget is gone, ends the intake.
|
||||
//
|
||||
// ONE path for every way a parse can fail, and that is the correction the cross-family review
|
||||
// forced: the engine maps ALL of its failures onto exit 1 (its own comment says so), so "this source
|
||||
// cannot be cut" is not distinguishable from "the disk was full", "an operator's own tmctl held the
|
||||
// lock", or "another pass of this platform took the claim and is holding it". Rejecting — and
|
||||
// DELETING the user's file — on the first exit 1 turned any of those into irreversible data loss.
|
||||
// Spending an attempt instead costs a broken host five engine calls spread over the claim grace and
|
||||
// costs a genuinely unreadable book the same five before it is rejected.
|
||||
//
|
||||
// ⚠ `not_configured` is the one reason that NEVER becomes terminal. The platform does not write
|
||||
// `book.yaml` (D39.110 §2b) and who does is a ratification this zone does not have, so a book whose
|
||||
// directory has no configuration is waiting on a DEPLOYMENT question. Rejecting it — and it would be
|
||||
// every uploaded book on a deployment without that answer — would destroy a user's upload over a
|
||||
// gap they cannot see. It stays `parsing`, visible in the intake metric, and an operator dropping
|
||||
// the file in is all it takes.
|
||||
func (s *Service) defer_(ctx context.Context, claim pgstore.ParseClaim, reason string) error {
|
||||
if !waitsForTheDeployment(reason) && claim.Attempts >= parseAttempts {
|
||||
s.log().ErrorContext(ctx, "book rejected: intake has spent its attempts on it",
|
||||
"attempts", claim.Attempts, "reason", reason)
|
||||
return s.reject(ctx, claim, reason)
|
||||
}
|
||||
// The claim is NOT given back: it is what spaces the retries. Released, the sweep would re-offer
|
||||
// the book on its very next tick (the staleness predicate falls back to `added_at`, which is
|
||||
// already old), and the attempts would burn in as many ticks — 75 seconds at the default sweep
|
||||
// interval — rather than over the time this budget is for.
|
||||
//
|
||||
// The ATTEMPT, though, is given back when the engine was never asked. `ClaimParse` counts every
|
||||
// claim, and a book waiting for its configuration claims once per grace forever — so without this
|
||||
// the budget was spent by WAITING, and the first real answer from the engine afterwards was
|
||||
// terminal on arrival. A typo in a hand-written `book.yaml` would then delete the user's upload
|
||||
// on the first attempt at reading it. Waiting must not bring deletion closer.
|
||||
if waitsForTheDeployment(reason) {
|
||||
c, cancel := writeCtx(ctx)
|
||||
defer cancel()
|
||||
if err := s.Store.RefundParseAttempt(c, claim.BookID, claim.At); err != nil {
|
||||
s.log().ErrorContext(ctx, "the parse attempt could not be refunded", "err", err)
|
||||
}
|
||||
}
|
||||
s.log().WarnContext(ctx, "parse deferred", "attempts", claim.Attempts, "reason", reason)
|
||||
return nil
|
||||
}
|
||||
|
||||
// waitsForTheDeployment reports the reasons that never end an intake and never spend its budget: the
|
||||
// engine was not asked at all, and what is missing belongs to the deployment — a configuration
|
||||
// nobody has written yet, or the storage root itself. Both are answerable by a human, and both would
|
||||
// otherwise destroy every upload on a host that is merely misconfigured.
|
||||
func waitsForTheDeployment(reason string) bool {
|
||||
return reason == ReasonNotConfigured || reason == ReasonStorageUnavailable
|
||||
}
|
||||
|
||||
// writeCtx is the context a TERMINAL write uses: detached from the caller's deadline and bounded on
|
||||
// its own.
|
||||
//
|
||||
// The engine call this follows can legitimately consume the whole budget of the pass — and then the
|
||||
// write that records what happened would run on an already-expired context and be lost, leaving the
|
||||
// book to be retried and the attempt to be spent again, forever. What must survive is the record.
|
||||
func writeCtx(ctx context.Context) (context.Context, context.CancelFunc) {
|
||||
return context.WithTimeout(context.WithoutCancel(ctx), writeBudget)
|
||||
}
|
||||
|
||||
// writeBudget is what a terminal write gets. Short: it is one statement against a database this
|
||||
// process is already connected to.
|
||||
const writeBudget = 30 * time.Second
|
||||
|
||||
// manifest asks the engine to cut the book, once its configuration is there to cut it against.
|
||||
func (s *Service) manifest(ctx context.Context, workdir string) (ingest.Manifest, error) {
|
||||
if s.Engine == nil {
|
||||
return ingest.Manifest{}, errors.New("books: no engine is configured")
|
||||
}
|
||||
if _, err := os.Stat(workdir); err != nil {
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
// Which of the two absences this is decides whether the book dies, so it is decided by the
|
||||
// SENTINEL rather than by the root's own existence (see storageIsThere).
|
||||
if !s.storageIsThere() {
|
||||
return ingest.Manifest{}, ErrStorageGone
|
||||
}
|
||||
return ingest.Manifest{}, ErrDirectoryGone
|
||||
}
|
||||
return ingest.Manifest{}, fmt.Errorf("books: read book directory: %w", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(workdir, runner.ConfigFile)); err != nil {
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
return ingest.Manifest{}, ErrNotProvisioned
|
||||
}
|
||||
return ingest.Manifest{}, fmt.Errorf("books: read book configuration: %w", err)
|
||||
}
|
||||
return s.Engine.Manifest(ctx, s.Cfg.EngineBinary, workdir)
|
||||
}
|
||||
|
||||
// refusedTheSource tells the engine's ANSWER from the engine's absence.
|
||||
//
|
||||
// The engine maps every failure that is not one of its two sentinels onto exit 1 (backend
|
||||
// cmd/tmctl/main.go exitCode), so "the source cannot be cut" arrives as an ExitError — a process
|
||||
// that ran and said no. Everything else here is this host: no binary at that path, no permission to
|
||||
// execute it, a working directory that is gone, a context that expired. Those retry; an answer does
|
||||
// not.
|
||||
func refusedTheSource(err error) bool {
|
||||
var exit *exec.ExitError
|
||||
if !errors.As(err, &exit) {
|
||||
return false
|
||||
}
|
||||
// A signal is not an answer: an engine the machine killed said nothing about the book.
|
||||
return exit.Exited()
|
||||
}
|
||||
|
||||
// reject records the terminal end of an intake and removes what is left of it.
|
||||
//
|
||||
// The source is deleted, and it is a decision rather than housekeeping: the file cannot be parsed,
|
||||
// no path in the contract re-parses or downloads it, and an authenticated route that writes bytes to
|
||||
// an operator's disk and never removes them is a hole this pack would otherwise be opening. The ROW
|
||||
// stays — the user has to be able to see that the book they uploaded did not make it.
|
||||
func (s *Service) reject(ctx context.Context, claim pgstore.ParseClaim, reason string) error {
|
||||
// The DIRECTORY goes first and the row second, and the order is the crash window: dying between
|
||||
// the two then leaves a book still `parsing` with no source, which the next attempt fails on and
|
||||
// rejects properly. The other order leaves a directory with a full source and a row that says
|
||||
// `rejected` — and nothing ever looks at a rejected book again, so those bytes stay forever.
|
||||
//
|
||||
// ⚠ Only where the source is the BOOK's fault. A deployment that could not run the engine keeps
|
||||
// the file: deleting a user's upload because this host was misconfigured is not a decision to
|
||||
// make on their behalf.
|
||||
if reason == ReasonSourceUnreadable {
|
||||
s.removeDir(claim.Workdir)
|
||||
}
|
||||
// On a context of its own: this write is the only record that the intake is over, and the engine
|
||||
// call that led here may have used up everything the pass had.
|
||||
c, cancel := writeCtx(ctx)
|
||||
defer cancel()
|
||||
return s.Store.RejectBook(c, claim.BookID, claim.At, reason)
|
||||
}
|
||||
|
||||
// Sweep finishes the intake walk of every book whose own process did not.
|
||||
//
|
||||
// The two halves are the two ways a walk stops: a request that went away mid-upload, and a parse
|
||||
// whose process is gone. Neither is reachable from the thing that started it, which is what makes
|
||||
// this the only cure — the same reason the run reconciler exists.
|
||||
//
|
||||
// One book's failure never stops the sweep: these are independent books of independent accounts.
|
||||
func (s *Service) Sweep(ctx context.Context) error {
|
||||
now := s.now()
|
||||
stuck, err := s.Store.StuckIntake(ctx, now.Add(-UploadGrace), now.Add(-claimGrace))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, b := range stuck {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
switch b.Status {
|
||||
case "uploading":
|
||||
s.log().InfoContext(ctx, "removing an upload that never finished")
|
||||
s.abandon(ctx, b.ID, b.Workdir)
|
||||
case "parsing":
|
||||
// Each book gets its OWN budget, and it has to be one a parse can live inside: the engine
|
||||
// call is the same one the queue gives fifteen minutes. Sharing the pass's deadline meant a
|
||||
// large book was killed by it, the kill was read as a host that cannot run the engine, and
|
||||
// the attempt was spent — every pass, until the book was rejected for being big.
|
||||
//
|
||||
// ⚠ A budget the book gets is not a budget the PASS still has, and that gap was the rest of
|
||||
// the same defect (re-check of the dofix): the second book of a pass inherited whatever the
|
||||
// first one left, so a slow first parse handed the second a stub of a deadline and burned
|
||||
// its attempt on the timeout. A book that cannot be given its whole budget is therefore not
|
||||
// STARTED — the claim is taken inside Parse, so a book left for the next tick has spent
|
||||
// nothing — and the pass says so once rather than per book.
|
||||
if deadline, ok := ctx.Deadline(); ok && time.Until(deadline) < jobs.JobTimeout {
|
||||
s.log().InfoContext(ctx, "intake pass ends early: what is left of it is shorter than a parse",
|
||||
"remaining", time.Until(deadline).String())
|
||||
return nil
|
||||
}
|
||||
c, cancel := context.WithTimeout(ctx, jobs.JobTimeout)
|
||||
err := s.Parse(c, b.ID)
|
||||
cancel()
|
||||
if err != nil {
|
||||
s.log().ErrorContext(ctx, "book could not be parsed", "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
|
@ -4,18 +4,22 @@ package config
|
|||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"textmachine/platform/internal/books"
|
||||
"textmachine/platform/internal/money"
|
||||
"textmachine/platform/internal/pricing"
|
||||
)
|
||||
|
||||
// Config is the whole configuration surface. Environment only: a control plane is deployed, not
|
||||
// hand-run, and a settings file is one more thing to keep in sync with the deployment.
|
||||
// hand-run, and a settings file is one more thing to keep in sync with the deployment. Ratified
|
||||
// 09.08 (register row PD-114) together with the obligation this file now also carries — an operator
|
||||
// must be able to read what actually applied, which is what Settings and LogEffective are for.
|
||||
type Config struct {
|
||||
Addr string
|
||||
// DSN is the Postgres connection string. Empty is allowed and means "start unready": the
|
||||
|
|
@ -54,9 +58,18 @@ type Config struct {
|
|||
// SignupGrantMicroUSD is the credit a new account is created with (the free tier, owner 05.08:
|
||||
// default five dollars, settable per account by granting a different amount).
|
||||
SignupGrantMicroUSD int64
|
||||
// MetricsAddr is where the telemetry listener binds. A SEPARATE listener from the API on
|
||||
// purpose — see the runner's own note — and empty turns it off.
|
||||
MetricsAddr string
|
||||
|
||||
// Runner is everything about starting engine runs.
|
||||
Runner RunnerConfig
|
||||
// Intake is everything about receiving a book.
|
||||
Intake IntakeConfig
|
||||
|
||||
// Settings is every value above with where it came from, in the order it was read. It is what
|
||||
// the boot line prints (PD-114) and it carries no secret and no money amount — see Setting.
|
||||
Settings []Setting
|
||||
}
|
||||
|
||||
// RunnerConfig is the runner's operator-facing surface.
|
||||
|
|
@ -95,25 +108,72 @@ type RunnerConfig struct {
|
|||
Workers int
|
||||
}
|
||||
|
||||
// IntakeConfig is the book upload's operator-facing surface.
|
||||
type IntakeConfig struct {
|
||||
// BooksDir is the root under which the platform creates one directory per uploaded book. Empty
|
||||
// means this deployment takes no uploads, and then POST /books is not mounted at all: an
|
||||
// instance that accepted files it has nowhere to put would reject every one of them after the
|
||||
// upload rather than before it.
|
||||
//
|
||||
// Absolute, for the same reason StateDir is: a relative root names a different directory to the
|
||||
// daemon and to anything else that reads a book's workdir out of the database.
|
||||
BooksDir string
|
||||
// MaxUploadBytes caps ONE upload's request body, and it is the per-route limit PD-72 was opened
|
||||
// for. It is a refusal and not a sizing: the acceptance corpus is a 23 MB book.
|
||||
MaxUploadBytes int64
|
||||
// UploadDeadline is how long one upload's body may take to arrive. It REPLACES the server's
|
||||
// 30-second ReadTimeout for that route alone and stays finite; clearing a deadline instead is
|
||||
// what re-created PD-2 and is forbidden (STACK_DECISIONS §12).
|
||||
UploadDeadline time.Duration
|
||||
}
|
||||
|
||||
// Setting is one resolved value as an operator may read it in a log.
|
||||
type Setting struct {
|
||||
// Key is the environment variable.
|
||||
Key string
|
||||
// Value is what applied — or a REDACTION where the value may not be logged. Two classes are
|
||||
// redacted and both are rules of this project rather than taste: a secret (a DSN carries a
|
||||
// password; §11 keeps it out of the process environment for the same reason), and any amount of
|
||||
// MONEY, which does not reach an INFO log in any form (D39.84, PD-99). For those the source is
|
||||
// the whole point — "did my file get picked up" is answerable without the value.
|
||||
Value string
|
||||
// Source is where the value came from: default, environment, or a *_FILE.
|
||||
Source string
|
||||
}
|
||||
|
||||
const (
|
||||
sourceDefault = "default"
|
||||
sourceEnv = "environment"
|
||||
sourceFile = "file"
|
||||
)
|
||||
|
||||
// redacted values. They are written out rather than left empty so that a line always says something:
|
||||
// an empty value column reads as "unset", which is a different fact.
|
||||
const (
|
||||
valueSet = "(set)"
|
||||
valueUnset = "(unset)"
|
||||
valueAmount = "(an amount; not logged)"
|
||||
)
|
||||
|
||||
// Load reads the environment.
|
||||
func Load() (Config, error) {
|
||||
var l loader
|
||||
c := Config{
|
||||
Addr: env("TM_PLATFORM_ADDR", "127.0.0.1:8080"),
|
||||
DSN: "", // read below: it may come from a file
|
||||
Addr: l.env("TM_PLATFORM_ADDR", "127.0.0.1:8080"),
|
||||
OIDCProvider: l.env("TM_PLATFORM_OIDC_PROVIDER", "google"),
|
||||
OIDCIssuer: l.env("TM_PLATFORM_OIDC_ISSUER", ""),
|
||||
OIDCClientID: l.env("TM_PLATFORM_OIDC_CLIENT_ID", ""),
|
||||
OIDCRedirectURL: l.env("TM_PLATFORM_OIDC_REDIRECT_URL", ""),
|
||||
AfterLogin: l.env("TM_PLATFORM_AFTER_LOGIN", "/"),
|
||||
MetricsAddr: metricsAddr(l.env("TM_PLATFORM_METRICS_ADDR", "127.0.0.1:9464")),
|
||||
SignupGrantMicroUSD: 5 * 1_000_000,
|
||||
// 14 days idle, 30 days absolute. The absolute one is the NIST SP 800-63B-4 AAL1 figure
|
||||
// ("SHOULD be no more than 30 days"), not a preference: it was 90 days, and a deviation from
|
||||
// a SHOULD needs a reason that survives inspection, which that one did not (PD-58, §13).
|
||||
SessionIdleTTL: 14 * 24 * time.Hour,
|
||||
SessionMaxAge: 30 * 24 * time.Hour,
|
||||
OIDCProvider: env("TM_PLATFORM_OIDC_PROVIDER", "google"),
|
||||
OIDCIssuer: os.Getenv("TM_PLATFORM_OIDC_ISSUER"),
|
||||
OIDCClientID: os.Getenv("TM_PLATFORM_OIDC_CLIENT_ID"),
|
||||
OIDCClientSecret: "", // read below: it may come from a file
|
||||
OIDCRedirectURL: os.Getenv("TM_PLATFORM_OIDC_REDIRECT_URL"),
|
||||
AfterLogin: env("TM_PLATFORM_AFTER_LOGIN", "/"),
|
||||
SignupGrantMicroUSD: 5 * 1_000_000,
|
||||
SessionIdleTTL: 14 * 24 * time.Hour,
|
||||
SessionMaxAge: 30 * 24 * time.Hour,
|
||||
}
|
||||
if raw := os.Getenv("TM_PLATFORM_TRUSTED_ORIGINS"); raw != "" {
|
||||
if raw := l.env("TM_PLATFORM_TRUSTED_ORIGINS", ""); raw != "" {
|
||||
for _, o := range strings.Split(raw, ",") {
|
||||
if o = strings.TrimSpace(o); o != "" {
|
||||
c.TrustedOrigins = append(c.TrustedOrigins, o)
|
||||
|
|
@ -121,38 +181,34 @@ func Load() (Config, error) {
|
|||
}
|
||||
}
|
||||
var err error
|
||||
if c.Migrate, err = boolean("TM_PLATFORM_MIGRATE"); err != nil {
|
||||
if c.Migrate, err = l.boolean("TM_PLATFORM_MIGRATE"); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if c.InsecureCookies, err = boolean("TM_PLATFORM_INSECURE_COOKIES"); err != nil {
|
||||
if c.InsecureCookies, err = l.boolean("TM_PLATFORM_INSECURE_COOKIES"); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if c.DSN, err = secret("TM_PLATFORM_DSN"); err != nil {
|
||||
if c.DSN, err = l.secret("TM_PLATFORM_DSN"); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if c.OIDCClientSecret, err = secret("TM_PLATFORM_OIDC_CLIENT_SECRET"); err != nil {
|
||||
if c.OIDCClientSecret, err = l.secret("TM_PLATFORM_OIDC_CLIENT_SECRET"); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if c.SessionIdleTTL, err = duration("TM_PLATFORM_SESSION_IDLE", c.SessionIdleTTL); err != nil {
|
||||
if c.SessionIdleTTL, err = l.duration("TM_PLATFORM_SESSION_IDLE", c.SessionIdleTTL); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if c.SessionMaxAge, err = duration("TM_PLATFORM_SESSION_MAX_AGE", c.SessionMaxAge); err != nil {
|
||||
if c.SessionMaxAge, err = l.duration("TM_PLATFORM_SESSION_MAX_AGE", c.SessionMaxAge); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if c.SessionIdleTTL > c.SessionMaxAge {
|
||||
return Config{}, fmt.Errorf("config: session idle TTL %s exceeds max age %s", c.SessionIdleTTL, c.SessionMaxAge)
|
||||
}
|
||||
if raw := os.Getenv("TM_PLATFORM_SIGNUP_GRANT_USD"); raw != "" {
|
||||
v, err := money.ParseUSD(raw)
|
||||
if err != nil {
|
||||
return Config{}, fmt.Errorf("config: TM_PLATFORM_SIGNUP_GRANT_USD: %w", err)
|
||||
}
|
||||
if v < 0 {
|
||||
return Config{}, errors.New("config: TM_PLATFORM_SIGNUP_GRANT_USD cannot be negative")
|
||||
}
|
||||
c.SignupGrantMicroUSD = int64(v)
|
||||
if c.SignupGrantMicroUSD, err = l.amount("TM_PLATFORM_SIGNUP_GRANT_USD", c.SignupGrantMicroUSD, false); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if err := c.loadRunner(); err != nil {
|
||||
if err := c.loadRunner(&l); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if err := c.loadIntake(&l); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
// Half a login configuration is worse than none: the surface would mount and fail at the first
|
||||
|
|
@ -167,9 +223,39 @@ func Load() (Config, error) {
|
|||
if set != 0 && set != len(oidc) {
|
||||
return Config{}, errors.New("config: OIDC needs all of TM_PLATFORM_OIDC_ISSUER, _CLIENT_ID, _CLIENT_SECRET, _REDIRECT_URL, or none")
|
||||
}
|
||||
c.Settings = l.settings
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// metricsAddr turns the operator's word for "do not serve metrics" into the empty address the
|
||||
// listener understands.
|
||||
//
|
||||
// A word is needed because an EMPTY environment variable is indistinguishable from an unset one to
|
||||
// os.Getenv, and unset takes the default — so without this the switch documented as "empty turns it
|
||||
// off" could not be reached from the environment at all.
|
||||
func metricsAddr(v string) string {
|
||||
if strings.EqualFold(v, "off") || strings.EqualFold(v, "none") {
|
||||
return ""
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// LogEffective prints what actually applied, one line per setting (PD-114, ratified 09.08).
|
||||
//
|
||||
// It exists because an environment-only configuration has no artifact an operator can read back: a
|
||||
// deployment whose EnvironmentFile= did not load looks exactly like one whose values were chosen,
|
||||
// and the difference used to be discoverable only from behaviour. The line carries the SOURCE as
|
||||
// well as the value, because "did my setting take effect" is the actual question and a value alone
|
||||
// cannot answer it — a default and an environment variable holding the same string are the same line
|
||||
// otherwise.
|
||||
//
|
||||
// Secrets and money amounts are redacted here rather than at the call site (see Setting.Value).
|
||||
func (c Config) LogEffective(log *slog.Logger) {
|
||||
for _, s := range c.Settings {
|
||||
log.Info("config", "key", s.Key, "value", s.Value, "source", s.Source)
|
||||
}
|
||||
}
|
||||
|
||||
// LoginEnabled reports whether a sign-in provider is configured.
|
||||
func (c Config) LoginEnabled() bool { return c.OIDCIssuer != "" }
|
||||
|
||||
|
|
@ -177,17 +263,22 @@ func (c Config) LoginEnabled() bool { return c.OIDCIssuer != "" }
|
|||
// engine binary still serves every read: the library and the book card are not run machinery.
|
||||
func (c Config) RunsEnabled() bool { return c.Runner.EngineBinary != "" }
|
||||
|
||||
// IntakeEnabled reports whether this instance can receive a book. It needs somewhere to put the file
|
||||
// AND an engine to cut it with: parsing is one call of tmctl, so a deployment without the binary
|
||||
// could only take uploads it would have to reject.
|
||||
func (c Config) IntakeEnabled() bool { return c.Intake.BooksDir != "" && c.RunsEnabled() }
|
||||
|
||||
// loadRunner reads the runner's settings.
|
||||
func (c *Config) loadRunner() error {
|
||||
func (c *Config) loadRunner(l *loader) error {
|
||||
r := RunnerConfig{
|
||||
EngineBinary: os.Getenv("TM_PLATFORM_ENGINE_BIN"),
|
||||
StateDir: env("TM_PLATFORM_STATE_DIR", "/var/lib/tmplatform"),
|
||||
MarkerBinary: os.Getenv("TM_PLATFORM_CTL_BIN"),
|
||||
CeilingArg: env("TM_PLATFORM_ENGINE_CEILING_ARG", "--ceiling-usd {{usd}}"),
|
||||
EngineBinary: l.env("TM_PLATFORM_ENGINE_BIN", ""),
|
||||
StateDir: l.env("TM_PLATFORM_STATE_DIR", "/var/lib/tmplatform"),
|
||||
MarkerBinary: l.env("TM_PLATFORM_CTL_BIN", ""),
|
||||
CeilingArg: l.env("TM_PLATFORM_ENGINE_CEILING_ARG", "--ceiling-usd {{usd}}"),
|
||||
// A run gets a generous but finite share of the machine. The figure is a BACKSTOP, not a
|
||||
// sizing: a translation is bounded by its ceiling in dollars, and this bounds the one way a
|
||||
// single run can hurt the others regardless of money (PD-13).
|
||||
MemoryMax: env("TM_PLATFORM_RUN_MEMORY_MAX", "4G"),
|
||||
MemoryMax: l.env("TM_PLATFORM_RUN_MEMORY_MAX", "4G"),
|
||||
TasksMax: 64,
|
||||
SweepEvery: 15 * time.Second,
|
||||
ResyncEvery: 5 * time.Minute,
|
||||
|
|
@ -203,56 +294,97 @@ func (c *Config) loadRunner() error {
|
|||
return fmt.Errorf("config: TM_PLATFORM_STATE_DIR must be an absolute path, got %q", r.StateDir)
|
||||
}
|
||||
var err error
|
||||
if r.TasksMax, err = number("TM_PLATFORM_RUN_TASKS_MAX", r.TasksMax); err != nil {
|
||||
if r.TasksMax, err = l.number("TM_PLATFORM_RUN_TASKS_MAX", r.TasksMax); err != nil {
|
||||
return err
|
||||
}
|
||||
if r.AllowEngineVersionChange, err = boolean("TM_PLATFORM_RESUME_MAY_CHANGE_ENGINE"); err != nil {
|
||||
if r.AllowEngineVersionChange, err = l.boolean("TM_PLATFORM_RESUME_MAY_CHANGE_ENGINE"); err != nil {
|
||||
return err
|
||||
}
|
||||
if r.Workers, err = number("TM_PLATFORM_RUN_WORKERS", r.Workers); err != nil {
|
||||
if r.Workers, err = l.number("TM_PLATFORM_RUN_WORKERS", r.Workers); err != nil {
|
||||
return err
|
||||
}
|
||||
if r.SweepEvery, err = duration("TM_PLATFORM_SWEEP_EVERY", r.SweepEvery); err != nil {
|
||||
if r.SweepEvery, err = l.duration("TM_PLATFORM_SWEEP_EVERY", r.SweepEvery); err != nil {
|
||||
return err
|
||||
}
|
||||
if r.ResyncEvery, err = duration("TM_PLATFORM_RESYNC_EVERY", r.ResyncEvery); err != nil {
|
||||
if r.ResyncEvery, err = l.duration("TM_PLATFORM_RESYNC_EVERY", r.ResyncEvery); err != nil {
|
||||
return err
|
||||
}
|
||||
if raw := os.Getenv("TM_PLATFORM_USD_PER_CHAPTER"); raw != "" {
|
||||
v, err := money.ParseUSD(raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("config: TM_PLATFORM_USD_PER_CHAPTER: %w", err)
|
||||
}
|
||||
if v <= 0 {
|
||||
return errors.New("config: TM_PLATFORM_USD_PER_CHAPTER must be positive")
|
||||
}
|
||||
r.PerChapterMicroUSD = int64(v)
|
||||
if r.PerChapterMicroUSD, err = l.amount("TM_PLATFORM_USD_PER_CHAPTER", r.PerChapterMicroUSD, true); err != nil {
|
||||
return err
|
||||
}
|
||||
c.Runner = r
|
||||
return nil
|
||||
}
|
||||
|
||||
func number(key string, def int) (int, error) {
|
||||
raw := os.Getenv(key)
|
||||
if raw == "" {
|
||||
return def, nil
|
||||
// loadIntake reads the book upload's settings.
|
||||
func (c *Config) loadIntake(l *loader) error {
|
||||
in := IntakeConfig{
|
||||
BooksDir: l.env("TM_PLATFORM_BOOKS_DIR", ""),
|
||||
// 64 MiB and ten minutes: the acceptance corpus is a 23 MB book, and ten minutes is what a
|
||||
// file that size takes on an uplink an ordinary reader has. Both are refusals rather than
|
||||
// promises — the numbers an operator raises when their users hit them.
|
||||
MaxUploadBytes: 64 << 20,
|
||||
UploadDeadline: 10 * time.Minute,
|
||||
}
|
||||
v, err := strconv.Atoi(raw)
|
||||
if err != nil || v <= 0 {
|
||||
return 0, fmt.Errorf("config: %s must be a positive whole number, got %q", key, raw)
|
||||
if in.BooksDir != "" && !filepath.IsAbs(in.BooksDir) {
|
||||
return fmt.Errorf("config: TM_PLATFORM_BOOKS_DIR must be an absolute path, got %q", in.BooksDir)
|
||||
}
|
||||
return v, nil
|
||||
var err error
|
||||
if in.MaxUploadBytes, err = l.bytes("TM_PLATFORM_MAX_UPLOAD_BYTES", in.MaxUploadBytes); err != nil {
|
||||
return err
|
||||
}
|
||||
if in.UploadDeadline, err = l.duration("TM_PLATFORM_UPLOAD_DEADLINE", in.UploadDeadline); err != nil {
|
||||
return err
|
||||
}
|
||||
// The two are a PAIR, and the boot is where a mismatched pair is cheap to find. The intake sweep
|
||||
// treats a book that has been `uploading` longer than its grace as an upload whose request is
|
||||
// gone — true only while one request cannot legitimately take that long. Longer, and the sweep
|
||||
// deletes the row and the directory under a request that is still writing into them.
|
||||
if in.UploadDeadline >= books.UploadGrace {
|
||||
return fmt.Errorf("config: TM_PLATFORM_UPLOAD_DEADLINE (%s) must be shorter than the intake sweep's grace (%s)",
|
||||
in.UploadDeadline, books.UploadGrace)
|
||||
}
|
||||
c.Intake = in
|
||||
return nil
|
||||
}
|
||||
|
||||
// loader reads the environment and REMEMBERS what it read, so that the boot line can say where each
|
||||
// value came from. Every setting of this service goes through it: a print that covers most of the
|
||||
// configuration is worse than none, because the variable an operator is hunting is exactly the one
|
||||
// nobody thought to record.
|
||||
type loader struct{ settings []Setting }
|
||||
|
||||
func (l *loader) record(key, value, source string) {
|
||||
l.settings = append(l.settings, Setting{Key: key, Value: value, Source: source})
|
||||
}
|
||||
|
||||
func (l *loader) env(key, def string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
l.record(key, v, sourceEnv)
|
||||
return v
|
||||
}
|
||||
if def == "" {
|
||||
l.record(key, valueUnset, sourceDefault)
|
||||
} else {
|
||||
l.record(key, def, sourceDefault)
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
// Secret is the shared way to read a credential: from KEY, or preferably from the file named by
|
||||
// KEY_FILE. Exported so the admin CLI reads the DSN the same way the daemon does — an operator who
|
||||
// followed the deploy notes has it in a file, not in the environment.
|
||||
func Secret(key string) (string, error) { return secret(key) }
|
||||
func Secret(key string) (string, error) {
|
||||
var l loader
|
||||
return l.secret(key)
|
||||
}
|
||||
|
||||
// secret reads a value from KEY, or — preferably — from the file named by KEY_FILE. A secret in a
|
||||
// file does not show up in /proc/<pid>/environ, is not inherited by child processes, and is exactly
|
||||
// what systemd's LoadCredential= hands over (deploy/tmplatformd.service).
|
||||
func secret(key string) (string, error) {
|
||||
//
|
||||
// What is recorded is that it was set and WHERE from, never the value: a DSN carries a password.
|
||||
func (l *loader) secret(key string) (string, error) {
|
||||
if path := os.Getenv(key + "_FILE"); path != "" {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
|
|
@ -260,35 +392,67 @@ func secret(key string) (string, error) {
|
|||
// the error goes to the log.
|
||||
return "", fmt.Errorf("config: %s_FILE: %w", key, err)
|
||||
}
|
||||
l.record(key, valueSet, sourceFile)
|
||||
return strings.TrimSpace(string(b)), nil
|
||||
}
|
||||
return os.Getenv(key), nil
|
||||
v := os.Getenv(key)
|
||||
switch v {
|
||||
case "":
|
||||
l.record(key, valueUnset, sourceDefault)
|
||||
default:
|
||||
l.record(key, valueSet, sourceEnv)
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// boolean reads a flag. strconv.ParseBool rather than a comparison with "1": an operator who wrote
|
||||
// `true` deserves an error or the truth, not a silent no.
|
||||
func boolean(key string) (bool, error) {
|
||||
func (l *loader) boolean(key string) (bool, error) {
|
||||
raw := os.Getenv(key)
|
||||
if raw == "" {
|
||||
l.record(key, "false", sourceDefault)
|
||||
return false, nil
|
||||
}
|
||||
v, err := strconv.ParseBool(raw)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("config: %s: %q is not a boolean", key, raw)
|
||||
}
|
||||
l.record(key, strconv.FormatBool(v), sourceEnv)
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func env(key, def string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func duration(key string, def time.Duration) (time.Duration, error) {
|
||||
func (l *loader) number(key string, def int) (int, error) {
|
||||
raw := os.Getenv(key)
|
||||
if raw == "" {
|
||||
l.record(key, strconv.Itoa(def), sourceDefault)
|
||||
return def, nil
|
||||
}
|
||||
v, err := strconv.Atoi(raw)
|
||||
if err != nil || v <= 0 {
|
||||
return 0, fmt.Errorf("config: %s must be a positive whole number, got %q", key, raw)
|
||||
}
|
||||
l.record(key, strconv.Itoa(v), sourceEnv)
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func (l *loader) bytes(key string, def int64) (int64, error) {
|
||||
raw := os.Getenv(key)
|
||||
if raw == "" {
|
||||
l.record(key, strconv.FormatInt(def, 10), sourceDefault)
|
||||
return def, nil
|
||||
}
|
||||
v, err := strconv.ParseInt(raw, 10, 64)
|
||||
if err != nil || v <= 0 {
|
||||
return 0, fmt.Errorf("config: %s must be a positive number of bytes, got %q", key, raw)
|
||||
}
|
||||
l.record(key, strconv.FormatInt(v, 10), sourceEnv)
|
||||
return v, nil
|
||||
}
|
||||
|
||||
func (l *loader) duration(key string, def time.Duration) (time.Duration, error) {
|
||||
raw := os.Getenv(key)
|
||||
if raw == "" {
|
||||
l.record(key, def.String(), sourceDefault)
|
||||
return def, nil
|
||||
}
|
||||
d, err := time.ParseDuration(raw)
|
||||
|
|
@ -298,5 +462,32 @@ func duration(key string, def time.Duration) (time.Duration, error) {
|
|||
if d <= 0 {
|
||||
return 0, fmt.Errorf("config: %s must be positive", key)
|
||||
}
|
||||
l.record(key, d.String(), sourceEnv)
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// amount reads a sum in dollars into micro-USD. positiveOnly refuses zero as well as negatives,
|
||||
// which is the difference between a rate (a zero rate makes every scale infinite) and a grant (a
|
||||
// deployment that grants nothing is a legitimate choice).
|
||||
//
|
||||
// The VALUE is never recorded — money does not reach an INFO log in any form (D39.84, PD-99) — and
|
||||
// the source is, which is what an operator needs to see that their figure was picked up.
|
||||
func (l *loader) amount(key string, def int64, positiveOnly bool) (int64, error) {
|
||||
raw := os.Getenv(key)
|
||||
if raw == "" {
|
||||
l.record(key, valueAmount, sourceDefault)
|
||||
return def, nil
|
||||
}
|
||||
v, err := money.ParseUSD(raw)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("config: %s: %w", key, err)
|
||||
}
|
||||
switch {
|
||||
case positiveOnly && v <= 0:
|
||||
return 0, fmt.Errorf("config: %s must be positive", key)
|
||||
case v < 0:
|
||||
return 0, fmt.Errorf("config: %s cannot be negative", key)
|
||||
}
|
||||
l.record(key, valueAmount, sourceEnv)
|
||||
return int64(v), nil
|
||||
}
|
||||
|
|
|
|||
192
platform/internal/config/effective_test.go
Normal file
192
platform/internal/config/effective_test.go
Normal file
|
|
@ -0,0 +1,192 @@
|
|||
package config
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"textmachine/platform/internal/books"
|
||||
)
|
||||
|
||||
// PD-114. An environment-only configuration leaves no artifact to read back, so the boot line IS the
|
||||
// artifact — and the two things it must never carry are a secret and a sum of money.
|
||||
|
||||
func effective(t *testing.T) (map[string]Setting, string) {
|
||||
t.Helper()
|
||||
c, err := Load()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
c.LogEffective(slog.New(slog.NewJSONHandler(&buf, nil)))
|
||||
by := map[string]Setting{}
|
||||
for _, s := range c.Settings {
|
||||
if _, dup := by[s.Key]; dup {
|
||||
t.Errorf("%s is printed twice", s.Key)
|
||||
}
|
||||
by[s.Key] = s
|
||||
}
|
||||
return by, buf.String()
|
||||
}
|
||||
|
||||
// The completeness gate, and the reason it is written against the SOURCE rather than a list: a print
|
||||
// that covers most of the configuration is worse than none, because the variable an operator is
|
||||
// hunting is exactly the one nobody remembered to record. A setting added without a record fails
|
||||
// here, in the same commit that adds it.
|
||||
func TestEverySettingThisServiceReadsIsPrinted(t *testing.T) {
|
||||
src, err := os.ReadFile("config.go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := map[string]bool{}
|
||||
for _, m := range regexp.MustCompile(`TM_PLATFORM_[A-Z0-9_]+`).FindAllString(string(src), -1) {
|
||||
// A *_FILE variable is the alternative SOURCE of the key beside it, not a setting of its own:
|
||||
// it is recorded as `source=file` on that key, and its path is never printed.
|
||||
want[strings.TrimSuffix(m, "_FILE")] = true
|
||||
}
|
||||
got, _ := effective(t)
|
||||
var missing []string
|
||||
for k := range want {
|
||||
if _, ok := got[k]; !ok {
|
||||
missing = append(missing, k)
|
||||
}
|
||||
}
|
||||
sort.Strings(missing)
|
||||
if len(missing) != 0 {
|
||||
t.Fatalf("read but never printed: %v", missing)
|
||||
}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("%d settings printed, %d read from the environment", len(got), len(want))
|
||||
}
|
||||
}
|
||||
|
||||
// A DSN carries a password and a client secret is a client secret. What the operator gets is the
|
||||
// fact and the SOURCE, which is what answers "did my file get picked up".
|
||||
func TestASecretIsNamedAndNeverPrinted(t *testing.T) {
|
||||
path := t.TempDir() + "/dsn"
|
||||
if err := os.WriteFile(path, []byte("postgres://u:hunter2@db/x\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("TM_PLATFORM_DSN_FILE", path)
|
||||
// The whole OIDC quartet, because half a login configuration is refused at boot.
|
||||
t.Setenv("TM_PLATFORM_OIDC_CLIENT_SECRET", "s3cret")
|
||||
t.Setenv("TM_PLATFORM_OIDC_ISSUER", "https://accounts.example.org")
|
||||
t.Setenv("TM_PLATFORM_OIDC_CLIENT_ID", "client")
|
||||
t.Setenv("TM_PLATFORM_OIDC_REDIRECT_URL", "https://app.example.org/auth/callback")
|
||||
got, line := effective(t)
|
||||
if got["TM_PLATFORM_DSN"].Source != sourceFile || got["TM_PLATFORM_DSN"].Value != valueSet {
|
||||
t.Errorf("DSN printed as %+v", got["TM_PLATFORM_DSN"])
|
||||
}
|
||||
if got["TM_PLATFORM_OIDC_CLIENT_SECRET"].Source != sourceEnv {
|
||||
t.Errorf("client secret printed as %+v", got["TM_PLATFORM_OIDC_CLIENT_SECRET"])
|
||||
}
|
||||
for _, leaked := range []string{"hunter2", "s3cret", path} {
|
||||
if strings.Contains(line, leaked) {
|
||||
t.Errorf("the boot line carries %q", leaked)
|
||||
}
|
||||
}
|
||||
// And the value really was read, not lost by being redacted.
|
||||
c, err := Load()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(c.DSN, "hunter2") || c.OIDCClientSecret != "s3cret" {
|
||||
t.Fatal("the secret was redacted out of the configuration itself, not only out of the log")
|
||||
}
|
||||
}
|
||||
|
||||
// Money does not reach an INFO log in ANY form (D39.84, PD-99) — including the amounts an operator
|
||||
// configured. The source is what they need, and the source is all they get.
|
||||
func TestAConfiguredAmountIsNeverPrinted(t *testing.T) {
|
||||
t.Setenv("TM_PLATFORM_SIGNUP_GRANT_USD", "7.50")
|
||||
t.Setenv("TM_PLATFORM_USD_PER_CHAPTER", "0.0425")
|
||||
got, line := effective(t)
|
||||
for _, k := range []string{"TM_PLATFORM_SIGNUP_GRANT_USD", "TM_PLATFORM_USD_PER_CHAPTER"} {
|
||||
if got[k].Source != sourceEnv {
|
||||
t.Errorf("%s: source %q, want the environment it came from", k, got[k].Source)
|
||||
}
|
||||
if got[k].Value != valueAmount {
|
||||
t.Errorf("%s: value %q, want it withheld", k, got[k].Value)
|
||||
}
|
||||
}
|
||||
for _, amount := range []string{"7.50", "7500000", "0.0425", "42500"} {
|
||||
if strings.Contains(line, amount) {
|
||||
t.Errorf("the boot line carries the amount %q", amount)
|
||||
}
|
||||
}
|
||||
c, err := Load()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if c.SignupGrantMicroUSD != 7_500_000 || c.Runner.PerChapterMicroUSD != 42_500 {
|
||||
t.Fatalf("the amounts did not apply: %d and %d", c.SignupGrantMicroUSD, c.Runner.PerChapterMicroUSD)
|
||||
}
|
||||
}
|
||||
|
||||
// A default and an environment variable holding the same string are the same line without the
|
||||
// source, and telling them apart is the whole question ("did my setting take effect").
|
||||
func TestTheSourceOfEveryValueIsPrintedBesideIt(t *testing.T) {
|
||||
t.Setenv("TM_PLATFORM_ADDR", "127.0.0.1:8080") // exactly the default
|
||||
got, line := effective(t)
|
||||
if got["TM_PLATFORM_ADDR"].Source != sourceEnv {
|
||||
t.Errorf("a value set to the default reads as %q", got["TM_PLATFORM_ADDR"].Source)
|
||||
}
|
||||
if got["TM_PLATFORM_AFTER_LOGIN"].Source != sourceDefault {
|
||||
t.Errorf("an unset value reads as %q", got["TM_PLATFORM_AFTER_LOGIN"].Source)
|
||||
}
|
||||
// One line per setting, as structured fields rather than a rendered sentence.
|
||||
var first map[string]any
|
||||
if err := json.Unmarshal([]byte(strings.SplitN(line, "\n", 2)[0]), &first); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, k := range []string{"key", "value", "source"} {
|
||||
if _, ok := first[k]; !ok {
|
||||
t.Errorf("the boot line has no %q field: %v", k, first)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// An intake root that is not absolute names a different directory to every process that reads a
|
||||
// book's workdir out of the database — the same class as PD-149 on the state directory, refused in
|
||||
// the same place.
|
||||
func TestARelativeBooksDirectoryIsRefusedAtBoot(t *testing.T) {
|
||||
t.Setenv("TM_PLATFORM_BOOKS_DIR", "books")
|
||||
if _, err := Load(); err == nil {
|
||||
t.Fatal("a relative TM_PLATFORM_BOOKS_DIR was accepted")
|
||||
}
|
||||
t.Setenv("TM_PLATFORM_BOOKS_DIR", "/srv/tm/books")
|
||||
c, err := Load()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if c.IntakeEnabled() {
|
||||
t.Fatal("intake is enabled without an engine binary: a deployment that cannot parse would take uploads it can only reject")
|
||||
}
|
||||
t.Setenv("TM_PLATFORM_ENGINE_BIN", "/opt/engine/2026.08.01/tmctl")
|
||||
if c, err = Load(); err != nil || !c.IntakeEnabled() {
|
||||
t.Fatalf("intake stayed off with a root and an engine: %v %+v", err, c.Intake)
|
||||
}
|
||||
}
|
||||
|
||||
// The upload deadline and the intake sweep's grace are a pair: the sweep treats a book that has been
|
||||
// `uploading` longer than its grace as an upload whose request is gone, which is only true while one
|
||||
// request cannot legitimately take that long. Longer, and the sweep deletes the row and the directory
|
||||
// out from under a request still writing into them — so the boot refuses the combination.
|
||||
func TestAnUploadDeadlineLongerThanTheSweepsGraceIsRefused(t *testing.T) {
|
||||
t.Setenv("TM_PLATFORM_BOOKS_DIR", "/srv/tm/books")
|
||||
t.Setenv("TM_PLATFORM_ENGINE_BIN", "/opt/engine/2026.08.01/tmctl")
|
||||
t.Setenv("TM_PLATFORM_UPLOAD_DEADLINE", (books.UploadGrace + time.Minute).String())
|
||||
if _, err := Load(); err == nil {
|
||||
t.Fatal("an upload deadline longer than the sweep's grace was accepted")
|
||||
}
|
||||
t.Setenv("TM_PLATFORM_UPLOAD_DEADLINE", (books.UploadGrace - time.Minute).String())
|
||||
if _, err := Load(); err != nil {
|
||||
t.Fatalf("a deadline inside the grace was refused: %v", err)
|
||||
}
|
||||
}
|
||||
97
platform/internal/gates/toolchain_test.go
Normal file
97
platform/internal/gates/toolchain_test.go
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
// Package gates holds the tests of the zone's BUILD gates — the checks that run before any code
|
||||
// does. They live in their own package because their subject is the Makefile and go.mod rather than
|
||||
// any package's behaviour, and a gate nobody tests is a gate that quietly stops gating: this file
|
||||
// exists because the toolchain floor was raised in three documents and one echo, while the check
|
||||
// itself went on accepting the version it was raised against (re-check of the P5 dofix, FP5-9).
|
||||
package gates
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// zoneRoot is where the Makefile and go.mod live, two levels up from this package.
|
||||
const zoneRoot = "../.."
|
||||
|
||||
// The gate is exercised with versions THIS HOST does not have. That is the whole point of asserting
|
||||
// on the comparison rather than on a run: a gate tested only against the toolchain that happens to be
|
||||
// installed proves nothing about the one it is meant to refuse.
|
||||
func TestTheToolchainGateComparesVersionsRatherThanMatchingThem(t *testing.T) {
|
||||
min := minVersion(t)
|
||||
if min != "1.26.6" {
|
||||
// Not a failure — the floor is allowed to move — but the table below is written against it.
|
||||
t.Logf("GO_MIN_VERSION is %s; the cases below assume the 1.26.x floor", min)
|
||||
}
|
||||
for _, c := range []struct {
|
||||
version string
|
||||
accept bool
|
||||
}{
|
||||
{"go1.26.6", true}, // the floor itself
|
||||
{"go1.26.7", true}, // a later patch
|
||||
{"go1.26.10", true}, // double digits: the regex this replaced ordered 1.26.10 BELOW 1.26.9
|
||||
{"go1.27.0", true}, // a later minor
|
||||
{"go2.0.0", true},
|
||||
{"go1.26.5", false}, // the release the floor was raised against — the case that used to pass
|
||||
{"go1.26.4", false},
|
||||
{"go1.25.9", false},
|
||||
{"go1.9.9", false}, // one-digit minor must not sort above 1.26
|
||||
{"go1.26.6rc1", false}, // a prerelease does not carry the fixes its number promises
|
||||
{"devel go1.27-abcdef", false},
|
||||
} {
|
||||
err := makeTarget(t, "version-check", "GO_VERSION="+c.version)
|
||||
if c.accept && err != nil {
|
||||
t.Errorf("the gate refused %s, which is at or above the floor %s: %v", c.version, min, err)
|
||||
}
|
||||
if !c.accept && err == nil {
|
||||
t.Errorf("the gate ACCEPTED %s, below the floor %s", c.version, min)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The other half of the same floor: the Makefile is a convenience, go.mod is what every build reads —
|
||||
// including one that never calls make. The two must name the same version, or a build that skips the
|
||||
// battery gets a toolchain the battery would have refused.
|
||||
func TestGoModPinsTheSameToolchainTheBatteryDemands(t *testing.T) {
|
||||
src, err := os.ReadFile(filepath.Join(zoneRoot, "go.mod"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m := regexp.MustCompile(`(?m)^toolchain go(\S+)$`).FindSubmatch(src)
|
||||
if m == nil {
|
||||
t.Fatal("go.mod carries no toolchain directive: the floor then lives only in the Makefile, and a build that does not call make ignores it")
|
||||
}
|
||||
if got, want := string(m[1]), minVersion(t); got != want {
|
||||
t.Errorf("go.mod pins the toolchain at %s and the battery demands %s", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func minVersion(t *testing.T) string {
|
||||
t.Helper()
|
||||
src, err := os.ReadFile(filepath.Join(zoneRoot, "Makefile"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m := regexp.MustCompile(`(?m)^GO_MIN_VERSION := (\S+)$`).FindSubmatch(src)
|
||||
if m == nil {
|
||||
t.Fatal("the Makefile no longer declares GO_MIN_VERSION")
|
||||
}
|
||||
return string(m[1])
|
||||
}
|
||||
|
||||
func makeTarget(t *testing.T, args ...string) error {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("make"); err != nil {
|
||||
t.Skip("make is not on this host: the build gates cannot be exercised")
|
||||
}
|
||||
cmd := exec.CommandContext(t.Context(), "make", args...)
|
||||
cmd.Dir = zoneRoot
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil && !strings.Contains(string(out), "required") {
|
||||
t.Fatalf("make %v failed for a reason other than the gate: %s", args, out)
|
||||
}
|
||||
return err
|
||||
}
|
||||
169
platform/internal/httpapi/control_test.go
Normal file
169
platform/internal/httpapi/control_test.go
Normal file
|
|
@ -0,0 +1,169 @@
|
|||
package httpapi
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"textmachine/platform/internal/metrics"
|
||||
"textmachine/platform/internal/pgstore"
|
||||
"textmachine/platform/internal/runs"
|
||||
)
|
||||
|
||||
// The stop and resume handles answer 202 with a Run, and the run they act on is the one in the path.
|
||||
func TestStopAndResumeAnswerWithTheRunTheContractDescribes(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
path string
|
||||
seen func(*fakeRuns) string
|
||||
}{
|
||||
{"stop", func(f *fakeRuns) string { return f.stopped }},
|
||||
{"resume", func(f *fakeRuns) string { return f.resumed }},
|
||||
} {
|
||||
rn := &fakeRuns{run: pgstore.Run{
|
||||
ID: "run_9", Revision: 12, Status: "stopped", VerifyBank: true, CeilingChapters: 100,
|
||||
StartedAt: time.Unix(0, 0).UTC(),
|
||||
}}
|
||||
w := call(t, v0Server(t, &fakeLibrary{}, rn), "POST", "/v0/runs/run_9/"+tc.path, "")
|
||||
if w.Code != http.StatusAccepted {
|
||||
t.Fatalf("%s: status %d, want 202: %s", tc.path, w.Code, w.Body)
|
||||
}
|
||||
got := decode(t, w)
|
||||
for _, k := range []string{"id", "revision", "status", "verify_bank", "ceiling_chapters",
|
||||
"paused_reason", "started_at"} {
|
||||
if _, ok := got[k]; !ok {
|
||||
t.Errorf("%s: the Run is missing the required field %q", tc.path, k)
|
||||
}
|
||||
}
|
||||
if got["paused_reason"] != nil {
|
||||
t.Errorf("%s: paused_reason is %v, want null outside a pause", tc.path, got["paused_reason"])
|
||||
}
|
||||
if id := tc.seen(rn); id != "run_9" {
|
||||
t.Errorf("%s: the service was asked about %q, not the run in the path", tc.path, id)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The four ways these two calls fail, each mapped to the code the contract names. 404 and 409 are
|
||||
// the pair that matters: a run of another account is NOT FOUND, and one that cannot be acted on is a
|
||||
// conflict — swapping them either tells a stranger the run exists or tells an owner theirs does not.
|
||||
func TestTheControlHandlesMapEveryRefusalToItsContractCode(t *testing.T) {
|
||||
cases := []struct {
|
||||
err error
|
||||
want int
|
||||
}{
|
||||
{pgstore.ErrNoRun, http.StatusNotFound},
|
||||
{fmt.Errorf("%w: it is ready", runs.ErrNotStoppable), http.StatusConflict},
|
||||
{fmt.Errorf("%w: the glossary is not signed", runs.ErrNotResumable), http.StatusConflict},
|
||||
{errors.New("pgstore: dial tcp 127.0.0.1:5432: connection refused"), http.StatusInternalServerError},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
h := v0Server(t, &fakeLibrary{}, &fakeRuns{err: tc.err})
|
||||
for _, path := range []string{"stop", "resume"} {
|
||||
w := call(t, h, "POST", "/v0/runs/run_9/"+path, "")
|
||||
if w.Code != tc.want {
|
||||
t.Errorf("%s on %v: %d, want %d", path, tc.err, w.Code, tc.want)
|
||||
}
|
||||
if w.Header().Get("Content-Type") != "application/problem+json" {
|
||||
t.Errorf("%s: content-type %q", path, w.Header().Get("Content-Type"))
|
||||
}
|
||||
if bytes.Contains(w.Body.Bytes(), []byte("5432")) {
|
||||
t.Errorf("%s: the response carries internals: %s", path, w.Body)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Both handles END or RESTART a paid run, so both are unsafe requests on the cookie path and both
|
||||
// need the CSRF header. The audit of P4 called this the weightiest of its findings: the check
|
||||
// survives everything else being right.
|
||||
func TestTheControlHandlesRequireTheCSRFHeaderOnTheCookiePath(t *testing.T) {
|
||||
h := v0Server(t, &fakeLibrary{}, &fakeRuns{})
|
||||
for _, path := range []string{"/v0/runs/run_9/stop", "/v0/runs/run_9/resume", "/v0/books"} {
|
||||
r := httptest.NewRequest("POST", path, nil)
|
||||
r.AddCookie(&http.Cookie{Name: "__Host-tm_session", Value: "token"})
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
if w.Code != http.StatusForbidden {
|
||||
t.Errorf("%s without X-TM-Client: %d, want 403", path, w.Code)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The upload's own deadline, on a REAL server: a book is minutes of a domestic uplink and the
|
||||
// server's ReadTimeout — the whole of what bounds a half-fed request — covers the entire body.
|
||||
//
|
||||
// The negative half is the point of the test. The same slow body on a route that does NOT extend the
|
||||
// deadline is cut off, which is what proves the extension is doing the work rather than the timeout
|
||||
// being generous.
|
||||
func TestTheUploadRouteExtendsItsOwnReadDeadlineAndOnlyItsOwn(t *testing.T) {
|
||||
in := &fakeIntake{book: pgstore.Book{ID: "bk_1", Status: "parsing"}}
|
||||
h := v0ServerWith(t, Deps{
|
||||
Library: &fakeLibrary{}, Runs: &fakeRuns{}, Intake: in,
|
||||
Upload: UploadLimits{MaxBytes: 1 << 20, Deadline: 10 * time.Second},
|
||||
// With the telemetry wrapper in the chain, exactly as the daemon builds it. It is here on
|
||||
// purpose: http.ResponseController walks the wrappers through Unwrap, and a wrapper that
|
||||
// forgets it turns the deadline extension into a silent no-op — the upload would then die at
|
||||
// the server's own ReadTimeout and nothing in this file would notice.
|
||||
Observe: metrics.New().Middleware(),
|
||||
})
|
||||
to := fastTimeouts() // Read: 250ms — far less than the body below takes to arrive
|
||||
ln := start(t, h, to)
|
||||
|
||||
ct, body := form(t, "book.txt", 2048, true)
|
||||
raw, err := io.ReadAll(body)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if code := slowPost(t, ln, "/v0/books", ct, raw); code != http.StatusCreated {
|
||||
t.Fatalf("a slow upload answered %d, want 201: the route's own deadline did not apply", code)
|
||||
}
|
||||
// The same shape of body on the JSON route keeps the server's short ReadTimeout and dies with it.
|
||||
payload := append([]byte(`{"verify_bank":false,"ceiling_chapters":1,"pad":"`),
|
||||
append(bytes.Repeat([]byte("a"), 2048), []byte(`"}`)...)...)
|
||||
if code := slowPost(t, ln, "/v0/books/bk_1/runs", "application/json", payload); code == http.StatusAccepted {
|
||||
t.Fatal("a slow request on an ordinary route was served: the upload's deadline leaked onto it")
|
||||
}
|
||||
}
|
||||
|
||||
// slowPost sends a request whose body arrives in two halves a second apart, which is a slow client
|
||||
// and not a malicious one: the bytes keep coming, just not fast enough for a 250ms deadline.
|
||||
func slowPost(t *testing.T, ln net.Listener, path, contentType string, body []byte) int {
|
||||
t.Helper()
|
||||
var d net.Dialer
|
||||
conn, err := d.DialContext(t.Context(), "tcp", ln.Addr().String())
|
||||
if err != nil {
|
||||
t.Fatalf("dial: %v", err)
|
||||
}
|
||||
defer conn.Close()
|
||||
head := fmt.Sprintf("POST %s HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer token\r\n"+
|
||||
"Content-Type: %s\r\nContent-Length: %d\r\n\r\n", path, contentType, len(body))
|
||||
if _, err := conn.Write([]byte(head)); err != nil {
|
||||
t.Fatalf("write head: %v", err)
|
||||
}
|
||||
half := len(body) / 2
|
||||
if _, err := conn.Write(body[:half]); err != nil {
|
||||
t.Fatalf("write first half: %v", err)
|
||||
}
|
||||
time.Sleep(time.Second)
|
||||
if _, err := conn.Write(body[half:]); err != nil {
|
||||
// A server that gave up has closed the connection; that is an answer too.
|
||||
return 0
|
||||
}
|
||||
if err := conn.SetReadDeadline(time.Now().Add(5 * time.Second)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp, err := http.ReadResponse(bufio.NewReader(conn), nil)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
_, _ = io.Copy(io.Discard, resp.Body)
|
||||
return resp.StatusCode
|
||||
}
|
||||
346
platform/internal/httpapi/intake_test.go
Normal file
346
platform/internal/httpapi/intake_test.go
Normal file
|
|
@ -0,0 +1,346 @@
|
|||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"textmachine/platform/internal/books"
|
||||
"textmachine/platform/internal/pgstore"
|
||||
)
|
||||
|
||||
// The upload is the only streaming route on this surface, and the only one whose body limit is not
|
||||
// the default. Both facts are asserted here on the REAL handler chain — through New, with the guard,
|
||||
// the CSRF layer and the per-route cap in place — because a limit tested on a bare handler is a
|
||||
// limit tested somewhere other than where it applies (the shape of PD-46).
|
||||
|
||||
type fakeIntake struct {
|
||||
book pgstore.Book
|
||||
err error
|
||||
got books.Intake
|
||||
// read is what the service actually pulled out of the part, which is how a test tells "the file
|
||||
// was streamed" from "the handler read it into memory and handed over a copy".
|
||||
read int64
|
||||
}
|
||||
|
||||
func (f *fakeIntake) Accept(_ context.Context, in books.Intake) (pgstore.Book, error) {
|
||||
f.got = in
|
||||
if in.File != nil {
|
||||
n, err := io.Copy(io.Discard, in.File)
|
||||
f.read = n
|
||||
if err != nil {
|
||||
return pgstore.Book{}, err
|
||||
}
|
||||
}
|
||||
if f.err != nil {
|
||||
return pgstore.Book{}, f.err
|
||||
}
|
||||
return f.book, nil
|
||||
}
|
||||
|
||||
// form builds a multipart body. The order of the parts is the caller's on purpose: the file MUST
|
||||
// come last, and the test that says so builds it the other way round.
|
||||
func form(t *testing.T, file string, size int, fieldsFirst bool) (string, io.Reader) {
|
||||
t.Helper()
|
||||
var buf bytes.Buffer
|
||||
w := multipart.NewWriter(&buf)
|
||||
writeFields := func() {
|
||||
for k, v := range map[string]string{"source_lang": "zh", "target_lang": "ru", "genre": "xianxia"} {
|
||||
if err := w.WriteField(k, v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
if fieldsFirst {
|
||||
writeFields()
|
||||
}
|
||||
part, err := w.CreateFormFile("file", file)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := part.Write(bytes.Repeat([]byte("a"), size)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !fieldsFirst {
|
||||
writeFields()
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return w.FormDataContentType(), &buf
|
||||
}
|
||||
|
||||
func upload(t *testing.T, h http.Handler, contentType string, body io.Reader) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
r := httptest.NewRequest("POST", "/v0/books", body)
|
||||
r.Header.Set("Content-Type", contentType)
|
||||
r.Header.Set("Authorization", "Bearer token")
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
return w
|
||||
}
|
||||
|
||||
func TestAnAcceptedUploadAnswersWithTheBookTheContractDescribes(t *testing.T) {
|
||||
in := &fakeIntake{book: pgstore.Book{
|
||||
ID: "bk_7c1", Title: "蛊真人", SourceLang: "zh", TargetLang: "ru", Genre: "xianxia",
|
||||
Status: "parsing", AddedAt: time.Unix(0, 0).UTC(),
|
||||
}}
|
||||
h := v0ServerWith(t, Deps{Library: &fakeLibrary{}, Intake: in})
|
||||
ct, body := form(t, "蛊真人.txt", 1024, true)
|
||||
w := upload(t, h, ct, body)
|
||||
if w.Code != http.StatusCreated {
|
||||
t.Fatalf("status %d: %s", w.Code, w.Body)
|
||||
}
|
||||
got := decode(t, w)
|
||||
// Every field the contract marks required on Book, present on the intake's own response.
|
||||
for _, k := range []string{"id", "title", "source_lang", "target_lang", "status", "chapter_count",
|
||||
"added_at", "progress", "note_count"} {
|
||||
if _, ok := got[k]; !ok {
|
||||
t.Errorf("the created book is missing the required field %q", k)
|
||||
}
|
||||
}
|
||||
if got["status"] != "parsing" {
|
||||
t.Errorf("status %v, want the state the book is actually in", got["status"])
|
||||
}
|
||||
if in.got.SourceLang != "zh" || in.got.TargetLang != "ru" || in.got.Genre != "xianxia" {
|
||||
t.Errorf("the form did not reach the service: %+v", in.got)
|
||||
}
|
||||
if in.got.Filename != "蛊真人.txt" {
|
||||
t.Errorf("filename %q did not reach the service", in.got.Filename)
|
||||
}
|
||||
if in.read != 1024 {
|
||||
t.Errorf("the service was handed %d bytes of the file, want 1024", in.read)
|
||||
}
|
||||
}
|
||||
|
||||
// PD-72, and the reason the register kept the row open until this route landed: the per-route cap is
|
||||
// only OBSERVABLE once a route asks for more than the default, and the test has to arrive with it.
|
||||
func TestAnUploadLargerThanTheRouteAllowsIsRefusedAsTooLarge(t *testing.T) {
|
||||
in := &fakeIntake{}
|
||||
h := v0ServerWith(t, Deps{
|
||||
Library: &fakeLibrary{}, Intake: in,
|
||||
Upload: UploadLimits{MaxBytes: 4096, Deadline: time.Minute},
|
||||
})
|
||||
ct, body := form(t, "book.txt", 16*1024, true)
|
||||
w := upload(t, h, ct, body)
|
||||
if w.Code != http.StatusRequestEntityTooLarge {
|
||||
t.Fatalf("status %d, want 413: %s", w.Code, w.Body)
|
||||
}
|
||||
if ct := w.Header().Get("Content-Type"); ct != "application/problem+json" {
|
||||
t.Errorf("content-type %q, want application/problem+json", ct)
|
||||
}
|
||||
problem := decode(t, w)
|
||||
if problem["status"] != float64(http.StatusRequestEntityTooLarge) || problem["title"] == "" {
|
||||
t.Errorf("problem body %v", problem)
|
||||
}
|
||||
// And a body UNDER the limit goes through the same route untouched: a cap that refuses
|
||||
// everything is not a cap.
|
||||
ct2, body2 := form(t, "book.txt", 1024, true)
|
||||
if w := upload(t, h, ct2, body2); w.Code != http.StatusCreated {
|
||||
t.Fatalf("a 1 KiB upload under a 4 KiB cap: %d %s", w.Code, w.Body)
|
||||
}
|
||||
}
|
||||
|
||||
// The other half of the same property (PD-35): raising the limit for ONE route must not raise it for
|
||||
// the rest of the surface.
|
||||
func TestTheUploadLimitBelongsToTheUploadRouteAlone(t *testing.T) {
|
||||
h := v0ServerWith(t, Deps{
|
||||
Library: &fakeLibrary{}, Runs: &fakeRuns{}, Intake: &fakeIntake{},
|
||||
Upload: UploadLimits{MaxBytes: 8 << 20, Deadline: time.Minute},
|
||||
})
|
||||
big := strings.Repeat("a", DefaultMaxBody+1)
|
||||
w := call(t, h, "POST", "/v0/books/bk_1/runs", `{"verify_bank":false,"ceiling_chapters":1,"pad":"`+big+`"}`)
|
||||
if w.Code == http.StatusAccepted {
|
||||
t.Fatal("a run request larger than the default body cap was accepted: the upload's limit leaked onto every route")
|
||||
}
|
||||
}
|
||||
|
||||
// A deployment with nowhere to put a file mounts no upload route, exactly like every other contract
|
||||
// route it cannot serve — and an anonymous caller still meets 401 before 404.
|
||||
func TestWithoutAnIntakeTheUploadRouteIsAGuarded404(t *testing.T) {
|
||||
h := v0ServerWith(t, Deps{Library: &fakeLibrary{}})
|
||||
ct, body := form(t, "book.txt", 16, true)
|
||||
if w := upload(t, h, ct, body); w.Code != http.StatusNotFound {
|
||||
t.Fatalf("status %d, want 404 where no intake is configured", w.Code)
|
||||
}
|
||||
r := httptest.NewRequest("POST", "/v0/books", nil)
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, r)
|
||||
if w.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("an anonymous caller got %d, want 401 before anything else", w.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// The wire rule a streaming reader forces: the languages have to be known before the file part is
|
||||
// handed over, so a form that sends them afterwards is refused rather than half-processed.
|
||||
func TestAFormWhoseFileComesBeforeItsFieldsIsRefused(t *testing.T) {
|
||||
in := &fakeIntake{}
|
||||
h := v0ServerWith(t, Deps{Library: &fakeLibrary{}, Intake: in})
|
||||
ct, body := form(t, "book.txt", 64, false)
|
||||
w := upload(t, h, ct, body)
|
||||
if w.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status %d, want 400: %s", w.Code, w.Body)
|
||||
}
|
||||
if in.read != 0 {
|
||||
t.Errorf("the file was streamed before the form was known to be usable (%d bytes)", in.read)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUploadWithNoFileAtAllIsRefused(t *testing.T) {
|
||||
h := v0ServerWith(t, Deps{Library: &fakeLibrary{}, Intake: &fakeIntake{}})
|
||||
var buf bytes.Buffer
|
||||
w := multipart.NewWriter(&buf)
|
||||
if err := w.WriteField("source_lang", "zh"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := upload(t, h, w.FormDataContentType(), &buf); got.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status %d, want 400", got.Code)
|
||||
}
|
||||
// And a body that is not multipart at all.
|
||||
r := httptest.NewRequest("POST", "/v0/books", strings.NewReader(`{"not":"multipart"}`))
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
r.Header.Set("Authorization", "Bearer token")
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, r)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("a JSON body on the intake route: %d, want 400", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// A field the contract does not name is IGNORED, like an unknown JSON property: inside 0.x a minor
|
||||
// bump is where optional fields appear, and refusing the upload would break a client generated
|
||||
// against a later version for something it was free to ignore.
|
||||
func TestAnUnknownFormFieldDoesNotRefuseTheUpload(t *testing.T) {
|
||||
in := &fakeIntake{book: pgstore.Book{ID: "bk_1", Status: "parsing"}}
|
||||
h := v0ServerWith(t, Deps{Library: &fakeLibrary{}, Intake: in})
|
||||
var buf bytes.Buffer
|
||||
w := multipart.NewWriter(&buf)
|
||||
for _, kv := range [][2]string{{"source_lang", "zh"}, {"target_lang", "ru"}, {"title", "as yet unratified"}} {
|
||||
if err := w.WriteField(kv[0], kv[1]); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
part, err := w.CreateFormFile("file", "b.txt")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := part.Write([]byte("text")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := upload(t, h, w.FormDataContentType(), &buf); got.Code != http.StatusCreated {
|
||||
t.Fatalf("status %d, want 201: an unknown field must not refuse the upload", got.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// A form field long enough to be a payload of its own is refused rather than truncated.
|
||||
func TestAFormFieldThatIsAPayloadIsRefused(t *testing.T) {
|
||||
h := v0ServerWith(t, Deps{Library: &fakeLibrary{}, Intake: &fakeIntake{},
|
||||
Upload: UploadLimits{MaxBytes: 8 << 20, Deadline: time.Minute}})
|
||||
var buf bytes.Buffer
|
||||
w := multipart.NewWriter(&buf)
|
||||
if err := w.WriteField("genre", strings.Repeat("g", maxIntakeField+1)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := upload(t, h, w.FormDataContentType(), &buf); got.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status %d, want 400", got.Code)
|
||||
}
|
||||
}
|
||||
|
||||
// The service's own refusals reach the wire as the contract's codes and never as engine or database
|
||||
// text (§Problem).
|
||||
func TestWhatTheIntakeRefusesBecomesAProductPhrase(t *testing.T) {
|
||||
cases := []struct {
|
||||
err error
|
||||
want int
|
||||
}{
|
||||
{fmt.Errorf("%w: the languages must be codes", books.ErrBadIntake), http.StatusBadRequest},
|
||||
{pgstore.ErrNoAccount, http.StatusNotFound},
|
||||
{errors.New("pgstore: connection refused: dial tcp 127.0.0.1:5432"), http.StatusInternalServerError},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
h := v0ServerWith(t, Deps{Library: &fakeLibrary{}, Intake: &fakeIntake{err: tc.err}})
|
||||
ct, body := form(t, "b.txt", 16, true)
|
||||
w := upload(t, h, ct, body)
|
||||
if w.Code != tc.want {
|
||||
t.Errorf("%v -> %d, want %d", tc.err, w.Code, tc.want)
|
||||
}
|
||||
if strings.Contains(w.Body.String(), "pgstore") || strings.Contains(w.Body.String(), "5432") {
|
||||
t.Errorf("the response carries internals: %s", w.Body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// FP5-8(в) (acceptance): the cap allowed one part past its own number. The assertion is not the
|
||||
// status — a form that simply runs out of parts is a 400 too, so a status-only test says nothing —
|
||||
// but that the part BEHIND the cap is never handed to the intake. The file sits at exactly
|
||||
// maxIntakeParts+1 here: with the cap off by one it is reached, accepted and streamed, which is the
|
||||
// work the cap exists to refuse.
|
||||
func TestAFormWithTooManyPartsIsRefusedAtTheCap(t *testing.T) {
|
||||
in := &fakeIntake{book: pgstore.Book{ID: "bk_cap"}}
|
||||
h := v0ServerWith(t, Deps{Library: &fakeLibrary{}, Intake: in,
|
||||
Upload: UploadLimits{MaxBytes: 8 << 20, Deadline: time.Minute}})
|
||||
var buf bytes.Buffer
|
||||
w := multipart.NewWriter(&buf)
|
||||
for k, v := range map[string]string{"source_lang": "zh", "target_lang": "ru"} {
|
||||
if err := w.WriteField(k, v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for i := range maxIntakeParts - 2 {
|
||||
if err := w.WriteField(fmt.Sprintf("filler-%d", i), "x"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
fw, err := w.CreateFormFile("file", "book.txt")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := fw.Write(bytes.Repeat([]byte("x"), 1024)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := upload(t, h, w.FormDataContentType(), &buf)
|
||||
if got.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status %d with %d parts, want 400 at the cap of %d", got.Code, maxIntakeParts+1, maxIntakeParts)
|
||||
}
|
||||
if in.read != 0 {
|
||||
t.Fatalf("the file behind the cap was streamed anyway: %d bytes", in.read)
|
||||
}
|
||||
}
|
||||
|
||||
// FP5-6 (acceptance): a body that outlives the route's own deadline is the client being slow, not
|
||||
// this service being broken — and a 500 tells the client the opposite of the truth about retrying.
|
||||
func TestAnUploadThatOutlivesItsDeadlineIsNotAnInternalError(t *testing.T) {
|
||||
h := v0ServerWith(t, Deps{Library: &fakeLibrary{}, Intake: &fakeIntake{err: os.ErrDeadlineExceeded}})
|
||||
ct, body := form(t, "book.txt", 64, true)
|
||||
w := upload(t, h, ct, body)
|
||||
if w.Code == http.StatusInternalServerError {
|
||||
t.Fatalf("an expired upload deadline answered 500: %s", w.Body)
|
||||
}
|
||||
if w.Code != http.StatusRequestTimeout {
|
||||
t.Fatalf("status %d, want 408", w.Code)
|
||||
}
|
||||
if w.Header().Get("Content-Type") != "application/problem+json" {
|
||||
t.Errorf("content-type %q", w.Header().Get("Content-Type"))
|
||||
}
|
||||
}
|
||||
|
|
@ -49,6 +49,40 @@ type Deps struct {
|
|||
// service started without a database serves — the routes below all read one.
|
||||
Library Library
|
||||
Runs Runs
|
||||
// Intake is the book upload. Nil where this deployment has nowhere to put a file or no engine to
|
||||
// cut it with, and then POST /books is a guarded 404 like every other unbuilt route: an instance
|
||||
// that accepted uploads it could only reject would be worse than one that says it takes none.
|
||||
Intake Intake
|
||||
// Upload bounds the one route that carries a file.
|
||||
Upload UploadLimits
|
||||
// Observe wraps every request in this deployment's telemetry. A plain middleware rather than an
|
||||
// interface so this package stays free of a metrics library; nil is a service that measures
|
||||
// nothing.
|
||||
Observe func(http.Handler) http.Handler
|
||||
}
|
||||
|
||||
// UploadLimits is what the intake route is allowed to cost.
|
||||
//
|
||||
// Both numbers are the operator's, and both are refusals rather than sizings: a book is tens of
|
||||
// megabytes and minutes of a domestic connection, and the defaults of the surface — a one-megabyte
|
||||
// body and a thirty-second read — are correct for every OTHER route and would make this one
|
||||
// impossible.
|
||||
type UploadLimits struct {
|
||||
// MaxBytes caps the request body of the intake route (http.MaxBytesReader, the second half of
|
||||
// PD-2 and the whole of PD-72).
|
||||
MaxBytes int64
|
||||
// Deadline is how long the body of ONE upload may take to arrive. It replaces the server's
|
||||
// ReadTimeout for this route through http.ResponseController and is still a finite bound — the
|
||||
// forbidden thing is CLEARING the deadline, which puts a half-fed request back to being
|
||||
// unbounded (STACK_DECISIONS §12, register row PD-51).
|
||||
Deadline time.Duration
|
||||
}
|
||||
|
||||
// DefaultUploadLimits are the intake's bounds when an operator chooses none: a book file large
|
||||
// enough for the acceptance corpus (23 MB) with room around it, and long enough to arrive over a
|
||||
// slow domestic uplink.
|
||||
func DefaultUploadLimits() UploadLimits {
|
||||
return UploadLimits{MaxBytes: 64 << 20, Deadline: 10 * time.Minute}
|
||||
}
|
||||
|
||||
// New builds the handler.
|
||||
|
|
@ -71,10 +105,16 @@ func New(d Deps) (http.Handler, error) {
|
|||
//
|
||||
// The body limit is per ROUTE, not a blanket outer layer: MaxBytesReader wrapping an already
|
||||
// wrapped body keeps the tighter limit, so an upload route could never raise its own above a
|
||||
// shared default. The book upload registers guard(maxUpload, …) when it lands.
|
||||
// shared default. The book upload is the route that needed it and registers guard(MaxBytes, …).
|
||||
guard := func(maxBody int64, h http.Handler) http.Handler {
|
||||
return LimitBody(maxBody)(csrf(d.Auth.Require(h)))
|
||||
}
|
||||
if d.Upload.MaxBytes <= 0 || d.Upload.Deadline <= 0 {
|
||||
// A zero here would be a route with no bound at all — MaxBytesReader treats a negative count
|
||||
// as zero and a zero deadline as none — so it is filled rather than trusted. The intake is the
|
||||
// one route where an unset limit is a hole and not an inconvenience.
|
||||
d.Upload = DefaultUploadLimits()
|
||||
}
|
||||
|
||||
mux := http.NewServeMux()
|
||||
mux.Handle("GET /healthz", http.HandlerFunc(healthz))
|
||||
|
|
@ -94,8 +134,13 @@ func New(d Deps) (http.Handler, error) {
|
|||
mux.Handle(APIPrefix+"/", guard(DefaultMaxBody, ProblemHandler(http.StatusNotFound, "Object not found")))
|
||||
|
||||
// Recover sits INSIDE AccessLog: a panic converted to a 500 still produces a log line, whereas
|
||||
// a panic unwinding past the logger produces none.
|
||||
return reqid.Middleware(SecurityHeaders(d.HSTS)(AccessLog(d.Log)(Recover(d.Log)(mux)))), nil
|
||||
// a panic unwinding past the logger produces none. Telemetry sits beside the log and for the same
|
||||
// reason — it reads the route the mux matched and the status the recovery produced.
|
||||
served := http.Handler(Recover(d.Log)(mux))
|
||||
if d.Observe != nil {
|
||||
served = d.Observe(served)
|
||||
}
|
||||
return reqid.Middleware(SecurityHeaders(d.HSTS)(AccessLog(d.Log)(served))), nil
|
||||
}
|
||||
|
||||
// healthz is liveness: the process is up and serving. It touches nothing, so a database outage
|
||||
|
|
|
|||
|
|
@ -4,12 +4,17 @@ import (
|
|||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"textmachine/platform/internal/auth"
|
||||
"textmachine/platform/internal/books"
|
||||
"textmachine/platform/internal/pgstore"
|
||||
"textmachine/platform/internal/pricing"
|
||||
"textmachine/platform/internal/runner"
|
||||
|
|
@ -28,6 +33,14 @@ type Library interface {
|
|||
type Runs interface {
|
||||
Bounds(ctx context.Context, userID, bookID string) (pricing.Bounds, error)
|
||||
Start(ctx context.Context, in runs.StartRequest) (pgstore.Run, error)
|
||||
Stop(ctx context.Context, userID, runID string) (pgstore.Run, error)
|
||||
Resume(ctx context.Context, userID, runID string) (pgstore.Run, error)
|
||||
}
|
||||
|
||||
// Intake is the book upload, as the HTTP layer needs to see it. The service takes a READER: the
|
||||
// file is streamed to its place on disk and is never held in this process's memory.
|
||||
type Intake interface {
|
||||
Accept(ctx context.Context, in books.Intake) (pgstore.Book, error)
|
||||
}
|
||||
|
||||
// contractRoutes registers the /v0 surface.
|
||||
|
|
@ -36,7 +49,7 @@ type Runs interface {
|
|||
// ops endpoints: a nested mux behind StripPrefix hands the inner handler a copy of the request and
|
||||
// the pattern never comes back, which would put raw paths carrying book ids into the access log.
|
||||
func contractRoutes(mux *http.ServeMux, d Deps, guard func(int64, http.Handler) http.Handler) {
|
||||
h := &v0{lib: d.Library, runs: d.Runs, log: d.Log}
|
||||
h := &v0{lib: d.Library, runs: d.Runs, intake: d.Intake, upload: d.Upload, log: d.Log}
|
||||
// The READS need a read model and nothing else. An instance with no engine binary is a read
|
||||
// replica, not a broken one, and mounting nothing unless it could ALSO start runs made it answer
|
||||
// 404 to a library it was holding — while its own boot line said it was serving one.
|
||||
|
|
@ -50,13 +63,26 @@ func contractRoutes(mux *http.ServeMux, d Deps, guard func(int64, http.Handler)
|
|||
if d.Runs != nil {
|
||||
mux.Handle("GET "+APIPrefix+"/books/{bookId}/run-options", guard(DefaultMaxBody, http.HandlerFunc(h.runOptions)))
|
||||
mux.Handle("POST "+APIPrefix+"/books/{bookId}/runs", guard(DefaultMaxBody, http.HandlerFunc(h.startRun)))
|
||||
mux.Handle("POST "+APIPrefix+"/runs/{runId}/stop", guard(DefaultMaxBody, http.HandlerFunc(h.stopRun)))
|
||||
mux.Handle("POST "+APIPrefix+"/runs/{runId}/resume", guard(DefaultMaxBody, http.HandlerFunc(h.resumeRun)))
|
||||
}
|
||||
// Intake needs somewhere to put a file and an engine to cut it with; an instance with neither is
|
||||
// the read replica again, and its /books stays a guarded 404 for POST.
|
||||
//
|
||||
// This is the ONE route whose body limit is not the default, and it is why the limit is a
|
||||
// per-route argument in the first place (PD-35/PD-72): a book is tens of megabytes and every
|
||||
// other route on this surface carries a few kilobytes of JSON.
|
||||
if d.Intake != nil {
|
||||
mux.Handle("POST "+APIPrefix+"/books", guard(d.Upload.MaxBytes, http.HandlerFunc(h.createBook)))
|
||||
}
|
||||
}
|
||||
|
||||
type v0 struct {
|
||||
lib Library
|
||||
runs Runs
|
||||
log *slog.Logger
|
||||
lib Library
|
||||
runs Runs
|
||||
intake Intake
|
||||
upload UploadLimits
|
||||
log *slog.Logger
|
||||
}
|
||||
|
||||
// The wire shapes below are the contract's, field for field (openapi 0.2.0). They are written out
|
||||
|
|
@ -169,14 +195,13 @@ func (h *v0) getBook(w http.ResponseWriter, r *http.Request) {
|
|||
h.fail(w, r, err)
|
||||
return
|
||||
}
|
||||
// ONE counter per book, and it is the book's (contract §Revision: "every book-scoped read and the
|
||||
// id of every stream frame of that book's run carry the same number"). Reading it off the run row
|
||||
// made the card lag: a unit_done bumps the book and the chapter and not the run, so a client that
|
||||
// had applied frame id=2 got revision 0 back and, obeying the contract, dropped the read.
|
||||
// ONE counter per book (contract §Revision: "every book-scoped read and the id of every stream
|
||||
// frame of that book's run carry the same number"). The store answers a run's revision from its
|
||||
// BOOK on every path that hands one out — the card here, and the stop and resume handles — so this
|
||||
// layer projects what it was given rather than carrying a second copy of the rule.
|
||||
out := wireBookDetail{Revision: book.Revision, Book: projectBook(book)}
|
||||
if run != nil {
|
||||
wr := projectRun(*run)
|
||||
wr.Revision = book.Revision
|
||||
out.Run = &wr
|
||||
}
|
||||
writeJSON(w, r, http.StatusOK, out, h.log)
|
||||
|
|
@ -238,6 +263,169 @@ func (h *v0) startRun(w http.ResponseWriter, r *http.Request) {
|
|||
writeJSON(w, r, http.StatusAccepted, projectRun(run), h.log)
|
||||
}
|
||||
|
||||
// stopRun is the product "stop" action (contract §stopRun). The engine stops gracefully on a signal
|
||||
// and the reconciler turns the exit into a status; what this call does is record that the stop was
|
||||
// OURS — which is the only way the exit can afterwards be told from a crash (register row PD-152).
|
||||
func (h *v0) stopRun(w http.ResponseWriter, r *http.Request) {
|
||||
user, ok := principal(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
run, err := h.runs.Stop(r.Context(), user, r.PathValue("runId"))
|
||||
if err != nil {
|
||||
h.fail(w, r, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, r, http.StatusAccepted, projectRun(run), h.log)
|
||||
}
|
||||
|
||||
// resumeRun continues a run that was stopped (contract §resumeRun).
|
||||
func (h *v0) resumeRun(w http.ResponseWriter, r *http.Request) {
|
||||
user, ok := principal(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
run, err := h.runs.Resume(r.Context(), user, r.PathValue("runId"))
|
||||
if err != nil {
|
||||
h.fail(w, r, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, r, http.StatusAccepted, projectRun(run), h.log)
|
||||
}
|
||||
|
||||
// maxIntakeField bounds one text field of the intake form, and maxIntakeParts how many parts may
|
||||
// arrive before the file. Neither is the contract's business: they are what keeps a form with a
|
||||
// megabyte-long genre, or a hundred thousand empty parts, from being work this process does.
|
||||
const (
|
||||
maxIntakeField = 1 << 10
|
||||
maxIntakeParts = 16
|
||||
)
|
||||
|
||||
// createBook receives a book (contract §createBook).
|
||||
//
|
||||
// It is the only route on this surface that streams. The body is read part by part through
|
||||
// r.MultipartReader and the file is handed to the intake as a READER, so a 60 MB upload costs a
|
||||
// buffer and not 60 MB of this process — which is what ParseMultipartForm, the reflex alternative,
|
||||
// would have cost in memory or in a second copy through a temporary file.
|
||||
//
|
||||
// ⚠ The FILE PART MUST COME LAST. A streaming reader hands parts over in wire order, and the book's
|
||||
// row — which is what makes an upload visible while it arrives and findable when it dies halfway —
|
||||
// cannot be written before the languages that row requires. The same constraint is how S3's own
|
||||
// browser upload is specified ("the file or content must be the last field in the form"). It is a
|
||||
// wire rule the contract does not yet state; raised to the contract's owner rather than resolved
|
||||
// here (zone journal, P5).
|
||||
func (h *v0) createBook(w http.ResponseWriter, r *http.Request) {
|
||||
user, ok := principal(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
// A book is minutes of upload on a domestic connection, and the server's ReadTimeout — the whole
|
||||
// of what bounds a half-fed request (PD-2) — covers the entire body. It is EXTENDED here for this
|
||||
// route and never cleared: net/http's own documentation points at a per-request deadline for
|
||||
// exactly this case, and clearing one instead is the mistake that re-created PD-2 (STACK §12).
|
||||
if err := http.NewResponseController(w).SetReadDeadline(time.Now().Add(h.upload.Deadline)); err != nil {
|
||||
// Not fatal: a ResponseWriter that cannot carry a deadline is a test double, and the real
|
||||
// server's own timeout then still applies.
|
||||
h.log.DebugContext(r.Context(), "upload deadline not extended", "err", err)
|
||||
}
|
||||
parts, err := r.MultipartReader()
|
||||
if err != nil {
|
||||
WriteProblem(w, http.StatusBadRequest, "Request could not be read", "")
|
||||
return
|
||||
}
|
||||
in := books.Intake{UserID: user}
|
||||
for n := 0; ; n++ {
|
||||
if n >= maxIntakeParts {
|
||||
WriteProblem(w, http.StatusBadRequest, "Request could not be read", "")
|
||||
return
|
||||
}
|
||||
part, err := parts.NextPart()
|
||||
if errors.Is(err, io.EOF) {
|
||||
// Every field and no file: the form the contract requires was not sent.
|
||||
WriteProblem(w, http.StatusBadRequest, "The upload is incomplete", "")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
h.uploadFailed(w, r, err)
|
||||
return
|
||||
}
|
||||
if part.FormName() == "file" {
|
||||
if in.SourceLang == "" || in.TargetLang == "" {
|
||||
// Either the languages were not sent, or they were sent AFTER the file — and the
|
||||
// second is indistinguishable from the first to a reader that streams.
|
||||
WriteProblem(w, http.StatusBadRequest, "The upload is incomplete", "")
|
||||
return
|
||||
}
|
||||
in.Filename, in.File = part.FileName(), part
|
||||
break
|
||||
}
|
||||
value, err := readField(part)
|
||||
if err != nil {
|
||||
h.uploadFailed(w, r, err)
|
||||
return
|
||||
}
|
||||
switch part.FormName() {
|
||||
case "source_lang":
|
||||
in.SourceLang = value
|
||||
case "target_lang":
|
||||
in.TargetLang = value
|
||||
case "genre":
|
||||
in.Genre = value
|
||||
}
|
||||
// An unknown field is IGNORED rather than refused, for the same reason an unknown JSON
|
||||
// property is: inside 0.x a minor bump is where optional fields appear, and a server that
|
||||
// rejected the whole upload would break a client generated against a later contract.
|
||||
}
|
||||
book, err := h.intake.Accept(r.Context(), in)
|
||||
if err != nil {
|
||||
h.uploadFailed(w, r, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, r, http.StatusCreated, projectBook(book), h.log)
|
||||
}
|
||||
|
||||
// readField reads one text field of the form, refusing one that is too long rather than silently
|
||||
// keeping its first kilobyte.
|
||||
func readField(part io.Reader) (string, error) {
|
||||
b, err := io.ReadAll(io.LimitReader(part, maxIntakeField+1))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(b) > maxIntakeField {
|
||||
return "", fmt.Errorf("%w: a form field is too long", books.ErrBadIntake)
|
||||
}
|
||||
return strings.TrimSpace(string(b)), nil
|
||||
}
|
||||
|
||||
// uploadFailed maps the ways an upload ends badly.
|
||||
//
|
||||
// The size refusal is the reason PD-72 was opened and is closed with this route: MaxBytesReader is
|
||||
// what enforces the cap, and *http.MaxBytesError is how it says so — the same error whether the
|
||||
// client announced the size or simply kept sending.
|
||||
func (h *v0) uploadFailed(w http.ResponseWriter, r *http.Request, err error) {
|
||||
var tooLarge *http.MaxBytesError
|
||||
switch {
|
||||
case errors.As(err, &tooLarge):
|
||||
WriteProblem(w, http.StatusRequestEntityTooLarge, "The file is larger than this service accepts", "")
|
||||
case errors.Is(err, os.ErrDeadlineExceeded):
|
||||
// The body did not finish inside the route's own deadline. 408 is what RFC 9110 §15.5.9 calls
|
||||
// exactly this ("the server did not receive a complete request message within the time it was
|
||||
// prepared to wait"), and it tells a client that RETRYING is the remedy — which a 500 does not.
|
||||
// ⚠ Outside the codes the spec enumerates for this operation; named in the question package of
|
||||
// PD-180 rather than chosen silently.
|
||||
h.log.InfoContext(r.Context(), "upload did not finish inside the route's deadline")
|
||||
WriteProblem(w, http.StatusRequestTimeout, "The upload did not finish in time", "")
|
||||
case errors.Is(err, books.ErrBadIntake):
|
||||
WriteProblem(w, http.StatusBadRequest, "The upload is incomplete", "")
|
||||
case errors.Is(err, context.Canceled), errors.Is(err, io.ErrUnexpectedEOF), errors.Is(err, io.EOF):
|
||||
// The client went away mid-body. Nothing reaches it; the line is what an operator sees.
|
||||
h.log.InfoContext(r.Context(), "upload did not finish", "err", err)
|
||||
WriteProblem(w, http.StatusBadRequest, "The upload is incomplete", "")
|
||||
default:
|
||||
h.fail(w, r, err)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *v0) usage(w http.ResponseWriter, r *http.Request) {
|
||||
user, ok := principal(w, r)
|
||||
if !ok {
|
||||
|
|
@ -331,12 +519,20 @@ func pageLimit(w http.ResponseWriter, r *http.Request) (int, bool) {
|
|||
// the sentence a reader gets.
|
||||
func (h *v0) fail(w http.ResponseWriter, r *http.Request, err error) {
|
||||
switch {
|
||||
case errors.Is(err, pgstore.ErrNoBook), errors.Is(err, pgstore.ErrNoAccount):
|
||||
case errors.Is(err, pgstore.ErrNoBook), errors.Is(err, pgstore.ErrNoAccount), errors.Is(err, pgstore.ErrNoRun):
|
||||
WriteProblem(w, http.StatusNotFound, "Object not found", "")
|
||||
case errors.Is(err, pgstore.ErrBadCursor):
|
||||
case errors.Is(err, pgstore.ErrBadCursor), errors.Is(err, books.ErrBadIntake):
|
||||
WriteProblem(w, http.StatusBadRequest, "Request could not be read", "")
|
||||
case errors.Is(err, pgstore.ErrRunInFlight):
|
||||
WriteProblem(w, http.StatusConflict, "This book is already being translated", "")
|
||||
case errors.Is(err, runs.ErrBookNotReady):
|
||||
// The book is still being received or was rejected. 409 and not 404: the book exists and the
|
||||
// client can see it — what it cannot do is start a translation of it yet.
|
||||
WriteProblem(w, http.StatusConflict, "This book is not ready to be translated", "")
|
||||
case errors.Is(err, runs.ErrNotStoppable):
|
||||
WriteProblem(w, http.StatusConflict, "This run is not running", "")
|
||||
case errors.Is(err, runs.ErrNotResumable):
|
||||
WriteProblem(w, http.StatusConflict, "This translation cannot be continued yet", "")
|
||||
case errors.Is(err, runs.ErrCeilingOutOfBounds), errors.Is(err, pgstore.ErrInsufficientCredit):
|
||||
// 409 and not 400: the request was legal when the bounds were read, and a hold taken for
|
||||
// another book between that read and this call is what moved them (contract §startRun).
|
||||
|
|
|
|||
|
|
@ -47,6 +47,10 @@ type fakeRuns struct {
|
|||
run pgstore.Run
|
||||
err error
|
||||
got runs.StartRequest
|
||||
// stopped and resumed record which run each control handle was called for, so a test can assert
|
||||
// that the path parameter reaches the service rather than only that the status code is right.
|
||||
stopped string
|
||||
resumed string
|
||||
}
|
||||
|
||||
func (f *fakeRuns) Bounds(context.Context, string, string) (pricing.Bounds, error) {
|
||||
|
|
@ -58,17 +62,32 @@ func (f *fakeRuns) Start(_ context.Context, in runs.StartRequest) (pgstore.Run,
|
|||
return f.run, f.err
|
||||
}
|
||||
|
||||
func (f *fakeRuns) Stop(_ context.Context, _, runID string) (pgstore.Run, error) {
|
||||
f.stopped = runID
|
||||
return f.run, f.err
|
||||
}
|
||||
|
||||
func (f *fakeRuns) Resume(_ context.Context, _, runID string) (pgstore.Run, error) {
|
||||
f.resumed = runID
|
||||
return f.run, f.err
|
||||
}
|
||||
|
||||
func v0Server(t *testing.T, lib Library, rn Runs) http.Handler {
|
||||
t.Helper()
|
||||
h, err := New(Deps{
|
||||
Log: slog.New(slog.NewJSONHandler(&bytes.Buffer{}, nil)),
|
||||
Auth: &auth.Authenticator{
|
||||
Sessions: liveSessions{}, IdleTTL: time.Hour,
|
||||
Deny: ProblemHandler(http.StatusUnauthorized, "Session missing or invalid"),
|
||||
},
|
||||
Library: lib,
|
||||
Runs: rn,
|
||||
})
|
||||
return v0ServerWith(t, Deps{Library: lib, Runs: rn})
|
||||
}
|
||||
|
||||
// v0ServerWith builds the real handler chain with whatever this test wants mounted on it. Every
|
||||
// wire test goes through New — not through a handler in isolation — because the guard, the body cap
|
||||
// and the security headers are part of the surface being asserted.
|
||||
func v0ServerWith(t *testing.T, d Deps) http.Handler {
|
||||
t.Helper()
|
||||
d.Log = slog.New(slog.NewJSONHandler(&bytes.Buffer{}, nil))
|
||||
d.Auth = &auth.Authenticator{
|
||||
Sessions: liveSessions{}, IdleTTL: time.Hour,
|
||||
Deny: ProblemHandler(http.StatusUnauthorized, "Session missing or invalid"),
|
||||
}
|
||||
h, err := New(d)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
|
@ -435,7 +454,16 @@ func TestAnUnknownRequestPropertyIsIgnoredRatherThanRefused(t *testing.T) {
|
|||
// ONE counter per book. Reading the card's revision off the RUN row made it lag: a unit_done bumps
|
||||
// the book and its chapter and not the run, so a client that had applied stream frame id=2 got 0
|
||||
// back and — obeying the contract — dropped the read.
|
||||
func TestTheCardsRevisionIsTheBooksAndNotTheRuns(t *testing.T) {
|
||||
// ONE counter per book on the wire. ⚠ The RULE moved in P5 and this test moved with it: the store now
|
||||
// answers a run's revision from its book on every path that hands one out — the card here and the
|
||||
// stop/resume handles, which have no book to override from — so the pin on the rule itself lives in
|
||||
// `pgstore.TestEveryRunTheStoreHandsOutCarriesItsBooksRevision` and `runs.TestEveryRunCarryingAnswerUsesTheBooksRevision`.
|
||||
// What THIS layer owes is that it projects the number it was given and invents nothing, which is
|
||||
// what a fake store can prove and the rule itself no longer is.
|
||||
func TestTheCardProjectsTheRevisionTheStoreGaveAndInventsNone(t *testing.T) {
|
||||
// The two numbers are DIFFERENT on purpose: equal ones cannot tell "projected what it was given"
|
||||
// from "overrode it with the book's", which is exactly the discriminating power an earlier
|
||||
// revision of this test lost when the fixture was levelled (found by acceptance).
|
||||
lib := &fakeLibrary{
|
||||
book: pgstore.Book{ID: "bk_1", Revision: 41},
|
||||
run: &pgstore.Run{ID: "run_1", Status: "translating", Revision: 12},
|
||||
|
|
@ -445,8 +473,9 @@ func TestTheCardsRevisionIsTheBooksAndNotTheRuns(t *testing.T) {
|
|||
t.Errorf("BookDetail.revision = %v, want the book's 41", got["revision"])
|
||||
}
|
||||
run, _ := got["run"].(map[string]any)
|
||||
if run["revision"] != float64(41) {
|
||||
t.Errorf("Run.revision = %v, want the same book counter", run["revision"])
|
||||
if run["revision"] != float64(12) {
|
||||
t.Errorf("Run.revision = %v, want the 12 the store handed over: this layer projects, it does not decide",
|
||||
run["revision"])
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
54
platform/internal/ingest/manifest.go
Normal file
54
platform/internal/ingest/manifest.go
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
package ingest
|
||||
|
||||
import (
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// Manifest is the ALLOWLISTED summary of the engine's chapter manifest (`tmctl manifest --json`,
|
||||
// unified backlog row 100, form ratified with D39.122 as `tm-manifest-v2`).
|
||||
//
|
||||
// What is deliberately absent is most of the document: the per-chapter and per-unit arrays carry the
|
||||
// engine's stable ids, its dense ordinals and its rendered headings, and none of them has a reader
|
||||
// on this side yet — the platform's chapter tree is written by the materializer from the stream, not
|
||||
// from here. Taking them now would store engine vocabulary nobody asks for and would have to be
|
||||
// maintained against a document that re-cuts itself.
|
||||
//
|
||||
// What IS taken is what intake decides with: how many chapters the book has (the ceiling scale is
|
||||
// built on it), and the two identifiers that say WHICH cut of WHICH bytes produced that number.
|
||||
type Manifest struct {
|
||||
// Version is `manifest_version`. Recorded and logged rather than gated: the fields below are
|
||||
// counts and identities whose meaning is stable, and the engine already refuses the version it
|
||||
// cannot produce. A shape this platform truly could not read arrives as a zero chapter count,
|
||||
// which is a loud failure and not a silent one.
|
||||
Version string `json:"manifest_version"`
|
||||
ChaptersTotal int `json:"chapters_total"`
|
||||
UnitsTotal int `json:"units_total"`
|
||||
// SourceSHA256 is the digest of the ingested source, hex. It answers "is the file on disk still
|
||||
// the one that was cut" without the platform reading the file again.
|
||||
SourceSHA256 string `json:"source_sha256"`
|
||||
SourceBytes int64 `json:"source_bytes"`
|
||||
// ChunkerVersion is one of the inputs of the manifest's validity key: a change re-numbers
|
||||
// chapters, so a stored tree is only comparable within one of these (register row PD-166).
|
||||
ChunkerVersion string `json:"chunker_version"`
|
||||
}
|
||||
|
||||
// SourceSHA256Bytes is the digest as the column stores it. An unparsable value is stored as nothing
|
||||
// rather than as garbage: the field is evidence, and evidence that cannot be decoded is absence.
|
||||
func (m Manifest) SourceSHA256Bytes() []byte {
|
||||
b, err := hex.DecodeString(m.SourceSHA256)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// DecodeManifest parses a manifest document.
|
||||
func DecodeManifest(b []byte) (Manifest, error) {
|
||||
var m Manifest
|
||||
if err := json.Unmarshal(b, &m); err != nil {
|
||||
return Manifest{}, fmt.Errorf("ingest: decode manifest: %w", err)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
183
platform/internal/jobs/jobs.go
Normal file
183
platform/internal/jobs/jobs.go
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
// Package jobs is the platform's queue: one River client, and the job kinds the control plane hands
|
||||
// to itself.
|
||||
//
|
||||
// It knows nothing about what a job MEANS. Both workers below call one method of one service and
|
||||
// return, because the work itself belongs to the package that owns the object — the run lifecycle to
|
||||
// `runs`, the intake to `books` — and a queue that also owned the work would be a second place where
|
||||
// a run can be started.
|
||||
//
|
||||
// One client for both kinds, deliberately: River runs its own maintenance (scheduler, rescuer,
|
||||
// cleaner) per client, and a second client in one process would be a second set of those against the
|
||||
// same tables for no gain.
|
||||
package jobs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"github.com/riverqueue/river"
|
||||
"github.com/riverqueue/river/riverdriver/riverpgxv5"
|
||||
|
||||
"textmachine/platform/internal/pgstore"
|
||||
)
|
||||
|
||||
// Spawner starts the engine for a run that has already been admitted.
|
||||
type Spawner interface {
|
||||
Spawn(ctx context.Context, runID string) error
|
||||
}
|
||||
|
||||
// Parser turns a received book file into a chapter tree.
|
||||
type Parser interface {
|
||||
Parse(ctx context.Context, bookID string) error
|
||||
}
|
||||
|
||||
// SpawnArgs is one queued permission to start a run.
|
||||
//
|
||||
// The job carries an id and nothing else. Everything about the run — the book, the ceiling, the
|
||||
// binary it is pinned to — is in Postgres, and a job that carried its own copy would be a second
|
||||
// answer that goes stale the moment the run is restarted with a smaller budget.
|
||||
type SpawnArgs struct {
|
||||
RunID string `json:"run_id"`
|
||||
}
|
||||
|
||||
// Kind is River's name for this job type.
|
||||
func (SpawnArgs) Kind() string { return "tm_spawn_run" }
|
||||
|
||||
// InsertOpts pins the queue-level policy.
|
||||
//
|
||||
// MaxAttempts is 1 on purpose, and it is the opposite of the usual queue reflex. A retry here does
|
||||
// not repeat lost work: the run row already exists and holds the account's money, and the thing that
|
||||
// would be repeated is spawning an engine. What recovers a run whose spawn failed is the reconciler,
|
||||
// which reads the world instead of assuming the job's view of it — and which is the ONLY component
|
||||
// that can tell "the unit never started" from "the unit is running and this platform was restarted".
|
||||
func (SpawnArgs) InsertOpts() river.InsertOpts {
|
||||
return river.InsertOpts{MaxAttempts: 1, Queue: river.QueueDefault}
|
||||
}
|
||||
|
||||
// ParseArgs is one queued book waiting to be cut into chapters.
|
||||
type ParseArgs struct {
|
||||
BookID string `json:"book_id"`
|
||||
}
|
||||
|
||||
// Kind is River's name for this job type.
|
||||
func (ParseArgs) Kind() string { return "tm_parse_book" }
|
||||
|
||||
// InsertOpts pins the queue-level policy.
|
||||
//
|
||||
// MaxAttempts is 1 for the same reason as the spawn's, arrived at from the other direction: a parse
|
||||
// is free and idempotent, so a retry would be harmless — but it would also be a SECOND recovery
|
||||
// mechanism next to the intake sweep, which already re-claims a parse whose process is gone and
|
||||
// which is the only one that can bound how many times a broken host re-runs the engine.
|
||||
func (ParseArgs) InsertOpts() river.InsertOpts {
|
||||
return river.InsertOpts{MaxAttempts: 1, Queue: river.QueueDefault}
|
||||
}
|
||||
|
||||
type spawnWorker struct {
|
||||
river.WorkerDefaults[SpawnArgs]
|
||||
svc Spawner
|
||||
}
|
||||
|
||||
func (w *spawnWorker) Work(ctx context.Context, job *river.Job[SpawnArgs]) error {
|
||||
return w.svc.Spawn(ctx, job.Args.RunID)
|
||||
}
|
||||
|
||||
type parseWorker struct {
|
||||
river.WorkerDefaults[ParseArgs]
|
||||
svc Parser
|
||||
}
|
||||
|
||||
func (w *parseWorker) Work(ctx context.Context, job *river.Job[ParseArgs]) error {
|
||||
return w.svc.Parse(ctx, job.Args.BookID)
|
||||
}
|
||||
|
||||
// JobTimeout bounds ONE job. Generous because the work is an engine call over a whole book, and
|
||||
// finite because a worker that never returns is a worker slot that never comes back.
|
||||
//
|
||||
// Exported because it is half of a pair: the intake's claim grace must OUTLIVE it, or a backstop
|
||||
// sweep takes the claim off a parse this queue is still running. The other half is written as this
|
||||
// constant plus a margin.
|
||||
const JobTimeout = 15 * time.Minute
|
||||
|
||||
// Queue is the River client, wired to the services that do the work.
|
||||
type Queue struct {
|
||||
client *river.Client[pgstore.Tx]
|
||||
}
|
||||
|
||||
// New builds the queue and its worker pool.
|
||||
//
|
||||
// Concurrency is deliberately small: a spawn worker's whole job is to create a transient unit, and
|
||||
// the runs themselves are bounded by their own cgroups and by the one-live-run-per-book index, not
|
||||
// by how many workers exist.
|
||||
func New(pool *pgxpool.Pool, spawner Spawner, parser Parser, log *slog.Logger, workers int) (*Queue, error) {
|
||||
if workers <= 0 {
|
||||
workers = 4
|
||||
}
|
||||
w := river.NewWorkers()
|
||||
if spawner != nil {
|
||||
if err := river.AddWorkerSafely(w, &spawnWorker{svc: spawner}); err != nil {
|
||||
return nil, fmt.Errorf("jobs: register spawn worker: %w", err)
|
||||
}
|
||||
}
|
||||
if parser != nil {
|
||||
if err := river.AddWorkerSafely(w, &parseWorker{svc: parser}); err != nil {
|
||||
return nil, fmt.Errorf("jobs: register parse worker: %w", err)
|
||||
}
|
||||
}
|
||||
c, err := river.NewClient(riverpgxv5.New(pool), &river.Config{
|
||||
Logger: log,
|
||||
Workers: w,
|
||||
Queues: map[string]river.QueueConfig{river.QueueDefault: {MaxWorkers: workers}},
|
||||
// CHOSEN, not inherited. River's own default is one minute, and both jobs here call the engine
|
||||
// against a whole book: a parse ingests and cuts a source of up to the intake's limit, and a
|
||||
// spawn reads the book's meter first (seconds of CPU on a 23 MB book, unified backlog row 100).
|
||||
// A minute would kill those, and a killed process is indistinguishable from a host that cannot
|
||||
// run the engine — so the intake would count a deployment fault against a book whose only sin
|
||||
// was being large.
|
||||
JobTimeout: JobTimeout,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("jobs: river client: %w", err)
|
||||
}
|
||||
return &Queue{client: c}, nil
|
||||
}
|
||||
|
||||
// EnqueueRun inserts the job in the CALLER's transaction, so the run row, its hold and its queue
|
||||
// entry commit together or not at all.
|
||||
func (q *Queue) EnqueueRun(ctx context.Context, tx pgstore.Tx, runID string) error {
|
||||
if _, err := q.client.InsertTx(ctx, tx, SpawnArgs{RunID: runID}, nil); err != nil {
|
||||
return fmt.Errorf("jobs: enqueue run %s: %w", runID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// EnqueueRunNow inserts the job on its own, outside any transaction.
|
||||
//
|
||||
// The caller is a RESUME: the run it names has already been re-opened and its money already held, in
|
||||
// a transaction that is committed by the time this runs. A failed insert therefore costs one sweep
|
||||
// interval and nothing else — the reconciler finds an attempt with no unit and spawns it — which is
|
||||
// exactly what a lost queue entry costs on the admission path too.
|
||||
func (q *Queue) EnqueueRunNow(ctx context.Context, runID string) error {
|
||||
if _, err := q.client.Insert(ctx, SpawnArgs{RunID: runID}, nil); err != nil {
|
||||
return fmt.Errorf("jobs: enqueue run %s: %w", runID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// EnqueueParse inserts the job in the CALLER's transaction, so a book that says it is being parsed
|
||||
// and the job that parses it commit together.
|
||||
func (q *Queue) EnqueueParse(ctx context.Context, tx pgstore.Tx, bookID string) error {
|
||||
if _, err := q.client.InsertTx(ctx, tx, ParseArgs{BookID: bookID}, nil); err != nil {
|
||||
return fmt.Errorf("jobs: enqueue parse %s: %w", bookID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Start begins working jobs.
|
||||
func (q *Queue) Start(ctx context.Context) error { return q.client.Start(ctx) }
|
||||
|
||||
// Stop drains the workers. The runs themselves are untouched: they are transient units, not
|
||||
// children, and outliving this process is what they are for.
|
||||
func (q *Queue) Stop(ctx context.Context) error { return q.client.Stop(ctx) }
|
||||
37
platform/internal/jobs/jobs_test.go
Normal file
37
platform/internal/jobs/jobs_test.go
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
package jobs
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/riverqueue/river"
|
||||
)
|
||||
|
||||
// Two properties of the queue that are one line each to write and expensive to get wrong, because
|
||||
// both fail SILENTLY: a renamed kind orphans every job already in the table (River dispatches by
|
||||
// that string, and a job whose kind no worker claims simply waits forever), and a MaxAttempts above
|
||||
// one turns "start a run" into "start it again" — the reflex this zone deliberately refuses, since
|
||||
// a retry here spawns a SECOND engine rather than finishing lost work.
|
||||
func TestTheQueuesKindsAndRetryPolicyAreWhatTheRestOfTheSystemAssumes(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
kind string
|
||||
opts river.InsertOpts
|
||||
}{
|
||||
{"spawn", SpawnArgs{}.Kind(), SpawnArgs{}.InsertOpts()},
|
||||
{"parse", ParseArgs{}.Kind(), ParseArgs{}.InsertOpts()},
|
||||
}
|
||||
want := map[string]string{"spawn": "tm_spawn_run", "parse": "tm_parse_book"}
|
||||
for _, tc := range cases {
|
||||
if tc.kind != want[tc.name] {
|
||||
t.Errorf("%s kind is %q, want %q — a renamed kind orphans the jobs already queued under the old one",
|
||||
tc.name, tc.kind, want[tc.name])
|
||||
}
|
||||
if tc.opts.MaxAttempts != 1 {
|
||||
t.Errorf("%s MaxAttempts is %d, want 1: recovery is the reconciler's job, not the queue's",
|
||||
tc.name, tc.opts.MaxAttempts)
|
||||
}
|
||||
if tc.opts.Queue != river.QueueDefault {
|
||||
t.Errorf("%s queue is %q, want the default one the client works", tc.name, tc.opts.Queue)
|
||||
}
|
||||
}
|
||||
}
|
||||
231
platform/internal/metrics/metrics.go
Normal file
231
platform/internal/metrics/metrics.go
Normal file
|
|
@ -0,0 +1,231 @@
|
|||
// Package metrics is the platform's telemetry: the numbers an operator needs to see that the
|
||||
// control plane is doing its job, in the form the ecosystem scrapes.
|
||||
//
|
||||
// # Why a Prometheus client and not expvar
|
||||
//
|
||||
// stdlib `expvar` is the alternative the zone's own norm points at first ("stdlib before a library",
|
||||
// ENGINEERING_STANDARDS §1), and it does not carry this job: it has no labels, so "requests by route
|
||||
// and status" cannot be expressed at all; no histograms, so a latency question is answerable only as
|
||||
// an average, which is the one statistic that hides the tail; and its JSON is not a format any
|
||||
// scraper reads without a translator. What it would save is a dependency, and what it would cost is
|
||||
// writing the missing three by hand — which is the self-written path the same norm rules out.
|
||||
//
|
||||
// `prometheus/client_golang` v1.24.1 (released 24.07.2026, pin verified live) is the de-facto
|
||||
// standard exposition, and the OpenTelemetry alternative is heavier for what this deployment is: a
|
||||
// collector process to run, an export protocol to configure, and a scrape endpoint at the end of it
|
||||
// anyway. The baseline this axis is measured against — the external one PD-115 asks for — is
|
||||
// Prometheus's own naming practices (base units, `_total` on counters, no units in labels) and the
|
||||
// four golden signals for WHAT to expose.
|
||||
//
|
||||
// # Cardinality
|
||||
//
|
||||
// Labels carry the route PATTERN, never the path: a raw path carries book and run ids, which is a
|
||||
// user's library in an operator's index (PD-3) and an unbounded number of time series besides.
|
||||
package metrics
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"github.com/prometheus/client_golang/prometheus/collectors"
|
||||
"github.com/prometheus/client_golang/prometheus/promhttp"
|
||||
)
|
||||
|
||||
// namespace prefixes every series of this service.
|
||||
const namespace = "tm_platform"
|
||||
|
||||
// Metrics is the registry and the instruments of one process.
|
||||
type Metrics struct {
|
||||
registry *prometheus.Registry
|
||||
|
||||
// The runner's state, refreshed by the sweep. Gauges rather than collectors that query on
|
||||
// scrape: the sweep already reads the database on a timer, and a scrape that ran its own queries
|
||||
// would let anyone with access to the endpoint set the load on the control plane's database.
|
||||
queueDepth prometheus.Gauge
|
||||
oldestHold prometheus.Gauge
|
||||
quarantined prometheus.Gauge
|
||||
liveRuns prometheus.Gauge
|
||||
tailerLag prometheus.Gauge
|
||||
booksInIntake *prometheus.GaugeVec
|
||||
sweepDuration *prometheus.HistogramVec
|
||||
sweepUnfinished *prometheus.CounterVec
|
||||
|
||||
requests *prometheus.CounterVec
|
||||
latency *prometheus.HistogramVec
|
||||
}
|
||||
|
||||
// New builds the instruments on a registry of this process's own.
|
||||
//
|
||||
// Its own, and not the package-global default: a global registry is shared with every library that
|
||||
// ever registers into it, and a duplicate registration there is a panic at init in a process whose
|
||||
// job is to stay up.
|
||||
func New() *Metrics {
|
||||
m := &Metrics{registry: prometheus.NewRegistry()}
|
||||
m.queueDepth = prometheus.NewGauge(prometheus.GaugeOpts{
|
||||
Namespace: namespace, Name: "queue_depth",
|
||||
Help: "Jobs in the queue that have not finished (pending, available, running, scheduled or retryable).",
|
||||
})
|
||||
m.oldestHold = prometheus.NewGauge(prometheus.GaugeOpts{
|
||||
Namespace: namespace, Name: "oldest_open_hold_seconds",
|
||||
Help: "Age of the oldest reservation still open. A hold outlives its run only when a settlement could not be made.",
|
||||
})
|
||||
m.quarantined = prometheus.NewGauge(prometheus.GaugeOpts{
|
||||
Namespace: namespace, Name: "quarantined_attempts",
|
||||
Help: "Live attempts whose journal is no longer being materialized. Such a run keeps going and keeps spending.",
|
||||
})
|
||||
m.liveRuns = prometheus.NewGauge(prometheus.GaugeOpts{
|
||||
Namespace: namespace, Name: "live_runs",
|
||||
Help: "Runs the database believes are still going.",
|
||||
})
|
||||
m.tailerLag = prometheus.NewGauge(prometheus.GaugeOpts{
|
||||
Namespace: namespace, Name: "tailer_lag_bytes",
|
||||
Help: "Largest number of bytes any live run's journal holds beyond the platform's cursor.",
|
||||
})
|
||||
m.booksInIntake = prometheus.NewGaugeVec(prometheus.GaugeOpts{
|
||||
Namespace: namespace, Name: "books_in_intake",
|
||||
Help: "Books that have not finished intake, by status.",
|
||||
}, []string{"status"})
|
||||
m.sweepDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
|
||||
Namespace: namespace, Name: "sweep_duration_seconds",
|
||||
Help: "How long one pass of a sweep took.",
|
||||
Buckets: sweepBuckets,
|
||||
}, []string{"sweep"})
|
||||
m.sweepUnfinished = prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Namespace: namespace, Name: "sweep_unfinished_total",
|
||||
Help: "Passes that ran out of their time budget with work left. A rising count is starvation: the tail of the list is never reached (register row PD-169).",
|
||||
}, []string{"sweep"})
|
||||
m.requests = prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||
Namespace: namespace, Name: "http_requests_total",
|
||||
Help: "Requests served, by route pattern, method and status.",
|
||||
}, []string{"route", "method", "code"})
|
||||
m.latency = prometheus.NewHistogramVec(prometheus.HistogramOpts{
|
||||
Namespace: namespace, Name: "http_request_duration_seconds",
|
||||
Help: "Time to serve a request, by route pattern and method.",
|
||||
Buckets: latencyBuckets,
|
||||
}, []string{"route", "method"})
|
||||
m.registry.MustRegister(m.queueDepth, m.oldestHold, m.quarantined, m.liveRuns, m.tailerLag,
|
||||
m.booksInIntake, m.sweepDuration, m.sweepUnfinished, m.requests, m.latency,
|
||||
// The runtime and the process itself: memory, goroutines, file descriptors, CPU. They are
|
||||
// what answers "is this instance healthy" when none of the numbers above has moved.
|
||||
collectors.NewGoCollector(), collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}))
|
||||
return m
|
||||
}
|
||||
|
||||
// sweepBuckets span one tick to well past the sweep's own budget: the question they answer is "is a
|
||||
// pass still finishing", and the interesting shape is entirely in the tail.
|
||||
var sweepBuckets = []float64{0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300}
|
||||
|
||||
// latencyBuckets carry both kinds of request this service serves: a read that must be milliseconds
|
||||
// and an upload that is legitimately minutes.
|
||||
var latencyBuckets = []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 300}
|
||||
|
||||
// Runner is what the reconciler's sweep observed about the world. Written as one struct because the
|
||||
// numbers are read together, in one query, and a set of them from different moments would describe a
|
||||
// state that never existed.
|
||||
type Runner struct {
|
||||
QueueDepth int64
|
||||
OldestHoldSeconds float64
|
||||
QuarantinedAttempts int64
|
||||
LiveRuns int64
|
||||
BooksUploading int64
|
||||
BooksParsing int64
|
||||
}
|
||||
|
||||
// ObserveRunner publishes one reading of the runner's state.
|
||||
func (m *Metrics) ObserveRunner(r Runner) {
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
m.queueDepth.Set(float64(r.QueueDepth))
|
||||
m.oldestHold.Set(r.OldestHoldSeconds)
|
||||
m.quarantined.Set(float64(r.QuarantinedAttempts))
|
||||
m.liveRuns.Set(float64(r.LiveRuns))
|
||||
m.booksInIntake.WithLabelValues("uploading").Set(float64(r.BooksUploading))
|
||||
m.booksInIntake.WithLabelValues("parsing").Set(float64(r.BooksParsing))
|
||||
}
|
||||
|
||||
// ObserveTailerLag publishes how far the furthest-behind live run's cursor is from the end of its
|
||||
// journal, in bytes.
|
||||
func (m *Metrics) ObserveTailerLag(bytes int64) {
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
m.tailerLag.Set(float64(bytes))
|
||||
}
|
||||
|
||||
// ObserveSweep records one pass. unfinished means the pass ran out of time with work left, which is
|
||||
// the starvation the per-run budget bounds and this counter makes visible.
|
||||
func (m *Metrics) ObserveSweep(name string, d time.Duration, unfinished bool) {
|
||||
if m == nil {
|
||||
return
|
||||
}
|
||||
m.sweepDuration.WithLabelValues(name).Observe(d.Seconds())
|
||||
if unfinished {
|
||||
m.sweepUnfinished.WithLabelValues(name).Inc()
|
||||
}
|
||||
}
|
||||
|
||||
// Handler serves the exposition endpoint.
|
||||
func (m *Metrics) Handler() http.Handler {
|
||||
return promhttp.HandlerFor(m.registry, promhttp.HandlerOpts{})
|
||||
}
|
||||
|
||||
// Middleware counts and times every request.
|
||||
//
|
||||
// It sits where the access log sits — around the mux, so `Request.Pattern` is filled by the time it
|
||||
// reads it, and outside the panic recovery, so a request that panicked is counted as the 500 it
|
||||
// became.
|
||||
func (m *Metrics) Middleware() func(http.Handler) http.Handler {
|
||||
return func(next http.Handler) http.Handler {
|
||||
if m == nil {
|
||||
return next
|
||||
}
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
start := time.Now()
|
||||
rec := &recorder{ResponseWriter: w, status: http.StatusOK}
|
||||
next.ServeHTTP(rec, r)
|
||||
route := r.Pattern
|
||||
if route == "" {
|
||||
// One series for everything unmatched, never the path: an unrouted request is where
|
||||
// an attacker chooses the label, and a label an attacker chooses is unbounded
|
||||
// cardinality.
|
||||
route = "(unmatched)"
|
||||
}
|
||||
method := knownMethod(r.Method)
|
||||
m.requests.WithLabelValues(route, method, strconv.Itoa(rec.status)).Inc()
|
||||
m.latency.WithLabelValues(route, method).Observe(time.Since(start).Seconds())
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// knownMethod folds anything that is not a method this service serves into one label. A method is a
|
||||
// token the CALLER chooses, and an unrouted request carries whatever they sent — so without this the
|
||||
// series count is theirs to grow, exactly like the raw path would be.
|
||||
func knownMethod(m string) string {
|
||||
switch m {
|
||||
case http.MethodGet, http.MethodHead, http.MethodPost, http.MethodPut, http.MethodPatch,
|
||||
http.MethodDelete, http.MethodOptions:
|
||||
return m
|
||||
default:
|
||||
return "(other)"
|
||||
}
|
||||
}
|
||||
|
||||
// recorder captures the status code.
|
||||
//
|
||||
// Unwrap is NOT optional: http.ResponseController walks the wrappers through it, and the upload
|
||||
// route reaches its own read deadline that way. A wrapper without it silently turns that call into
|
||||
// "not supported" — and the upload then dies at the server's 30-second ReadTimeout instead.
|
||||
type recorder struct {
|
||||
http.ResponseWriter
|
||||
status int
|
||||
}
|
||||
|
||||
func (r *recorder) WriteHeader(code int) {
|
||||
r.status = code
|
||||
r.ResponseWriter.WriteHeader(code)
|
||||
}
|
||||
|
||||
func (r *recorder) Unwrap() http.ResponseWriter { return r.ResponseWriter }
|
||||
147
platform/internal/metrics/metrics_test.go
Normal file
147
platform/internal/metrics/metrics_test.go
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
package metrics
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// scrape renders the exposition the way a scraper would read it.
|
||||
func scrape(t *testing.T, m *Metrics) string {
|
||||
t.Helper()
|
||||
w := httptest.NewRecorder()
|
||||
m.Handler().ServeHTTP(w, httptest.NewRequest("GET", "/metrics", nil))
|
||||
if w.Code != http.StatusOK {
|
||||
t.Fatalf("scrape: %d %s", w.Code, w.Body)
|
||||
}
|
||||
return w.Body.String()
|
||||
}
|
||||
|
||||
// Labels carry the route PATTERN and never the path. A raw path is a user's library in an operator's
|
||||
// index (PD-3) and an unbounded number of series besides — the label would be chosen by whoever
|
||||
// sends the request.
|
||||
func TestRequestsAreCountedByRoutePatternAndNeverByPath(t *testing.T) {
|
||||
m := New()
|
||||
mux := http.NewServeMux()
|
||||
mux.Handle("GET /v0/books/{bookId}", m.Middleware()(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
})))
|
||||
mux.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/v0/books/bk_secret", nil))
|
||||
|
||||
body := scrape(t, m)
|
||||
if !strings.Contains(body, `tm_platform_http_requests_total{code="404",method="GET",route="GET /v0/books/{bookId}"} 1`) {
|
||||
t.Errorf("the counter does not carry the route pattern:\n%s", body)
|
||||
}
|
||||
if strings.Contains(body, "bk_secret") {
|
||||
t.Errorf("a book id reached the exposition:\n%s", body)
|
||||
}
|
||||
if !strings.Contains(body, "tm_platform_http_request_duration_seconds_bucket") {
|
||||
t.Error("no latency histogram: an average is the one statistic that hides the tail")
|
||||
}
|
||||
}
|
||||
|
||||
// An unrouted request is where the label would be chosen by an attacker, so there is exactly one
|
||||
// series for all of them.
|
||||
func TestAnUnroutedRequestGetsOneSeriesAndNotOnePerPath(t *testing.T) {
|
||||
m := New()
|
||||
h := m.Middleware()(http.NotFoundHandler())
|
||||
for _, p := range []string{"/../etc/passwd", "/random-1", "/random-2"} {
|
||||
h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", p, nil))
|
||||
}
|
||||
body := scrape(t, m)
|
||||
if !strings.Contains(body, `route="(unmatched)"} 3`) {
|
||||
t.Errorf("unmatched requests are not folded into one series:\n%s", body)
|
||||
}
|
||||
if strings.Contains(body, "random-1") {
|
||||
t.Errorf("a caller-chosen path became a label:\n%s", body)
|
||||
}
|
||||
// The METHOD is a token the caller chooses too, and on an unrouted request it is theirs to invent.
|
||||
for _, verb := range []string{"FOOBAR", "WHATEVER"} {
|
||||
r := httptest.NewRequest("GET", "/nope", nil)
|
||||
r.Method = verb
|
||||
h.ServeHTTP(httptest.NewRecorder(), r)
|
||||
}
|
||||
body = scrape(t, m)
|
||||
for _, verb := range []string{"FOOBAR", "WHATEVER"} {
|
||||
if strings.Contains(body, verb) {
|
||||
t.Errorf("a caller-chosen method became a label:\n%s", body)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(body, `method="(other)"`) {
|
||||
t.Errorf("unknown methods are not folded into one series:\n%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
// The wrapper must stay transparent to http.ResponseController, which is how the upload route
|
||||
// reaches its own read deadline. Asserted here as well as through the real server, because this is
|
||||
// where the property lives.
|
||||
func TestTheWrapperKeepsTheResponseControllerReachable(t *testing.T) {
|
||||
m := New()
|
||||
var sawFlush bool
|
||||
h := m.Middleware()(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
// httptest's recorder supports Flush and nothing else, which is enough: what is being
|
||||
// asserted is that the controller finds the wrapped writer at all.
|
||||
sawFlush = http.NewResponseController(w).Flush() == nil
|
||||
}))
|
||||
h.ServeHTTP(httptest.NewRecorder(), httptest.NewRequest("GET", "/", nil))
|
||||
if !sawFlush {
|
||||
t.Fatal("the wrapper hides the ResponseWriter from http.ResponseController: Unwrap is missing")
|
||||
}
|
||||
}
|
||||
|
||||
// The numbers П-11 was opened for: each one used to be answerable only in psql.
|
||||
func TestTheRunnersStateIsExposedWithItsUnits(t *testing.T) {
|
||||
m := New()
|
||||
m.ObserveRunner(Runner{QueueDepth: 3, OldestHoldSeconds: 61.5, QuarantinedAttempts: 1,
|
||||
LiveRuns: 2, BooksUploading: 4, BooksParsing: 5})
|
||||
m.ObserveTailerLag(4096)
|
||||
m.ObserveSweep("runs", 1500*time.Millisecond, true)
|
||||
body := scrape(t, m)
|
||||
for _, want := range []string{
|
||||
"tm_platform_queue_depth 3",
|
||||
"tm_platform_oldest_open_hold_seconds 61.5",
|
||||
"tm_platform_quarantined_attempts 1",
|
||||
"tm_platform_live_runs 2",
|
||||
"tm_platform_tailer_lag_bytes 4096",
|
||||
`tm_platform_books_in_intake{status="uploading"} 4`,
|
||||
`tm_platform_books_in_intake{status="parsing"} 5`,
|
||||
`tm_platform_sweep_unfinished_total{sweep="runs"} 1`,
|
||||
`tm_platform_sweep_duration_seconds_sum{sweep="runs"} 1.5`,
|
||||
} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the exposition is missing %q", want)
|
||||
}
|
||||
}
|
||||
// Base units and no unit inside a label, per the naming practices this axis is measured against.
|
||||
// Judged over OUR namespace only: the Go collector's own series are its to name (and one of them
|
||||
// is `go_memstats_mspan_inuse_bytes`, which is how this assertion first failed).
|
||||
for _, line := range strings.Split(body, "\n") {
|
||||
if !strings.HasPrefix(line, namespace+"_") {
|
||||
continue
|
||||
}
|
||||
for _, wrong := range []string{"_ms ", "_ms{", "_millis", `unit="`} {
|
||||
if strings.Contains(line, wrong) {
|
||||
t.Errorf("%q is not a base-unit name", line)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A process that measures nothing must still serve: every instrument is nil-safe, because the
|
||||
// alternative is a telemetry failure taking down the thing it measures.
|
||||
func TestAServiceWithoutTelemetryStillWorks(t *testing.T) {
|
||||
var m *Metrics
|
||||
m.ObserveRunner(Runner{QueueDepth: 1})
|
||||
m.ObserveTailerLag(1)
|
||||
m.ObserveSweep("runs", time.Second, false)
|
||||
h := m.Middleware()(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusTeapot)
|
||||
}))
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest("GET", "/", nil))
|
||||
if w.Code != http.StatusTeapot {
|
||||
t.Fatalf("status %d through a nil registry", w.Code)
|
||||
}
|
||||
}
|
||||
|
|
@ -57,11 +57,11 @@ type NewBook struct {
|
|||
// AddBook registers a book. `not_started` and not `uploading`: the file is already on disk, and a
|
||||
// status describing an upload this route never performed would be a lie the library then shows.
|
||||
func (s *Store) AddBook(ctx context.Context, in NewBook) (string, error) {
|
||||
id := newID("bk")
|
||||
id := NewBookID()
|
||||
const q = `
|
||||
insert into books (id, owner_id, title, source_lang, target_lang, genre, status,
|
||||
chapter_count, character_count, added_at, workdir, engine_book_id)
|
||||
values ($1, $2, $3, $4, $5, $6, 'not_started', $7, $8, $9, $10, $11)`
|
||||
chapter_count, character_count, added_at, workdir, engine_book_id, revision)
|
||||
values ($1, $2, $3, $4, $5, $6, 'not_started', $7, $8, $9, $10, $11, ` + nextLibraryRevision + `)`
|
||||
// engine_book_id is the platform's id until the engine's handshake reports its own. It is not
|
||||
// left empty: the column is the join back to whatever the engine calls this book, and an empty
|
||||
// one would read as "the engine has no name for it" rather than "we have not been told yet".
|
||||
|
|
@ -77,6 +77,323 @@ func (s *Store) AddBook(ctx context.Context, in NewBook) (string, error) {
|
|||
return id, nil
|
||||
}
|
||||
|
||||
// NewBookID mints a book identifier. Exported because the intake needs one BEFORE the row exists:
|
||||
// the book's directory is named after it, and the row records that directory.
|
||||
func NewBookID() string { return newID("bk") }
|
||||
|
||||
// nextLibraryRevision is the revision a book JOINS the library with, as a SQL fragment over $2 (the
|
||||
// owner).
|
||||
//
|
||||
// The library's revision is derived as the maximum over the account's books, and the contract asks
|
||||
// a client to DROP a read whose revision is not above what it already applied. A book inserted at
|
||||
// revision 0 therefore left the library's number unchanged, and the screen that had just uploaded
|
||||
// it dropped the very read that carried it (register row PD-122). Starting above the account's
|
||||
// current maximum is what makes an addition visible.
|
||||
//
|
||||
// It is the greatest of BOTH halves the library's own revision is read from — the maximum over the
|
||||
// books and the account's floor — because taking only the maximum reproduced the very defect the
|
||||
// floor exists to prevent, one step later: after a cancelled upload raised the floor above the
|
||||
// maximum, the NEXT book joined below it, and the library then answered the same number for three
|
||||
// different states of itself. Found by acceptance; the first version of this constant read only the
|
||||
// maximum.
|
||||
//
|
||||
// Two concurrent additions can compute the same number, and that is harmless: the value still rises,
|
||||
// so the client still refetches and still sees both. What stays open in PD-122 is the account-scope
|
||||
// counter itself — one writer for membership and statuses — which is what would let two SIMULTANEOUS
|
||||
// writers get distinct numbers instead of merely rising ones.
|
||||
const nextLibraryRevision = `(select greatest(
|
||||
coalesce((select max(revision) from books where owner_id = $2), 0),
|
||||
coalesce((select library_revision from users where id = $2), 0)) + 1)`
|
||||
|
||||
// nextRevisionOfThisBooksLibrary is the same number for a statement that has the book but not its
|
||||
// owner: the owner is read from the row being updated.
|
||||
//
|
||||
// ⚠ A status change may NOT simply do `revision + 1`, and that is the same defect as the one above
|
||||
// wearing different clothes. A book whose own counter sits below the account's floor — it was
|
||||
// uploaded before another upload was cancelled — would walk `uploading → parsing → not_started`
|
||||
// three increments deep and still be under the floor, so `greatest(max, floor)` would not move at
|
||||
// all. A client obeying the contract drops every read that is not ABOVE what it applied, and the
|
||||
// screen that uploaded the book would keep showing it as arriving until some unrelated event pushed
|
||||
// the number. Taking the library's own next number instead makes every status visible exactly once.
|
||||
const nextRevisionOfThisBooksLibrary = `(select greatest(
|
||||
coalesce((select max(b.revision) from books b where b.owner_id = books.owner_id), 0),
|
||||
coalesce((select u.library_revision from users u where u.id = books.owner_id), 0)) + 1)`
|
||||
|
||||
// NewUpload is one book arriving through the contract's intake, before its file has been received.
|
||||
type NewUpload struct {
|
||||
OwnerID string
|
||||
Title string
|
||||
SourceLang string
|
||||
TargetLang string
|
||||
Genre string
|
||||
// Workdir is the engine project directory this book will own. It is named after the book id and
|
||||
// created before the row, so a row always points at a directory that exists.
|
||||
Workdir string
|
||||
Now time.Time
|
||||
}
|
||||
|
||||
// CreateUpload registers a book whose file is still arriving.
|
||||
//
|
||||
// `uploading` is a real, observable state and not a formality: the row is written BEFORE the body is
|
||||
// read, so a 60 MB upload is visible in the library — with its languages, without its size — while
|
||||
// it is still on the wire. The alternative, inserting once the bytes have landed, gives the status
|
||||
// no writer at all and makes a half-finished upload invisible to everything, including the sweep
|
||||
// that has to clean it up.
|
||||
func (s *Store) CreateUpload(ctx context.Context, id string, in NewUpload) (Book, error) {
|
||||
const q = `
|
||||
insert into books (id, owner_id, title, source_lang, target_lang, genre, status,
|
||||
added_at, workdir, engine_book_id, revision)
|
||||
values ($1, $2, $3, $4, $5, $6, 'uploading', $7, $8, $1, ` + nextLibraryRevision + `)
|
||||
returning id, title, source_lang, target_lang, genre, status, chapter_count, character_count,
|
||||
note_count, added_at, revision`
|
||||
var b Book
|
||||
err := s.pool.QueryRow(ctx, q, id, in.OwnerID, in.Title, in.SourceLang, in.TargetLang, in.Genre,
|
||||
in.Now, in.Workdir).
|
||||
Scan(&b.ID, &b.Title, &b.SourceLang, &b.TargetLang, &b.Genre, &b.Status, &b.ChapterCount,
|
||||
&b.CharacterCount, &b.NoteCount, &b.AddedAt, &b.Revision)
|
||||
if err != nil {
|
||||
var pg *pgconn.PgError
|
||||
if errors.As(err, &pg) && pg.ConstraintName == "books_owner_id_fkey" {
|
||||
return Book{}, ErrNoAccount
|
||||
}
|
||||
return Book{}, fmt.Errorf("pgstore: create upload: %w", err)
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
|
||||
// StartParsing hands a received file to the parser: `uploading → parsing`, with the size the intake
|
||||
// counted while the bytes went past.
|
||||
//
|
||||
// Conditional on the status, so an upload the sweep has already given up on cannot be resurrected by
|
||||
// a request that finally finished.
|
||||
//
|
||||
// enqueue joins the caller's transaction for the same reason the run admission's does: a status that
|
||||
// says "being parsed" with no job to do it waits for the backstop sweep's grace, and a job for a
|
||||
// book that never became `parsing` is a worker with nothing to claim.
|
||||
func (s *Store) StartParsing(ctx context.Context, id string, characters int64,
|
||||
enqueue func(context.Context, Tx, string) error) (Book, error) {
|
||||
var b Book
|
||||
err := s.inTx(ctx, func(tx pgx.Tx) error {
|
||||
const q = `
|
||||
update books set status = 'parsing', character_count = $2,
|
||||
revision = ` + nextRevisionOfThisBooksLibrary + `
|
||||
where id = $1 and status = 'uploading'
|
||||
returning id, title, source_lang, target_lang, genre, status, chapter_count,
|
||||
character_count, note_count, added_at, revision`
|
||||
err := tx.QueryRow(ctx, q, id, characters).
|
||||
Scan(&b.ID, &b.Title, &b.SourceLang, &b.TargetLang, &b.Genre, &b.Status, &b.ChapterCount,
|
||||
&b.CharacterCount, &b.NoteCount, &b.AddedAt, &b.Revision)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return ErrNoBook
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("pgstore: start parsing: %w", err)
|
||||
}
|
||||
if enqueue != nil {
|
||||
return enqueue(ctx, tx, id)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return Book{}, err
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
|
||||
// ParseClaim is one book this process may parse.
|
||||
type ParseClaim struct {
|
||||
BookID string
|
||||
Workdir string
|
||||
Attempts int
|
||||
// At is the stamp this claim was taken with. Every write that ENDS the parse carries it back, so
|
||||
// a pass whose claim was taken over by another cannot record a verdict about work it is no longer
|
||||
// doing — the same re-check-under-the-write the money paths use.
|
||||
At time.Time
|
||||
}
|
||||
|
||||
// ErrParseClaimed is a parse someone else is already doing, or a book that is no longer waiting for
|
||||
// one. The ordinary case, not a failure: the queue job and the backstop sweep both aim at the same
|
||||
// book on purpose.
|
||||
var ErrParseClaimed = errors.New("pgstore: the parse of this book is already claimed")
|
||||
|
||||
// ClaimParse takes the right to parse a book, once.
|
||||
//
|
||||
// A compare-and-set rather than a plain read, for the same reason the spawn claim is one: the queue
|
||||
// worker and the backstop sweep legitimately arrive together, and two `tmctl manifest` processes on
|
||||
// one project directory would be two writers of one SQLite file — which is exactly what the engine's
|
||||
// exclusive lock exists to refuse, noisily and after the work.
|
||||
//
|
||||
// staleAfter is what makes the claim recoverable: a process that died holding one leaves a stamp
|
||||
// nothing clears, and without a grace the book would wait for parsing forever.
|
||||
func (s *Store) ClaimParse(ctx context.Context, id string, now, staleBefore time.Time) (ParseClaim, error) {
|
||||
const q = `
|
||||
update books set parse_started_at = $2, parse_attempts = parse_attempts + 1
|
||||
where id = $1 and status = 'parsing'
|
||||
and (parse_started_at is null or parse_started_at < $3)
|
||||
returning id, workdir, parse_attempts`
|
||||
var c ParseClaim
|
||||
err := s.pool.QueryRow(ctx, q, id, now, staleBefore).Scan(&c.BookID, &c.Workdir, &c.Attempts)
|
||||
c.At = now
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return ParseClaim{}, ErrParseClaimed
|
||||
}
|
||||
if err != nil {
|
||||
return ParseClaim{}, fmt.Errorf("pgstore: claim parse: %w", err)
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// RefundParseAttempt gives back an attempt that was claimed but never spent on the engine.
|
||||
//
|
||||
// The budget exists to bound how many times a broken HOST is asked to parse a book, and a book that
|
||||
// is only waiting for its configuration asks nobody: counting those claims meant the budget ran out
|
||||
// while nothing had been tried, and the first real answer from the engine was terminal the moment it
|
||||
// arrived. Conditional on the claim, like every other write that ends a pass.
|
||||
func (s *Store) RefundParseAttempt(ctx context.Context, id string, claimedAt time.Time) error {
|
||||
_, err := s.pool.Exec(ctx, `
|
||||
update books set parse_attempts = greatest(parse_attempts - 1, 0)
|
||||
where id = $1 and status = 'parsing' and parse_started_at = $2`, id, claimedAt)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pgstore: refund parse attempt: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ParsedBook is what the engine's manifest told the platform about a book it has now cut.
|
||||
type ParsedBook struct {
|
||||
Chapters int
|
||||
SourceSHA256 []byte
|
||||
ChunkerVersion string
|
||||
}
|
||||
|
||||
// FinishParse records a parsed book: `parsing → not_started`.
|
||||
//
|
||||
// Conditional on the CLAIM still being this caller's: a pass whose claim was taken over is not the
|
||||
// pass whose answer counts, and two answers about one book are one answer too many.
|
||||
func (s *Store) FinishParse(ctx context.Context, id string, claimedAt time.Time, in ParsedBook) error {
|
||||
const q = `
|
||||
update books set status = 'not_started', chapter_count = $2, source_sha256 = $3,
|
||||
chunker_version = $4, parse_started_at = null,
|
||||
revision = ` + nextRevisionOfThisBooksLibrary + `
|
||||
where id = $1 and status = 'parsing' and parse_started_at = $5`
|
||||
tag, err := s.pool.Exec(ctx, q, id, in.Chapters, in.SourceSHA256, in.ChunkerVersion, claimedAt)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pgstore: finish parse: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrNoBook
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// RejectBook is the terminal end of an intake that failed: `parsing → rejected`.
|
||||
//
|
||||
// The reason is the platform's own word for the class of failure and is not projected — contract v0
|
||||
// has no field for it (migration 00013).
|
||||
//
|
||||
// Conditional on the CLAIM, like FinishParse and for a sharper reason: rejecting is destructive —
|
||||
// the book's source goes with it — and a pass whose claim was stolen may be holding an exit code
|
||||
// that says nothing about the book. The engine maps every failure onto exit 1, including a lock held
|
||||
// by the parse that took the claim over.
|
||||
func (s *Store) RejectBook(ctx context.Context, id string, claimedAt time.Time, reason string) error {
|
||||
const q = `
|
||||
update books set status = 'rejected', reject_reason = $2, parse_started_at = null,
|
||||
revision = ` + nextRevisionOfThisBooksLibrary + `
|
||||
where id = $1 and status = 'parsing' and parse_started_at = $3`
|
||||
tag, err := s.pool.Exec(ctx, q, id, reason, claimedAt)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pgstore: reject book: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrNoBook
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// DeleteUpload removes a book whose file never arrived.
|
||||
//
|
||||
// Not `rejected`: that status means "the file could not be parsed" (contract §BookStatus), and a
|
||||
// request the client abandoned produced no file to judge. The user's library must not accumulate a
|
||||
// permanent row for every cancelled upload — there is no delete handle in the contract to clear one
|
||||
// with.
|
||||
//
|
||||
// Narrow on purpose: only a book still in `uploading`, and only one no run was ever started for, so
|
||||
// this can never become a way to delete a book that has cost money.
|
||||
//
|
||||
// ⚠ It also carries the library's revision ACROSS the deletion, and that is not bookkeeping: the
|
||||
// library's number is derived as the maximum over the account's books, so removing the newest one
|
||||
// would lower it — and the contract asks a client to DROP a read whose revision is below what it
|
||||
// applied (§Revision). The screen would then keep rendering a book that no longer exists until some
|
||||
// other book's counter grew past it. The floor lives in `users.library_revision`, the account-scope
|
||||
// counter that has existed unused since 00001 (register row PD-122).
|
||||
func (s *Store) DeleteUpload(ctx context.Context, id string) error {
|
||||
return s.inTx(ctx, func(tx pgx.Tx) error {
|
||||
// Book first, as in every transaction that touches two of these tables (lockBook). `users` is
|
||||
// taken after it, and no path in this package takes them the other way round.
|
||||
if err := lockBook(ctx, tx, id); err != nil {
|
||||
return err
|
||||
}
|
||||
var owner string
|
||||
var revision int64
|
||||
err := tx.QueryRow(ctx, `
|
||||
delete from books where id = $1 and status = 'uploading'
|
||||
and not exists (select 1 from runs where book_id = $1)
|
||||
returning owner_id, revision`, id).Scan(&owner, &revision)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return ErrNoBook
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("pgstore: delete upload: %w", err)
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
update users set library_revision = greatest(library_revision, $2) where id = $1`,
|
||||
owner, revision+1); err != nil {
|
||||
return fmt.Errorf("pgstore: carry the library revision: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// IntakeBook is a book the backstop sweep has to make a decision about.
|
||||
type IntakeBook struct {
|
||||
ID string
|
||||
Status string
|
||||
Workdir string
|
||||
Attempts int
|
||||
}
|
||||
|
||||
// StuckIntake lists books that stopped moving through intake.
|
||||
//
|
||||
// Both halves have the same cause and different cures: an `uploading` row whose request is gone is
|
||||
// litter to remove, a `parsing` row whose claim went stale is work to retry. Neither is reachable
|
||||
// from the request that created it — the process holding it is the one that died — so the sweep is
|
||||
// the only thing that can finish the walk.
|
||||
func (s *Store) StuckIntake(ctx context.Context, uploadingBefore, claimedBefore time.Time) ([]IntakeBook, error) {
|
||||
// coalesce(claimed, added): an unclaimed parse is stuck when the BOOK has been waiting, a claimed
|
||||
// one when the CLAIM has. Written as one expression because they are the same question — how long
|
||||
// has this book had nobody working on it — and a NULL claim is simply the first answer to it.
|
||||
const q = `
|
||||
select id, status, workdir, parse_attempts
|
||||
from books
|
||||
where (status = 'uploading' and added_at < $1)
|
||||
or (status = 'parsing' and coalesce(parse_started_at, added_at) < $2)
|
||||
order by added_at`
|
||||
rows, err := s.pool.Query(ctx, q, uploadingBefore, claimedBefore)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pgstore: list stuck intake: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []IntakeBook
|
||||
for rows.Next() {
|
||||
var b IntakeBook
|
||||
if err := rows.Scan(&b.ID, &b.Status, &b.Workdir, &b.Attempts); err != nil {
|
||||
return nil, fmt.Errorf("pgstore: scan stuck intake: %w", err)
|
||||
}
|
||||
out = append(out, b)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// Progress is per phase, in units (contract §Progress).
|
||||
type Progress struct {
|
||||
DraftDone, DraftTotal int
|
||||
|
|
@ -158,8 +475,15 @@ func (s *Store) ListBooks(ctx context.Context, userID string, limit int, cursor
|
|||
}
|
||||
// The revision of the LIBRARY scope — membership and statuses — and never compared with the
|
||||
// revision of a book (contract §Revision).
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`select coalesce(max(revision), 0) from books where owner_id = $1`, userID).Scan(&out.Revision); err != nil {
|
||||
//
|
||||
// The maximum over the account's books, FLOORED by the account's own counter. The floor is what
|
||||
// makes the number monotonic across a deletion: the maximum alone falls when the newest book goes
|
||||
// (DeleteUpload raises the floor as it deletes), and a library revision that goes backwards is a
|
||||
// library the client stops rendering.
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
select greatest(coalesce((select max(revision) from books where owner_id = $1), 0),
|
||||
coalesce((select library_revision from users where id = $1), 0))`,
|
||||
userID).Scan(&out.Revision); err != nil {
|
||||
return Library{}, fmt.Errorf("pgstore: library revision: %w", err)
|
||||
}
|
||||
return out, nil
|
||||
|
|
@ -191,6 +515,13 @@ func (s *Store) GetBook(ctx context.Context, userID, bookID string) (Book, *Run,
|
|||
}
|
||||
if run != nil {
|
||||
b.Progress = run.progress
|
||||
// ONE counter per book, and this is the single place that says so for every run this store
|
||||
// hands out (§Revision: "every book-scoped read … carry the same number"). Read off the run's
|
||||
// own column the card lagged — a unit_done bumps the book and the chapter and not the run, so
|
||||
// a client that had applied frame id=2 got revision 0 back and, obeying the contract, dropped
|
||||
// the read. The stop and resume handles answer from ReadRun/RequestStop, which select the
|
||||
// book's revision for the same reason.
|
||||
run.Revision = b.Revision
|
||||
return b, &run.Run, nil
|
||||
}
|
||||
return b, nil, nil
|
||||
|
|
@ -225,6 +556,10 @@ func (s *Store) lastRun(ctx context.Context, bookID string) (*runWithProgress, e
|
|||
// ownership so a stranger's book is indistinguishable from a missing one.
|
||||
type BookRunContext struct {
|
||||
Workdir string
|
||||
// Status is the book's own state. A run may only be started on a book whose intake finished:
|
||||
// the directory of a book still `uploading` holds half a file and the directory of a `rejected`
|
||||
// one may hold nothing at all.
|
||||
Status string
|
||||
// ChaptersLeft is what is still untranslated, and it clamps the ceiling scale. Until the engine
|
||||
// persists a chapter manifest (unified backlog row 100) the platform knows the chapter COUNT
|
||||
// from intake and nothing per chapter, so a book with no materialized chapters answers with its
|
||||
|
|
@ -236,14 +571,14 @@ type BookRunContext struct {
|
|||
// ReadBookForRun gathers the facts a run start is judged on.
|
||||
func (s *Store) ReadBookForRun(ctx context.Context, userID, bookID string) (BookRunContext, error) {
|
||||
const q = `
|
||||
select b.workdir,
|
||||
select b.workdir, b.status,
|
||||
b.chapter_count - (select count(*) from chapters c
|
||||
where c.book_id = b.id and c.units_total > 0
|
||||
and c.units_done >= c.units_total),
|
||||
exists (select 1 from runs r where r.book_id = b.id and r.finished_at is null)
|
||||
from books b where b.id = $1 and b.owner_id = $2`
|
||||
var out BookRunContext
|
||||
err := s.pool.QueryRow(ctx, q, bookID, userID).Scan(&out.Workdir, &out.ChaptersLeft, &out.HasLiveRun)
|
||||
err := s.pool.QueryRow(ctx, q, bookID, userID).Scan(&out.Workdir, &out.Status, &out.ChaptersLeft, &out.HasLiveRun)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return BookRunContext{}, ErrNoBook
|
||||
}
|
||||
|
|
|
|||
287
platform/internal/pgstore/intake_test.go
Normal file
287
platform/internal/pgstore/intake_test.go
Normal file
|
|
@ -0,0 +1,287 @@
|
|||
package pgstore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The store half of the two walks this pack added: a book through intake, and a run through a stop
|
||||
// the user asked for. What is asserted here is what only SQL can get wrong — the conditions on the
|
||||
// updates, which are what keep two processes from finishing one walk twice.
|
||||
|
||||
func upload(t *testing.T, s *Store, ctx context.Context, owner string, now time.Time) Book {
|
||||
t.Helper()
|
||||
id := NewBookID()
|
||||
b, err := s.CreateUpload(ctx, id, NewUpload{OwnerID: owner, Title: "蛊真人", SourceLang: "zh",
|
||||
TargetLang: "ru", Workdir: "/srv/books/" + id, Now: now})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func TestABookWalksThroughIntakeOnlyForwards(t *testing.T) {
|
||||
s, ctx := testDB(t)
|
||||
now := fundedAccount(t, s, ctx, "u1", "10")
|
||||
b := upload(t, s, ctx, "u1", now)
|
||||
if b.Status != "uploading" {
|
||||
t.Fatalf("a created upload is %q", b.Status)
|
||||
}
|
||||
if _, err := s.StartParsing(ctx, b.ID, 42, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The same call again finds no book in `uploading`: a request that finally finished must not be
|
||||
// able to resurrect an upload the sweep has already given up on.
|
||||
if _, err := s.StartParsing(ctx, b.ID, 42, nil); !errors.Is(err, ErrNoBook) {
|
||||
t.Fatalf("a second StartParsing gave %v, want ErrNoBook", err)
|
||||
}
|
||||
claim, err := s.ClaimParse(ctx, b.ID, now, now.Add(-time.Hour))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// A verdict may only be written by the pass that still HOLDS the claim: an answer from a pass
|
||||
// whose claim was taken over is an answer about work it is no longer doing — and for a rejection
|
||||
// it would also delete the book's source.
|
||||
stale := claim.At.Add(-time.Minute)
|
||||
if err := s.FinishParse(ctx, b.ID, stale, ParsedBook{Chapters: 7}); !errors.Is(err, ErrNoBook) {
|
||||
t.Fatalf("a stale claim finished the parse: %v, want ErrNoBook", err)
|
||||
}
|
||||
if err := s.RejectBook(ctx, b.ID, stale, ReasonForTest); !errors.Is(err, ErrNoBook) {
|
||||
t.Fatalf("a stale claim rejected the book: %v, want ErrNoBook", err)
|
||||
}
|
||||
if err := s.FinishParse(ctx, b.ID, claim.At, ParsedBook{Chapters: 500, ChunkerVersion: "chunk-1"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.FinishParse(ctx, b.ID, claim.At, ParsedBook{Chapters: 7}); !errors.Is(err, ErrNoBook) {
|
||||
t.Fatalf("a second FinishParse gave %v, want ErrNoBook", err)
|
||||
}
|
||||
// And a parsed book can no longer be rejected: `rejected` is an END of intake, not a state a
|
||||
// later pass may drop a working book into.
|
||||
if err := s.RejectBook(ctx, b.ID, claim.At, ReasonForTest); !errors.Is(err, ErrNoBook) {
|
||||
t.Fatalf("rejecting a parsed book gave %v, want ErrNoBook", err)
|
||||
}
|
||||
got, _, err := s.GetBook(ctx, "u1", b.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Status != "not_started" || got.ChapterCount != 500 || got.CharacterCount != 42 {
|
||||
t.Fatalf("after the walk: %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
// ReasonForTest stands for one of the platform's own reject reasons; the vocabulary itself lives in
|
||||
// the intake package, and this file only needs a value.
|
||||
const ReasonForTest = "source_unreadable"
|
||||
|
||||
// The claim is what keeps two `tmctl manifest` processes off one project directory, and the grace is
|
||||
// what keeps a claim from being permanent when the process holding it dies.
|
||||
func TestAParseClaimIsTakenOnceAndRecoveredAfterTheGrace(t *testing.T) {
|
||||
s, ctx := testDB(t)
|
||||
now := fundedAccount(t, s, ctx, "u1", "10")
|
||||
b := upload(t, s, ctx, "u1", now)
|
||||
if _, err := s.StartParsing(ctx, b.ID, 1, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stale := now.Add(-time.Hour)
|
||||
first, err := s.ClaimParse(ctx, b.ID, now, stale)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if first.Attempts != 1 || first.Workdir == "" {
|
||||
t.Fatalf("first claim: %+v", first)
|
||||
}
|
||||
if _, err := s.ClaimParse(ctx, b.ID, now, stale); !errors.Is(err, ErrParseClaimed) {
|
||||
t.Fatalf("a second claim gave %v, want ErrParseClaimed", err)
|
||||
}
|
||||
// The holder is gone and its claim has aged past the grace: the next pass may take it, and the
|
||||
// attempt counter — which is what bounds a broken host — keeps counting. An hour later, with the
|
||||
// grace the intake actually uses, the first claim is well behind the line.
|
||||
later := now.Add(time.Hour)
|
||||
again, err := s.ClaimParse(ctx, b.ID, later, later.Add(-10*time.Minute))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if again.Attempts != 2 {
|
||||
t.Fatalf("the attempt counter reads %d after a recovered claim", again.Attempts)
|
||||
}
|
||||
// ⚠ And the claim is what SPACES the retries: it stays held after a failed parse, so the book is
|
||||
// not claimable again until the grace passes. Without that the sweep re-offers it on its very next
|
||||
// tick — the staleness predicate falls back to `added_at`, which is already old — and the attempt
|
||||
// budget burns in five ticks of the sweep instead of in the time it is meant to bound.
|
||||
if _, err := s.ClaimParse(ctx, b.ID, later, later.Add(-10*time.Minute)); !errors.Is(err, ErrParseClaimed) {
|
||||
t.Fatalf("a fresh claim was retakeable at once: %v, want ErrParseClaimed", err)
|
||||
}
|
||||
}
|
||||
|
||||
// DeleteUpload is narrow on purpose: it is the cure for a request that went away, and it must never
|
||||
// become a way to delete a book that has cost money.
|
||||
func TestOnlyAnUnfinishedUploadCanBeDeleted(t *testing.T) {
|
||||
s, ctx := testDB(t)
|
||||
now := fundedAccount(t, s, ctx, "u1", "10")
|
||||
b := upload(t, s, ctx, "u1", now)
|
||||
if _, err := s.StartParsing(ctx, b.ID, 1, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.DeleteUpload(ctx, b.ID); !errors.Is(err, ErrNoBook) {
|
||||
t.Fatalf("deleting a book that is being parsed gave %v, want ErrNoBook", err)
|
||||
}
|
||||
// A book back in `uploading` but with a run against it stays as well: the run's money is keyed to
|
||||
// it, and the cascade would take the row out from under the ledger.
|
||||
if _, err := s.pool.Exec(ctx, `update books set status = 'uploading' where id = $1`, b.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.StartRun(ctx, StartRunInput{UserID: "u1", BookID: b.ID, CeilingChapters: 1,
|
||||
Ceiling: 1_000_000, Now: now}, 0, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.DeleteUpload(ctx, b.ID); !errors.Is(err, ErrNoBook) {
|
||||
t.Fatalf("deleting a book with a run gave %v, want ErrNoBook", err)
|
||||
}
|
||||
other := upload(t, s, ctx, "u1", now)
|
||||
if err := s.DeleteUpload(ctx, other.ID); err != nil {
|
||||
t.Fatalf("an untouched upload could not be deleted: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The stop intent: idempotent, first timestamp kept, and the two refusals told apart.
|
||||
func TestAStopRequestIsIdempotentAndKeepsTheFirstTime(t *testing.T) {
|
||||
s, ctx := testDB(t)
|
||||
now := fundedAccount(t, s, ctx, "u1", "10")
|
||||
fundedAccount(t, s, ctx, "u2", "10")
|
||||
bookID, err := s.AddBook(ctx, NewBook{OwnerID: "u1", Title: "b", SourceLang: "zh", TargetLang: "ru",
|
||||
ChapterCount: 10, Workdir: "/srv/books/b", Now: now})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
started, err := s.StartRun(ctx, StartRunInput{UserID: "u1", BookID: bookID, CeilingChapters: 1,
|
||||
Ceiling: 1_000_000, Now: now}, 0, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := s.RequestStop(ctx, "u2", started.ID, now); !errors.Is(err, ErrNoRun) {
|
||||
t.Fatalf("a stranger's stop gave %v, want ErrNoRun", err)
|
||||
}
|
||||
run, unit, err := s.RequestStop(ctx, "u1", started.ID, now)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if run.ID != started.ID || unit != "" {
|
||||
t.Fatalf("stop returned %+v with unit %q (nothing was spawned)", run, unit)
|
||||
}
|
||||
if _, _, err := s.RequestStop(ctx, "u1", started.ID, now.Add(time.Hour)); err != nil {
|
||||
t.Fatalf("a second stop: %v", err)
|
||||
}
|
||||
var at time.Time
|
||||
if err := s.pool.QueryRow(ctx, `select stop_requested_at from runs where id = $1`, started.ID).Scan(&at); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !at.Equal(now) {
|
||||
t.Fatalf("the stop is stamped %s, want the FIRST request at %s", at, now)
|
||||
}
|
||||
// Over means over: a finished run is a conflict, not a missing one.
|
||||
if _, err := s.FinishRun(ctx, started.ID, started.AttemptID, "stopped", "success", nil, now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := s.RequestStop(ctx, "u1", started.ID, now); !errors.Is(err, ErrRunNotLive) {
|
||||
t.Fatalf("stopping a finished run gave %v, want ErrRunNotLive", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A resumed run is LIVE again, and the three columns that say otherwise have to go with it: an
|
||||
// unfinished run the one-live-per-book index cannot see, money marked resolved that the new attempt
|
||||
// has not spent, and a stop request from the previous life that would classify this attempt's ending
|
||||
// as a stop nobody asked for.
|
||||
func TestReopeningARunClearsWhatSaidItWasOver(t *testing.T) {
|
||||
s, ctx := testDB(t)
|
||||
now := fundedAccount(t, s, ctx, "u1", "20")
|
||||
bookID, err := s.AddBook(ctx, NewBook{OwnerID: "u1", Title: "b", SourceLang: "zh", TargetLang: "ru",
|
||||
ChapterCount: 10, Workdir: "/srv/books/b", Now: now})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
started, err := s.StartRun(ctx, StartRunInput{UserID: "u1", BookID: bookID, CeilingChapters: 1,
|
||||
Ceiling: 1_000_000, Now: now}, 0, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := s.RequestStop(ctx, "u1", started.ID, now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.FinishRun(ctx, started.ID, started.AttemptID, "stopped", "exit-code", nil, now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.MarkSettled(ctx, started.ID, now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.RestartRun(ctx, RestartInput{RunID: started.ID, AttemptID: started.AttemptID,
|
||||
UserID: "u1", BookID: bookID, Ceiling: 500_000, Now: now}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var finished, settled, stop *time.Time
|
||||
var status, bookStatus string
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
select r.finished_at, r.settled_at, r.stop_requested_at, r.status, b.status
|
||||
from runs r join books b on b.id = r.book_id where r.id = $1`, started.ID).
|
||||
Scan(&finished, &settled, &stop, &status, &bookStatus); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if finished != nil || settled != nil || stop != nil {
|
||||
t.Fatalf("a reopened run still says it is over: finished %v settled %v stop %v", finished, settled, stop)
|
||||
}
|
||||
if status != "translating" || bookStatus != "translating" {
|
||||
t.Fatalf("run %q, book %q after a reopen", status, bookStatus)
|
||||
}
|
||||
// The verdict of the attempt that ended is NOT overwritten: how the run the user stopped actually
|
||||
// ended is the one fact a resume must not erase.
|
||||
var result string
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`select exit_result from run_attempts where id = $1`, started.AttemptID).Scan(&result); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if result != "exit-code" {
|
||||
t.Fatalf("the previous attempt's exit_result became %q", result)
|
||||
}
|
||||
}
|
||||
|
||||
// One counter per book, and this store is where that rule lives: every Run it hands out — from the
|
||||
// card, from a stop, from a plain read — carries its BOOK's revision. A run's own column lags by
|
||||
// construction (a materialized unit bumps the book and the chapter, not the run), and the contract
|
||||
// tells a client to DROP a read whose revision is below one it applied, so a handle answering from
|
||||
// the run's column hands back a number the client is required to throw away.
|
||||
func TestEveryRunTheStoreHandsOutCarriesItsBooksRevision(t *testing.T) {
|
||||
s, ctx := testDB(t)
|
||||
now := fundedAccount(t, s, ctx, "u1", "10")
|
||||
bookID, err := s.AddBook(ctx, NewBook{OwnerID: "u1", Title: "b", SourceLang: "zh", TargetLang: "ru",
|
||||
ChapterCount: 10, Workdir: "/srv/books/b", Now: now})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
started, err := s.StartRun(ctx, StartRunInput{UserID: "u1", BookID: bookID, CeilingChapters: 1,
|
||||
Ceiling: 1_000_000, Now: now}, 0, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Something materializes: the book's counter moves and the run's own column stays where it was.
|
||||
if _, err := s.pool.Exec(ctx, `update books set revision = revision + 7 where id = $1`, bookID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
book, card, err := s.GetBook(ctx, "u1", bookID)
|
||||
if err != nil || card == nil {
|
||||
t.Fatalf("card: %v", err)
|
||||
}
|
||||
read, err := s.ReadRun(ctx, "u1", started.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stop, _, err := s.RequestStop(ctx, "u1", started.ID, now)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for name, got := range map[string]int64{"card": card.Revision, "read": read.Revision, "stop": stop.Revision} {
|
||||
if got != book.Revision {
|
||||
t.Errorf("%s answered revision %d, the book is at %d", name, got, book.Revision)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -25,3 +25,5 @@ c225e1a12bab8c62669848976096453aa84d66263fe211da0a5eb4b173ff2eff 00007_credits.
|
|||
905b89bcecffd53cda7c628f7e1c3b9dad522a8947fbcd7dbe31dfb9644c644e 00010_spend_baseline.sql
|
||||
76f3bb045cf3a2d232065418357f534752ddb06e28e4122e1d45c8f59c23abf8 00011_ceiling_argument.sql
|
||||
282ff2d515d9de16f631a5cd77f676c7742337f96406b8876c900f0b44ea19a0 00012_resync_stamp.sql
|
||||
1cb0d8127ea8ad0391b6829c2d09a7b776ebc9729002ae5db76afe5daa7a790b 00013_book_intake.sql
|
||||
4281cfce2a34179333843f72257f3d9e62c71480a129168d613066aa8ad3687e 00014_stop_intent.sql
|
||||
|
|
|
|||
39
platform/internal/pgstore/migrations/00013_book_intake.sql
Normal file
39
platform/internal/pgstore/migrations/00013_book_intake.sql
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
-- +goose Up
|
||||
|
||||
-- The intake half of a book: what `POST /books` needs to move a row through
|
||||
-- `uploading → parsing → not_started | rejected` and what a backstop sweep needs to finish that
|
||||
-- walk when the process doing it died halfway.
|
||||
--
|
||||
-- Until this pack the three intake statuses existed in the contract and in the DDL check and had no
|
||||
-- writer at all: a book could only be registered by the dev CLI, already parsed, straight into
|
||||
-- `not_started`.
|
||||
|
||||
alter table books
|
||||
-- How many times parsing has been CLAIMED for this book. Parsing is one $0 call of the engine
|
||||
-- (`tmctl manifest`), and it fails in two different ways: the source cannot be cut (the book's
|
||||
-- own fault, terminal) or the engine could not be run at all (the deployment's fault, and the
|
||||
-- next sweep may well succeed). Only the second is retried, and only this counter bounds that
|
||||
-- retry — without it a misconfigured host would re-run the engine over every uploaded book
|
||||
-- forever.
|
||||
add column parse_attempts integer not null default 0,
|
||||
-- When the current parse was claimed. It is the CLAIM, not a timestamp for display: a book in
|
||||
-- `parsing` whose claim is older than the grace is a parse whose process is gone, and the sweep
|
||||
-- may take it again. Null means no claim is outstanding.
|
||||
add column parse_started_at timestamptz,
|
||||
-- Why a book was rejected, in the platform's own closed vocabulary — never the engine's text,
|
||||
-- which reads like pipeline internals and must not cross the seam (contract §Problem, PT-33).
|
||||
-- ⚠ NOT on the wire: contract v0 gives a book a `rejected` status and no reason field, so this
|
||||
-- is written for an operator and for whatever later pack adds one. Inventing the field here is
|
||||
-- not this zone's right (the precedent is last_resync_at, register row PD-150).
|
||||
add column reject_reason text not null default '';
|
||||
|
||||
-- The sweep asks one question — "which books are stuck in intake" — every few seconds, and the
|
||||
-- answer is a handful of rows in a table that grows with every book of every account.
|
||||
create index books_intake_idx on books (status, added_at) where status in ('uploading', 'parsing');
|
||||
|
||||
-- +goose Down
|
||||
drop index books_intake_idx;
|
||||
alter table books
|
||||
drop column reject_reason,
|
||||
drop column parse_started_at,
|
||||
drop column parse_attempts;
|
||||
20
platform/internal/pgstore/migrations/00014_stop_intent.sql
Normal file
20
platform/internal/pgstore/migrations/00014_stop_intent.sql
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
-- +goose Up
|
||||
|
||||
-- When THIS platform asked the run to stop.
|
||||
--
|
||||
-- It exists because the exit marker cannot answer "who ended this run". The engine catches SIGTERM
|
||||
-- and exits with code 1, so `ExecStopPost` records `exit-code/exited/1` for a stop a user pressed
|
||||
-- and for a crash alike (register row PD-152, measured on the stand). Guessing from the code would
|
||||
-- be reading someone else's tea leaves; this column is the platform's OWN knowledge, written before
|
||||
-- the signal is sent, and the reconciler classifies the marker by it.
|
||||
--
|
||||
-- The order matters and is the whole point: the intent is committed BEFORE systemd is asked, so a
|
||||
-- platform that dies between the two still knows, on its next sweep, that the run it finds stopped
|
||||
-- was stopped on purpose — and that it must not restart it.
|
||||
--
|
||||
-- Null means nobody asked. Cleared when a run is re-opened, because a resumed run has not been
|
||||
-- asked to stop; a stale intent would classify its next ending as a stop nobody requested.
|
||||
alter table runs add column stop_requested_at timestamptz;
|
||||
|
||||
-- +goose Down
|
||||
alter table runs drop column stop_requested_at;
|
||||
48
platform/internal/pgstore/observe.go
Normal file
48
platform/internal/pgstore/observe.go
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
package pgstore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
// Observations is one reading of the state an operator cannot otherwise see.
|
||||
//
|
||||
// Every figure here used to be answerable only by opening psql (zone backlog П-11): how much work is
|
||||
// queued, whether money is stuck in a hold nothing closes, whether a live run's projection has been
|
||||
// blinded, how many books are stuck in intake. They are read in ONE statement because a set of them
|
||||
// taken at different moments describes a state the system was never in.
|
||||
type Observations struct {
|
||||
QueueDepth int64
|
||||
OldestHoldSeconds float64
|
||||
QuarantinedAttempts int64
|
||||
LiveRuns int64
|
||||
BooksUploading int64
|
||||
BooksParsing int64
|
||||
}
|
||||
|
||||
// Observe reads the control plane's own state.
|
||||
//
|
||||
// The queue's table is asked for through to_regclass rather than assumed: River owns that schema and
|
||||
// migrates it separately (STACK_DECISIONS §19), so an instance whose queue migration has not run
|
||||
// answers a readiness probe about it and must not fail its telemetry over it as well.
|
||||
func (s *Store) Observe(ctx context.Context) (Observations, error) {
|
||||
const q = `
|
||||
select
|
||||
(select case when to_regclass('river_job') is null then 0 else
|
||||
(select count(*) from river_job
|
||||
where state in ('available', 'running', 'scheduled', 'retryable', 'pending')) end),
|
||||
(select coalesce(extract(epoch from (now() - min(opened_at))), 0)
|
||||
from reservations where state = 'open'),
|
||||
(select count(*) from run_attempts
|
||||
where quarantine_reason is not null and ended_at is null),
|
||||
(select count(*) from runs where finished_at is null),
|
||||
(select count(*) from books where status = 'uploading'),
|
||||
(select count(*) from books where status = 'parsing')`
|
||||
var o Observations
|
||||
err := s.pool.QueryRow(ctx, q).Scan(&o.QueueDepth, &o.OldestHoldSeconds, &o.QuarantinedAttempts,
|
||||
&o.LiveRuns, &o.BooksUploading, &o.BooksParsing)
|
||||
if err != nil {
|
||||
return Observations{}, fmt.Errorf("pgstore: observe: %w", err)
|
||||
}
|
||||
return o, nil
|
||||
}
|
||||
|
|
@ -105,7 +105,8 @@ func (s *Store) StartRun(ctx context.Context, in StartRunInput, journalOffset in
|
|||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
update books set status = 'translating', revision = revision + 1 where id = $1`, in.BookID); err != nil {
|
||||
update books set status = 'translating',
|
||||
revision = `+nextRevisionOfThisBooksLibrary+` where id = $1`, in.BookID); err != nil {
|
||||
return fmt.Errorf("pgstore: mark book translating: %w", err)
|
||||
}
|
||||
if enqueue != nil {
|
||||
|
|
@ -168,6 +169,13 @@ type LiveRun struct {
|
|||
// PausedReason is what the stream already said about this run. A run that reported a ceiling halt
|
||||
// is `paused` however its process then ended (contract §BookStatus: never `failed`).
|
||||
PausedReason string
|
||||
// Status is the run's product status. The reconciler works on live runs, where it is
|
||||
// `translating`; the resume path reads a run that has already ended and decides by it.
|
||||
Status string
|
||||
// StopRequestedAt is when THIS platform asked the run to stop, and it is the only thing that can
|
||||
// tell a stop from a crash: the engine catches SIGTERM and exits 1, so the marker says
|
||||
// `exit-code/exited/1` for both (register row PD-152). Nil means nobody asked.
|
||||
StopRequestedAt *time.Time
|
||||
// StartedAt is the RUN's start; AttemptStartedAt is THIS attempt's. The reconciler's grace is
|
||||
// measured against the second: a restarted attempt inherits a start time hours old, and the
|
||||
// grace then expires before systemd has had a chance to create anything.
|
||||
|
|
@ -202,7 +210,7 @@ func (s *Store) ListLiveRuns(ctx context.Context) ([]LiveRun, error) {
|
|||
// differ only in which runs they select, and a scan list copied twice is a scan list that drifts.
|
||||
const runColumns = `
|
||||
select r.id, r.book_id, b.owner_id, b.workdir, r.verify_bank, r.ceiling_chapters,
|
||||
coalesce(r.paused_reason, ''), r.started_at,
|
||||
coalesce(r.paused_reason, ''), r.started_at, r.status, r.stop_requested_at,
|
||||
a.id, a.attempt_no, coalesce(a.unit_name, ''), coalesce(a.engine_run_id, ''),
|
||||
a.last_offset, a.last_seq, a.last_line_sha256, a.quarantine_reason is not null,
|
||||
coalesce(res.amount_micro_usd, 0), a.engine_binary, a.spend_baseline_micro_usd,
|
||||
|
|
@ -210,8 +218,8 @@ const runColumns = `
|
|||
from runs r
|
||||
join books b on b.id = r.book_id`
|
||||
|
||||
func (s *Store) queryRuns(ctx context.Context, tail string) ([]LiveRun, error) {
|
||||
rows, err := s.pool.Query(ctx, runColumns+tail)
|
||||
func (s *Store) queryRuns(ctx context.Context, tail string, args ...any) ([]LiveRun, error) {
|
||||
rows, err := s.pool.Query(ctx, runColumns+tail, args...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pgstore: list runs: %w", err)
|
||||
}
|
||||
|
|
@ -222,7 +230,7 @@ func (s *Store) queryRuns(ctx context.Context, tail string) ([]LiveRun, error) {
|
|||
var ceiling, ceilingArg int64
|
||||
var baseline *int64
|
||||
if err := rows.Scan(&l.RunID, &l.BookID, &l.UserID, &l.Workdir, &l.VerifyBank, &l.CeilingChapters,
|
||||
&l.PausedReason, &l.StartedAt,
|
||||
&l.PausedReason, &l.StartedAt, &l.Status, &l.StopRequestedAt,
|
||||
&l.AttemptID, &l.AttemptNo, &l.UnitName, &l.EngineRunID,
|
||||
&l.Position.Offset, &l.Position.LastSeq, &l.Position.LastHash, &l.Quarantined, &ceiling,
|
||||
&l.EngineBinary, &baseline, &ceilingArg, &l.AttemptStartedAt); err != nil {
|
||||
|
|
@ -282,6 +290,114 @@ func (s *Store) SpendBound(ctx context.Context, bookID string, attemptID int64)
|
|||
return &bound, nil
|
||||
}
|
||||
|
||||
// ErrRunNotLive is a run that cannot be stopped because it is already over. Distinct from ErrNoRun,
|
||||
// which is a run this account cannot see at all: the first is the contract's 409 and the second its
|
||||
// 404, and answering the wrong one either tells a stranger that a run exists or tells an owner that
|
||||
// theirs does not.
|
||||
var ErrRunNotLive = errors.New("pgstore: the run is not live")
|
||||
|
||||
// ErrStopRequested is a restart refused because the run it would restart has been asked to stop. Not
|
||||
// an error of the caller: the reconciler ends the run instead.
|
||||
var ErrStopRequested = errors.New("pgstore: the run has been asked to stop")
|
||||
|
||||
// RequestStop records that THIS platform asked a run to stop, and hands back the unit to ask.
|
||||
//
|
||||
// The order is the whole design (migration 00014): the intent is COMMITTED before systemd is
|
||||
// touched, so a platform that dies between the two still knows on its next sweep that the run it
|
||||
// finds ended was stopped on purpose — and, just as important, that it must not restart it.
|
||||
//
|
||||
// Idempotent by coalesce: pressing stop twice is one intent with the FIRST timestamp, because the
|
||||
// timestamp is evidence about which of the two events came first and a later one would erase that.
|
||||
//
|
||||
// The revision it answers with is the BOOK's — see ReadRun for why every run-carrying response uses
|
||||
// one counter.
|
||||
func (s *Store) RequestStop(ctx context.Context, userID, runID string, now time.Time) (Run, string, error) {
|
||||
const q = `
|
||||
update runs r set stop_requested_at = coalesce(r.stop_requested_at, $3)
|
||||
from books b
|
||||
where r.id = $1 and b.id = r.book_id and b.owner_id = $2 and r.finished_at is null
|
||||
returning r.id, r.book_id, b.revision, r.status, r.verify_bank, r.ceiling_chapters,
|
||||
coalesce(r.paused_reason, ''), r.started_at, r.finished_at,
|
||||
coalesce((select a.unit_name from run_attempts a
|
||||
where a.run_id = r.id and a.ended_at is null
|
||||
order by a.attempt_no desc limit 1), '')`
|
||||
var out Run
|
||||
var unit string
|
||||
err := s.pool.QueryRow(ctx, q, runID, userID, now).Scan(&out.ID, &out.BookID, &out.Revision,
|
||||
&out.Status, &out.VerifyBank, &out.CeilingChapters, &out.PausedReason, &out.StartedAt,
|
||||
&out.FinishedAt, &unit)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
// Nothing matched, and the two reasons need different answers. Asked separately and only on
|
||||
// this path, so the ordinary stop stays one round trip.
|
||||
return Run{}, "", s.whyNotLive(ctx, userID, runID)
|
||||
}
|
||||
if err != nil {
|
||||
return Run{}, "", fmt.Errorf("pgstore: request stop: %w", err)
|
||||
}
|
||||
return out, unit, nil
|
||||
}
|
||||
|
||||
func (s *Store) whyNotLive(ctx context.Context, userID, runID string) error {
|
||||
var visible bool
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
select exists (select 1 from runs r join books b on b.id = r.book_id
|
||||
where r.id = $1 and b.owner_id = $2)`, runID, userID).Scan(&visible); err != nil {
|
||||
return fmt.Errorf("pgstore: read run: %w", err)
|
||||
}
|
||||
if visible {
|
||||
return ErrRunNotLive
|
||||
}
|
||||
return ErrNoRun
|
||||
}
|
||||
|
||||
// ReadRunForResume loads a run and its LAST attempt, whether or not either is still live.
|
||||
//
|
||||
// The reconciler's own list is deliberately not reusable here: it selects the attempt that has not
|
||||
// ended, and every run this call is about has ended. What resume needs is the state the run stopped
|
||||
// in and the attempt whose money and journal position it stopped at.
|
||||
func (s *Store) ReadRunForResume(ctx context.Context, userID, runID string) (LiveRun, error) {
|
||||
rows, err := s.queryRuns(ctx, `
|
||||
join run_attempts a on a.run_id = r.id
|
||||
and a.attempt_no = (select max(attempt_no) from run_attempts
|
||||
where run_id = r.id)
|
||||
left join reservations res on res.engine_run_id = r.id || '#' || a.attempt_no
|
||||
and res.state = 'open'
|
||||
where r.id = $1 and b.owner_id = $2`, runID, userID)
|
||||
if err != nil {
|
||||
return LiveRun{}, err
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
return LiveRun{}, ErrNoRun
|
||||
}
|
||||
return rows[0], nil
|
||||
}
|
||||
|
||||
// ReadRun is the run row as the contract projects it, read under the caller's ownership.
|
||||
//
|
||||
// ⚠ The revision is the BOOK's, not the `runs.revision` column, and that is the contract rather than
|
||||
// a shortcut: "the counter is PER BOOK — every book-scoped read and the id of every stream frame of
|
||||
// that book's run carry the same number" (§Revision), and a client MUST DROP a read whose revision is
|
||||
// below one it has applied. The book card already answers this way (httpapi getBook); a handle that
|
||||
// answered from the run's own column would hand the client a number below the card's and the client,
|
||||
// obeying the contract, would drop the answer to the button it just pressed.
|
||||
func (s *Store) ReadRun(ctx context.Context, userID, runID string) (Run, error) {
|
||||
const q = `
|
||||
select r.id, r.book_id, b.revision, r.status, r.verify_bank, r.ceiling_chapters,
|
||||
coalesce(r.paused_reason, ''), r.started_at, r.finished_at
|
||||
from runs r join books b on b.id = r.book_id
|
||||
where r.id = $1 and b.owner_id = $2`
|
||||
var out Run
|
||||
err := s.pool.QueryRow(ctx, q, runID, userID).Scan(&out.ID, &out.BookID, &out.Revision, &out.Status,
|
||||
&out.VerifyBank, &out.CeilingChapters, &out.PausedReason, &out.StartedAt, &out.FinishedAt)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return Run{}, ErrNoRun
|
||||
}
|
||||
if err != nil {
|
||||
return Run{}, fmt.Errorf("pgstore: read run: %w", err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// RestartInput is one interrupted attempt being replaced.
|
||||
type RestartInput struct {
|
||||
RunID string
|
||||
|
|
@ -311,12 +427,48 @@ func (s *Store) RestartRun(ctx context.Context, in RestartInput) (LiveRun, error
|
|||
if err := lockBook(ctx, tx, in.BookID); err != nil {
|
||||
return err
|
||||
}
|
||||
// The RUN row second — the order lockBook writes down, which this transaction used to take
|
||||
// last, after the money. Taking it here also does the work below: what is read from the row is
|
||||
// read under the lock that will do the writing.
|
||||
var stopRequested, finished *time.Time
|
||||
if err := tx.QueryRow(ctx,
|
||||
`select stop_requested_at, finished_at from runs where id = $1 for update`, in.RunID).
|
||||
Scan(&stopRequested, &finished); err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return ErrNoRun
|
||||
}
|
||||
return fmt.Errorf("pgstore: lock run: %w", err)
|
||||
}
|
||||
// A stop asked for on a run that is STILL LIVE outranks a restart, and the check belongs here
|
||||
// rather than in the caller's snapshot: the reconciler decides to restart, spends seconds
|
||||
// settling (a `tmctl status` call), and the user presses stop inside that window. Restarting
|
||||
// then cleared the fresh intent, opened a second attempt and took a new hold — the user had a
|
||||
// 202 for a stop that never happened and paid for the work they had just cancelled.
|
||||
//
|
||||
// On a FINISHED run the same column is history: a resume re-opens a run that was stopped, and
|
||||
// the intent belongs to the life that ended — it is cleared below with the rest of what said
|
||||
// the run was over.
|
||||
//
|
||||
// A stop arriving DURING this transaction is not lost either: it waits on this row lock and
|
||||
// lands on the re-opened run, which the next sweep then stops.
|
||||
if stopRequested != nil && finished == nil {
|
||||
return ErrStopRequested
|
||||
}
|
||||
// The previous attempt is closed if it is still open, and left exactly as it is if it is not.
|
||||
// Both callers arrive here: the reconciler replaces an attempt that was INTERRUPTED and is
|
||||
// still open, while a resume continues a run whose attempt already ended with a verdict of its
|
||||
// own — and overwriting that verdict would erase how the run the user stopped actually ended.
|
||||
//
|
||||
// What serializes two callers is no longer this row but the next one: `unique (run_id,
|
||||
// attempt_no)` lets exactly one of them insert attempt N+1, and the loser gets the same
|
||||
// ErrNoRun it always got.
|
||||
var attemptNo int
|
||||
if err := tx.QueryRow(ctx, `
|
||||
update run_attempts set ended_at = $2, exit_result = 'interrupted'
|
||||
where id = $1 and ended_at is null returning attempt_no`, in.AttemptID, in.Now).Scan(&attemptNo); err != nil {
|
||||
update run_attempts set ended_at = coalesce(ended_at, $2),
|
||||
exit_result = coalesce(exit_result, 'interrupted')
|
||||
where id = $1 returning attempt_no`, in.AttemptID, in.Now).Scan(&attemptNo); err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return ErrNoRun // another sweep got there first
|
||||
return ErrNoRun // the attempt is gone
|
||||
}
|
||||
return fmt.Errorf("pgstore: close interrupted attempt: %w", err)
|
||||
}
|
||||
|
|
@ -333,17 +485,41 @@ func (s *Store) RestartRun(ctx context.Context, in RestartInput) (LiveRun, error
|
|||
from run_attempts prev where prev.id = $6
|
||||
returning id, engine_binary`, in.RunID, next, in.Now, in.Offset, in.EngineBinary, in.AttemptID).
|
||||
Scan(&id, &pinned); err != nil {
|
||||
if isUnique(err, "run_attempts_run_id_attempt_no_key") {
|
||||
return ErrNoRun // another caller opened this attempt first
|
||||
}
|
||||
return fmt.Errorf("pgstore: open next attempt: %w", err)
|
||||
}
|
||||
if err := holdTx(ctx, tx, in.UserID, in.BookID, engineRunKey(in.RunID, next), in.Ceiling, in.Now); err != nil {
|
||||
return err
|
||||
}
|
||||
// finished_at, settled_at and stop_requested_at are CLEARED, and each for its own reason. The
|
||||
// reconciler's restart works on a run where all three are already null, so there it changes
|
||||
// nothing; a RESUME re-opens a run that ended, and leaving them would mean a live run the
|
||||
// one-live-per-book index does not see, money marked resolved that this attempt has not spent
|
||||
// yet, and a stop request from the previous life that would classify this attempt's ending as
|
||||
// a stop nobody asked for.
|
||||
if _, err := tx.Exec(ctx, `
|
||||
update runs set status = 'translating', paused_reason = null,
|
||||
update runs set status = 'translating', paused_reason = null, finished_at = null,
|
||||
settled_at = null, stop_requested_at = null,
|
||||
revision = (select revision + 1 from books where id = runs.book_id)
|
||||
where id = $1`, in.RunID); err != nil {
|
||||
// Clearing finished_at puts the run back under the one-live-run-per-book index, and the
|
||||
// book may already have a NEWER live run — nothing stops an account starting one after it
|
||||
// stopped this one. That is a conflict and not a failure: the whole transaction rolls back,
|
||||
// so the hold taken three lines above is undone with it, and the caller gets the same error
|
||||
// a second admission would have got.
|
||||
if isUnique(err, "runs_one_live_per_book") {
|
||||
return ErrRunInFlight
|
||||
}
|
||||
return fmt.Errorf("pgstore: reopen run: %w", err)
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
update books set status = 'translating',
|
||||
revision = `+nextRevisionOfThisBooksLibrary+` where id = $1`,
|
||||
in.BookID); err != nil {
|
||||
return fmt.Errorf("pgstore: mark book translating: %w", err)
|
||||
}
|
||||
out = LiveRun{RunID: in.RunID, BookID: in.BookID, UserID: in.UserID, AttemptID: id,
|
||||
AttemptNo: next, Ceiling: in.Ceiling, StartedAt: in.Now, AttemptStartedAt: in.Now,
|
||||
EngineBinary: pinned, Position: Position{Offset: in.Offset}}
|
||||
|
|
@ -403,21 +579,49 @@ func (s *Store) PauseRun(ctx context.Context, runID string, attemptID int64, rea
|
|||
}
|
||||
return fmt.Errorf("pgstore: lock book: %w", err)
|
||||
}
|
||||
// A stop the user asked for outranks a pause, and the check belongs inside this TRANSACTION,
|
||||
// under the same locks: the reconciler decides to pause after settling — seconds of a
|
||||
// `tmctl status` call — and a stop landing inside that window would otherwise be answered with
|
||||
// `paused/credit_exhausted`, which
|
||||
// says the money ran out when what happened is that its owner stopped it.
|
||||
var stopRequested *time.Time
|
||||
if err := tx.QueryRow(ctx,
|
||||
`select stop_requested_at from runs where id = $1 for update`, runID).Scan(&stopRequested); err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("pgstore: lock run: %w", err)
|
||||
}
|
||||
if stopRequested != nil {
|
||||
return ErrStopRequested
|
||||
}
|
||||
// The attempt being closed must still be a LIVE attempt OF THIS RUN — the third path to carry
|
||||
// the guard `FinishRun` (PD-181) and `FinishUnspawnedStop` (FP5-2) already have, and it was
|
||||
// missing here. Without it a pass holding an old snapshot pauses a run whose attempt has since
|
||||
// been restarted: the run reads `finished`, the SECOND attempt's hold stays open and falls out
|
||||
// of both `ListLiveRuns` (the run is finished) and `UnsettledRuns` (the attempt is not), and an
|
||||
// engine keeps spending under a run its owner is told ran out of money. Found by cross-family
|
||||
// review of the acceptance dofix.
|
||||
if err := tx.QueryRow(ctx, `
|
||||
update runs set status = 'paused', paused_reason = $2, finished_at = $3,
|
||||
revision = (select revision + 1 from books where id = runs.book_id)
|
||||
where id = $1 and finished_at is null returning book_id`, runID, reason, now).Scan(&bookID); err != nil {
|
||||
where id = $1 and finished_at is null
|
||||
and exists (select 1 from run_attempts a
|
||||
where a.id = $4 and a.run_id = runs.id and a.ended_at is null)
|
||||
returning book_id`, runID, reason, now, attemptID).Scan(&bookID); err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("pgstore: pause run: %w", err)
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
update run_attempts set ended_at = coalesce(ended_at, $2) where id = $1`, attemptID, now); err != nil {
|
||||
update run_attempts set ended_at = coalesce(ended_at, $3)
|
||||
where id = $1 and run_id = $2`, attemptID, runID, now); err != nil {
|
||||
return fmt.Errorf("pgstore: end attempt: %w", err)
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
update books set status = 'paused', revision = revision + 1 where id = $1`, bookID); err != nil {
|
||||
update books set status = 'paused',
|
||||
revision = `+nextRevisionOfThisBooksLibrary+` where id = $1`, bookID); err != nil {
|
||||
return fmt.Errorf("pgstore: pause book: %w", err)
|
||||
}
|
||||
return nil
|
||||
|
|
@ -467,12 +671,24 @@ type SpawnRecord struct {
|
|||
// Written BEFORE the unit is created, so a crash between the two leaves a record to reconcile rather
|
||||
// than an unattributable process.
|
||||
//
|
||||
// claimed is false when the attempt already has a unit name. It is a COMPARE-AND-SET rather than a
|
||||
// plain update because two callers legitimately reach here at once — the queue worker that was
|
||||
// handed the run and the reconciler that found it unspawned — and the loser must not start a second
|
||||
// engine. systemd would refuse the duplicate NAME, so the accident was survivable; surviving by
|
||||
// someone else's uniqueness rule is not the same as being correct, and the day a resume changes the
|
||||
// naming it stops holding.
|
||||
// claimed is false when the attempt already has a unit name, when the attempt has ENDED, or when its
|
||||
// run is over. It is a COMPARE-AND-SET rather than a plain update because two callers legitimately
|
||||
// reach here at once — the queue worker that was handed the run and the reconciler that found it
|
||||
// unspawned — and the loser must not start a second engine. systemd would refuse the duplicate NAME,
|
||||
// so that accident was survivable; surviving by someone else's uniqueness rule is not the same as
|
||||
// being correct, and the day a resume changes the naming it stops holding.
|
||||
//
|
||||
// ⚠ The three conditions BESIDES the unit name are money, and they were not here until a run could be
|
||||
// stopped before it ever spawned. The window: a worker sits inside `bookMeter` for the seconds a
|
||||
// `tmctl status` takes, the user stops the run, the sweep ends it and gives the whole hold back
|
||||
// (nothing was spawned) — and the worker then wakes up and creates a unit for a run that is finished
|
||||
// and settled. That engine would spend against its own book cap with NO open reservation, and
|
||||
// nothing would ever look at it: the reconciler lists runs by `finished_at is null` and settlements
|
||||
// by an open reservation, so it is in neither list. The stop INTENT is asked about one step earlier
|
||||
// for the same reason: it lands while the worker is inside that same `bookMeter`, before any sweep
|
||||
// could have finished the run, and a claim granted then starts an engine for work its owner
|
||||
// cancelled before it began. All three are asked in the statement that claims, because a check made
|
||||
// before it is a check with a window after it.
|
||||
func (s *Store) RecordSpawn(ctx context.Context, r SpawnRecord) (claimed bool, err error) {
|
||||
// What a PREVIOUS claim of this attempt decided is kept. The claim is given back when the unit
|
||||
// could not be created (ReleaseSpawnClaim), and "could not be created" is not the same as "was not
|
||||
|
|
@ -487,7 +703,9 @@ func (s *Store) RecordSpawn(ctx context.Context, r SpawnRecord) (claimed bool, e
|
|||
ceiling_arg_micro_usd = case when spend_baseline_micro_usd is null
|
||||
then $5 else ceiling_arg_micro_usd end,
|
||||
spend_baseline_micro_usd = coalesce(spend_baseline_micro_usd, $6)
|
||||
where id = $1 and unit_name is null`,
|
||||
where id = $1 and unit_name is null and ended_at is null
|
||||
and exists (select 1 from runs r where r.id = run_attempts.run_id
|
||||
and r.finished_at is null and r.stop_requested_at is null)`,
|
||||
r.AttemptID, r.Unit, r.Binary, int64(r.Ceiling), int64(r.CeilingArg), int64(r.Baseline))
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("pgstore: record spawn: %w", err)
|
||||
|
|
|
|||
|
|
@ -425,11 +425,11 @@ func TestFinishingARunTwiceChangesNothingTheSecondTime(t *testing.T) {
|
|||
t.Fatal(err)
|
||||
}
|
||||
code := 0
|
||||
if err := s.FinishRun(ctx, run.ID, run.AttemptID, "ready", "success", &code, now); err != nil {
|
||||
if _, err := s.FinishRun(ctx, run.ID, run.AttemptID, "ready", "success", &code, now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
later := now.Add(time.Hour)
|
||||
if err := s.FinishRun(ctx, run.ID, run.AttemptID, "failed", "exit-code", nil, later); err != nil {
|
||||
if _, err := s.FinishRun(ctx, run.ID, run.AttemptID, "failed", "exit-code", nil, later); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var status string
|
||||
|
|
@ -465,7 +465,7 @@ func TestAFinishedRunWithAnOpenHoldIsListedAsUnsettled(t *testing.T) {
|
|||
t.Fatal(err)
|
||||
}
|
||||
code := 0
|
||||
if err := s.FinishRun(ctx, run.ID, run.AttemptID, "ready", "success", &code, now); err != nil {
|
||||
if _, err := s.FinishRun(ctx, run.ID, run.AttemptID, "ready", "success", &code, now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
open, err := s.UnsettledRuns(ctx)
|
||||
|
|
@ -750,7 +750,7 @@ func TestTheSettlementWorklistAndItsStampAreBothIdempotent(t *testing.T) {
|
|||
t.Fatal(err)
|
||||
}
|
||||
code := 0
|
||||
if err := s.FinishRun(ctx, run.ID, run.AttemptID, "ready", "success", &code, now); err != nil {
|
||||
if _, err := s.FinishRun(ctx, run.ID, run.AttemptID, "ready", "success", &code, now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.Settle(ctx, ReservationKey(run.ID, 1), money.MicroUSD(100_000), now); err != nil {
|
||||
|
|
@ -904,3 +904,106 @@ func TestOnlyOneSweepCanRestartAnInterruptedAttempt(t *testing.T) {
|
|||
t.Errorf("%d attempts after four concurrent restarts", attempts)
|
||||
}
|
||||
}
|
||||
|
||||
// Cross-family review of the acceptance dofix (M1): a pause is a CLOSING path, and every closing path
|
||||
// must check that the attempt it closes is still a live attempt of this run. `FinishRun` and
|
||||
// `FinishUnspawnedStop` do; `PauseRun` did not, so a pass holding a snapshot of the attempt that was
|
||||
// already restarted answered "the credit ran out" for a run whose second attempt is spending right
|
||||
// now — and that second hold then belongs to no list at all.
|
||||
func TestAPauseFromAnOldSnapshotDoesNotCloseARunOverALiveAttempt(t *testing.T) {
|
||||
s, ctx := testDB(t)
|
||||
now := fundedAccount(t, s, ctx, "u1", "10")
|
||||
seedBook(t, s, ctx, "bk1", "u1", 500)
|
||||
run, err := s.StartRun(ctx, StartRunInput{UserID: "u1", BookID: "bk1", CeilingChapters: 100,
|
||||
Ceiling: money.MicroUSD(3_000_000), Now: now}, 0, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.Settle(ctx, ReservationKey(run.ID, 1), money.MicroUSD(500_000), now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
next, err := s.RestartRun(ctx, RestartInput{RunID: run.ID, AttemptID: run.AttemptID, UserID: "u1",
|
||||
BookID: "bk1", Ceiling: money.MicroUSD(1_000_000), Now: now})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.PauseRun(ctx, run.ID, run.AttemptID, PausedCreditExhausted, now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var status string
|
||||
var finished *time.Time
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`select status, finished_at from runs where id = $1`, run.ID).Scan(&status, &finished); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if finished != nil {
|
||||
t.Fatalf("a stale pause closed the run as %q while attempt %d is live", status, next.AttemptID)
|
||||
}
|
||||
var ended *time.Time
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`select ended_at from run_attempts where id = $1`, next.AttemptID).Scan(&ended); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ended != nil {
|
||||
t.Fatal("a stale pause ended the LIVE attempt")
|
||||
}
|
||||
// The money half of the same fact: the live attempt's hold must be findable. A closed run with an
|
||||
// open hold is in neither list — not in the live runs (the run is finished) and not in the
|
||||
// unsettled ones (the attempt is not) — which is how a hold is lost for good.
|
||||
live, err := s.ListLiveRuns(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
found := false
|
||||
for _, l := range live {
|
||||
if l.AttemptID == next.AttemptID {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("the live attempt %d is not in ListLiveRuns after a stale pause", next.AttemptID)
|
||||
}
|
||||
}
|
||||
|
||||
// Re-check of the dofix (FP5-11): the rule the intake writers got — take the LIBRARY's next number,
|
||||
// not your own plus one — has to hold for the run's writers too. A book that is not the account's
|
||||
// newest carries a counter below the library's, so `translating`, `paused` and `stopped` on it left
|
||||
// `greatest(max, floor)` exactly where it was and the library screen never saw the run start.
|
||||
func TestARunTransitionOnAnOlderBookStillMovesTheLibrary(t *testing.T) {
|
||||
s, ctx := testDB(t)
|
||||
now := fundedAccount(t, s, ctx, "u1", "10")
|
||||
seedBook(t, s, ctx, "bk1", "u1", 500)
|
||||
// A second, NEWER book carries the library's maximum, so bk1's own counter is not it.
|
||||
seedBook(t, s, ctx, "bk2", "u1", 500)
|
||||
exec(t, s, ctx, `update books set revision = 41 where id = 'bk2'`)
|
||||
before, err := s.ListBooks(ctx, "u1", 50, "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
run, err := s.StartRun(ctx, StartRunInput{UserID: "u1", BookID: "bk1", CeilingChapters: 10,
|
||||
Ceiling: money.MicroUSD(300_000), Now: now}, 0, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
after, err := s.ListBooks(ctx, "u1", 50, "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if after.Revision <= before.Revision {
|
||||
t.Fatalf("a run started on an older book and the library still answers %d (was %d): a client "+
|
||||
"obeying the contract drops that read and never sees the book start translating",
|
||||
after.Revision, before.Revision)
|
||||
}
|
||||
// And the same for the closing half of the lifecycle.
|
||||
mid := after.Revision
|
||||
if err := s.PauseRun(ctx, run.ID, run.AttemptID, PausedCreditExhausted, now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
end, err := s.ListBooks(ctx, "u1", 50, "")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if end.Revision <= mid {
|
||||
t.Fatalf("the run was paused and the library still answers %d (was %d)", end.Revision, mid)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -34,15 +34,23 @@ func (s *Store) NewRunSink(attemptID int64, runID, bookID string) *RunSink {
|
|||
// The binding is refused if the attempt already carries a DIFFERENT id: that means two engine
|
||||
// processes wrote into one journal under one attempt, and materializing either of them would mix
|
||||
// two runs' counters into one projection.
|
||||
//
|
||||
// It is refused for an ENDED attempt too, and that half is newer than the rule above. A stop that
|
||||
// landed before its engine was spawned leaves an attempt that is closed and carries no engine run id
|
||||
// — a shape this zone could not produce before P5 — and an unbound, ended attempt would otherwise
|
||||
// ADOPT the handshake of the attempt that replaced it: the same events would then be materialized
|
||||
// twice, once per attempt, and every counter they increment would be counted twice. Found by
|
||||
// cross-family review of the acceptance dofix.
|
||||
func (r *RunSink) Begin(ctx context.Context, h ingest.Hello) error {
|
||||
tag, err := r.store.pool.Exec(ctx, `
|
||||
update run_attempts set engine_run_id = $2
|
||||
where id = $1 and (engine_run_id is null or engine_run_id = $2)`, r.attemptID, h.EngineRunID)
|
||||
where id = $1 and ended_at is null
|
||||
and (engine_run_id is null or engine_run_id = $2)`, r.attemptID, h.EngineRunID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pgstore: bind engine run: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return fmt.Errorf("pgstore: attempt %d is already bound to another engine run", r.attemptID)
|
||||
return fmt.Errorf("pgstore: attempt %d is over, or already bound to another engine run", r.attemptID)
|
||||
}
|
||||
if h.ChunkerVersion != "" {
|
||||
if _, err := r.store.pool.Exec(ctx,
|
||||
|
|
@ -257,11 +265,23 @@ func (s *Store) ApplyStatus(ctx context.Context, runID, bookID string, rep inges
|
|||
|
||||
// FinishRun closes a run and its attempt: the read-model status, the end of the attempt and what
|
||||
// systemd said about it. Money is NOT touched here — see Settlement.
|
||||
func (s *Store) FinishRun(ctx context.Context, runID string, attemptID int64, status, exitResult string, exitCode *int, now time.Time) error {
|
||||
//
|
||||
// closed is false when the write did not apply, and the caller must then do nothing else: the run was
|
||||
// already finished, or — the case that made this a bool — the attempt it was asked to close is no
|
||||
// longer the run's live one.
|
||||
//
|
||||
// ⚠ That second guard is money. A sweep decides from a snapshot and writes seconds later; between the
|
||||
// two, the user can stop the run and RESUME it, and the resumed run is live again with a second
|
||||
// attempt holding a second reservation. The old attempt's exit marker is still on disk — nothing
|
||||
// deletes markers of attempts that ended — so the stale pass would close the run from it, leaving
|
||||
// attempt 2's hold in NO worklist: `ListLiveRuns` selects runs with `finished_at is null` and
|
||||
// `UnsettledRuns` attempts with `ended_at is not null`, and the resumed run matches neither once it
|
||||
// has been re-finished. Reachable only since a run can come back to life at all, which is this pack.
|
||||
func (s *Store) FinishRun(ctx context.Context, runID string, attemptID int64, status, exitResult string, exitCode *int, now time.Time) (closed bool, err error) {
|
||||
if !validRunStatus(status) {
|
||||
return fmt.Errorf("pgstore: %q is not a run status", status)
|
||||
return false, fmt.Errorf("pgstore: %q is not a run status", status)
|
||||
}
|
||||
return s.inTx(ctx, func(tx pgx.Tx) error {
|
||||
err = s.inTx(ctx, func(tx pgx.Tx) error {
|
||||
// The BOOK's row is locked first, here and in the materializer. The two used to take them in
|
||||
// opposite orders — the materializer books-then-runs, this one runs-then-books — and two
|
||||
// reconcilers on one run (overlapping deploy generations) then deadlocked in both directions;
|
||||
|
|
@ -278,9 +298,14 @@ func (s *Store) FinishRun(ctx context.Context, runID string, attemptID int64, st
|
|||
if err := tx.QueryRow(ctx, `
|
||||
update runs set status = $2, finished_at = $3,
|
||||
revision = (select revision + 1 from books where id = runs.book_id)
|
||||
where id = $1 and finished_at is null returning book_id`, runID, status, now).Scan(&bookID); err != nil {
|
||||
where id = $1 and finished_at is null
|
||||
and exists (select 1 from run_attempts a
|
||||
where a.id = $4 and a.run_id = runs.id and a.ended_at is null)
|
||||
returning book_id`, runID, status, now, attemptID).Scan(&bookID); err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil // already finished: finishing is idempotent by design, the reconciler retries
|
||||
// Already finished, or asked about an attempt that is no longer the live one. Both are
|
||||
// "someone else got here first"; finishing is idempotent by refusal, not by repetition.
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("pgstore: finish run: %w", err)
|
||||
}
|
||||
|
|
@ -290,11 +315,14 @@ func (s *Store) FinishRun(ctx context.Context, runID string, attemptID int64, st
|
|||
return fmt.Errorf("pgstore: finish attempt: %w", err)
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
update books set status = $2, revision = revision + 1 where id = $1`, bookID, status); err != nil {
|
||||
update books set status = $2,
|
||||
revision = `+nextRevisionOfThisBooksLibrary+` where id = $1`, bookID, status); err != nil {
|
||||
return fmt.Errorf("pgstore: finish book: %w", err)
|
||||
}
|
||||
closed = true
|
||||
return nil
|
||||
})
|
||||
return closed, err
|
||||
}
|
||||
|
||||
func validRunStatus(s string) bool {
|
||||
|
|
@ -305,6 +333,82 @@ func validRunStatus(s string) bool {
|
|||
return false
|
||||
}
|
||||
|
||||
// FinishUnspawnedStop closes a run that was stopped before its unit ever existed, and only if that is
|
||||
// still true when the write happens.
|
||||
//
|
||||
// The re-check is the whole method. The reconciler decides from a snapshot in which the attempt had
|
||||
// no unit; between that read and this write the queue worker can claim the attempt and create one,
|
||||
// and closing the run then leaves an engine spending against a book with no open reservation and no
|
||||
// list that looks at it — the reconciler lists by `finished_at is null` and settlement by an open
|
||||
// reservation, so it would be in neither. `ReleaseUnspawned` already guards the MONEY of exactly this
|
||||
// window under a lock (PD-159); this is the same guard for the lifecycle the money follows.
|
||||
//
|
||||
// finished is false when the attempt was spawned after all: the caller does nothing, and the next
|
||||
// sweep meets an ordinary live run — with a unit to signal and an intent that says to.
|
||||
func (s *Store) FinishUnspawnedStop(ctx context.Context, runID string, attemptID int64, now time.Time) (finished bool, err error) {
|
||||
err = s.inTx(ctx, func(tx pgx.Tx) error {
|
||||
// Book first, then the attempt: the order every transaction in this package takes (lockBook).
|
||||
var bookID string
|
||||
if err := tx.QueryRow(ctx,
|
||||
`select id from books where id = (select book_id from runs where id = $1) for update`,
|
||||
runID).Scan(&bookID); err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil // the run is gone
|
||||
}
|
||||
return fmt.Errorf("pgstore: lock book: %w", err)
|
||||
}
|
||||
var unit *string
|
||||
var ended *time.Time
|
||||
if err := tx.QueryRow(ctx,
|
||||
`select unit_name, ended_at from run_attempts where id = $1 and run_id = $2 for update`,
|
||||
attemptID, runID).Scan(&unit, &ended); err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("pgstore: lock attempt: %w", err)
|
||||
}
|
||||
if unit != nil && *unit != "" {
|
||||
return nil // spawned inside the window; not this path's business any more
|
||||
}
|
||||
if ended != nil {
|
||||
// The attempt this pass is holding is over, which means the run moved on without it — a stop
|
||||
// and a RESUME can both have happened since the snapshot was taken, and the run is live
|
||||
// again on a second attempt holding a second reservation. Closing it from here would put
|
||||
// that hold in no worklist at all. The same guard FinishRun carries (PD-181); acceptance
|
||||
// found this path missing it.
|
||||
return nil
|
||||
}
|
||||
tag, err := tx.Exec(ctx, `
|
||||
update runs set status = 'stopped', finished_at = $2,
|
||||
revision = (select revision + 1 from books where id = runs.book_id)
|
||||
where id = $1 and finished_at is null`, runID, now)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pgstore: finish stopped run: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return nil // already finished by an earlier pass
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
update run_attempts set ended_at = $2, exit_result = $3
|
||||
where id = $1 and ended_at is null`, attemptID, now, StopRequestedResult); err != nil {
|
||||
return fmt.Errorf("pgstore: finish attempt: %w", err)
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
update books set status = 'stopped',
|
||||
revision = `+nextRevisionOfThisBooksLibrary+` where id = $1`, bookID); err != nil {
|
||||
return fmt.Errorf("pgstore: finish book: %w", err)
|
||||
}
|
||||
finished = true
|
||||
return nil
|
||||
})
|
||||
return finished, err
|
||||
}
|
||||
|
||||
// StopRequestedResult is the exit_result of an attempt that ended by this platform's request without
|
||||
// systemd getting to write a marker. Deliberately a value systemd cannot produce: an operator reading
|
||||
// the column must be able to tell what the machine saw from what the platform concluded.
|
||||
const StopRequestedResult = "stop-requested"
|
||||
|
||||
// MarkSettled records that the money of a run has been resolved.
|
||||
func (s *Store) MarkSettled(ctx context.Context, runID string, now time.Time) error {
|
||||
_, err := s.pool.Exec(ctx, `update runs set settled_at = $2 where id = $1 and settled_at is null`, runID, now)
|
||||
|
|
|
|||
|
|
@ -194,7 +194,8 @@ func TestAMaterializerAndAReconcilerOnOneRunDoNotDeadlock(t *testing.T) {
|
|||
}()
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
errs <- s.FinishRun(ctx, runID, run.AttemptID, "ready", "exit-code", nil, now)
|
||||
_, err := s.FinishRun(ctx, runID, run.AttemptID, "ready", "exit-code", nil, now)
|
||||
errs <- err
|
||||
}()
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
|
|
@ -378,3 +379,33 @@ func TestAResyncRecordsWhenItWasTaken(t *testing.T) {
|
|||
t.Errorf("the second resync left the stamp at %v, want %s", stamp, later)
|
||||
}
|
||||
}
|
||||
|
||||
// Cross-family review of the acceptance dofix (M2): a stop that lands before its engine was spawned
|
||||
// leaves an attempt that is ENDED and carries no engine run id — a shape this zone could not produce
|
||||
// before the stop path existed. An old materializer still holding that attempt would then bind it to
|
||||
// the handshake of the attempt that REPLACED it and apply the same journal a second time, counting
|
||||
// every unit and chapter twice. Binding is refused for an attempt that is over.
|
||||
func TestAMaterializerOfAnEndedAttemptDoesNotAdoptTheNextAttemptsEngine(t *testing.T) {
|
||||
s, ctx := testDB(t)
|
||||
now := fundedAccount(t, s, ctx, "u1", "10")
|
||||
seedBook(t, s, ctx, "bk1", "u1", 500)
|
||||
run, err := s.StartRun(ctx, StartRunInput{UserID: "u1", BookID: "bk1", CeilingChapters: 10,
|
||||
Ceiling: money.MicroUSD(300_000), Now: now}, 0, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The stop-before-spawn shape: the attempt is closed without ever having been bound.
|
||||
exec(t, s, ctx, `update run_attempts set ended_at = now() where id = $1`, run.AttemptID)
|
||||
stale := s.NewRunSink(run.AttemptID, run.ID, "bk1")
|
||||
if err := stale.Begin(ctx, ingest.Hello{EngineRunID: "eng-of-the-next-attempt"}); err == nil {
|
||||
t.Fatal("an ended attempt adopted an engine run: its journal would be materialized twice")
|
||||
}
|
||||
var bound *string
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`select engine_run_id from run_attempts where id = $1`, run.AttemptID).Scan(&bound); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if bound != nil {
|
||||
t.Fatalf("the ended attempt was bound to %q", *bound)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import (
|
|||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
|
@ -96,6 +97,57 @@ func StatusArgs(workdir string) []string {
|
|||
return []string{"status", "--config", filepath.Join(workdir, ConfigFile), "--json"}
|
||||
}
|
||||
|
||||
// ManifestArgs is the intake channel: `tmctl manifest --config <book.yaml> --json`.
|
||||
//
|
||||
// $0 and key-less like `status` — it ingests and cuts the source and writes a sidecar, with no
|
||||
// provider call and no wave — and it is the ONE engine command the platform runs before a book has
|
||||
// ever been translated. ⚠ Its first touch of a project CREATES and migrates the engine's database
|
||||
// (ratified D39.122), which is why intake needs the book's directory writable and why it is this
|
||||
// command rather than `status`: `status` has the same side effect and answers a different question.
|
||||
func ManifestArgs(workdir string) []string {
|
||||
return []string{"manifest", "--config", filepath.Join(workdir, ConfigFile), "--json"}
|
||||
}
|
||||
|
||||
// maxManifest bounds what the platform will read from the manifest command. The document is counts
|
||||
// and identities — no source text — so a real one is single-digit megabytes even for a 2283-chapter
|
||||
// book; the cap is against a process that is not the engine, not against the engine.
|
||||
const maxManifest = 64 << 20
|
||||
|
||||
// Manifest builds the book's chapter tree and returns the allowlisted summary of it.
|
||||
func (r *Runner) Manifest(ctx context.Context, binary, workdir string) (ingest.Manifest, error) {
|
||||
cmd := exec.CommandContext(ctx, binary, ManifestArgs(workdir)...)
|
||||
cmd.Dir = workdir
|
||||
var errOut bytes.Buffer
|
||||
cmd.Stderr = &errOut
|
||||
out, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return ingest.Manifest{}, fmt.Errorf("runner: tmctl manifest: %w", err)
|
||||
}
|
||||
if err := cmd.Start(); err != nil {
|
||||
return ingest.Manifest{}, fmt.Errorf("runner: tmctl manifest: %w", err)
|
||||
}
|
||||
doc, readErr := io.ReadAll(io.LimitReader(out, maxManifest+1))
|
||||
// Whatever is left is DRAINED, and that is not tidiness: a pipe nobody reads blocks the child on
|
||||
// its next write and Wait then blocks on the child. Stopping at the cap without draining does not
|
||||
// truncate the document — it hangs the caller, or, once the pipe is closed, kills the engine with
|
||||
// EPIPE and reports that as a host which cannot run the engine.
|
||||
_, _ = io.Copy(io.Discard, out)
|
||||
if err := cmd.Wait(); err != nil {
|
||||
// The engine's stderr is engine vocabulary and must not become a platform log line verbatim,
|
||||
// let alone a product phrase; what travels is the fact and the first line. The error is
|
||||
// returned UNWRAPPED as an *exec.ExitError underneath, because the caller tells "the engine
|
||||
// answered no" from "the engine could not be run" by exactly that type.
|
||||
return ingest.Manifest{}, fmt.Errorf("runner: tmctl manifest: %w: %s", err, firstLine(errOut.Bytes()))
|
||||
}
|
||||
if readErr != nil {
|
||||
return ingest.Manifest{}, fmt.Errorf("runner: read manifest: %w", readErr)
|
||||
}
|
||||
if int64(len(doc)) > maxManifest {
|
||||
return ingest.Manifest{}, fmt.Errorf("runner: the manifest is over %d bytes, which no book produces", maxManifest)
|
||||
}
|
||||
return ingest.DecodeManifest(doc)
|
||||
}
|
||||
|
||||
// Status runs the reconciliation channel and decodes the allowlisted subset of its report.
|
||||
func (r *Runner) Status(ctx context.Context, binary, workdir string) (ingest.StatusReport, error) {
|
||||
cmd := exec.CommandContext(ctx, binary, StatusArgs(workdir)...)
|
||||
|
|
|
|||
127
platform/internal/runner/manifest_test.go
Normal file
127
platform/internal/runner/manifest_test.go
Normal file
|
|
@ -0,0 +1,127 @@
|
|||
package runner
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"go/parser"
|
||||
"go/printer"
|
||||
"go/token"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The intake channel is the one engine command the platform runs before a book has ever been
|
||||
// translated. These tests run a REAL process — a shell script standing in for tmctl — because what
|
||||
// the caller decides on is the difference between "the engine answered no" and "the engine could not
|
||||
// be run", and that difference is a type from os/exec rather than a value the platform chooses.
|
||||
|
||||
// fakeEngine writes an executable that behaves as told and returns its path.
|
||||
func fakeEngine(t *testing.T, script string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "tmctl")
|
||||
if err := os.WriteFile(path, []byte("#!/bin/sh\n"+script+"\n"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestTheManifestCommandIsTheZeroCostOneAndCarriesTheBooksConfig(t *testing.T) {
|
||||
got := ManifestArgs("/srv/books/bk_1")
|
||||
want := []string{"manifest", "--config", "/srv/books/bk_1/book.yaml", "--json"}
|
||||
if strings.Join(got, " ") != strings.Join(want, " ") {
|
||||
t.Fatalf("argv %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAManifestIsReadForTheFourThingsIntakeDecidesWith(t *testing.T) {
|
||||
// A document with far more in it than the platform takes: the per-chapter and per-unit arrays are
|
||||
// the engine's and have no reader on this side.
|
||||
bin := fakeEngine(t, `cat <<'JSON'
|
||||
{"manifest_version":"tm-manifest-v2","book_id":"gu","key":"abcdef",
|
||||
"chunker_version":"chunk-2026.07","source_sha256":"`+strings.Repeat("ab", 32)+`",
|
||||
"source_bytes":24117248,"source_lang":"zh","target_lang":"ru","encoding":"utf-8",
|
||||
"chapters_total":2283,"units_total":4402,"chunks_total":9130,
|
||||
"chapters":[{"id":"0011223344556677","number":1,"heading":"Глава 1","units_total":2,
|
||||
"chunks_total":4,"units":[{"id":"0011223344556677:aabbccdd:0","first_chunk_idx":0,
|
||||
"chunk_count":2,"edit_unit_id":1}]}]}
|
||||
JSON`)
|
||||
r := New(nil)
|
||||
m, err := r.Manifest(t.Context(), bin, t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if m.ChaptersTotal != 2283 || m.UnitsTotal != 4402 {
|
||||
t.Errorf("counts %d/%d", m.ChaptersTotal, m.UnitsTotal)
|
||||
}
|
||||
if m.Version != "tm-manifest-v2" || m.ChunkerVersion != "chunk-2026.07" {
|
||||
t.Errorf("identity %q/%q", m.Version, m.ChunkerVersion)
|
||||
}
|
||||
if len(m.SourceSHA256Bytes()) != 32 {
|
||||
t.Errorf("the source digest did not decode: %q", m.SourceSHA256)
|
||||
}
|
||||
}
|
||||
|
||||
// The classifier the intake keys on: a process that RAN and refused carries *exec.ExitError, and one
|
||||
// that could not be run does not. Getting this backwards either retries a book forever or rejects a
|
||||
// user's file because a host was misconfigured.
|
||||
func TestARefusalAndAnAbsenceAreDifferentErrors(t *testing.T) {
|
||||
r := New(nil)
|
||||
refused := fakeEngine(t, `echo "chunk: ingest txt: not a book" >&2; exit 1`)
|
||||
_, err := r.Manifest(t.Context(), refused, t.TempDir())
|
||||
var exit *exec.ExitError
|
||||
if !errors.As(err, &exit) {
|
||||
t.Fatalf("a refusal came back as %T: %v", err, err)
|
||||
}
|
||||
_, err = r.Manifest(t.Context(), filepath.Join(t.TempDir(), "no-such-binary"), t.TempDir())
|
||||
if errors.As(err, &exit) {
|
||||
t.Fatalf("a missing binary came back as an exit status: %v", err)
|
||||
}
|
||||
// The engine's own words stay out of the platform's error, which is what a product phrase is
|
||||
// eventually written from (contract §Problem).
|
||||
if _, err := r.Manifest(t.Context(), refused, t.TempDir()); !strings.Contains(err.Error(), "tmctl manifest") {
|
||||
t.Errorf("the error does not name what failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAManifestThatIsNotJSONIsAnError(t *testing.T) {
|
||||
r := New(nil)
|
||||
bin := fakeEngine(t, `echo "=== MANIFEST: gu ==="`)
|
||||
if _, err := r.Manifest(t.Context(), bin, t.TempDir()); err == nil {
|
||||
t.Fatal("a human summary was accepted as a manifest document")
|
||||
}
|
||||
}
|
||||
|
||||
// FP5-7 (acceptance): the gate on these tests must ask whether this host CAN run them, not whether
|
||||
// systemd phrased its refusal the way it did in 2026. It skipped on "Failed to connect to bus" and
|
||||
// systemd 259 answers "Failed to connect to user scope bus", so on a host with no user manager the
|
||||
// tests failed instead of skipping — a gate that stopped gating when somebody else edited a string.
|
||||
func TestTheSystemdGateAsksAboutTheCapabilityAndNotAMessage(t *testing.T) {
|
||||
src, err := os.ReadFile("systemd_test.go")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The CODE only: the comments there explain the history of this very defect and quote the message
|
||||
// the gate used to match, which a plain text scan would read as the defect itself.
|
||||
file, err := parser.ParseFile(token.NewFileSet(), "systemd_test.go", src, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var code bytes.Buffer
|
||||
if err := printer.Fprint(&code, token.NewFileSet(), file); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// ⚠ The assertion is on the WORDS systemd could change, not on one spelling of the old check:
|
||||
// pinning two exact expressions is a gate that a third spelling walks straight past (found by
|
||||
// cross-family review of the acceptance dofix). Any occurrence of systemd's refusal text in this
|
||||
// file means the gate is reading messages again.
|
||||
if bytes.Contains(code.Bytes(), []byte("Failed to connect")) ||
|
||||
bytes.Contains(code.Bytes(), []byte("No medium found")) {
|
||||
t.Fatal("the systemd gate matches a message again: it must skip on the failure to reach the manager, whatever systemd calls it")
|
||||
}
|
||||
if !bytes.Contains(code.Bytes(), []byte("systemctl")) {
|
||||
t.Fatal("the systemd gate no longer asks the manager anything: a gate that probes nothing cannot know whether these tests can run")
|
||||
}
|
||||
}
|
||||
|
|
@ -1,6 +1,7 @@
|
|||
package runner
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
|
|
@ -21,9 +22,16 @@ func systemdOrSkip(t *testing.T) {
|
|||
if _, err := exec.LookPath("systemd-run"); err != nil {
|
||||
t.Skip("no systemd-run: the transient-unit properties cannot be measured on this host")
|
||||
}
|
||||
// The gate is the CAPABILITY, not a sentence: this used to skip on the words "Failed to connect to
|
||||
// bus" and systemd 259 says "Failed to connect to user scope bus", so on a host without a manager
|
||||
// the tests FAILED instead of skipping — a gate that stopped gating when someone else edited a
|
||||
// string. What is asked instead is the thing these tests need: can this process reach its own user
|
||||
// manager at all. The skip is loud, like the database one, because `make check` names skips and a
|
||||
// silent one reads as coverage.
|
||||
out, err := exec.CommandContext(t.Context(), "systemctl", "--user", "show", "--property=Version").CombinedOutput()
|
||||
if err != nil && strings.Contains(string(out), "Failed to connect to bus") {
|
||||
t.Skip("no systemd user manager: the transient-unit properties cannot be measured on this host")
|
||||
if err != nil {
|
||||
t.Skipf("no reachable systemd user manager (%s): the transient-unit properties cannot be measured on this host",
|
||||
bytes.TrimSpace(out))
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
938
platform/internal/runs/control_test.go
Normal file
938
platform/internal/runs/control_test.go
Normal file
|
|
@ -0,0 +1,938 @@
|
|||
package runs
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"textmachine/platform/internal/ingest"
|
||||
"textmachine/platform/internal/money"
|
||||
"textmachine/platform/internal/pgstore"
|
||||
"textmachine/platform/internal/runner"
|
||||
)
|
||||
|
||||
// The control handles — stop and resume — are the two places where a USER's decision reaches a run
|
||||
// that is not this process's child. What they have to get right is not the systemd call, which is
|
||||
// one line, but the record around it: a stop that is not written down before the signal cannot be
|
||||
// told from a crash afterwards, and a resume that forgets the previous attempt's hold reserves the
|
||||
// run's ceiling twice.
|
||||
|
||||
// stopped drives a fixture to a run the user stopped: admitted, spawned, asked to stop, and then
|
||||
// reconciled from the marker the unit left. It is the state resume starts from.
|
||||
//
|
||||
// spent is what the engine reports AFTER the spawn, which is the only order that models the real
|
||||
// one: the spawn reads the book's meter to take this attempt's baseline, so a figure set before it
|
||||
// would be read as money some earlier run had spent and this attempt would settle at nothing.
|
||||
func (f *fixture) stopped(t *testing.T, chapters int, spent money.MicroUSD, marker runner.Marker) string {
|
||||
t.Helper()
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: chapters})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.svc.Spawn(f.ctx, run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
live := f.live(t)
|
||||
marker.Unit = live.UnitName
|
||||
if err := runner.WriteMarker(f.svc.markerPath(live.RunID, live.AttemptNo), marker); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.engine.set(statusSpending(spent), nil)
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return run.ID
|
||||
}
|
||||
|
||||
// secondBook registers another book of the same account, with a directory of its own.
|
||||
func (f *fixture) secondBook(t *testing.T) string {
|
||||
t.Helper()
|
||||
id, err := f.store.AddBook(f.ctx, pgstore.NewBook{OwnerID: "u1", Title: "second", SourceLang: "zh",
|
||||
TargetLang: "ru", ChapterCount: 100, Workdir: t.TempDir(), Now: f.now})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
// lastAttempt is the snapshot a slow sweep pass would still be holding: the run as it was, with the
|
||||
// attempt that was live at the time.
|
||||
func (f *fixture) lastAttempt(t *testing.T, runID string) pgstore.LiveRun {
|
||||
t.Helper()
|
||||
l, err := f.store.ReadRunForResume(f.ctx, "u1", runID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return l
|
||||
}
|
||||
|
||||
func (f *fixture) run(t *testing.T, runID string) pgstore.Run {
|
||||
t.Helper()
|
||||
r, err := f.store.ReadRun(f.ctx, "u1", runID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
// The heart of PD-152. The engine catches SIGTERM and exits 1, so the marker of a stop and the
|
||||
// marker of a crash are the same bytes; what tells them apart is that the platform wrote down its
|
||||
// own intent first.
|
||||
func TestARunTheUserStoppedIsNotReportedAsFailed(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
runID := f.stopped(t, 100, 0, runner.Marker{Result: "exit-code", Code: "exited", Status: "1",
|
||||
At: f.now.Add(time.Second)})
|
||||
if got := f.run(t, runID); got.Status != "stopped" {
|
||||
t.Fatalf("a stop the user asked for came back as %q, want stopped", got.Status)
|
||||
}
|
||||
// And the same marker WITHOUT the intent is still a failure: the discriminator has to be the
|
||||
// record, not a new reading of the exit code.
|
||||
g := newFixture(t, "10", 500)
|
||||
run, err := g.svc.Start(g.ctx, StartRequest{UserID: "u1", BookID: g.bookID(t), CeilingChapters: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := g.svc.Spawn(g.ctx, run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
live := g.live(t)
|
||||
if err := runner.WriteMarker(g.svc.markerPath(live.RunID, live.AttemptNo),
|
||||
runner.Marker{Unit: live.UnitName, Result: "exit-code", Code: "exited", Status: "1"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := g.svc.Sweep(g.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := g.run(t, run.ID); got.Status != "failed" {
|
||||
t.Fatalf("a run nobody stopped came back as %q, want failed", got.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// The race the design owes an answer to: a run that finished by itself a moment before someone
|
||||
// pressed stop was not stopped by them, and calling it `stopped` would hide a completed translation
|
||||
// behind a cancelled one.
|
||||
func TestARunThatEndedBeforeTheStopKeepsItsOwnOutcome(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
// The marker is written a minute BEFORE the stop request the fixture makes.
|
||||
runID := f.stopped(t, 100, 0, runner.Marker{Result: "exit-code", Code: "exited", Status: "1",
|
||||
At: f.now.Add(-time.Minute)})
|
||||
if got := f.run(t, runID); got.Status != "failed" {
|
||||
t.Fatalf("a run that had already ended came back as %q, want failed", got.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// The engine's own clean answers outrank a stop that arrived while it was already finishing: a
|
||||
// translation that COMPLETED must not be shown as cancelled.
|
||||
func TestACleanExitOutranksAStopThatArrivedTooLate(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
runID := f.stopped(t, 100, 0, runner.Marker{Result: "exit-code", Code: "exited", Status: "0",
|
||||
At: f.now.Add(time.Second)})
|
||||
if got := f.run(t, runID); got.Status != "ready" {
|
||||
t.Fatalf("a run that finished cleanly came back as %q, want ready", got.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// The intent is committed BEFORE systemd is asked, and that order is the whole mechanism: a platform
|
||||
// that died between the two must still know, on its next sweep, that the run was stopped on purpose.
|
||||
func TestTheStopIsRecordedBeforeSystemdIsAsked(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 10})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.svc.Spawn(f.ctx, run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var recordedFirst bool
|
||||
f.runner.onStop = func(string) {
|
||||
var at *time.Time
|
||||
if err := f.store.Pool().QueryRow(f.ctx,
|
||||
`select stop_requested_at from runs where id = $1`, run.ID).Scan(&at); err != nil {
|
||||
t.Error(err)
|
||||
return
|
||||
}
|
||||
recordedFirst = at != nil
|
||||
}
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !recordedFirst {
|
||||
t.Fatal("systemd was asked to stop the unit before the intent was committed")
|
||||
}
|
||||
}
|
||||
|
||||
// A stop that systemd did not take must not be lost. The intent is durable, so the next sweep asks
|
||||
// again — which is the only thing that closes "the platform died between the write and the call".
|
||||
func TestAStopTheUnitDidNotTakeIsAskedAgainByTheSweep(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 10})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.svc.Spawn(f.ctx, run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.runner.alive = true
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := len(f.runner.stops()); n != 2 {
|
||||
t.Fatalf("the unit was asked to stop %d times, want 2 (the handle and the sweep)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// The dangerous half: a stopped unit that left no marker looks exactly like a reboot, and the reboot
|
||||
// path RESTARTS the run — which would spend the account's money on work its owner had just
|
||||
// cancelled.
|
||||
func TestAStoppedRunIsNotRestartedWhenItsMarkerIsMissing(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.svc.Spawn(f.ctx, run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The unit is gone, no marker was written, and the spawn grace is long past.
|
||||
f.runner.alive = false
|
||||
f.svc.Now = func() time.Time { return f.now.Add(2 * time.Hour) }
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := f.run(t, run.ID); got.Status != "stopped" {
|
||||
t.Fatalf("run is %q, want stopped", got.Status)
|
||||
}
|
||||
if n := len(f.runner.starts()); n != 1 {
|
||||
t.Fatalf("%d units were started; a stopped run must not be restarted", n)
|
||||
}
|
||||
if acct := f.account(t); acct.Reserved != 0 {
|
||||
t.Fatalf("the hold of a stopped run is still open: %s", acct.Reserved.USD())
|
||||
}
|
||||
}
|
||||
|
||||
// A run stopped before its unit ever existed: there is nothing to signal, and the money must come
|
||||
// back whole rather than wait for a unit that will never be created.
|
||||
func TestStoppingARunThatNeverSpawnedGivesTheHoldBackWhole(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := len(f.runner.stops()); n != 0 {
|
||||
t.Fatalf("systemd was asked about a unit that does not exist (%d times)", n)
|
||||
}
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := f.run(t, run.ID); got.Status != "stopped" {
|
||||
t.Fatalf("run is %q, want stopped", got.Status)
|
||||
}
|
||||
acct := f.account(t)
|
||||
if acct.Reserved != 0 || acct.Balance != money.MicroUSD(10_000_000) {
|
||||
t.Fatalf("the hold of a run that never started did not come back whole: balance %s reserved %s",
|
||||
acct.Balance.USD(), acct.Reserved.USD())
|
||||
}
|
||||
if n := len(f.runner.starts()); n != 0 {
|
||||
t.Fatalf("a stopped run was spawned anyway (%d units)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// Ownership, on the handle that ENDS a paid run: another account's run is not "forbidden", it is not
|
||||
// there at all (API1/BOLA, and the same rule the library reads by).
|
||||
func TestARunOfAnotherAccountCannotBeStoppedOrResumed(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 10})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.store.Pool().Exec(f.ctx,
|
||||
`insert into users (id, email) values ('u2','u2@example.org')`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.svc.Stop(f.ctx, "u2", run.ID); !errors.Is(err, pgstore.ErrNoRun) {
|
||||
t.Fatalf("stop by a stranger: %v, want ErrNoRun", err)
|
||||
}
|
||||
if _, err := f.svc.Resume(f.ctx, "u2", run.ID); !errors.Is(err, pgstore.ErrNoRun) {
|
||||
t.Fatalf("resume by a stranger: %v, want ErrNoRun", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Stopping a run that is already over is a conflict and NOT a not-found: the owner can see the run,
|
||||
// and answering 404 would tell them their own run does not exist.
|
||||
func TestStoppingARunThatIsOverIsAConflict(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
runID := f.stopped(t, 10, 0, runner.Marker{Result: "exit-code", Code: "exited", Status: "1",
|
||||
At: f.now.Add(time.Second)})
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", runID); !errors.Is(err, ErrNotStoppable) {
|
||||
t.Fatalf("stopping a finished run: %v, want ErrNotStoppable", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Resume is the reconciler's restart with a user behind it: a NEW attempt, holding what is LEFT of
|
||||
// the run's budget. The money is the assertion — one run may not reserve its ceiling twice.
|
||||
func TestResumeContinuesTheRunWithWhatIsLeftOfItsBudget(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
spent := money.MicroUSD(500_000)
|
||||
runID := f.stopped(t, 100, spent, runner.Marker{Result: "exit-code", Code: "exited", Status: "1",
|
||||
At: f.now.Add(time.Second)})
|
||||
budget := f.svc.Pricing.Ceiling(100)
|
||||
if acct := f.account(t); acct.Reserved != 0 || acct.Balance != money.MicroUSD(10_000_000)-spent {
|
||||
t.Fatalf("before the resume: balance %s reserved %s", acct.Balance.USD(), acct.Reserved.USD())
|
||||
}
|
||||
got, err := f.svc.Resume(f.ctx, "u1", runID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Status != "translating" || got.FinishedAt != nil {
|
||||
t.Fatalf("the resumed run is %+v, want a live translating run", got)
|
||||
}
|
||||
// The new attempt holds the REMAINDER, not the whole ceiling: the first attempt already spent
|
||||
// part of it and settled at what it spent.
|
||||
if acct := f.account(t); acct.Reserved != budget-spent {
|
||||
t.Fatalf("the resumed run holds %s, want %s (the ceiling less what was already spent)",
|
||||
acct.Reserved.USD(), (budget - spent).USD())
|
||||
}
|
||||
// The engine is QUEUED, not started inside the request: reading the book's meter costs seconds of
|
||||
// the engine's CPU and creating a unit is a round trip to systemd, and a call that answers 202
|
||||
// must not hold either. (Measured, not reasoned about: the live probe's resume held its request
|
||||
// until the engine exited.) With no queue wired here, the reconciler is the backstop — which is
|
||||
// the same backstop the admission path relies on.
|
||||
live := f.live(t)
|
||||
if live.AttemptNo != 2 || live.UnitName != "" {
|
||||
t.Fatalf("the resumed run's attempt is %+v, want a second attempt waiting to be spawned", live)
|
||||
}
|
||||
if n := len(f.runner.starts()); n != 1 {
|
||||
t.Fatalf("%d units started; the resume spawned the engine inside the request", n)
|
||||
}
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := len(f.runner.starts()); n != 2 {
|
||||
t.Fatalf("%d units started across the run, want 2 once the backstop ran", n)
|
||||
}
|
||||
// The book follows the run back into translation, and the library's revision moves with it.
|
||||
book, _, err := f.store.GetBook(f.ctx, "u1", f.bookID(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if book.Status != "translating" {
|
||||
t.Fatalf("the book of a resumed run is %q, want translating", book.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// The bank stop is the one resume the contract describes in detail, and its condition — a COMPLETE
|
||||
// set of decisions — cannot be met by any deployment today: nothing materializes the bank at all
|
||||
// (companion §3). Answering 202 would start a run that walks straight into the same stop.
|
||||
func TestResumeRefusesABankStopWhileTheGlossaryIsUnsigned(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
runID := f.stopped(t, 100, 0, runner.Marker{Result: "exit-code", Code: "exited", Status: "3",
|
||||
At: f.now.Add(time.Second)})
|
||||
if got := f.run(t, runID); got.Status != "awaiting_bank" {
|
||||
t.Fatalf("run after the bank stop is %q", got.Status)
|
||||
}
|
||||
if _, err := f.svc.Resume(f.ctx, "u1", runID); !errors.Is(err, ErrNotResumable) {
|
||||
t.Fatalf("resume of a bank stop: %v, want ErrNotResumable", err)
|
||||
}
|
||||
}
|
||||
|
||||
// "Resume on a run paused by a ceiling returns it to the same state" (contract §resumeRun). It is
|
||||
// answered rather than refused, and nothing moves: no second hold, no second unit — and, the half
|
||||
// that is easy to get wrong, the run's own state is not rewritten on the way out. A run the user
|
||||
// stopped comes back stopped; calling it `paused` because a resume failed would replace what
|
||||
// happened to it with why it could not continue.
|
||||
func TestResumeOfARunWithNothingLeftReturnsItUnchanged(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
// The attempt spends its entire ceiling, so the run has no budget left at all.
|
||||
runID := f.stopped(t, 10, f.svc.Pricing.Ceiling(10), runner.Marker{Result: "exit-code",
|
||||
Code: "exited", Status: "1", At: f.now.Add(time.Second)})
|
||||
before := f.account(t)
|
||||
got, err := f.svc.Resume(f.ctx, "u1", runID)
|
||||
if err != nil {
|
||||
t.Fatalf("resume of an exhausted run: %v, want the run back unchanged", err)
|
||||
}
|
||||
if got.Status != "stopped" || got.PausedReason != "" {
|
||||
t.Fatalf("the run came back as %q/%q, want the stopped run it was", got.Status, got.PausedReason)
|
||||
}
|
||||
if acct := f.account(t); acct.Reserved != 0 || acct.Balance != before.Balance {
|
||||
t.Fatalf("a resume that changed nothing moved money: %+v -> %+v", before, acct)
|
||||
}
|
||||
if n := len(f.runner.starts()); n != 1 {
|
||||
t.Fatalf("%d units started, want 1: nothing was resumed", n)
|
||||
}
|
||||
}
|
||||
|
||||
// The reconciler's half of the same decision, which the shared body must not have taken away: a run
|
||||
// that was LIVE and has nothing left is `paused` with the reason the contract has a word for, and a
|
||||
// resume of that comes back paused too.
|
||||
func TestAnInterruptedRunWithNothingLeftIsPausedAndStaysPausedThroughAResume(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 10})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.svc.Spawn(f.ctx, run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The engine spent the whole ceiling and then its unit vanished without a marker, which is what a
|
||||
// reboot leaves behind.
|
||||
f.engine.set(statusSpending(f.svc.Pricing.Ceiling(10)), nil)
|
||||
f.runner.alive = false
|
||||
f.svc.Now = func() time.Time { return f.now.Add(2 * time.Hour) }
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := f.run(t, run.ID)
|
||||
if got.Status != "paused" || got.PausedReason != pgstore.PausedCreditExhausted {
|
||||
t.Fatalf("an interrupted run with no budget is %q/%q, want paused/credit_exhausted",
|
||||
got.Status, got.PausedReason)
|
||||
}
|
||||
resumed, err := f.svc.Resume(f.ctx, "u1", run.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("resume of a paused run: %v, want it back unchanged", err)
|
||||
}
|
||||
if resumed.Status != "paused" || resumed.PausedReason != pgstore.PausedCreditExhausted {
|
||||
t.Fatalf("the paused run came back as %q/%q", resumed.Status, resumed.PausedReason)
|
||||
}
|
||||
if n := len(f.runner.starts()); n != 1 {
|
||||
t.Fatalf("%d units started, want 1", n)
|
||||
}
|
||||
}
|
||||
|
||||
// A settlement is allowed to defer, and a resume that ignored that would open a SECOND hold on a run
|
||||
// whose first is still reserved.
|
||||
func TestResumeRefusesWhileThePreviousAttemptIsStillHoldingMoney(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.svc.Spawn(f.ctx, run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
live := f.live(t)
|
||||
if err := runner.WriteMarker(f.svc.markerPath(live.RunID, live.AttemptNo),
|
||||
runner.Marker{Unit: live.UnitName, Result: "exit-code", Code: "exited", Status: "1",
|
||||
At: f.now.Add(time.Second)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The engine cannot be asked what it spent, so the run finishes and its hold stays open.
|
||||
f.engine.set(statusSpending(0), errors.New("status unavailable"))
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if acct := f.account(t); acct.Reserved == 0 {
|
||||
t.Fatal("the hold was released even though the engine could not be asked")
|
||||
}
|
||||
if _, err := f.svc.Resume(f.ctx, "u1", run.ID); !errors.Is(err, ErrNotResumable) {
|
||||
t.Fatalf("resume while unsettled: %v, want ErrNotResumable", err)
|
||||
}
|
||||
if n := len(f.runner.starts()); n != 1 {
|
||||
t.Fatalf("%d units started: an unsettled run must not be resumed", n)
|
||||
}
|
||||
}
|
||||
|
||||
// A book that is still arriving, or one the engine refused, has no chapter tree to translate — and
|
||||
// while it is `uploading` it has half a file on disk. The refusal happens BEFORE the money moves.
|
||||
func TestARunIsRefusedOnABookThatIsNotThroughIntake(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
for _, status := range []string{"uploading", "parsing", "rejected"} {
|
||||
if _, err := f.store.Pool().Exec(f.ctx,
|
||||
`update books set status = $1 where id = $2`, status, f.bookID(t)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 10})
|
||||
if !errors.Is(err, ErrBookNotReady) {
|
||||
t.Fatalf("starting a run on a %s book: %v, want ErrBookNotReady", status, err)
|
||||
}
|
||||
if acct := f.account(t); acct.Reserved != 0 {
|
||||
t.Fatalf("a refused run held money: %s", acct.Reserved.USD())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// statusSpending is a report of an engine that has committed exactly this much on the book.
|
||||
func statusSpending(v money.MicroUSD) ingest.StatusReport {
|
||||
return ingest.StatusReport{Spend: usd(v), Reserved: usd(0)}
|
||||
}
|
||||
|
||||
// PD-169. A pass of the sweep has one budget for every run in it, and the list is ordered the same
|
||||
// way every time: a run whose engine hangs used to eat the pass, and the tail of the list — other
|
||||
// people's books — was never reached at all. The per-run budget is what bounds that, and this is the
|
||||
// assertion that it does.
|
||||
func TestOneSlowRunDoesNotEatThePassOfTheWholeSweep(t *testing.T) {
|
||||
f := newFixture(t, "20", 500)
|
||||
f.svc.Cfg.RunBudget = 100 * time.Millisecond
|
||||
// Two books, two live runs, and the sweep visits them in the order they started.
|
||||
first, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 10})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
second := f.secondBook(t)
|
||||
if _, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: second, CeilingChapters: 10}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The engine hangs for the FIRST run only — until the context that carries it is cancelled.
|
||||
f.engine.block(first.ID, f.workdir)
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The second run was spawned in the same pass, which is the whole property: the slow one cost its
|
||||
// own budget and nothing else's.
|
||||
started := f.runner.starts()
|
||||
if len(started) != 1 {
|
||||
t.Fatalf("%d units started; the second run was starved by the first", len(started))
|
||||
}
|
||||
}
|
||||
|
||||
// Two clicks on one button. Both calls pass the state check together and race for attempt N+1 on the
|
||||
// unique index; the loser must answer with the run the winner re-opened, not with a not-found — and
|
||||
// exactly ONE hold may be taken.
|
||||
func TestTwoResumesOfOneRunTakeOneHoldAndBothAnswer(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
runID := f.stopped(t, 100, money.MicroUSD(500_000), runner.Marker{Result: "exit-code",
|
||||
Code: "exited", Status: "1", At: f.now.Add(time.Second)})
|
||||
var wg sync.WaitGroup
|
||||
runs := make([]pgstore.Run, 2)
|
||||
errs := make([]error, 2)
|
||||
for i := range runs {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
runs[i], errs[i] = f.svc.Resume(f.ctx, "u1", runID)
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
for i, err := range errs {
|
||||
if err != nil {
|
||||
t.Fatalf("resume %d: %v, want the run back", i, err)
|
||||
}
|
||||
if runs[i].Status != "translating" {
|
||||
t.Errorf("resume %d answered %q, want translating", i, runs[i].Status)
|
||||
}
|
||||
}
|
||||
if acct := f.account(t); acct.Reserved != f.svc.Pricing.Ceiling(100)-money.MicroUSD(500_000) {
|
||||
t.Fatalf("two resumes reserved %s, want exactly one remainder", acct.Reserved.USD())
|
||||
}
|
||||
if acct := f.account(t); acct.Balance != acct.LedgerSum {
|
||||
t.Fatalf("the cached balance and the ledger disagree: %s vs %s", acct.Balance.USD(), acct.LedgerSum.USD())
|
||||
}
|
||||
var attempts int
|
||||
if err := f.store.Pool().QueryRow(f.ctx,
|
||||
`select count(*) from run_attempts where run_id = $1`, runID).Scan(&attempts); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if attempts != 2 {
|
||||
t.Fatalf("%d attempts after two resumes, want 2", attempts)
|
||||
}
|
||||
}
|
||||
|
||||
// A stopped run can be resumed — unless the account started ANOTHER run of the same book in the
|
||||
// meantime. Re-opening then puts two live runs under an index that forbids exactly that, and the
|
||||
// answer has to be the conflict a second admission gets, not an internal error. The money is the
|
||||
// other half: the hold of the refused resume is taken in the same transaction and dies with it.
|
||||
func TestResumeIsRefusedWhenTheBookHasAnotherLiveRun(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
stoppedRun := f.stopped(t, 10, money.MicroUSD(10_000), runner.Marker{Result: "exit-code",
|
||||
Code: "exited", Status: "1", At: f.now.Add(time.Second)})
|
||||
if _, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 10}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
before := f.account(t)
|
||||
if _, err := f.svc.Resume(f.ctx, "u1", stoppedRun); !errors.Is(err, pgstore.ErrRunInFlight) {
|
||||
t.Fatalf("resume while another run is live: %v, want ErrRunInFlight", err)
|
||||
}
|
||||
acct := f.account(t)
|
||||
if acct.Reserved != before.Reserved || acct.Balance != before.Balance {
|
||||
t.Fatalf("a refused resume moved money: %+v -> %+v", before, acct)
|
||||
}
|
||||
if acct.Balance != acct.LedgerSum {
|
||||
t.Fatalf("the cached balance and the ledger disagree: %s vs %s", acct.Balance.USD(), acct.LedgerSum.USD())
|
||||
}
|
||||
}
|
||||
|
||||
// One counter per book, and every run-carrying answer uses it (contract §Revision). The stop handle
|
||||
// used to answer from the run's own column, which sits BELOW the card's number the moment anything
|
||||
// materializes — and a client that has applied the card's revision is required by the contract to
|
||||
// drop the lower one, i.e. to drop the answer to the button it just pressed.
|
||||
func TestEveryRunCarryingAnswerUsesTheBooksRevision(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 10})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Something materializes: the book's counter moves and the run's own column does not.
|
||||
if _, err := f.store.Pool().Exec(f.ctx,
|
||||
`update books set revision = revision + 5 where id = $1`, f.bookID(t)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
book, card, err := f.store.GetBook(f.ctx, "u1", f.bookID(t))
|
||||
if err != nil || card == nil {
|
||||
t.Fatalf("card: %v", err)
|
||||
}
|
||||
if card.Revision != book.Revision {
|
||||
t.Fatalf("the card's run carries revision %d and its book %d: one counter per book", card.Revision, book.Revision)
|
||||
}
|
||||
stopped, err := f.svc.Stop(f.ctx, "u1", run.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if stopped.Revision != book.Revision {
|
||||
t.Fatalf("stop answered revision %d, the card answers %d — the client drops the lower one",
|
||||
stopped.Revision, book.Revision)
|
||||
}
|
||||
read, err := f.store.ReadRun(f.ctx, "u1", run.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if read.Revision != book.Revision {
|
||||
t.Fatalf("ReadRun answered revision %d, the card answers %d", read.Revision, book.Revision)
|
||||
}
|
||||
}
|
||||
|
||||
// A run that is not the caller's, or one that does not exist, is a 404 — even on a deployment that
|
||||
// could not resume anything anyway. The other order answers "this service cannot start runs" to a
|
||||
// question about someone else's run, which is both a worse answer and a small oracle.
|
||||
func TestOwnershipIsJudgedBeforeTheDeploymentsHealth(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 10})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.svc.Cfg.MarkerArgv = nil // a deployment that cannot record how a run ends
|
||||
if _, err := f.svc.Resume(f.ctx, "u1", "run_does_not_exist"); !errors.Is(err, pgstore.ErrNoRun) {
|
||||
t.Fatalf("resume of a missing run on an unwired deployment: %v, want ErrNoRun", err)
|
||||
}
|
||||
if _, err := f.svc.Resume(f.ctx, "u1", run.ID); !errors.Is(err, ErrRunnerIncomplete) {
|
||||
t.Fatalf("resume of an OWN run on an unwired deployment: %v, want the deployment's refusal", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The reconciler decides from a SNAPSHOT and writes seconds later; in between, the user presses
|
||||
// stop. Three windows, one cure — the row is re-read under the lock that does the write. Each of
|
||||
// these was reproduced by an independent reviewer before it was closed.
|
||||
|
||||
// Window 1 (money): a stop pressed while the reconciler is settling an interrupted attempt. The
|
||||
// restart used to clear the fresh intent, open a second attempt and take a new hold — a 202 for a
|
||||
// stop that never happened, and the user paying for the work they had just cancelled.
|
||||
func TestAStopPressedWhileTheReconcilerRestartsIsNotLost(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.svc.Spawn(f.ctx, run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The unit is gone with no marker — the shape of a reboot — and the grace is past, so the
|
||||
// reconciler will restart. The stop lands while it is settling.
|
||||
f.runner.alive = false
|
||||
f.svc.Now = func() time.Time { return f.now.Add(2 * time.Hour) }
|
||||
f.engine.onStatus = func() {
|
||||
f.engine.onStatus = nil
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := f.run(t, run.ID)
|
||||
if got.Status != "stopped" {
|
||||
t.Fatalf("the run is %q, want stopped: the stop outranks the restart", got.Status)
|
||||
}
|
||||
if n := len(f.runner.starts()); n != 1 {
|
||||
t.Fatalf("%d units started; the restart went ahead over a stop", n)
|
||||
}
|
||||
if acct := f.account(t); acct.Reserved != 0 {
|
||||
t.Fatalf("a run stopped mid-restart still holds %s", acct.Reserved.USD())
|
||||
}
|
||||
if acct := f.account(t); acct.Balance != acct.LedgerSum {
|
||||
t.Fatalf("balance %s and ledger %s disagree", acct.Balance.USD(), acct.LedgerSum.USD())
|
||||
}
|
||||
}
|
||||
|
||||
// Window 2 (money): the sweep is about to close a run that was stopped before it spawned, and the
|
||||
// queue worker creates the unit in between. Closing it anyway strands a live engine that no list
|
||||
// looks at — not the live runs (finished), not the unsettled ones (its hold was released).
|
||||
func TestARunSpawnedWhileTheSweepWasClosingItIsNotAbandoned(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The snapshot a sweep would decide from: taken while the attempt still had no unit.
|
||||
live := f.live(t)
|
||||
// The worker claims and creates the unit — BEFORE the stop, because the claim itself refuses once
|
||||
// an intent is standing. This is the window that remains: the sweep is holding a snapshot older
|
||||
// than the unit.
|
||||
claimed, err := f.store.RecordSpawn(f.ctx, pgstore.SpawnRecord{AttemptID: live.AttemptID,
|
||||
Unit: "tm-run-raced", Binary: "/opt/engine/tmctl", Ceiling: live.Ceiling,
|
||||
CeilingArg: live.Ceiling, Baseline: 1})
|
||||
if err != nil || !claimed {
|
||||
t.Fatalf("the worker could not claim the attempt: %v %v", claimed, err)
|
||||
}
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.svc.finishStopped(f.ctx, live); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := f.run(t, run.ID)
|
||||
if got.Status == "stopped" || got.FinishedAt != nil {
|
||||
t.Fatalf("a run whose engine had just started was closed anyway: %+v", got)
|
||||
}
|
||||
if acct := f.account(t); acct.Reserved == 0 {
|
||||
t.Fatal("the hold of a run whose engine is running was released")
|
||||
}
|
||||
// And the next pass does the right thing with it: a live unit plus a standing intent is a stop.
|
||||
f.runner.alive = true
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := len(f.runner.stops()); n == 0 {
|
||||
t.Fatal("the raced run was never asked to stop")
|
||||
}
|
||||
}
|
||||
|
||||
// Window 3: a stop pressed on a run whose queue job has not been picked up yet. The worker must not
|
||||
// start an engine for it — the stop arrived before the run did anything at all.
|
||||
func TestTheQueueWorkerDoesNotStartARunThatWasAlreadyStopped(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.svc.Spawn(f.ctx, run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := len(f.runner.starts()); n != 0 {
|
||||
t.Fatalf("%d units started for a run that was stopped before its worker ran", n)
|
||||
}
|
||||
// The claim itself refuses too, which is the half that covers an intent written after the read.
|
||||
live := f.live(t)
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
claimed, err := f.store.RecordSpawn(f.ctx, pgstore.SpawnRecord{AttemptID: live.AttemptID,
|
||||
Unit: "tm-run-late", Binary: "/opt/engine/tmctl", Ceiling: live.Ceiling, CeilingArg: live.Ceiling})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if claimed {
|
||||
t.Fatal("the spawn claim was granted for an attempt of a run that is over and settled")
|
||||
}
|
||||
if acct := f.account(t); acct.Reserved != 0 || acct.Balance != money.MicroUSD(10_000_000) {
|
||||
t.Fatalf("the hold did not come back whole: balance %s reserved %s",
|
||||
acct.Balance.USD(), acct.Reserved.USD())
|
||||
}
|
||||
}
|
||||
|
||||
// The window a RESUME opened and nothing else could: a run comes back to life, and the exit marker of
|
||||
// the attempt that ended is still on disk — nothing deletes markers of attempts that are over. A pass
|
||||
// holding the old snapshot would close the resumed run from it, and attempt 2's hold would then be
|
||||
// in NO worklist at all: live runs are selected by `finished_at is null`, unsettled ones by an
|
||||
// attempt with `ended_at`, and a re-finished resumed run matches neither.
|
||||
func TestAStaleSweepDoesNotReFinishAResumedRunFromTheOldAttemptsMarker(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
spent := money.MicroUSD(100_000)
|
||||
runID := f.stopped(t, 100, spent, runner.Marker{Result: "exit-code", Code: "exited", Status: "1",
|
||||
At: f.now.Add(time.Second)})
|
||||
stale := f.lastAttempt(t, runID) // the snapshot a slow pass is still holding: attempt 1
|
||||
if _, err := f.svc.Resume(f.ctx, "u1", runID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
held := f.account(t).Reserved
|
||||
if held == 0 {
|
||||
t.Fatal("the resumed run holds nothing")
|
||||
}
|
||||
// The slow pass arrives with attempt 1 and its marker, which is still on disk.
|
||||
if err := f.svc.reconcile(f.ctx, stale); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := f.run(t, runID)
|
||||
if got.Status != "translating" || got.FinishedAt != nil {
|
||||
t.Fatalf("a stale pass closed the resumed run: %+v", got)
|
||||
}
|
||||
if acct := f.account(t); acct.Reserved != held {
|
||||
t.Fatalf("the resumed run's hold moved: %s → %s", held.USD(), acct.Reserved.USD())
|
||||
}
|
||||
// And the run is still findable, which is the half that makes the money recoverable at all.
|
||||
live, err := f.store.ListLiveRuns(f.ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(live) != 1 || live[0].AttemptNo != 2 {
|
||||
t.Fatalf("live runs after the stale pass: %+v", live)
|
||||
}
|
||||
}
|
||||
|
||||
// A claim that was GIVEN BACK is not proof that no engine exists: `systemd-run` killed after it had
|
||||
// already asked leaves one running, which is why the spend baseline is kept on such an attempt. The
|
||||
// stop path must ask systemd about the unit's deterministic name rather than close over it.
|
||||
func TestAStopDoesNotCloseARunWhoseGivenBackClaimLeftAnEngineRunning(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The unit could "not be created" — but it was: the claim is given back and the baseline stays.
|
||||
f.runner.startErr = errors.New("systemd-run was killed after it had asked")
|
||||
if err := f.svc.Spawn(f.ctx, run.ID); err == nil {
|
||||
t.Fatal("a failed spawn reported success")
|
||||
}
|
||||
live := f.live(t)
|
||||
if live.UnitName != "" || live.SpendBaseline == nil {
|
||||
t.Fatalf("after a given-back claim: unit %q, baseline %v", live.UnitName, live.SpendBaseline)
|
||||
}
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// systemd says the orphan is alive.
|
||||
f.runner.alive = true
|
||||
f.runner.startErr = nil
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := f.run(t, run.ID)
|
||||
if got.FinishedAt != nil {
|
||||
t.Fatalf("the run was closed while its engine was still running: %+v", got)
|
||||
}
|
||||
if n := len(f.runner.stops()); n == 0 {
|
||||
t.Fatal("the orphaned unit was never asked to stop")
|
||||
}
|
||||
if acct := f.account(t); acct.Reserved == 0 {
|
||||
t.Fatal("the hold of a run whose engine is alive was released")
|
||||
}
|
||||
}
|
||||
|
||||
// The window the queue worker opens: it reads the run, then spends seconds in `bookMeter` asking the
|
||||
// engine what the book has cost. A stop committed inside that window has no unit to signal — so the
|
||||
// only thing that can refuse the engine is the claim itself.
|
||||
func TestAStopCommittedWhileTheWorkerReadsTheMeterStopsTheSpawn(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The stop lands while the worker is inside the meter call, i.e. after it read its snapshot.
|
||||
f.engine.onStatus = func() {
|
||||
f.engine.onStatus = nil
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
if err := f.svc.Spawn(f.ctx, run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := len(f.runner.starts()); n != 0 {
|
||||
t.Fatalf("%d units started for a run stopped while the worker was reading the meter", n)
|
||||
}
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := f.run(t, run.ID); got.Status != "stopped" {
|
||||
t.Fatalf("the run is %q, want stopped", got.Status)
|
||||
}
|
||||
if acct := f.account(t); acct.Reserved != 0 || acct.Balance != money.MicroUSD(10_000_000) {
|
||||
t.Fatalf("the hold did not come back whole: balance %s reserved %s",
|
||||
acct.Balance.USD(), acct.Reserved.USD())
|
||||
}
|
||||
}
|
||||
|
||||
// FP5-2 (acceptance, money). The guard FinishRun received (PD-181) was missing from the path that
|
||||
// closes a stop which never spawned: a stale pass could close a RESUMED run through it, and attempt
|
||||
// 2's hold would then be in no worklist at all — with the added twist that every later resume answers
|
||||
// 409 forever, because the run it would continue is finished.
|
||||
func TestAStaleUnspawnedStopDoesNotCloseAResumedRun(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stale := f.live(t) // attempt 1, no unit — the snapshot a slow pass holds
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := f.svc.Resume(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
held := f.account(t).Reserved
|
||||
if held == 0 {
|
||||
t.Fatal("the resumed run holds nothing")
|
||||
}
|
||||
// The slow pass arrives with attempt 1 and the intent that was on it.
|
||||
if err := f.svc.finishStopped(f.ctx, stale); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := f.run(t, run.ID)
|
||||
if got.Status != "translating" || got.FinishedAt != nil {
|
||||
t.Fatalf("a stale unspawned-stop closed the resumed run: %+v", got)
|
||||
}
|
||||
if acct := f.account(t); acct.Reserved != held {
|
||||
t.Fatalf("the resumed run's hold moved: %s → %s", held.USD(), acct.Reserved.USD())
|
||||
}
|
||||
live, err := f.store.ListLiveRuns(f.ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(live) != 1 || live[0].AttemptNo != 2 {
|
||||
t.Fatalf("live runs after the stale pass: %+v", live)
|
||||
}
|
||||
}
|
||||
|
||||
// FP5-8(а) (acceptance). A stop that lands while the reconciler is settling an exhausted run must not
|
||||
// come back as `paused/credit_exhausted`: that says the money ran out, when what happened is that the
|
||||
// owner stopped it.
|
||||
func TestAStopDuringSettlementOutranksThePause(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 10})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.svc.Spawn(f.ctx, run.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The attempt spends the whole ceiling, so the restart path will find nothing left; the stop lands
|
||||
// while that settlement is in flight.
|
||||
f.engine.set(statusSpending(f.svc.Pricing.Ceiling(10)), nil)
|
||||
f.engine.onStatus = func() {
|
||||
f.engine.onStatus = nil
|
||||
if _, err := f.svc.Stop(f.ctx, "u1", run.ID); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
f.runner.alive = false
|
||||
f.svc.Now = func() time.Time { return f.now.Add(2 * time.Hour) }
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := f.run(t, run.ID)
|
||||
if got.Status != "stopped" || got.PausedReason != "" {
|
||||
t.Fatalf("the run came back as %q/%q, want stopped", got.Status, got.PausedReason)
|
||||
}
|
||||
if acct := f.account(t); acct.Balance != acct.LedgerSum {
|
||||
t.Fatalf("balance %s and ledger %s disagree", acct.Balance.USD(), acct.LedgerSum.USD())
|
||||
}
|
||||
}
|
||||
|
|
@ -1,91 +0,0 @@
|
|||
package runs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"github.com/riverqueue/river"
|
||||
"github.com/riverqueue/river/riverdriver/riverpgxv5"
|
||||
|
||||
"textmachine/platform/internal/pgstore"
|
||||
)
|
||||
|
||||
// SpawnArgs is one queued permission to start a run.
|
||||
//
|
||||
// The job carries an id and nothing else. Everything about the run — the book, the ceiling, the
|
||||
// binary it is pinned to — is in Postgres, and a job that carried its own copy would be a second
|
||||
// answer that goes stale the moment the run is restarted with a smaller budget.
|
||||
type SpawnArgs struct {
|
||||
RunID string `json:"run_id"`
|
||||
}
|
||||
|
||||
// Kind is River's name for this job type.
|
||||
func (SpawnArgs) Kind() string { return "tm_spawn_run" }
|
||||
|
||||
// InsertOpts pins the queue-level policy.
|
||||
//
|
||||
// MaxAttempts is 1 on purpose, and it is the opposite of the usual queue reflex. A retry here does
|
||||
// not repeat lost work: the run row already exists and holds the account's money, and the thing that
|
||||
// would be repeated is spawning an engine. What recovers a run whose spawn failed is the reconciler,
|
||||
// which reads the world instead of assuming the job's view of it — and which is the ONLY component
|
||||
// that can tell "the unit never started" from "the unit is running and this platform was restarted".
|
||||
func (SpawnArgs) InsertOpts() river.InsertOpts {
|
||||
return river.InsertOpts{MaxAttempts: 1, Queue: river.QueueDefault}
|
||||
}
|
||||
|
||||
// spawnWorker runs SpawnArgs jobs.
|
||||
type spawnWorker struct {
|
||||
river.WorkerDefaults[SpawnArgs]
|
||||
svc *Service
|
||||
}
|
||||
|
||||
func (w *spawnWorker) Work(ctx context.Context, job *river.Job[SpawnArgs]) error {
|
||||
return w.svc.Spawn(ctx, job.Args.RunID)
|
||||
}
|
||||
|
||||
// Queue is the River client, wired to this service.
|
||||
type Queue struct {
|
||||
client *river.Client[pgstore.Tx]
|
||||
}
|
||||
|
||||
// NewQueue builds the queue and its worker pool.
|
||||
//
|
||||
// Concurrency is deliberately small: a worker's whole job is to create a transient unit, and the
|
||||
// runs themselves are bounded by their own cgroups and by the one-live-run-per-book index, not by
|
||||
// how many workers exist.
|
||||
func NewQueue(pool *pgxpool.Pool, svc *Service, log *slog.Logger, workers int) (*Queue, error) {
|
||||
if workers <= 0 {
|
||||
workers = 4
|
||||
}
|
||||
w := river.NewWorkers()
|
||||
if err := river.AddWorkerSafely(w, &spawnWorker{svc: svc}); err != nil {
|
||||
return nil, fmt.Errorf("runs: register worker: %w", err)
|
||||
}
|
||||
c, err := river.NewClient(riverpgxv5.New(pool), &river.Config{
|
||||
Logger: log,
|
||||
Workers: w,
|
||||
Queues: map[string]river.QueueConfig{river.QueueDefault: {MaxWorkers: workers}},
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("runs: river client: %w", err)
|
||||
}
|
||||
return &Queue{client: c}, nil
|
||||
}
|
||||
|
||||
// EnqueueRun inserts the job in the CALLER's transaction, so the run row, its hold and its queue
|
||||
// entry commit together or not at all.
|
||||
func (q *Queue) EnqueueRun(ctx context.Context, tx pgstore.Tx, runID string) error {
|
||||
if _, err := q.client.InsertTx(ctx, tx, SpawnArgs{RunID: runID}, nil); err != nil {
|
||||
return fmt.Errorf("runs: enqueue %s: %w", runID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Start begins working jobs.
|
||||
func (q *Queue) Start(ctx context.Context) error { return q.client.Start(ctx) }
|
||||
|
||||
// Stop drains the workers. The runs themselves are untouched: they are transient units, not
|
||||
// children, and outliving this process is what they are for.
|
||||
func (q *Queue) Stop(ctx context.Context) error { return q.client.Stop(ctx) }
|
||||
|
|
@ -38,7 +38,7 @@ func (s *Service) Sweep(ctx context.Context) error {
|
|||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.reconcile(ctx, l); err != nil {
|
||||
if err := s.withBudget(ctx, func(ctx context.Context) error { return s.reconcile(ctx, l) }); err != nil {
|
||||
s.log().ErrorContext(ctx, "run could not be reconciled", "run", l.RunID, "err", err)
|
||||
}
|
||||
}
|
||||
|
|
@ -50,13 +50,63 @@ func (s *Service) Sweep(ctx context.Context) error {
|
|||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.settle(ctx, u); err != nil {
|
||||
if err := s.withBudget(ctx, func(ctx context.Context) error { return s.settle(ctx, u) }); err != nil {
|
||||
s.log().ErrorContext(ctx, "run could not be settled", "run", u.RunID, "err", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// runBudget is what ONE run may cost a pass of the sweep.
|
||||
//
|
||||
// It exists because the pass has a budget of its own and the work is not uniform: a spawn or a
|
||||
// settlement is a `tmctl status` call, seconds of the engine's CPU each, and a handful of runs whose
|
||||
// engine hangs used to spend the whole pass — after which the list, which is ordered the same way
|
||||
// every time, was never reached past them. That is starvation with no upper bound, and it was
|
||||
// invisible (register row PD-169; the counter that makes it visible is the sweep's own metric).
|
||||
//
|
||||
// Generous rather than tight: the call it bounds legitimately takes seconds on a large book, and the
|
||||
// point is to bound the pathological case, not to race the ordinary one.
|
||||
const defaultRunBudget = 60 * time.Second
|
||||
|
||||
func (s *Service) runBudget() time.Duration {
|
||||
if s.Cfg.RunBudget > 0 {
|
||||
return s.Cfg.RunBudget
|
||||
}
|
||||
return defaultRunBudget
|
||||
}
|
||||
|
||||
// withBudget runs one item of a sweep under its own deadline. A cancelled item is not a failed one:
|
||||
// the next pass reads the same world and tries again, and the money paths inside are transactional,
|
||||
// so an item cut off mid-write leaves nothing half-done.
|
||||
func (s *Service) withBudget(ctx context.Context, fn func(context.Context) error) error {
|
||||
c, cancel := context.WithTimeout(ctx, s.runBudget())
|
||||
defer cancel()
|
||||
return fn(c)
|
||||
}
|
||||
|
||||
// Lag is how far the furthest-behind live run's journal is beyond this platform's cursor, in bytes.
|
||||
//
|
||||
// It is the tailer's own health, and until this pack nothing reported it: a projection that has
|
||||
// stopped moving and a run that is simply quiet look the same from outside. Read separately from the
|
||||
// sweep rather than folded into it because it is telemetry — a failure to measure must not be able
|
||||
// to affect what the sweep does.
|
||||
func (s *Service) Lag(ctx context.Context) (int64, error) {
|
||||
live, err := s.Store.ListLiveRuns(ctx)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var worst int64
|
||||
for _, l := range live {
|
||||
size, err := journalSize(l.Workdir)
|
||||
if err != nil {
|
||||
continue // a book whose directory is unreadable is the reconciler's problem, not this one's
|
||||
}
|
||||
worst = max(worst, size-l.Position.Offset)
|
||||
}
|
||||
return worst, nil
|
||||
}
|
||||
|
||||
func (s *Service) reconcile(ctx context.Context, l pgstore.LiveRun) error {
|
||||
// The cursor this call MOVED, not the one the sweep's snapshot was taken with. Everything below
|
||||
// that asks "has the stream said anything" has to ask about now: the snapshot is read before the
|
||||
|
|
@ -83,6 +133,12 @@ func (s *Service) reconcile(ctx context.Context, l pgstore.LiveRun) error {
|
|||
return err
|
||||
}
|
||||
if l.UnitName == "" {
|
||||
if l.StopRequestedAt != nil {
|
||||
// Stopped before it ever started: the request was admitted, the money was held, and the
|
||||
// unit was never created. There is nothing to signal and nothing to wait for, so the run
|
||||
// ends here and its hold comes back whole through the ordinary settlement.
|
||||
return s.finishStopped(ctx, l)
|
||||
}
|
||||
// Admitted and never spawned: the platform stopped between the transaction and the unit, or
|
||||
// the queue entry was lost. The reconciler is the backstop for both.
|
||||
return s.spawnAttempt(ctx, l)
|
||||
|
|
@ -94,14 +150,84 @@ func (s *Service) reconcile(ctx context.Context, l pgstore.LiveRun) error {
|
|||
return err
|
||||
}
|
||||
if alive {
|
||||
if l.StopRequestedAt != nil {
|
||||
// The intent is committed and the unit is still there. Either the signal never went out —
|
||||
// the platform died between writing the intent and asking systemd — or the engine is
|
||||
// finishing the chunk it has already paid for. Asking again is free and idempotent, and it
|
||||
// is the only thing that closes the first case.
|
||||
if err := s.Runner.Stop(ctx, l.UnitName); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return s.maybeResync(ctx, l)
|
||||
}
|
||||
if s.now().Sub(l.AttemptStartedAt) < spawnGrace {
|
||||
return nil // systemd has not started it yet
|
||||
}
|
||||
if l.StopRequestedAt != nil {
|
||||
// The unit is gone and left no marker, which is the shape of a reboot — but this run was
|
||||
// asked to stop, and restarting it would spend the account's money on work its owner had
|
||||
// just cancelled. The intent decides, because it is the one fact here that is ours.
|
||||
return s.finishStopped(ctx, l)
|
||||
}
|
||||
return s.restart(ctx, l)
|
||||
}
|
||||
|
||||
// finishStopped closes a run whose end this platform asked for and whose unit left no marker to read:
|
||||
// it was never created, or the manager went away with it.
|
||||
//
|
||||
// The two cases are not the same write. Where a unit EXISTED the ordinary finish applies. Where it
|
||||
// never did, the close has to re-check that under a lock: the queue worker may have claimed the
|
||||
// attempt between this sweep's snapshot and now, and closing the run then strands a live engine that
|
||||
// no list looks at any more.
|
||||
func (s *Service) finishStopped(ctx context.Context, l pgstore.LiveRun) error {
|
||||
if l.StopRequestedAt == nil {
|
||||
// The SNAPSHOT may predate the request — that is the whole reason one of the two callers is
|
||||
// here: the reconciler read the run, spent seconds settling, and the stop landed inside that
|
||||
// window. Reaching this function is itself the establishment of the fact, so the local copy is
|
||||
// made to say so rather than letting `outcome` read a nil and call the ending a failure.
|
||||
now := s.now()
|
||||
l.StopRequestedAt = &now
|
||||
}
|
||||
if l.UnitName != "" {
|
||||
return s.finish(ctx, l, runner.Marker{Unit: l.UnitName, Result: pgstore.StopRequestedResult})
|
||||
}
|
||||
// ⚠ An empty unit name is not proof that no process exists. `ReleaseSpawnClaim` clears the name
|
||||
// when the unit could NOT be created — and "could not be created" is not "was not created": a
|
||||
// systemd-run killed after it had already asked leaves an engine running, which is why the spend
|
||||
// baseline is deliberately kept on such an attempt (RecordSpawn). That baseline is the tombstone,
|
||||
// and where it exists the unit's name is derivable, so systemd is asked rather than assumed.
|
||||
// Closing over a live engine would leave it spending against a run this platform calls finished.
|
||||
if l.SpendBaseline != nil {
|
||||
unit := unitName(l.RunID, l.AttemptNo)
|
||||
alive, err := s.Runner.Alive(ctx, unit)
|
||||
if err != nil {
|
||||
// "I could not ask" is never "the run is gone" — the same rule the reconciler follows
|
||||
// everywhere else.
|
||||
return err
|
||||
}
|
||||
if alive {
|
||||
s.log().InfoContext(ctx, "a claim was given back but its unit exists; stopping it instead of closing the run",
|
||||
"run", l.RunID, "attempt", l.AttemptNo)
|
||||
return s.Runner.Stop(ctx, unit)
|
||||
}
|
||||
}
|
||||
finished, err := s.Store.FinishUnspawnedStop(ctx, l.RunID, l.AttemptID, s.now())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !finished {
|
||||
// It was spawned after all. Nothing to do: the next pass sees a live unit and an intent that
|
||||
// says to stop it.
|
||||
s.log().InfoContext(ctx, "the stopped run was spawned after this sweep read it", "run", l.RunID)
|
||||
return nil
|
||||
}
|
||||
s.log().InfoContext(ctx, "run finished", "run", l.RunID, "status", "stopped",
|
||||
"result", pgstore.StopRequestedResult)
|
||||
return s.settle(ctx, l)
|
||||
}
|
||||
|
||||
// drainJournal applies whatever the engine has written since the cursor, and returns where the
|
||||
// cursor now stands.
|
||||
func (s *Service) drainJournal(ctx context.Context, l pgstore.LiveRun) (int64, error) {
|
||||
|
|
@ -216,9 +342,18 @@ func (s *Service) dueForResync(runID string) bool {
|
|||
func (s *Service) finish(ctx context.Context, l pgstore.LiveRun, m runner.Marker) error {
|
||||
delete(s.resynced, l.RunID) // a finished run keeps no rate-limit entry: the map is per process
|
||||
status, exit := outcome(l, m)
|
||||
if err := s.Store.FinishRun(ctx, l.RunID, l.AttemptID, status, m.Result, exit, s.now()); err != nil {
|
||||
closed, err := s.Store.FinishRun(ctx, l.RunID, l.AttemptID, status, m.Result, exit, s.now())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !closed {
|
||||
// The run was already finished, or this pass is holding a snapshot of an attempt that is no
|
||||
// longer the live one — a stop and a resume can both have happened since it was taken. Doing
|
||||
// anything further here would settle the money of a run somebody else is now running.
|
||||
s.log().InfoContext(ctx, "the run moved on since this pass read it; nothing to finish",
|
||||
"run", l.RunID, "attempt", l.AttemptNo)
|
||||
return nil
|
||||
}
|
||||
s.log().InfoContext(ctx, "run finished", "run", l.RunID, "status", status, "result", m.Result)
|
||||
// Settling immediately rather than waiting for the next sweep: the hold is the account's money
|
||||
// and every second it stays reserved is a second the user cannot start another book.
|
||||
|
|
@ -238,29 +373,57 @@ func (s *Service) finish(ctx context.Context, l pgstore.LiveRun, m runner.Marker
|
|||
// verdict returns errReserveCeiling). The contract requires a ceiling stop to be `paused` and never
|
||||
// `failed`, so until the emitter lands the only ceiling stop this platform can report correctly is
|
||||
// one it heard about on the stream — which is exactly the branch below that reads paused_reason.
|
||||
//
|
||||
// A stop this platform ASKED for is a different question and is answered here, because the answer is
|
||||
// ours: the engine catches SIGTERM and exits 1, so the marker cannot tell a stop from a crash — but
|
||||
// the request was written down before the signal went out, and that record is evidence rather than
|
||||
// inference (register row PD-152).
|
||||
func outcome(l pgstore.LiveRun, m runner.Marker) (status string, exitCode *int) {
|
||||
if l.PausedReason != "" {
|
||||
return "paused", nil
|
||||
}
|
||||
code, exited := m.Exited()
|
||||
if !exited {
|
||||
if m.Result == "success" {
|
||||
return "stopped", nil // asked to stop, and it did
|
||||
if exited {
|
||||
switch code {
|
||||
case 0, 2:
|
||||
// 2 is "completed with flagged units": a finished translation whose notes carry the flags.
|
||||
// The engine's own clean answer wins over a stop that arrived while it was already done.
|
||||
return "ready", &code
|
||||
case 3:
|
||||
// The bank-signing stop. The ATTEMPT is over — the engine exits — and the run is left in
|
||||
// a state `resume` continues from once the decisions are complete.
|
||||
return "awaiting_bank", &code
|
||||
}
|
||||
return "failed", nil
|
||||
}
|
||||
switch code {
|
||||
case 0, 2:
|
||||
// 2 is "completed with flagged units": a finished translation whose notes carry the flags.
|
||||
return "ready", &code
|
||||
case 3:
|
||||
// The bank-signing stop. The ATTEMPT is over — the engine exits — and so is this run row;
|
||||
// the work is resumable and resuming it starts a new run, because the contract's resume
|
||||
// handle is not part of this pack.
|
||||
return "awaiting_bank", &code
|
||||
default:
|
||||
if stoppedOnRequest(l, m) {
|
||||
// Exit 1 after our SIGTERM is the ordinary shape of a graceful stop; a kill after the stop
|
||||
// timeout, or an OOM in the same window, is an ungraceful one. Both are the stop the user
|
||||
// asked for, and the machine's own word for what happened stays in exit_result.
|
||||
return "stopped", nil
|
||||
}
|
||||
if !exited && m.Result == "success" {
|
||||
// Killed by a signal and systemd calls the result success: a stop nobody recorded, which is
|
||||
// `systemctl stop` by hand on the host.
|
||||
return "stopped", nil
|
||||
}
|
||||
if exited {
|
||||
return "failed", &code
|
||||
}
|
||||
return "failed", nil
|
||||
}
|
||||
|
||||
// stoppedOnRequest reports whether this platform's own stop is what ended the attempt.
|
||||
//
|
||||
// The timestamp comparison is the race guard the design owes: a run that finished by itself a moment
|
||||
// before someone pressed stop was not stopped by them, and calling it `stopped` would hide a
|
||||
// completed translation behind a cancelled one. Marker.At is written by the unit as it dies, on the
|
||||
// same host and by the same clock the request was stamped with. A marker with no timestamp is
|
||||
// evidence we do not have, and then the intent — the only fact left — decides.
|
||||
func stoppedOnRequest(l pgstore.LiveRun, m runner.Marker) bool {
|
||||
if l.StopRequestedAt == nil {
|
||||
return false
|
||||
}
|
||||
return m.At.IsZero() || !l.StopRequestedAt.After(m.At)
|
||||
}
|
||||
|
||||
// settle resolves the money of an attempt whose process is gone.
|
||||
|
|
@ -279,6 +442,23 @@ func (s *Service) settle(ctx context.Context, l pgstore.LiveRun) error {
|
|||
return nil
|
||||
}
|
||||
key := pgstore.ReservationKey(l.RunID, l.AttemptNo)
|
||||
// An attempt with no unit never reached the engine, so there is nothing to ask it about — and
|
||||
// asking anyway is what kept such a hold reserved for good on the very hosts that produce this
|
||||
// case: a deployment whose engine cannot be run fails `Status` on every pass, and the money of a
|
||||
// run that never ran would wait for an answer that never comes. The row is re-checked under the
|
||||
// lock inside ReleaseUnspawned, so a snapshot that was spawned in the meantime is refused there
|
||||
// and settles the ordinary way below. Found by cross-family review of the acceptance dofix.
|
||||
if l.UnitName == "" && l.SpendBaseline == nil {
|
||||
switch err := s.Store.ReleaseUnspawned(ctx, key, l.AttemptID, s.now()); {
|
||||
case err == nil, errors.Is(err, pgstore.ErrNoReservation):
|
||||
return s.Store.MarkSettled(ctx, l.RunID, s.now())
|
||||
case !errors.Is(err, pgstore.ErrAttemptSpawned):
|
||||
return err
|
||||
}
|
||||
s.log().InfoContext(ctx, "settlement deferred: the attempt was spawned after this sweep read it",
|
||||
"run", l.RunID, "attempt", l.AttemptNo)
|
||||
return nil // the next sweep sees the unit and settles against its baseline
|
||||
}
|
||||
rep, err := s.Engine.Status(ctx, s.engineBinary(l), l.Workdir)
|
||||
if err != nil {
|
||||
s.log().WarnContext(ctx, "settlement deferred: the engine's committed spend could not be read",
|
||||
|
|
@ -367,34 +547,89 @@ func (s *Service) restart(ctx context.Context, l pgstore.LiveRun) error {
|
|||
if err := s.settle(ctx, l); err != nil {
|
||||
return err
|
||||
}
|
||||
next, v, err := s.reopen(ctx, l)
|
||||
if errors.Is(err, pgstore.ErrStopRequested) {
|
||||
// The user pressed stop while this pass was settling. The intent outranks the restart, and it
|
||||
// is still there — the refusal is what kept it — so the run ends here instead.
|
||||
s.log().InfoContext(ctx, "restart abandoned: the run was asked to stop", "run", l.RunID)
|
||||
return s.finishStopped(ctx, l)
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch v {
|
||||
case exhausted:
|
||||
// `paused` WITH its reason: the contract describes PausedReason as the reason a run is paused,
|
||||
// and a paused run with a null reason gives the screen nothing to say. PauseRun is the path
|
||||
// that sets it; FinishRun does not. Done HERE and not inside reopen because the other caller —
|
||||
// a resume — works on a run that has already ended, and pausing one of those would rewrite how
|
||||
// it ended.
|
||||
//
|
||||
// A stop that arrived while this pass was settling outranks the pause: the store refuses and
|
||||
// the run ends as what it is, stopped.
|
||||
if err := s.Store.PauseRun(ctx, l.RunID, l.AttemptID, pgstore.PausedCreditExhausted, s.now()); err != nil {
|
||||
if errors.Is(err, pgstore.ErrStopRequested) {
|
||||
s.log().InfoContext(ctx, "pause abandoned: the run was asked to stop", "run", l.RunID)
|
||||
return s.finishStopped(ctx, l)
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
case deferred:
|
||||
return nil
|
||||
}
|
||||
s.log().InfoContext(ctx, "interrupted run restarted", "run", l.RunID, "attempt", next.AttemptNo)
|
||||
return s.spawnAttempt(ctx, next)
|
||||
}
|
||||
|
||||
// verdict is what reopening a run came to. Three outcomes because the callers want different things
|
||||
// from the two that are not "it started": the reconciler treats both as "nothing more this pass",
|
||||
// while a user's resume must tell "your money is still being counted, ask again" from "there is
|
||||
// nothing left of this run's budget".
|
||||
type verdict int
|
||||
|
||||
const (
|
||||
reopened verdict = iota
|
||||
// deferred — the previous attempt's money is not settled yet. Temporary by construction: the
|
||||
// settlement is retried by every sweep.
|
||||
deferred
|
||||
// exhausted — nothing is left of the run's budget, or the account cannot carry a new hold. What
|
||||
// to DO about it differs by caller, so reopen only reports it.
|
||||
exhausted
|
||||
)
|
||||
|
||||
// reopen closes an attempt and opens the next one with what is LEFT of the run's budget.
|
||||
//
|
||||
// It is the shared body of the reconciler's restart and of the contract's resume, and sharing it is
|
||||
// the point: both are "this run continues in a new process", both must give the new attempt the
|
||||
// remainder rather than the whole ceiling — one run may not spend its ceiling twice — and both must
|
||||
// refuse to open a second hold while the first is still open.
|
||||
func (s *Service) reopen(ctx context.Context, l pgstore.LiveRun) (pgstore.LiveRun, verdict, error) {
|
||||
// Settling is allowed to DEFER — the engine's figure may not be readable — and a deferral must not
|
||||
// become a second reservation. Asked rather than assumed: proceeding on an unsettled attempt holds
|
||||
// the ceiling twice and strands the first hold where no later sweep looks for it.
|
||||
open, err := s.Store.AttemptReservationOpen(ctx, l.RunID, l.AttemptNo)
|
||||
if err != nil {
|
||||
return err
|
||||
return pgstore.LiveRun{}, deferred, err
|
||||
}
|
||||
if open {
|
||||
s.log().WarnContext(ctx, "restart deferred: the interrupted attempt is not settled yet",
|
||||
s.log().WarnContext(ctx, "restart deferred: the previous attempt is not settled yet",
|
||||
"run", l.RunID, "attempt", l.AttemptNo)
|
||||
return nil
|
||||
return pgstore.LiveRun{}, deferred, nil
|
||||
}
|
||||
budget := s.Pricing.Ceiling(l.CeilingChapters)
|
||||
spent, err := s.Store.RunSpent(ctx, l.RunID)
|
||||
if err != nil {
|
||||
return err
|
||||
return pgstore.LiveRun{}, deferred, err
|
||||
}
|
||||
remaining := budget - spent
|
||||
if remaining <= 0 {
|
||||
// `paused` WITH its reason: the contract describes PausedReason as the reason a run is paused,
|
||||
// and a paused run with a null reason gives the screen nothing to say. PauseRun is the path
|
||||
// that sets it; FinishRun does not.
|
||||
s.log().InfoContext(ctx, "interrupted run has no budget left", "run", l.RunID)
|
||||
return s.Store.PauseRun(ctx, l.RunID, l.AttemptID, pgstore.PausedCreditExhausted, s.now())
|
||||
s.log().InfoContext(ctx, "the run has no budget left", "run", l.RunID)
|
||||
return pgstore.LiveRun{}, exhausted, nil
|
||||
}
|
||||
offset, err := journalSize(l.Workdir)
|
||||
if err != nil {
|
||||
return err
|
||||
return pgstore.LiveRun{}, deferred, err
|
||||
}
|
||||
// Nil = inherit the pinned build. A different one only when an operator asked for it (row 139).
|
||||
var version *string
|
||||
|
|
@ -412,25 +647,142 @@ func (s *Service) restart(ctx context.Context, l pgstore.LiveRun) error {
|
|||
Now: s.now(),
|
||||
})
|
||||
if errors.Is(err, pgstore.ErrInsufficientCredit) {
|
||||
// The balance went elsewhere while this run was down. Honest state, not a failure: the run is
|
||||
// paused for the reason the contract has a word for.
|
||||
s.log().InfoContext(ctx, "interrupted run cannot be resumed on the current balance", "run", l.RunID)
|
||||
return s.Store.PauseRun(ctx, l.RunID, l.AttemptID, pgstore.PausedCreditExhausted, s.now())
|
||||
// The balance went elsewhere while this run was down. Honest state, not a failure.
|
||||
s.log().InfoContext(ctx, "the run cannot be continued on the current balance", "run", l.RunID)
|
||||
return pgstore.LiveRun{}, exhausted, nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
return pgstore.LiveRun{}, deferred, err
|
||||
}
|
||||
s.log().InfoContext(ctx, "interrupted run restarted", "run", l.RunID, "attempt", next.AttemptNo)
|
||||
return s.spawnAttempt(ctx, next)
|
||||
return next, reopened, nil
|
||||
}
|
||||
|
||||
// Stop asks a live run to shut down. The engine stops gracefully on SIGTERM and finishes the chunk
|
||||
// it has already paid for; the reconciler turns the exit into a status.
|
||||
func (s *Service) Stop(ctx context.Context, l pgstore.LiveRun) error {
|
||||
if l.UnitName == "" {
|
||||
return fmt.Errorf("runs: run %s has no unit to stop", l.RunID)
|
||||
func (s *Service) enqueueNow(ctx context.Context, runID string) error {
|
||||
if s.Queue == nil {
|
||||
return nil // no queue configured: the reconciler still picks the run up on its next sweep
|
||||
}
|
||||
return s.Runner.Stop(ctx, l.UnitName)
|
||||
return s.Queue.EnqueueRunNow(ctx, runID)
|
||||
}
|
||||
|
||||
// Stop is the contract's stop handle (§stopRun).
|
||||
//
|
||||
// What it does is record the INTENT and then ask systemd, in that order, and the order is the whole
|
||||
// mechanism: the engine catches SIGTERM and exits 1, so the exit marker says the same thing for a
|
||||
// stop and for a crash (register row PD-152). The row written before the signal is what tells them
|
||||
// apart afterwards — and what stops the reconciler restarting a run whose owner just cancelled it.
|
||||
//
|
||||
// The systemd call is allowed to fail without failing the request: the intent is committed, the
|
||||
// reconciler re-issues the stop on its next pass, and answering an error to a stop that WILL happen
|
||||
// would invite the user to press it again.
|
||||
func (s *Service) Stop(ctx context.Context, userID, runID string) (pgstore.Run, error) {
|
||||
run, unit, err := s.Store.RequestStop(ctx, userID, runID, s.now())
|
||||
if errors.Is(err, pgstore.ErrRunNotLive) {
|
||||
// No status in the message: RequestStop answers a zero Run alongside this error, so anything
|
||||
// read off it would be an empty string dressed up as a fact.
|
||||
return pgstore.Run{}, ErrNotStoppable
|
||||
}
|
||||
if err != nil {
|
||||
return pgstore.Run{}, err
|
||||
}
|
||||
s.log().InfoContext(ctx, "run stop requested", "run", runID)
|
||||
if unit == "" {
|
||||
// Admitted and not yet spawned. There is nothing to signal; the reconciler ends the run and
|
||||
// gives the hold back on its next pass.
|
||||
return run, nil
|
||||
}
|
||||
if err := s.Runner.Stop(ctx, unit); err != nil {
|
||||
s.log().ErrorContext(ctx, "the stop was recorded but systemd did not take it; the reconciler will retry",
|
||||
"run", runID, "err", err)
|
||||
}
|
||||
return run, nil
|
||||
}
|
||||
|
||||
// Resume is the contract's resume handle (§resumeRun): it continues a run that was stopped.
|
||||
//
|
||||
// The mechanics are the reconciler's — a new attempt with what is LEFT of the run's budget — because
|
||||
// a resume and a restart are the same event seen from two sides, and a second implementation of
|
||||
// "give this run another process" is a second place for the money to be got wrong.
|
||||
//
|
||||
// What this adds is the state machine the contract describes, and its shape is dictated by that text
|
||||
// rather than chosen:
|
||||
// - a run the user stopped continues;
|
||||
// - a bank stop answers 409 while the set of decisions is incomplete — and it is incomplete for
|
||||
// every book today, because nothing materializes the bank at all (companion §3: the export
|
||||
// artifact the read needs does not exist);
|
||||
// - a run paused by a ceiling "returns it to the same state", so it answers 202 with the run
|
||||
// unchanged rather than pretending an action happened;
|
||||
// - anything else — already going, finished, failed — is a 409.
|
||||
func (s *Service) Resume(ctx context.Context, userID, runID string) (pgstore.Run, error) {
|
||||
// Ownership FIRST, deployment health second. The other order answers "this service cannot start
|
||||
// runs" to a caller asking about a run that is not theirs and to one asking about a run that does
|
||||
// not exist — which is both a worse answer and a small oracle about the shape of the deployment.
|
||||
l, err := s.Store.ReadRunForResume(ctx, userID, runID)
|
||||
if err != nil {
|
||||
return pgstore.Run{}, err
|
||||
}
|
||||
if err := s.runnable(); err != nil {
|
||||
return pgstore.Run{}, err
|
||||
}
|
||||
switch l.Status {
|
||||
case "stopped", "paused":
|
||||
case "awaiting_bank":
|
||||
// The stop clears ONLY on a complete set of decisions (contract §resumeRun, §BankDecisions).
|
||||
// There is no channel that could make one complete yet, so this is the honest answer rather
|
||||
// than a placeholder: resuming would hit the same stop and spend the start of a run doing it.
|
||||
return pgstore.Run{}, fmt.Errorf("%w: the glossary is not signed", ErrNotResumable)
|
||||
default:
|
||||
return pgstore.Run{}, fmt.Errorf("%w: it is %s", ErrNotResumable, l.Status)
|
||||
}
|
||||
next, v, err := s.reopen(ctx, l)
|
||||
if errors.Is(err, pgstore.ErrStopRequested) {
|
||||
// Cannot happen through this path today — a resume works on a run that has ENDED and the guard
|
||||
// only fires on a live one — but the answer is written rather than left to fall through as an
|
||||
// internal error if some later path reaches it.
|
||||
return pgstore.Run{}, fmt.Errorf("%w: it has been asked to stop", ErrNotResumable)
|
||||
}
|
||||
if errors.Is(err, pgstore.ErrNoRun) {
|
||||
// Another caller re-opened this run first — two clicks on the same button reach here together,
|
||||
// and the second loses the race for attempt N+1 on the unique index. Its answer is the WORK OF
|
||||
// THE FIRST: the run is continuing, so reporting "no such run" would be both false and
|
||||
// alarming, and re-opening again would take a second hold.
|
||||
return s.Store.ReadRun(ctx, userID, runID)
|
||||
}
|
||||
if err != nil {
|
||||
return pgstore.Run{}, err
|
||||
}
|
||||
switch v {
|
||||
case deferred:
|
||||
// The previous attempt's money is still being resolved. A second hold now would reserve the
|
||||
// ceiling twice, so the answer is "not yet" and the sweep clears it.
|
||||
return pgstore.Run{}, fmt.Errorf("%w: its previous attempt is still being settled", ErrNotResumable)
|
||||
case exhausted:
|
||||
// Nothing left to continue with: the call returns the run in the state it was in, and the state
|
||||
// is NOT rewritten on the way out — a run the user stopped stays stopped, because a resume that
|
||||
// changed nothing must not change what the run was.
|
||||
//
|
||||
// ⚠ This is NOT the whole of what the contract asks about `paused`. It says a resume must not be
|
||||
// the remedy for a ceiling pause, and this branch only produces that answer when there is
|
||||
// nothing left to spend. Today the only reachable `paused` is `credit_exhausted` from the
|
||||
// reconciler, where continuing after a grant IS the right thing; a CEILING pause needs the
|
||||
// engine's ceiling event, which does not exist (register row PD-113). When it lands, resume has
|
||||
// to refuse for that reason instead of re-opening the run — a run whose ceiling nothing raised
|
||||
// would otherwise walk into the same stop and, with no event to name it, come back `failed`.
|
||||
return s.Store.ReadRun(ctx, userID, runID)
|
||||
}
|
||||
s.log().InfoContext(ctx, "run resumed", "run", runID, "attempt", next.AttemptNo)
|
||||
// QUEUED, not spawned here. Starting the engine inline would put two slow things inside a request
|
||||
// that answers 202: reading the book's meter costs seconds of the engine's CPU, and creating the
|
||||
// unit is a round trip to systemd. The admission path queues for the same reason, and the
|
||||
// reconciler is the backstop if the entry is lost.
|
||||
if err := s.enqueueNow(ctx, next.RunID); err != nil {
|
||||
// The run IS re-opened and the money IS held: the row says `translating` and the reconciler
|
||||
// picks up an attempt with no unit on its next pass, exactly as it does after an admission.
|
||||
// Reporting a failure here would tell the user nothing had started while the row said
|
||||
// otherwise.
|
||||
s.log().ErrorContext(ctx, "the resumed run could not be queued; the reconciler will retry",
|
||||
"run", runID, "err", err)
|
||||
}
|
||||
return s.Store.ReadRun(ctx, userID, runID)
|
||||
}
|
||||
|
||||
// attemptSpend turns the engine's figure into what THIS attempt cost.
|
||||
|
|
|
|||
|
|
@ -30,6 +30,9 @@ type fakeRunner struct {
|
|||
alive bool
|
||||
aliveErr error
|
||||
startErr error
|
||||
// onStop runs INSIDE the fake's Stop, which is what makes the ORDER of "write the intent, then
|
||||
// ask systemd" assertable: by the time this runs, the database must already know.
|
||||
onStop func(unit string)
|
||||
}
|
||||
|
||||
func (f *fakeRunner) Start(_ context.Context, s runner.Spec) error {
|
||||
|
|
@ -44,11 +47,21 @@ func (f *fakeRunner) Start(_ context.Context, s runner.Spec) error {
|
|||
|
||||
func (f *fakeRunner) Stop(_ context.Context, unit string) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
hook := f.onStop
|
||||
f.stopped = append(f.stopped, unit)
|
||||
f.mu.Unlock()
|
||||
if hook != nil {
|
||||
hook(unit)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeRunner) stops() []string {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]string(nil), f.stopped...)
|
||||
}
|
||||
|
||||
func (f *fakeRunner) Alive(context.Context, string) (bool, error) { return f.alive, f.aliveErr }
|
||||
|
||||
func (f *fakeRunner) starts() []runner.Spec {
|
||||
|
|
@ -64,13 +77,38 @@ type fakeEngine struct {
|
|||
report ingest.StatusReport
|
||||
err error
|
||||
calls int
|
||||
// hangs is the workdir the engine refuses to answer about.
|
||||
hangs string
|
||||
// onStatus runs INSIDE the call, which is how a test opens the window the reconciler decides in:
|
||||
// a status call is seconds of the engine's CPU, and a user can press stop inside it.
|
||||
onStatus func()
|
||||
}
|
||||
|
||||
func (f *fakeEngine) Status(context.Context, string, string) (ingest.StatusReport, error) {
|
||||
func (f *fakeEngine) Status(ctx context.Context, _, workdir string) (ingest.StatusReport, error) {
|
||||
f.mu.Lock()
|
||||
f.calls++
|
||||
hang := f.hangs != "" && f.hangs == workdir
|
||||
hook := f.onStatus
|
||||
rep, err := f.report, f.err
|
||||
f.mu.Unlock()
|
||||
if hook != nil {
|
||||
hook()
|
||||
}
|
||||
if hang {
|
||||
// An engine that never answers, which is what a hung `tmctl status` looks like from here. It
|
||||
// ends when the CALLER's budget does, and that is the property being measured.
|
||||
<-ctx.Done()
|
||||
return ingest.StatusReport{}, ctx.Err()
|
||||
}
|
||||
return rep, err
|
||||
}
|
||||
|
||||
// block makes the engine hang for one book's directory. The run id is taken only to make the call
|
||||
// site read like the thing it models.
|
||||
func (f *fakeEngine) block(_ string, workdir string) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.calls++
|
||||
return f.report, f.err
|
||||
f.hangs = workdir
|
||||
}
|
||||
|
||||
func (f *fakeEngine) set(rep ingest.StatusReport, err error) {
|
||||
|
|
|
|||
|
|
@ -59,6 +59,10 @@ type Config struct {
|
|||
// the engine is deployed more often than a translation finishes, so the quiet behaviour would be
|
||||
// for a resumed run to continue under a program nobody chose for it.
|
||||
AllowEngineVersionChange bool
|
||||
// RunBudget is what ONE run may cost a pass of the sweep; zero takes the default. It bounds the
|
||||
// starvation the pass itself cannot: the list is ordered the same way every time, so a handful of
|
||||
// runs whose engine hangs used to mean the tail was never reached (register row PD-169).
|
||||
RunBudget time.Duration
|
||||
// ResyncEvery is how often a live run is reconciled from `tmctl status --json`. It is a SLOW
|
||||
// poll on purpose: every call re-ingests and re-chunks the source (~1.4-1.5 s of CPU on a 23 MB
|
||||
// book, unified backlog row 100). Until the event emitter exists (row 103) it is the only thing
|
||||
|
|
@ -83,9 +87,16 @@ type Service struct {
|
|||
resynced map[string]time.Time
|
||||
}
|
||||
|
||||
// Enqueuer hands a run to the queue inside the caller's transaction.
|
||||
// Enqueuer hands a run to the queue.
|
||||
//
|
||||
// Two methods because the two callers need different atomicity, not different queues: an ADMISSION
|
||||
// writes the run, its hold and its queue entry in one transaction (a job for a run that does not
|
||||
// exist, or a hold with no job, are both ways to lose money or work), while a RESUME re-opens a run
|
||||
// that already exists and only then asks for a worker. A lost entry costs a sweep interval either
|
||||
// way — the reconciler is the backstop for both.
|
||||
type Enqueuer interface {
|
||||
EnqueueRun(ctx context.Context, tx pgstore.Tx, runID string) error
|
||||
EnqueueRunNow(ctx context.Context, runID string) error
|
||||
}
|
||||
|
||||
func (s *Service) now() time.Time {
|
||||
|
|
@ -115,6 +126,18 @@ var ErrCeilingOutOfBounds = errors.New("runs: the requested ceiling is outside t
|
|||
// one during exactly the operation that causes it, a binary upgrade.
|
||||
var ErrRunnerIncomplete = errors.New("runs: this deployment cannot record how a run ends")
|
||||
|
||||
// ErrBookNotReady is a run asked for on a book whose intake has not finished — one still arriving,
|
||||
// still being cut, or rejected. The contract's 409: the book exists and the client can see it, and
|
||||
// what cannot happen yet is a translation of it.
|
||||
var ErrBookNotReady = errors.New("runs: the book is not ready to be translated")
|
||||
|
||||
// ErrNotStoppable is a stop asked for on a run that is already over.
|
||||
var ErrNotStoppable = errors.New("runs: the run is not live")
|
||||
|
||||
// ErrNotResumable is a resume that would not move the run: a run that is already going, one that
|
||||
// finished, or a bank stop whose set of decisions is not complete (contract §resumeRun, 409).
|
||||
var ErrNotResumable = errors.New("runs: the run cannot be continued")
|
||||
|
||||
// Bounds computes the run-ceiling scale for a book.
|
||||
func (s *Service) Bounds(ctx context.Context, userID, bookID string) (pricing.Bounds, error) {
|
||||
book, err := s.Store.ReadBookForRun(ctx, userID, bookID)
|
||||
|
|
@ -154,6 +177,13 @@ func (s *Service) Start(ctx context.Context, in StartRequest) (pgstore.Run, erro
|
|||
if err != nil {
|
||||
return pgstore.Run{}, err
|
||||
}
|
||||
if !readyToTranslate(book.Status) {
|
||||
// A book whose intake has not finished has no chapter tree to translate and, while it is
|
||||
// `uploading`, half a file on disk. Refused here rather than discovered by the engine inside a
|
||||
// transient unit, where the only trace would be a marker reading "exit-code 1" — and after the
|
||||
// account's money had already been held for it.
|
||||
return pgstore.Run{}, fmt.Errorf("%w: it is %s", ErrBookNotReady, book.Status)
|
||||
}
|
||||
if book.HasLiveRun {
|
||||
return pgstore.Run{}, pgstore.ErrRunInFlight
|
||||
}
|
||||
|
|
@ -187,6 +217,21 @@ func (s *Service) Start(ctx context.Context, in StartRequest) (pgstore.Run, erro
|
|||
return started.Run, nil
|
||||
}
|
||||
|
||||
// readyToTranslate is the half of the book vocabulary a run may start from: everything a finished
|
||||
// intake can leave behind, plus every state a previous run can leave a parsed book in.
|
||||
//
|
||||
// Written as an allowlist and not as "not uploading, not parsing, not rejected": a status added
|
||||
// later is one this function has never thought about, and refusing it is the safe direction — the
|
||||
// user gets a 409 they can report, rather than an engine started against a book in a state nobody
|
||||
// designed for.
|
||||
func readyToTranslate(status string) bool {
|
||||
switch status {
|
||||
case "not_started", "translating", "awaiting_bank", "finalizing", "ready", "paused", "stopped", "failed":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (s *Service) enqueue(ctx context.Context, tx pgstore.Tx, runID string) error {
|
||||
if s.Queue == nil {
|
||||
return nil // no queue configured: the reconciler still picks the run up on its next sweep
|
||||
|
|
|
|||
|
|
@ -52,6 +52,14 @@ func (s *Service) spawnAttempt(ctx context.Context, l pgstore.LiveRun) error {
|
|||
if err := s.runnable(); err != nil {
|
||||
return err
|
||||
}
|
||||
// A run the user has already asked to stop is not started. The claim in RecordSpawn refuses the
|
||||
// window this cannot see — the intent written after this read — but the ordinary case is a stop
|
||||
// pressed on a queued run before its worker got to it, and starting an engine there would spend
|
||||
// the account's money on work that was cancelled before it began.
|
||||
if l.StopRequestedAt != nil {
|
||||
s.log().InfoContext(ctx, "not starting a run that has been asked to stop", "run", l.RunID)
|
||||
return nil
|
||||
}
|
||||
// The engine's own money state for this book, read BEFORE anything is started, because afterwards
|
||||
// both of its numbers have already moved. It decides two different things: what this attempt will
|
||||
// owe when it ends (the difference from the committed figure) and what limit it may be given (the
|
||||
|
|
|
|||
|
|
@ -1431,3 +1431,38 @@ func TestAResumeMovesToANewEngineBuildOnlyWhenItIsAllowed(t *testing.T) {
|
|||
t.Fatalf("the resumed attempt ran %q, want the new build", starts[len(starts)-1].Binary)
|
||||
}
|
||||
}
|
||||
|
||||
// Cross-family review of the acceptance dofix (M3): the same hold, on the host that actually produces
|
||||
// this case. An attempt is unspawned BECAUSE the engine could not be run — and settlement used to ask
|
||||
// that same engine for a committed spend before it would give the money back, so every pass failed on
|
||||
// the call and the hold of a run that never ran stayed reserved for good.
|
||||
func TestTheHoldOfARunThatNeverStartedComesBackOnAHostWhoseEngineCannotAnswer(t *testing.T) {
|
||||
f := newFixture(t, "10", 500)
|
||||
run, err := f.svc.Start(f.ctx, StartRequest{UserID: "u1", BookID: f.bookID(t), CeilingChapters: 100})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
live := f.live(t)
|
||||
if live.UnitName != "" {
|
||||
t.Fatal("the attempt was spawned; this test is about one that never was")
|
||||
}
|
||||
f.engine.set(ingest.StatusReport{}, errors.New("tmctl status: exec: no such file or directory"))
|
||||
if err := runner.WriteMarker(f.svc.markerPath(run.ID, live.AttemptNo),
|
||||
runner.Marker{Unit: "u", Result: "exit-code", Code: "exited", Status: "1"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := f.svc.Sweep(f.ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
acct := f.account(t)
|
||||
if acct.Reserved != 0 {
|
||||
t.Fatalf("reserved %s: the hold of a run that never started waits on an engine that cannot answer",
|
||||
acct.Reserved.USD())
|
||||
}
|
||||
if acct.Balance != money.MicroUSD(10_000_000) {
|
||||
t.Errorf("balance %s: a run that never started must cost nothing", acct.Balance.USD())
|
||||
}
|
||||
if f.engine.called() != 0 {
|
||||
t.Errorf("the engine was asked %d times about an attempt that never reached it", f.engine.called())
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue