diff --git a/backend/PACK11_CONT_REPORT.md b/backend/PACK11_CONT_REPORT.md new file mode 100644 index 0000000..164e967 --- /dev/null +++ b/backend/PACK11_CONT_REPORT.md @@ -0,0 +1,250 @@ +# PACK11_CONT_REPORT — бэкенд-сессия ПРОДОЛЖЕНИЯ, резидуал R1–R5 (2026-07-19) + +> Продолжение `PACK11_REPORT.md` (Block A, `85f5b9e`). Строил по `docs/architecture/11-implementation-plan.md` +> (ратифицирован D39.12) + `PACK11_REPORT.md` §Резидуал (пошаговые дизайны R1–R5). **НЕ коммитил** — сдача +> на приёмку. Верификация исполнением: `go build ./... && go vet ./... && go test -race ./...` — ВСЁ +> ЗЕЛЁНОЕ на каждом чекпойнте (логи в §Верификация). Block A НЕ трогал (под тестами). + +## 0. Санкционированный порядок и итог + +**Развилка порядка эскалирована владельцу/оркестратору в начале сессии** (R1 = крупный каскад, меняющий +ГРАНУЛЯРНОСТЬ вывода → ~весь e2e-суит + read-models + golden; риск красного дерева vs жёсткое правило +«суит зелёный на каждом чекпойнте»). **Оркестратор санкционировал вариант 1:** R2→R3→R4→R5 сначала (все +самодостаточны и зелёно-достижимы независимо от R1), **R1 — капстоуном на остатке бюджета**, с условиями: +чекпойнт-ревью после R2+R3 и после R4; R3 на текущем последовательном раннере; каждый снапшот-двигающий +лендинг — свой golden re-capture с masked-диффом, СТРУКТУРНЫЙ рерайт — один раз в R1; R1 сначала аддитивные +под-шаги без каскада, переключение драйвера — финальный атомарный шаг; не дотягивает до зелёного → дерево +зелёное, R1 точным резидуалом (запланированный исход). + +| WS / шаг | Статус | Верификация | +|---|---|---| +| **R2 · WS3 Go-майнер** | ✅ **СДАНО** | Go↔Python паритет **EXACT**: 13618 канд-СЕТ, катастроф {方源:0,蛊:1,蛊师:2,古月:22}, recall@proposed **0.9649**; alias-ярус-1 (5 R4-блоков+компози-гард); сид-дельта+`MinedDeltaYAML`; `Source:mined`-путь; `tmctl seed-lint`. | +| **R3 · WS4 банкнота 12 точек** | ✅ **СДАНО** (на посл. раннере) | срез-до-classify (fresh+resume); `tm-banknote-v1:` derived + re-point после эскалации; `banknoteSnap` (omitempty); v9-телеметрия; finish=stop-only гейт. 5 e2e-пинов. | +| **R4a · `tmctl export --pairs`** | ✅ **СДАНО** | source-столбец (пред-условие §5(д)-FP DC1/DC2). | +| **R4b · чекеры DC1/DC2/DC6 + zh-ru пакет** | ✅ **СДАНО** | эмпир-позитивы пойманы; DC2 **байт-верно референсу** (ok_re-supress + case-sensitive — фикс ревью); `cheapGateVersion` v3; golden re-capture masked-чист. | +| **R4c · DC3 gender-инъекция** | ✅ **СДАНО** | `memoryEntry.gender`→editor-констрейнт; `renderFormatVersion` v2; golden re-capture masked-чист. | +| **R5 · WS6 арм-конфиги** | ✅ **СДАНО** | `pipeline-arm-mistral.yaml`/`pipeline-arm-deepseek-pro.yaml`; `echo_mine_test` под варианты; rate-guard связка. | +| **R1 · волновой ИСПОЛНИТЕЛЬ** | ⚠️ **АДДИТИВ СДАН · драйвер-свитч РЕЗИДУАЛ** | `snapshotIDForWave` (пер-волновой снапшот, «переоплата ОДНА» пиннута) + W0 sticky-precompute (**байт-идентично, golden-доказано**) — СДАНЫ, зелёные. Параллельный dispatch W1/W1.5/W2 — резидуал (§R1-residual, дизайн ниже). | +| **Чекпойнт-ревью** | ✅ ×3 | R2+R3 (4 minor/nit, фикс) · R4 (2 DC2-parity, фикс) · финал (см. §Ревью). Адверсариал-воркфлоу, author≠reviewer, refute-by-default. | + +--- + +## R2 — WS3 Go-майнер (план §3) — ✅ СДАНО + +Новый пакет `internal/pipeline/miner*.go` (substrate/detect/patterns/palladius/alias/emit) — **байт-верный +порт frozen exp16** (`exp16_common/detectors/patterns/arms/alias/palladius/emit_owner_sheets.py`), +**дефолт-лемматизатор B**. Per-буллет (§R2): + +- ✅ **Порт V-A→V-C по ТОЧНЫМ порогам §3(б):** c-value `g(L)=log₂(L+1)` (не log₂L — пин + `TestMinerLengthMultNonDegenerate`/`CValueSingleCharNested`); subsumption α=0.80 (`TestMinerSubsumption`); + weirdness `p_book/max(word_rel,char_indep_rel,1e-9)`, N_book = полная пре-флор масса; n-граммы Han 1–6 + overlapping, freq-floor 3; паттерн-каналы (surname/百家姓−凝, title-suffix/bare, ordinal_title, rank_grade, + topo, авто-формант over_rep≥15∧partners≥3 — `TestMinerFormantDetection`); arm A3 λ=0, bonus·pw, + subfloor `bonus+40·f·pw`. Детерминизм: все сорты `(-score,-freq,src)`, map-итерация только ПЕРЕД тотал-сортом. +- ✅ **Дефолт B — дроп Палладий-подканала V-C** (плюс `dst_variants`, `is_name_lemma`, spread λ, alias-R3, + canon-dst-recovery — все pymorphy3-зависимые; dst доставляет банкнота WS4). `is_palladius_token` порт как + **детерминированный** артефакт (Палладий-таблица, без pymorphy3) + тестируем (`TestMinerPalladiusToken`). + **Палладий-ЗАВИСИМЫЙ ранг 古月:21→22** — как предписано §10-14/§3(д); катастроф-скрин ∈top-50 PASS. +- ✅ **Палладий-ИНВАРИАНТНЫЕ гарантии — EXACT** (`TestMinerFullBookParity`, data-gated `TM_MINER_PARITY=1` на + стенд-данных: jieba-контраст + `rerun/records.json` + `guzhenren-seed-v2.yaml`): **n=13618** (SET-членство), + recall@proposed **0.9649** (55/57 GT), катастроф {方源:0,蛊:1,蛊师:2,古月:22}. Скипается при отсутствии данных + (как `TM_LIVE`). **Python = референс: byte-verified, расхождений 0 по инвариантам.** +- ✅ **Алиас-ярус-1 (R1–R4, R4 hard-blocks ПЕРВЫМИ):** `TestMinerAliasRules` (5 сабтестов): R1-extension + (方源⊂古月方源), компози-гард (古月+族长=фраза-не-алиас, но 古月+方源=fullname-алиас), R4-iii/v (族长≠四代族长 по + разному approved-dst), R4-ii (разный пол блок, hidden не блокирует), R4-i/R2 (одна фамилия+разные имена→family). + **R3 (shared-ru-rendering) ОМИТ в дефолте B** (нужен `dst_lemmas`/pymorphy3) — задокументировано в коде. +- ✅ **Эмиссия сид-дельты §C2-7:** `MineBank`→`MinedTerm` с фильтрами `build_precision30` (тип∈{name,place,title}, + freq≥5, ∉subsumed, len≥2, ∉фрагмент по PARTICLE) — НЕ сырой 13618-дамп; alias-кластеризация (репрезентант = + топ-ранг; кластер с seed-поверхностью = алиас-of-existing, не эмитится). `MinedDeltaYAML` → seedTerm-схема + БЕЗ новых полей. `TestMineBankDeterministicAndNonSeed`/`TestMinerEmissionFilters`. +- ✅ **`Source:"mined"` mined-write-путь:** `minedToCandidates` (штамп `Source:mined`, status:auto, no-dst, + зеркало `rubyToCandidates`, НЕ через `loadGlossarySeed`). `TestMinedToCandidatesStampsMinedSource` + + `TestMinedDeltaStaysBaseBankStable` (Source:mined НЕ двигает `BaseVersion` — «переоплата ОДНА» через реальный путь). +- ✅ **`tmctl seed-lint`:** `SeedLint(path)` = сухой прогон РЕАЛЬНОГО `loadGlossarySeed`+`approvedSharedKeyCollisions`; + CLI `seed-lint --seed ` (без `--config`). `seedlint_test.go`: чистый→OK; дефекты (approved-без-dst / + unknown-status / duplicate / shared-key) → фейл-лауд; **эмитированная дельта линтует чисто** (`TestSeedLintEmittedMinedDelta`). + +**Артефакт-зависимость:** прод-майнер грузит jieba-контраст (5MB, SHA `7197c321…`, ВНЕ git) через +`LoadContrast(io.Reader)` — путь конфигурируется вызывающим. Тесты — синтетический малый контраст; +паритет — реальный jieba со стенда. + +**Не-каскадная зона R2:** `MineBank`/`minedToCandidates`/`MinedDeltaYAML` — библиотека пак-3; ВЫЗОВ на +W1.5-границе (реseed через `ReplaceGlossary`) — часть волнового W1.5-стопа (R1-residual). Сдано как +самодостаточное, тестируемое ядро (санкция оркестратора: R2 самодостаточен, R1 вайрит). + +--- + +## R3 — WS4 банкнота: 12 интеграционных точек (план §4) — ✅ СДАНО (на посл. раннере) + +Ядро `banknote.go` (Block A) + интеграция. **Все точки пер-стадийные → в волновую модель переезжают КАК ЕСТЬ** +(применяются к W1/draft-стадии; условие оркестратора). Per-точка: + +1. ✅ **Санитайзер:** банкнота-срез — ПЕРВЫЙ рубеж (`applyBanknote` до `classifyOutput`); trailing-note НЕ + реюзим (ревью-1 F3). Финал-бэкстоп = телеметрия п.10. +2. ✅ **cjk_leak:** src-колонка Han по построению → срез ДО санитайзера снимает ложь. +3. ✅ **classify/echo (п.3, ЖЁСТКИЙ порядок):** `applyBanknote` режет блок на ОБОИХ путях runAttempt (fresh-call + И checkpoint-hit/resume) ДО `classifyOutput` → echo/coverage/length видят ОЧИЩЕННЫЙ перевод, не Han-сноску. + Пин `TestBanknoteClassifyOverCleaned` (клин RU + Han-тяжёлый блок → OK, не cjk_artifact). +4. ✅ **Coverage:** числитель по очищенному (срез раньше coverageCheck). +5. ✅ **max_tokens += `bankTokenBudget`** (≤12×12) для translator при enabled (перевод первым, блок хвостом → + трункация бьёт блок). Пин: draft `max_tokens=656` (512+144) в e2e. +6. ✅ **request_hash/снапшот:** `banknoteSnap{enabled,parser_version}` в `snapshotID()`. **ДЕВИАЦИЯ-с-обоснованием: + omitempty-указатель (nil при off) вместо безусловного `{enabled:false}` зеркала sanitizerSnap** — включение + канала И bump парсера ВСЁ РАВНО громкий сдвиг (несущая loudness §4а-п6 сохранена), но ДОБАВЛЕНИЕ фичи НЕ + инвалидирует banknote-OFF книги (в т.ч. golden-фикстуру) → R3 golden БАЙТ-СТАБИЛЕН, re-capture не нужен. + Пин `TestBanknoteSnapshotFold`. +7. ✅ **Чекпоинт/резюм:** derived content-addressed → resume re-derives id бесплатно; сырой чекпоинт хранит + RAW-ответ, resume пере-режет. +8. ✅ **Re-point (главная правка):** OK-путь draft `final_hash` → `tm-banknote-v1:` derived-чекпоинт с ОЧИЩЕННЫМ + черновиком (`commitBanknoteExport`, формула §4б `sha256("tm-banknote-v1\x00"+reqHash+"\x00"+stripped)`); стоит + ПОСЛЕ резолва эскалации, потребляет `last.bankStripped` (эскалированный черновик тоже очищен). Пины + `TestBanknoteSliceReTelemetryResume` (final_hash=derived, derived=clean, resume $0) + `TestBanknoteEscalatedDraftStripped`. +9. ✅ **Пост-чек глоссария:** не затронут. +10. ✅ **Телеметрия (лауд):** store v9 (`n_banknote_lines`/`banknote_parse_fail`/`banknote_truncated`) в + retrieval_state; тред draft→translateChunk→persistRetrievalState. +11. ✅ **DeepSeek thinking-ON нетронут** (echoMineViolation). +12. ✅ **Экспорт:** final без сноски автоматически (срез раньше). + +**§4б:** толерантный парсер (Block A) + **finish=stop-only гейт** — кандидаты ПАРСЯТСЯ только при `finish=="stop"`; +срез идёт РЕГАРДЛЕСС (echo/length classify всегда видят clean). Пин `TestBanknoteFinishStopOnlyGate`. + +--- + +## R4 — WS5 чекеры + данные пакета + `export --pairs` (план §5) — ✅ ЯДРО СДАНО + +- ✅ **R4a `tmctl export --pairs`:** `Export(pairs bool)` + `ChunkExport.Source` (omitempty, $0 из манифеста); + дефолт-экспорт target-only (обратно-совместим). CLI `--pairs`. `TestExportPairsIncludesSource`. + Пред-условие §5(д)-FP DC1/DC2 (src↔target). **Read-only проекция — снапшот/wire/деньги не трогает.** +- ✅ **R4b DC1/DC2/DC6 + zh-ru пакет** (`checkers_zh_ru.go`, порт `ws5_checkers_verify.py`): наблюдаемость + (НЕ диспозиция), в `runCheapGates`→retrieval_state; самогейтятся на контенте (0 на не-zh/чистом). **DC2 — + байт-верно референсу** (после ревью-фикса): ok_re-suppression гард (数十万 не фаерит при верном «сотни тысяч») + + case-SENSITIVE inner (референс без re.I). `TestDC1/DC2/DC6` (эмпир-позитивы + non-firing + parity-кейсы). +- ✅ **R4c DC3 gender-инъекция** (§5(б), «фикс Бай Нинбин»): `memoryEntry.gender`←`row.Gender`; + `genderConstraintNote` в `renderEditorConstraintBlock` (male/female жёсткие формы; hidden — мандат безродовых); + ДАННЫЕ пола уже в `MemoryVersion`, РЕНДЕР — в `renderFormatVersion` v2. `TestDC3GenderInjection`. +- **Пакет данных = код-консты, версия `cheapGateVersion`** (та же дисциплина, что существующие cheap-данные): + правка данных = громкий resnapshot. Конфиг-загружаемый per-pair пакет — чище, **резидуал** (§Резидуал). + +**Golden re-capture ×2 (masked-чист, §Golden):** (1) `cheapGateVersion` v2→v3 — единственная payload-правка = +`style_check_version`, wire/вердикты байт-идентичны после hash-маскинга; (2) `renderFormatVersion` v1→v2 — +payload-версия + 7 editor-wire-тел получают gender-note (gender:m терм фикстуры), НИ ОДНОЙ вердикт/final-правки. + +--- + +## R5 — WS6 арм-конфиги (план §6) — ✅ СДАНО + +- ✅ **Армы = КОНФИГ:** `configs/pipeline-arm-mistral.yaml` (editor=mistral-large-2512) / + `pipeline-arm-deepseek-pro.yaml` (editor=deepseek-v4-pro, `few_shot:false`). Оба — копии боевого C1 с + ЕДИНСТВЕННОЙ правкой editor-model → другой `stageSnap.Model` → другой снапшот (арм-механика Block A). draft/ + translator неизменны. +- ✅ **`echo_mine_test` под варианты** (`TestSwapArmConfigs`): каждый арм грузится; editor=арм-модель; НИ ОДИН не + вооружает эхо-мину (deepseek-v4-pro → ReasoningNone, thinking ON, `echoMineViolation` пуст на echo-prone + провайдере); few_shot (deepseek off / mistral nil=ON); **mistral-large-2512 несёт `rate_limit`** (WS6 пред-условие — + свап-арм за rate-guard WS1b). +- ✅ **Атрибуция арма:** editor-model уже в `chunk_status`/checkpoint `model_actual` → export/report несут её + БЕЗ новой кодовой правки (план: «уже в chunk_status»). Пере-прогон армов = отдельные джобы (разные снапшоты). + +--- + +## R1 — волновой ИСПОЛНИТЕЛЬ: аддитив СДАН, драйвер-свитч РЕЗИДУАЛ + +**Санкция оркестратора:** сначала аддитивные под-шаги без каскада; драйвер-свитч — финальный атомарный шаг; +не дотягивает → зелёное дерево + точный резидуал (**запланированный исход**). Сделано: + +- ✅ **Пер-волновой снапшот (`snapshotIDForWave`, §1в — несущее):** рефактор `snapshotID()`→`buildSnapshotID(stages, + memVersion)` (**байт-идентичен** для all-stages+enriched → golden/последовательный драйвер НЕ двигаются). + `snapshotIDForWave(W1)` = translator-стадии + BASE-версия (excl mined); `(W2)` = editor-стадии + ENRICHED. + `waveStages` (партиция по роли), `baseMemoryVersion`. Пин `TestSnapshotIDForWave`: W1≠W2; **mined-approved + добавка двигает ТОЛЬКО W2, W1 стабилен** («переоплата ОДНА» на снапшот-уровне) + `TestWaveStagesPartition`. +- ✅ **W0 sticky-precompute (`wave.go precomputeSticky`, §1б):** sticky-цепочка — кросс-чанковая зависимость + (не пере-считать внутри параллельной волны) → предвычислена в W0 одним проходом (пер-глава reset, `Select`, + advance). **Последовательный драйвер ТЕПЕРЬ ЕЁ КОНСУМИТ** (`bookrun.go` W0-precompute; `translateChunk` + принимает готовый `memSel`, снят inline-`Select` + `activeIDs`-return). **БАЙТ-ИДЕНТИЧНО** — `TestGolden` + зелёный без правок (несущий байт-паритет-пруф). Это ТОЧНЫЙ precompute, который консумит волновой исполнитель. + +**РЕЗИДУАЛ R1 — драйвер-свитч (точный дизайн, `PACK11_REPORT.md §R1` + план §1):** +1. Партиция стадий по волне — **ГОТОВА** (`waveStages`); пер-волновой снапшот-апсерт (два UpsertSnapshot) — + рефактор `bookrun.go` W0-прелюдии (сейчас один `snapshotID`; заменить на `snapshotIDForWave(W1/W2)`, пинить + стадии волны к своему через `EnsureJob(snapshot_Wx)`). +2. W1-dispatch: N воркеров тянут draft-чанки, `runStage(draft)` под `snapshot_W1`, `memSel`=precompute[i] — + **ГОТОВЫ**: precompute, rate-guard (Block A), single-writer деньги (Block A), eager-clients (Block A). +3. W1.5-стоп: пустая mined-дельта → авто-продолжение; иначе `MineBank`→карта-подписи→подпись→`minedToCandidates` + →`ReplaceGlossary`(reseed)→`snapshot_W2` — **компоненты ГОТОВЫ** (R2), нужна оркестровка стопа/резюма. +4. W2-dispatch: N воркеров тянут edit-ЕДИНИЦЫ (`EditUnitID`, Block A), `prev`=конкатенация draft-чекпоинтов + единицы по `final_hash`, СВЕЖИЙ `Select` по тексту ВСЕЙ единицы над enriched-банком, `runStage(edit)` под + `snapshot_W2`. **W2 НИКОГДА не пере-рендерит draft** (несущий инвариант — редактор читает по `final_hash`). +5. **Каскад (почему не в этой сессии):** edit=единица меняет ГРАНУЛЯРНОСТЬ вывода (edit per-UNIT, адресуется + `(chapter, firstChunkIdx, "edit")`) → `ChunkOutcome`-форма + export/status/quality read-models + ~весь + e2e-суит (asserts per-chunk 2-стадии + call-counts) + СТРУКТУРНЫЙ golden-рерайт. Оркестратор явно + санкционировал структурный рерайт «один раз в R1»; аддитив-пруф (golden байт-стабилен) даёт уверенность, + что все несущие R2–R5 компоненты на месте для атомарного свитча в отдельной сессии/окне. +6. kill-9 с N резервами: порт `store/kill9_test.go` на N воркеров (money-инвариант Block A уже single-writer-safe). + +--- + +## Открытые эскалации / решения (флаг оркестратору при лендинге) + +1. **Дефолт-B майнер** — ратифицированный §10-14 дроп pymorphy3-подканалов (spread λ / dst_variants / + is_name_lemma / Палладий-конфирм / alias-R3 / canon-dst). 古月:22 (не 21). Все Палладий-ИНВАРИАНТНЫЕ гарантии + воспроизведены EXACT. dst доставляет банкнота. **Faithful-A (полный pymorphy3-порт) — только при готовой + поддерживаемой Go-морфологии** (не строил). +2. **`banknoteSnap` omitempty** (не безусловный sanitizerSnap-зеркало) — де­виация-с-обоснованием (§R3-п6): + loudness сохранена, off-книги не re-bill-ятся, R3 golden байт-стабилен. **Требует подтверждения при лендинге.** +3. **DC-пакет данных = код-консты** (версия cheapGateVersion), не конфиг-загружаемый per-pair файл — конфиг-форма + резидуал (§Резидуал). Дисциплина resnapshot сохранена. +4. **R4 резидуал (честный, бюджет→R1):** DC7 grade/role-чекер (нужен promote 甲乙丙丁/四代族长 approved — owner-only, + §5-A2 UNVERIFIED); DC3 male/female-ЧЕКЕР (нужна coref-lite, §5(д)-FP-гейт); omission-бэкстоп (лёгкий Go-ru-стеммер + без pymorphy3 + лемма-матч по {dst,decl.forms,aliases}); DC2 word↔word дроби (q4a rule_b1/b2). Дизайны — план §5(б). +5. **R1 драйвер-свитч резидуал** (запланированный исход, санкция оркестратора) — дизайн §R1 выше. +6. **§5(д)-FP-замер DC1/2/3/6** — платный пере-прогон (не в этой сессии); `export --pairs` — пред-условие СДАНО. + ⚠ FP-замер прогонять через **Go-код** (не `ws5_checkers_verify.py`), т.к. чекеры теперь байт-верны референсу, + но лендинг-решение должно отражать что реально шипается. + +--- + +## Структурные переносы +- Новые файлы `internal/pipeline/miner_*.go` (9 шт — новый детектор-пакет, не рефактор), `checkers_zh_ru.go`, + `wave.go` (W0-precompute). Все — новые оси. +- `snapshotID()` → тонкая обёртка над извлечённым `buildSnapshotID(stages, memVersion)` (байт-идентично). +- `translateChunk` сигнатура: `stickyPrev map[...]` → `memSel memorySelection` (готовый из W0); снят + `activeIDs`-return (precompute владеет sticky-advance). Внутренний вызов, вне-скоуп поведение не затронуто. +- `Export()` → `Export(pairs bool)` (6 тест-вызовов обновлены на `Export(false)`). + +## Багфиксы вне плана +- Нет вне-плановых. Правки по ревью: **R2+R3 (4):** derived-namespace `banknote-v1`→`tm-banknote-v1` (§4б EXACT-формула, + отдельная const от parser_version), мёртвая `rank`-карта в `MineBank`, doc-коммент, error-wrap + `commitBanknoteExport`. **R4 (2):** DC2 ok_re-suppression-гард + case-sensitive inner (байт-паритет референсу). + Все верифицированы адверсариально, пофикшены, суит зелёный. + +## Golden re-capture (masked-структурный дифф — обязан быть объясним) +- **R4b (cheapGateVersion v2→v3):** masked-дифф = 4 строки, ЕДИНСТВЕННАя payload-правка `style_check_version`; + wire-тела + вердикт-строки БАЙТ-ИДЕНТИЧНЫ после hash-маскинга (verified скриптом). +- **R4c (renderFormatVersion v1→v2):** masked-дифф = 18 строк: 4 версия-payload + 7×(old/new) editor-wire-тел с + gender-note (gender:m терм). НИ ОДНОЙ вердикт/disposition/final-text правки (Counter: version/gender-note/wire только). +- **R3/R5/R1-аддитив:** golden НЕ двигается (banknote omitempty off; армы — отдельные файлы; snapshot-рефактор + байт-идентичен). СТРУКТУРНЫЙ golden-рерайт — событие драйвер-свитча (R1-residual), не этой сессии. + +## Верификация (исполнением — логи, не слова) +``` +$ go build ./... → OK +$ go vet ./... → OK (чисто) +$ go test -race ./... → ВСЁ ЗЕЛЁНОЕ: + ok cmd/tmctl · config · ledger · llm · pipeline (~28s -race) · store +$ gofmt -l <мои файлы> → пусто (репо несёт ПРЕ-существующий gofmt-долг в не-моих файлах — не трогал) +$ TM_MINER_PARITY=1 go test -run TestMinerFullBookParity + → n=13618 · катастроф {方源:0,蛊:1,蛊师:2,古月:22} · recall@proposed=0.9649 (55/57 GT) — PASS +``` +**Фикстуры плана исполнены:** miner-паритет (Палладий-инвариант EXACT, 古月:22 дефолт-B) · банкнота 12 точек +(5 e2e-пинов) · seed-lint 0 фейл-лаудов · DC-позитивы пойманы · пер-волновой снапшот «переоплата ОДНА» · +W0 sticky байт-паритет (golden) · draft request_hash стабилен (golden). + +## Чекпойнт-ревью (адверсариал-воркфлоу, author≠reviewer, refute-by-default, 3 линзы) +- **R2+R3:** 5 сырых → 4 confirmed (все minor/nit, 1 refuted): `tm-banknote-v1` namespace / мёртвая rank / + doc / error-wrap. **НИ ОДНОГО major correctness/wire/money.** Все пофикшены. +- **R4:** 2 confirmed (оба DC2 reference-divergence): ok_re-supress-гард (MAJOR — реальный FP-дивердж на + ch2/chunk0) + case-sensitivity (MINOR). Оба пофикшены — DC2 байт-верно референсу. +- **Финал (R5 + R1-аддитив + кросс-каттинг инвариантов, 2-мерный воркфлоу + verify):** **0 находок — ЧИСТЫЙ + вердикт.** Обе линзы (R1-аддитив байт-паритет/пер-волновой снапшот/precompute-эквивалентность; R5-армы + + кросс-каттинг: деньги/снапшот-дисциплина/детерминизм/store-v9) вернули пустой список. Ни одного invariant- + break, un-folded wire/verdict-change, byte-parity-break или map-order-в-выводе. `go test -race ./...` зелёный, + golden байт-стабилен, arm/wave-тесты проходят — clean bill. diff --git a/backend/cmd/tmctl/invocation.go b/backend/cmd/tmctl/invocation.go index f503fb9..52100f4 100644 --- a/backend/cmd/tmctl/invocation.go +++ b/backend/cmd/tmctl/invocation.go @@ -17,9 +17,11 @@ import ( type invocation struct { cmd string cfgPath string + seedPath string // seed-lint: the glossary seed YAML to validate (no --config) resnapshot bool asJSON bool asPlaintext bool + asPairs bool // export: include the source column (--pairs) for the DC1/DC2 FP-measure sel pipeline.RedriveSelector } @@ -48,18 +50,27 @@ func parseInvocation(args []string, flagOut io.Writer) (invocation, error) { resnapshot := fs.Bool("resnapshot", false, "re-pin existing jobs to the current config snapshot (пере-перевод оплаченных чанков — явное согласие)") asJSON := fs.Bool("json", false, "status: emit the projection as JSON (stable disposition/flag_reason enums) for CI/IDE") asPlaintext := fs.Bool("plaintext", false, "export: emit the concatenated human text instead of the default stable JSON") + asPairs := fs.Bool("pairs", false, "export: include the source text per chunk (src↔target column for the DC1/DC2 FP-measure, WS5)") chapter := fs.Int("chapter", -1, "redrive: restrict to this chapter (default: any)") chunk := fs.Int("chunk", -1, "redrive: restrict to this chunk index within the chapter (default: any)") reason := fs.String("reason", "", "redrive: restrict to this flag_reason (default: any)") dryRun := fs.Bool("dry-run", false, "redrive: report what would be re-attacked without touching anything") + seed := fs.String("seed", "", "seed-lint: path to the glossary seed YAML to validate ($0, no --config)") if err := fs.Parse(rest); err != nil { return invocation{}, err } + // seed-lint validates a standalone seed YAML — it takes --seed, not --config (no book/store/keys). + if cmd == "seed-lint" { + if *seed == "" { + return invocation{}, fmt.Errorf("--seed is required for seed-lint") + } + return invocation{cmd: cmd, seedPath: *seed}, nil + } if *cfgPath == "" { return invocation{}, fmt.Errorf("--config book.yaml is required") } return invocation{ - cmd: cmd, cfgPath: *cfgPath, resnapshot: *resnapshot, asJSON: *asJSON, asPlaintext: *asPlaintext, + cmd: cmd, cfgPath: *cfgPath, resnapshot: *resnapshot, asJSON: *asJSON, asPlaintext: *asPlaintext, asPairs: *asPairs, sel: pipeline.RedriveSelector{ Chapter: *chapter, ChunkIdx: *chunk, Reason: *reason, DryRun: *dryRun, }, diff --git a/backend/cmd/tmctl/main.go b/backend/cmd/tmctl/main.go index de42964..71bab93 100644 --- a/backend/cmd/tmctl/main.go +++ b/backend/cmd/tmctl/main.go @@ -69,11 +69,13 @@ func run() error { case "status": return status(ctx, inv.cfgPath, inv.asJSON) case "export": - return export(inv.cfgPath, inv.asPlaintext) + return export(inv.cfgPath, inv.asPlaintext, inv.asPairs) case "redrive": return redrive(ctx, inv.cfgPath, inv.resnapshot, inv.sel) + case "seed-lint": + return seedLint(inv.seedPath) default: - return fmt.Errorf("unknown command %q (want translate|report|status|export|redrive)", inv.cmd) + return fmt.Errorf("unknown command %q (want translate|report|status|export|redrive|seed-lint)", inv.cmd) } } @@ -94,6 +96,17 @@ func translate(ctx context.Context, cfgPath string, resnapshot bool) error { }) } +// seedLint validates a glossary seed YAML (a manual seed or the emitted mined delta) through the REAL +// loadGlossarySeed fail-louds + the shared-key collision check (WS3). $0, no store, no provider keys — a +// pre-flight the operator runs on the mined delta before the W1.5 reseed. Prints "OK" on a clean seed. +func seedLint(seedPath string) error { + if err := pipeline.SeedLint(seedPath); err != nil { + return err + } + fmt.Fprintf(os.Stdout, "seed-lint OK: %s is loadable (0 fail-louds, 0 shared-key collisions)\n", seedPath) + return nil +} + func report(cfgPath string) error { // Read-only ($0): no API keys required (D20.4 — a store audit must not demand provider keys). r, err := pipeline.NewReadOnlyRunner(cfgPath, obs.NewLogger()) @@ -125,14 +138,14 @@ func report(cfgPath string) error { // extractor / reader-samples read the SAME bytes the backend ships instead of the raw checkpoint. Like // report/status it is $0 and needs no provider keys (D20.4 — an audit surface must not demand keys). // Default output is stable JSON (the machine surface); --plaintext emits the human concatenation. -func export(cfgPath string, asPlaintext bool) error { +func export(cfgPath string, asPlaintext, asPairs bool) error { r, err := pipeline.NewReadOnlyRunner(cfgPath, obs.NewLogger()) if err != nil { return err } defer r.Close() - exp, err := r.Export() + exp, err := r.Export(asPairs) if err != nil { return err } diff --git a/backend/configs/pipeline-arm-deepseek-pro.yaml b/backend/configs/pipeline-arm-deepseek-pro.yaml new file mode 100644 index 0000000..fdac041 --- /dev/null +++ b/backend/configs/pipeline-arm-deepseek-pro.yaml @@ -0,0 +1,76 @@ +# Свап-арм редактора: DeepSeek-v4-pro reasoning (WS6 / D39.9). Копия боевого C1 (pipeline-c1.yaml) с +# ЕДИНСТВЕННЫМ изменением: editor glm-5 → deepseek-v4-pro + few_shot:false. Арм = КОНФИГ (другой +# editor-model → другой snapshot_W2). Атрибуция едет в chunk_status.model_actual → export/report. +# +# ⚠️ ЭХО-МИНА (несущая): deepseek — echo-prone провайдер (thinking-off на CJK → эхо исходника). deepseek- +# v4-pro резолвится в ReasoningNone (нет reasoning-capability → reasoning:off = НО-ОП, thinking остаётся +# ON) → echoMineViolation НЕ срабатывает, thinking-ON держится (эхо-мина НЕ вооружается). Вход редактора — +# русский черновик (не CJK), так что эхо-класс тут вырожден, но thinking-ON держим по дисциплине провайдера. +# +# few_shot:false (D38.4): у reasoning-редактора собственный CoT нарушается рукописными few-shot-примерами +# (exp14 §2а) → ---FEWSHOT---блок editor.md дропается. Фолдится в снапшот (snapshot.go FewShot) — осознанный +# --resnapshot. min_max_tokens 8000-флор deepseek-v4-pro (D24.3) держит thinking внутри бюджета. +core: C1 +version: 1 + +defaults: + max_output_ratio: 2.2 + min_max_tokens: 2048 + +context: + glossary_injection: selective + glossary_token_budget: 800 + cache_ttl: "5m" + +segmentation: + draft_budget_out: 1797 + edit_ceiling_out: 3200 + fertility: + cjk: 1.1978 + other: 0.3852 + +retries: + regenerate_before_escalate: 1 + +stages: + - name: draft + role: translator + model: deepseek-v4-flash + prompts: + zh-ru: ../prompts/translator.md + prompt_version: v1-reflow + temperature: 0.3 + reasoning: "off" + escalate_to: deepseek-v4-pro + - name: edit + role: editor + # СВАП-АРМ: deepseek-v4-pro reasoning-редактор (COGS $0.117 vs glm $0.335 WS6(г); thinking-ON держится + # через ReasoningNone no-op). Пиннут (editor не эскалирует — D12). + model: deepseek-v4-pro + prompts: + zh-ru: ../prompts/editor.md + prompt_version: v3-discourse-reflow + temperature: 0.4 + reasoning: "off" # NO-OP на deepseek-v4-pro (ReasoningNone) → thinking ON; НЕ вооружает эхо-мину + few_shot: false # reasoning-редактор: собственный CoT vs рукописные примеры (exp14 §2а) → дроп ---FEWSHOT--- + +gates: + coverage: + enabled: false + len_ratio_bounds: + zh-ru: [2.2, 4.2] + ja-ru: [1.4, 2.6] + en-ru: [0.70, 1.4] + sent_cov_min: 0.75 + min_chunk_chars: 500 + sanitizer: + enabled: true + +escalation: + chains: + default: [deepseek-v4-pro, glm-5.1, gemini-3.1-pro-preview] + adult: [grok-4.3] + budget_usd: 0 + +fanout: + candidates: 1 diff --git a/backend/configs/pipeline-arm-mistral.yaml b/backend/configs/pipeline-arm-mistral.yaml new file mode 100644 index 0000000..b177d33 --- /dev/null +++ b/backend/configs/pipeline-arm-mistral.yaml @@ -0,0 +1,74 @@ +# Свап-арм редактора: Mistral (WS6 / D39.9 — слабое звено = редактор). Копия боевого C1 +# (pipeline-c1.yaml) с ЕДИНСТВЕННЫМ изменением: editor.model glm-5 → mistral-large-2512. Арм = КОНФИГ, +# не код: другой editor-model → другой stageSnap.Model → другой snapshot_W2 (отдельный прогон, не +# конфликтует с базлайном). Атрибуция арма едет в chunk_status.model_actual → export/report. +# +# ⚠️ ПРЕД-УСЛОВИЕ (WS1 §1б / ревью-2 F4): mistral-large — агрессивный rate-limiter (~48% retry-fails под +# N-∥ без guard'а — 00-provider-quirks §Транспорт). models.yaml несёт rate_limit:{max_concurrency:2} на +# mistral-large-2512; пер-модельный семафор волнового раннера (WS1b, сдан Block A) реально ограничивает +# конкуренцию. Свап-арм mistral НЕ идёт через прод-путь без этого guard. +core: C1 +version: 1 + +defaults: + max_output_ratio: 2.2 + min_max_tokens: 2048 + +context: + glossary_injection: selective + glossary_token_budget: 800 + cache_ttl: "5m" + +# WS2 (слой 1): бюджет нарезки в ВЫХОДНЫХ (ru) токенах — те же ратифицированные zh-ru дефолты, что C1. +segmentation: + draft_budget_out: 1797 + edit_ceiling_out: 3200 + fertility: + cjk: 1.1978 + other: 0.3852 + +retries: + regenerate_before_escalate: 1 + +stages: + - name: draft + role: translator + model: deepseek-v4-flash # черновик — тот же, что базлайн (свап только редактора) + prompts: + zh-ru: ../prompts/translator.md + prompt_version: v1-reflow + temperature: 0.3 + reasoning: "off" + escalate_to: deepseek-v4-pro + - name: edit + role: editor + # СВАП-АРМ: mistral-large-2512 вместо glm-5 (D39.9 — редактор = слабое звено; COGS $0.205 vs glm $0.335 + # WS6(г); не reasoning-модель → эхо-мины нет). Идёт ТОЛЬКО за rate-guard (models.yaml rate_limit). + model: mistral-large-2512 + prompts: + zh-ru: ../prompts/editor.md + prompt_version: v3-discourse-reflow + temperature: 0.4 + reasoning: "off" + # few_shot по дефолту ON: mistral НЕ reasoning-модель, few-shot-примеры P1a-дискурса не мешают её CoT. + +gates: + coverage: + enabled: false + len_ratio_bounds: + zh-ru: [2.2, 4.2] + ja-ru: [1.4, 2.6] + en-ru: [0.70, 1.4] + sent_cov_min: 0.75 + min_chunk_chars: 500 + sanitizer: + enabled: true # премиум-редактор за санитайзером — как C1 (утечки преамбул ловятся) + +escalation: + chains: + default: [deepseek-v4-pro, glm-5.1, gemini-3.1-pro-preview] + adult: [grok-4.3] + budget_usd: 0 + +fanout: + candidates: 1 diff --git a/backend/internal/config/echo_mine_test.go b/backend/internal/config/echo_mine_test.go index a4b3f70..9070369 100644 --- a/backend/internal/config/echo_mine_test.go +++ b/backend/internal/config/echo_mine_test.go @@ -213,3 +213,63 @@ func TestBoevoyConfigEditorGlm5AndGrokReasoning(t *testing.T) { } } } + +// TestSwapArmConfigs guards the WS6 editor swap-arms (D39.9 — редактор = слабое звено). The arms are +// CONFIG, not code: each pipeline-arm-*.yaml swaps ONLY the editor model, so a different editor-model +// resolves a different stageSnap.Model → a distinct snapshot_W2 (a separate run). This pins: (1) each arm +// loads and its editor is the arm model; (2) NEITHER arm re-arms the echo mine — the deepseek-pro editor +// resolves to ReasoningNone (thinking ON, off is a no-op) so echoMineViolation stays empty on an echo- +// prone provider; (3) the deepseek reasoning-editor drops few-shot (few_shot:false), the mistral arm +// keeps it (nil = ON); (4) the mistral arm's PRE-CONDITION — mistral-large-2512 carries a rate_limit so +// the WS1 per-model rate-guard actually throttles it (§6(б) F4). All three boevoy pipelines (c1 + two +// arms) still pin the same translator+draft; only the editor differs. +func TestSwapArmConfigs(t *testing.T) { + m, err := LoadModels(filepath.Join("..", "..", "configs", "models.yaml")) + if err != nil { + t.Fatalf("load boevoy models.yaml: %v", err) + } + arms := []struct { + file, editorModel string + fewShotOff bool // the deepseek reasoning editor drops few-shot; mistral keeps it + }{ + {"pipeline-arm-mistral.yaml", "mistral-large-2512", false}, + {"pipeline-arm-deepseek-pro.yaml", "deepseek-v4-pro", true}, + } + for _, a := range arms { + t.Run(a.file, func(t *testing.T) { + p, err := LoadPipeline(filepath.Join("..", "..", "configs", a.file), m) + if err != nil { + t.Fatalf("load %s: %v", a.file, err) + } + var edit *Stage + for i := range p.Stages { + if p.Stages[i].Role == "editor" { + edit = &p.Stages[i] + } + } + if edit == nil { + t.Fatalf("%s has no editor stage", a.file) + } + if edit.Model != a.editorModel { + t.Errorf("%s editor model = %q, want the swap-arm %q", a.file, edit.Model, a.editorModel) + } + // (2) echo-mine safety: the arm editor's model must not re-arm the mine. + if why := m.echoMineViolation(edit.Model); why != "" { + t.Errorf("%s editor %q re-arms the echo mine: %s", a.file, edit.Model, why) + } + // (3) few-shot policy: deepseek reasoning-editor drops the block; mistral keeps it (nil = ON). + if a.fewShotOff { + if edit.FewShot == nil || *edit.FewShot { + t.Errorf("%s deepseek reasoning-editor must set few_shot:false (drop hand examples), got %v", a.file, edit.FewShot) + } + } else if edit.FewShot != nil { + t.Errorf("%s mistral editor must keep few_shot default (nil=ON), got %v", a.file, edit.FewShot) + } + }) + } + // (4) mistral rate-guard pre-condition: mistral-large-2512 must carry a rate_limit so the WS1 wave + // per-model semaphore actually throttles it (mistral ~48% retry-fails under N-∥ without it). + if rl := m.Models["mistral-large-2512"].RateLimit; rl.MaxConcurrency <= 0 { + t.Errorf("mistral-large-2512 must configure rate_limit.max_concurrency>0 (WS6 pre-condition — the swap-arm needs the WS1 rate-guard), got %d", rl.MaxConcurrency) + } +} diff --git a/backend/internal/config/pipeline.go b/backend/internal/config/pipeline.go index 93553a0..690a533 100644 --- a/backend/internal/config/pipeline.go +++ b/backend/internal/config/pipeline.go @@ -137,6 +137,21 @@ type Gates struct { Glossary GlossaryGate `yaml:"glossary"` Sanitizer SanitizerGate `yaml:"sanitizer"` RegressionGuard RegressionGuardGate `yaml:"regression_guard"` + Banknote BanknoteGate `yaml:"banknote"` +} + +// BanknoteGate controls the banknote-v1 in-band footnote channel (WS4, RATIFIED D39.10): when enabled, +// the translator MAY emit a versioned ⟦TM-BANK-v1⟧ separator + tab-delimited term lines for NEW terms +// after the translation — the DIRECT dst delivery the co-occurrence miner could not extract (蛊→гу). +// Enabling it is TWO coordinated changes: this backend gate (the runner slices the block off BEFORE the +// gates/editor, commits the cleaned draft as a derived export checkpoint, and folds banknoteSnap into the +// snapshot) AND the footnote INSTRUCTION baked into the translator prompt file (which moves PromptSHA256). +// Opt-in (default false), like the sanitizer/coverage gates; a no-op unless the prompt actually instructs +// the model to emit banknotes (a normal draft has no separator → the slice is a no-op). Its parser/slice +// VERSION is folded into banknoteSnap (verdict-axis) only when enabled, so a parser change is a loud +// --resnapshot even without a prompt edit (§4в point 6). +type BanknoteGate struct { + Enabled bool `yaml:"enabled"` } // RegressionGuardGate controls the post-reflow regression guard (D38 infra-pack, diff --git a/backend/internal/pipeline/banknote.go b/backend/internal/pipeline/banknote.go index c0af0ca..c3e5192 100644 --- a/backend/internal/pipeline/banknote.go +++ b/backend/internal/pipeline/banknote.go @@ -1,9 +1,15 @@ package pipeline import ( + "crypto/sha256" + "encoding/hex" + "fmt" "regexp" "strings" "unicode" + + "textmachine/backend/internal/config" + "textmachine/backend/internal/store" ) // banknote.go: the banknote-v1 in-band footnote channel (WS4, план §4 / research/20 §B3 / exp16 @@ -30,6 +36,12 @@ const bankMaxLines = 12 // re-resolves the stripped draft, so it must be a loud --resnapshot even without a prompt edit. const bankParserVersion = "banknote-v1" +// bankDerivedNS is the derived-checkpoint id NAMESPACE (§4б EXACT formula) — the "tm--v1" prefix +// convention the sibling commitSanitizedExport uses ("tm-sanitized-v1"), kept DISTINCT from the parser +// version above exactly as sanitizerSnap.Version (sanitizer-v6) is distinct from the "tm-sanitized-v1" +// namespace: it embeds the channel version so a version change yields a fresh (never colliding) id. +const bankDerivedNS = "tm-banknote-v1" + // bankTypeOK is the accepted type set; anything else falls back to "term" (a benign default). var bankTypeOK = map[string]bool{"name": true, "place": true, "title": true, "term": true, "nickname": true} @@ -133,3 +145,63 @@ func parseBanknote(block string, truncatedGeneration bool) ([]bankEntry, bankFla flags.ParseFail = bad > 0 return entries, flags } + +// --- integration seam (план §4(а) 12 points) ---------------------------------------------------- + +// bankTokenBudget is the extra max_tokens the translator draft reserves for the ≤12-line footnote block +// (integration point 5): the translation comes FIRST and the block LAST, so a length cut hits the block +// (tolerated), not the translation. ~12 tokens per line. Added ONLY when the channel is enabled. +const bankTokenBudget = bankMaxLines * 12 + +// applyBanknote slices the banknote block off a translator draft (integration points 1–5, 8). It runs +// ONLY on the translator role with the channel enabled — the editor never emits banknotes, and a normal +// draft has no separator (the slice is then a no-op that returns the raw text unchanged, so a +// banknote-OFF run is byte-identical). When a ⟦TM-BANK-v1⟧ block IS present it returns the CLEANED +// translation (what classify/coverage/echo/editor/export all see — points 2/3/4) plus that same cleaned +// text as the `stripped` export to commit as a derived checkpoint (point 8), and the per-chunk telemetry +// (point 10). Candidates are PARSED/accepted only under finish=="stop" (the finish=stop-only gate §4б — +// a truncated block is not trusted); the slice itself runs regardless, so the length/echo classify never +// sees the footnote's Han src column. Deterministic (a pure function of the raw text + gate state). +func (r *Runner) applyBanknote(role, rawText, finish string) (clean, stripped string, flags bankFlags) { + if role != roleTranslator || !r.Pipeline.Gates.Banknote.Enabled { + return rawText, "", bankFlags{} + } + cleanText, block := splitBanknote(rawText) + if block == "" { + return rawText, "", bankFlags{} // no separator → no strip, no derived checkpoint (byte-identical path) + } + if finish == "stop" { + // finish=stop-only gate: accept candidates + count telemetry only for a complete generation. + // truncatedGeneration=false is unreachable-otherwise in prod (a truncated gen is finish≠stop). + _, flags = parseBanknote(block, false) + } + return cleanText, cleanText, flags +} + +// bankDerivedHash is the CONTENT-ADDRESSED id of a banknote-stripped export checkpoint (§4б EXACT +// formula, mirroring commitSanitizedExport's namespacing): sha256("tm-banknote-v1\x00"+reqHash+"\x00"+ +// stripped), prefixed "tm-banknote-v1:" so it can never collide with a real hex attempt hash and a +// resume re-derives the identical id for free. The namespace embeds the channel version → a version +// change yields a new id. +func bankDerivedHash(reqHash, stripped string) string { + sum := sha256.Sum256([]byte(bankDerivedNS + "\x00" + reqHash + "\x00" + stripped)) + return bankDerivedNS + ":" + hex.EncodeToString(sum[:]) +} + +// commitBanknoteExport persists the banknote-stripped CLEAN draft as a $0 derived checkpoint and returns +// its request_hash (→ chunk_status.final_hash), so the OK-path final_hash→checkpoint export contract +// (and the editor's resume read) yields the CLEANED draft instead of the raw one carrying the footnote +// (integration point 8). Mirrors commitSanitizedExport: cost 0, escalation 0, idempotent, written BEFORE +// chunk_status references it. `att` is the terminal (possibly escalated) OK attempt whose block was +// sliced; att.bankStripped is the cleaned text. +func (r *Runner) commitBanknoteExport(st config.Stage, ch Chunk, job *store.Job, att stageAttempt) (string, error) { + derivedHash := bankDerivedHash(att.reqHash, att.bankStripped) + if err := r.Store.PutDerivedCheckpoint(store.Checkpoint{ + RequestHash: derivedHash, JobID: job.ID, ChunkIdx: ch.ChunkIdx, Attempt: att.attempt, + Stage: st.Name, Role: st.Role, ModelRequested: att.modelActual, ModelActual: att.modelActual, + ResponseText: att.bankStripped, UsageJSON: "{}", FinishReason: "banknote_export", + }); err != nil { + return "", fmt.Errorf("pipeline: commit banknote export ch%d/chunk%d/%s: %w", ch.Chapter, ch.ChunkIdx, st.Name, err) + } + return derivedHash, nil +} diff --git a/backend/internal/pipeline/banknote_integration_test.go b/backend/internal/pipeline/banknote_integration_test.go new file mode 100644 index 0000000..261bbf9 --- /dev/null +++ b/backend/internal/pipeline/banknote_integration_test.go @@ -0,0 +1,240 @@ +package pipeline + +import ( + "context" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" +) + +// banknote_integration_test.go: the WS4 (д) end-to-end pins for the banknote channel wired into the +// runner (banknote ON), covering the load-bearing integration points §4(а): the block is SLICED before +// the editor/gates (points 1-4), the cleaned draft is committed as a derived export and final_hash is +// re-pointed (point 8), resume re-serves the cleaned draft at $0 (point 7), the echo/length classify +// runs over the CLEANED text (point 3), the telemetry is persisted (point 10), and banknoteSnap folds +// into the snapshot only when enabled (point 6 / §4в). The banknote-OFF golden fixture stays byte-stable +// (the TestGolden guard), so these ON-only cells live here rather than in the shared golden. + +// bankBlock is a well-formed 2-line banknote block a translator emits after a clean RU translation. +const bankBlock = bankSeparator + "\n龙公\tЛун Гун\tname\n蛊\tгу\tterm" + +func TestBanknoteSliceReTelemetryResume(t *testing.T) { + rec := &reqRec{} + const cleanDraft = "Тихое утро в библиотеке." + srv := newJSONProvider(rec, func(body string) (string, string) { + if isEditBody(body) { + // The editor MUST receive the CLEANED draft — the block was sliced off upstream (point 8). + if strings.Contains(body, bankSeparator) || strings.Contains(body, "龙公") { + t.Errorf("editor request leaked the banknote block:\n%s", body) + } + return "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД", "stop" + } + return cleanDraft + "\n" + bankBlock, "stop" + }) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{banknote: true}) + ctx := context.Background() + + r1 := newRunner(t, bookPath) + res1, err := r1.TranslateBook(ctx) + if err != nil { + t.Fatal(err) + } + ch := res1.Chunks[0] + // The draft stage's usable text is the CLEANED translation (block removed, points 1-3). + if got := ch.Stages[0].Text; got != cleanDraft { + t.Fatalf("draft text must be the cleaned translation %q, got %q", cleanDraft, got) + } + if ch.FinalText != "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД" { + t.Fatalf("final text must come from the edit over the cleaned draft, got %q", ch.FinalText) + } + // The draft chunk_status.final_hash must be RE-POINTED at the tm-banknote-v1 derived checkpoint (point 8). + cs, _ := r1.Store.GetChunkStatus("test-book", 1, 0, "draft") + if cs == nil || !strings.HasPrefix(cs.FinalHash, bankDerivedNS+":") { + t.Fatalf("draft final_hash must be a tm-banknote-v1 derived checkpoint, got %+v", cs) + } + // The derived checkpoint holds the CLEANED draft (so export/resume yield it, not the raw block). + dcp, err := r1.Store.GetCheckpoint(cs.FinalHash) + if err != nil || dcp == nil { + t.Fatalf("derived checkpoint missing: %v", err) + } + if dcp.ResponseText != cleanDraft || dcp.FinishReason != "banknote_export" { + t.Fatalf("derived checkpoint = %q/%q, want cleaned draft / banknote_export", dcp.ResponseText, dcp.FinishReason) + } + // Telemetry (point 10): 2 accepted lines, no parse fail / truncation. + rs, _ := r1.Store.GetRetrievalState("test-book", 1, 0) + if rs == nil || rs.NBanknoteLines != 2 || rs.BanknoteParseFail != 0 || rs.BanknoteTruncated != 0 { + t.Fatalf("banknote telemetry wrong: %+v", rs) + } + callsAfterRun1 := rec.count() + r1.Close() + + // Resume (new process): $0, NO new provider calls, cleaned draft re-served identically (point 7). + r2 := newRunner(t, bookPath) + defer r2.Close() + res2, err := r2.TranslateBook(ctx) + if err != nil { + t.Fatal(err) + } + if rec.count() != callsAfterRun1 { + t.Fatalf("resume must not re-call the provider, calls %d -> %d", callsAfterRun1, rec.count()) + } + if res2.TotalUSD != 0 || res2.Chunks[0].FinalText != res1.Chunks[0].FinalText { + t.Fatalf("resume must be $0 and byte-identical, got %+v", res2.Chunks[0]) + } +} + +// A clean RU translation followed by a Han-heavy banknote block must classify OK — the echo/CJK check +// runs over the CLEANED text (point 3). WITHOUT the slice, the block's Han src column would push +// cjk_share over the 15% threshold and false-flag cjk_artifact. +func TestBanknoteClassifyOverCleaned(t *testing.T) { + rec := &reqRec{} + // A short RU translation + a LONG Han-heavy block: raw cjk_share > 0.15, cleaned cjk_share ≈ 0. + bigBlock := bankSeparator + "\n" + + strings.Repeat("龙公\tЛун Гун\tname\n古月\tГу Юэ\tname\n方源\tФан Юань\tname\n", 3) + srv := newJSONProvider(rec, func(body string) (string, string) { + if isEditBody(body) { + return "ред", "stop" + } + return "Он ушёл." + "\n" + bigBlock, "stop" + }) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{banknote: true}) + + r := newRunner(t, bookPath) + defer r.Close() + res, err := r.TranslateBook(context.Background()) + if err != nil { + t.Fatal(err) + } + ch := res.Chunks[0] + if ch.Stages[0].Disposition != DispOK { + t.Fatalf("a clean RU draft + Han banknote block must classify OK over the cleaned text, got %+v", ch.Stages[0]) + } + if ch.Stages[0].Text != "Он ушёл." { + t.Fatalf("draft text must be the cleaned translation, got %q", ch.Stages[0].Text) + } +} + +// Enabling the banknote channel folds banknoteSnap into the snapshot (point 6 / §4в); a banknote-OFF +// project renders a DIFFERENT snapshot (so a channel flip is a loud --resnapshot). Off omits the field +// (omitempty), on adds banknote:{enabled,parser_version}. +func TestBanknoteSnapshotFold(t *testing.T) { + srv := httptest.NewServer(nil) + defer srv.Close() + off := newRunner(t, setupProjectOpts(t, srv.URL, projectOpts{})) + defer off.Close() + on := newRunner(t, setupProjectOpts(t, srv.URL, projectOpts{banknote: true})) + defer on.Close() + + offID, offPayload, err := off.snapshotID() + if err != nil { + t.Fatal(err) + } + onID, onPayload, err := on.snapshotID() + if err != nil { + t.Fatal(err) + } + if offID == onID { + t.Fatalf("enabling the banknote channel must move the snapshot (loud --resnapshot)") + } + if strings.Contains(offPayload, "banknote") { + t.Fatalf("a banknote-OFF snapshot must omit the field (omitempty), got: %s", offPayload) + } + if !strings.Contains(onPayload, `"banknote":{"enabled":true,"parser_version":"`+bankParserVersion+`"}`) { + t.Fatalf("a banknote-ON snapshot must fold {enabled,parser_version}, got: %s", onPayload) + } +} + +// A truncated banknote generation (finish=length) is a length flag (retryable) — the block is not parsed +// (finish=stop-only gate §4б), and with the tolerant parser held for robustness it never counts a parse +// fail. Here a channel-off run is unaffected; this pins that the stop-only gate does not accept a +// truncated block's candidates. With regenerate=0 the length draft flags immediately. +func TestBanknoteFinishStopOnlyGate(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, func(body string) (string, string) { + if isEditBody(body) { + return "ред", "stop" + } + // A truncated generation: clean text + a partial block, finish=length. + return "Незаконченный перевод" + "\n" + bankSeparator + "\n龙公\tЛун", "length" + }) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{banknote: true, regenerate: 0}) + + r := newRunner(t, bookPath) + defer r.Close() + res, err := r.TranslateBook(context.Background()) + if err != nil { + t.Fatal(err) + } + ch := res.Chunks[0] + // finish=length → flagged length (the block is not accepted; the translation is retried in prod). + if ch.Stages[0].Disposition != DispFlagged || ch.Stages[0].FlagReason != FlagLength { + t.Fatalf("a truncated generation must flag length (stop-only gate rejects the block), got %+v", ch.Stages[0]) + } + // No candidates accepted under a non-stop finish → zero telemetry. + rs, _ := r.Store.GetRetrievalState("test-book", 1, 0) + if rs != nil && rs.NBanknoteLines != 0 { + t.Fatalf("a non-stop generation must accept 0 banknote lines, got %+v", rs) + } +} + +// The banknote slice + derived-commit + re-point must run AFTER the escalation resolve and consume the +// ESCALATED draft's stripped text (WS4 point 8's escalation-ordering note): the primary echoes CJK +// (cjk_artifact) → escalates → the fallback returns a clean RU translation + a banknote block, which is +// sliced and committed as a derived checkpoint keyed on the ESCALATED attempt's request_hash. +func TestBanknoteEscalatedDraftStripped(t *testing.T) { + rec := &reqRec{} + const fbClean = "Тихое утро в библиотеке." + srv := newJSONProvider(rec, func(body string) (string, string) { + if isEditBody(body) { + if strings.Contains(body, bankSeparator) { + t.Errorf("editor leaked the escalated banknote block:\n%s", body) + } + return "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД", "stop" + } + if strings.Contains(body, "fake-fallback") { + return fbClean + "\n" + bankBlock, "stop" // the fallback emits a clean translation + banknote + } + return "静かな図書館の朝。", "stop" // the primary echoes the CJK source → cjk_artifact + }) + defer srv.Close() + bookPath := setupEscalationProject(t, srv.URL, 1.0, false, "") + pipePath := filepath.Join(filepath.Dir(bookPath), "pipeline.yaml") + raw, err := os.ReadFile(pipePath) + if err != nil { + t.Fatal(err) + } + writeFile(t, pipePath, string(raw)+"\ngates:\n banknote:\n enabled: true\n") + + r := newRunner(t, bookPath) + defer r.Close() + res, err := r.TranslateBook(context.Background()) + if err != nil { + t.Fatal(err) + } + ch := res.Chunks[0] + draft := ch.Stages[0] + if !draft.Escalated || draft.EscalationModel != "fake-fallback" { + t.Fatalf("the draft must have escalated to fake-fallback, got %+v", draft) + } + // The escalated draft's usable text is the CLEANED fallback translation (block sliced). + if draft.Text != fbClean { + t.Fatalf("escalated draft text must be the cleaned fallback translation %q, got %q", fbClean, draft.Text) + } + // final_hash re-pointed at a tm-banknote-v1 derived checkpoint holding the cleaned ESCALATED draft. + cs, _ := r.Store.GetChunkStatus("test-book", 1, 0, "draft") + if cs == nil || !strings.HasPrefix(cs.FinalHash, bankDerivedNS+":") { + t.Fatalf("escalated draft final_hash must be a derived checkpoint, got %+v", cs) + } + dcp, _ := r.Store.GetCheckpoint(cs.FinalHash) + if dcp == nil || dcp.ResponseText != fbClean { + t.Fatalf("derived checkpoint must hold the cleaned escalated draft, got %+v", dcp) + } + if ch.FinalText != "ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД" { + t.Fatalf("final text must come from the edit over the cleaned escalated draft, got %q", ch.FinalText) + } +} diff --git a/backend/internal/pipeline/bookrun.go b/backend/internal/pipeline/bookrun.go index d7fefcb..b42690e 100644 --- a/backend/internal/pipeline/bookrun.go +++ b/backend/internal/pipeline/bookrun.go @@ -36,6 +36,9 @@ type StageResult struct { // fallback passed the re-gate, Model above is the fallback (it answered). Escalated bool EscalationModel string // the fallback model when Escalated ("" otherwise) + // BankFlags carries the translator draft's banknote telemetry (WS4 point 10): accepted line count + + // parse-fail / truncation flags. Zero-valued on every non-translator stage and every channel-off run. + BankFlags bankFlags } // ChunkOutcome is one chunk's result across the stage list. @@ -142,22 +145,23 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) { "chapters", len(doc.Chapters), "chunks", len(chunks), "stages_per_chunk", len(r.Pipeline.Stages)) res := &BookResult{BookID: r.Book.BookID} - // Sticky scene-inertia state (A5): the exact-matched ids of the recent chunks in - // the CURRENT chapter, reset at a chapter boundary (a new chapter is a scene change). - // Rebuilt deterministically each run because translateChunk recomputes the ($0) - // selection for EVERY chunk, resumed ones included — so sticky is resume-stable. - var stickyWin []map[string]injectionDisposition + // W0: pre-compute the sticky-chain memory selection for EVERY chunk (WS1 §1б, precomputeSticky). The + // sticky window (A5 scene-inertia — the recent chunks' exact matches in the current chapter, reset at + // a chapter boundary) is a CROSS-CHUNK sequential dependency, so it is computed once here in W0 and + // consumed per-chunk below. This is byte-identical to the old inline computation (Select is $0 and + // pure; the golden test proves the injected bytes did not move) — and it is the exact precompute the + // parallel wave executor will consume (the cross-chunk state cannot be recomputed inside a wave). + stickySel := precomputeSticky(chunks, r.memory, r.Pipeline.Context.GlossaryTokenBudget) prevChapter := 0 for i, ch := range chunks { if ch.Chapter != prevChapter { - stickyWin = nil prevChapter = ch.Chapter // Пульс прогресса на границе главы: done/total + деньги ЭТОГО прогона — // оператор длинной книги видит движение, не открывая status. r.Log.InfoContext(ctx, "chapter started", "chapter", ch.Chapter, "chunks_done", i, "chunks_total", len(chunks), "run_usd", fmt.Sprintf("%.6f", res.TotalUSD)) } - outcome, activeIDs, err := r.translateChunk(ctx, snapID, ch, unionSticky(stickyWin)) + outcome, err := r.translateChunk(ctx, snapID, ch, stickySel[i]) if err != nil { // Infra failure: abort. Chunks already committed are checkpointed; // resume continues from here at $0 for the done work. @@ -168,11 +172,6 @@ func (r *Runner) TranslateBook(ctx context.Context) (*BookResult, error) { if outcome.Disposition == DispFlagged { res.Flagged++ } - // Advance the sticky window (keep the last stickyDepth chunks' exact matches). - stickyWin = append(stickyWin, activeIDs) - if len(stickyWin) > stickyDepth { - stickyWin = stickyWin[len(stickyWin)-stickyDepth:] - } } r.Log.InfoContext(ctx, "book run finished", "book", r.Book.BookID, "chunks", len(res.Chunks), "flagged", res.Flagged, "run_usd", fmt.Sprintf("%.6f", res.TotalUSD)) diff --git a/backend/internal/pipeline/cheapgates.go b/backend/internal/pipeline/cheapgates.go index 3658c69..ca58aaf 100644 --- a/backend/internal/pipeline/cheapgates.go +++ b/backend/internal/pipeline/cheapgates.go @@ -38,7 +38,12 @@ import ( // false-flagged against a STRAY output integer (a year, a count) — only a mismatching magnitude // WORD triggers, bare integers can merely confirm; (3) same mechanism covers a 万-in-a-name + // unrelated output number. No live book has run under v1 (D18 acceptance pending), so the re-pin is free. -const cheapGateVersion = "cheapgate-v2" +// +// v3 (WS5, R4) adds the defect-class checkers DC1 (时辰 double-hour units), DC2 (千万/数十万 magnitude +// scale) and DC6 (register negative-list — the zh-ru pack, checkers_zh_ru.go). They are observability +// (never a disposition), tuned precision-over-recall; the bump is a loud --resnapshot as the discipline +// requires (a rule/pack edit shifts recorded counts). They fire 0 on a non-zh source / non-register text. +const cheapGateVersion = "cheapgate-v3-dc-checkers" // cheapGateConfig carries the brief-derived knobs: the ё-policy and the per-project allowlist of // surfaces that look like a blocklisted interjection but are legitimate here (e.g. a character @@ -55,20 +60,26 @@ type cheapGateConfig struct { // cheapGateResult is the per-chunk outcome: a count per flagger plus human-readable detail lines // (deterministic order) for the report. total() is what the passport surfaces. type cheapGateResult struct { - DialogueDash int `json:"dialogue_dash,omitempty"` - YoInconsistent int `json:"yo,omitempty"` - TranslitInterj int `json:"translit_interj,omitempty"` - NumberMagnitude int `json:"number_magnitude,omitempty"` + DialogueDash int `json:"dialogue_dash,omitempty"` + YoInconsistent int `json:"yo,omitempty"` + TranslitInterj int `json:"translit_interj,omitempty"` + NumberMagnitude int `json:"number_magnitude,omitempty"` // LengthCollapse / NumberDrift are the opt-in post-reflow regression guard (D38, // regressionguard.go), folded into this observability result. They stay 0 unless // cfg.regressionEnabled, so a book that does not enable the guard serialises identically. - LengthCollapse int `json:"length_collapse,omitempty"` - NumberDrift int `json:"number_drift,omitempty"` - Detail []string `json:"detail,omitempty"` + LengthCollapse int `json:"length_collapse,omitempty"` + NumberDrift int `json:"number_drift,omitempty"` + // DC1TimeUnits / DC2Magnitude / DC6Register are the WS5 defect-class checkers (checkers_zh_ru.go), + // observability like the others. Zero on a non-zh source / non-register final (the golden fixture). + DC1TimeUnits int `json:"dc1_time_units,omitempty"` + DC2Magnitude int `json:"dc2_magnitude,omitempty"` + DC6Register int `json:"dc6_register,omitempty"` + Detail []string `json:"detail,omitempty"` } func (c cheapGateResult) total() int { - return c.DialogueDash + c.YoInconsistent + c.TranslitInterj + c.NumberMagnitude + c.LengthCollapse + c.NumberDrift + return c.DialogueDash + c.YoInconsistent + c.TranslitInterj + c.NumberMagnitude + c.LengthCollapse + c.NumberDrift + + c.DC1TimeUnits + c.DC2Magnitude + c.DC6Register } // runCheapGates runs the four always-on style flaggers over one chunk's source and FINAL text, plus @@ -87,6 +98,16 @@ func runCheapGates(source, draft, final string, cfg cheapGateConfig) cheapGateRe n, det = lintNumberMagnitude(source, final) r.NumberMagnitude = n r.Detail = append(r.Detail, det...) + // WS5 defect-class checkers (DC1/DC2/DC6, checkers_zh_ru.go) — src↔target observability flaggers. + n, det = lintTimeUnits(source, final) + r.DC1TimeUnits = n + r.Detail = append(r.Detail, det...) + n, det = lintMagnitudeScale(source, final) + r.DC2Magnitude = n + r.Detail = append(r.Detail, det...) + n, det = lintRegisterLexicon(final) + r.DC6Register = n + r.Detail = append(r.Detail, det...) if cfg.regressionEnabled { rg := runRegressionGuard(draft, final) r.LengthCollapse = rg.LengthCollapse @@ -210,7 +231,7 @@ func quoteShape(after []rune) bool { // needs a ё-dictionary (deferred, B-tier). Names — the primary target (Пётр/Петр) — are never // homographs, so they are always caught regardless. var yoHomographEForms = map[string]bool{ - "все": true, "всех": true, "всем": true, "всеми":true, + "все": true, "всех": true, "всем": true, "всеми": true, "небо": true, "небом": true, "узнаем": true, "узнаете": true, "узнает": true, "падеж": true, "падежа": true, diff --git a/backend/internal/pipeline/checkers_zh_ru.go b/backend/internal/pipeline/checkers_zh_ru.go new file mode 100644 index 0000000..70accf1 --- /dev/null +++ b/backend/internal/pipeline/checkers_zh_ru.go @@ -0,0 +1,168 @@ +package pipeline + +import ( + "fmt" + "regexp" + "sort" + "strconv" + "strings" + "unicode" +) + +// checkers_zh_ru.go: the WS5 defect-class checkers DC1 (时辰 double-hour units), DC2 (千万/数十万 magnitude +// scale) and DC6 (register negative-list) — deterministic, $0 flaggers on the source↔FINAL text, ported +// from the frozen ws5_checkers_verify.py. Like the four cheap style gates they are OBSERVABILITY, NEVER a +// disposition (research/20: judges too noisy → deterministic gates; a hit is recorded in the retrieval- +// state / report, never dropping a chunk). They are tuned PRECISION over recall — DC1/DC2 fire only on an +// EXPLICIT src↔target mismatch; the LANDING (whether the flag is trusted) is gated by a §5(д) false- +// positive measure on a fresh rerun, not by this build. Their rule VERSION rides cheapGateVersion (folded +// into the snapshot as StyleCheckVersion), so editing a rule / the pack is a loud --resnapshot. +// +// PER-PAIR PACK (слой-2 data, §5(в)): the unit table + negative-list below are the zh-ru convention pack. +// They are code consts versioned by cheapGateVersion (the same discipline the existing cheap-gate data +// uses), not a config-loaded per-pair file — a config-loaded versioned pack is a cleaner future form +// (noted residual). The checkers self-gate on source content (时辰/千万 are zh-specific → no fire on a +// non-zh source), and DC6 is target-side (gated behind ru like the other readability flaggers). + +// --- DC-1: 时辰 (double-hour) unit checker (ws5.shichen_checker) ----------------------------------- + +// dcCNNum maps the small CJK numerals the 时辰 pattern accepts to their value (1 时辰 = 2 modern hours). +var dcCNNum = map[rune]int{ + '一': 1, '二': 2, '两': 2, '三': 3, '四': 4, '五': 5, '六': 6, '七': 7, '八': 8, '九': 9, '十': 10, +} + +// dcShichenRE captures the count before 时辰 (an Arabic digit or a small CJK numeral), tolerating 个. +var dcShichenRE = regexp.MustCompile(`([0-9一二两三四五六七八九十])\s*个?\s*时辰`) + +// dcRuHoursRE captures a Russian " час…" rendering; the alternatives mirror the reference map. +var dcRuHoursRE = regexp.MustCompile(`(\d+|один|два|двух|три|трёх|трех|четыре|пять|шесть)\s+час`) + +// dcRuHourWord maps the Russian count words dcRuHoursRE captures to their value. +var dcRuHourWord = map[string]int{ + "один": 1, "два": 2, "двух": 2, "три": 3, "трёх": 3, "трех": 3, "четыре": 4, "пять": 5, "шесть": 6, +} + +// lintTimeUnits flags a 时辰 (=2h) unit error: N个时辰 rendered as N часов (the count copied as hours) +// instead of ~2N hours (三个时辰 → «три часа» should be ~6h). It fires ONLY on an explicit mismatch — a +// paraphrase with no hours count is a valid rendering, not a defect (the reference dropped the "no hours +// found" branch that false-flagged 1.8%). Pure and deterministic. +func lintTimeUnits(source, final string) (int, []string) { + m := dcShichenRE.FindStringSubmatch(source) + if m == nil { + return 0, nil + } + n, ok := dcParseCount(m[1]) + if !ok { + return 0, nil + } + hm := dcRuHoursRE.FindStringSubmatch(final) + if hm == nil { + return 0, nil // no explicit hours rendering → a valid paraphrase, not a defect + } + ruNum, ok := dcParseRuHours(hm[1]) + if !ok { + return 0, nil + } + expectedHours := n * 2 + if ruNum == n && ruNum != expectedHours { + return 1, []string{fmt.Sprintf("DC1 时辰: %d个时辰 передано как «%d час…» (счёт как часы) вместо ~%d ч (1 时辰 = 2 ч)", n, ruNum, expectedHours)} + } + return 0, nil +} + +// dcParseCount parses the DC1 count group: an Arabic digit string or a single small CJK numeral. +func dcParseCount(s string) (int, bool) { + if v, err := strconv.Atoi(s); err == nil { + return v, true + } + r := []rune(s) + if len(r) == 1 { + if v, ok := dcCNNum[r[0]]; ok { + return v, true + } + } + return 0, false +} + +// dcParseRuHours parses the DC1 hours group: an Arabic digit string or a Russian count word. +func dcParseRuHours(s string) (int, bool) { + if v, err := strconv.Atoi(s); err == nil { + return v, true + } + if v, ok := dcRuHourWord[s]; ok { + return v, true + } + return 0, false +} + +// --- DC-2: number-scale magnitude checker (ws5.magnitude_checker, 千万 / 数十万) -------------------- + +// DC2 regexes — a BYTE-FAITHFUL port of ws5.magnitude_checker (_MAG). The ok_re SUPPRESSION guards are +// case-INsensitive (the reference passes re.I); the FIRE predicates are case-SENSITIVE (the reference +// does NOT pass re.I to the inner тысяч/десятки-тысяч searches). \w is ported as \p{L}* (RE2's \w is +// ASCII; the Cyrillic word-continuation the reference intends is letters). +var ( + dcQianwanOkRE = regexp.MustCompile(`(?i)десят\p{L}* миллион|10\s*000\s*000|10000000`) // 千万 correct render → suppress + dcShushiwanOkRE = regexp.MustCompile(`(?i)сотн\p{L}* тысяч|нескольк\p{L}* сот\p{L}* тысяч|[1-9]00\s*000`) // 数十万 correct → suppress + dcDesyatkiTysRE = regexp.MustCompile(`десятк\p{L}* тысяч`) // 数十万 under-render (case-SENSITIVE, per reference) +) + +// lintMagnitudeScale flags a 千万 (10^7) / 数十万 (~several×10^5) magnitude rendered at a WRONG smaller +// scale (a разряд error DC2 targets). 千万 → «тысячи» without «миллион» = 10000× under; 数十万 → «десятки +// тысяч» = 10× under. A CORRECT rendering anywhere in the chunk (the ok_re guard) SUPPRESSES the flag, so +// a chunk that renders 数十万 as «сотни тысяч» does not fire even if «десятки тысяч» appears elsewhere as +// an unrelated quantity (the ws5 reference parity — required so the §5(д) FP-measure taken against the +// frozen reference matches what ships). ⚠ 千万 is also a stock HYPERBOLE («несметно») whose «тысячи» +// rendering is in-register literary, NOT a hard error (§5 A4) — hyperbole-exposed, landing FP-gated §5(д). +// Observability only. (Word↔word fraction inversion — 四成四=44% via q4a rule_b1/b2 — is a residual +// sub-checker, not built here.) +func lintMagnitudeScale(source, final string) (int, []string) { + var flags []string + if strings.Contains(source, "千万") && !dcQianwanOkRE.MatchString(final) { + if strings.Contains(final, "тысяч") && !strings.Contains(final, "миллион") { // case-sensitive, per reference + flags = append(flags, "DC2 千万=10^7 передано как «тысячи» (≈10000× занижение) — возможна ошибка разряда (гипербола-риск, §5-A4)") + } + } + if strings.Contains(source, "数十万") && !dcShushiwanOkRE.MatchString(final) { + if dcDesyatkiTysRE.MatchString(final) { + flags = append(flags, "DC2 数十万≈неск.×10^5 передано как «десятки тысяч» (≈10× занижение)") + } + } + return len(flags), flags +} + +// --- DC-6: register-lexicon negative-list (ws5.register_checker) ---------------------------------- + +// dcRegisterNegList is the zh-ru register negative-list (слой-2 pack): fairy-tale-Russian / chancery +// lexemes that break the xianxia register («терем» in a cultivation novel is a hard register error). The +// owner extends this per corpus finding. Lower-cased; whole-word matched. +var dcRegisterNegList = []string{"терем", "терема", "тереме", "теремом", "терему", "теремах"} + +// lintRegisterLexicon flags whole-word occurrences of a register negative-list lexeme in the FINAL text. +func lintRegisterLexicon(final string) (int, []string) { + low := []rune(strings.ToLower(final)) + hitSet := map[string]bool{} + for _, w := range dcRegisterNegList { + wr := []rune(w) + for i := 0; i+len(wr) <= len(low); i++ { + if !runesEqual(low[i:i+len(wr)], wr) { + continue + } + if (i == 0 || !isCyrLetter(low[i-1])) && (i+len(wr) == len(low) || !isCyrLetter(low[i+len(wr)])) { + hitSet[w] = true + } + } + } + if len(hitSet) == 0 { + return 0, nil + } + hits := make([]string, 0, len(hitSet)) + for w := range hitSet { + hits = append(hits, w) + } + sort.Strings(hits) + return len(hits), []string{"DC6 регистр: сказочно-русская лексика вне жанра сянься: " + strings.Join(hits, ", ")} +} + +// isCyrLetter reports whether r is a Cyrillic letter (the word boundary for the register match). +func isCyrLetter(r rune) bool { return unicode.IsLetter(r) && unicode.Is(unicode.Cyrillic, r) } diff --git a/backend/internal/pipeline/checkers_zh_ru_test.go b/backend/internal/pipeline/checkers_zh_ru_test.go new file mode 100644 index 0000000..d63846a --- /dev/null +++ b/backend/internal/pipeline/checkers_zh_ru_test.go @@ -0,0 +1,107 @@ +package pipeline + +import ( + "strings" + "testing" +) + +// checkers_zh_ru_test.go: WS5 (г) — the DC1/DC2/DC6 checkers must CATCH the empirical trap positives +// (ws5_checkers_verify.py §a) and stay SILENT on clean/in-register text (precision over recall). These +// are the regression fixtures the plan requires (positives from exp15 §7.9 / exp14b). + +func TestDC1TimeUnits(t *testing.T) { + cases := []struct { + name, src, tgt string + wantFlag bool + }{ + {"count-as-hours", "他闭关了三个时辰。", "Он затворился на три часа.", true}, // 三时辰=6h rendered «три часа» + {"correct-conversion", "他闭关了三个时辰。", "Он затворился на шесть часов.", false}, // 6h — correct, no flag + {"paraphrase-no-hours", "他闭关了三个时辰。", "Он затворился надолго.", false}, // no hours count → valid + {"no-shichen", "他走了三里。", "Он прошёл три ли.", false}, + {"arabic-count", "过了2个时辰。", "Прошло 2 часа.", true}, // 2时辰=4h rendered «2 часа» + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + n, _ := lintTimeUnits(c.src, c.tgt) + if (n > 0) != c.wantFlag { + t.Fatalf("lintTimeUnits(%q,%q) fired=%v, want %v", c.src, c.tgt, n > 0, c.wantFlag) + } + }) + } +} + +func TestDC2MagnitudeScale(t *testing.T) { + cases := []struct { + name, src, tgt string + wantFlag bool + }{ + {"qianwan-as-thousands", "有千万条蛊虫。", "Там были тысячи гу-червей.", true}, // 千万=10M as «тысячи» + {"qianwan-correct", "有千万条蛊虫。", "Там были десятки миллионов гу-червей.", false}, // «миллион» present → ok + {"shushiwan-as-tens", "聚集了数十万人。", "Собрались десятки тысяч человек.", true}, // 数十万 as «десятки тысяч» + {"shushiwan-correct", "聚集了数十万人。", "Собрались сотни тысяч человек.", false}, // «сотни тысяч» → ok + // ws5 parity (R4-review MAJOR): a CORRECT rendering «сотни тысяч» SUPPRESSES the flag even when + // «десятки тысяч» appears elsewhere as an unrelated quantity (the ok_re guard). + {"shushiwan-okre-suppress", "聚集了数十万人。", "Здесь сотни тысяч воинов, а не десятки тысяч слуг.", false}, + // ws5 parity (R4-review MINOR): the inner fire check is CASE-SENSITIVE (reference no re.I), so a + // sentence-initial capitalized «Десятки тысяч» does NOT fire. + {"case-sensitive-capitalized", "聚集了数十万人。", "Десятки тысяч человек собрались.", false}, + {"no-magnitude", "他有三个朋友。", "У него три друга.", false}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + n, _ := lintMagnitudeScale(c.src, c.tgt) + if (n > 0) != c.wantFlag { + t.Fatalf("lintMagnitudeScale(%q,%q) fired=%v, want %v", c.src, c.tgt, n > 0, c.wantFlag) + } + }) + } +} + +func TestDC6RegisterLexicon(t *testing.T) { + cases := []struct { + name, tgt string + wantN int + }{ + {"terem", "Он вошёл в высокий терем.", 1}, + {"terem-inflected", "В тереме было тихо, у терема стоял страж.", 2}, // тереме + терема → 2 distinct register forms + {"clean", "Он вошёл в высокий зал павильона.", 0}, + {"substring-not-word", "Термин был странным.", 0}, // «терм» inside «термин» must NOT fire (whole-word) + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + n, det := lintRegisterLexicon(c.tgt) + if n != c.wantN { + t.Fatalf("lintRegisterLexicon(%q) = %d (%v), want %d", c.tgt, n, det, c.wantN) + } + }) + } +} + +// The DC checkers must stay SILENT on a non-zh / clean chunk (they self-gate on content), so a +// non-Chinese book never accrues style flags from them. +func TestDCCheckersSilentOnCleanChunk(t *testing.T) { + r := runCheapGates("静かな図書館の朝。", "", "Тихое утро в библиотеке.", cheapGateConfig{}) + if r.DC1TimeUnits != 0 || r.DC2Magnitude != 0 || r.DC6Register != 0 { + t.Fatalf("DC checkers must be silent on a clean ja→ru chunk, got %+v", r) + } +} + +// TestDC3GenderInjection pins the DC3 gender constraint annotation on the editor block (WS5 §5(б)): a +// gendered CONFIRMED term carries a gender directive; a genderless term does not (byte-identical line). +func TestDC3GenderInjection(t *testing.T) { + mk := func(gender string) []pickedEntry { + return []pickedEntry{{entry: &memoryEntry{src: "方源", dst: "Фан Юань", status: "approved", gender: gender}, via: "方源", disp: memConfirmed}} + } + male := renderEditorConstraintBlock(mk("male")) + if !strings.Contains(male, "方源 → «Фан Юань» (муж.") { + t.Fatalf("male term must carry a masculine directive, got:\n%s", male) + } + hidden := renderEditorConstraintBlock(mk("hidden")) + if !strings.Contains(hidden, "пол СКРЫТ") { + t.Fatalf("hidden term must carry the gender-avoidance mandate, got:\n%s", hidden) + } + none := renderEditorConstraintBlock(mk("")) + if !strings.HasSuffix(none, "«Фан Юань»") { + t.Fatalf("a genderless term must have NO gender note (line ends at the dst), got:\n%s", none) + } +} diff --git a/backend/internal/pipeline/chunkrun.go b/backend/internal/pipeline/chunkrun.go index 54d9350..a4aa378 100644 --- a/backend/internal/pipeline/chunkrun.go +++ b/backend/internal/pipeline/chunkrun.go @@ -87,7 +87,7 @@ func (r *Runner) classifyOutput(role, source, output, finish string, isFinal boo // persists the per-chunk retrieval-state (observability). Returns the chunk outcome, the // exact-matched entity ids (the next chunk's sticky_prev, A5), and an error only on an // infra failure. stickyPrev is the prior chunks' exact matches in this chapter. -func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, stickyPrev map[string]injectionDisposition) (*ChunkOutcome, map[string]injectionDisposition, error) { +func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, memSel memorySelection) (*ChunkOutcome, error) { out := &ChunkOutcome{Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Disposition: DispOK} prev := "" flagged := false @@ -100,19 +100,20 @@ func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, st // draftText is the FIRST (translator) stage's output — the pre-reflow baseline for the opt-in // regression guard's draft→final comparison (== final in a single-stage pipeline). draftText := "" + // bankTelem carries the translator draft's banknote telemetry (WS4 point 10) into the per-chunk + // retrieval-state. Zero for every channel-off / non-banknote chunk. + var bankTelem bankFlags - // Hot path: select the glossary records for this chunk ONCE (deterministic, $0), and serialize the - // per-role injection blocks via the role→renderer registry (D39 слой 7). Recomputed on every run - // (resumed chunks included) so the sticky window and the injected bytes are resume-stable. - var memSel memorySelection + // Hot path: the glossary selection for this chunk is PRE-COMPUTED in W0 (precomputeSticky, WS1 §1б) + // and passed in — the sticky chain is a cross-chunk dependency the parallel waves cannot recompute. + // It is a pure deterministic function of (chunk, sticky_prev, frozen bank), so a resumed chunk + // reproduces the identical injected bytes. Serialize the per-role injection blocks via the + // role→renderer registry (D39 слой 7). injectionByRole := map[string]string{} - activeIDs := map[string]injectionDisposition{} if r.memory != nil { - memSel = r.memory.Select(ch.Text, ch.Chapter, stickyPrev, r.Pipeline.Context.GlossaryTokenBudget) for role, render := range roleInjectionRenderers { injectionByRole[role] = render(memSel.injected) // pure → per-role result is map-order-independent } - activeIDs = memSel.activeIDs // The disposition-gated suppressor refused to let a lower-trust longer key eat a higher-trust // nested one (D39 слой 4): the approved term survives, but surface the seed-hygiene collision // loudly so an operator can reconcile the draft-nesting-over-approved seed (research/13 §7). @@ -132,7 +133,7 @@ func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, st SnapshotID: snapID, Disposition: string(DispSkipped), FlagReason: string(flagReason), Detail: detail, }); err != nil { - return out, activeIDs, fmt.Errorf("pipeline: record skipped chunk_status: %w", err) + return out, fmt.Errorf("pipeline: record skipped chunk_status: %w", err) } out.Stages = append(out.Stages, StageResult{ Stage: st.Name, Role: st.Role, Model: st.Model, @@ -149,10 +150,13 @@ func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, st injection := injectionByRole[st.Role] sr, err := r.runStage(ctx, st, stageIdx, snapID, ch, prev, injection) if err != nil { - return out, activeIDs, err + return out, err } out.Stages = append(out.Stages, *sr) out.CostUSD += sr.CostUSD + if st.Role == roleTranslator { + bankTelem = sr.BankFlags // translator draft's banknote telemetry (WS4 point 10) + } if sr.Disposition == DispFlagged { flagged = true flagReason = sr.FlagReason @@ -211,8 +215,8 @@ func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, st // idempotent, so a resume re-derives the identical row. Only when a glossary is materialized // (always true in a normal run — seedGlossary sets an at-least-empty bank). if r.memory != nil { - if err := r.persistRetrievalState(snapID, ch, memSel, postMisses, outputChecked, cheap); err != nil { - return out, activeIDs, err + if err := r.persistRetrievalState(snapID, ch, memSel, postMisses, outputChecked, cheap, bankTelem); err != nil { + return out, err } } @@ -229,7 +233,7 @@ func (r *Runner) translateChunk(ctx context.Context, snapID string, ch Chunk, st } else { out.FinalText = exportNormalize(prev) } - return out, activeIDs, nil + return out, nil } // injectionRenderer serializes a chunk's selected memory records into a role's injection message. @@ -249,10 +253,13 @@ var roleInjectionRenderers = map[string]injectionRenderer{ // selection + the post-check result. n_exact_hits/n_sticky/n_ambiguous count the INJECTED // records (what the model saw); spoiler/eviction are the dropped-and-logged totals; // post-check misses are recorded only when the translator actually produced text. -func (r *Runner) persistRetrievalState(snapID string, ch Chunk, sel memorySelection, misses []postcheckMiss, outputChecked bool, cheap cheapGateResult) error { +func (r *Runner) persistRetrievalState(snapID string, ch Chunk, sel memorySelection, misses []postcheckMiss, outputChecked bool, cheap cheapGateResult, bank bankFlags) error { rs := store.RetrievalState{ BookID: r.Book.BookID, Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, SnapshotID: snapID, - NStyleFlags: cheap.total(), + NStyleFlags: cheap.total(), + NBanknoteLines: bank.NLines, + BanknoteParseFail: boolToStoreInt(bank.ParseFail), + BanknoteTruncated: boolToStoreInt(bank.Truncated), } if cheap.total() > 0 { if b, err := json.Marshal(cheap); err == nil { @@ -304,6 +311,14 @@ func (r *Runner) persistRetrievalState(snapID string, ch Chunk, sel memorySelect return r.Store.UpsertRetrievalState(rs) } +// boolToStoreInt maps a bool telemetry flag to the store's 0/1 integer column form. +func boolToStoreInt(b bool) int { + if b { + return 1 + } + return 0 +} + // cheapGateConfig builds the cheap-gate knobs from the book brief: the ё-policy and the // lower-cased per-project interjection allowlist. Pure, no store access. func (r *Runner) cheapGateConfig() cheapGateConfig { diff --git a/backend/internal/pipeline/export.go b/backend/internal/pipeline/export.go index 8d88409..c836785 100644 --- a/backend/internal/pipeline/export.go +++ b/backend/internal/pipeline/export.go @@ -35,6 +35,11 @@ type ChunkExport struct { // substantive flag / upstream-skip / pending chunk (exportNormalize("")=="", so the paths coincide // byte-for-byte with ChunkOutcome.FinalText). FinalText string `json:"final_text"` + // Source is the chunk's normalized source text — populated ONLY under `tmctl export --pairs` (WS5 + // §5(д) / research/20 §E-3), the src↔target column the polygon needs to measure the DC1 (时辰) / DC2 + // (千万) source-vs-target checkers' false-positive rate. Omitted by default (the export contract is + // target-only); the source is the $0 manifest re-chunk, never a stored/billed artifact. + Source string `json:"source,omitempty"` } // BookExport is the whole-book export projection (chunks in manifest — (chapter, chunk_idx) — order, so @@ -70,7 +75,7 @@ type BookExport struct { // - MANIFEST (F4): a chunk absent from the current source is a GHOST row (dropped + counted + WARNed); // a manifest chunk with no final row is PENDING; both keep a partial/edited book from exporting as // if complete (the D37-skew the polygon extractor must not inherit). -func (r *Runner) Export() (*BookExport, error) { +func (r *Runner) Export(pairs bool) (*BookExport, error) { statuses, err := r.Store.ChunkStatusesForBook(r.Book.BookID) if err != nil { return nil, err @@ -124,14 +129,21 @@ func (r *Runner) Export() (*BookExport, error) { exp.TotalChunks++ cs, ok := finalRow[k] if !ok { + pend := ChunkExport{Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Disposition: exportPending} + if pairs { + pend.Source = ch.Text + } exp.PendingChunks++ - exp.Chunks = append(exp.Chunks, ChunkExport{Chapter: ch.Chapter, ChunkIdx: ch.ChunkIdx, Disposition: exportPending}) + exp.Chunks = append(exp.Chunks, pend) continue } ce, err := r.chunkExport(cs, gateOn, missByChunk[k]) if err != nil { return nil, err } + if pairs { + ce.Source = ch.Text // the src↔target column for the DC1/DC2 FP-measure (--pairs) + } exp.Chunks = append(exp.Chunks, ce) } // F4: GHOST rows — a stored final row whose chunk is NOT in the current manifest (source shrunk since diff --git a/backend/internal/pipeline/export_pairs_test.go b/backend/internal/pipeline/export_pairs_test.go new file mode 100644 index 0000000..131ca0f --- /dev/null +++ b/backend/internal/pipeline/export_pairs_test.go @@ -0,0 +1,61 @@ +package pipeline + +import ( + "context" + "testing" +) + +// export_pairs_test.go: WS5 `tmctl export --pairs` — the src↔target column the polygon needs for the +// DC1/DC2 false-positive measure (§5(д)). Default export is target-only (Source omitted); --pairs adds +// the $0 manifest source per chunk, matching what the runner chunked. +func TestExportPairsIncludesSource(t *testing.T) { + rec := &reqRec{} + srv := newJSONProvider(rec, draftEdit) + defer srv.Close() + bookPath := setupProjectOpts(t, srv.URL, projectOpts{source: "静かな図書館の朝。"}) + ctx := context.Background() + + r := newRunner(t, bookPath) + defer r.Close() + if _, err := r.TranslateBook(ctx); err != nil { + t.Fatal(err) + } + // Recover the manifest source the runner chunked, to compare against the --pairs column. + manifest, err := r.bookChunks() + if err != nil { + t.Fatal(err) + } + bySource := map[[2]int]string{} + for _, ch := range manifest { + bySource[[2]int{ch.Chapter, ch.ChunkIdx}] = ch.Text + } + + // Default export: Source omitted (target-only contract). + plain, err := r.Export(false) + if err != nil { + t.Fatal(err) + } + for _, ce := range plain.Chunks { + if ce.Source != "" { + t.Fatalf("default export must omit Source, got %q for ch%d/%d", ce.Source, ce.Chapter, ce.ChunkIdx) + } + } + + // --pairs export: Source populated with the exact manifest source per chunk. + pairs, err := r.Export(true) + if err != nil { + t.Fatal(err) + } + if len(pairs.Chunks) == 0 { + t.Fatal("export produced no chunks") + } + for _, ce := range pairs.Chunks { + want := bySource[[2]int{ce.Chapter, ce.ChunkIdx}] + if ce.Source != want { + t.Fatalf("--pairs ch%d/%d Source = %q, want the manifest source %q", ce.Chapter, ce.ChunkIdx, ce.Source, want) + } + if ce.Source == "" { + t.Fatalf("--pairs must populate a non-empty Source for a real chunk (ch%d/%d)", ce.Chapter, ce.ChunkIdx) + } + } +} diff --git a/backend/internal/pipeline/export_test.go b/backend/internal/pipeline/export_test.go index 6bc9739..08047b3 100644 --- a/backend/internal/pipeline/export_test.go +++ b/backend/internal/pipeline/export_test.go @@ -95,7 +95,7 @@ func TestExportMatchesTranslateFinalText(t *testing.T) { assertExportEquals := func(tag string, r *Runner) { t.Helper() - exp, err := r.Export() + exp, err := r.Export(false) if err != nil { t.Fatalf("%s: Export: %v", tag, err) } @@ -165,7 +165,7 @@ func TestExportGlossaryGateWithheld(t *testing.T) { if len(res.Chunks) != 1 || res.Chunks[0].FlagReason != FlagGlossaryMiss || res.Chunks[0].FinalText != "" { t.Fatalf("precondition: want 1 chunk flagged glossary_miss with empty text, got %+v", res.Chunks) } - exp, err := r.Export() + exp, err := r.Export(false) if err != nil { t.Fatalf("Export: %v", err) } @@ -197,7 +197,7 @@ func TestExportManifestPending(t *testing.T) { // The ceiling stop is an infra error — expected; we only need the partial store state it leaves. _, _ = r.TranslateBook(ctx) - exp, err := r.Export() + exp, err := r.Export(false) if err != nil { t.Fatalf("Export: %v", err) } @@ -240,7 +240,7 @@ func TestExportDetectsConfigDrift(t *testing.T) { // Unchanged config → no drift. r2 := newRunner(t, bookPath) - exp, err := r2.Export() + exp, err := r2.Export(false) if err != nil { t.Fatal(err) } @@ -263,7 +263,7 @@ func TestExportDetectsConfigDrift(t *testing.T) { r3 := newRunner(t, bookPath) defer r3.Close() - exp3, err := r3.Export() + exp3, err := r3.Export(false) if err != nil { t.Fatal(err) } @@ -297,7 +297,7 @@ func TestExportFailLoudOnInconsistentStore(t *testing.T) { if err := r.Store.UpsertChunkStatus(*cs); err != nil { t.Fatal(err) } - if _, err := r.Export(); err == nil { + if _, err := r.Export(false); err == nil { t.Fatal("Export must fail loud on a DispOK row with an empty final_hash") } else if !strings.Contains(err.Error(), "empty final_hash") { t.Fatalf("unexpected error: %v", err) diff --git a/backend/internal/pipeline/memory.go b/backend/internal/pipeline/memory.go index c627920..25e96eb 100644 --- a/backend/internal/pipeline/memory.go +++ b/backend/internal/pipeline/memory.go @@ -106,6 +106,7 @@ type memoryEntry struct { dst string // raw approved translation (may be "" for a ruby candidate) status string // auto|draft|approved sense string + gender string // male|female|hidden|"" (C2) — feeds the DC3 gender constraint on the editor block sinceCh int untilCh int // normKeys are the normalized source surfaces (src + aliases) ELIGIBLE to fire @@ -203,6 +204,7 @@ func materializeMemory(rows []store.GlossaryEntry, gateOn bool) *MemoryBank { dst: row.Dst, status: row.Status, sense: row.Sense, + gender: row.Gender, sinceCh: row.SinceCh, untilCh: row.UntilCh, allowShort: row.AllowShort, @@ -526,11 +528,13 @@ func renderGlossaryBlock(injected []pickedEntry) string { // renderFormatVersion versions the FORMAT of the role-injection renderers whose output is NOT // captured by memoryMatchVersion (a scope/matcher-ALGORITHM version, not a render-layout version): -// today the editor constraint block's src→dst layout (WS2 §2в) and, later, the gender-constraint -// render (WS5). A format change shifts the injected bytes → the wire, so it must be a loud -// --resnapshot; folding it as a dedicated snapshot component (snapshot.go) keeps that loudness a -// MECHANISM, not discipline — memoryMatchVersion (const memory.go) would not move on a render edit. -const renderFormatVersion = "renderfmt-v1-editor-src2dst" +// the editor constraint block's src→dst layout (WS2 §2в) and the DC3 gender-constraint annotation +// (WS5 §5(б)). A format change shifts the injected bytes → the wire, so it must be a loud --resnapshot; +// folding it as a dedicated snapshot component (snapshot.go) keeps that loudness a MECHANISM, not +// discipline — memoryMatchVersion (const memory.go) would not move on a render edit. +// v2 (WS5 R4): the editor constraint line now carries a gender annotation for a gendered CONFIRMED +// term (DC3 injection — the Bai Ninbing fix: the injection DIRECTS the editor, no coreference needed). +const renderFormatVersion = "renderfmt-v2-editor-src2dst+dc3-gender" // editorConstraintHeader introduces the editor's canonical-constraint block. It gives the BILINGUAL // editor (D30.1) the approved src→dst bindings as consistency constraints — it must render each @@ -570,7 +574,7 @@ func renderEditorConstraintBlock(injected []pickedEntry) string { continue } seen[key] = true - lines = append(lines, "- "+src+" → «"+dst+"»") + lines = append(lines, "- "+src+" → «"+dst+"»"+genderConstraintNote(p.entry.gender)) } if len(lines) == 0 { return "" @@ -578,6 +582,23 @@ func renderEditorConstraintBlock(injected []pickedEntry) string { return editorConstraintHeader + "\n" + strings.Join(lines, "\n") } +// genderConstraintNote is the DC3 gender directive appended to a CONFIRMED editor-constraint line (WS5 +// §5(б)): the injection DIRECTS the editor to the character's grammatical gender, which is the fix for +// the Bai Ninbing class — no coreference needed. male/female ⇒ hard gender forms; hidden ⇒ a mandate to +// AVOID gender-marking constructions until the reveal (a masculine default when unavoidable, D19.3). "" +// for a term with no gender datum (the common case → the line is unchanged, byte-identical to before). +func genderConstraintNote(gender string) string { + switch gender { + case "male", "m": + return " (муж. — мужские родовые формы)" + case "female", "f": + return " (жен. — женские родовые формы)" + case "hidden": + return " (пол СКРЫТ до раскрытия — избегай родовых форм; при неизбежности — мужские)" + } + return "" +} + // --- Aho-Corasick multi-pattern automaton over runes ---------------------------- type acMatch struct { diff --git a/backend/internal/pipeline/memseed.go b/backend/internal/pipeline/memseed.go index 1c6637a..b1cdfe6 100644 --- a/backend/internal/pipeline/memseed.go +++ b/backend/internal/pipeline/memseed.go @@ -501,6 +501,58 @@ func hasAliasSurface(aliases []store.GlossaryAlias, surface string) bool { return false } +// SeedLint dry-runs the REAL loadGlossarySeed fail-louds + the approved shared-key collision check over +// a glossary seed YAML (a manual seed OR the emitted mined delta), returning nil on a clean seed or a +// loud error listing every problem (WS3 (д): "сухой прогон фейл-лаудов реального loadGlossarySeed по +// дельте"). $0, no store, no LLM — it runs the same loader translate would, so a delta that lints clean +// here is guaranteed loadable, and a shared-key livelock / duplicate / missing-dst is caught BEFORE the +// W1.5 reseed instead of aborting a paid run. +func SeedLint(path string) error { + entries, err := loadGlossarySeed(path) + if err != nil { + return err + } + if problems := approvedSharedKeyCollisions(entries); len(problems) > 0 { + return fmt.Errorf("glossary seed %s shared-key collisions:\n - %s", path, strings.Join(problems, "\n - ")) + } + return nil +} + +// --- mined → candidates (WS3 W1.5 mined-write path) ----------------------------- + +// minedToCandidates converts the miner's WHICH proposals (MineBank) into store candidates for the W1.5 +// reseed (WS3 / §1в mined-write path). It is the DEDICATED mined path the plan requires: it stamps +// Source:"mined" (mirroring rubyToCandidates' Source:"ruby"), NOT the Source:"seed" that loadGlossarySeed +// hardcodes — so a mined row is EXCLUDED from the base-bank-version (BaseVersion) and the W1.5 bank +// enrichment moves ONLY snapshot_W2, keeping W1 checkpoints valid («переоплата ОДНА», §1в/F2). Every +// candidate is status=auto with NO dst (the WHAT is delivered by the banknote/owner sign, WS4): inert on +// the hot path until a human/batch promotes it, never a blind name-lock. A src already covered by the +// manual seed is skipped (the curated entry wins), exactly like rubyToCandidates. Deterministic (MineBank +// emits a src-sorted delta; aliases are re-sorted on read by GlossaryForBook). +func minedToCandidates(mined []MinedTerm, manualSrcs map[string]bool) []store.GlossaryEntry { + var out []store.GlossaryEntry + for _, m := range mined { + if manualSrcs[m.Src] { + continue // the curated seed already owns this src + } + var aliases []store.GlossaryAlias + seen := map[string]bool{} + for _, a := range m.Aliases { + if a == "" || a == m.Src || seen[a] { + continue + } + seen[a] = true + aliases = append(aliases, store.GlossaryAlias{Alias: a, AliasType: "mined"}) + } + out = append(out, store.GlossaryEntry{ + Src: m.Src, Dst: "", Type: m.Type, Status: "auto", Source: "mined", + SinceCh: m.SinceCh, Confidence: m.Freq, Aliases: aliases, + Note: "mined WHICH candidate (miner-v1); dst delivered by banknote/owner sign — promote before use", + }) + } + return out +} + // ruby classifier classes (deterministic HINTS for human promotion). const ( rubyClassName = "name_candidate" // all-Han base + all-kana reading: the furigana-name SHAPE diff --git a/backend/internal/pipeline/miner.go b/backend/internal/pipeline/miner.go new file mode 100644 index 0000000..ab29c41 --- /dev/null +++ b/backend/internal/pipeline/miner.go @@ -0,0 +1,152 @@ +package pipeline + +import "sort" + +// miner.go: the WS3 bank-mining detector orchestration (a Go port of the frozen exp16 arm A3 = V-C, +// LEMMATIZER-DEFAULT-B). Default B drops the pymorphy3-backed sub-channels the ratified §10-14 default +// removes: the spread multiplier (λ=0 already zeroed it), the dst-variants co-occurrence, and the +// Palladius ru-side name confirmation (+56 bonus). The dst (WHAT) is delivered by the banknote (WS4), +// never by co-occurrence, so exact rank is not a product value — the invariant guarantees are the +// candidate SET (membership), recall@proposed, and the catastrophe screen (∈top-50); dropping the +// Palladius sub-channel moves ONLY 古月's exact rank (21→22 vs the pymorphy3 reference), screen still +// PASS. Pure and deterministic ($0, no LLM, no network, no time/rand). + +// MinerConfig is the frozen miner-v1 config (thresholds tuned on chapters 1–15, §D1). Values mirror +// arms.FROZEN; the pymorphy3-dependent knobs (lam, spread_freq_min) are absent by default-B design. +type MinerConfig struct { + FreqFloor int + SubsumeAlpha float64 + FormantMinPartners int + FormantMinOverRep float64 + Bonus map[string]float64 // typ → base bonus (name/place/title/term) + PatternWeight map[string]float64 // evidence-source → weight (max over a candidate's evidence) + SubfloorFreqScale float64 + TopK int + PackVersion string +} + +// frozenMinerConfig is the ratified default-B config (versionable — a threshold edit is a mining-config +// bump the caller folds; a ±50% sweep is invariant, A3@f≥3=1.0 holds by floor/over_rep/subsume/partners). +func frozenMinerConfig() MinerConfig { + return MinerConfig{ + FreqFloor: minerFreqFloor, + SubsumeAlpha: minerSubsumeAlpha, + FormantMinPartners: 3, + FormantMinOverRep: 15.0, + Bonus: map[string]float64{"name": 140.0, "place": 140.0, "title": 120.0, "term": 80.0}, + PatternWeight: map[string]float64{ + "surname": 1.0, "ordinal_title": 1.0, "title_suffix": 0.9, "topo_suffix": 0.9, + "rank_grade": 1.0, "formant_suffix": 0.6, "formant_prefix": 0.5, "title_bare": 0.4, + "palladius": 0.8, // retained for parity of the weight table; the sub-channel is off in default B + }, + SubfloorFreqScale: 40.0, + TopK: 90, + PackVersion: minerPackVersion, + } +} + +// ScoredCand is one candidate after the V-C arm assembly (arms.ScoredCand, minus the default-B-dropped +// dst_variants/spread). types/evidence are in deterministic first-seen order; fromPattern marks a +// pattern-channel contribution. +type ScoredCand struct { + Src string + Score float64 + Freq int + Types []string + Evidence []string + FromPattern bool +} + +// mineResult carries the detector outputs the alias/emit layers consume. +type mineResult struct { + ranked []ScoredCand // A3 ranking, sorted (-score,-freq,src) + subsumed map[string]bool // V-A ranking-dropped boundary fragments + formants map[rune]formantInfo + cfg MinerConfig +} + +// mineDetect runs V-A → V-C (arm A3, default B) over the normalized chunks and returns the ranked +// candidate list. contrast is the general-zh reference (loaded once). Deterministic. +func mineDetect(chunks []MinerChunk, contrast *Contrast, cfg MinerConfig) mineResult { + va := runVA(chunks, contrast, cfg.FreqFloor, cfg.SubsumeAlpha) + pats, formants := patternCandidates(chunks, va.candFreq, contrast, cfg.FormantMinPartners, cfg.FormantMinOverRep) + + merged := make(map[string]*ScoredCand, len(va.ranked)+len(pats)) + for _, c := range va.ranked { + merged[c.src] = &ScoredCand{Src: c.src, Score: c.score, Freq: c.freq} // score·(1+λ·sp) with λ=0 + } + for cand, info := range pats { + pw := 0.3 + for _, ev := range info.evidence { + if w, ok := cfg.PatternWeight[evSource(ev)]; ok && w > pw { + pw = w + } + } + typ := "term" + if len(info.types) > 0 { + typ = info.types[0] + } + base, ok := cfg.Bonus[typ] + if !ok { + base = cfg.Bonus["term"] + } + bonus := base * pw + if sc, ok := merged[cand]; ok { + sc.Score += bonus + for _, t := range info.types { + if !containsStr(sc.Types, t) { + sc.Types = append(sc.Types, t) + } + } + sc.Evidence = append(sc.Evidence, firstN(info.evidence, 3)...) + sc.FromPattern = true + } else { + f := countOccurrences(cand, chunks) + merged[cand] = &ScoredCand{ + Src: cand, Score: bonus + cfg.SubfloorFreqScale*float64(f)*pw, Freq: f, + Types: append([]string(nil), info.types...), Evidence: firstN(info.evidence, 3), + FromPattern: true, + } + } + } + // Palladius ru-side confirmation loop is INTENTIONALLY OMITTED (default B — it needs dst_variants + + // is_name_lemma, both pymorphy3; it only shifts 古月's exact rank, which is not a product value). + + ranked := make([]ScoredCand, 0, len(merged)) + for _, sc := range merged { + ranked = append(ranked, *sc) + } + sortScored(ranked) + return mineResult{ranked: ranked, subsumed: va.subsumed, formants: formants, cfg: cfg} +} + +// sortScored orders the arm ranking by (-score, -freq, src) — the frozen key. +func sortScored(cs []ScoredCand) { + sort.Slice(cs, func(i, j int) bool { + if cs[i].Score != cs[j].Score { + return cs[i].Score > cs[j].Score + } + if cs[i].Freq != cs[j].Freq { + return cs[i].Freq > cs[j].Freq + } + return cs[i].Src < cs[j].Src + }) +} + +// evSource is the pattern-weight key: the evidence tag before its ':' (arms' ev.split(":")[0]). +func evSource(ev string) string { + for i := 0; i < len(ev); i++ { + if ev[i] == ':' { + return ev[:i] + } + } + return ev +} + +// firstN returns up to n elements of ss (a fresh slice, never aliasing the input). +func firstN(ss []string, n int) []string { + if len(ss) < n { + n = len(ss) + } + return append([]string(nil), ss[:n]...) +} diff --git a/backend/internal/pipeline/miner_alias.go b/backend/internal/pipeline/miner_alias.go new file mode 100644 index 0000000..ecd685b --- /dev/null +++ b/backend/internal/pipeline/miner_alias.go @@ -0,0 +1,194 @@ +package pipeline + +import ( + "sort" + "strings" +) + +// miner_alias.go: alias tier-1 clustering (WS3 — a Go port of exp16 alias.py, DEFAULT-B). Precision-safe +// PROP-layer rules over a set of src surfaces: +// +// R1 surface containment X ⊂ Y, |X|≥2 → 'extension' edge (方源 ⊂ 古月方源), COMPOSITIONAL guard +// (古月+族长 = a phrase, not an alias) blocks the clan↔title chaining +// R2 surname anchor+compose shared surname → FAMILY supercluster (NOT identity) +// R4 NEGATIVE hard-blocks (checked FIRST): (ii) different confirmed gender; (iii/v) different approved +// dst → different entities EVEN under containment (族长⊂四代族长, 古月⊂古月方源); (i) same surname + +// different given names → family, not identity; (iv) co-presence in one source sentence → not identity +// +// The R3 "shared ru rendering → identity" edge is OMITTED in default B (it needs the pymorphy3-backed +// dst-variant lemmas, which the ratified default drops). Precision is measured entity-level (B³, §A5). + +// aliasParticle is the trailing-particle set that marks a boundary FRAGMENT (alias.PARTICLE): a candidate +// = a known surface + one of these is 方源在/方源心, not an entity. +var aliasParticle = buildRuneSet2("的了在是和就也都不心面前后上下今少多大小与之其") + +// aliasSurface is one src surface with the seed metadata the rules read (alias.Surface, minus the +// default-B-dropped dst_lemmas). +type aliasSurface struct { + src string // normalized src surface + gender string + approvedDst string + typ string +} + +// aliasEdge is one proposed edge between two surfaces. +type aliasEdge struct { + a, b string + rule string + kind string +} + +// isFragment reports whether src is a known surface + a trailing particle (alias.frag): a boundary +// artifact, not an independent entity. seedSurfaces is the normalized seed src/alias set. +func isFragment(src string, seedSurfaces map[string]bool) bool { + r := []rune(src) + if len(r) < 2 { + return false + } + return seedSurfaces[string(r[:len(r)-1])] && aliasParticle[r[len(r)-1]] +} + +// cooccurSameSentence reports whether a and b appear in one source sentence anywhere (alias. +// cooccur_same_sentence: split on 。！？\n). +func cooccurSameSentence(chunks []MinerChunk, aNorm, bNorm string) bool { + for _, c := range chunks { + for _, sent := range splitMinerSentences(c.NSource) { + if strings.Contains(sent, aNorm) && strings.Contains(sent, bNorm) { + return true + } + } + } + return false +} + +func splitMinerSentences(s string) []string { + return strings.FieldsFunc(s, func(r rune) bool { + return r == '。' || r == '！' || r == '？' || r == '\n' + }) +} + +// proposeAliasEdges applies the tier-1 rules over the surfaces, returning identity, family and weak +// edges (alias.propose_edges, default B). seedSurfaces is the normalized seed src/alias set (the R1 +// entity-vs-fragment guard). Deterministic: surfaces are iterated in a fixed sorted order. +func proposeAliasEdges(surfaces map[string]aliasSurface, chunks []MinerChunk, seedSurfaces map[string]bool) (ident, family, weak []aliasEdge) { + keys := make([]string, 0, len(surfaces)) + for k := range surfaces { + keys = append(keys, k) + } + sort.Strings(keys) + for i := 0; i < len(keys); i++ { + for j := i + 1; j < len(keys); j++ { + a, b := surfaces[keys[i]], surfaces[keys[j]] + // R4-ii different confirmed gender (hidden never blocks). + if a.gender != "" && b.gender != "" && a.gender != b.gender && a.gender != "hidden" && b.gender != "hidden" { + continue + } + // R4-iii/v different approved dst → different entities, even under containment. + if a.approvedDst != "" && b.approvedDst != "" && normalizeSourceKey(a.approvedDst) != normalizeSourceKey(b.approvedDst) { + continue + } + surA, surB := isSurnameStart([]rune(a.src)), isSurnameStart([]rune(b.src)) + // R1 containment → extension edge (same entity, fuller vs shorter surface). + if runeLen(a.src) >= 2 && runeLen(b.src) >= 2 && a.src != b.src && (strings.Contains(b.src, a.src) || strings.Contains(a.src, b.src)) { + short, long := a, b + if runeLen(b.src) < runeLen(a.src) { + short, long = b, a + } + shortIsEntity := short.typ != "" || seedSurfaces[short.src] + rest := containmentRest(short.src, long.src) + restEnt, restKnown := surfaces[rest] + compositional := rest != "" && seedSurfaces[rest] && + (!restKnown || restEnt.typ == "title" || restEnt.typ == "place" || restEnt.typ == "term") + switch { + case compositional: + weak = append(weak, aliasEdge{a.src, b.src, "R1-compositional", "phrase_not_alias"}) + case shortIsEntity: + ident = append(ident, aliasEdge{a.src, b.src, "R1", "extension"}) + default: + weak = append(weak, aliasEdge{a.src, b.src, "R1-fragment?", "containment_weak"}) + } + continue + } + // R2 surname anchor → family (NOT identity). + if surA != "" && surB != "" && surA == surB && a.src != b.src { + givenA := trimRunePrefix(a.src, surA) + givenB := trimRunePrefix(b.src, surB) + if givenA != givenB && givenA != "" && givenB != "" { // R4-i different given names → family only + if cooccurSameSentence(chunks, a.src, b.src) { + family = append(family, aliasEdge{a.src, b.src, "R2+R4iv", "family_copresent"}) + } else { + family = append(family, aliasEdge{a.src, b.src, "R2", "family"}) + } + continue + } + } + // R3 (shared ru rendering → identity) is OMITTED in default B. + } + } + return ident, family, weak +} + +// containmentRest is alias.py's rest = long[len(short):] if long.startswith(short) else long[:-len(short)] +// (by character). short is contained in long. +func containmentRest(short, long string) string { + sr, lr := []rune(short), []rune(long) + if len(sr) > len(lr) { + return "" + } + if runesEqual(lr[:len(sr)], sr) { + return string(lr[len(sr):]) + } + return string(lr[:len(lr)-len(sr)]) +} + +// trimRunePrefix removes the leading prefix (a surname) from src, by character. +func trimRunePrefix(src, prefix string) string { + sr, pr := []rune(src), []rune(prefix) + if len(pr) <= len(sr) && runesEqual(sr[:len(pr)], pr) { + return string(sr[len(pr):]) + } + return src +} + +// clusterAlias runs union-find over the identity edges and returns clusters of size > 1, each sorted +// (alias.cluster). allSurfaces bounds the universe. +func clusterAlias(ident []aliasEdge, allSurfaces []string) [][]string { + parent := make(map[string]string, len(allSurfaces)) + for _, s := range allSurfaces { + parent[s] = s + } + var find func(string) string + find = func(x string) string { + for parent[x] != x { + parent[x] = parent[parent[x]] + x = parent[x] + } + return x + } + for _, e := range ident { + if _, ok := parent[e.a]; ok { + if _, ok := parent[e.b]; ok { + parent[find(e.a)] = find(e.b) + } + } + } + groups := map[string][]string{} + for _, s := range allSurfaces { + r := find(s) + groups[r] = append(groups[r], s) + } + var out [][]string + roots := make([]string, 0, len(groups)) + for r := range groups { + roots = append(roots, r) + } + sort.Strings(roots) + for _, r := range roots { + if len(groups[r]) > 1 { + g := append([]string(nil), groups[r]...) + sort.Strings(g) + out = append(out, g) + } + } + return out +} diff --git a/backend/internal/pipeline/miner_detect.go b/backend/internal/pipeline/miner_detect.go new file mode 100644 index 0000000..ac99268 --- /dev/null +++ b/backend/internal/pipeline/miner_detect.go @@ -0,0 +1,160 @@ +package pipeline + +import ( + "math" + "sort" + "strings" + "unicode/utf8" +) + +// miner_detect.go: the deterministic V-A detector (WS3 — a Go port of exp16 detectors.py). Frequency × +// contrast(weirdness) × c-value(nested discount). Output: ranked src candidates. The thresholds (freq +// floor 3, subsume α 0.80) are frozen miner-v1 (tuned on chapters 1–15, §D1). Pure and deterministic — +// map iteration only ever precedes a total sort by (-score, -freq, src). + +// minerSubsumeAlpha: drop candidate a if a longer container b has f(b) >= α·f(a) — a is a boundary +// fragment, not an independent term (detectors.SUBSUME_ALPHA). It only cleans the candidate RANKING; +// the occurrence axis (countOccurrences) is suppress-free (D24.2). +const minerSubsumeAlpha = 0.80 + +// runeLen is the character length of a candidate (Han n-grams are ≤6 runes). +func runeLen(s string) int { return utf8.RuneCountInString(s) } + +// minerCandidate is one V-A-scored candidate (detectors.Candidate; the V-B/V-C enrichments live on the +// arm-level ScoredCand, not here). +type minerCandidate struct { + src string + freq int + cvalue float64 + weirdness float64 + termhood float64 + score float64 + length int +} + +// computeCValue is the c-value with nested discount (detectors.compute_cvalue). For a candidate a nested +// in longer candidates T_a: cval(a) = g(|a|)·(f(a) − (1/|T_a|)·Σ_{b∈T_a} f(b)); non-nested: g(|a|)·f(a). +// g(L)=log2(L+1). Deterministic (accumulates into maps, order-independent). +func computeCValue(candFreq map[string]int) map[string]float64 { + tcount := make(map[string]int, len(candFreq)) + tfreqsum := make(map[string]int, len(candFreq)) + for b, fb := range candFreq { + br := []rune(b) + Lb := len(br) + if Lb < 2 { + continue + } + // Distinct proper substrings a of b that are themselves candidates. + subs := map[string]bool{} + for n := 1; n < Lb; n++ { + for i := 0; i+n <= Lb; i++ { + a := string(br[i : i+n]) + if a == b { + continue + } + if _, ok := candFreq[a]; ok { + subs[a] = true + } + } + } + for a := range subs { + tcount[a]++ + tfreqsum[a] += fb + } + } + cval := make(map[string]float64, len(candFreq)) + for a, fa := range candFreq { + g := lengthMult(runeLen(a)) + if tcount[a] > 0 { + cval[a] = g * (float64(fa) - float64(tfreqsum[a])/float64(tcount[a])) + } else { + cval[a] = g * float64(fa) + } + } + return cval +} + +// subsumedCandidates returns the boundary-fragment candidates to drop from the RANKING (detectors. +// subsumed_candidates): a whose every occurrence is (almost) inside a strictly longer candidate b with +// f(b) ≥ α·f(a). 方源(559) survives 方源的(85) [ratio 0.15]; 花酒行(65) is dropped by 花酒行者(65) [1.0]. +func subsumedCandidates(candFreq map[string]int, alpha float64) map[string]bool { + byLen := map[int][]string{} + for c := range candFreq { + L := runeLen(c) + byLen[L] = append(byLen[L], c) + } + drop := map[string]bool{} + for a, fa := range candFreq { + La := runeLen(a) + found := false + for Lb := La + 1; Lb <= minerNgramMax && !found; Lb++ { + for _, b := range byLen[Lb] { + if strings.Contains(b, a) && float64(candFreq[b]) >= alpha*float64(fa) { + found = true + break + } + } + } + if found { + drop[a] = true + } + } + return drop +} + +// vaResult carries the V-A build outputs the arms/patterns layers need. +type vaResult struct { + candFreq map[string]int // freq ≥ floor + nBook int // total n-gram token mass (weirdness denominator scale) + subsumed map[string]bool // ranking-dropped boundary fragments + cval map[string]float64 // per-candidate c-value + ranked []minerCandidate // V-A ranking (subsumed excluded), sorted (-score,-freq,src) +} + +// runVA builds and scores V-A over the chunks (detectors.VA.build + score_all). candFreq is the +// freq≥floor multiset; nBook is the FULL (pre-floor) token mass. score = max(cval,0)·termhood, where +// termhood=log1p(weirdness), weirdness = P_book / max(word_rel, char_indep_rel, 1e-9). +func runVA(chunks []MinerChunk, contrast *Contrast, freqFloor int, alpha float64) vaResult { + allFreq, nBook := enumerateCandidates(chunks) + candFreq := make(map[string]int, len(allFreq)) + for k, v := range allFreq { + if v >= freqFloor { + candFreq[k] = v + } + } + subsumed := subsumedCandidates(candFreq, alpha) + cval := computeCValue(candFreq) + + const weirdFloor = 1e-9 + var ranked []minerCandidate + for a, f := range candFreq { + if subsumed[a] { + continue + } + pBook := float64(f) / float64(nBook) + gen := math.Max(math.Max(contrast.wordRel(a), contrast.charIndepRel(a)), weirdFloor) + w := pBook / gen + termhood := math.Log1p(w) + cv := cval[a] + score := math.Max(cv, 0.0) * termhood + ranked = append(ranked, minerCandidate{ + src: a, freq: f, cvalue: cv, weirdness: w, termhood: termhood, score: score, length: runeLen(a), + }) + } + sortCandidates(ranked) + return vaResult{candFreq: candFreq, nBook: nBook, subsumed: subsumed, cval: cval, ranked: ranked} +} + +// sortCandidates orders by (-score, -freq, src) — the frozen miner-v1 ranking key (stable across Go/ +// Python via the same tie-break). +func sortCandidates(cs []minerCandidate) { + sort.Slice(cs, func(i, j int) bool { + if cs[i].score != cs[j].score { + return cs[i].score > cs[j].score + } + if cs[i].freq != cs[j].freq { + return cs[i].freq > cs[j].freq + } + return cs[i].src < cs[j].src + }) +} diff --git a/backend/internal/pipeline/miner_emit.go b/backend/internal/pipeline/miner_emit.go new file mode 100644 index 0000000..e2bf16d --- /dev/null +++ b/backend/internal/pipeline/miner_emit.go @@ -0,0 +1,227 @@ +package pipeline + +import ( + "fmt" + "sort" + + "gopkg.in/yaml.v3" + + "textmachine/backend/internal/store" +) + +// miner_emit.go: the WS3 seed-delta emission (§C2-7) — the miner's owner-facing output. It runs the +// default-B detector, applies the subsumption + fragment + type + freq FILTERS (never a raw 13618-dump, +// which carries §B2 fragment noise), clusters aliases (tier-1), and emits one MinedTerm per NEW entity. +// +// Discipline: the miner NEVER writes `approved` — every emitted term is a `draft`/`auto` PROPOSAL for the +// owner to sign (§C2). In default B the WHAT (dst) is delivered by the banknote (WS4) at the W1.5 sign +// boundary, so the miner emits WHICH-only candidates (status:auto, no dst — inert until a dst + owner +// promotion). A candidate that clusters with an existing seed surface is an alias-of-existing and is left +// for the owner to attach (not emitted as a new entity), so the delta is genuinely new terms. + +// emitMinFreq is the emission frequency floor (emit_owner_sheets.build_precision30: freq ≥ 5). +const emitMinFreq = 5 + +// MinedTerm is one WHICH candidate the miner proposes (a seed-delta entry). The miner never writes a dst +// (default B) or `approved`; the caller persists it via the mined-write path (Source:"mined", status +// auto) and the owner signs it at W1.5. +type MinedTerm struct { + Src string + Type string // name|place|title|term (the candidate's first pattern type; "" → term) + SinceCh int // first chapter of appearance across the term + its aliases (auto) + Freq int + Aliases []string // identity-cluster co-surfaces (normalized), sorted; the entity's other surfaces + Evidence []string // up to 3 pattern-evidence tags (for the owner sidecar) +} + +// MineBank runs the full default-B miner over the normalized chunks and emits the alias-clustered, +// filtered mined delta (WHICH candidates for owner sign). seed is the current glossary (its surfaces +// scope the non-seed filter, the fragment guard, and the alias entity/metadata). Deterministic and $0. +func MineBank(chunks []MinerChunk, contrast *Contrast, seed []store.GlossaryEntry, cfg MinerConfig) []MinedTerm { + mr := mineDetect(chunks, contrast, cfg) + + // Seed surfaces (normalized) + their metadata for the alias rules and the non-seed / fragment guards. + seedSurfaces := map[string]bool{} + seedMeta := map[string]aliasSurface{} + for _, e := range seed { + for _, surf := range append([]string{e.Src}, aliasStrings(e.Aliases)...) { + nk := normalizeSourceKey(surf) + if nk == "" { + continue + } + seedSurfaces[nk] = true + // The primary src carries the dst/gender/type; an alias inherits them (same entity). + if _, ok := seedMeta[nk]; !ok { + meta := aliasSurface{src: nk, typ: e.Type} + if e.Status == "approved" { + meta.approvedDst = e.Dst + } + meta.gender = e.Gender + seedMeta[nk] = meta + } + } + } + + // Qualifying candidate pool (build_precision30 filters), in ranked order (the ranked order is the + // representative-selection order below — the top-ranked cluster member owns the aliases). + var pool []ScoredCand + for _, c := range mr.ranked { + if !emissionEligible(c, mr.subsumed, seedSurfaces) { + continue + } + pool = append(pool, c) + } + + // Alias universe = qualifying candidate surfaces ∪ seed surfaces; run tier-1 clustering. + surfaces := map[string]aliasSurface{} + for _, c := range pool { + typ := "" + if len(c.Types) > 0 { + typ = c.Types[0] + } + surfaces[c.Src] = aliasSurface{src: c.Src, typ: typ} + } + for nk, meta := range seedMeta { + surfaces[nk] = meta // seed metadata wins (gender/approved_dst) + } + universe := make([]string, 0, len(surfaces)) + for s := range surfaces { + universe = append(universe, s) + } + sort.Strings(universe) + ident, _, _ := proposeAliasEdges(surfaces, chunks, seedSurfaces) + clusters := clusterAlias(ident, universe) + + // clusterOf maps a surface to its identity cluster (members); a surface not in a multi-cluster maps + // to a singleton. + clusterOf := map[string][]string{} + for _, cl := range clusters { + for _, s := range cl { + clusterOf[s] = cl + } + } + + var out []MinedTerm + emitted := map[string]bool{} + for _, c := range pool { // ranked order → the top-ranked cluster member is the representative + if seedSurfaces[c.Src] || emitted[c.Src] { + continue + } + cl := clusterOf[c.Src] + // A cluster touching a seed surface is an alias-of-existing entity → the owner attaches it; skip. + if clusterHasSeed(cl, seedSurfaces) { + markEmitted(cl, emitted) + continue + } + var aliases []string + if len(cl) > 1 { + for _, m := range cl { + if m != c.Src && !seedSurfaces[m] { + aliases = append(aliases, m) + } + } + sort.Strings(aliases) + markEmitted(cl, emitted) // the whole cluster is represented by this one term + } else { + emitted[c.Src] = true + } + typ := "term" + if len(c.Types) > 0 { + typ = c.Types[0] + } + out = append(out, MinedTerm{ + Src: c.Src, Type: typ, Freq: c.Freq, Aliases: aliases, Evidence: c.Evidence, + SinceCh: entitySinceCh(append([]string{c.Src}, aliases...), chunks), + }) + } + // Deterministic output order: by src (the caller may re-sort, but pin a stable delta). + sort.Slice(out, func(i, j int) bool { return out[i].Src < out[j].Src }) + return out +} + +// MinedDeltaYAML serializes the mined delta into the seedTerm YAML schema (§C2-7) — the owner-sign +// artifact and the `tmctl seed-lint` input. Every term is status:auto with NO dst (WHICH-only, default +// B; the WHAT is delivered by the banknote at W1.5). It reuses the EXISTING seedTerm schema (no new +// fields — §3(б)); evidence / zones live in the sidecar sign-map, not the seed. The output is guaranteed +// loadable by loadGlossarySeed (status:auto may lack a dst); seed-lint proves it against the real loader. +func MinedDeltaYAML(mined []MinedTerm) (string, error) { + sf := seedFile{} + for _, m := range mined { + st := seedTerm{Src: m.Src, Type: m.Type, Status: "auto", SinceCh: m.SinceCh} + for _, a := range m.Aliases { + st.Aliases = append(st.Aliases, seedAlias{Alias: a, Type: "mined"}) + } + if len(m.Evidence) > 0 { + st.Note = "mined WHICH candidate; evidence: " + fmt.Sprint(m.Evidence) + } + sf.Terms = append(sf.Terms, st) + } + b, err := yaml.Marshal(sf) + if err != nil { + return "", fmt.Errorf("pipeline: marshal mined delta: %w", err) + } + return string(b), nil +} + +// emissionEligible applies the build_precision30 filters: type ∈ {name,place,title}, freq ≥ 5, src not +// subsumed, len ≥ 2 rune, not a boundary fragment. +func emissionEligible(c ScoredCand, subsumed, seedSurfaces map[string]bool) bool { + if !hasAnyType(c.Types, "name", "place", "title") { + return false + } + if c.Freq < emitMinFreq || subsumed[c.Src] || runeLen(c.Src) < 2 { + return false + } + if isFragment(c.Src, seedSurfaces) { + return false + } + return true +} + +func hasAnyType(types []string, want ...string) bool { + for _, t := range types { + for _, w := range want { + if t == w { + return true + } + } + } + return false +} + +func clusterHasSeed(cl []string, seedSurfaces map[string]bool) bool { + for _, s := range cl { + if seedSurfaces[s] { + return true + } + } + return false +} + +func markEmitted(cl []string, emitted map[string]bool) { + for _, s := range cl { + emitted[s] = true + } +} + +// entitySinceCh is the earliest chapter any of the entity's surfaces appears in (canon.since_ch = first +// chapter of appearance). +func entitySinceCh(surfaces []string, chunks []MinerChunk) int { + min := 0 + for _, s := range surfaces { + for ch := range candidateChapters(s, chunks) { + if min == 0 || ch < min { + min = ch + } + } + } + return min +} + +func aliasStrings(as []store.GlossaryAlias) []string { + out := make([]string, len(as)) + for i, a := range as { + out[i] = a.Alias + } + return out +} diff --git a/backend/internal/pipeline/miner_palladius.go b/backend/internal/pipeline/miner_palladius.go new file mode 100644 index 0000000..3e3e331 --- /dev/null +++ b/backend/internal/pipeline/miner_palladius.go @@ -0,0 +1,138 @@ +package pipeline + +import ( + "sort" + "strings" + "unicode" +) + +// miner_palladius.go: the Palladius (Палладий) transliteration syllable table + ru-side name-shape +// detector (WS3 — a Go port of exp16 palladius.py). A ru token that segments cleanly into Palladius +// syllables looks like a transliterated Chinese NAME. The pinyin→Cyrillic table is generated +// deterministically from initials × finals + the documented irregularities (a versioned artifact). +// +// In the ratified DEFAULT-B miner the ru-side name CONFIRMATION sub-channel is OFF (it also needs the +// pymorphy3 is_name_lemma gate to block the «найти»=най+ти false-positive class), so isPalladiusToken is +// not wired into the default-B detector pipeline. It is kept as a deterministic, tested artifact — the +// name-shape signal a future faithful-A variant (with a Go morphology backend) would consume, and the +// pattern P4 transliteration-by-pair seam (§B5). + +// palladius initials (声母) → Palladius Cyrillic (palladius.INITIALS). +var palladiusInitials = map[string]string{ + "b": "б", "p": "п", "m": "м", "f": "ф", "d": "д", "t": "т", "n": "н", "l": "л", + "g": "г", "k": "к", "h": "х", "j": "цз", "q": "ц", "x": "с", + "zh": "чж", "ch": "ч", "sh": "ш", "r": "ж", "z": "цз", "c": "ц", "s": "с", +} + +// palladius finals (韵母) → Palladius Cyrillic (palladius.FINALS). ü is written as v. +var palladiusFinals = map[string]string{ + "a": "а", "o": "о", "e": "э", "ai": "ай", "ei": "эй", "ao": "ао", "ou": "оу", + "an": "ань", "en": "энь", "ang": "ан", "eng": "эн", "er": "эр", "ong": "ун", + "i": "и", "ia": "я", "ie": "е", "iao": "яо", "iu": "ю", "ian": "янь", + "in": "инь", "iang": "ян", "ing": "ин", "iong": "юн", + "u": "у", "ua": "уа", "uo": "о", "uai": "уай", "ui": "уй", "uan": "уань", + "un": "унь", "uang": "уан", "ueng": "ун", + "v": "юй", "ve": "юэ", "van": "юань", "vn": "юнь", +} + +// whole zero-initial (y-/w-) syllables (palladius.Y_W). +var palladiusYW = map[string]string{ + "yi": "и", "ya": "я", "ye": "е", "yao": "яо", "you": "ю", "yan": "янь", "yin": "инь", + "yang": "ян", "ying": "ин", "yong": "юн", "yu": "юй", "yue": "юэ", "yuan": "юань", "yun": "юнь", + "wu": "у", "wa": "ва", "wo": "во", "wai": "вай", "wei": "вэй", "wan": "вань", "wen": "вэнь", + "wang": "ван", "weng": "вэн", +} + +// retroflex/sibilant + i → -ы/-и class (palladius.SPECIAL_I). +var palladiusSpecialI = map[string]string{ + "zhi": "чжи", "chi": "чи", "shi": "ши", "ri": "жи", "zi": "цзы", "ci": "цы", "si": "сы", +} + +var palladiusJQX = map[string]bool{"j": true, "q": true, "x": true} + +// palladiusCyrSyllables is the distinct Cyrillic syllable set, longest-first, for greedy segmentation +// (palladius._CYR_SYL). Ties in length never both match a position (distinct strings), so the order +// among equal-length forms is immaterial — sorted (len desc, then value) for a stable artifact. +var palladiusCyrSyllables = buildPalladiusCyrSyllables() + +func buildPalladiusCyrSyllables() []string { + set := map[string]bool{} + for _, v := range palladiusYW { + set[v] = true + } + for _, v := range palladiusSpecialI { + set[v] = true + } + retroflex := map[string]bool{"zh": true, "ch": true, "sh": true, "r": true, "z": true, "c": true, "s": true} + for pi, ci := range palladiusInitials { + for pf, cf := range palladiusFinals { + // ü finals (written v) are valid only after j/q/x (pinyin writes ü as plain u) or l/n. + switch pf { + case "v", "ve", "van", "vn": + if !palladiusJQX[pi] && pi != "l" && pi != "n" { + continue + } + } + // skip the retroflex/sibilant + bare i (handled by SPECIAL_I). + if pf == "i" && retroflex[pi] { + continue + } + set[ci+cf] = true + } + } + out := make([]string, 0, len(set)) + for s := range set { + if s != "" { + out = append(out, s) + } + } + sort.Slice(out, func(i, j int) bool { + if len([]rune(out[i])) != len([]rune(out[j])) { + return len([]rune(out[i])) > len([]rune(out[j])) + } + return out[i] < out[j] + }) + return out +} + +// isPalladiusToken reports whether a ru word segments fully into Palladius syllables (greedy longest- +// match, palladius.is_palladius_token): a high-precision "this looks like a transliterated Chinese name" +// signal. ъ is dropped before segmentation. minSyllables is the minimum syllable count. +func isPalladiusToken(token string, minSyllables int) bool { + t := strings.ToLower(strings.TrimSpace(token)) + t = strings.ReplaceAll(t, "ъ", "") + if t == "" || !allCyrillic(t) { + return false + } + tr := []rune(t) + nSyl := 0 + for i := 0; i < len(tr); { + matched := false + for _, s := range palladiusCyrSyllables { + sr := []rune(s) + if i+len(sr) <= len(tr) && runesEqual(tr[i:i+len(sr)], sr) { + i += len(sr) + nSyl++ + matched = true + break + } + } + if !matched { + return false + } + } + return nSyl >= minSyllables +} + +// allCyrillic reports whether every rune of s is Cyrillic (palladius._RE_CYR_WORD). +func allCyrillic(s string) bool { + if s == "" { + return false + } + for _, r := range s { + if !unicode.Is(unicode.Cyrillic, r) { + return false + } + } + return true +} diff --git a/backend/internal/pipeline/miner_parity_test.go b/backend/internal/pipeline/miner_parity_test.go new file mode 100644 index 0000000..1cbb30e --- /dev/null +++ b/backend/internal/pipeline/miner_parity_test.go @@ -0,0 +1,121 @@ +package pipeline + +import ( + "encoding/json" + "os" + "testing" +) + +// miner_parity_test.go: the WS3 (д) Go↔Python full-book parity check. It re-runs the default-B miner on +// the PINNED exp16 inputs (jieba contrast + the 25-chapter records.json + the seed GT) and asserts the +// Palladius-INVARIANT guarantees the Go port must reproduce (research/20 §D, ws3_miner_verify.py): +// +// - candidate SET count = 13618 (membership, integer-determined — Palladius-invariant); +// - recall@proposed (overall) = 0.965 (set membership — Palladius-invariant); +// - catastrophe screen: 方源/蛊/蛊师 at ranks 0/1/2 (invariant), 古月 ∈ top-50 (rank 22 in default B — +// the +56 Palladius bonus is dropped; the SCREEN still passes, exact rank is not a product value). +// +// It is DATA-GATED: the jieba artifact + book records are OUT of git (CLAUDE.md), so the test SKIPS when +// they are absent (CI / a fresh checkout) and runs on the stand (or when TM_MINER_PARITY=1 forces it, +// failing loud if the data is missing). $0, deterministic — Python is the reference; a divergence means +// the Go port is wrong (fix Go), never the reference. +const ( + minerParityContrast = "/home/ubuntu/projects/textmachine/eval/exp16/data/jieba_dict_general_zh.txt" + minerParityRecords = "/home/ubuntu/books/gu-zhenren/rerun/records.json" + minerParitySeed = "/home/ubuntu/books/gu-zhenren/guzhenren-seed-v2.yaml" +) + +func TestMinerFullBookParity(t *testing.T) { + force := os.Getenv("TM_MINER_PARITY") == "1" + for _, p := range []string{minerParityContrast, minerParityRecords, minerParitySeed} { + if _, err := os.Stat(p); err != nil { + if force { + t.Fatalf("TM_MINER_PARITY=1 but required data is missing: %s", p) + } + t.Skipf("stand data absent (%s) — skipping full-book parity (set TM_MINER_PARITY=1 to force)", p) + } + } + + // Contrast (jieba dict). + cf, err := os.Open(minerParityContrast) + if err != nil { + t.Fatal(err) + } + defer cf.Close() + contrast, err := LoadContrast(cf) + if err != nil { + t.Fatal(err) + } + + // Chunks (records.json → normalized miner chunks). + raw, err := os.ReadFile(minerParityRecords) + if err != nil { + t.Fatal(err) + } + var recs []struct { + Chapter int `json:"chapter"` + ChunkIdx int `json:"chunk_idx"` + Source string `json:"source"` + } + if err := json.Unmarshal(raw, &recs); err != nil { + t.Fatal(err) + } + chunks := make([]MinerChunk, 0, len(recs)) + for _, r := range recs { + chunks = append(chunks, MinerChunk{Chapter: r.Chapter, ChunkIdx: r.ChunkIdx, NSource: normalizeSourceKey(r.Source)}) + } + + // GT (the seed, parsed by the REAL loader — a divergence would fail seed-lint too). + gt, err := loadGlossarySeed(minerParitySeed) + if err != nil { + t.Fatal(err) + } + + mr := mineDetect(chunks, contrast, frozenMinerConfig()) + + // 1) candidate SET count. + n := len(mr.ranked) + if n != 13618 { + t.Errorf("candidate SET count = %d, want 13618 (Palladius-invariant)", n) + } + + // 2) catastrophe screen. + rank := map[string]int{} + for i, c := range mr.ranked { + rank[c.Src] = i + } + catExpect := map[string]int{"方源": 0, "蛊": 1, "蛊师": 2} + for src, want := range catExpect { + if got, ok := rank[normalizeSourceKey(src)]; !ok || got != want { + t.Errorf("catastrophe rank[%s] = %d (present=%v), want %d (Palladius-invariant)", src, got, ok, want) + } + } + if got, ok := rank[normalizeSourceKey("古月")]; !ok || got >= 50 { + t.Errorf("catastrophe rank[古月] = %d (present=%v), want ∈top-50 (default B ≈22)", got, ok) + } else { + t.Logf("古月 rank = %d (default B — Palladius sub-channel dropped; reference A3 = 21)", got) + } + + // 3) recall@proposed (overall): fraction of GT entities with any normalized surface in the SET. + candSet := make(map[string]bool, n) + for _, c := range mr.ranked { + candSet[c.Src] = true + } + hits := 0 + for _, e := range gt { + surfaces := append([]string{e.Src}, aliasStrings(e.Aliases)...) + for _, s := range surfaces { + if nk := normalizeSourceKey(s); nk != "" && candSet[nk] { + hits++ + break + } + } + } + recall := float64(hits) / float64(len(gt)) + if recall < 0.960 || recall > 0.970 { + t.Errorf("recall@proposed = %.4f (%d/%d), want ≈0.965", recall, hits, len(gt)) + } + t.Logf("PARITY: n=%d catastrophe{方源:%d 蛊:%d 蛊师:%d 古月:%d} recall@proposed=%.4f (%d/%d GT)", + n, rank[normalizeSourceKey("方源")], rank[normalizeSourceKey("蛊")], rank[normalizeSourceKey("蛊师")], + rank[normalizeSourceKey("古月")], recall, hits, len(gt)) +} diff --git a/backend/internal/pipeline/miner_patterns.go b/backend/internal/pipeline/miner_patterns.go new file mode 100644 index 0000000..aabaf16 --- /dev/null +++ b/backend/internal/pipeline/miner_patterns.go @@ -0,0 +1,354 @@ +package pipeline + +// miner_patterns.go: the V-C pattern channels (WS3 — a Go port of exp16 patterns.py). A language×genre +// pattern pack for zh (§B5 plugin P3, universal set only — owner decision 18.07: genre packs are NOT +// built). Each channel proposes TYPED candidates (name/title/place/term) with evidence, closing V-A's +// frequency-blind classes (rare surname-anchored names, rank/grade titles, one-off realia). Versioned +// data (surnames / title affixes / topo suffixes) + a book-adaptive productive-morphology detector +// (channel 4 — auto-detects the domain formant, NOT hardcoded 蛊). Pure and deterministic. + +// minerPackVersion is the pattern-inventory version (patterns.PACK_VERSION). +const minerPackVersion = "zh-universal-v1" + +// surnamesSingle is the 百家姓 single-char subset + this book's cast (patterns.SURNAMES_SINGLE, with 凝 +// discarded — it is not a surname). surnamesCompound are the explicit two-char compound surnames. +var ( + surnamesSingle = buildRuneSet(surnamesSingleRaw) + surnamesCompound = map[string]bool{ + "古月": true, "欧阳": true, "司马": true, "上官": true, "夏侯": true, "诸葛": true, "东方": true, + "皇甫": true, "尉迟": true, "公孙": true, "慕容": true, "长孙": true, "宇文": true, "司徒": true, + "鲜于": true, "南宫": true, + } +) + +const surnamesSingleRaw = "赵钱孙李周吴郑王冯陈褚卫蒋沈韩杨朱秦尤许何吕施张孔曹严华金魏陶姜" + + "戚谢邹喻柏水窦章云苏潘葛奚范彭郎鲁韦昌马苗凤花方俞任袁柳酆鲍史唐" + + "费廉岑薛雷贺倪汤滕殷罗毕郝邬安常乐于时傅皮卞齐康伍余元卜顾孟平黄" + + "和穆萧尹姚邵湛汪祁毛禹狄米贝明臧计伏成戴谈宋茅庞熊纪舒屈项祝董梁" + + "杜阮蓝闵席季麻强贾路娄危江童颜郭梅盛林刁钟徐邱骆高夏蔡田樊胡凌霍" + + "虞万支柯昝管卢莫经房裘缪干解应宗丁宣贲邓郁单杭洪包诸左石崔吉钮龚" + + "白凝" // 凝 removed below via buildRuneSet's caller — kept literal for provenance parity + +// buildRuneSet builds a rune set, then removes 凝 (patterns.SURNAMES_SINGLE.discard("凝")). +func buildRuneSet(s string) map[rune]bool { + m := map[rune]bool{} + for _, r := range s { + m[r] = true + } + delete(m, '凝') + return m +} + +// Title / ordinal / topo / rank inventories (patterns.TITLE_SUFFIX/ORDINAL/TOPO_SUFFIX/GRADE_PREFIX/ +// NUMERAL/RANK_WORD). TITLE_PREFIX is defined in the reference but unused by pattern_candidates, so it +// is omitted here. +var ( + titleSuffix = []string{"公子", "大人", "长老", "前辈", "族长", "家老", "老祖", "祖师", "真人", "上人", + "道人", "先生", "夫人", "娘子", "姑娘", "嬷嬷", "师傅", "师父", "掌门", "宗主", + "少爷", "老爷", "小姐", "婆婆", "大娘", "大爷"} + ordinalTitle = []string{"一代", "二代", "三代", "四代", "五代", "六代", "七代", "八代", "九代", "十代", + "第一", "第二", "第三", "第四", "第五"} + topoSuffix = buildRuneSet2("山寨村疆谷城门宗府洞峰岭河江湖海林原岛潭殿阁楼院堂祠") + gradePrefix = buildRuneSet2("甲乙丙丁戊己庚辛壬癸") + numeral = buildRuneSet2("一二三四五六七八九十零百千") + rankWord = []string{"等", "转", "阶", "级", "重", "品", "段", "层"} +) + +// buildRuneSet2 builds a rune set with NO discard (topo/grade/numeral inventories). +func buildRuneSet2(s string) map[rune]bool { + m := map[rune]bool{} + for _, r := range s { + m[r] = true + } + return m +} + +// isSurnameStart returns the surname prefix (compound preferred) if s starts with one, else "" (patterns. +// is_surname_start). s is a []rune (a Han run suffix). +func isSurnameStart(s []rune) string { + if len(s) >= 2 { + if pre := string(s[:2]); surnamesCompound[pre] { + return pre + } + } + if len(s) >= 1 && surnamesSingle[s[0]] { + return string(s[0]) + } + return "" +} + +// patternInfo is one candidate's channel-proposed types + evidence, in first-seen (deterministic) order. +type patternInfo struct { + types []string + evidence []string +} + +// formantInfo describes a detected productive-morphology formant (patterns.detect_formants entry). +type formantInfo struct { + role string // suffix|prefix|both + overRep float64 + nSuffix int + nPrefix int +} + +// runeHasPrefixAt reports whether run[i:] starts with p (patterns' run.startswith(suf, i)). +func runeHasPrefixAt(run []rune, i int, p []rune) bool { + if i+len(p) > len(run) { + return false + } + for k := range p { + if run[i+k] != p[k] { + return false + } + } + return true +} + +// formantType maps a formant char to a candidate type (patterns.formant_type): topo→place, 等/转/阶→ +// title, else term. +func formantType(c rune) string { + if topoSuffix[c] { + return "place" + } + if c == '等' || c == '转' || c == '阶' { + return "title" + } + return "term" +} + +// bookCharFreq counts Han chars over the normalized sources (patterns.book_char_freq). +func bookCharFreq(chunks []MinerChunk) map[rune]int { + bc := map[rune]int{} + for _, c := range chunks { + for _, r := range c.NSource { + if isMinerHan(r) { + bc[r]++ + } + } + } + return bc +} + +// detectFormants auto-detects the productive DOMAIN formants (patterns.detect_formants): a Han char that +// binds (suffix OR prefix) with ≥ minPartners DISTINCT content morphemes among candidates AND is +// over-represented in-book vs general zh (over_rep ≥ minOverRep). Over-representation — NOT char-rarity +// — isolates 蛊/窍/虫 while rejecting common chars 师/花/等. Deterministic (per-char, order-independent). +func detectFormants(chunks []MinerChunk, candFreq map[string]int, contrast *Contrast, minPartners int, minOverRep float64) map[rune]formantInfo { + bc := bookCharFreq(chunks) + totBook := 0 + for _, v := range bc { + totBook += v + } + if totBook == 0 { + totBook = 1 + } + suf := map[rune]map[string]bool{} + pre := map[rune]map[string]bool{} + addPartner := func(m map[rune]map[string]bool, key rune, part string) { + if m[key] == nil { + m[key] = map[string]bool{} + } + m[key][part] = true + } + for a := range candFreq { + ar := []rune(a) + if len(ar) < 2 { + continue + } + addPartner(suf, ar[len(ar)-1], string(ar[:len(ar)-1])) + addPartner(pre, ar[0], string(ar[1:])) + } + chars := map[rune]bool{} + for c := range suf { + chars[c] = true + } + for c := range pre { + chars[c] = true + } + formants := map[rune]formantInfo{} + for c := range chars { + pBook := float64(bc[c]) / float64(totBook) + overRep := contrast.charOverRep(c, pBook) + if overRep < minOverRep { + continue + } + ns, npr := len(suf[c]), len(pre[c]) + role := "" + if ns >= minPartners { + role = "suffix" + } + if npr >= minPartners { + if role != "" { + role = "both" + } else { + role = "prefix" + } + } + if role != "" { + formants[c] = formantInfo{role: role, overRep: overRep, nSuffix: ns, nPrefix: npr} + } + } + return formants +} + +// patternCandidates produces the typed pattern candidates from the source (patterns.pattern_candidates). +// It operates over the ALREADY-normalized source and may propose candidates BELOW the V-A freq floor +// (that is the point — patterns close the rare-term blind spot). Returns the candidate→info map and the +// detected formants. Deterministic (fixed chunk/run/position iteration; add() preserves first-seen order). +func patternCandidates(chunks []MinerChunk, candFreq map[string]int, contrast *Contrast, minPartners int, minOverRep float64) (map[string]*patternInfo, map[rune]formantInfo) { + out := map[string]*patternInfo{} + add := func(cand []rune, typ, ev string) { + cn := normalizeSourceKey(string(cand)) + if cn == "" || !minerHanOnly(cn) { + return + } + info := out[cn] + if info == nil { + info = &patternInfo{} + out[cn] = info + } + if !containsStr(info.types, typ) { + info.types = append(info.types, typ) + } + if !containsStr(info.evidence, ev) { + info.evidence = append(info.evidence, ev) + } + } + + titleSufR := toRuneSlices(titleSuffix) + ordinalR := toRuneSlices(ordinalTitle) + rankR := toRuneSlices(rankWord) + + for _, c := range chunks { + for _, run := range hanRuns(c.NSource) { + L := len(run) + for i := 0; i < L; i++ { + // (1) surname anchor: surname + 1–2 Han given-name window → name + if sn := isSurnameStart(run[i:]); sn != "" { + base := i + len([]rune(sn)) + for _, gl := range [2]int{1, 2} { + if base+gl <= L { + full := run[i : base+gl] + if len(full) >= 2 && len(full) <= 4 { + add(full, "name", "surname:"+sn) + } + } + } + } + // (2) title suffix: content + suffix → title + for si, suf := range titleSufR { + if runeHasPrefixAt(run, i, suf) { + for _, left := range [4]int{3, 2, 1, 0} { + if i-left >= 0 { + cand := run[i-left : i+len(suf)] + if len(cand) >= 2 && len(cand) <= 6 { + add(cand, "title", "title_suffix:"+titleSuffix[si]) + } + } + } + add([]rune(titleSuffix[si]), "title", "title_bare:"+titleSuffix[si]) + } + } + // (2b) ordinal + title (四代族长-class) → title + for oi, od := range ordinalR { + if runeHasPrefixAt(run, i, od) { + for si, suf := range titleSufR { + end := i + len(od) + if runeHasPrefixAt(run, end, suf) { + add(run[i:end+len(suf)], "title", "ordinal_title:"+ordinalTitle[oi]+"+"+titleSuffix[si]) + } + } + } + } + // (2c) rank/grade compositional: (numeral|grade-prefix) + rank-word → title + for ri, rw := range rankR { + if runeHasPrefixAt(run, i, rw) && i >= 1 { + left := run[i-1] + if numeral[left] || gradePrefix[left] { + add(run[i-1:i+len(rw)], "title", "rank_grade:"+string(left)+"+"+rankWord[ri]) + } + } + } + // (3) topo suffix: 1–3 Han base + topo char → place + if topoSuffix[run[i]] && i >= 1 { + for _, left := range [3]int{3, 2, 1} { + if i-left >= 0 { + cand := run[i-left : i+1] + if len(cand) >= 2 && len(cand) <= 4 { + add(cand, "place", "topo_suffix:"+string(run[i])) + } + } + } + } + } + } + } + + // (4) productive morphology: for each detected formant, propose all binding n-grams. + formants := detectFormants(chunks, candFreq, contrast, minPartners, minOverRep) + for _, c := range chunks { + for _, run := range hanRuns(c.NSource) { + L := len(run) + for i := 0; i < L; i++ { + info, ok := formants[run[i]] + if !ok { + continue + } + typ := formantType(run[i]) + if info.role == "suffix" || info.role == "both" { + for _, left := range [3]int{3, 2, 1} { + if i-left >= 0 { + cand := run[i-left : i+1] + if len(cand) >= 2 && len(cand) <= 4 { + add(cand, typ, "formant_suffix:"+string(run[i])) + } + } + } + } + if info.role == "prefix" || info.role == "both" { + for _, r := range [2]int{2, 3} { + if i+r <= L { + cand := run[i : i+r] + if len(cand) >= 2 && len(cand) <= 4 { + add(cand, typ, "formant_prefix:"+string(run[i])) + } + } + } + } + } + } + } + return out, formants +} + +// minerHanOnly reports whether s is non-empty and all Han (exp16_common.han_only). +func minerHanOnly(s string) bool { + if s == "" { + return false + } + for _, r := range s { + if !isMinerHan(r) { + return false + } + } + return true +} + +// toRuneSlices converts a []string inventory to []([]rune) once (avoids re-decoding per scan position). +func toRuneSlices(ss []string) [][]rune { + out := make([][]rune, len(ss)) + for i, s := range ss { + out[i] = []rune(s) + } + return out +} + +// containsStr reports whether ss contains s. +func containsStr(ss []string, s string) bool { + for _, x := range ss { + if x == s { + return true + } + } + return false +} diff --git a/backend/internal/pipeline/miner_substrate.go b/backend/internal/pipeline/miner_substrate.go new file mode 100644 index 0000000..13ba466 --- /dev/null +++ b/backend/internal/pipeline/miner_substrate.go @@ -0,0 +1,224 @@ +package pipeline + +import ( + "bufio" + "io" + "math" + "strconv" + "strings" + "unicode" +) + +// miner_substrate.go: the deterministic bank-mining substrate (WS3, слой 4 — a Go port of the FROZEN +// exp16 miner-v1: exp16_common.py). $0, no LLM, no embeddings. It provides the candidate generator (Han +// char n-grams 1–6 over the memnorm-normalized source, NO word segmentation — §B1/Qiu&Zhang), the +// general-zh contrast corpus (a versioned jieba word-freq artifact), overlapping occurrence counting +// (Aho-Corasick-equivalent WITHOUT the hot-path suppressContained — D24.2), and the frequency strata. +// +// It reuses the VALIDATED normalizeSourceKey (memnorm.go, byte-faithful to the eval mem_select port), so +// a candidate key and a seed/GT key live in the SAME normalized space (trad→simp, NFKC, kana-fold, lower) +// — comparable to the jieba dict (simplified zh). The whole detector (this + miner_detect/patterns/arms) +// is the WHICH channel; the dst (WHAT) is delivered by the banknote (WS4), never by co-occurrence +// (canon-recovery 0.20–0.68 < 70% — research/20). Determinism is load-bearing: no time/rand, fixed +// iteration; the mined delta is a pure function of (chunks, contrast, config). + +// Miner n-gram bounds and the V-A frequency floor (exp16_common: NGRAM_MIN/MAX, FREQ_FLOOR). +const ( + minerNgramMin = 1 + minerNgramMax = 6 + minerFreqFloor = 3 // §A1/§B1: below this V-A does not score (a conscious blind spot patterns close) +) + +// MinerChunk is one source chunk for mining: its chapter and the memnorm-normalized zh source. The +// caller normalizes once (normalizeSourceKey) so the candidate space matches the glossary/GT space. +type MinerChunk struct { + Chapter int + ChunkIdx int + NSource string // normalizeSourceKey(raw source) +} + +// isMinerHan reports whether r is a Han ideograph — the candidate alphabet (§B1). Mirrors the Python +// reference's \p{Han}: candidate n-grams are runs of Han characters only. +func isMinerHan(r rune) bool { return unicode.Is(unicode.Han, r) } + +// hanRuns returns the maximal runs of Han characters in normalized text (each run a []rune slice so +// n-gram slicing is by CHARACTER, exactly like the Python str slicing). +func hanRuns(ntext string) [][]rune { + var runs [][]rune + var cur []rune + for _, r := range ntext { + if isMinerHan(r) { + cur = append(cur, r) + } else if len(cur) > 0 { + runs = append(runs, cur) + cur = nil + } + } + if len(cur) > 0 { + runs = append(runs, cur) + } + return runs +} + +// enumerateCandidates counts every Han n-gram (length minerNgramMin..minerNgramMax) with OVERLAPPING +// counts across the chunk sources (exp16_common.enumerate_candidates). The annotation chunk is included +// (blurb terms live there; the GT/seed filter handles the blurb rule separately). Returns the full +// multiset (before the freq floor) — its total token mass is the weirdness denominator (N_book). +func enumerateCandidates(chunks []MinerChunk) (freq map[string]int, nBook int) { + freq = make(map[string]int, 1<<16) + for _, c := range chunks { + for _, run := range hanRuns(c.NSource) { + L := len(run) + for n := minerNgramMin; n <= minerNgramMax; n++ { + for i := 0; i+n <= L; i++ { + freq[string(run[i:i+n])]++ + nBook++ + } + } + } + } + return freq, nBook +} + +// countOccurrences totals the OVERLAPPING occurrences of a normalized needle across the chunk sources +// (exp16_common.count_occurrences: pos advances by 1, so 重重 counts twice in 重重重). Suppress-free +// (D24.2) — this is the occurrence axis, never the hot-path longest-match. +func countOccurrences(needleNorm string, chunks []MinerChunk) int { + if needleNorm == "" { + return 0 + } + total := 0 + for _, c := range chunks { + hay := c.NSource + for pos := 0; pos <= len(hay); { + rel := strings.Index(hay[pos:], needleNorm) + if rel < 0 { + break + } + total++ + pos = pos + rel + 1 // overlap advance by ONE byte-position (mirrors Python find(needle, pos+1)) + } + } + return total +} + +// candidateChapters returns the set of chapters whose normalized source contains the candidate (for +// since_ch = the first chapter of appearance, exp16_common.candidate_chapters). Deterministic. +func candidateChapters(candNorm string, chunks []MinerChunk) map[int]bool { + chs := map[int]bool{} + if candNorm == "" { + return chs + } + for _, c := range chunks { + if strings.Contains(c.NSource, candNorm) { + chs[c.Chapter] = true + } + } + return chs +} + +// freqStratum buckets an occurrence count (exp16_common.freq_stratum). +func freqStratum(f int) string { + switch { + case f >= 10: + return "f>=10" + case f >= minerFreqFloor: + return "f3-9" + default: + return "f<3" + } +} + +// --- general-zh contrast corpus (jieba dict.txt: "word freq POS" per line) ------------------------ + +// Contrast is the general-domain zh reference: normalized word frequencies + derived char frequencies, +// used for the weirdness (domain-specificity) signal. A versioned artifact (the jieba 0.42.1 dict.txt, +// SHA-pinned, kept OUT of git — reproducible from jieba). Load it once (miner-build time, $0). +type Contrast struct { + wordFreq map[string]int // normalized word → max freq (trad/simp collisions fold to the max) + charFreq map[rune]int // char → summed freq across normalized words + totalWord int64 + totalChar int64 + nDistinct int // len(charFreq), the Laplace |distinct| term +} + +// LoadContrast reads a jieba-style "word freq POS" corpus from r and builds the contrast model. Each +// word is normalized into the candidate space (normalizeSourceKey) and, on a normalization collision, +// the MAX frequency is kept (mirrors exp16_common.Contrast). Malformed / non-integer-freq lines are +// skipped, exactly like the reference. +func LoadContrast(r io.Reader) (*Contrast, error) { + c := &Contrast{wordFreq: map[string]int{}, charFreq: map[rune]int{}} + sc := bufio.NewScanner(r) + sc.Buffer(make([]byte, 0, 64*1024), 1<<20) + for sc.Scan() { + fields := strings.Fields(sc.Text()) + if len(fields) < 2 { + continue + } + w := fields[0] + fr, err := strconv.Atoi(fields[1]) + if err != nil { + continue + } + wn := normalizeSourceKey(w) + if wn == "" { + continue + } + if prev, ok := c.wordFreq[wn]; !ok || fr > prev { + c.wordFreq[wn] = fr + } + c.totalWord += int64(fr) + for _, ch := range wn { + c.charFreq[ch] += fr + } + } + if err := sc.Err(); err != nil { + return nil, err + } + for _, v := range c.charFreq { + c.totalChar += int64(v) + } + c.nDistinct = len(c.charFreq) + return c, nil +} + +// wordRel is P_general(ngram) treated as a whole word (0 for OOV). +func (c *Contrast) wordRel(ngramNorm string) float64 { + if c.totalWord == 0 { + return 0 + } + return float64(c.wordFreq[ngramNorm]) / float64(c.totalWord) +} + +// charIndepRel is the expected P_general(ngram) under char-independence with Laplace smoothing +// (exp16_common.char_indep_rel): pc = (cf+1)/(total_char+|distinct|), product over chars. +func (c *Contrast) charIndepRel(ngramNorm string) float64 { + denom := float64(c.totalChar) + float64(c.nDistinct) + if denom == 0 { + return 0 + } + p := 1.0 + for _, ch := range ngramNorm { + pc := float64(c.charFreq[ch]+1) / denom + p *= pc + } + return p +} + +// charOverRep is p_book(char)/p_general(char) — the domain over-representation used by the formant +// detector (patterns). p_general uses the same Laplace denominator as charIndepRel. +func (c *Contrast) charOverRep(ch rune, pBook float64) float64 { + denom := float64(c.totalChar) + float64(c.nDistinct) + if denom == 0 { + return 0 + } + pGen := float64(c.charFreq[ch]+1) / denom + if pGen == 0 { + return 0 + } + return pBook / pGen +} + +// lengthMult is the c-value g(L)=log2(L+1) length multiplier (miner_detect uses it; kept here beside +// the substrate math). §B1: NOT log2(L) — that zeros |a|=1 (蛊/转) and breaks the catastrophe screen. +func lengthMult(L int) float64 { return math.Log2(float64(L) + 1) } diff --git a/backend/internal/pipeline/miner_test.go b/backend/internal/pipeline/miner_test.go new file mode 100644 index 0000000..40fe49b --- /dev/null +++ b/backend/internal/pipeline/miner_test.go @@ -0,0 +1,337 @@ +package pipeline + +import ( + "reflect" + "strings" + "testing" + + "textmachine/backend/internal/store" +) + +// miner_test.go: synthetic, in-repo unit fixtures for the WS3 default-B miner (no external book data — +// the full-book Go↔Python parity is the data-gated TestMinerFullBookParity). Each fixture pins one +// load-bearing algorithm piece: the c-value length-multiplier (蛊/转 non-degenerate), subsumption α, the +// pattern channels, is_palladius_token, the alias tier-1 rules (the plan's required §3(д) fixtures), the +// emission filters, determinism, and the Source:"mined" write path. + +// mchunks builds miner chunks from raw zh strings (one chunk per chapter, normalized like the runner). +func mchunks(sources ...string) []MinerChunk { + out := make([]MinerChunk, len(sources)) + for i, s := range sources { + out[i] = MinerChunk{Chapter: i + 1, ChunkIdx: 0, NSource: normalizeSourceKey(s)} + } + return out +} + +// smallContrast is a tiny general-zh reference: common function/name chars carry high freq; the domain +// formant 蛊 is deliberately ABSENT so its book over-representation is high (the formant signal). +func smallContrast(t *testing.T) *Contrast { + t.Helper() + const data = "的 100000 uj\n是 80000 v\n人 50000 n\n出 9000 v\n现 9000 v\n" + + "月 4000 n\n光 3800 n\n希 900 n\n望 3000 v\n影 800 n\n在 20000 p\n此 1200 r\n" + + "古 400 nr\n方 900 nr\n源 300 n\n族 500 n\n长 6000 a\n青 700 n\n茅 20 n\n山 5000 n\n" + + "四 8000 m\n代 3000 n\n甲 300 n\n等 12000 u\n正 4000 a\n" + c, err := LoadContrast(strings.NewReader(data)) + if err != nil { + t.Fatalf("LoadContrast: %v", err) + } + return c +} + +func TestMinerLengthMultNonDegenerate(t *testing.T) { + // §B1: g(L)=log2(L+1), NOT log2(L) — a single-char candidate (蛊/转) must keep a NON-ZERO length + // multiplier, else the catastrophe screen zeros out and 蛊 sinks to rank ~4753. + if got := lengthMult(1); got != 1.0 { + t.Fatalf("lengthMult(1) = %v, want 1.0 (log2(2)); log2(1)=0 would degenerate |a|=1 candidates", got) + } + if lengthMult(2) <= lengthMult(1) { + t.Fatalf("length multiplier must be monotone in L") + } +} + +func TestMinerCValueSingleCharNested(t *testing.T) { + // 蛊 (len 1) nested in 蛊师 (len 2): its c-value is g(1)·(f(蛊) − f(蛊师)) — computed, finite, and its + // magnitude is non-degenerate because g(1)=1 (not 0). Pins the length-mult fix at the c-value layer. + cf := map[string]int{"蛊": 10, "蛊师": 4} + cv := computeCValue(cf) + want := 1.0 * (10.0 - 4.0/1.0) // g(1)·(f(蛊) − avg over its 1 container) + if got := cv["蛊"]; got != want { + t.Fatalf("cval[蛊] = %v, want %v (g(1)=1 non-degenerate)", got, want) + } + if cv["蛊师"] != lengthMult(2)*4 { // non-nested → g(2)·f + t.Fatalf("cval[蛊师] = %v, want %v", cv["蛊师"], lengthMult(2)*4) + } +} + +func TestMinerSubsumption(t *testing.T) { + // 方源(559) survives 方源的(85) [ratio 0.15 < 0.80]; 花酒行(65) is dropped by 花酒行者(65) [ratio 1.0]. + cf := map[string]int{"方源": 559, "方源的": 85, "花酒行": 65, "花酒行者": 65} + drop := subsumedCandidates(cf, minerSubsumeAlpha) + if drop["方源"] { + t.Fatalf("方源 must NOT be subsumed by 方源的 (85 < 0.8·559)") + } + if !drop["花酒行"] { + t.Fatalf("花酒行 must be subsumed by 花酒行者 (65 >= 0.8·65)") + } +} + +func TestMinerPatternChannels(t *testing.T) { + chunks := mchunks( + "古月方源走进青茅山。四代族长甲等弟子。", // surname+name, topo, ordinal_title, rank_grade + ) + // candFreq is only needed for the formant channel; the structural channels scan the source directly. + cf := map[string]int{} + pats, _ := patternCandidates(chunks, cf, smallContrast(t), 3, 15.0) + cases := []struct { + src, typ string + }{ + {"古月方源", "name"}, // surname anchor (compound 古月 + 方源) + {"青茅山", "place"}, // topo suffix 山 + {"四代族长", "title"}, // ordinal_title 四代+族长 + {"甲等", "title"}, // rank_grade 甲+等 + {"族长", "title"}, // title_suffix / title_bare + } + for _, c := range cases { + info, ok := pats[normalizeSourceKey(c.src)] + if !ok { + t.Errorf("pattern channel missed %q", c.src) + continue + } + if !containsStr(info.types, c.typ) { + t.Errorf("%q types = %v, want to include %q", c.src, info.types, c.typ) + } + } +} + +func TestMinerFormantDetection(t *testing.T) { + // 蛊 binds ≥3 distinct morphemes among freq≥3 candidates AND is over-represented in-book vs the + // contrast (absent there) → auto-detected as a suffix formant; a common char (的) is NOT. + chunks := mchunks(strings.Repeat("月光蛊出现。希望蛊出现。月影蛊出现。", 3)) + va := runVA(chunks, smallContrast(t), minerFreqFloor, minerSubsumeAlpha) + formants := detectFormants(chunks, va.candFreq, smallContrast(t), 3, 15.0) + if _, ok := formants['蛊']; !ok { + t.Fatalf("蛊 must be auto-detected as a productive formant, got %v", formantKeys(formants)) + } + if _, ok := formants['的']; ok { + t.Fatalf("的 (common char) must NOT be a formant") + } +} + +func TestMinerPalladiusToken(t *testing.T) { + // Positives: transliterated Chinese-name shapes segment cleanly. + for _, tok := range []string{"юань", "фан", "гу", "цзюй"} { + if !isPalladiusToken(tok, 1) { + t.Errorf("isPalladiusToken(%q) = false, want true (Palladius name shape)", tok) + } + } + // Negative control (palladius.py): common Russian words are NOT Palladius tokens. + for _, tok := range []string{"человек", "который", "деревня", "камень"} { + if isPalladiusToken(tok, 1) { + t.Errorf("isPalladiusToken(%q) = true, want false (common ru word)", tok) + } + } + // The «найти»=най+ти FALSE-POSITIVE the pymorphy3 is_name_lemma gate exists to block: it IS + // Palladius-shaped, which is exactly why default B drops the whole ru-side name confirmation channel + // (no morphology backend to run the gate) rather than trust the token shape alone. + if !isPalladiusToken("найти", 2) { + t.Fatalf("найти must be Palladius-shaped (documents why default-B drops the ru name channel)") + } +} + +// aliasSurf builds an aliasSurface for the rule tests. +func aliasSurf(src, typ, gender, approvedDst string) aliasSurface { + return aliasSurface{src: normalizeSourceKey(src), typ: typ, gender: gender, approvedDst: approvedDst} +} + +func TestMinerAliasRules(t *testing.T) { + seedSurf := func(ss ...string) map[string]bool { + m := map[string]bool{} + for _, s := range ss { + m[normalizeSourceKey(s)] = true + } + return m + } + hasIdent := func(edges []aliasEdge, a, b string) bool { + an, bn := normalizeSourceKey(a), normalizeSourceKey(b) + for _, e := range edges { + if (e.a == an && e.b == bn) || (e.a == bn && e.b == an) { + return true + } + } + return false + } + + // R1 extension: 方源 (seed entity) ⊂ 古月方源 → identity (same person, fuller surface). + t.Run("R1_extension", func(t *testing.T) { + surf := map[string]aliasSurface{ + normalizeSourceKey("方源"): aliasSurf("方源", "name", "", ""), + normalizeSourceKey("古月方源"): aliasSurf("古月方源", "name", "", ""), + } + ident, _, _ := proposeAliasEdges(surf, nil, seedSurf("方源")) + if !hasIdent(ident, "方源", "古月方源") { + t.Fatalf("方源 ⊂ 古月方源 must be an identity extension, got %+v", ident) + } + }) + + // Composite guard: 古月 (seed clan) + 族长 (seed title) = 古月族长 is a PHRASE, not an alias (blocks + // the clan↔title chaining). But 古月 + 方源 (name) stays a fullname alias. + t.Run("composite_guard", func(t *testing.T) { + surf := map[string]aliasSurface{ + normalizeSourceKey("古月"): aliasSurf("古月", "name", "", ""), + normalizeSourceKey("族长"): aliasSurf("族长", "title", "", ""), + normalizeSourceKey("古月族长"): aliasSurf("古月族长", "title", "", ""), + normalizeSourceKey("古月方源"): aliasSurf("古月方源", "name", "", ""), + normalizeSourceKey("方源"): aliasSurf("方源", "name", "", ""), + } + seeds := seedSurf("古月", "族长", "方源") + ident, _, weak := proposeAliasEdges(surf, nil, seeds) + if hasIdent(ident, "古月", "古月族长") { + t.Fatalf("古月+族长 (clan+title) must be a phrase, NOT an identity alias") + } + phraseFlagged := false + for _, e := range weak { + if e.kind == "phrase_not_alias" && e.a == normalizeSourceKey("古月") && e.b == normalizeSourceKey("古月族长") { + phraseFlagged = true + } + } + if !phraseFlagged { + t.Fatalf("古月+族长 must be flagged phrase_not_alias, got weak=%+v", weak) + } + if !hasIdent(ident, "古月", "古月方源") { + t.Fatalf("古月+方源 (clan+name) must stay a fullname identity alias, got %+v", ident) + } + }) + + // R4-iii/v: 族长 (approved «глава клана») ⊂ 四代族长 (approved «четвёртый глава клана») — DIFFERENT + // approved dst → different entities EVEN under containment (D38 §3). No identity edge. + t.Run("R4_different_approved_dst", func(t *testing.T) { + surf := map[string]aliasSurface{ + normalizeSourceKey("族长"): aliasSurf("族长", "title", "", "глава клана"), + normalizeSourceKey("四代族长"): aliasSurf("四代族长", "title", "", "четвёртый глава клана"), + } + ident, _, _ := proposeAliasEdges(surf, nil, seedSurf("族长", "四代族长")) + if hasIdent(ident, "族长", "四代族长") { + t.Fatalf("族长 vs 四代族长 with different approved dst must NOT be identity, got %+v", ident) + } + }) + + // R4-ii: different confirmed gender blocks identity; hidden never blocks. + t.Run("R4_gender", func(t *testing.T) { + surf := map[string]aliasSurface{ + normalizeSourceKey("方源"): aliasSurf("方源", "name", "male", ""), + normalizeSourceKey("方源儿"): aliasSurf("方源儿", "name", "female", ""), + } + ident, _, _ := proposeAliasEdges(surf, nil, seedSurf("方源")) + if hasIdent(ident, "方源", "方源儿") { + t.Fatalf("different confirmed gender must block identity, got %+v", ident) + } + }) + + // R4-i / R2: same surname, DIFFERENT given names → family, not identity (方源 vs 方正, surname 方). + t.Run("R2_family_R4i", func(t *testing.T) { + surf := map[string]aliasSurface{ + normalizeSourceKey("方源"): aliasSurf("方源", "name", "", ""), + normalizeSourceKey("方正"): aliasSurf("方正", "name", "", ""), + } + ident, family, _ := proposeAliasEdges(surf, nil, seedSurf("方源", "方正")) + if hasIdent(ident, "方源", "方正") { + t.Fatalf("same surname + different given names must be family, NOT identity") + } + if len(family) == 0 { + t.Fatalf("方源 ~ 方正 must be a family edge, got none") + } + }) +} + +func TestMinerEmissionFilters(t *testing.T) { + subsumed := map[string]bool{"花酒行": true} + seeds := map[string]bool{normalizeSourceKey("方源"): true} + cases := []struct { + name string + c ScoredCand + want bool + }{ + {"typed name freq5", ScoredCand{Src: "龙公", Types: []string{"name"}, Freq: 5}, true}, + {"below freq floor", ScoredCand{Src: "龙公", Types: []string{"name"}, Freq: 4}, false}, + {"untyped", ScoredCand{Src: "什么", Types: []string{"term"}, Freq: 20}, false}, + {"subsumed", ScoredCand{Src: "花酒行", Types: []string{"place"}, Freq: 9}, false}, + {"single char", ScoredCand{Src: "蛊", Types: []string{"term", "name"}, Freq: 99}, false}, + {"fragment", ScoredCand{Src: normalizeSourceKey("方源心"), Types: []string{"name"}, Freq: 9}, false}, + } + for _, tc := range cases { + if got := emissionEligible(tc.c, subsumed, seeds); got != tc.want { + t.Errorf("%s: emissionEligible = %v, want %v", tc.name, got, tc.want) + } + } +} + +func TestMineBankDeterministicAndNonSeed(t *testing.T) { + chunks := mchunks(strings.Repeat("龙公来到青茅山。龙公很强。青茅山很高。", 4)) + seed := []store.GlossaryEntry{{Src: "青茅山", Dst: "гора Цинмао", Type: "place", Status: "approved", Source: "seed"}} + cfg := frozenMinerConfig() + a := MineBank(chunks, smallContrast(t), seed, cfg) + b := MineBank(chunks, smallContrast(t), seed, cfg) + if !reflect.DeepEqual(a, b) { + t.Fatalf("MineBank must be deterministic byte-for-byte:\n a=%+v\n b=%+v", a, b) + } + // The seed surface 青茅山 must NOT be re-emitted as a new mined term (the curated entry owns it). + for _, m := range a { + if m.Src == normalizeSourceKey("青茅山") { + t.Fatalf("MineBank re-emitted a seed surface as a new term: %+v", m) + } + } +} + +func TestMinedToCandidatesStampsMinedSource(t *testing.T) { + mined := []MinedTerm{ + {Src: "龙公", Type: "name", SinceCh: 3, Freq: 7, Aliases: []string{"龙公子"}}, + {Src: "青茅山", Type: "place", SinceCh: 1, Freq: 9}, // covered by manual seed → skipped + } + manual := map[string]bool{"青茅山": true} + got := minedToCandidates(mined, manual) + if len(got) != 1 { + t.Fatalf("want 1 mined candidate (青茅山 skipped as manual), got %d", len(got)) + } + e := got[0] + if e.Source != "mined" { + t.Fatalf("mined candidate Source = %q, want \"mined\" (NOT seed — else it moves base-bank-version)", e.Source) + } + if e.Status != "auto" || e.Dst != "" { + t.Fatalf("mined candidate must be status=auto with no dst, got status=%q dst=%q", e.Status, e.Dst) + } + if e.SinceCh != 3 || e.Confidence != 7 { + t.Fatalf("mined candidate metadata wrong: %+v", e) + } + if len(e.Aliases) != 1 || e.Aliases[0].Alias != "龙公子" || e.Aliases[0].AliasType != "mined" { + t.Fatalf("mined alias wrong: %+v", e.Aliases) + } +} + +func TestMinedDeltaStaysBaseBankStable(t *testing.T) { + // The mined delta (Source:"mined") must NOT move BaseVersion — the load-bearing «переоплата ОДНА» + // invariant (§1в): a W1.5 mined-row addition moves only the enriched version. (Sibling of the + // Block-A TestBaseEnrichedMemoryVersionSplit, here through the real mined-write path.) + seed := []store.GlossaryEntry{{Src: "方源", Dst: "Фан Юань", Status: "approved", Source: "seed"}} + base0 := materializeMemory(seed, false) + mined := minedToCandidates([]MinedTerm{{Src: "龙公", Type: "name", SinceCh: 1, Freq: 5}}, map[string]bool{"方源": true}) + // A mined candidate is status=auto (not folded even into enriched when gate off) — promote it to + // approved to exercise the split, as the owner sign would at W1.5. + mined[0].Status = "approved" + mined[0].Dst = "Лун Гун" + enriched := append(append([]store.GlossaryEntry{}, seed...), mined...) + bank1 := materializeMemory(enriched, false) + if base0.BaseVersion() != bank1.BaseVersion() { + t.Fatalf("base-bank-version moved on a Source:mined addition — snapshot_W1 would re-bill W1") + } + if base0.Version() == bank1.Version() { + t.Fatalf("enriched version must move on a mined approved addition") + } +} + +func formantKeys(m map[rune]formantInfo) []string { + out := make([]string, 0, len(m)) + for r := range m { + out = append(out, string(r)) + } + return out +} diff --git a/backend/internal/pipeline/runner_test.go b/backend/internal/pipeline/runner_test.go index da42eb4..f17deef 100644 --- a/backend/internal/pipeline/runner_test.go +++ b/backend/internal/pipeline/runner_test.go @@ -111,6 +111,7 @@ type projectOpts struct { gatesYAML string // optional gates:/... block appended to pipeline.yaml glossarySeed string // optional glossary seed YAML content; "" = no glossary postcheckGate bool // enable the memory post-check hard gate (assumes gatesYAML is empty) + banknote bool // enable the banknote channel gate (WS4; assumes gatesYAML/postcheckGate empty) } func setupProjectOpts(t *testing.T, providerURL string, o projectOpts) string { @@ -149,6 +150,9 @@ models: if o.postcheckGate { gatesBlock += "\ngates:\n glossary:\n postcheck_gate: true\n" } + if o.banknote { + gatesBlock += "\ngates:\n banknote:\n enabled: true\n" + } writeFile(t, filepath.Join(dir, "pipeline.yaml"), fmt.Sprintf(` core: C1 version: 1 diff --git a/backend/internal/pipeline/seedlint_test.go b/backend/internal/pipeline/seedlint_test.go new file mode 100644 index 0000000..86d6c6e --- /dev/null +++ b/backend/internal/pipeline/seedlint_test.go @@ -0,0 +1,66 @@ +package pipeline + +import ( + "strings" + "testing" + + "textmachine/backend/internal/store" +) + +// seedlint_test.go: WS3 (д) `tmctl seed-lint` — the dry-run of the REAL loadGlossarySeed fail-louds over +// a seed YAML (the mined delta or a manual seed). Both directions: a clean seed lints OK; a defect fails +// loud; and the emitted mined delta (MinedDeltaYAML) round-trips through the real loader with 0 fails. + +func TestSeedLintCleanPasses(t *testing.T) { + p := writeSeed(t, ` +terms: + - src: 方源 + dst: Фан Юань + type: name + status: approved + - src: 蛊 + status: auto +`) + if err := SeedLint(p); err != nil { + t.Fatalf("a clean seed must lint OK, got: %v", err) + } +} + +func TestSeedLintCatchesDefects(t *testing.T) { + cases := []struct { + name, yaml, want string + }{ + {"approved without dst", "terms:\n - src: 方源\n status: approved\n", "non-empty dst"}, + {"unknown status", "terms:\n - src: 方源\n dst: X\n status: bogus\n", "status must be"}, + {"duplicate key", "terms:\n - src: 方源\n dst: A\n - src: 方源\n dst: B\n", "duplicate"}, + {"shared-key collision", "terms:\n - src: 老赵\n dst: Старина Чжао\n status: approved\n - src: 赵大\n dst: Чжао Да\n status: approved\n aliases:\n - alias: 老赵\n", "shared"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + p := writeSeed(t, tc.yaml) + err := SeedLint(p) + if err == nil { + t.Fatalf("%s: seed-lint must fail loud", tc.name) + } + if !strings.Contains(err.Error(), tc.want) { + t.Fatalf("%s: error %q must mention %q", tc.name, err.Error(), tc.want) + } + }) + } +} + +func TestSeedLintEmittedMinedDelta(t *testing.T) { + // The emitted mined delta must be loadable by the REAL loader (0 fail-louds) — the pre-condition for + // the W1.5 reseed. + chunks := mchunks(strings.Repeat("龙公来到青茅山。龙公很强。青茅山很高。", 4)) + seed := []store.GlossaryEntry{{Src: "青茅山", Dst: "гора Цинмао", Type: "place", Status: "approved", Source: "seed"}} + mined := MineBank(chunks, smallContrast(t), seed, frozenMinerConfig()) + yamlStr, err := MinedDeltaYAML(mined) + if err != nil { + t.Fatal(err) + } + p := writeSeed(t, yamlStr) + if err := SeedLint(p); err != nil { + t.Fatalf("the emitted mined delta must lint clean (loadable), got:\n%v\n--- delta ---\n%s", err, yamlStr) + } +} diff --git a/backend/internal/pipeline/snapshot.go b/backend/internal/pipeline/snapshot.go index 3102202..60c4b3a 100644 --- a/backend/internal/pipeline/snapshot.go +++ b/backend/internal/pipeline/snapshot.go @@ -4,6 +4,8 @@ import ( "crypto/sha256" "encoding/hex" "encoding/json" + + "textmachine/backend/internal/config" ) // snapshot.go: материализация snapshotID (§3.2/D5.2/D8) — контент-хеш всего, что @@ -97,6 +99,28 @@ func (r *Runner) sanitizerSnapshot() sanitizerSnap { return s } +// banknoteSnap freezes the banknote channel (WS4 §4в, F9) inside the snapshot — the VERDICT-AXIS that +// governs the resolved draft (the slice + the derived-checkpoint), mirroring sanitizerSnap. parser_version +// is the split/parse algorithm version: bumping it re-resolves the stripped draft, so it must be a loud +// --resnapshot even without a prompt edit (§4а point 6). DEVIATION-with-rationale from "mirror +// sanitizerSnap exactly": it is folded via a POINTER with omitempty (nil when the channel is off) rather +// than an unconditional {enabled:false} — so ADDING this optional feature does NOT invalidate every +// banknote-OFF book (incl. the golden fixture), while ENABLING the channel or bumping the parser is still +// a loud snapshot move (the substantive loudness point 6 requires). A snapshot re-pin is never billed for +// a feature a book does not use. +type banknoteSnap struct { + Enabled bool `json:"enabled"` + ParserVersion string `json:"parser_version,omitempty"` +} + +// banknoteSnapshot returns the fold ONLY when the channel is enabled (nil otherwise → omitempty drops it). +func (r *Runner) banknoteSnapshot() *banknoteSnap { + if !r.Pipeline.Gates.Banknote.Enabled { + return nil + } + return &banknoteSnap{Enabled: true, ParserVersion: bankParserVersion} +} + // memoryVersion is the content-hash of the deterministically materialized injected // memory (the frozen APPROVED glossary rows + the normalization/matcher algorithm // versions), the memory component of the snapshot (D5.2/D8, F1 CLOSED). It is the @@ -114,12 +138,65 @@ func (r *Runner) memoryVersion() string { return computeMemoryVersion(nil, r.Pipeline.Gates.Glossary.PostcheckGate) } -// snapshotID materializes the job-context snapshot (§3.2): brief_hash, chunker -// version, context-assembly knobs, the memory version and the full stage plan -// (model, prompt version + CONTENT hash, sampling, resolved capability). -// Payload is rendered with a fixed field order — the id is a content hash, so -// identical context re-upserts idempotently. +// wave selects which pipeline wave a per-wave snapshot folds (WS1 §1в). +type wave int + +const ( + waveW1 wave = iota // DRAFT wave: translator-role stages + the BASE bank version (excl Source:mined) + waveW2 // EDIT wave: non-translator stages + the ENRICHED bank version (all approved incl mined) +) + +// snapshotID materializes the BOOK-GLOBAL job-context snapshot (§3.2): brief_hash, chunker version, +// context-assembly knobs, the (enriched) memory version and the FULL stage plan. It is the snapshot the +// SEQUENTIAL driver pins every job to; the byte output is unchanged (buildSnapshotID over all stages + +// the enriched memory version reproduces it exactly). Payload is a fixed-field-order content hash. func (r *Runner) snapshotID() (id, payload string, err error) { + return r.buildSnapshotID(r.Pipeline.Stages, r.memoryVersion()) +} + +// snapshotIDForWave computes the PER-WAVE snapshot (WS1 §1в, ADDITIVE — the wave executor residual pins +// each wave's jobs to it; the sequential driver still uses snapshotID). W1 folds the DRAFT stages +// (translator role) + the BASE bank version (Source∈{seed,ruby,auto}, EXCL mined); W2 folds the EDIT +// stages (non-translator) + the ENRICHED bank version (all approved incl Source:mined). A W1.5 bank +// enrichment (adding mined rows) moves ONLY the enriched version → ONLY snapshot_W2, keeping W1 +// checkpoints valid («переоплата ОДНА»). Byte-consistent with snapshotID's building — the SAME payload +// struct, only the stage subset and the memory version differ. +func (r *Runner) snapshotIDForWave(w wave) (id, payload string, err error) { + memVer := r.memoryVersion() // enriched (W2) + if w == waveW1 { + memVer = r.baseMemoryVersion() + } + return r.buildSnapshotID(waveStages(r.Pipeline.Stages, w), memVer) +} + +// waveStages partitions the pipeline stages by role for a wave: W1 = the translator-role (draft) stages, +// W2 = every non-translator (editor/other) stage. Order preserved. The prod C1 core is one draft + one +// edit, so this yields [draft] and [edit] respectively. +func waveStages(stages []config.Stage, w wave) []config.Stage { + var out []config.Stage + for _, st := range stages { + isDraft := st.Role == roleTranslator + if (w == waveW1) == isDraft { + out = append(out, st) + } + } + return out +} + +// baseMemoryVersion is the DRAFT-wave (W1) memory component: the BASE bank version (excl Source:mined). +// nil bank → the empty-materialization base hash (a stable constant), mirroring memoryVersion(). +func (r *Runner) baseMemoryVersion() string { + if r.memory != nil { + return r.memory.BaseVersion() + } + return computeMemoryVersionScoped(nil, r.Pipeline.Gates.Glossary.PostcheckGate, true) +} + +// buildSnapshotID materializes the snapshot payload for a given stage subset + memory version (§3.2). +// Extracted from snapshotID so the book-global snapshot (all stages + enriched version) AND the two +// per-wave snapshots (WS1 §1в) share one byte-exact builder — a stage subset / memory version is the +// ONLY axis that differs. Fixed field order (a content hash). +func (r *Runner) buildSnapshotID(stages []config.Stage, memVersion string) (id, payload string, err error) { type stageSnap struct { Name string `json:"name"` Role string `json:"role"` @@ -238,7 +315,10 @@ func (r *Runner) snapshotID() (id, payload string, err error) { // but determines a checkpoint's RESOLVED verdict (a defect flips the chunk to flagged), so a // gate flip / rule edit is a loud --resnapshot. Folded only when enabled (sanitizerSnapshot). Sanitizer sanitizerSnap `json:"sanitizer"` - Stages []stageSnap `json:"stages"` + // Banknote — the banknote channel (WS4). Verdict-axis (governs the sliced/derived draft), folded + // only when ENABLED (nil→omitempty otherwise), so adding the feature never re-bills an off book. + Banknote *banknoteSnap `json:"banknote,omitempty"` + Stages []stageSnap `json:"stages"` }{ BriefHash: r.Book.BriefHash(), ChunkerVersion: chunkerVersion, @@ -255,13 +335,14 @@ func (r *Runner) snapshotID() (id, payload string, err error) { }, Segmentation: r.segmentationSnapshot(), RenderFormatVersion: renderFormatVersion, - MemoryVersion: r.memoryVersion(), + MemoryVersion: memVersion, PostcheckGate: r.Pipeline.Gates.Glossary.PostcheckGate, Coverage: r.coverageSnapshot(), StyleCheckVersion: cheapGateVersion, Sanitizer: r.sanitizerSnapshot(), + Banknote: r.banknoteSnapshot(), } - for _, st := range r.Pipeline.Stages { + for _, st := range stages { ss := stageSnap{ Name: st.Name, Role: st.Role, Model: st.Model, PromptVersion: st.PromptVersion, PromptSHA256: r.templates[st.Name].SHA256, diff --git a/backend/internal/pipeline/snapshot_wave_test.go b/backend/internal/pipeline/snapshot_wave_test.go new file mode 100644 index 0000000..f9d626b --- /dev/null +++ b/backend/internal/pipeline/snapshot_wave_test.go @@ -0,0 +1,71 @@ +package pipeline + +import ( + "strings" + "testing" + + "textmachine/backend/internal/store" +) + +// snapshot_wave_test.go: WS1 §1в per-wave snapshot mechanics (R1 ADDITIVE sub-step — the wave executor +// itself is residual; this pins the snapshot machinery it will use, WITHOUT switching the driver). The +// book-global snapshotID() is byte-unchanged (TestGolden proves it); these tests exercise the new +// snapshotIDForWave. +func TestSnapshotIDForWave(t *testing.T) { + r := newRunner(t, setupProject(t, "http://127.0.0.1:1")) + defer r.Close() + + seed := []store.GlossaryEntry{ + {Src: "方源", Dst: "Фан Юань", Status: "approved", Source: "seed"}, + {Src: "古月", Dst: "Гу Юэ", Status: "approved", Source: "ruby"}, + } + r.memory = materializeMemory(seed, false) + + w1id, w1p, err := r.snapshotIDForWave(waveW1) + if err != nil { + t.Fatal(err) + } + w2id, w2p, err := r.snapshotIDForWave(waveW2) + if err != nil { + t.Fatal(err) + } + // W1 and W2 differ (different stage subset AND different bank version). + if w1id == w2id { + t.Fatalf("snapshot_W1 must differ from snapshot_W2 (stage subset + bank version)") + } + // The stage PARTITION: W1 folds the draft (translator) stage, W2 folds the edit (editor) stage. + if !strings.Contains(w1p, `"name":"draft"`) || strings.Contains(w1p, `"name":"edit"`) { + t.Fatalf("snapshot_W1 must fold ONLY the draft stage, payload: %s", w1p) + } + if !strings.Contains(w2p, `"name":"edit"`) || strings.Contains(w2p, `"name":"draft"`) { + t.Fatalf("snapshot_W2 must fold ONLY the edit stage, payload: %s", w2p) + } + + // «Переоплата ОДНА» at the snapshot level (§1в): a W1.5 mined-approved addition moves ONLY W2. + enriched := append(append([]store.GlossaryEntry{}, seed...), store.GlossaryEntry{ + Src: "蛊", Dst: "гу", Status: "approved", Source: "mined", + }) + r.memory = materializeMemory(enriched, false) + w1id2, _, _ := r.snapshotIDForWave(waveW1) + w2id2, _, _ := r.snapshotIDForWave(waveW2) + if w1id2 != w1id { + t.Fatalf("snapshot_W1 moved on a mined-approved addition — W1 checkpoints would re-bill («переоплата ОДНА» broken)") + } + if w2id2 == w2id { + t.Fatalf("snapshot_W2 did NOT move on a mined-approved addition — the edit wave would miss the mined canon") + } +} + +// TestWaveStagesPartition pins the role partition independently of the runner. +func TestWaveStagesPartition(t *testing.T) { + r := newRunner(t, setupProject(t, "http://127.0.0.1:1")) + defer r.Close() + w1 := waveStages(r.Pipeline.Stages, waveW1) + w2 := waveStages(r.Pipeline.Stages, waveW2) + if len(w1) != 1 || w1[0].Role != roleTranslator { + t.Fatalf("W1 must be the translator stage(s), got %+v", w1) + } + if len(w2) != 1 || w2[0].Role != roleEditor { + t.Fatalf("W2 must be the editor stage(s), got %+v", w2) + } +} diff --git a/backend/internal/pipeline/stagerun.go b/backend/internal/pipeline/stagerun.go index a2b932b..a4da9a6 100644 --- a/backend/internal/pipeline/stagerun.go +++ b/backend/internal/pipeline/stagerun.go @@ -99,6 +99,13 @@ func (r *Runner) runStage(ctx context.Context, st config.Stage, stageIdx int, sn // inherited 2048/3291 and returned length. baseMaxTokens = r.applyModelFloor(baseMaxTokens, st.Model) + // Banknote (WS4 point 5): reserve room for the ≤12-line footnote block so a length cut hits the + // block (tolerated), not the translation. Translator-only + channel-enabled; wire-affecting + // (max_tokens ∈ request_hash) and reproducible on resume (banknoteSnap gates the enabled state). + if st.Role == roleTranslator && r.Pipeline.Gates.Banknote.Enabled { + baseMaxTokens += bankTokenBudget + } + maxRegen := r.Pipeline.Retries.RegenerateBeforeEscalate if maxRegen < 0 { maxRegen = 0 @@ -167,6 +174,19 @@ func (r *Runner) runStage(ctx context.Context, st config.Stage, stageIdx int, sn recovered := "" if disposition == DispOK { finalHash = last.reqHash + // Banknote OK-path (WS4 point 8, the main code change): if the translator draft carried a + // ⟦TM-BANK-v1⟧ block, commit the CLEANED draft as a $0 derived checkpoint and re-point final_hash + // at it — extending the flagged→ok derived-export precedent to the OK path — so the editor + the + // export read the cleaned draft on resume, never the raw footnote-bearing one. `last` is the + // terminal (possibly escalated) attempt whose block was sliced, so this runs AFTER the escalation + // resolve and consumes the escalated draft's stripped text (point 8's escalation-ordering note). + if last.bankStripped != "" { + dh, err := r.commitBanknoteExport(st, ch, job, last) + if err != nil { + return nil, err + } + finalHash = dh + } } else if last.cls.Reason == FlagSanitizerStripped { // Cosmetic leak (D35.4a): COMMIT the cleaned remainder as a $0 derived export checkpoint, then // point final_hash at it — so the standard final_hash→checkpoint.response_text export @@ -211,6 +231,7 @@ func (r *Runner) runStage(ctx context.Context, st config.Stage, stageIdx int, sn Attempts: attemptsMade, Escalated: escalated, EscalationModel: escModel, + BankFlags: last.bankFlags, } if disposition == DispOK { sr.Text = last.text @@ -243,20 +264,26 @@ func (r *Runner) commitSanitizedExport(st config.Stage, ch Chunk, job *store.Job // stageAttempt is the result of one attempt (a checkpoint hit or a fresh call). type stageAttempt struct { reqHash string + attempt int // the attempt index (for a derived checkpoint's Attempt column) cls classification // stripped is the resolved cosmetic-strip EXPORT text classifyOutput computed ONCE (T3.4): // non-empty only when cls.Reason == FlagSanitizerStripped. runStage ships it verbatim instead of // re-deriving stripCosmetic, so the "non-empty & clean" invariant classifyOutput asserts is the // exact text committed to the derived export checkpoint. - stripped string - text string - usage llm.Usage - finish string - modelActual string - latency int - runCost float64 // billed THIS run (0 on a checkpoint hit) - cumCost float64 // this attempt's cost (checkpoint cost on a hit, fresh cost on a call) - freshCall bool // a provider call was made this run + stripped string + // bankStripped is the banknote-CLEANED draft (WS4 point 8): non-empty only for a translator draft + // that carried a ⟦TM-BANK-v1⟧ block, when the channel is enabled. runStage commits it as a derived + // export checkpoint and re-points final_hash so the editor/export read the cleaned draft on resume. + bankStripped string + bankFlags bankFlags // per-chunk banknote telemetry (point 10) for the translator draft + text string + usage llm.Usage + finish string + modelActual string + latency int + runCost float64 // billed THIS run (0 on a checkpoint hit) + cumCost float64 // this attempt's cost (checkpoint cost on a hit, fresh cost on a call) + freshCall bool // a provider call was made this run } // runAttempt executes exactly one attempt on the request-hash axis: a checkpoint @@ -267,7 +294,7 @@ type stageAttempt struct { func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID string, ch Chunk, job *store.Job, attempt, maxTokens int, msgs []llm.Message, escalation, isFinal bool) (stageAttempt, error) { reqHash := RequestHash(r.Book.BookID, ch.Chapter, ch.ChunkIdx, attempt, st.Name, st.Role, model, st.Temperature, st.Reasoning, false, maxTokens, snapID, msgs) - att := stageAttempt{reqHash: reqHash, modelActual: model} + att := stageAttempt{reqHash: reqHash, attempt: attempt, modelActual: model} // Resume on the attempt axis: a checkpoint means THIS attempt already happened // and was billed — classify its text and never re-bill (kill -9 loses ≤1 call; @@ -282,11 +309,16 @@ func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID r.Log.DebugContext(ctx, "checkpoint usage_json unreadable; tokens report as zero", "hash", reqHash[:12], "err", uerr) } att.usage = usage - att.text = cp.ResponseText att.finish = cp.FinishReason att.modelActual = cp.ModelActual att.cumCost = cp.CostUSD - att.cls, att.stripped = r.classifyOutput(st.Role, ch.Text, cp.ResponseText, cp.FinishReason, isFinal) + // Banknote: slice the block off the RAW checkpoint text BEFORE classify + before feeding the + // editor (WS4 points 1-3,8). The checkpoint stores the RAW draft; resume re-derives the clean + // text deterministically (a no-op returning the raw text for a channel-off / no-separator draft). + cleanText, bankStripped, bankFlags := r.applyBanknote(st.Role, cp.ResponseText, cp.FinishReason) + att.text = cleanText + att.bankStripped, att.bankFlags = bankStripped, bankFlags + att.cls, att.stripped = r.classifyOutput(st.Role, ch.Text, cleanText, cp.FinishReason, isFinal) rl := r.baseRequestLog(st, ch, model, reqHash) rl.ModelActual = cp.ModelActual rl.TMHit, rl.OK, rl.FinishReason, rl.Degraded = true, att.cls.ok(), cp.FinishReason, degradedTag(att.cls) @@ -463,8 +495,13 @@ func (r *Runner) runAttempt(ctx context.Context, st config.Stage, model, snapID // Classify AFTER the money is durably settled+checkpointed. F4 lives here: a // non-empty truncated length draft is now classified (flagged/retried), never // silently passed downstream as OK; an empty completion is flagged too, not a - // run-crash. - att.cls, att.stripped = r.classifyOutput(st.Role, ch.Text, resp.Text, resp.FinishReason, isFinal) + // run-crash. The banknote block is sliced off the RAW draft FIRST (WS4 points 1-3,8): the + // checkpoint above holds the raw response, but classify/coverage/echo and the editor see the + // CLEANED translation, and the cleaned text is committed as a derived export in runStage. + cleanText, bankStripped, bankFlags := r.applyBanknote(st.Role, resp.Text, resp.FinishReason) + att.text = cleanText + att.bankStripped, att.bankFlags = bankStripped, bankFlags + att.cls, att.stripped = r.classifyOutput(st.Role, ch.Text, cleanText, resp.FinishReason, isFinal) rl := r.baseRequestLog(st, ch, model, reqHash) rl.ModelActual = modelActual diff --git a/backend/internal/pipeline/testdata/golden/capture.golden b/backend/internal/pipeline/testdata/golden/capture.golden index 5d0beeb..9a119d0 100644 --- a/backend/internal/pipeline/testdata/golden/capture.golden +++ b/backend/internal/pipeline/testdata/golden/capture.golden @@ -1,25 +1,25 @@ ==== run 1 (fresh) ==== -snapshot_id: 36dc3139d83a84b4b1aca5bb1b24287831aa2c4944192b7756b5c4df2f52fd8c -snapshot_payload: {"brief_hash":"c2021b1b5e6c20f6084e425b9a70bdc5f682801be347a57eb1c591deb8cab0df","chunker_version":"chunker-v5-output-budget-editunit","estimator_version":"estimator-v0","max_tokens_policy":"maxtok-v1-double-per-attempt","classifier_version":"classify-v1-refusal+echo015+loop","pipeline_core":"C1","max_output_ratio":2,"min_max_tokens":512,"context_assembly":{"glossary_injection":"selective","glossary_token_budget":800,"cache_ttl":""},"segmentation":{"draft_budget_out":1797,"edit_ceiling_out":3200,"fertility_cjk":1.1978,"fertility_other":0.3852},"render_format_version":"renderfmt-v1-editor-src2dst","memory_version":"636d2a6993285e433ce25bad2f5da01c1bcc19a4682dc9fe1efb15e5feb92f53","postcheck_gate":false,"coverage":{"enabled":false},"style_check_version":"cheapgate-v2","sanitizer":{"enabled":true,"version":"sanitizer-v6"},"stages":[{"name":"draft","role":"translator","model":"fake-model","prompt_version":"v-golden","prompt_sha256":"26d0245a56d180eafd200ebe5311aab2488142c4a98dcddee21ea457814aa21a","temperature":0.3,"reasoning":"off","capability":{"Budget":"max_tokens","Temp":"send","TempValue":0,"Reasoning":{"Control":"none","OffEffort":"","OffExtraBody":null,"OnExtraBody":null},"MinMaxTokens":4000},"escalate_to":"fake-fallback","escalate_capability":{"Budget":"max_tokens","Temp":"send","TempValue":0,"Reasoning":{"Control":"none","OffEffort":"","OffExtraBody":null,"OnExtraBody":null},"MinMaxTokens":6000}},{"name":"edit","role":"editor","model":"fake-model","prompt_version":"v-golden","prompt_sha256":"6a8e7f71139d6bc09a95159d1cb895e05b56d2f2f7417c5e20a7073c517c79f6","temperature":0.4,"reasoning":"off","capability":{"Budget":"max_tokens","Temp":"send","TempValue":0,"Reasoning":{"Control":"none","OffEffort":"","OffExtraBody":null,"OnExtraBody":null},"MinMaxTokens":4000}}]} +snapshot_id: 263b1c4baad731258ef34bcfee11c12de767737c3f282ec2561f66b8ce90f669 +snapshot_payload: {"brief_hash":"c2021b1b5e6c20f6084e425b9a70bdc5f682801be347a57eb1c591deb8cab0df","chunker_version":"chunker-v5-output-budget-editunit","estimator_version":"estimator-v0","max_tokens_policy":"maxtok-v1-double-per-attempt","classifier_version":"classify-v1-refusal+echo015+loop","pipeline_core":"C1","max_output_ratio":2,"min_max_tokens":512,"context_assembly":{"glossary_injection":"selective","glossary_token_budget":800,"cache_ttl":""},"segmentation":{"draft_budget_out":1797,"edit_ceiling_out":3200,"fertility_cjk":1.1978,"fertility_other":0.3852},"render_format_version":"renderfmt-v2-editor-src2dst+dc3-gender","memory_version":"636d2a6993285e433ce25bad2f5da01c1bcc19a4682dc9fe1efb15e5feb92f53","postcheck_gate":false,"coverage":{"enabled":false},"style_check_version":"cheapgate-v3-dc-checkers","sanitizer":{"enabled":true,"version":"sanitizer-v6"},"stages":[{"name":"draft","role":"translator","model":"fake-model","prompt_version":"v-golden","prompt_sha256":"26d0245a56d180eafd200ebe5311aab2488142c4a98dcddee21ea457814aa21a","temperature":0.3,"reasoning":"off","capability":{"Budget":"max_tokens","Temp":"send","TempValue":0,"Reasoning":{"Control":"none","OffEffort":"","OffExtraBody":null,"OnExtraBody":null},"MinMaxTokens":4000},"escalate_to":"fake-fallback","escalate_capability":{"Budget":"max_tokens","Temp":"send","TempValue":0,"Reasoning":{"Control":"none","OffEffort":"","OffExtraBody":null,"OnExtraBody":null},"MinMaxTokens":6000}},{"name":"edit","role":"editor","model":"fake-model","prompt_version":"v-golden","prompt_sha256":"6a8e7f71139d6bc09a95159d1cb895e05b56d2f2f7417c5e20a7073c517c79f6","temperature":0.4,"reasoning":"off","capability":{"Budget":"max_tokens","Temp":"send","TempValue":0,"Reasoning":{"Control":"none","OffEffort":"","OffExtraBody":null,"OnExtraBody":null},"MinMaxTokens":4000}}]} brief_hash: c2021b1b5e6c20f6084e425b9a70bdc5f682801be347a57eb1c591deb8cab0df memory_version: 636d2a6993285e433ce25bad2f5da01c1bcc19a4682dc9fe1efb15e5feb92f53 -- book result -- chunks=9 flagged=4 exit=2 total_usd=0.041859999999999994 -chunk ch1/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 745fead48c9f. Судзуки шёл по коридорам Академии магии." cost=0.00364 +chunk ch1/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 2de18cd1c124. Судзуки шёл по коридорам Академии магии." cost=0.00364 stage=draft role=translator model=fake-model resume=false disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" stage_text="ЧЕРНОВИК ПЕРЕВОДА a62fff0a94c2. Судзуки шёл по коридорам Академии магии." recovered="" stage=edit role=editor model=fake-model resume=false disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" - stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 745fead48c9f. Судзуки шёл по коридорам Академии магии." recovered="" -chunk ch1/1 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 560db1759bc2. Судзуки шёл по коридорам Академии магии." cost=0.00364 + stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 2de18cd1c124. Судзуки шёл по коридорам Академии магии." recovered="" +chunk ch1/1 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 838537705d66. Судзуки шёл по коридорам Академии магии." cost=0.00364 stage=draft role=translator model=fake-model resume=false disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" stage_text="ЧЕРНОВИК ПЕРЕВОДА f543c12a3d62. Судзуки шёл по коридорам Академии магии." recovered="" stage=edit role=editor model=fake-model resume=false disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" - stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 560db1759bc2. Судзуки шёл по коридорам Академии магии." recovered="" -chunk ch2/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 1ed88001fc0d. Судзуки шёл по коридорам Академии магии." cost=0.0091 + stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 838537705d66. Судзуки шёл по коридорам Академии магии." recovered="" +chunk ch2/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 5abc35ddfb65. Судзуки шёл по коридорам Академии магии." cost=0.0091 stage=draft role=translator model=fake-fallback resume=false disp=ok flag="" attempts=1 escalated=true esc_model="fake-fallback" finish="stop" cum_usd=0.00728 detail="" stage_text="ЭСКАЛАЦИОННЫЙ ПЕРЕВОД 0ab6e1368c17. Судзуки вынес Драконью печать из Академии магии." recovered="" stage=edit role=editor model=fake-model resume=false disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" - stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 1ed88001fc0d. Судзуки шёл по коридорам Академии магии." recovered="" + stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 5abc35ddfb65. Судзуки шёл по коридорам Академии магии." recovered="" chunk ch3/0 disposition=flagged flag="hard_refusal" final_text="" cost=0.00728 stage=draft role=translator model=fake-model resume=false disp=flagged flag="hard_refusal" attempts=1 escalated=true esc_model="fake-fallback" finish="refusal" cum_usd=0.00728 detail="provider finish_reason=refusal" stage_text="" recovered="" @@ -30,11 +30,11 @@ chunk ch4/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫ stage_text="ЧЕРНОВИК ПЕРЕВОДА 2eee3ef795d7. Судзуки шёл по коридорам Академии магии." recovered="" stage=edit role=editor model=fake-model resume=false disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 035dad4c3958. Судзуки шёл по коридорам Академии магии." recovered="" -chunk ch5/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД b086edefe69c. Судзуки увидел, как в тени ждал старик." cost=0.00364 +chunk ch5/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 546481fe42fe. Судзуки увидел, как в тени ждал старик." cost=0.00364 stage=draft role=translator model=fake-model resume=false disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" stage_text="ЧЕРНОВИК ПЕРЕВОДА bb921e6e678b. СТАРИК-СЦЕНА Судзуки увидел старика." recovered="" stage=edit role=editor model=fake-model resume=false disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" - stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД b086edefe69c. Судзуки увидел, как в тени ждал старик." recovered="" + stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 546481fe42fe. Судзуки увидел, как в тени ждал старик." recovered="" chunk ch6/0 disposition=flagged flag="sanitizer_defect" final_text="" cost=0.00364 stage=draft role=translator model=fake-model resume=false disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" stage_text="Вот перевод фрагмента: ПРЕАМБУЛА-СЦЕНА Судзуки открыл дверь." recovered="" @@ -51,44 +51,44 @@ chunk ch8/0 disposition=flagged flag="sanitizer_stripped" final_text="Судзу stage=edit role=editor model=fake-model resume=false disp=flagged flag="sanitizer_stripped" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="CJK-утечка в ru-выходе: 特产" stage_text="" recovered="Судзуки нашёл древний камень на каменном алтаре." -- chunk_status -- -ch1/0 draft snap_match=true content_hash=331c6f32c7f3ed15447e2639a2621d24a180e561a2aee92115324f492db5de3e disp=ok flag="" attempts=1 final_hash=2f3ac0b87516a3843e9ecb3e58d732fad5aa327a134d5222661ad87e5fc189b3 cost=0.00182 escalated=false esc_model="" detail="" -ch1/0 edit snap_match=true content_hash=32e9974cbc5a1df1384a9f79b74d930044996e300e987c61188c57e2f1cca8d6 disp=ok flag="" attempts=1 final_hash=8013609b4fa7e48edf7c2e2e4cc78f57ad39e28bb417ab184ecc9da5b1ad70a1 cost=0.00182 escalated=false esc_model="" detail="" -ch1/1 draft snap_match=true content_hash=12fe3ccdf062f43e011363a33ff85c9b6842a8f23d790991fe9afab51a8becc8 disp=ok flag="" attempts=1 final_hash=162d61c7ea223237705da4325094eeaaee2ef8b9d5a20d9876c523a7476779cd cost=0.00182 escalated=false esc_model="" detail="" -ch1/1 edit snap_match=true content_hash=64d62818b100f43f8c5230a3ee1b32f62189eab97e5cddd31e0710b357857cda disp=ok flag="" attempts=1 final_hash=d0cdedefb8f492af8a7a3a840af6afb7ef996973f40d15c0c67e0adb5ec36aa2 cost=0.00182 escalated=false esc_model="" detail="" -ch2/0 draft snap_match=true content_hash=2693325bc747687acf090bc263fe8c17783726b949c6d8a2d072da3401be6827 disp=ok flag="" attempts=1 final_hash=24a9cfda087e0cb0732b34b6b6d53fc4d0c70de5a73ffe6f9087402da1b14898 cost=0.00728 escalated=true esc_model="fake-fallback" detail="" -ch2/0 edit snap_match=true content_hash=8e1d544574e1709c45377cf8d781056ddf1833cf90cb738543689f4a887d2795 disp=ok flag="" attempts=1 final_hash=68473338f2542411f864f34e52e7712ccc1bc8d9a6164c939c3a3b103ca206e4 cost=0.00182 escalated=false esc_model="" detail="" +ch1/0 draft snap_match=true content_hash=331c6f32c7f3ed15447e2639a2621d24a180e561a2aee92115324f492db5de3e disp=ok flag="" attempts=1 final_hash=0dcdc0ec5804cb4724522fa5430c82e11e63fe268551831a525223e12d986773 cost=0.00182 escalated=false esc_model="" detail="" +ch1/0 edit snap_match=true content_hash=3c8d82d49806a85aee173bbbf0fb2136c73238056cab87df0127fdc7650a999c disp=ok flag="" attempts=1 final_hash=6eade8f5698c8eb88f5d1262a94056be4cae227b6ec56e1dda601eace14ee6e5 cost=0.00182 escalated=false esc_model="" detail="" +ch1/1 draft snap_match=true content_hash=12fe3ccdf062f43e011363a33ff85c9b6842a8f23d790991fe9afab51a8becc8 disp=ok flag="" attempts=1 final_hash=73d8ca5c2b4686c7af6325355e39dfa97b1c08514920cad5300bc227c3e67f7e cost=0.00182 escalated=false esc_model="" detail="" +ch1/1 edit snap_match=true content_hash=d80847cc32e983dd19a71c2972a58e1b31a87363adcec42fdccc14a21d0ac9dc disp=ok flag="" attempts=1 final_hash=0fb66113726b6614be47ce6c90a101cbc9bfce1d547479c27ecdabeab1ca8c02 cost=0.00182 escalated=false esc_model="" detail="" +ch2/0 draft snap_match=true content_hash=2693325bc747687acf090bc263fe8c17783726b949c6d8a2d072da3401be6827 disp=ok flag="" attempts=1 final_hash=75c33fde9dfdefa90047f6879603bafc2648a33023195937f3f716a58a3e0b4e cost=0.00728 escalated=true esc_model="fake-fallback" detail="" +ch2/0 edit snap_match=true content_hash=5a5397d41bcfa9326b5fe67c70aca8d9ef14ef89af2dbe1a758e25d364226fc5 disp=ok flag="" attempts=1 final_hash=f21f5490bcba6e6eb8810b2fbce38a9e0c91be87705326c8a642aab7a1405a22 cost=0.00182 escalated=false esc_model="" detail="" ch3/0 draft snap_match=true content_hash=733fc6a89215efe112cfadd1d1fb361bb8a7ab1013994248a49da0dbd137548e disp=flagged flag="hard_refusal" attempts=1 final_hash= cost=0.00728 escalated=true esc_model="fake-fallback" detail="provider finish_reason=refusal" ch3/0 edit snap_match=true content_hash= disp=skipped flag="hard_refusal" attempts=0 final_hash= cost=0 escalated=false esc_model="" detail="skipped: an upstream stage was flagged (hard_refusal)" -ch4/0 draft snap_match=true content_hash=7aabdcb89dea365c95363b2662144ca1a10b307d377f47a8e7cc19b5b872d287 disp=ok flag="" attempts=1 final_hash=e68690f3abc3ece2b5202cc10bbfc0a8dce2b79e69c705a73a33bbe801653b38 cost=0.00182 escalated=false esc_model="" detail="" -ch4/0 edit snap_match=true content_hash=2854dfe1648b0cf0cbb683cb7ce94c5eedc9223b2ce20c99dc008c7add6317ef disp=ok flag="" attempts=1 final_hash=2f8e3bcc66fa095ee443a6172193cf488140d53256e4f7514f1aca1d63b81395 cost=0.00182 escalated=false esc_model="" detail="" -ch5/0 draft snap_match=true content_hash=d8aba42cabb41853a81875ebc725bb516dc2412e29de3a03bc1a68e90f8cf157 disp=ok flag="" attempts=1 final_hash=a02083ed96d74f0ebf589ef4d27ab6109152baae9ffc437b1477aad4d5aa2d75 cost=0.00182 escalated=false esc_model="" detail="" -ch5/0 edit snap_match=true content_hash=10f63e759c9c0cb78b3c2ff51523308b3f9c6c49b5755c27c59c7c91546412d9 disp=ok flag="" attempts=1 final_hash=99d93354bd65f17d859ac749a19befc9c8d2c5cc82f858b6b484434790001603 cost=0.00182 escalated=false esc_model="" detail="" -ch6/0 draft snap_match=true content_hash=9375b05780724cc354eba15a595da9a5f964219c0887042484256eed11fb566f disp=ok flag="" attempts=1 final_hash=abec9d62bee829a2dd5af959a3fccd72fd6ebd968f265389b706854f3432dab1 cost=0.00182 escalated=false esc_model="" detail="" -ch6/0 edit snap_match=true content_hash=457fd53dd68c11665ea5d2ec489f966dd80e0232db5f10880cd13ebf865ef972 disp=flagged flag="sanitizer_defect" attempts=1 final_hash= cost=0.00182 escalated=false esc_model="" detail="ведущая служебная преамбула: Вот перевод фрагмента:" -ch7/0 draft snap_match=true content_hash=d90c335cba3d1547a22bb3f656443cb432f84c12a577d43509cfe472cf86a2ac disp=ok flag="" attempts=1 final_hash=2189c766693381a223938e3c8036b2c9bba19dc16e93d1a26671a5a397d577c0 cost=0.00182 escalated=false esc_model="" detail="" -ch7/0 edit snap_match=true content_hash=fe693bebbf5cfce3ff75d2756c511f4bd2a037256908ec4d758d9a6272535b5f disp=flagged flag="sanitizer_stripped" attempts=1 final_hash=tm-sanitized-v1:8ef72b20eac059984209728d34e0ef742b2fab2a8aef57a37dd5d5cf55545f4f cost=0.00182 escalated=false esc_model="" detail="markdown-заголовок в выходе: ### Глава 7" -ch8/0 draft snap_match=true content_hash=fbd27af11da2275cd1d47a0224970a5f8d073c6164f614efb6522ea96fa1e021 disp=ok flag="" attempts=1 final_hash=59118bacb3968e766e4c75d2ed770bc2a08d81231e4a01cf04e7c8fe7a722621 cost=0.00182 escalated=false esc_model="" detail="" -ch8/0 edit snap_match=true content_hash=d88a05fd553703e758d69278748152959d2c4b08d969d1099f1fce0d48476c1d disp=flagged flag="sanitizer_stripped" attempts=1 final_hash=tm-sanitized-v1:faf1ca5e8f561d183dad262f013b9820425201cdd0ac5682c54d20176f1f59d8 cost=0.00182 escalated=false esc_model="" detail="CJK-утечка в ru-выходе: 特产" +ch4/0 draft snap_match=true content_hash=7aabdcb89dea365c95363b2662144ca1a10b307d377f47a8e7cc19b5b872d287 disp=ok flag="" attempts=1 final_hash=368928e6b6ebc6422df723a726b99152b999e293dbcd2b38ae1a977d3e3740be cost=0.00182 escalated=false esc_model="" detail="" +ch4/0 edit snap_match=true content_hash=2854dfe1648b0cf0cbb683cb7ce94c5eedc9223b2ce20c99dc008c7add6317ef disp=ok flag="" attempts=1 final_hash=685dbe9923e245ceef1f0237ee7fbc7558fcf93590a88aef1bef310f83aaa8c2 cost=0.00182 escalated=false esc_model="" detail="" +ch5/0 draft snap_match=true content_hash=d8aba42cabb41853a81875ebc725bb516dc2412e29de3a03bc1a68e90f8cf157 disp=ok flag="" attempts=1 final_hash=59e252c7d50d6e2e146ef3bbcb6910e98824dfa68b13dd564791a54f1506f426 cost=0.00182 escalated=false esc_model="" detail="" +ch5/0 edit snap_match=true content_hash=82b05e4c3f5ca8432d7eaa7d7d63c4a7766f47a08398953c9a9bc3dc8437039b disp=ok flag="" attempts=1 final_hash=2916b384d1b23b5e9439fb9d9baab299d16898bf1e1c2dbe25f52c434b2212c0 cost=0.00182 escalated=false esc_model="" detail="" +ch6/0 draft snap_match=true content_hash=9375b05780724cc354eba15a595da9a5f964219c0887042484256eed11fb566f disp=ok flag="" attempts=1 final_hash=4bccc97fdab953a881aea64c33305d529d6f54692fd7048d455db543ae0af5bb cost=0.00182 escalated=false esc_model="" detail="" +ch6/0 edit snap_match=true content_hash=592353321cc8000e792e0897446e4dd4f88a0280b9368b62cbd5c82b0c433193 disp=flagged flag="sanitizer_defect" attempts=1 final_hash= cost=0.00182 escalated=false esc_model="" detail="ведущая служебная преамбула: Вот перевод фрагмента:" +ch7/0 draft snap_match=true content_hash=d90c335cba3d1547a22bb3f656443cb432f84c12a577d43509cfe472cf86a2ac disp=ok flag="" attempts=1 final_hash=64cc7c596cfb048cf48b2202d7dea01f07e7f3536884c93d2fc5cb5719abed9d cost=0.00182 escalated=false esc_model="" detail="" +ch7/0 edit snap_match=true content_hash=e12521b6eb6dbc2cd35758d664d4d31894c528f092083b5cfe0e79c51be2663e disp=flagged flag="sanitizer_stripped" attempts=1 final_hash=tm-sanitized-v1:2bef56171e2a1a6dd901a283e6d4f9f1f2aca106735e2a5ff638fbe366433c0a cost=0.00182 escalated=false esc_model="" detail="markdown-заголовок в выходе: ### Глава 7" +ch8/0 draft snap_match=true content_hash=fbd27af11da2275cd1d47a0224970a5f8d073c6164f614efb6522ea96fa1e021 disp=ok flag="" attempts=1 final_hash=bb2e0697351fc22258ff3f163d986a8ebff1bc6fff4ee5186b3cf3e579b222e6 cost=0.00182 escalated=false esc_model="" detail="" +ch8/0 edit snap_match=true content_hash=0c6030a943cb341390509e2ebed4bbfd52af328d36916ce2e341f43665cf9183 disp=flagged flag="sanitizer_stripped" attempts=1 final_hash=tm-sanitized-v1:d699a1e04616de214bf266a3e610ce060c5afc1fd640a537d19e49e8ad69c139 cost=0.00182 escalated=false esc_model="" detail="CJK-утечка в ru-выходе: 特产" -- request_log (insertion order) -- -ch1/0 draft role=translator req=fake-model actual=fake-model hash=2f3ac0b87516a3843e9ecb3e58d732fad5aa327a134d5222661ad87e5fc189b3 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch1/0 edit role=editor req=fake-model actual=fake-model hash=8013609b4fa7e48edf7c2e2e4cc78f57ad39e28bb417ab184ecc9da5b1ad70a1 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch1/1 draft role=translator req=fake-model actual=fake-model hash=162d61c7ea223237705da4325094eeaaee2ef8b9d5a20d9876c523a7476779cd tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch1/1 edit role=editor req=fake-model actual=fake-model hash=d0cdedefb8f492af8a7a3a840af6afb7ef996973f40d15c0c67e0adb5ec36aa2 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch2/0 draft role=translator req=fake-model actual=fake-model hash=98d51f7b179f0e736e347038a5bc8c5d5673adc93632dbcbdd026d26d090130e tm_hit=0 ok=0 finish="stop" degraded="cjk_artifact" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch2/0 draft role=translator req=fake-fallback actual=fake-fallback hash=24a9cfda087e0cb0732b34b6b6d53fc4d0c70de5a73ffe6f9087402da1b14898 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.0054600000000000004 tokens=1000/200/0/500/0 err="" -ch2/0 edit role=editor req=fake-model actual=fake-model hash=68473338f2542411f864f34e52e7712ccc1bc8d9a6164c939c3a3b103ca206e4 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch3/0 draft role=translator req=fake-model actual=fake-model hash=e6f9a3e8cd8a7d70444efc8ce55f961545e2c8f9c21b332bf62be6f7bac0187e tm_hit=0 ok=0 finish="refusal" degraded="hard_refusal" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch3/0 draft role=translator req=fake-fallback actual=fake-fallback hash=d437eebecbb9ea8b18b36096f4afc30d0a8da5d39071bc65777b2d3936b3cef7 tm_hit=0 ok=0 finish="refusal" degraded="hard_refusal" cost=0.0054600000000000004 tokens=1000/200/0/500/0 err="" -ch4/0 draft role=translator req=fake-model actual=fake-model hash=e68690f3abc3ece2b5202cc10bbfc0a8dce2b79e69c705a73a33bbe801653b38 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch4/0 edit role=editor req=fake-model actual=fake-model hash=2f8e3bcc66fa095ee443a6172193cf488140d53256e4f7514f1aca1d63b81395 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch5/0 draft role=translator req=fake-model actual=fake-model hash=a02083ed96d74f0ebf589ef4d27ab6109152baae9ffc437b1477aad4d5aa2d75 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch5/0 edit role=editor req=fake-model actual=fake-model hash=99d93354bd65f17d859ac749a19befc9c8d2c5cc82f858b6b484434790001603 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch6/0 draft role=translator req=fake-model actual=fake-model hash=abec9d62bee829a2dd5af959a3fccd72fd6ebd968f265389b706854f3432dab1 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch6/0 edit role=editor req=fake-model actual=fake-model hash=ef0f4b20ad89a3e5eacb7736083a9bedf598bdb2b0e904ab4f5af751ca16ae43 tm_hit=0 ok=0 finish="stop" degraded="sanitizer_defect" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch7/0 draft role=translator req=fake-model actual=fake-model hash=2189c766693381a223938e3c8036b2c9bba19dc16e93d1a26671a5a397d577c0 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch7/0 edit role=editor req=fake-model actual=fake-model hash=d102ddcda67e7971cbfbcf4ef31cfb403ec525cbaba5d1faabacc0cdc7953971 tm_hit=0 ok=0 finish="stop" degraded="sanitizer_stripped" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch8/0 draft role=translator req=fake-model actual=fake-model hash=59118bacb3968e766e4c75d2ed770bc2a08d81231e4a01cf04e7c8fe7a722621 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch8/0 edit role=editor req=fake-model actual=fake-model hash=3b705de15f457ced93f8649c995966183921fb25e8bfd4b1e37753ef267d3c47 tm_hit=0 ok=0 finish="stop" degraded="sanitizer_stripped" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch1/0 draft role=translator req=fake-model actual=fake-model hash=0dcdc0ec5804cb4724522fa5430c82e11e63fe268551831a525223e12d986773 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch1/0 edit role=editor req=fake-model actual=fake-model hash=6eade8f5698c8eb88f5d1262a94056be4cae227b6ec56e1dda601eace14ee6e5 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch1/1 draft role=translator req=fake-model actual=fake-model hash=73d8ca5c2b4686c7af6325355e39dfa97b1c08514920cad5300bc227c3e67f7e tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch1/1 edit role=editor req=fake-model actual=fake-model hash=0fb66113726b6614be47ce6c90a101cbc9bfce1d547479c27ecdabeab1ca8c02 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch2/0 draft role=translator req=fake-model actual=fake-model hash=62824a48a6143b1ee8d16aa167f45389050cae0b36488db3cf08dd5c75c44ced tm_hit=0 ok=0 finish="stop" degraded="cjk_artifact" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch2/0 draft role=translator req=fake-fallback actual=fake-fallback hash=75c33fde9dfdefa90047f6879603bafc2648a33023195937f3f716a58a3e0b4e tm_hit=0 ok=1 finish="stop" degraded="" cost=0.0054600000000000004 tokens=1000/200/0/500/0 err="" +ch2/0 edit role=editor req=fake-model actual=fake-model hash=f21f5490bcba6e6eb8810b2fbce38a9e0c91be87705326c8a642aab7a1405a22 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch3/0 draft role=translator req=fake-model actual=fake-model hash=eecbe6ed41153e35103989ba5dade978d20d97613c3b858fa2b24e20e8042523 tm_hit=0 ok=0 finish="refusal" degraded="hard_refusal" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch3/0 draft role=translator req=fake-fallback actual=fake-fallback hash=46a90aa896830bcc1c308d109d365d7b95f570ab0f347b3fd5895eb22cf71fe4 tm_hit=0 ok=0 finish="refusal" degraded="hard_refusal" cost=0.0054600000000000004 tokens=1000/200/0/500/0 err="" +ch4/0 draft role=translator req=fake-model actual=fake-model hash=368928e6b6ebc6422df723a726b99152b999e293dbcd2b38ae1a977d3e3740be tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch4/0 edit role=editor req=fake-model actual=fake-model hash=685dbe9923e245ceef1f0237ee7fbc7558fcf93590a88aef1bef310f83aaa8c2 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch5/0 draft role=translator req=fake-model actual=fake-model hash=59e252c7d50d6e2e146ef3bbcb6910e98824dfa68b13dd564791a54f1506f426 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch5/0 edit role=editor req=fake-model actual=fake-model hash=2916b384d1b23b5e9439fb9d9baab299d16898bf1e1c2dbe25f52c434b2212c0 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch6/0 draft role=translator req=fake-model actual=fake-model hash=4bccc97fdab953a881aea64c33305d529d6f54692fd7048d455db543ae0af5bb tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch6/0 edit role=editor req=fake-model actual=fake-model hash=0cb5a1c9a33dd9fe176d2b477f34de09681d8c191f6f8fb85f6fe4a2a60372bc tm_hit=0 ok=0 finish="stop" degraded="sanitizer_defect" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch7/0 draft role=translator req=fake-model actual=fake-model hash=64cc7c596cfb048cf48b2202d7dea01f07e7f3536884c93d2fc5cb5719abed9d tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch7/0 edit role=editor req=fake-model actual=fake-model hash=c1f1f7432def336d03507bca7cc63d79dba7e792b08820b63977f3cfeb2d3540 tm_hit=0 ok=0 finish="stop" degraded="sanitizer_stripped" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch8/0 draft role=translator req=fake-model actual=fake-model hash=bb2e0697351fc22258ff3f163d986a8ebff1bc6fff4ee5186b3cf3e579b222e6 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch8/0 edit role=editor req=fake-model actual=fake-model hash=012cf140c68edece8ad9819388854aef0002f34de9062746bda78c932353cf7f tm_hit=0 ok=0 finish="stop" degraded="sanitizer_stripped" cost=0.00182 tokens=1000/200/0/500/0 err="" -- retrieval_state -- ch1/0 snap_match=true exact=3 sticky=0 ambiguous=0 spoiler=1 evicted=0 postcheck_miss=1 style_flags=0 trust_gated=0 injected_ids=["紋章\u001f\u001f0\u001f0","鈴木\u001f\u001f0\u001f0","魔法学院\u001f\u001f0\u001f0"] postcheck_detail=[{"src":"紋章","dst":"герб","disp":"confirmed"}] style_detail= trust_gate_detail= @@ -110,46 +110,46 @@ ch8/0 snap_match=true exact=1 sticky=0 ambiguous=0 spoiler=0 evicted=0 postcheck injected_ids=["鈴木\u001f\u001f0\u001f0"] postcheck_detail= style_detail= trust_gate_detail= -- wire bodies (call order) -- [0] {"max_tokens":4000,"messages":[{"role":"system","content":"Переводи художественный текст с ja на ru. Жанр: ранобэ. Аудитория: взрослые. Баланс Венути: 0.50. Хонорифики: keep. Транскрипция: polivanov."},{"role":"system","content":"ГЛОССАРИЙ (используй эти утверждённые переводы имён и терминов последовательно; строки с пометкой ⟨проверить⟩ — неподтверждённые кандидаты):\n紋章 → герб\n鈴木 → Судзуки\n魔法学院 → Академия магии"},{"role":"user","content":"第一章　図書館の秘密\n\n一番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ一層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。鈴木の指先に刻まれた印は、書庫に近づくたびに淡く光った。司書たちはその光を恐れ、誰も彼に声をかけなかった。\n\n二番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ二層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。回廊の壁には歴代の院長の肖像が並び、その視線は生きているかのように彼の背中を追った。埃の匂いが記憶を刺激した。\n\n三番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ三層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。鈴木の指先に刻まれた印は、書庫に近づくたびに淡く光った。司書たちはその光を恐れ、誰も彼に声をかけなかった。\n\n四番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ四層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。回廊の壁には歴代の院長の肖像が並び、その視線は生きているかのように彼の背中を追った。埃の匂いが記憶を刺激した。\n\n五番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ五層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。鈴木の指先に刻まれた印は、書庫に近づくたびに淡く光った。司書たちはその光を恐れ、誰も彼に声をかけなかった。\n\n六番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ六層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。回廊の壁には歴代の院長の肖像が並び、その視線は生きているかのように彼の背中を追った。埃の匂いが記憶を刺激した。扉の奥で彼が見たものは、台座に置かれた竜の紋章だった。それは禁じられた歴史の最後の欠片だと、後に彼は知ることになる。\n\n七番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ七層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。鈴木の指先に刻まれた印は、書庫に近づくたびに淡く光った。司書たちはその光を恐れ、誰も彼に声をかけなかった。"}],"model":"fake-model","stream":false,"temperature":0.3} -[1] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 紋章 → «герб»\n- 鈴木 → «Судзуки»\n- 魔法学院 → «Академия магии»"},{"role":"user","content":"Черновик перевода для редактуры: ЧЕРНОВИК ПЕРЕВОДА a62fff0a94c2. Судзуки шёл по коридорам Академии магии."}],"model":"fake-model","stream":false,"temperature":0.4} +[1] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 紋章 → «герб»\n- 鈴木 → «Судзуки» (муж. — мужские родовые формы)\n- 魔法学院 → «Академия магии»"},{"role":"user","content":"Черновик перевода для редактуры: ЧЕРНОВИК ПЕРЕВОДА a62fff0a94c2. Судзуки шёл по коридорам Академии магии."}],"model":"fake-model","stream":false,"temperature":0.4} [2] {"max_tokens":4000,"messages":[{"role":"system","content":"Переводи художественный текст с ja на ru. Жанр: ранобэ. Аудитория: взрослые. Баланс Венути: 0.50. Хонорифики: keep. Транскрипция: polivanov."},{"role":"system","content":"ГЛОССАРИЙ (используй эти утверждённые переводы имён и терминов последовательно; строки с пометкой ⟨проверить⟩ — неподтверждённые кандидаты):\n鈴木 → Судзуки\n魔法学院 → Академия магии\n紋章 → герб"},{"role":"user","content":"八番目の朝、鈴木は魔法学院の長い廊下をゆっくりと歩きながら、昨夜の講義の内容を思い返していた。窓の外では銀色の雨が降り続き、石畳の中庭に小さな川を作っていた。図書館の塔は霧に包まれ、その頂は見えなかった。「今日こそ八層書庫の扉を開ける」と彼はつぶやき、胸元の古い鍵を握りしめた。鍵は冷たく、微かに震えているようだった。回廊の壁には歴代の院長の肖像が並び、その視線は生きているかのように彼の背中を追った。埃の匂いが記憶を刺激した。"}],"model":"fake-model","stream":false,"temperature":0.3} -[3] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 鈴木 → «Судзуки»\n- 魔法学院 → «Академия магии»\n- 紋章 → «герб»"},{"role":"user","content":"Черновик перевода для редактуры: ЧЕРНОВИК ПЕРЕВОДА f543c12a3d62. Судзуки шёл по коридорам Академии магии."}],"model":"fake-model","stream":false,"temperature":0.4} +[3] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 鈴木 → «Судзуки» (муж. — мужские родовые формы)\n- 魔法学院 → «Академия магии»\n- 紋章 → «герб»"},{"role":"user","content":"Черновик перевода для редактуры: ЧЕРНОВИК ПЕРЕВОДА f543c12a3d62. Судзуки шёл по коридорам Академии магии."}],"model":"fake-model","stream":false,"temperature":0.4} [4] {"max_tokens":4000,"messages":[{"role":"system","content":"Переводи художественный текст с ja на ru. Жанр: ранобэ. Аудитория: взрослые. Баланс Венути: 0.50. Хонорифики: keep. Транскрипция: polivanov."},{"role":"system","content":"ГЛОССАРИЙ (используй эти утверждённые переводы имён и терминов последовательно; строки с пометкой ⟨проверить⟩ — неподтверждённые кандидаты):\n竜の紋章 → Драконья печать\n鈴木 → Судзуки\n魔法学院 → Академия магии"},{"role":"user","content":"第二章　響動計画\n\n夜明け前、鈴木は再び書庫に戻り、竜の紋章を布に包んで持ち出した。魔法学院の鐘が三度鳴り、計画の始まりを告げた。\n\n「響動計画は今夜動き出す」と老司書が言った。その声は震えていたが、目は確かな決意に満ちていた。"}],"model":"fake-model","stream":false,"temperature":0.3} [5] {"max_tokens":6000,"messages":[{"role":"system","content":"Переводи художественный текст с ja на ru. Жанр: ранобэ. Аудитория: взрослые. Баланс Венути: 0.50. Хонорифики: keep. Транскрипция: polivanov."},{"role":"system","content":"ГЛОССАРИЙ (используй эти утверждённые переводы имён и терминов последовательно; строки с пометкой ⟨проверить⟩ — неподтверждённые кандидаты):\n竜の紋章 → Драконья печать\n鈴木 → Судзуки\n魔法学院 → Академия магии"},{"role":"user","content":"第二章　響動計画\n\n夜明け前、鈴木は再び書庫に戻り、竜の紋章を布に包んで持ち出した。魔法学院の鐘が三度鳴り、計画の始まりを告げた。\n\n「響動計画は今夜動き出す」と老司書が言った。その声は震えていたが、目は確かな決意に満ちていた。"}],"model":"fake-fallback","stream":false,"temperature":0.3} -[6] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 竜の紋章 → «Драконья печать»\n- 鈴木 → «Судзуки»\n- 魔法学院 → «Академия магии»"},{"role":"user","content":"Черновик перевода для редактуры: ЭСКАЛАЦИОННЫЙ ПЕРЕВОД 0ab6e1368c17. Судзуки вынес Драконью печать из Академии магии."}],"model":"fake-model","stream":false,"temperature":0.4} +[6] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 竜の紋章 → «Драконья печать»\n- 鈴木 → «Судзуки» (муж. — мужские родовые формы)\n- 魔法学院 → «Академия магии»"},{"role":"user","content":"Черновик перевода для редактуры: ЭСКАЛАЦИОННЫЙ ПЕРЕВОД 0ab6e1368c17. Судзуки вынес Драконью печать из Академии магии."}],"model":"fake-model","stream":false,"temperature":0.4} [7] {"max_tokens":4000,"messages":[{"role":"system","content":"Переводи художественный текст с ja на ru. Жанр: ранобэ. Аудитория: взрослые. Баланс Венути: 0.50. Хонорифики: keep. Транскрипция: polivanov."},{"role":"system","content":"ГЛОССАРИЙ (используй эти утверждённые переводы имён и терминов последовательно; строки с пометкой ⟨проверить⟩ — неподтверждённые кандидаты):\n鈴木 → Судзуки"},{"role":"user","content":"第三章　拒絶計画\n\nその手紙には拒絶計画という言葉だけが記されていた。鈴木は封を閉じ、暖炉の火にかざした。\n\n炎は紙を包み、言葉は灰になった。しかし灰の中で、何かが微かに光り続けていた。"}],"model":"fake-model","stream":false,"temperature":0.3} [8] {"max_tokens":6000,"messages":[{"role":"system","content":"Переводи художественный текст с ja на ru. Жанр: ранобэ. Аудитория: взрослые. Баланс Венути: 0.50. Хонорифики: keep. Транскрипция: polivanov."},{"role":"system","content":"ГЛОССАРИЙ (используй эти утверждённые переводы имён и терминов последовательно; строки с пометкой ⟨проверить⟩ — неподтверждённые кандидаты):\n鈴木 → Судзуки"},{"role":"user","content":"第三章　拒絶計画\n\nその手紙には拒絶計画という言葉だけが記されていた。鈴木は封を閉じ、暖炉の火にかざした。\n\n炎は紙を包み、言葉は灰になった。しかし灰の中で、何かが微かに光り続けていた。"}],"model":"fake-fallback","stream":false,"temperature":0.3} [9] {"max_tokens":4000,"messages":[{"role":"system","content":"Переводи художественный текст с ja на ru. Жанр: ранобэ. Аудитория: взрослые. Баланс Венути: 0.50. Хонорифики: keep. Транскрипция: polivanov."},{"role":"user","content":"第四章　静かな終わり\n\n夜が明けると、街は霧に包まれていた。人々は何も知らないまま、いつもの朝を迎えた。\n\n遠くの丘の上で、旅人がひとり、東の空を眺めていた。風は冷たく、道はまだ長い。"}],"model":"fake-model","stream":false,"temperature":0.3} [10] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"user","content":"Черновик перевода для редактуры: ЧЕРНОВИК ПЕРЕВОДА 2eee3ef795d7. Судзуки шёл по коридорам Академии магии."}],"model":"fake-model","stream":false,"temperature":0.4} [11] {"max_tokens":4000,"messages":[{"role":"system","content":"Переводи художественный текст с ja на ru. Жанр: ранобэ. Аудитория: взрослые. Баланс Венути: 0.50. Хонорифики: keep. Транскрипция: polivanov."},{"role":"system","content":"ГЛОССАРИЙ (используй эти утверждённые переводы имён и терминов последовательно; строки с пометкой ⟨проверить⟩ — неподтверждённые кандидаты):\n老人 → старик\n鈴木 → Судзуки"},{"role":"user","content":"第五章　老人の秘密\n\n鈴木は書庫の奥で老人に会った。老人は静かに古い巻物を差し出した。"}],"model":"fake-model","stream":false,"temperature":0.3} -[12] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 老人 → «старик»\n- 鈴木 → «Судзуки»"},{"role":"user","content":"Черновик перевода для редактуры: ЧЕРНОВИК ПЕРЕВОДА bb921e6e678b. СТАРИК-СЦЕНА Судзуки увидел старика."}],"model":"fake-model","stream":false,"temperature":0.4} +[12] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 老人 → «старик»\n- 鈴木 → «Судзуки» (муж. — мужские родовые формы)"},{"role":"user","content":"Черновик перевода для редактуры: ЧЕРНОВИК ПЕРЕВОДА bb921e6e678b. СТАРИК-СЦЕНА Судзуки увидел старика."}],"model":"fake-model","stream":false,"temperature":0.4} [13] {"max_tokens":4000,"messages":[{"role":"system","content":"Переводи художественный текст с ja на ru. Жанр: ранобэ. Аудитория: взрослые. Баланс Венути: 0.50. Хонорифики: keep. Транскрипция: polivanov."},{"role":"system","content":"ГЛОССАРИЙ (используй эти утверждённые переводы имён и терминов последовательно; строки с пометкой ⟨проверить⟩ — неподтверждённые кандидаты):\n鈴木 → Судзуки"},{"role":"user","content":"第六章　序文漏洩\n\n鈴木は最後の扉の前に立ち、深く息を吸い込んだ。その先に何があるのか、誰も知らなかった。"}],"model":"fake-model","stream":false,"temperature":0.3} -[14] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 鈴木 → «Судзуки»"},{"role":"user","content":"Черновик перевода для редактуры: Вот перевод фрагмента: ПРЕАМБУЛА-СЦЕНА Судзуки открыл дверь."}],"model":"fake-model","stream":false,"temperature":0.4} +[14] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 鈴木 → «Судзуки» (муж. — мужские родовые формы)"},{"role":"user","content":"Черновик перевода для редактуры: Вот перевод фрагмента: ПРЕАМБУЛА-СЦЕНА Судзуки открыл дверь."}],"model":"fake-model","stream":false,"temperature":0.4} [15] {"max_tokens":4000,"messages":[{"role":"system","content":"Переводи художественный текст с ja на ru. Жанр: ранобэ. Аудитория: взрослые. Баланс Венути: 0.50. Хонорифики: keep. Транскрипция: polivanov."},{"role":"system","content":"ГЛОССАРИЙ (используй эти утверждённые переводы имён и терминов последовательно; строки с пометкой ⟨проверить⟩ — неподтверждённые кандидаты):\n鈴木 → Судзуки"},{"role":"user","content":"第七章　見出漏洩\n\n七層目の朝、鈴木は最後の書庫の扉を開けた。扉の奥には見出しだけが刻まれた石板が静かに待っていた。誰もその文字を読めなかった。"}],"model":"fake-model","stream":false,"temperature":0.3} -[16] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 鈴木 → «Судзуки»"},{"role":"user","content":"Черновик перевода для редактуры: ЧЕРНОВИК ПЕРЕВОДА ec12f3d7a1df. МАРКДАУН-СЦЕНА Судзуки открыл седьмую дверь."}],"model":"fake-model","stream":false,"temperature":0.4} +[16] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 鈴木 → «Судзуки» (муж. — мужские родовые формы)"},{"role":"user","content":"Черновик перевода для редактуры: ЧЕРНОВИК ПЕРЕВОДА ec12f3d7a1df. МАРКДАУН-СЦЕНА Судзуки открыл седьмую дверь."}],"model":"fake-model","stream":false,"temperature":0.4} [17] {"max_tokens":4000,"messages":[{"role":"system","content":"Переводи художественный текст с ja на ru. Жанр: ранобэ. Аудитория: взрослые. Баланс Венути: 0.50. Хонорифики: keep. Транскрипция: polivanov."},{"role":"system","content":"ГЛОССАРИЙ (используй эти утверждённые переводы имён и терминов последовательно; строки с пометкой ⟨проверить⟩ — неподтверждённые кандидаты):\n鈴木 → Судзуки"},{"role":"user","content":"第八章　漢字漏洩\n\n鈴木は石板の文字をなぞり、古い漢字の意味を思い出そうとした。だが記憶は霧のように薄れ、指先だけが淡く光っていた。"}],"model":"fake-model","stream":false,"temperature":0.3} -[18] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 鈴木 → «Судзуки»"},{"role":"user","content":"Черновик перевода для редактуры: ЧЕРНОВИК ПЕРЕВОДА 0bc6a4817acb. ИЕРОГЛИФ-СЦЕНА Судзуки нашёл камень."}],"model":"fake-model","stream":false,"temperature":0.4} +[18] {"max_tokens":4000,"messages":[{"role":"system","content":"Ты — монолингвальный редактор русского текста. Правь стиль, не меняя смысла. Книга: Золотая книга."},{"role":"system","content":"КАНОНИЧЕСКИЕ ПЕРЕВОДЫ имён и терминов (в черновике термин исходника слева ДОЛЖЕН быть передан именно указанной формой справа — приводи к ней любые расхождения, склоняя по контексту; не вводи иных вариантов и не меняй ничего другого):\n- 鈴木 → «Судзуки» (муж. — мужские родовые формы)"},{"role":"user","content":"Черновик перевода для редактуры: ЧЕРНОВИК ПЕРЕВОДА 0bc6a4817acb. ИЕРОГЛИФ-СЦЕНА Судзуки нашёл камень."}],"model":"fake-model","stream":false,"temperature":0.4} ==== run 2 (resume) ==== -snapshot_id: 36dc3139d83a84b4b1aca5bb1b24287831aa2c4944192b7756b5c4df2f52fd8c -snapshot_payload: {"brief_hash":"c2021b1b5e6c20f6084e425b9a70bdc5f682801be347a57eb1c591deb8cab0df","chunker_version":"chunker-v5-output-budget-editunit","estimator_version":"estimator-v0","max_tokens_policy":"maxtok-v1-double-per-attempt","classifier_version":"classify-v1-refusal+echo015+loop","pipeline_core":"C1","max_output_ratio":2,"min_max_tokens":512,"context_assembly":{"glossary_injection":"selective","glossary_token_budget":800,"cache_ttl":""},"segmentation":{"draft_budget_out":1797,"edit_ceiling_out":3200,"fertility_cjk":1.1978,"fertility_other":0.3852},"render_format_version":"renderfmt-v1-editor-src2dst","memory_version":"636d2a6993285e433ce25bad2f5da01c1bcc19a4682dc9fe1efb15e5feb92f53","postcheck_gate":false,"coverage":{"enabled":false},"style_check_version":"cheapgate-v2","sanitizer":{"enabled":true,"version":"sanitizer-v6"},"stages":[{"name":"draft","role":"translator","model":"fake-model","prompt_version":"v-golden","prompt_sha256":"26d0245a56d180eafd200ebe5311aab2488142c4a98dcddee21ea457814aa21a","temperature":0.3,"reasoning":"off","capability":{"Budget":"max_tokens","Temp":"send","TempValue":0,"Reasoning":{"Control":"none","OffEffort":"","OffExtraBody":null,"OnExtraBody":null},"MinMaxTokens":4000},"escalate_to":"fake-fallback","escalate_capability":{"Budget":"max_tokens","Temp":"send","TempValue":0,"Reasoning":{"Control":"none","OffEffort":"","OffExtraBody":null,"OnExtraBody":null},"MinMaxTokens":6000}},{"name":"edit","role":"editor","model":"fake-model","prompt_version":"v-golden","prompt_sha256":"6a8e7f71139d6bc09a95159d1cb895e05b56d2f2f7417c5e20a7073c517c79f6","temperature":0.4,"reasoning":"off","capability":{"Budget":"max_tokens","Temp":"send","TempValue":0,"Reasoning":{"Control":"none","OffEffort":"","OffExtraBody":null,"OnExtraBody":null},"MinMaxTokens":4000}}]} +snapshot_id: 263b1c4baad731258ef34bcfee11c12de767737c3f282ec2561f66b8ce90f669 +snapshot_payload: {"brief_hash":"c2021b1b5e6c20f6084e425b9a70bdc5f682801be347a57eb1c591deb8cab0df","chunker_version":"chunker-v5-output-budget-editunit","estimator_version":"estimator-v0","max_tokens_policy":"maxtok-v1-double-per-attempt","classifier_version":"classify-v1-refusal+echo015+loop","pipeline_core":"C1","max_output_ratio":2,"min_max_tokens":512,"context_assembly":{"glossary_injection":"selective","glossary_token_budget":800,"cache_ttl":""},"segmentation":{"draft_budget_out":1797,"edit_ceiling_out":3200,"fertility_cjk":1.1978,"fertility_other":0.3852},"render_format_version":"renderfmt-v2-editor-src2dst+dc3-gender","memory_version":"636d2a6993285e433ce25bad2f5da01c1bcc19a4682dc9fe1efb15e5feb92f53","postcheck_gate":false,"coverage":{"enabled":false},"style_check_version":"cheapgate-v3-dc-checkers","sanitizer":{"enabled":true,"version":"sanitizer-v6"},"stages":[{"name":"draft","role":"translator","model":"fake-model","prompt_version":"v-golden","prompt_sha256":"26d0245a56d180eafd200ebe5311aab2488142c4a98dcddee21ea457814aa21a","temperature":0.3,"reasoning":"off","capability":{"Budget":"max_tokens","Temp":"send","TempValue":0,"Reasoning":{"Control":"none","OffEffort":"","OffExtraBody":null,"OnExtraBody":null},"MinMaxTokens":4000},"escalate_to":"fake-fallback","escalate_capability":{"Budget":"max_tokens","Temp":"send","TempValue":0,"Reasoning":{"Control":"none","OffEffort":"","OffExtraBody":null,"OnExtraBody":null},"MinMaxTokens":6000}},{"name":"edit","role":"editor","model":"fake-model","prompt_version":"v-golden","prompt_sha256":"6a8e7f71139d6bc09a95159d1cb895e05b56d2f2f7417c5e20a7073c517c79f6","temperature":0.4,"reasoning":"off","capability":{"Budget":"max_tokens","Temp":"send","TempValue":0,"Reasoning":{"Control":"none","OffEffort":"","OffExtraBody":null,"OnExtraBody":null},"MinMaxTokens":4000}}]} brief_hash: c2021b1b5e6c20f6084e425b9a70bdc5f682801be347a57eb1c591deb8cab0df memory_version: 636d2a6993285e433ce25bad2f5da01c1bcc19a4682dc9fe1efb15e5feb92f53 -- book result -- chunks=9 flagged=4 exit=2 total_usd=0 -chunk ch1/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 745fead48c9f. Судзуки шёл по коридорам Академии магии." cost=0 +chunk ch1/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 2de18cd1c124. Судзуки шёл по коридорам Академии магии." cost=0 stage=draft role=translator model=fake-model resume=true disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" stage_text="ЧЕРНОВИК ПЕРЕВОДА a62fff0a94c2. Судзуки шёл по коридорам Академии магии." recovered="" stage=edit role=editor model=fake-model resume=true disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" - stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 745fead48c9f. Судзуки шёл по коридорам Академии магии." recovered="" -chunk ch1/1 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 560db1759bc2. Судзуки шёл по коридорам Академии магии." cost=0 + stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 2de18cd1c124. Судзуки шёл по коридорам Академии магии." recovered="" +chunk ch1/1 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 838537705d66. Судзуки шёл по коридорам Академии магии." cost=0 stage=draft role=translator model=fake-model resume=true disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" stage_text="ЧЕРНОВИК ПЕРЕВОДА f543c12a3d62. Судзуки шёл по коридорам Академии магии." recovered="" stage=edit role=editor model=fake-model resume=true disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" - stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 560db1759bc2. Судзуки шёл по коридорам Академии магии." recovered="" -chunk ch2/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 1ed88001fc0d. Судзуки шёл по коридорам Академии магии." cost=0 + stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 838537705d66. Судзуки шёл по коридорам Академии магии." recovered="" +chunk ch2/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 5abc35ddfb65. Судзуки шёл по коридорам Академии магии." cost=0 stage=draft role=translator model=fake-fallback resume=true disp=ok flag="" attempts=1 escalated=true esc_model="fake-fallback" finish="stop" cum_usd=0.00728 detail="" stage_text="ЭСКАЛАЦИОННЫЙ ПЕРЕВОД 0ab6e1368c17. Судзуки вынес Драконью печать из Академии магии." recovered="" stage=edit role=editor model=fake-model resume=true disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" - stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 1ed88001fc0d. Судзуки шёл по коридорам Академии магии." recovered="" + stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 5abc35ddfb65. Судзуки шёл по коридорам Академии магии." recovered="" chunk ch3/0 disposition=flagged flag="hard_refusal" final_text="" cost=0 stage=draft role=translator model=fake-model resume=true disp=flagged flag="hard_refusal" attempts=1 escalated=true esc_model="fake-fallback" finish="" cum_usd=0.00728 detail="provider finish_reason=refusal" stage_text="" recovered="" @@ -160,11 +160,11 @@ chunk ch4/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫ stage_text="ЧЕРНОВИК ПЕРЕВОДА 2eee3ef795d7. Судзуки шёл по коридорам Академии магии." recovered="" stage=edit role=editor model=fake-model resume=true disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 035dad4c3958. Судзуки шёл по коридорам Академии магии." recovered="" -chunk ch5/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД b086edefe69c. Судзуки увидел, как в тени ждал старик." cost=0 +chunk ch5/0 disposition=ok flag="" final_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 546481fe42fe. Судзуки увидел, как в тени ждал старик." cost=0 stage=draft role=translator model=fake-model resume=true disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" stage_text="ЧЕРНОВИК ПЕРЕВОДА bb921e6e678b. СТАРИК-СЦЕНА Судзуки увидел старика." recovered="" stage=edit role=editor model=fake-model resume=true disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" - stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД b086edefe69c. Судзуки увидел, как в тени ждал старик." recovered="" + stage_text="ОТРЕДАКТИРОВАННЫЙ ПЕРЕВОД 546481fe42fe. Судзуки увидел, как в тени ждал старик." recovered="" chunk ch6/0 disposition=flagged flag="sanitizer_defect" final_text="" cost=0 stage=draft role=translator model=fake-model resume=true disp=ok flag="" attempts=1 escalated=false esc_model="" finish="stop" cum_usd=0.00182 detail="" stage_text="Вот перевод фрагмента: ПРЕАМБУЛА-СЦЕНА Судзуки открыл дверь." recovered="" @@ -181,61 +181,61 @@ chunk ch8/0 disposition=flagged flag="sanitizer_stripped" final_text="Судзу stage=edit role=editor model=fake-model resume=true disp=flagged flag="sanitizer_stripped" attempts=1 escalated=false esc_model="" finish="sanitized_export" cum_usd=0.00182 detail="CJK-утечка в ru-выходе: 特产" stage_text="" recovered="Судзуки нашёл древний камень на каменном алтаре." -- chunk_status -- -ch1/0 draft snap_match=true content_hash=331c6f32c7f3ed15447e2639a2621d24a180e561a2aee92115324f492db5de3e disp=ok flag="" attempts=1 final_hash=2f3ac0b87516a3843e9ecb3e58d732fad5aa327a134d5222661ad87e5fc189b3 cost=0.00182 escalated=false esc_model="" detail="" -ch1/0 edit snap_match=true content_hash=32e9974cbc5a1df1384a9f79b74d930044996e300e987c61188c57e2f1cca8d6 disp=ok flag="" attempts=1 final_hash=8013609b4fa7e48edf7c2e2e4cc78f57ad39e28bb417ab184ecc9da5b1ad70a1 cost=0.00182 escalated=false esc_model="" detail="" -ch1/1 draft snap_match=true content_hash=12fe3ccdf062f43e011363a33ff85c9b6842a8f23d790991fe9afab51a8becc8 disp=ok flag="" attempts=1 final_hash=162d61c7ea223237705da4325094eeaaee2ef8b9d5a20d9876c523a7476779cd cost=0.00182 escalated=false esc_model="" detail="" -ch1/1 edit snap_match=true content_hash=64d62818b100f43f8c5230a3ee1b32f62189eab97e5cddd31e0710b357857cda disp=ok flag="" attempts=1 final_hash=d0cdedefb8f492af8a7a3a840af6afb7ef996973f40d15c0c67e0adb5ec36aa2 cost=0.00182 escalated=false esc_model="" detail="" -ch2/0 draft snap_match=true content_hash=2693325bc747687acf090bc263fe8c17783726b949c6d8a2d072da3401be6827 disp=ok flag="" attempts=1 final_hash=24a9cfda087e0cb0732b34b6b6d53fc4d0c70de5a73ffe6f9087402da1b14898 cost=0.00728 escalated=true esc_model="fake-fallback" detail="" -ch2/0 edit snap_match=true content_hash=8e1d544574e1709c45377cf8d781056ddf1833cf90cb738543689f4a887d2795 disp=ok flag="" attempts=1 final_hash=68473338f2542411f864f34e52e7712ccc1bc8d9a6164c939c3a3b103ca206e4 cost=0.00182 escalated=false esc_model="" detail="" +ch1/0 draft snap_match=true content_hash=331c6f32c7f3ed15447e2639a2621d24a180e561a2aee92115324f492db5de3e disp=ok flag="" attempts=1 final_hash=0dcdc0ec5804cb4724522fa5430c82e11e63fe268551831a525223e12d986773 cost=0.00182 escalated=false esc_model="" detail="" +ch1/0 edit snap_match=true content_hash=3c8d82d49806a85aee173bbbf0fb2136c73238056cab87df0127fdc7650a999c disp=ok flag="" attempts=1 final_hash=6eade8f5698c8eb88f5d1262a94056be4cae227b6ec56e1dda601eace14ee6e5 cost=0.00182 escalated=false esc_model="" detail="" +ch1/1 draft snap_match=true content_hash=12fe3ccdf062f43e011363a33ff85c9b6842a8f23d790991fe9afab51a8becc8 disp=ok flag="" attempts=1 final_hash=73d8ca5c2b4686c7af6325355e39dfa97b1c08514920cad5300bc227c3e67f7e cost=0.00182 escalated=false esc_model="" detail="" +ch1/1 edit snap_match=true content_hash=d80847cc32e983dd19a71c2972a58e1b31a87363adcec42fdccc14a21d0ac9dc disp=ok flag="" attempts=1 final_hash=0fb66113726b6614be47ce6c90a101cbc9bfce1d547479c27ecdabeab1ca8c02 cost=0.00182 escalated=false esc_model="" detail="" +ch2/0 draft snap_match=true content_hash=2693325bc747687acf090bc263fe8c17783726b949c6d8a2d072da3401be6827 disp=ok flag="" attempts=1 final_hash=75c33fde9dfdefa90047f6879603bafc2648a33023195937f3f716a58a3e0b4e cost=0.00728 escalated=true esc_model="fake-fallback" detail="" +ch2/0 edit snap_match=true content_hash=5a5397d41bcfa9326b5fe67c70aca8d9ef14ef89af2dbe1a758e25d364226fc5 disp=ok flag="" attempts=1 final_hash=f21f5490bcba6e6eb8810b2fbce38a9e0c91be87705326c8a642aab7a1405a22 cost=0.00182 escalated=false esc_model="" detail="" ch3/0 draft snap_match=true content_hash=733fc6a89215efe112cfadd1d1fb361bb8a7ab1013994248a49da0dbd137548e disp=flagged flag="hard_refusal" attempts=1 final_hash= cost=0.00728 escalated=true esc_model="fake-fallback" detail="provider finish_reason=refusal" ch3/0 edit snap_match=true content_hash= disp=skipped flag="hard_refusal" attempts=0 final_hash= cost=0 escalated=false esc_model="" detail="skipped: an upstream stage was flagged (hard_refusal)" -ch4/0 draft snap_match=true content_hash=7aabdcb89dea365c95363b2662144ca1a10b307d377f47a8e7cc19b5b872d287 disp=ok flag="" attempts=1 final_hash=e68690f3abc3ece2b5202cc10bbfc0a8dce2b79e69c705a73a33bbe801653b38 cost=0.00182 escalated=false esc_model="" detail="" -ch4/0 edit snap_match=true content_hash=2854dfe1648b0cf0cbb683cb7ce94c5eedc9223b2ce20c99dc008c7add6317ef disp=ok flag="" attempts=1 final_hash=2f8e3bcc66fa095ee443a6172193cf488140d53256e4f7514f1aca1d63b81395 cost=0.00182 escalated=false esc_model="" detail="" -ch5/0 draft snap_match=true content_hash=d8aba42cabb41853a81875ebc725bb516dc2412e29de3a03bc1a68e90f8cf157 disp=ok flag="" attempts=1 final_hash=a02083ed96d74f0ebf589ef4d27ab6109152baae9ffc437b1477aad4d5aa2d75 cost=0.00182 escalated=false esc_model="" detail="" -ch5/0 edit snap_match=true content_hash=10f63e759c9c0cb78b3c2ff51523308b3f9c6c49b5755c27c59c7c91546412d9 disp=ok flag="" attempts=1 final_hash=99d93354bd65f17d859ac749a19befc9c8d2c5cc82f858b6b484434790001603 cost=0.00182 escalated=false esc_model="" detail="" -ch6/0 draft snap_match=true content_hash=9375b05780724cc354eba15a595da9a5f964219c0887042484256eed11fb566f disp=ok flag="" attempts=1 final_hash=abec9d62bee829a2dd5af959a3fccd72fd6ebd968f265389b706854f3432dab1 cost=0.00182 escalated=false esc_model="" detail="" -ch6/0 edit snap_match=true content_hash=457fd53dd68c11665ea5d2ec489f966dd80e0232db5f10880cd13ebf865ef972 disp=flagged flag="sanitizer_defect" attempts=1 final_hash= cost=0.00182 escalated=false esc_model="" detail="ведущая служебная преамбула: Вот перевод фрагмента:" -ch7/0 draft snap_match=true content_hash=d90c335cba3d1547a22bb3f656443cb432f84c12a577d43509cfe472cf86a2ac disp=ok flag="" attempts=1 final_hash=2189c766693381a223938e3c8036b2c9bba19dc16e93d1a26671a5a397d577c0 cost=0.00182 escalated=false esc_model="" detail="" -ch7/0 edit snap_match=true content_hash=fe693bebbf5cfce3ff75d2756c511f4bd2a037256908ec4d758d9a6272535b5f disp=flagged flag="sanitizer_stripped" attempts=1 final_hash=tm-sanitized-v1:8ef72b20eac059984209728d34e0ef742b2fab2a8aef57a37dd5d5cf55545f4f cost=0.00182 escalated=false esc_model="" detail="markdown-заголовок в выходе: ### Глава 7" -ch8/0 draft snap_match=true content_hash=fbd27af11da2275cd1d47a0224970a5f8d073c6164f614efb6522ea96fa1e021 disp=ok flag="" attempts=1 final_hash=59118bacb3968e766e4c75d2ed770bc2a08d81231e4a01cf04e7c8fe7a722621 cost=0.00182 escalated=false esc_model="" detail="" -ch8/0 edit snap_match=true content_hash=d88a05fd553703e758d69278748152959d2c4b08d969d1099f1fce0d48476c1d disp=flagged flag="sanitizer_stripped" attempts=1 final_hash=tm-sanitized-v1:faf1ca5e8f561d183dad262f013b9820425201cdd0ac5682c54d20176f1f59d8 cost=0.00182 escalated=false esc_model="" detail="CJK-утечка в ru-выходе: 特产" +ch4/0 draft snap_match=true content_hash=7aabdcb89dea365c95363b2662144ca1a10b307d377f47a8e7cc19b5b872d287 disp=ok flag="" attempts=1 final_hash=368928e6b6ebc6422df723a726b99152b999e293dbcd2b38ae1a977d3e3740be cost=0.00182 escalated=false esc_model="" detail="" +ch4/0 edit snap_match=true content_hash=2854dfe1648b0cf0cbb683cb7ce94c5eedc9223b2ce20c99dc008c7add6317ef disp=ok flag="" attempts=1 final_hash=685dbe9923e245ceef1f0237ee7fbc7558fcf93590a88aef1bef310f83aaa8c2 cost=0.00182 escalated=false esc_model="" detail="" +ch5/0 draft snap_match=true content_hash=d8aba42cabb41853a81875ebc725bb516dc2412e29de3a03bc1a68e90f8cf157 disp=ok flag="" attempts=1 final_hash=59e252c7d50d6e2e146ef3bbcb6910e98824dfa68b13dd564791a54f1506f426 cost=0.00182 escalated=false esc_model="" detail="" +ch5/0 edit snap_match=true content_hash=82b05e4c3f5ca8432d7eaa7d7d63c4a7766f47a08398953c9a9bc3dc8437039b disp=ok flag="" attempts=1 final_hash=2916b384d1b23b5e9439fb9d9baab299d16898bf1e1c2dbe25f52c434b2212c0 cost=0.00182 escalated=false esc_model="" detail="" +ch6/0 draft snap_match=true content_hash=9375b05780724cc354eba15a595da9a5f964219c0887042484256eed11fb566f disp=ok flag="" attempts=1 final_hash=4bccc97fdab953a881aea64c33305d529d6f54692fd7048d455db543ae0af5bb cost=0.00182 escalated=false esc_model="" detail="" +ch6/0 edit snap_match=true content_hash=592353321cc8000e792e0897446e4dd4f88a0280b9368b62cbd5c82b0c433193 disp=flagged flag="sanitizer_defect" attempts=1 final_hash= cost=0.00182 escalated=false esc_model="" detail="ведущая служебная преамбула: Вот перевод фрагмента:" +ch7/0 draft snap_match=true content_hash=d90c335cba3d1547a22bb3f656443cb432f84c12a577d43509cfe472cf86a2ac disp=ok flag="" attempts=1 final_hash=64cc7c596cfb048cf48b2202d7dea01f07e7f3536884c93d2fc5cb5719abed9d cost=0.00182 escalated=false esc_model="" detail="" +ch7/0 edit snap_match=true content_hash=e12521b6eb6dbc2cd35758d664d4d31894c528f092083b5cfe0e79c51be2663e disp=flagged flag="sanitizer_stripped" attempts=1 final_hash=tm-sanitized-v1:2bef56171e2a1a6dd901a283e6d4f9f1f2aca106735e2a5ff638fbe366433c0a cost=0.00182 escalated=false esc_model="" detail="markdown-заголовок в выходе: ### Глава 7" +ch8/0 draft snap_match=true content_hash=fbd27af11da2275cd1d47a0224970a5f8d073c6164f614efb6522ea96fa1e021 disp=ok flag="" attempts=1 final_hash=bb2e0697351fc22258ff3f163d986a8ebff1bc6fff4ee5186b3cf3e579b222e6 cost=0.00182 escalated=false esc_model="" detail="" +ch8/0 edit snap_match=true content_hash=0c6030a943cb341390509e2ebed4bbfd52af328d36916ce2e341f43665cf9183 disp=flagged flag="sanitizer_stripped" attempts=1 final_hash=tm-sanitized-v1:d699a1e04616de214bf266a3e610ce060c5afc1fd640a537d19e49e8ad69c139 cost=0.00182 escalated=false esc_model="" detail="CJK-утечка в ru-выходе: 特产" -- request_log (insertion order) -- -ch1/0 draft role=translator req=fake-model actual=fake-model hash=2f3ac0b87516a3843e9ecb3e58d732fad5aa327a134d5222661ad87e5fc189b3 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch1/0 edit role=editor req=fake-model actual=fake-model hash=8013609b4fa7e48edf7c2e2e4cc78f57ad39e28bb417ab184ecc9da5b1ad70a1 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch1/1 draft role=translator req=fake-model actual=fake-model hash=162d61c7ea223237705da4325094eeaaee2ef8b9d5a20d9876c523a7476779cd tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch1/1 edit role=editor req=fake-model actual=fake-model hash=d0cdedefb8f492af8a7a3a840af6afb7ef996973f40d15c0c67e0adb5ec36aa2 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch2/0 draft role=translator req=fake-model actual=fake-model hash=98d51f7b179f0e736e347038a5bc8c5d5673adc93632dbcbdd026d26d090130e tm_hit=0 ok=0 finish="stop" degraded="cjk_artifact" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch2/0 draft role=translator req=fake-fallback actual=fake-fallback hash=24a9cfda087e0cb0732b34b6b6d53fc4d0c70de5a73ffe6f9087402da1b14898 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.0054600000000000004 tokens=1000/200/0/500/0 err="" -ch2/0 edit role=editor req=fake-model actual=fake-model hash=68473338f2542411f864f34e52e7712ccc1bc8d9a6164c939c3a3b103ca206e4 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch3/0 draft role=translator req=fake-model actual=fake-model hash=e6f9a3e8cd8a7d70444efc8ce55f961545e2c8f9c21b332bf62be6f7bac0187e tm_hit=0 ok=0 finish="refusal" degraded="hard_refusal" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch3/0 draft role=translator req=fake-fallback actual=fake-fallback hash=d437eebecbb9ea8b18b36096f4afc30d0a8da5d39071bc65777b2d3936b3cef7 tm_hit=0 ok=0 finish="refusal" degraded="hard_refusal" cost=0.0054600000000000004 tokens=1000/200/0/500/0 err="" -ch4/0 draft role=translator req=fake-model actual=fake-model hash=e68690f3abc3ece2b5202cc10bbfc0a8dce2b79e69c705a73a33bbe801653b38 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch4/0 edit role=editor req=fake-model actual=fake-model hash=2f8e3bcc66fa095ee443a6172193cf488140d53256e4f7514f1aca1d63b81395 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch5/0 draft role=translator req=fake-model actual=fake-model hash=a02083ed96d74f0ebf589ef4d27ab6109152baae9ffc437b1477aad4d5aa2d75 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch5/0 edit role=editor req=fake-model actual=fake-model hash=99d93354bd65f17d859ac749a19befc9c8d2c5cc82f858b6b484434790001603 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch6/0 draft role=translator req=fake-model actual=fake-model hash=abec9d62bee829a2dd5af959a3fccd72fd6ebd968f265389b706854f3432dab1 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch6/0 edit role=editor req=fake-model actual=fake-model hash=ef0f4b20ad89a3e5eacb7736083a9bedf598bdb2b0e904ab4f5af751ca16ae43 tm_hit=0 ok=0 finish="stop" degraded="sanitizer_defect" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch7/0 draft role=translator req=fake-model actual=fake-model hash=2189c766693381a223938e3c8036b2c9bba19dc16e93d1a26671a5a397d577c0 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch7/0 edit role=editor req=fake-model actual=fake-model hash=d102ddcda67e7971cbfbcf4ef31cfb403ec525cbaba5d1faabacc0cdc7953971 tm_hit=0 ok=0 finish="stop" degraded="sanitizer_stripped" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch8/0 draft role=translator req=fake-model actual=fake-model hash=59118bacb3968e766e4c75d2ed770bc2a08d81231e4a01cf04e7c8fe7a722621 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch8/0 edit role=editor req=fake-model actual=fake-model hash=3b705de15f457ced93f8649c995966183921fb25e8bfd4b1e37753ef267d3c47 tm_hit=0 ok=0 finish="stop" degraded="sanitizer_stripped" cost=0.00182 tokens=1000/200/0/500/0 err="" -ch1/0 draft role=translator req=fake-model actual=fake-model hash=2f3ac0b87516a3843e9ecb3e58d732fad5aa327a134d5222661ad87e5fc189b3 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" -ch1/0 edit role=editor req=fake-model actual=fake-model hash=8013609b4fa7e48edf7c2e2e4cc78f57ad39e28bb417ab184ecc9da5b1ad70a1 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" -ch1/1 draft role=translator req=fake-model actual=fake-model hash=162d61c7ea223237705da4325094eeaaee2ef8b9d5a20d9876c523a7476779cd tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" -ch1/1 edit role=editor req=fake-model actual=fake-model hash=d0cdedefb8f492af8a7a3a840af6afb7ef996973f40d15c0c67e0adb5ec36aa2 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" -ch2/0 draft role=translator req=fake-model actual=fake-fallback hash=24a9cfda087e0cb0732b34b6b6d53fc4d0c70de5a73ffe6f9087402da1b14898 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" -ch2/0 edit role=editor req=fake-model actual=fake-model hash=68473338f2542411f864f34e52e7712ccc1bc8d9a6164c939c3a3b103ca206e4 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch1/0 draft role=translator req=fake-model actual=fake-model hash=0dcdc0ec5804cb4724522fa5430c82e11e63fe268551831a525223e12d986773 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch1/0 edit role=editor req=fake-model actual=fake-model hash=6eade8f5698c8eb88f5d1262a94056be4cae227b6ec56e1dda601eace14ee6e5 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch1/1 draft role=translator req=fake-model actual=fake-model hash=73d8ca5c2b4686c7af6325355e39dfa97b1c08514920cad5300bc227c3e67f7e tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch1/1 edit role=editor req=fake-model actual=fake-model hash=0fb66113726b6614be47ce6c90a101cbc9bfce1d547479c27ecdabeab1ca8c02 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch2/0 draft role=translator req=fake-model actual=fake-model hash=62824a48a6143b1ee8d16aa167f45389050cae0b36488db3cf08dd5c75c44ced tm_hit=0 ok=0 finish="stop" degraded="cjk_artifact" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch2/0 draft role=translator req=fake-fallback actual=fake-fallback hash=75c33fde9dfdefa90047f6879603bafc2648a33023195937f3f716a58a3e0b4e tm_hit=0 ok=1 finish="stop" degraded="" cost=0.0054600000000000004 tokens=1000/200/0/500/0 err="" +ch2/0 edit role=editor req=fake-model actual=fake-model hash=f21f5490bcba6e6eb8810b2fbce38a9e0c91be87705326c8a642aab7a1405a22 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch3/0 draft role=translator req=fake-model actual=fake-model hash=eecbe6ed41153e35103989ba5dade978d20d97613c3b858fa2b24e20e8042523 tm_hit=0 ok=0 finish="refusal" degraded="hard_refusal" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch3/0 draft role=translator req=fake-fallback actual=fake-fallback hash=46a90aa896830bcc1c308d109d365d7b95f570ab0f347b3fd5895eb22cf71fe4 tm_hit=0 ok=0 finish="refusal" degraded="hard_refusal" cost=0.0054600000000000004 tokens=1000/200/0/500/0 err="" +ch4/0 draft role=translator req=fake-model actual=fake-model hash=368928e6b6ebc6422df723a726b99152b999e293dbcd2b38ae1a977d3e3740be tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch4/0 edit role=editor req=fake-model actual=fake-model hash=685dbe9923e245ceef1f0237ee7fbc7558fcf93590a88aef1bef310f83aaa8c2 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch5/0 draft role=translator req=fake-model actual=fake-model hash=59e252c7d50d6e2e146ef3bbcb6910e98824dfa68b13dd564791a54f1506f426 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch5/0 edit role=editor req=fake-model actual=fake-model hash=2916b384d1b23b5e9439fb9d9baab299d16898bf1e1c2dbe25f52c434b2212c0 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch6/0 draft role=translator req=fake-model actual=fake-model hash=4bccc97fdab953a881aea64c33305d529d6f54692fd7048d455db543ae0af5bb tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch6/0 edit role=editor req=fake-model actual=fake-model hash=0cb5a1c9a33dd9fe176d2b477f34de09681d8c191f6f8fb85f6fe4a2a60372bc tm_hit=0 ok=0 finish="stop" degraded="sanitizer_defect" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch7/0 draft role=translator req=fake-model actual=fake-model hash=64cc7c596cfb048cf48b2202d7dea01f07e7f3536884c93d2fc5cb5719abed9d tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch7/0 edit role=editor req=fake-model actual=fake-model hash=c1f1f7432def336d03507bca7cc63d79dba7e792b08820b63977f3cfeb2d3540 tm_hit=0 ok=0 finish="stop" degraded="sanitizer_stripped" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch8/0 draft role=translator req=fake-model actual=fake-model hash=bb2e0697351fc22258ff3f163d986a8ebff1bc6fff4ee5186b3cf3e579b222e6 tm_hit=0 ok=1 finish="stop" degraded="" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch8/0 edit role=editor req=fake-model actual=fake-model hash=012cf140c68edece8ad9819388854aef0002f34de9062746bda78c932353cf7f tm_hit=0 ok=0 finish="stop" degraded="sanitizer_stripped" cost=0.00182 tokens=1000/200/0/500/0 err="" +ch1/0 draft role=translator req=fake-model actual=fake-model hash=0dcdc0ec5804cb4724522fa5430c82e11e63fe268551831a525223e12d986773 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch1/0 edit role=editor req=fake-model actual=fake-model hash=6eade8f5698c8eb88f5d1262a94056be4cae227b6ec56e1dda601eace14ee6e5 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch1/1 draft role=translator req=fake-model actual=fake-model hash=73d8ca5c2b4686c7af6325355e39dfa97b1c08514920cad5300bc227c3e67f7e tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch1/1 edit role=editor req=fake-model actual=fake-model hash=0fb66113726b6614be47ce6c90a101cbc9bfce1d547479c27ecdabeab1ca8c02 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch2/0 draft role=translator req=fake-model actual=fake-fallback hash=75c33fde9dfdefa90047f6879603bafc2648a33023195937f3f716a58a3e0b4e tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch2/0 edit role=editor req=fake-model actual=fake-model hash=f21f5490bcba6e6eb8810b2fbce38a9e0c91be87705326c8a642aab7a1405a22 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" ch3/0 draft role=translator req=fake-model actual=fake-model hash= tm_hit=1 ok=0 finish="" degraded="hard_refusal" cost=0 tokens=0/0/0/0/0 err="" -ch4/0 draft role=translator req=fake-model actual=fake-model hash=e68690f3abc3ece2b5202cc10bbfc0a8dce2b79e69c705a73a33bbe801653b38 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" -ch4/0 edit role=editor req=fake-model actual=fake-model hash=2f8e3bcc66fa095ee443a6172193cf488140d53256e4f7514f1aca1d63b81395 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" -ch5/0 draft role=translator req=fake-model actual=fake-model hash=a02083ed96d74f0ebf589ef4d27ab6109152baae9ffc437b1477aad4d5aa2d75 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" -ch5/0 edit role=editor req=fake-model actual=fake-model hash=99d93354bd65f17d859ac749a19befc9c8d2c5cc82f858b6b484434790001603 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" -ch6/0 draft role=translator req=fake-model actual=fake-model hash=abec9d62bee829a2dd5af959a3fccd72fd6ebd968f265389b706854f3432dab1 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch4/0 draft role=translator req=fake-model actual=fake-model hash=368928e6b6ebc6422df723a726b99152b999e293dbcd2b38ae1a977d3e3740be tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch4/0 edit role=editor req=fake-model actual=fake-model hash=685dbe9923e245ceef1f0237ee7fbc7558fcf93590a88aef1bef310f83aaa8c2 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch5/0 draft role=translator req=fake-model actual=fake-model hash=59e252c7d50d6e2e146ef3bbcb6910e98824dfa68b13dd564791a54f1506f426 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch5/0 edit role=editor req=fake-model actual=fake-model hash=2916b384d1b23b5e9439fb9d9baab299d16898bf1e1c2dbe25f52c434b2212c0 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch6/0 draft role=translator req=fake-model actual=fake-model hash=4bccc97fdab953a881aea64c33305d529d6f54692fd7048d455db543ae0af5bb tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" ch6/0 edit role=editor req=fake-model actual=fake-model hash= tm_hit=1 ok=0 finish="" degraded="sanitizer_defect" cost=0 tokens=0/0/0/0/0 err="" -ch7/0 draft role=translator req=fake-model actual=fake-model hash=2189c766693381a223938e3c8036b2c9bba19dc16e93d1a26671a5a397d577c0 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" -ch7/0 edit role=editor req=fake-model actual=fake-model hash=tm-sanitized-v1:8ef72b20eac059984209728d34e0ef742b2fab2a8aef57a37dd5d5cf55545f4f tm_hit=1 ok=0 finish="sanitized_export" degraded="sanitizer_stripped" cost=0 tokens=0/0/0/0/0 err="" -ch8/0 draft role=translator req=fake-model actual=fake-model hash=59118bacb3968e766e4c75d2ed770bc2a08d81231e4a01cf04e7c8fe7a722621 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" -ch8/0 edit role=editor req=fake-model actual=fake-model hash=tm-sanitized-v1:faf1ca5e8f561d183dad262f013b9820425201cdd0ac5682c54d20176f1f59d8 tm_hit=1 ok=0 finish="sanitized_export" degraded="sanitizer_stripped" cost=0 tokens=0/0/0/0/0 err="" +ch7/0 draft role=translator req=fake-model actual=fake-model hash=64cc7c596cfb048cf48b2202d7dea01f07e7f3536884c93d2fc5cb5719abed9d tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch7/0 edit role=editor req=fake-model actual=fake-model hash=tm-sanitized-v1:2bef56171e2a1a6dd901a283e6d4f9f1f2aca106735e2a5ff638fbe366433c0a tm_hit=1 ok=0 finish="sanitized_export" degraded="sanitizer_stripped" cost=0 tokens=0/0/0/0/0 err="" +ch8/0 draft role=translator req=fake-model actual=fake-model hash=bb2e0697351fc22258ff3f163d986a8ebff1bc6fff4ee5186b3cf3e579b222e6 tm_hit=1 ok=1 finish="stop" degraded="" cost=0 tokens=0/0/0/0/0 err="" +ch8/0 edit role=editor req=fake-model actual=fake-model hash=tm-sanitized-v1:d699a1e04616de214bf266a3e610ce060c5afc1fd640a537d19e49e8ad69c139 tm_hit=1 ok=0 finish="sanitized_export" degraded="sanitizer_stripped" cost=0 tokens=0/0/0/0/0 err="" -- retrieval_state -- ch1/0 snap_match=true exact=3 sticky=0 ambiguous=0 spoiler=1 evicted=0 postcheck_miss=1 style_flags=0 trust_gated=0 injected_ids=["紋章\u001f\u001f0\u001f0","鈴木\u001f\u001f0\u001f0","魔法学院\u001f\u001f0\u001f0"] postcheck_detail=[{"src":"紋章","dst":"герб","disp":"confirmed"}] style_detail= trust_gate_detail= diff --git a/backend/internal/pipeline/wave.go b/backend/internal/pipeline/wave.go new file mode 100644 index 0000000..3d5a0b4 --- /dev/null +++ b/backend/internal/pipeline/wave.go @@ -0,0 +1,42 @@ +package pipeline + +// wave.go: the W0 (deterministic, $0) pre-compute of the wave runner (WS1 §1б). This is the FIRST piece +// of the wave executor built ahead of the driver switch (R1 residual): the sticky-chain memory selection +// is a CROSS-CHUNK sequential dependency (a chunk's sticky_prev is the union of the prior chunks' exact +// matches in the same chapter), so the parallel waves cannot recompute it per-chunk — it must be +// precomputed once in W0. precomputeSticky replicates EXACTLY the sequential driver's inline computation +// (bookrun.go's per-chapter reset + memory.Select + sticky advance), and the sequential driver now +// CONSUMES it — so it is byte-identical by construction, and the golden test proves the injection bytes +// did not move. When the driver switches to waves (residual), the waves consume this same precompute. +// +// Select is a PURE deterministic function of (chunk, chapter, sticky_prev, budget, frozen-bank) with NO +// model output, so precomputing it reproduces byte-for-byte the injection the sequential runner rendered +// (parallelism does not change the bytes — the order is fixed and the input carries no model state). + +// precomputeSticky computes the per-chunk memory selection for EVERY chunk in one ordered W0 pass +// (WS1 §1б): the sticky window is reset at each chapter boundary (a new chapter is a scene change) and +// advanced by each chunk's exact matches, exactly as the sequential loop did inline. Returns a slice +// aligned index-for-index with `chunks`. A nil bank (report path / no glossary) yields zero-valued +// selections (no injection), so a caller can index it unconditionally. +func precomputeSticky(chunks []Chunk, mem *MemoryBank, budget int) []memorySelection { + out := make([]memorySelection, len(chunks)) + if mem == nil { + return out + } + var stickyWin []map[string]injectionDisposition + prevChapter := 0 + for i, ch := range chunks { + if ch.Chapter != prevChapter { + stickyWin = nil // chapter boundary: reset scene-inertia (A5) + prevChapter = ch.Chapter + } + sel := mem.Select(ch.Text, ch.Chapter, unionSticky(stickyWin), budget) + out[i] = sel + // Advance the sticky window with THIS chunk's exact matches (keep the last stickyDepth). + stickyWin = append(stickyWin, sel.activeIDs) + if len(stickyWin) > stickyDepth { + stickyWin = stickyWin[len(stickyWin)-stickyDepth:] + } + } + return out +} diff --git a/backend/internal/store/glossary.go b/backend/internal/store/glossary.go index a18c4d2..8ec82d0 100644 --- a/backend/internal/store/glossary.go +++ b/backend/internal/store/glossary.go @@ -79,6 +79,12 @@ type RetrievalState struct { // count is a loud seed-hygiene signal (a draft term nesting over an approved term to reconcile). NTrustGatedSuppress int TrustGateDetail string // JSON of the refused suppressor→protected pairs + // NBanknoteLines / BanknoteParseFail / BanknoteTruncated are the banknote channel's per-chunk + // telemetry (WS4 point 10): the accepted ⟦TM-BANK-v1⟧ line count and the parse-fail / truncation + // flags (0/1) of the translator draft. Zero for every channel-off / non-banknote chunk. + NBanknoteLines int + BanknoteParseFail int + BanknoteTruncated int } // ReplaceGlossary replaces a book's ENTIRE glossary (rows + aliases) with the given @@ -244,8 +250,9 @@ func (s *Store) UpsertRetrievalState(rs RetrievalState) error { book_id, chapter, chunk_idx, snapshot_id, n_exact_hits, n_sticky, n_ambiguous_flagged, n_spoiler_blocked, n_evicted, embedding_tier_used, n_postcheck_miss, postcheck_detail, injected_ids, - n_style_flags, style_detail, n_trust_gated_suppress, trust_gate_detail, updated_at - ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,datetime('now')) + n_style_flags, style_detail, n_trust_gated_suppress, trust_gate_detail, + n_banknote_lines, banknote_parse_fail, banknote_truncated, updated_at + ) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,datetime('now')) ON CONFLICT (book_id, chapter, chunk_idx) DO UPDATE SET snapshot_id = excluded.snapshot_id, n_exact_hits = excluded.n_exact_hits, @@ -261,11 +268,15 @@ func (s *Store) UpsertRetrievalState(rs RetrievalState) error { style_detail = excluded.style_detail, n_trust_gated_suppress = excluded.n_trust_gated_suppress, trust_gate_detail = excluded.trust_gate_detail, + n_banknote_lines = excluded.n_banknote_lines, + banknote_parse_fail = excluded.banknote_parse_fail, + banknote_truncated = excluded.banknote_truncated, updated_at = excluded.updated_at`, rs.BookID, rs.Chapter, rs.ChunkIdx, rs.SnapshotID, rs.NExactHits, rs.NSticky, rs.NAmbiguousFlagged, rs.NSpoilerBlocked, rs.NEvicted, rs.EmbeddingTierUsed, rs.NPostcheckMiss, rs.PostcheckDetail, rs.InjectedIDs, - rs.NStyleFlags, rs.StyleDetail, rs.NTrustGatedSuppress, rs.TrustGateDetail) + rs.NStyleFlags, rs.StyleDetail, rs.NTrustGatedSuppress, rs.TrustGateDetail, + rs.NBanknoteLines, rs.BanknoteParseFail, rs.BanknoteTruncated) return err } @@ -277,12 +288,14 @@ func (s *Store) GetRetrievalState(bookID string, chapter, chunkIdx int) (*Retrie err := s.r.QueryRowContext(ctx, ` SELECT snapshot_id, n_exact_hits, n_sticky, n_ambiguous_flagged, n_spoiler_blocked, n_evicted, embedding_tier_used, n_postcheck_miss, postcheck_detail, injected_ids, - n_style_flags, style_detail, n_trust_gated_suppress, trust_gate_detail + n_style_flags, style_detail, n_trust_gated_suppress, trust_gate_detail, + n_banknote_lines, banknote_parse_fail, banknote_truncated FROM retrieval_state WHERE book_id = ? AND chapter = ? AND chunk_idx = ?`, bookID, chapter, chunkIdx).Scan( &rs.SnapshotID, &rs.NExactHits, &rs.NSticky, &rs.NAmbiguousFlagged, &rs.NSpoilerBlocked, &rs.NEvicted, &rs.EmbeddingTierUsed, &rs.NPostcheckMiss, &rs.PostcheckDetail, &rs.InjectedIDs, - &rs.NStyleFlags, &rs.StyleDetail, &rs.NTrustGatedSuppress, &rs.TrustGateDetail) + &rs.NStyleFlags, &rs.StyleDetail, &rs.NTrustGatedSuppress, &rs.TrustGateDetail, + &rs.NBanknoteLines, &rs.BanknoteParseFail, &rs.BanknoteTruncated) if errors.Is(err, sql.ErrNoRows) { return nil, nil } @@ -298,7 +311,8 @@ func (s *Store) RetrievalStatesForBook(bookID string) ([]RetrievalState, error) return queryAll(s.r, ` SELECT chapter, chunk_idx, snapshot_id, n_exact_hits, n_sticky, n_ambiguous_flagged, n_spoiler_blocked, n_evicted, embedding_tier_used, n_postcheck_miss, postcheck_detail, injected_ids, - n_style_flags, style_detail, n_trust_gated_suppress, trust_gate_detail + n_style_flags, style_detail, n_trust_gated_suppress, trust_gate_detail, + n_banknote_lines, banknote_parse_fail, banknote_truncated FROM retrieval_state WHERE book_id = ? ORDER BY chapter, chunk_idx`, func(rows *sql.Rows) (RetrievalState, error) { @@ -306,7 +320,8 @@ func (s *Store) RetrievalStatesForBook(bookID string) ([]RetrievalState, error) err := rows.Scan(&rs.Chapter, &rs.ChunkIdx, &rs.SnapshotID, &rs.NExactHits, &rs.NSticky, &rs.NAmbiguousFlagged, &rs.NSpoilerBlocked, &rs.NEvicted, &rs.EmbeddingTierUsed, &rs.NPostcheckMiss, &rs.PostcheckDetail, &rs.InjectedIDs, - &rs.NStyleFlags, &rs.StyleDetail, &rs.NTrustGatedSuppress, &rs.TrustGateDetail) + &rs.NStyleFlags, &rs.StyleDetail, &rs.NTrustGatedSuppress, &rs.TrustGateDetail, + &rs.NBanknoteLines, &rs.BanknoteParseFail, &rs.BanknoteTruncated) return rs, err }, bookID) } diff --git a/backend/internal/store/migrate.go b/backend/internal/store/migrate.go index 570d6ed..0360840 100644 --- a/backend/internal/store/migrate.go +++ b/backend/internal/store/migrate.go @@ -282,6 +282,16 @@ var migrations = []string{ ALTER TABLE retrieval_state ADD COLUMN n_trust_gated_suppress INTEGER NOT NULL DEFAULT 0; ALTER TABLE retrieval_state ADD COLUMN trust_gate_detail TEXT NOT NULL DEFAULT ''; `, + // v9 (WS4): the banknote channel's per-chunk telemetry (integration point 10) — the accepted line + // count and the parse-fail / truncation flags of the translator draft's ⟦TM-BANK-v1⟧ block, riding + // the same per-chunk observability row as the glossary post-check and the cheap style flaggers. + // Recomputed deterministically each run (self-heals on resume), NOT wire-load-bearing (observability, + // never a disposition). Zero for every channel-off / non-banknote chunk (the common case). + ` + ALTER TABLE retrieval_state ADD COLUMN n_banknote_lines INTEGER NOT NULL DEFAULT 0; + ALTER TABLE retrieval_state ADD COLUMN banknote_parse_fail INTEGER NOT NULL DEFAULT 0; + ALTER TABLE retrieval_state ADD COLUMN banknote_truncated INTEGER NOT NULL DEFAULT 0; + `, } // DESIGN NOTE (D21.10 — reserved memory-bank v2 record types; Ф2, NOT a migration, NOT code). diff --git a/docs/BACKEND_PLAN11_SESSION_PROMPT.md b/docs/BACKEND_PLAN11_SESSION_PROMPT.md index a4313d3..bf11630 100644 --- a/docs/BACKEND_PLAN11_SESSION_PROMPT.md +++ b/docs/BACKEND_PLAN11_SESSION_PROMPT.md @@ -1,7 +1,19 @@ -> **⚠ АКТУАЛИЗИРОВАН ПОД ПРОДОЛЖЕНИЕ (19.07, пост-D39.13; исходная версия — в git-истории).** Сессия №1 сдала -> Block A + ядра — ПРИНЯТО И ЗАЛЕНДЕНО (`85f5b9e`): WS2 ✅ · WS1a/b/c-инфра ✅ · WS4-core ✅ · WS6-конфиг-тай ✅. -> **Эта версия — стартовая для СЕССИИ-ПРОДОЛЖЕНИЯ (резидуал R1–R5).** Сданное — под тестами, НЕ пере-строить; -> дизайны резидуала — `backend/PACK11_REPORT.md` §Резидуал (авторитет наравне с планом; исполнять, не пере-изобретать). +> **⚠ АКТУАЛИЗИРОВАН ПОД R1-ПРОДОЛЖЕНИЕ (19.07, пост-D39.14; предыдущие версии — в git-истории).** Сессия №1 +> сдала Block A (`85f5b9e`, D39.13); сессия №2 сдала R2–R5 + R1-аддитив (D39.14). **Осталось: R1 драйвер-свитч + +> 3-пунктовый фикс-лист рубежа-2 (ниже).** Дизайны — `backend/PACK11_CONT_REPORT.md §R1` + `PACK11_REPORT.md §Резидуал` +> (авторитет наравне с планом). Сданное — под тестами, НЕ пере-строить. +> +> **ФИКС-ЛИСТ рубежа-2 (D39.14; все латентны — банкнота off по дефолту, эмиссия за R1 — лендинг не блокировали; +> закрыть В ЭТОЙ сессии, они трогают ровно R1-файлы):** +> - **FL-1 (MINOR):** свернуть `bankMaxLines`/`bankTokenBudget` в `banknoteSnap` (`snapshot.go`) — сейчас +> `max_tokens += bankTokenBudget` wire-affecting (∈RequestHash), но не свёрнут → правка конста без бампа +> `bankParserVersion` = тихий re-bill книги на резюме (loudness дисциплиной, не механизмом). +> - **FL-2 (MINOR, observability):** `resumeFromChunkStatus` (`resume.go:22`) не ставит `BankFlags` → банкнота- +> телеметрия (`n_banknote_lines/parse_fail/truncated`) обнуляется на резюме готовых чанков. Re-derive из сырого +> чекпоинта, ИЛИ не обнулять эти колонки на резюме, ИЛИ поправить ложный коммент «self-heals on resume». +> - **FL-3 (PARTIAL, порт-fidelity):** `miner_emit.go:68` идёт по полному `mr.ranked`; референс капит `a3[:200]`/ +> `[:30]` ДО фильтров (`eval/exp16/{alias.py:165, emit_owner_sheets.py:143,162,169}`) → карта подписи шумнее +> референса. Добавить top-N кап перед emission-фильтрами. **Закрыть ДО живого W1.5** (владелец подписывает карту). # Промт: сессия БЭКЕНД — единый пак по плану 11 (волновой раннер · чанкер/единицы · банк W1.5 · банкнота · чекеры · армы), 2026-07-19 diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index 9bee5e8..56ac937 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -9,6 +9,10 @@ > - **Ждём от владельца:** запуск exp15 (бюджет ≤$15 подтверждён) · research/20 §E: мини-голд алиасов (~20–30 мин) / бюджет ≤$5 полигон-стадии / политика канона / жанр-паки / реплика ja→ru · чтение пере-прогона (после очереди) · старые висящие: планка запуска (лучше-фана/гибрид/издательский) · билингв-якорь пилота (D25.9-Q1) · контаминация пилот-корпуса (D27.4) · publishable/waiver (D25.1) · FN-bound L3 (D25.4) · юр-пакет · провенанс 12-*-доков. > - Архивы хроники: `archive/PROGRESS-2026-07-04-10.md` (D31) · `archive/PROGRESS-2026-07-10-13.md` (D39.6-гигиена). Записи ниже — живая эра D39. +## Оркестратор №7 — пак-11 continuation принят и залендён (D39.14), рубеж-2-долг погашен, 19.07 (четвёртая фаза) + +Continuation-сессия сдала R2–R5 + R1-аддитив (санкц. порядок вариант-1). **Залендено:** R2 Go-майнер (паритет EXACT: 13618, 古月:22, recall 0.9649) · R3 банкнота 12-точек (на посл. раннере, переезжают в волны как есть) · R4 DC1/DC2/DC6+DC3-gender+`export --pairs` (2 golden re-capture masked-чист) · R5 2 арм-конфига · R1-аддитив (snapshotIDForWave + W0-precompute байт-идентично). R1 драйвер-свитч (W1/W1.5/W2 + структурный golden-рерайт) — санкц. резидуал. **Приёмка исполнением:** build/vet/race сам, ~40 пинов свежим `-count=1` (майнер-паритет TM_MINER_PARITY EXACT), независимая masked-golden репликация (0 изменений final_text/disposition), построчное чтение деньги-пути (settle пишет raw, срез после durable settle — деньги нетронуты). **Долг рубежа-2 (D39.13) ПОГАШЕН:** 8-агентный адверсариал по обоим кускам, 5 линз refute-by-default, $0 — **3 находки (все MINOR/PARTIAL, 0 refuted); все CRITICAL/MAJOR-линзы пусты** (деньги/wire/порт/резюм/base-enriched/residual — clean bill). Фикс-лист FL-1 (bankTokenBudget не свёрнут в banknoteSnap) · FL-2 (banknote-телеметрия обнуляется на резюме, observability) · FL-3 (miner emission без top-N капа референса) — все латентны (банкнота off по дефолту, эмиссия за R1), лендинг не блокируют → R1-продолжение (промт активен, фикс-лист вписан). §0.1-заметка владельцу: DC-данные = код-консты за isRuTarget-гейтом (сеам защищает; config-extraction — открытый вопрос, увязать с ja→ru). Очередь: R1-продолжение → приёмка → resnapshot → пере-прогон → чтение. + ## Оркестратор №7 — пак-11 Block A принят и залендён (D39.13), 19.07 (третья фаза сессии) Бэкенд-сессия сдала Block A + ядра с ЧЕСТНЫМ резидуалом (мандат «резидуал > сломанный суит»): WS2 целиком (output-бюджет чанкера, edit-единицы grouping-of-whole-chunks [девиация ратифицирована: 37 единиц / 0 straddle воспроизведены, гарантия W2-реконструкции по построению], src→dst-блок редактора = закрытие D30.1, `render-format-version`, golden re-capture) · WS1a eager-clients (гонка закрыта) · WS1b rate-guard (`ratelimit.go`) · WS1c base/enriched `MemoryVersion`-split (enriched байт-идентичен — снапшот не двинулся) · WS4-core банкнота-парсер (байт-верный порт). **Приёмка — верификация исполнением:** build/vet/race сам, 27 пинов свежим `-count=1` вербозно, независимая masked-golden репликация (все категории объяснимы), построчное чтение несущих диффов. Сессионный 9-агентный чекпойнт: 3 confirmed пофикшены / 3 refuted. **Долг: мульти-агентный рубеж-2 приёмки не гонялся (лимит-режим) — гасится при приёмке продолжения (прецедент D39.2).** Резидуал R1–R5 с точными дизайнами — `backend/PACK11_REPORT.md`; промт пака АКТИВЕН со статус-баннером. Записка продолжению-сессии: стартовать по промту + отчёту, сданное не пере-строить. diff --git a/docs/architecture/05-decisions-log.md b/docs/architecture/05-decisions-log.md index d616dbc..be868b4 100644 --- a/docs/architecture/05-decisions-log.md +++ b/docs/architecture/05-decisions-log.md @@ -3,7 +3,7 @@ > **⟶ КАРТА АКТУАЛЬНОСТИ (ревизия D31, продлена до D38.2 [12.07]; исторические записи ниже НЕ переписываются — дисциплина D23.3).** Читая контракт целиком, держи под рукой, что чем перекрыто: > - **Полностью superseded:** **D1 (моно-редактор) → D17 → D30.1 (редактор БИЛИНГВ)** · D9 (ja-приёмка) → D18 (蛊真人 zh→ru; ja — второй прогон) · D17 → D30.1 · скобка D19.4 «ключ без data-sharing» и D20.3 «чистый ключ = блокер пилота» → **D27** (единый ключ С data-sharing, отключение перед продом) · exp04-дефолт «editor grok-4.3» (D3) → D30.1 (grok reasoning-off из редакторских ролей СНЯТ — no-op; кандидат glm-5-билингв; gemini — премиум-эскалация только за санитайзером D30.3). > - **Частично амендировано:** D3 (канал B → D14.1/D19.1 + оговорки D22.5/D22.6; апекс-слаг `-preview` — D22.4; draft под вопросом exp13 — D30.6) · D6 («off/minimal для draft/edit» эродирован: draft thinking-ON принудительно, editor-off снят D30.1; живы per-роль принцип и D6.2-буфер) · D10 (+D24.2 alias-слепое пятно, истинная консистентность ≈99.5%) · D11-числа → exp08 → **D30.4** (флип D1 = +15–25%, ~$0.85/ранобэ; «$1.5–2» = неподписанная опция Б) · D12 (+D24.3 флоры max_tokens; +D29.1в rollup — амендмент вердикт-правила) · D13.1-арм из решающего → ПОДТВЕРЖДАЮЩИЙ (D30.1); +D25.3 prefix-анализ судьи · D14.2 закрыт D19.1/D22.4; D14.4 закрыт D22.1 · D15.3 исполнен; спека D15.2: v2→v3→v3.1 (D22.2), реализация = текущий пакет (D30.9) · D24.4 +D28.1 (precision/recall-трейдофф; hard-gate требует пере-замера) · порядок D24.5 отложен D26.1 (этап B — после читаемых 25 глав). -> - **Живое ядро без изменений:** D2 (+новый гейт-класс D30.3), D4, D5, D7, D8, D10-механизм, D12-таксономия отказов, D16, D18, D19.1–19.2, D21 (Ф2-механизмы voice/address/reveal), D22.5–22.7, D23 (golden = инвариант №8), D24–D39.13 — действующая голова контракта (**D39.13 = пак-11 Block A ПРИНЯТ и залендён [WS2 целиком (вкл. src→dst-блок редактора = закрытие D30.1) + WS1a/b/c-инфра + WS4-ядро; 2 девиации ратифицированы; резидуал R1–R5 в PACK11_REPORT; промт остаётся активным; рубеж-2 приёмки — долг]**; **D39.12 = план `architecture/11-implementation-plan.md` РАТИФИЦИРОВАН [три рубежа: ревью-2 F1–F11 → 7-агентный адверсариал → подтверждающий пас; пер-волновой снапшот + инвариант секвенирования = «переоплата ОДНА»; EditCeilingOut=3200; лемматизатор дефолт B; 4 гейтнутых арма]; бэкенд-пак ЕДИНОЙ сессией — `BACKEND_PLAN11_SESSION_PROMPT.md`**; **D39.11 = директива владельца: стройка ТОЛЬКО после дизайн-синтез сессии с верифицированными алгоритмами; хендофф №5**; **D39.10 = exp16: WHICH подтверждена [V-C recall 0.97, $0] / WHAT→банкнота+Палладий [canon-recovery <70%], 9B-споттер не нужен, Z1→облако, ре-аудит exp14b чист [errata не требуется]; исследовательская программа ЗАВЕРШЕНА → бэкенд-пак слоёв 1+4**; **D39.9 = Q4a: верность НЕ на translate-стадии [flash уже верен, pro ×3.67 не ратифицирован], смысл ломает glm-РЕДАКТОР [свап-арм приоритетен], DET-пол 0.261; exp14b-правила дефектны → $0-ре-аудит в exp16**; **D39.8 = слепое чтение: планка не пройдена никем, дефекты распределены [читатель подтвердил floor-null], битва качества → дефект-классы [时辰-юниты, числа, gender-enforce, стих, регистр]**; **D39.7 = exp15 залендён REV.2: «граница вредит» отозван [артефакт окна судьи], все когезия-эффекты под floor 0.126, «sequential нигде не лучше» LOO-stable [не ратифицированный зелёный свет], фертильность 1.20/0.39, $12.79/$15; гейт = слепое чтение владельца**; **D39.6 = research/20 банк-майнинг залендён [детектор-лестница V-A/B/C, алиас-ярусы, banknote-v1 с 12 гейт-точками, §B5 плагины P1–P6, §D-пре-рег]; полигон-стадия после exp15**; **D39.5 = пак-1.5+F1–F9 залендены [tmctl export с manifest/drift-гардами · fold-first санитайзер · bounded глосс · пример zh-ru]; экстракция полигона впредь через tmctl export**; **D39.4 = адверсариал-долг ПОГАШЕН [8 осей: 0 CRIT/1 HIGH/6 MED], пак-1.5 придержан под фикс-лист F1–F9 в §T4**; **D39.3 = пакет банк-майнинга W1.5 [диспозиция H15/V4-п4] ратифицирован, research/20-промт с HOLD до фриза exp15**; **D39.2 = трек-A пак-1 залендён `d3f6b34`: trust-gated suppressor [терм-дрейф закрыт в КОДЕ] + export-contract нормализация + pair-сеам + реестры + quality-report; адверсариал-долг за 529**; **D39.1 = research/19 залендён: конфликт exp14↔research/18 разрешён [эмиссия≠окно≠связность], Go-спека чанкера+когезии, волновая архитектура W0–W3 [вводная владельца: параллелизм], пре-рег эмпирики Q0–Q5; оркестратор-ревью снято владельцем**; **D39 = АРХ-РЕСЕТ: синк-аудит сломанного телефона [65 находок, 0 refuted, `08-sync-audit-ledger.md`] + целевая 7-слойная архитектура [`09-target-architecture.md`] + фазовый план [трек A build-now ∥ трек B ресёрч, пере-прогон после]; формализует фантомный «D35.7» как цель слоя 1; терм-дрейф код-корень жив [сид-воркэраунд]; редактор 4-мандата-full-regen = причина инверсий**; D35 = пивот качество-первым; D36 = приёмка research/18; D37 = приёмка exp14: претензия-1=промпт-рычаг; D38 = приёмка exp14b: «только gpt-5.4» ОПРОВЕРГНУТО [mistral/deepseek-pro чинят], фронтир в дефолт НЕ нужен, корень терм-дрейфа = статус-draft сида; **D38.1 = слепой h2h P1a↔F-disc залендён [ChatGPT «фронтир сильнее» / Claude «near-parity» расходятся, СХОДЯТСЯ: дефекты = инфра] → РАЗВОРОТ «мерить→строить», выдан BACKEND_INFRA_PACK; D38.2 = консолидация/док-гигиена: спент exp-скрипты в `eval/exp12|13|14|14b/`, статус-доки+.puml актуализированы; D38.3 = инфра-пак читаемости залендён (санитайзер-классы strip+export + число/omission-гард), верифицирован исполнением [build/vet/test зелёные, 3 span-дефекта запинены]; D38.4 = аудит полноты exp14/14b: чэнъюй-рычаг забыт near-term→дописан в дискурс-промпт; inversion-гард D37 §2г не строился (число/omission ≠ полярность)→editor-swap = защита от инверсий (приоритет↑); нарезка = re-run-арм; гард-тумблер вкл. при resnapshot; D38.5 = компоненты resnapshot залендены+верифицированы [reseed-сид eb409f2 · дискурс+чэнъюй editor v3 0ac5f7a с few_shot-тумблером] → следующий шаг единый resnapshot + пере-прогон**). +> - **Живое ядро без изменений:** D2 (+новый гейт-класс D30.3), D4, D5, D7, D8, D10-механизм, D12-таксономия отказов, D16, D18, D19.1–19.2, D21 (Ф2-механизмы voice/address/reveal), D22.5–22.7, D23 (golden = инвариант №8), D24–D39.14 — действующая голова контракта (**D39.14 = пак-11 continuation ПРИНЯТ и залендён [R2 майнер паритет-EXACT · R3 банкнота 12-точек · R4 DC-чекеры+export--pairs · R5 арм-конфиги · R1-аддитив; рубеж-2 D39.13-долга погашен: 3 MINOR/PARTIAL-находки [FL-1 bankTokenBudget-фолд · FL-2 banknote-телеметрия-резюм · FL-3 emission-top-N-кап], все латентны → R1-продолжение; R1 драйвер-свитч резидуал; clean bill по деньгам/wire/порту]**; **D39.13 = пак-11 Block A ПРИНЯТ и залендён [WS2 целиком (вкл. src→dst-блок редактора = закрытие D30.1) + WS1a/b/c-инфра + WS4-ядро; 2 девиации ратифицированы; резидуал R1–R5 в PACK11_REPORT; промт остаётся активным; рубеж-2 приёмки — долг]**; **D39.12 = план `architecture/11-implementation-plan.md` РАТИФИЦИРОВАН [три рубежа: ревью-2 F1–F11 → 7-агентный адверсариал → подтверждающий пас; пер-волновой снапшот + инвариант секвенирования = «переоплата ОДНА»; EditCeilingOut=3200; лемматизатор дефолт B; 4 гейтнутых арма]; бэкенд-пак ЕДИНОЙ сессией — `BACKEND_PLAN11_SESSION_PROMPT.md`**; **D39.11 = директива владельца: стройка ТОЛЬКО после дизайн-синтез сессии с верифицированными алгоритмами; хендофф №5**; **D39.10 = exp16: WHICH подтверждена [V-C recall 0.97, $0] / WHAT→банкнота+Палладий [canon-recovery <70%], 9B-споттер не нужен, Z1→облако, ре-аудит exp14b чист [errata не требуется]; исследовательская программа ЗАВЕРШЕНА → бэкенд-пак слоёв 1+4**; **D39.9 = Q4a: верность НЕ на translate-стадии [flash уже верен, pro ×3.67 не ратифицирован], смысл ломает glm-РЕДАКТОР [свап-арм приоритетен], DET-пол 0.261; exp14b-правила дефектны → $0-ре-аудит в exp16**; **D39.8 = слепое чтение: планка не пройдена никем, дефекты распределены [читатель подтвердил floor-null], битва качества → дефект-классы [时辰-юниты, числа, gender-enforce, стих, регистр]**; **D39.7 = exp15 залендён REV.2: «граница вредит» отозван [артефакт окна судьи], все когезия-эффекты под floor 0.126, «sequential нигде не лучше» LOO-stable [не ратифицированный зелёный свет], фертильность 1.20/0.39, $12.79/$15; гейт = слепое чтение владельца**; **D39.6 = research/20 банк-майнинг залендён [детектор-лестница V-A/B/C, алиас-ярусы, banknote-v1 с 12 гейт-точками, §B5 плагины P1–P6, §D-пре-рег]; полигон-стадия после exp15**; **D39.5 = пак-1.5+F1–F9 залендены [tmctl export с manifest/drift-гардами · fold-first санитайзер · bounded глосс · пример zh-ru]; экстракция полигона впредь через tmctl export**; **D39.4 = адверсариал-долг ПОГАШЕН [8 осей: 0 CRIT/1 HIGH/6 MED], пак-1.5 придержан под фикс-лист F1–F9 в §T4**; **D39.3 = пакет банк-майнинга W1.5 [диспозиция H15/V4-п4] ратифицирован, research/20-промт с HOLD до фриза exp15**; **D39.2 = трек-A пак-1 залендён `d3f6b34`: trust-gated suppressor [терм-дрейф закрыт в КОДЕ] + export-contract нормализация + pair-сеам + реестры + quality-report; адверсариал-долг за 529**; **D39.1 = research/19 залендён: конфликт exp14↔research/18 разрешён [эмиссия≠окно≠связность], Go-спека чанкера+когезии, волновая архитектура W0–W3 [вводная владельца: параллелизм], пре-рег эмпирики Q0–Q5; оркестратор-ревью снято владельцем**; **D39 = АРХ-РЕСЕТ: синк-аудит сломанного телефона [65 находок, 0 refuted, `08-sync-audit-ledger.md`] + целевая 7-слойная архитектура [`09-target-architecture.md`] + фазовый план [трек A build-now ∥ трек B ресёрч, пере-прогон после]; формализует фантомный «D35.7» как цель слоя 1; терм-дрейф код-корень жив [сид-воркэраунд]; редактор 4-мандата-full-regen = причина инверсий**; D35 = пивот качество-первым; D36 = приёмка research/18; D37 = приёмка exp14: претензия-1=промпт-рычаг; D38 = приёмка exp14b: «только gpt-5.4» ОПРОВЕРГНУТО [mistral/deepseek-pro чинят], фронтир в дефолт НЕ нужен, корень терм-дрейфа = статус-draft сида; **D38.1 = слепой h2h P1a↔F-disc залендён [ChatGPT «фронтир сильнее» / Claude «near-parity» расходятся, СХОДЯТСЯ: дефекты = инфра] → РАЗВОРОТ «мерить→строить», выдан BACKEND_INFRA_PACK; D38.2 = консолидация/док-гигиена: спент exp-скрипты в `eval/exp12|13|14|14b/`, статус-доки+.puml актуализированы; D38.3 = инфра-пак читаемости залендён (санитайзер-классы strip+export + число/omission-гард), верифицирован исполнением [build/vet/test зелёные, 3 span-дефекта запинены]; D38.4 = аудит полноты exp14/14b: чэнъюй-рычаг забыт near-term→дописан в дискурс-промпт; inversion-гард D37 §2г не строился (число/omission ≠ полярность)→editor-swap = защита от инверсий (приоритет↑); нарезка = re-run-арм; гард-тумблер вкл. при resnapshot; D38.5 = компоненты resnapshot залендены+верифицированы [reseed-сид eb409f2 · дискурс+чэнъюй editor v3 0ac5f7a с few_shot-тумблером] → следующий шаг единый resnapshot + пере-прогон**). Ответ на вопросы бэкенд-сессии (PROGRESS §«Вопросы от бэкенда», §«[НУЖНО РЕШЕНИЕ] перед Фазой 1» — с ревизии D31 в `archive/PROGRESS-2026-07-04-10.md`) и полигона (think-режим). Каждое решение прошло адверсариальную панель из 3 критиков (research / экономика / исполнимость в коде Фазы 0); ни одно не отклонено, все уточнены. Это **контракт Фазы 1** — бэкенд исполняет отсюда; при конфликте с буквой 01-decisions/02-mvp-plan — источник истины здесь (потом сольём в основные доки). @@ -550,6 +550,19 @@ exp14b (D37-мандат) прогнал батарею DET-смысл-трап **ИТОГ: все 3 компонента resnapshot ГОТОВЫ** — дискурс+чэнъюй editor (v3) · reseed-сид · инфра-пак (D38.3). **Следующий шаг = единый resnapshot** (D30.9, одна переоплата книги) + пере-прогон 3–10 глав: конфиг = glm=P1a-few-shot (дефолт) + армы {mistral few-shot · deepseek-pro `few_shot:false` = инверсия-защита D38.4 · крупная edit-единица = нарезка-арм D38.4} + `RegressionGuard` ON (D38.4). ПАРНО с чтением владельца (интерим-планка 2 претензии, D35). Крупная edit-единица — НЕ реализована (нарезка-арм, не промпт-порт). **[⟶ ПРИОСТАНОВЛЕНО D39: владелец остановил раунд для арх-ресета; пере-прогон гейтится треком B, см. D39.]** +## D39.14 — Приёмка пака-11 continuation (R2–R5 + R1-аддитив) ЗАЛЕНДЕНА; рубеж-2 D39.13-долга погашен: clean bill по несущим осям, 3-пунктовый MINOR-фикс-лист → R1-продолжение (19.07, оркестратор №7). ✅ + +Бэкенд-сессия сдала continuation по санкционированному порядку (вариант 1: R2→R3→R4→R5, R1-аддитив, драйвер-свитч честным резидуалом). **Залендено** (`git`): **R2 WS3 Go-майнер** (порт frozen exp16, дефолт-B; паритет EXACT — 13618-СЕТ, катастроф {方源:0,蛊:1,蛊师:2,古月:22}, recall 0.9649; алиас-ярус-1, `MinedDeltaYAML`, `Source:mined`-путь, `tmctl seed-lint`) · **R3 WS4 банкнота 12 точек** (срез-до-classify fresh+resume, `tm-banknote-v1:` derived + re-point после эскалации, `banknoteSnap` omitempty, store v9-телеметрия, finish=stop-гейт) · **R4 WS5** (`export --pairs`, DC1/DC2/DC6 байт-верны референсу, DC3 gender-инъекция, 2 golden re-capture masked-чист) · **R5 WS6** (2 арм-конфига mistral/deepseek-pro + `echo_mine_test` под варианты + rate-guard связка) · **R1-аддитив** (`snapshotIDForWave` «переоплата ОДНА» + W0 sticky-precompute байт-идентично — golden доказал). **R1 драйвер-свитч (W1/W1.5/W2 dispatch + структурный golden-рерайт) — резидуал** (санкционированный исход; точный 6-шаговый дизайн `PACK11_CONT_REPORT.md §R1`). + +**Приёмка — ПРЯМАЯ верификация исполнением:** `go build/vet/test -race` зелёные (сам); ~40 пинов свежим `-count=1` вербозно (майнер-паритет `TM_MINER_PARITY=1` EXACT, банкнота 12-точек, DC-чекеры, seed-lint, snapshotIDForWave, алиас-правила); **независимая masked-golden репликация** (18 строк, все объяснимы: render-format v1→v2 + cheapgate v2→v3 + gender-note в src→dst-блоке; **0 изменений final_text/disposition**); построчное чтение несущих диффов (stagerun деньги-путь: settle пишет `ResponseText: resp.Text` RAW, applyBanknote срез ПОСЛЕ durable settle, чекпоинт хранит raw, резюм re-slice детерминирован — деньги нетронуты; re-point зеркалит sanitizer-прецедент, стоит после резолва эскалации; W0-precompute байт-идентичен по построению; base≠enriched домен-сепарированы). + +**Погашение долга рубежа-2 (D39.13):** мульти-агентный адверсариал по ОБОИМ кускам пака (Block A + continuation), 5 линз × refute-by-default, 8 агентов, $0 к провайдерам. **Итог: 3 находки, все MINOR/PARTIAL, 0 refuted; ВСЕ CRITICAL/MAJOR-линзы пусты** (деньги/durability · wire-детерминизм · порт майнера [инварианты EXACT] · banknote-резюм-в-редактор · инвариант base/enriched · честность residual — clean bill). Спот-верифицировал каждую находку сам против кода+референса. **Фикс-лист (→ R1-продолжение, все латентны/observability — лендинг НЕ блокируют; банкнота выключена по дефолту `BanknoteGate.Enabled=false`, майнер-эмиссия ещё не в живом прогоне за R1):** +- **FL-1 (MINOR, F1):** `bankTokenBudget` (144, wire-affecting via max_tokens∈RequestHash) НЕ свёрнут в `banknoteSnap{enabled,parser_version}` → правка конста `bankMaxLines` без бампа parser_version = тихий re-bill книги на резюме (loudness дисциплиной, не механизмом — единственный такой пропуск; снапшот фолдит MaxOutputRatio/MinMaxTokens/etc). Фикс: свернуть `bankMaxLines`/бюджет в `banknoteSnap`. +- **FL-2 (MINOR, F2, observability-only):** `resumeFromChunkStatus` (`resume.go:22`) не репопулирует `BankFlags` (ставится только fresh-путём `stagerun.go:234`) → `persistRetrievalState` обнуляет `n_banknote_lines/parse_fail/truncated` на КАЖДОМ резюме готовых чанков; коммент «self-heals on resume» ложен для этих 3 колонок (сырой блок — в непрочитанном attempt-чекпоинте, derived хранит очищенный). Фикс: re-derive из сырого чекпоинта ИЛИ не обнулять эти колонки на резюме ИЛИ поправить коммент. +- **FL-3 (PARTIAL, F3, порт-fidelity):** `miner_emit.go:68` идёт по полному `mr.ranked`, референс капит `a3[:200]` (alias-пул) / `[:30]` (precision-лист) ДО фильтров (`alias.py:165`, `emit_owner_sheets.py:143/162/169`) → эмитируемая сид-дельта/карта подписи шумнее референса (НЕ затрагивает WHICH-инварианты SET/recall/катастроф — только объём подписываемого владельцем). Фикс: добавить top-N кап перед emission-фильтрами. Гасить ДО живого W1.5-прогона (владелец подписывает эту карту). + +**Ратифицированные девиации (обе с эскалация-нотой, приняты):** (1) `banknoteSnap` omitempty-указатель (не безусловное `{enabled:false}` зеркало sanitizerSnap) — добавление опц. фичи не инвалидирует banknote-OFF книги/golden, включение/бамп-парсера всё равно громкий сдвиг (несущая loudness §4а-п6 цела); (2) DC-пакет = код-консты за `isRuTarget`-гейтом (конфиг-форма — резидуал; §0.1-сеам защищает от неверного на новой паре, extraction-в-конфиг честно не доделан — открытый вопрос владельцу, увязать с ja→ru-репликой). **Промт `BACKEND_PLAN11_SESSION_PROMPT.md` остаётся АКТИВНЫМ** (R1 драйвер-свитч + FL-1/2/3); фикс-лист вписан в него. Очередь: R1-продолжение (драйвер-свитч + фикс-лист) → приёмка → единый resnapshot → пере-прогон 3–10 глав → чтение. + ## D39 — Арх-ресет: синк-аудит «сломанного телефона» + целевая 7-слойная архитектура + фазовый план (13.07, оркестратор №6). ✅ Владелец остановил build-раунд: тактическое латание вместо архитектуры с первых принципов + «сломанный телефон» полигон(ресёрч)→оркестратор(выводы)→бэкенд(реализация) (триггер — чуть не потеряли «умное чанкование»). Хендофф `ORCHESTRATOR_ARCH_RESET_PROMPT.md` (5 концернов). **Проведён синк-аудит** (многоагентный воркфлоу, 42 агента, $0/только-чтение, адверсариальная верификация refute-by-default): **65 находок, 0 REFUTED, 25 CONFIRMED** — `docs/architecture/08-sync-audit-ledger.md`. Целевая архитектура и дорожная карта — `docs/architecture/09-target-architecture.md`.